Abstract:  This talk will discuss the massive shift that has come about in the vision and ML community as a result of the large pre-trained language and language and vision models such as Flamingo, GPT-4, and other models. We begin by looking at the work on knowledge-based systems in CV and robotics before the large model revolution and discuss the impact it had. This impact can be broken down into three areas in which world knowledge should be studied in the context of these new models: evaluation, harnessing large models, and building outside knowledge. First, evaluating world knowledge is even more important as the large model revolution gives more easy access to world knowledge. Next, we discuss recent work in harnessing models such as Flamingo and Chinchilla for visual and procedural knowledge. Finally, the talk discusses how, by focusing on knowledge acquisition as an agent-centric problem, we can make developments in retrieving and collecting world knowledge.

BIO: Kenneth Marino is a Research Scientist at Google DeepMind in NYC, focusing on improving knowledge-based systems such as retrieval and information extraction as well as embodied reasoning with language. He graduated in 2021 from Carnegie Mellon University advised by Abhinav Gupta, where his thesis focused on incorporating knowledge into embodied systems. He has an adjunct appointment at Columbia University where he teaches a class focused on the impact of datasets on machine learning and how to collect good datasets. He received his undergraduate degree from the Georgia Institute of Technology where he studied Computer Engineering and Computer Science.

Sponsored in part by:   Meta Reality Labs Pittsburgh