Event Location: NSH 1305

Abstract: Recognizing and reasoning about the objects found in an image is one of the key problems in computer vision. This thesis is based on the idea that in order to understand a novel object, it is often not enough to recognize the object category it belongs to (i.e., answering “What is this?”). We argue that a more meaningful interpretation can be obtained by linking the input object with a similar representation in memory (i.e., asking “What is this like?”). We present a memory-based system for recognizing and interpreting objects in images by learning how to create visual associations between a large database of object exemplars and an input image. These visual associations can then be used to predict properties of the novel object which cannot be deduced solely from category membership (e.g., which way is it facing? what is its segmentation? is there a person riding it?).

In this thesis, we build the Visual Memex, a vast graph defined over exemplars which encodes visual associations, contextual associations, as well as any meta-data (such as segmentation) accompanying each exemplar. Our large-scale object recognition experiments suggest that the Visual Memex performs competitively on the standard PASCAL VOC object category detection task as well as enables a host of prediction/interpretation tasks which are not well-handled by traditional category-based systems.

Committee:Alexei A. Efros, Chair

Martial Hebert

Takeo Kanade

Pietro Perona, California Institute of Technology