Abstract:
Panoptic segmentation is a recent problem in computer vision that attempts to classify each pixel in an image according to its semantic and instance label (accomplishing both semantic segmentation and instance segmentation respectively). Most existing panoptic and instance segmentation methods run a detection-first pipeline, where a bounding box is placed around an object and a mask is produced for that object. These methods have known failure modes (e.g – when an object’s shape is such that its center cannot identify it, or when multiple objects of the same type are clustered together). We propose to reduce these failure modes by jointly reasoning about objects through direct mask prediction of multiple instances.

Similarly, we aim to improve anomaly detection by considering methods that are able to jointly reason about relationships between objects in the scene (or objects and their surroundings). Anomaly detection is an understudied problem in the computer vision community but is becoming increasingly important as we deploy systems and hope to report (and then handle) the most rare or unexpected scenarios. While the vision-based anomaly detection community is primarily advancing through the use of more sophisticated features, it is challenging to incorporate methods from traditional supervised tasks because the features cannot be explicitly learned through supervision. In addition, anomaly detection datasets do not have held-out test sets, making it difficult to evaluate the generalization power of existing methods. We demonstrate a method that jointly reasons about spatiotemporal events in a video without any training data and discuss a simple way to remove ‘distracting features’ in a scene. We also propose to introduce a new dataset that addresses the issue of a held-out test set and a baseline that considers joint interactions between objects in the scene.

More Information

Thesis Committee Members:
Martial Hebert, Chair
J. Andrew Bagnell
David Held
Barnabas Poczos
Karteek Alahari, Inria Grenoble – Rhône-Alpes