Abstract:
Computer vision models have proven to be tremendously capable of recognizing and detecting several classes and objects. They succeed in classes widely ranging in type and scale from humans to cans to pens. However, the best performing classes have abundant examples in large-scale datasets today. In unbalanced datasets, where some categories are seen in large frequency while others are rarely seen, models struggle to perform well particularly on less represented classes. While several current methods have focused on improving performance for unbalanced dataset benchmarks, performance for rarer classes still lag far behind performance for frequent classes. Furthermore, the scarcity of large-scale unbalanced dataset benchmarks exacerbates the issue. This thesis focuses on improving models for unbalanced datasets through simple yet effective techniques and exploring datasets in hopes of growing and analyzing unbalanced datasets.

In the first part of this thesis, we focus on approaches to improve models for unbalanced datasets. Our first approach starts in visual classification task where we aim to increase performance on rarer classes. In this work, we create new stronger classifiers for rarer classes by leveraging the representations and classifiers learnt for common categories. Our simple method can be applied on top of any existing set of classifiers, thus showcasing that learning better classifiers does not require extensive, complicated approaches. Our second approach ventures into visual detection, where the additional localization task makes it difficult to train better rare detectors. We take a closer look at the basic resampling approach used widely in detection for unbalanced datasets. Notably, we showcase that the fundamental resampling strategy in detection can be simply improved by not only resampling whole images but also resampling only objects.

While model improvements for unbalanced datasets prove to be successful, we note that high-quality unbalanced dataset benchmarks, critical for progress, are still elusive. Unbalanced datasets are difficult to curate because exploring huge swaths of unlabeled data to find and label tail classes is prohibitively expensive. We propose to scale exploring large-scale unbalanced datasets in order to better label and analyze class distributions in datasets. We aim first to build a database of unlabeled data that can be rapidly searched through with multi-modal query. Multi-modal query allows for fine-grain and advanced search that would be otherwise impossible with single modal query. Finally, we propose to build a toolkit based on our database so that we may achieve two goals: 1) analyze labeled datasets to discover detailed class to class relations, 2) find any objects missing appropriate labels.

More Information

Thesis Committee Members:
Martial Hebert, Co-chair
Michael J Tarr, Co-chair
Deva Ramanan
Alexei Efros, UC Berkeley
Ross Girshick, Facebook AI Research