Building and Leveraging Category Hierarchies for Large-scale Image Classification

Abstract

In image classification, visual separability between different object categories is highly uneven, and some categories are more difficult to distinguish than others. Such difficult categories demand more dedicated classifiers. However, existing deep convolutional neural networks (CNN) are trained as flat N-way classifiers, and few efforts have been made to leverage the hierarchical structure of categories. Naturally, incorporating external knowledge from category hierarchies presents a major opportunity to improve the task. However, traditional methods of manually constructing category hierarchies by experts (e.g. WordNet, ImageNet) and interest communities (e.g. Wikipedia) are either knowledge or time intensive, and the results have limited coverage. In this report, we study the problem of automatically learning and utilizing category hierarchies (taxonomies) for large-scale image classification. First, we present a probabilistic model for taxonomy induction by jointly leveraging text corpus and images from the web. The model is discriminatively trained given a small set of existing ontologies and is capable of building full category hierarchies from scratch for a collection of unseen conceptual labels with associated images. Then, we introduce hierarchical deep convolutional neural networks (HD-CNNs), which embeds deep convolutional neural networks into a category hierarchy. An HD-CNN separates easy classes using a coarse category classifier while distinguishing difficult classes using fine category classifiers. We reported state-of-the-art results on both taxonomy induction and image classification tasks.