Bilinear Classifiers for Visual Recognition

Abstract

We describe an algorithm for learning bilinear SVMs. Bilinear classifiers are a discriminative variant of bilinear models, which capture the dependence of data on multiple factors. Such models are particularly appropriate for visual data that is better represented as a matrix or tensor, rather than a vector. Matrix encod- ings allow for more natural regularization through rank restriction. For example, a rank-one restriction produces a bilinear classifier that can be interpreted as a sep- arable filter. We also use bilinear classifiers for transfer learning by sharing linear factors between different tasks. Finally, we show that bilinear classifiers can be trained with biconvex programs. Such programs are optimized with coordinate descent, where each step is equivalent to a standard convex problem. This allows us to leverage existing SVM solvers during learning. We demonstrate bilinear SVMs on difficult problems of people detection in video sequences and action classification of video sequences, achieving state-of-the-art results in both.