Joint Patch and Multi-label Learning for Facial Action Unit and Holistic Expression Recognition - Robotics Institute Carnegie Mellon University

Joint Patch and Multi-label Learning for Facial Action Unit and Holistic Expression Recognition

Kaili Zhao, Wen-Sheng Chu, Fernando De la Torre Frade, Jeffrey Cohn, and Honggang Zhang
Journal Article, IEEE Transactions on Image Processing, Vol. 25, No. 8, pp. 3931 - 3946, August, 2016

Abstract

Most action unit (AU) detection methods use one-vs-all classifiers without considering dependencies between features or AUs. In this paper, we introduce a Joint Patch and Multilabel Learning (JPML) framework that models the structured joint dependency behind features, AUs, and their interplay. Specifically, JPML leverages group sparsity to identify important facial patches, and learns a multi-label classifier constrained by the likelihood of co-occurring AUs. To describe such likelihood, we derive two AU relations, positive correlation and negative competition, by statistically analyzing more than 350,000 video frames annotated with multiple AUs. To the best of our knowledge, this is the first work that jointly addresses patch learning and multi-label learning for AU detection. In addition, we show that JPML can be extended to recognize holistic expressions by learning common and specific patches, which afford a more compact representation than standard expression recognition methods. We evaluate JPML on three benchmark datasets CK+, BP4D and GFT, using within- and cross-dataset scenarios. In four of five experiments, JPML achieved the highest averaged F1 scores in comparison with baseline and alternative methods with either patch learning or multi-label learning alone.

BibTeX

@article{Zhao-2016-5555,
author = {Kaili Zhao and Wen-Sheng Chu and Fernando De la Torre Frade and Jeffrey Cohn and Honggang Zhang},
title = {Joint Patch and Multi-label Learning for Facial Action Unit and Holistic Expression Recognition},
journal = {IEEE Transactions on Image Processing},
year = {2016},
month = {August},
volume = {25},
number = {8},
pages = {3931 - 3946},
}