Abstract

Recognizing human emotion/expressions automatically is quite an expected ability for intelligent robotics, as it can promote better communication and cooperation with humans. Current deep-learning-based algorithms may achieve impressive performance in some lab-controlled environments, but they always fail to recognize the expressions accurately for the uncontrolled in-the-wild situation. Fortunately, facial action units (AU) describe subtle facial behaviors, and they can help distinguish uncertain and ambiguous expressions. In this work, we explore the correlations among the action units and facial expressions, and devise an AU-Expression Knowledge Constrained Representation Learning (AUE-CRL) framework to learn the AU representations without AU annotations and adaptively use representations to facilitate facial expression recognition. Specifically, it leverages AU-expression correlations to guide the learning of the AU classifiers, and thus it can obtain AU representations without incurring any AU annotations. Then, it introduces a knowledge-guided attention mechanism that mines useful AU representations under the constraint of AU-expression correlations. In this way, the framework can capture local discriminative and complementary features to enhance facial representation for facial expression recognition. We conduct experiments on the challenging uncontrolled datasets to demonstrate the superiority of the proposed framework over current state-of-the-art methods.

Framework

Experiment

Conclusion

In this paper, we propose an AU-Expression Knowledge Constrained Representation Learning framework that exploits prior knowledge to help mining AU information to promote facial expression recognition. It first leverages relationships between AUs and expressions to guide learning domain-adaptive AU representation without any additional AU annotations. Then, it introduces an attentional mechanism to adaptively select useful AU representation under the constraint of the dependencies among AUs. We conduct an experiment on two in-the-wild datasets and show that our method outperforms current state-of-the-art competitors by a sizable margin.