Open Access

ARTICLE

Facial Expression Recognition Based on the Fusion of Infrared and Visible Image

Jiancheng Zou¹, Jiaxin Li^1,*, Juncun Wei¹, Zhengzheng Li¹, Xin Yang²

1 North China University of Technology, Beijing, 100144, China
2 Department of Computer Science, Middle Tennessee State University, Murfreesboro, 37132, USA

* Corresponding Author: Jiaxin Li. Email:

Journal on Artificial Intelligence 2021, 3(3), 123-134. https://doi.org/10.32604/jai.2021.027069

Received 10 January 2022; Accepted 14 January 2022; Issue published 25 January 2022

Abstract

Facial expression recognition is a research hot spot in the fields of computer vision and pattern recognition. However, the existing facial expression recognition models are mainly concentrated in the visible light environment. They have insufficient generalization ability and low recognition accuracy, and are vulnerable to environmental changes such as illumination and distance. In order to solve these problems, we combine the advantages of the infrared and visible images captured simultaneously by array equipment our developed with two infrared and two visible lens, so that the fused image not only has the texture information of visible image, but also has the contrast information of infrared image. On the other hand, we improved the WGAN by adding SSIM and LBP loss functions to ensure the structural similarity between the fused image and infrared image, and also the texture similarity between the fused image and visible image respectively. Finally, a facial expression recognition model Pyconv-SE18 with pyramid convolution and attention mechanism module is designed to extract the important feature information of facial expression in multiple scales. We add cosine distance loss function to reduce the feature difference within the class. Experiment results show that the robustness of expression recognition algorithm to illumination is improved based on the fused images. The accuracy of this model on FER2013 and CK+ public data sets are 69.3% and 94.6%, respectively.

---

Keywords

---