Table of Content

Open AccessOpen Access


Facial Expression Recognition Based on the Fusion of Infrared and Visible Image

Jiancheng Zou1, Jiaxin Li1,*, Juncun Wei1, Zhengzheng Li1, Xin Yang2

1 North China University of Technology, Beijing, 100144, China
2 Department of Computer Science, Middle Tennessee State University, Murfreesboro, 37132, USA

* Corresponding Author: Jiaxin Li. Email:

Journal on Artificial Intelligence 2021, 3(3), 123-134.


Facial expression recognition is a research hot spot in the fields of computer vision and pattern recognition. However, the existing facial expression recognition models are mainly concentrated in the visible light environment. They have insufficient generalization ability and low recognition accuracy, and are vulnerable to environmental changes such as illumination and distance. In order to solve these problems, we combine the advantages of the infrared and visible images captured simultaneously by array equipment our developed with two infrared and two visible lens, so that the fused image not only has the texture information of visible image, but also has the contrast information of infrared image. On the other hand, we improved the WGAN by adding SSIM and LBP loss functions to ensure the structural similarity between the fused image and infrared image, and also the texture similarity between the fused image and visible image respectively. Finally, a facial expression recognition model Pyconv-SE18 with pyramid convolution and attention mechanism module is designed to extract the important feature information of facial expression in multiple scales. We add cosine distance loss function to reduce the feature difference within the class. Experiment results show that the robustness of expression recognition algorithm to illumination is improved based on the fused images. The accuracy of this model on FER2013 and CK+ public data sets are 69.3% and 94.6%, respectively.


Cite This Article

Zou, J., Li, J., Wei, J., Li, Z., Yang, X. (2021). Facial Expression Recognition Based on the Fusion of Infrared and Visible Image. Journal on Artificial Intelligence, 3(3), 123–134.

This work is licensed under a Creative Commons Attribution 4.0 International License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
  • 695


  • 440


  • 0


Share Link