Open Access

ARTICLE

Human-Computer Interaction Using Deep Fusion Model-Based Facial Expression Recognition System

Saiyed Umer^1,*, Ranjeet Kumar Rout², Shailendra Tiwari³, Ahmad Ali AlZubi⁴, Jazem Mutared Alanazi⁴, Kulakov Yurii⁵

1 Department of Computer Science & Engineering, Aliah University, Kolkata, 700156, India
2 Department of Computer Science and Engineering, National Institute of Technology, Srinagar, Jammu and Kashmir, 190006, India
3 Department of Computer Science & Engineering, Thapar University, Patiala, 147004, India
4 Computer Science Department, King Saud University, Riyadh, 11451, Saudi Arabia
5 Department of Computer Engineering, National Technical University of Ukraine, Igor Sikorsky Kyiv Polytechnic Institute, Kyiv, 03056, Ukraine

* Corresponding Author: Saiyed Umer. Email:

Computer Modeling in Engineering & Sciences 2023, 135(2), 1165-1185. https://doi.org/10.32604/cmes.2022.023312

Received 19 April 2022; Accepted 21 June 2022; Issue published 27 October 2022

Abstract

A deep fusion model is proposed for facial expression-based human-computer Interaction system. Initially, image preprocessing, i.e., the extraction of the facial region from the input image is utilized. Thereafter, the extraction of more discriminative and distinctive deep learning features is achieved using extracted facial regions. To prevent overfitting, in-depth features of facial images are extracted and assigned to the proposed convolutional neural network (CNN) models. Various CNN models are then trained. Finally, the performance of each CNN model is fused to obtain the final decision for the seven basic classes of facial expressions, i.e., fear, disgust, anger, surprise, sadness, happiness, neutral. For experimental purposes, three benchmark datasets, i.e., SFEW, CK+, and KDEF are utilized. The performance of the proposed system is compared with some state-of-the-art methods concerning each dataset. Extensive performance analysis reveals that the proposed system outperforms the competitive methods in terms of various performance metrics. Finally, the proposed deep fusion model is being utilized to control a music player using the recognized emotions of the users.

---

Keywords

---