Open Access

ARTICLE

Virtual Keyboard: A Real-Time Hand Gesture Recognition-Based Character Input System Using LSTM and Mediapipe Holistic

by Bijon Mallik¹, Md Abdur Rahim¹, Abu Saleh Musa Miah², Keun Soo Yun^3,*, Jungpil Shin²

1 Department of Computer Science and Engineering, Pabna University of Science and Technology, Pabna, 6600, Bangladesh
2 School of Computer Science and Engineering, The University of Aizu, Aizuwakamatsu, Fukushima, 965-8580, Japan
3 School of Computer and Information Technology, Ulsan College, Dong-gu, Ulsan, 44610, South Korea

* Corresponding Author: Keun Soo Yun. Email:

Computer Systems Science and Engineering 2024, 48(2), 555-570. https://doi.org/10.32604/csse.2023.045981

Received 13 September 2023; Accepted 12 December 2023; Issue published 19 March 2024

Abstract

In the digital age, non-touch communication technologies are reshaping human-device interactions and raising security concerns. A major challenge in current technology is the misinterpretation of gestures by sensors and cameras, often caused by environmental factors. This issue has spurred the need for advanced data processing methods to achieve more accurate gesture recognition and predictions. Our study presents a novel virtual keyboard allowing character input via distinct hand gestures, focusing on two key aspects: hand gesture recognition and character input mechanisms. We developed a novel model with LSTM and fully connected layers for enhanced sequential data processing and hand gesture recognition. We also integrated CNN, max-pooling, and dropout layers for improved spatial feature extraction. This model architecture processes both temporal and spatial aspects of hand gestures, using LSTM to extract complex patterns from frame sequences for a comprehensive understanding of input data. Our unique dataset, essential for training the model, includes 1,662 landmarks from dynamic hand gestures, 33 postures, and 468 face landmarks, all captured in real-time using advanced pose estimation. The model demonstrated high accuracy, achieving 98.52% in hand gesture recognition and over 97% in character input across different scenarios. Its excellent performance in real-time testing underlines its practicality and effectiveness, marking a significant advancement in enhancing human-device interactions in the digital age.

---

Keywords

---