Open Access

ARTICLE

Rotation, Translation and Scale Invariant Sign Word Recognition Using Deep Learning

by Abu Saleh Musa Miah¹, Jungpil Shin^1,*, Md. Al Mehedi Hasan¹, Md Abdur Rahim², Yuichi Okuyama¹

1 School of Computer Science and Engineering, The University of Aizu, Aizuwakamatsu, Fukushima, 965-8580, Japan
2 Department of Computer Science and Engineering, Pabna University of Science and Technology, Pabna, Bangladesh

* Corresponding Author: Jungpil Shin. Email:

Computer Systems Science and Engineering 2023, 44(3), 2521-2536. https://doi.org/10.32604/csse.2023.029336

Received 01 March 2022; Accepted 07 April 2022; Issue published 01 August 2022

Abstract

Communication between people with disabilities and people who do not understand sign language is a growing social need and can be a tedious task. One of the main functions of sign language is to communicate with each other through hand gestures. Recognition of hand gestures has become an important challenge for the recognition of sign language. There are many existing models that can produce a good accuracy, but if the model test with rotated or translated images, they may face some difficulties to make good performance accuracy. To resolve these challenges of hand gesture recognition, we proposed a Rotation, Translation and Scale-invariant sign word recognition system using a convolutional neural network (CNN). We have followed three steps in our work: rotated, translated and scaled (RTS) version dataset generation, gesture segmentation, and sign word classification. Firstly, we have enlarged a benchmark dataset of 20 sign words by making different amounts of Rotation, Translation and Scale of the original images to create the RTS version dataset. Then we have applied the gesture segmentation technique. The segmentation consists of three levels, i) Otsu Thresholding with YCbCr, ii) Morphological analysis: dilation through opening morphology and iii) Watershed algorithm. Finally, our designed CNN model has been trained to classify the hand gesture as well as the sign word. Our model has been evaluated using the twenty sign word dataset, five sign word dataset and the RTS version of these datasets. We achieved 99.30% accuracy from the twenty sign word dataset evaluation, 99.10% accuracy from the RTS version of the twenty sign word evolution, 100% accuracy from the five sign word dataset evaluation, and 98.00% accuracy from the RTS version five sign word dataset evolution. Furthermore, the influence of our model exists in competitive results with state-of-the-art methods in sign word recognition.

---

Keywords

---