Real-Time Sign Language Fingerspelling Recognition System Using 2D Deep CNN with Two-Stream Feature Extraction Approach

Abstract: This research paper introduces a novel sign language recognition system developed using advanced deep learning (DL) techniques aimed at enhancing communication capabilities between deaf and hearing individuals. The system leverages a convolutional neural network (CNN) architecture, optimized for the real-time interpretation of dynamic hand gestures that constitute sign language. A comprehensive dataset was employed to train and validate the model, encompassing a diverse range of gestures across different environmental settings. Comparative analysis revealed that the deep learning-based model significantly outperforms traditional machine learning techniques in terms of recognition accuracy, particularly with the increase in the volume of training data. This was illustrated through various performance metrics, including a detailed confusion matrix and Levenshtein distance measurements, highlighting the system’s efficacy in accurately identifying complex gestures. Real-time application tests further demonstrated the model's robustness and adaptability to varying lighting conditions and backgrounds, essential for practical deployment. Key challenges identified include the need for broader linguistic diversity in training datasets and enhanced model sensitivity to subtle gestural distinctions. The paper concludes with suggestions for future research directions, emphasizing algorithm optimization, data diversification, and user-centric design improvements to foster wider adoption and usability. This study underscores the potential of deep learning technologies to revolutionize assistive communication tools, making them more accessible and effective for the deaf community.

Aziza Zhidebayeva, Gulira Nurmukhanbetova, Sapargali Aldeshov, Kamshat Zhamalova, Satmyrza Mamikov and Nursaule Torebay. “Real-Time Sign Language Fingerspelling Recognition System Using 2D Deep CNN with Two-Stream Feature Extraction Approach”. International Journal of Advanced Computer Science and Applications (IJACSA) 15.9 (2024). http://dx.doi.org/10.14569/IJACSA.2024.01509108

@article{Zhidebayeva2024,
title = {Real-Time Sign Language Fingerspelling Recognition System Using 2D Deep CNN with Two-Stream Feature Extraction Approach},
journal = {International Journal of Advanced Computer Science and Applications},
doi = {10.14569/IJACSA.2024.01509108},
url = {http://dx.doi.org/10.14569/IJACSA.2024.01509108},
year = {2024},
publisher = {The Science and Information Organization},
volume = {15},
number = {9},
author = {Aziza Zhidebayeva and Gulira Nurmukhanbetova and Sapargali Aldeshov and Kamshat Zhamalova and Satmyrza Mamikov and Nursaule Torebay}
}