Real-Time Indian Sign Language Translation Using Deep Learning and Multilingual Speech Technologies

Abstract

This paper introduces a novel hybrid framework for Indian Sign Language (ISL) translation that performs real-time recognition of both static and dynamic gestures and generates multilingual outputs in both text and speech. Unlike existing systems that are limited to either static classification or single-language outputs, our approach integrates a fine-tuned ResNet50V2 model for static gesture classification (98.7% accuracy) and a YOLOv8m detector for dynamic word recognition (88.7% mAP@50). The system employs MediaPipe for efficient hand landmark extraction and incorporates frame-skipping and cooldown strategies to optimize real-time performance on CPU-based devices, achieving an average of 3.4 FPS without GPU acceleration. Recognized gestures are mapped to sequences, translated into eight Indian languages using Google Translate, and converted into synthesized speech using gTTS. Experimental results validate the system's robustness across gesture types and linguistic outputs. The proposed work is the first to offer a complete ISL-to-text-and-speech pipeline with integrated multi-language support,via a desktop User-interface. This makes it a scalable, low-cost assistive tool designed to enhance accessibility and communication for the hearing-impaired community in multilingual contexts.

Country : India

1 V. Vishnu Shankar2 B. Tharun Kumar3 N. Vignesh Kumar4 V. Venkat Charan Reddy

  1. Department of CSE (AI), Madanapalle Institute of Technology and Science, Madanapalle, India
  2. Department of CSE (AI), Madanapalle Institute of Technology and Science, Madanapalle, India
  3. Department of CSE (AI), Madanapalle Institute of Technology and Science, Madanapalle, India
  4. Department of CSE (AI), Madanapalle Institute of Technology and Science, Madanapalle, India

IRJIET, Volume 9, Special Issue of ICCIS-2025 May 2025 pp. 155-161

doi.org/10.47001/IRJIET/2025.ICCIS-202525

References

  1. E. Abraham, A. Nayak, and A. Iqbal, "Real-time translation of Indian Sign Language using LSTM," *in Proc. of the International Conference on Machine Learning*, 2019.
  2. M. J. C. Samonte, C. J. M. Guingab, R. A. Relayo, M. J. C. Sheng, and J. R. D. Tamayo, "Using deep learning in sign language translation to text," *Proc. Int. Conf. Ind. Eng. Oper. Manag. *, 2022.
  3. S. Thakar, S. Shah, B. Shah, and A. V. Nimkar, "Sign language to text conversion in real time using transfer learning," *arXiv preprint arXiv:2211.14446v2*, 2022. [Online]. Available: https://arxiv.org/abs/2211.14446v2.
  4. K. Goyal, G. V., and Vellore Institute of Technology, "Indian sign language recognition using MediaPipe holistic," *in Proc. of the International Conference on Artificial Intelligence*, 2021.
  5. A.C. Punekar, "A translator for Indian sign language to text and speech," *Int. J. Res. Appl. Sci. Eng. Technol.*, vol. 8, no. 6, pp. 1640–1646, 2020. doi: 10.22214/ijraset.2020.6267.
  6. D. Kothadiya, C. Bhatt, K. Sapariya, K. Patel, A. Gil-González, and J. M. Corchado, "DeepSign: Sign language detection and recognition using deep learning," *Electronics*, vol. 11, no. 11, p. 1780, 2022. doi: 10.3390/electronics11111780.
  7. P. C. Badhe and V. Kulkarni, "Indian sign language translator using gesture recognition algorithm," *in Proc. 2015 IEEE Int. Conf. Computer Graphics, Vision and Information Security (CGVIS)*, 2015.
  8. H. K. Vashisth, T. Tarafder, R. Aziz, M. Arora, and A. Alpana, "Hand gesture recognition in Indian Sign Language using deep learning," *Engineering Proceedings*, vol. 96, 2023. doi: 10.3390/engproc2023059096.
  9. R. Janani, R. Harini, S. Keerthana, S. Madhubala, and S. Venkatasubramanian, "Sign language recognition system using computer vision," *Engineering Proceedings*, 2019.
  10. A.Kamble, "Conversion of sign language to text," *Int. J. Res. Appl. Sci. Eng. Technol.*, vol. 11, no. 5, pp. 1963–1968, 2023. doi: 10.22214/ijraset.2023.51981.
  11. "Sign Language-to-Text Dictionary with Lightweight Transformer Models," *in Proc. 32nd Int. Joint Conf. on Artificial Intelligence (IJCAI 2023)*, 2023.
  12. S. Sabharwal and P. Singla, "Optimized machine learning-based translation of Indian sign language to text," *Int. J. Intelligent Engineering and Systems*, vol. 16, no. 4, pp. 398–408, 2023. doi: 10.22266/ijies2023.0831.32.
  13. A.S. Ghotkar, R. Khatal, S. Khupase, S. Asati, and M. Hadap, "Hand gesture recognition for Indian sign language," *in Proc. 2012 Int. Conf. on Computer Communication and Informatics (ICCCI)*, 2012.
  14. S. S. Prakash, B. M. Devi, P. Arulprakash, M. Bandlamudi, and R. Radhakrishnan, "Educating and communicating with deaf learners using CNN-based sign language prediction system," *Int. J. Early Child. Spec. Educ. (INT-JECSE)*, vol. 14, no. 2, 2022. doi: 10.9756/INT-JECSE/V14I2.245.
  15. S. Ezhil Tharsan, S. Dharshan, G. Dinesh, and S. Saraswathi, "Real-time Indian sign language detection using LSTM and keypoint extraction," *Int. J. Computer Applications*, 2022.