Journal of Ubiquitous Computing and Communication Technologies
IRO Journals
Volume 6 — Issue 4 — December 2024

Machine Learning based ISL Identification and Translation

https://doi.org/10.36548/jucct.2024.4.003
Thirumahal R.
Assistant Professor,Department of Computer Science and Engineering, PSG College of Technology, Coimbatore, India
Aswath Harish Jayaprakash
Under Graduate, Department of Computer Science and Engineering, PSG College of Technology, Coimbatore, India
Shiva Prakash P.
Under Graduate, Department of Computer Science and Engineering, PSG College of Technology, Coimbatore, India
Yuvaraj Kesavan P.
Under Graduate, Department of Computer Science and Engineering, PSG College of Technology, Coimbatore, India
Chirenjeevi M.
Under Graduate, Department of Computer Science and Engineering, PSG College of Technology, Coimbatore, India
Siva M.
Under Graduate, Department of Computer Science and Engineering, PSG College of Technology, Coimbatore, India
PDF
PDF

How to Cite

R., Thirumahal, Aswath Harish Jayaprakash, Shiva Prakash P., Yuvaraj Kesavan P., Chirenjeevi M., and Siva M. 2024. “Machine Learning Based ISL Identification and Translation”. Journal of Ubiquitous Computing and Communication Technologies 6 (4): 353-67. https://doi.org/10.36548/jucct.2024.4.003.
ACM ACS APA ABNT Chicago Harvard IEEE MLA Turabian Vancouver AMA

Download Citation

Endnote/Zotero/Mendeley (RIS) BibTeX

Keywords

— Sign Language Recognition
— Indian Sign Language (ISL)
— Machine Learning
— LSTM (Long Short-Term Memory)
— Transformers
— Real-time Gesture Recognition
— Feature Extraction
— Computer Vision
— ISL Dataset
— Gesture-to-Text Translation
— Human-Computer Interaction (HCI)
Published: 30-12-2024

Abstract

Sign language is an essential means of communication for the deaf and hard-of-hearing community. However, effective communication between sign language users and those unfamiliar with sign language can be challenging. The primary goal is to utilize the machine learning to automatically identify sign language gestures and translate them into easily understandable formats. This research presents a comprehensive sign language detection system that captures sign language gestures, detects them, and provides output in text using LSTM (Long Short-Term Memory) and Transformers with an accuracy of 79%. This multimodal approach ensures the system helps in understanding the sign language.

References

  1. Goel, Pragati, Ashutosh Sharma, Vikas Goel, and Vikas Jain. "Real-time sign language to text and speech translation and hand gesture recognition using the LSTM model." In 2022 3rd International Conference on Issues and Challenges in Intelligent Computing Techniques (ICICT),Ghaziabad, India. IEEE, 2022. 1-6.
  2. Mallika, Garg, Debashis Ghosh, and Pyari Mohan Pradhan. "A two-stage convolutional neural network for hand gesture recognition." In Proceedings of the 6th International Conference on Advance Computing and Intelligent Engineering: ICACIE 2021, Singapore: Springer Nature Singapore, 2022. 383-392.
  3. Tiku, Kohsheen, Jayshree Maloo, Aishwarya Ramesh, and R. Indra. "Real-time conversion of sign language to text and speech." In 2020 Second International Conference on Inventive Research in Computing Applications (ICIRCA), Coimbatore, India. IEEE, 2020. 346-351.
  4. Srininvas, Lokavarapu V., Chitri Raminaidu, Devareddi Ravibabu, and Shiva Shankar Reddy. "A framework to recognize the sign language system for deaf and dumb using mining techniques." Indonesian Journal of Electrical Engineering and Computer Science 29, no. 2 (2023): 1006-1016.
  5. Ojha, Ankit, Ayush Pandey, Shubham Maurya, Abhishek Thakur, and P. Dayananda. "Sign language to text and speech translation in real time using convolutional neural network." International Journal of Engineering Research & Tecnology.(IJERT) 8, no. 15 (2020): 191-196.
  6. Joshi, Abhinav, Susmit Agrawal, and Ashutosh Modi. "ISLTranslate: Dataset for Translating Indian Sign Language." arXiv preprint arXiv:2307.05440 (2023).
  7. Patil, Rachana, Vivek Patil, Abhishek Bahuguna, and Gaurav Datkhile. "Indian sign language recognition using convolutional neural network." In ITM web of conferences, vol. 40, p. 03004. EDP Sciences, 2021.
  8. Dafnis, Konstantinos M., Evgenia Chroni, Carol Neidle, and Dimitris N. Metaxas. "Isolated sign recognition using ASL datasets with consistent text-based gloss labeling and curriculum learning." In Proceedings of the 7th International Workshop on Sign Language Translation and Avatar Technology (STLAT 7), LREC 2022. European Language Resources Association (ELRA), 2022.
  9. Sridevi, Parama, Tahmida Islam, Urmi Debnath, Noor A. Nazia, Rajat Chakraborty, and Celia Shahnaz. "Sign Language recognition for Speech and Hearing Impaired by Image processing in matlab." In 2018 IEEE Region 10 Humanitarian Technology Conference (R10-HTC),Malambe, Sri Lanka. IEEE, 2018. 1-4.
  10. Akshatha Rani K and Dr. N Manjanaik, "Sign Language to Text-Speech Translator Using Machine Learning", International Journal of Emerging Trends in Engineering Research, Vol. 9, no. 7, 2021.
  11. Nandhini, A. Sunitha, D. Shiva Roopan, S. Shiyaam, and S. Yogesh. "Sign language recognition using convolutional neural network." In Journal of Physics: Conference Series, vol. 1916, no. 1, p. 012091. IOP Publishing, 2021.
  12. Limaye, Heramba, Shraddha Shinde, Anurag Bapat, and Nimish Samant. "Sign Language Recognition using Convolutional Neural Network with Customization." Available at SSRN 4169172 (2022).
  13. Wibawa, Adhi Dharma, and Surya Sumpeno. "Gesture recognition for Indonesian Sign Language Systems (ISLS) using multimodal sensor leap motion and myo armband controllers based-on naïve bayes classifier." In 2017 International Conference on Soft Computing, Intelligent System and Information Technology (ICSIIT), Changa, India pp. 1-6. IEEE, 2017.
  14. Rajalakshmi, E., R. Elakkiya, V. Subramaniyaswamy, L. Prikhodko Alexey, Grif Mikhail, Maxim Bakaev, Ketan Kotecha, Lubna Abdelkareim Gabralla, and Ajith Abraham. "Multi-semantic discriminative feature learning for sign gesture recognition using hybrid deep neural architecture." IEEE Access 11 (2023): 2226-2238.
  15. Mrs. S Chandra Gandhi, Aakash raj R, Muhammed Shamil ML, and Akhil S, "Real Time Translation Of Sign Language To Speech And Text", International Advance Research Journal in Science, Engineering, and Technology, Vol. 8, no 4, 2021. 56-58
  16. Victoria Adebimpe Akano and Adejoke O Olamiti, Conversion of Sign Language To Text And Speech Using Machine Learning Techniques", Journal Of Research And Review In Science, Vol. 5, 58-65, 2018.