Abstract
A sign language recognition system for low-resource Sinhala Sign Language using Leap Motion (LM) and Deep Neural Networks (DNN) has been presented in this paper. The study extracts static and dynamic features of hand movements of Sinhala Sign Language (SSL) using a LM controller which acquires the position of the palm, radius of hand sphere and positions of five fingers, and the proposed system is tested with the selected 24 letters and 6 words. The experimental results prove that the proposed DNN model with an average testing accuracy of 89.2% outperforms a Naïve Bayes model with 73.3% testing accuracy and a Support Vector Machine (SVM) based model with 81.2% testing accuracy. Therefore, the proposed system which uses 3D non-contact LM Controller and machine learning model has a great potential to be an affordable solution for people with hearing impairment when they communicate with normal people in their day-to-day life in all service sectors.
References
World Health Organization, “World Report on Hearing”, Geneva; BY-NC-SA 3.0 IGO: Licensee: CC, (2021)
M. Ranaweera Premasiri, H. A. D. K. N. Hettiarachchi, J. C. Delmar, P. H. S. Dias, G. S. Mallika, W. G. S. A. K. Rajapaksha, A. N. Jayasekara, S. M. W. Samarakoon et al. (Eds.). “Sri Lankan Sign Dictionary: Sinhala Language”. National Institute of Education, Maharagama, Sri Lanka (2015)
B. H. Juang and L. R. Rabiner, “Hidden Markov models for speech recognition,” Technometrics, vol. 33, no. 3, pp. 251–272, (1991)
H. Bunke, M. Roth, and E. G. Schukat-Talamazzini, “Off-line cursive handwriting recognition using hidden Markov models,” Pattern recognition, vol. 28, no. 9, pp. 1399–1413, (1995)
K. Grobel and M. Assan, “Isolated sign language recognition using hidden Markov models,” in Proceedings of the IEEE International Conference on System, Man and Cybernetics, pp. 162-167, (1996)
J. Wu, W. Gao, Y. Song, W. Liu, and B. Pang, “A simple sign language recognition system based on data glove,” in Proceedings of the 4th International Conference on Signal Processing, Vol. 2, pp. 1257-1260, (1998)
A.A Aliaa, A Youssif, Amal Elsayed Aboutabl and Heba Hamdy Ali, “Arabic Sign Language (ArSL) Recognition System Using HMM” International Journal of Advanced Computer Science and Applications (IJACSA), Vol. 2(11), (2011)
R-H. Liang and M. Ouhyoung, “A Sign Language Recognition System Using Hidden Markov Model and Context Sensitive Search” Proceedings of the ACM Symposium on Virtual Reality Software and Technology, Hong Kong, pp. 59-66, July (1996)
Saldaña González, G., Cerezo Sánchez, J., Bustillo Díaz, M.M., Ata Pérez, A., 2018. “Recognition and Classification of Sign Language for Spanish” Computer Systems (2018)
Zafar Ahmed Ansari and Gaurav Harit, “Nearest Neighbor Classification of Indian Sign Language Gestures using Kinect Camera”, in Sadhana, Vol. 41, No. 2, pp. 161-182, (2016)
Kang, B., Tripathi, S., & Nguyen, T. Q. “Real-time sign language fingerspelling recognition using convolutional neural networks from depth map”, 3rd IAPR Asian Conference on Pattern Recognition (ACPR) (pp. 136-140). IEEE, (2015)
P. Fernando & P. Wimalaratne, “Sign Language Translation Approach to Sinhalese Language” GSTF Journal of Computing, Vol. 5 (1) pp. 52-59, (2016)
S.D. Hettiarachchi & R.G.N. Meegama, “Machine Learning Approach for Real Time Translation of Sinhala Sign Language into Text”, Proceedings of International Conference on Advances in Computing and Technology (ICACT-2020), pp 23-25 (2020)
H. Wang, M. C. Leu, and C. Oz, “American sign language recognition using multi-dimensional hidden Markov models,” Journal of Information Science and Engineering, vol. 22, no. 5, pp. 1109–1123, (2006)
S. Lang, M. Block, and R. Rojas, “Sign language recognition using Kinect,” Artificial Intelligence and Soft Computing, pp. 394–402, Springer Berline Heidelberg, (2012)
A. Stone, “An introduction to Sri Lankan Sign Language.” Matara, Sri Lanka : Karunaratne & Sons, Sri Lanka, (2007)
H.C.M. Herath, W.A.L.V.Kumari, W.A.P.B Senevirathne and M.B Dissanayake, “Image based Sign Language Recognition System for Sinhala Sign Language”, Proceedings of SAITM Research Symposium on Engineering Advancements, Colombo, Sri Lanka (2013)
C. Vogler and D. Metaxas. “Adapting Hidden Markov Models for ASL recognition by using three-dimensional computer vision methods”, In Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, pages 156–161, Orlando, FL, (1997)
Y. Cui and J. Weng. “Learning-based hand sign recognition.” In Proc. of the Intl. Workshop on Automatic Face- and Gesture-Recognition, Zurich, (1995)
S. G. Wysoski, M. V. Lamar, S. Kuroyanagi and Akira Iwata, “A Rotation Invariant Approach On Static-Gesture Recognition Using Boundary Histograms Networks”, IEEE Proceedings of the 9th International Conference on Neural Information Processing, (2002)
C. Keskin, F. Kırac¸, Y. E. Kara, and L. Akarun, “Real time hand pose estimation using depth sensors,” in Consumer Depth Cameras for Computer Vision. Springer, pp. 119–137 (2013)
C. Zhang, X. Yang, and Y. Tian, “Histogram of 3d facets: A characteristic descriptor for hand gesture recognition,” in Automatic Face and Gesture Recognition (FG), 2013 10th IEEE International Conference and Workshops on. IEEE, pp. 1–8 (2013)
K. Fok, N. Ganganath, C. Cheng and C. K. Tse, "A Real-Time ASL Recognition System Using Leap Motion Sensors," International Conference on Cyber-Enabled Distributed, (2015)
M.A. Almasre and H. Al-Nuaim, “Real-time Letter Recognition Model for Arabic Sign Language Using Kinect and Leap Motion Controller v2”, International journal of Advanced Engineering, Management and Science, Vol. 2(5) pp 514-522 (2016)
Mohandes, M., Aliyu, S., & Deriche, M. “Arabic sign language recognition using the Leap Motion Controller”. Paper presented at King Fahd University of Petroleum & Minerals Dhahran, 31261, Saudi Arabia., IEEE Xplore (2014)