Vol. 2 No. 2 (2020), Articles
Volume 2 — Issue 2 — June 2020

Real Time Sign Language Recognition and Speech Generation

Published 03-06-2020
https://doi.org/10.36548/jiip.2020.2.001
Amrita Thakur
Department of Electronics and Computer Engineering, Pulchowk Campus
Pujan Budhathoki
Department of Electronics and Computer Engineering, Pulchowk Campus
Sarmila Upreti
Department of Electronics and Computer Engineering, Pulchowk Campus
Shirish Shrestha
Department of Electronics and Computer Engineering, Pulchowk Campus
Subarna Shakya
Department of Electronics and Computer Engineering, Pulchowk Campus
PDF
PDF

How to Cite

Thakur, Amrita, Pujan Budhathoki, Sarmila Upreti, Shirish Shrestha, and Subarna Shakya. 2020. “Real Time Sign Language Recognition and Speech Generation”. Journal of Innovative Image Processing 2 (2): 65-76. https://doi.org/10.36548/jiip.2020.2.001.
ACM ACS APA ABNT Chicago Harvard IEEE MLA Turabian Vancouver AMA

Download Citation

Endnote/Zotero/Mendeley (RIS) BibTeX

Keywords

  • American Sign Language (ASL)
  • Sign Language Recognition
  • Convolutional Neural Network (CNN)
  • Python
  • VGG-16

Abstract

Sign Language is the method of communication of deaf and dumb people all over the world. However, it has always been a difficulty in communication between a verbal impaired person and a normal person. Sign Language Recognition is a breakthrough for helping deaf-mute people to communicate with others. The commercialization of an economical and accurate recognition system is today's concern of researchers all over the world. Thus, sign language recognition systems based on Image processing and neural networks are preferred over gadget system as they are more accurate and easier to make. The aim of this paper is to build a user friendly and accurate sign language recognition system trained by neural network thereby generating text and speech of the input gesture. This paper also presents text to sign language generation model that enables a way to establish a two-way communication without the need of a translator.

References

Pavlovic, V, Sharma, R., &Huang T. (1997). Visual Interpretation of Hand Gestures for Human-Computer Interaction (HCI): A Review. IEEE TOPAMI, VOL. 19, NO. 7.

Jiyong, M.W., Gao, Jiangqin, &Wu Chunli, Wang (2000). A Continuous Chinese sign language recognition system in Automatic Face and Gesture Recognition. Fourth IEEE International Conference on 2000.

Vogler, C.M., &Dimitris (2004). Handshapes and Movements: Multiple-Channel American Sign Language Recognition Gesture-Based Communication in Human-Computer Interaction. Ed. A.V.Camurri, Gualtiero. Vol. 2915: Springer Berlin Heidelberg. 247-258.

Gunasekaran, K., &Manikandan, R. (2013). Sign Language to Speech Translation System Using PIC Microcontroller. International Journal of Engineering and Technology (IJET), Vol 5 No 2.

Kalidolda, N., &Sandygulova, A. (2018). Towards Interpreting Robotic System for Finger spelling Recognition in Real-Time. HRI Companion: 2018 ACM/IEEE International Conference on Human-Robot Interaction Companion.

Liang, R., & Ouhyoung, M. (1998). Real-time continuous gesture recognition system for sign language. Proc Third. IEEE International Conf: on Automatic Face and Gesture Recognition, pp. 558-567.

LeCun, Yann, Lon, Bottou, Yoshua, Bengio, & Haffner, Patrick (1998). Gradient-based learning applied to document recognition’, Proceedings of the IEEE 86, vol. 11, pages 22782324.

Simonyan, Karen, & Zisserman, Andrew(2015). Very Deep.... Recognition. Conference paper ICLR.