Proposal of a Real-time American Sign Language Detector using MediaPipe and Recurrent Neural Network - Ghosh, Souradeep (International Journal of Computer Sciences and Engineering, 2021)
The predominant vocabulary of the deaf and dumb, Sign Language serves as a natural, visual language which our brain is capable of processing and deciphering linguistic details. For the past two decades, scientists have been researching the automated recognition of sign language using translating gloves and complex systems with several cameras. Most of these systems can provide partial or complete recognition of the vocabulary but aren’t cost-effective for the average and below-average section of the demographic. With the advent of AI, we’re trying to overcome this biasness in technology. Google’s MediaPipe, which is an open-source framework for multimodal (video, audio, time-series) features with applied ML pipelines, came into existence in 2019. Using MediaPipe’s Multi-hand Tracking model pipeline we can get landmarks of our fingers. This paper advocates the use of MediaPipe Hand Tracking to get hand landmarks, training a Keras RNN-LSTM model with that data to detect Sign Language of 5 trained words in real-time.