Development of a Low-Cost Flex-Sensor and Machine Learning-Based Sign Language Interpreter
Abstract
Over the years, sign language has been a paramount approach for bridging the communication gap between people with normal hearing and those with hearing challenges, thereby achieving inclusive education and social integration. With the diversity of modern-day gestures and body positions during communication, developing automated Signed Language Interpreters (SLIs) and recognition systems is complex and challenging. Thus, scientists and researchers worldwide are developing simpler and more affordable SLI systems. This research addresses the issue of complex, unaffordable, and ineffective communication between hearing- and hearing-impaired communities by developing a wearable SLI using a flex-sensor-based glove and a machine learning algorithm. The primary aim is to assist individuals with hearing impairments by converting American Sign Language (ASL) finger spelling into text. The study focused on ASL due to its extensive research base, global reach, and historical ties with Nigerian sign language. The methodology involved designing and integrating flex sensors into a glove to capture hand gestures, which were then processed using a machine learning algorithm to interpret the ASL finger spelling. The hardware components included flex sensors, a NodeMCU-32S LUA development board (based on the ESP32 microcontroller), and an MPU6050 accelerometer. At the same time, the software implementation utilised Python and the Arduino Integrated Development Environment (IDE) for data acquisition and model development. The results demonstrated the system's effectiveness in accurately interpreting ASL finger spelling, with significant improvements in training and test accuracy over multiple epochs. The system achieved satisfactory performance, indicating its potential for real-world applications in assisting communication for people who are deaf or hard of hearing. The implications of an accuracy of 92% indicate that a low-cost wearable device can interpret sign language correctly, bridging communication gaps for deaf people, thereby enhancing SLI availability and enabling hearing-challenged individuals to interact more freely and confidently across various social settings.