A system capable of interpreting hand gestures to foster the inclusion of persons with language and speech disabilities

In this interview, we catch up with Boris and Hamza to learn all about their automatic sign language recognition system which can identify hand gestures and translate them into text and then into speech. The project aims to foster the inclusion of persons with language and speech disabilities, helping them to communicate more effectively.

How did this project come about?

[Boris answer in the video here below]

So how does the software work concretely?

Boris: The concept is based on two systems, the first being; a framework that can detect movement of the five fingers of each hand. Basically, once we have key reference points identified on each hand and the recording of the movements of each hand, we can associate them to a dictionary. For example, expressing the word “hello” is linked to a particular movement in sign language. On our part, we record these movements of the fingers and the system is capable of associating each sign to the equivalent word in spoken language.

While this system is able to aggregate words on the basis of their corresponding signs, the words together do not form a complete sentence in conventional spoken language. This is where the second system comes in, arranging the words in the right order so as to form a comprehensible sentence, after which is converted into audio format using a speech synthesizer.

All of this relies on Artificial Intelligence which requires an enormous number of recordings of each sign. Typically, we need between 100 and 500 sign movements made by different people. This will serve as the model for the AI which will then be able to interpret the signs made by any user. As it stands, we have already validated the proof of concept, with the AI training and the recognition stages finalized.  However, we need to continue enriching the AI system with more data and working on improving the user interface.

What use cases do you envisage for this software?

Boris: There are two axes, one personal with the application embedded on a smartphone, capable of interpreting sign language. The second involves developing a box with a recognition camera which can be installed in stores, a bakery for example. This system can allow the inclusion of persons who have difficulty communicating, giving them the opportunity to interact with the store attendants, using sign language.

Tell us a bit about yourselves

So how was the experience working together?

Boris: I’m very proud to have managed Hamza during his internship. His background in image and signal data management was instrumental to the advancement of the project. He was responsible for the research on what frameworks to use, how to train the AI, and how to gather data. Following this, we worked together on all the other phases of the project until now, and it was a pleasure working with him.

Hamza: It’s been an excellent opportunity for me to be able to work on such a project as my first experience in a professional environment. I particularly enjoyed collaborating with people like Boris who taught me a lot and helped me to realise my learning objectives. I think this internship was a really solid base, preparing me to take the next steps in my career.