AI4Bharat achieves more data processing with less costs in a critical research project on sign language with Microsoft Azure

Inclusive language key to social participation of people with hearing impairments

The Deaf and Hard of Hearing community (DHH) currently uses several sign languages for communication.

There are 300 sign languages in use around the world, each different from the others. For example, American Sign Language is different from British Sign Language. In India, the Indian sign language community is diverse, with different signs used for the same word in different regions. 

This poses a huge challenge to the DHH community’s efforts to ensure better social and workforce participation and equity in the quality of life.

With an eye on making the world more accessible to the DHH community in India, AI4Bharat launched project ASSIST to improve their quality of life using inclusive AI. While the ultimate aim would be to have solutions for real-time sign language translations, ASSIST is a research project towards that direction. 

Dr. Pratyush Kumar, Co-founder, AI4Bharat, says, “Microsoft firmly believes that accessible technology is a fundamental building block to unlock opportunities in every section of society while bridging the disability divide in innovative and new ways. Our partnership with them is allowing us to resolve many obstacles in our quest to build a more inclusive world for the DHH community.”

Data collection for sign language

The lack of data presented a significant problem for AI4Bharat, as AI models require huge amounts of data to learn from and perform. Hence, the team invested considerable effort in approaching people and collating data of them signing words. Working with the team at Microsoft Research, AI4Bharat adapted Project Karya, a crowdsourcing platform for data collection, to address its requirements. It extended Karya to collect sign language-related data via mobile phones. This was a first, as there was no other smartphone application that collected data from people using signs for words. It would show the user a word and record them signing it. The DHH community played a critical role as a co-creator in the project, helping with valuable inputs. With Microsoft’s support, AI4Bharat collaborated with several institutions, reaching out to people who understood sign language in multiple spoken languages. This allowed AI4Bharat to define the problem statement better.

More computing with less

As signs used across India are vastly different, the solution often requires a massive amount of data to train AI models, which in turn means using immense computing power.  

In this scenario, Azure Compute and Azure Cognitive Services proved effective. The solution had to collect and store data from videos containing sign language and publicly available online courses for Indian Sign Language. 

Overcoming AI skepticism  

A key challenge was the perception of AI as a futuristic technology that would be irrelevant in the current context.

Gokul NC, Team member, AI4Bharat, says, “Many people perceive AI as a futuristic notion of humanoid robots and holograms. There is a lot of skepticism around AI as well. AI is grounded in reality. If used correctly, it can revolutionize accessibility and inclusion. We hope to provide an AI-based solution that the DHH community can use to sign. It is also encouraging that there are efforts by the government to standardize sign language.” 

Harnessing innovation with Azure

While the research may still be away from becoming an AI-only solution for the DHH community, it is already being used in innovative ways. It is being leveraged to build other applications that can engage the community better. Along with Microsoft Research, AI4Bharat is developing a video conferencing application that will allow the DHH community to reap the benefits of remote work, remote learning, and file sharing capabilities, as well as a collaborative quiz application for members of the community to create videos for questions and answers.