Students Develop Augmented Reality Software For Deaf and Hard of Hearing Individuals

Approximately 466 million people worldwide have “disabling hearing loss,” according to the World Health Organization. That’s more than five percent of the world’s population.

Christopher Caulfield and Devon Bain, both candidates for the Technion-Cornell Dual Master’s Degrees in Connective Media ‘19, have personal connections with some of the communication barriers facing people who are deaf or hard of hearing. Caulfield was born profoundly deaf and Bain’s mother has hearing loss.

For their specialization project (a requirement of all Jacobs Technion-Cornell Institute master’s students), Caulfield and Bain are creating technology to make one-on-one conversations easier for people who are deaf or hard of hearing.

Using Research to Develop Technology

“Connective Media is really unique because it doesn’t just teach us about software development, it also teaches about user research and design,” said Bain.

The pair is working with Assistant Professor Shiri Azenkot, whose research focuses on accessible technology. Azenkot is an assistant professor at the Jacobs Technion-Cornell Institute at Cornell Tech who also specializes in research and technology for people who are blind or, like herself, have low vision. Azenkot has provided valuable feedback on the project based on her accessibility research. The fact that she is not deaf or hard of hearing has given them an additional perspective, said Caulfield.

Caulfield and Bain interviewed seven deaf or hard of hearing individuals of all backgrounds to learn about the technology they currently used, the challenges they faced with one-on-one conversations, and the technology that would be most helpful. The people ranged from students and senior citizens to working professionals. The primary difficulties most participants faced included difficulties hearing and participating in conversations in noisy locations like restaurants, subways, and public places. Some of the individuals felt frustrated, angry, and less motivated to socialize even though they wanted to participate in conversations. People also expressed worry about how others would interpret their hearing loss and accessible technology use. For example, one woman was nervous to disclose her hearing loss at work because she was nervous about how it would be received.

Most of the people they interviewed wanted a wearable augmented reality captioning system that could be used for lengthier conversations and ones in loud places. The insight motivated Caulfield and Bain to show captions in augmented reality glasses because they found that most of the current technology shows captions on computers or handheld devices. “Our research focuses on how this is a barrier on one-on-one conversations,” said Caulfield, “We want to make conversations more seamless using augmented reality.”

Building Commercial Projects

Students meet with company advisors throughout two semesters to create a presentation and demo which they pitch to company stakeholders at their mentoring company. Caulfield and Bain are working closely with a Verizon executive who is interested in augmented reality. They have people test their prototypes to provide feedback for iterations. “Hopefully that will lead to a fully-functional commercial project that people will use,” said Bain.

Caulfield and Bain built a prototype that uses augmented reality headsets. They used ARKit, the iOS platform for augmented reality, and a text box to populate captions for the user, who inserts their smartphone into a MERGE augmented reality headset. Right now they are using a script to show the overall experience, but later they plan to integrate automated speech recognition. They are using computer vision and face recognition to locate a person’s face and to position the captions directly under the speaker’s chin and the text size and color changes based on the speaker’s tone and emphasis.

Building More Accessibility Products at Cornell Tech

“Our project has really inspired a lot of people to think about accessibility at Cornell Tech,” said Bain. Now a group of Cornell Tech students meets regularly to discuss accessibility projects and Caulfield and Bain think it is likely that one of the ideas will be refined in Startup Studio.

“We were surprised about how each of them [was] interested in accessibility because they were related to someone with a disability or good friends with someone with a disability,” said Caulfield. And Bain said that even people without a personal connection to someone with a disability cared because they are focused on social good.

When Caulfield decided to attend Cornell Tech, his mission was to complete a specialization project related to a painful point he experienced in his life — hearing loss. Bain had a shared vision because he wanted to create technology that makes it easier for his mother and other people with hearing loss to have more seamless discussions. Their dedication has motivated other Cornell Tech students to use their skills to build accessibility technology to help people with a range of needs.