Cornell Tech Students Pioneer New Insights in Health Tech

From consumer wellness apps to clinical decision-making tools to care operations, digital technologies are transforming today’s healthcare landscape. With guidance from clinical and industry experts, students participating in Cornell Tech’s Product Studio program have pioneered innovative new products to support patients, healthcare providers, and people with disabilities.

Memorial Sloan Kettering Cancer Center asked: How might we free clinicians from computer workstations, allowing them to focus directly on their patients during clinical encounters?

Eshann Toteja, Master in Computer Science ’19, said he was inspired to take on this challenge in part by his and his teammates’ personal experiences in clinical settings: “As patients at hospitals, our whole team has felt ignored or unimportant at some point in the exam room.”

In addition to improving patients’ experiences, Toteja said their goal was to stay “valuable and true to the biggest problems we saw providers facing in exam rooms.”

The team started by learning as much as they could from as many people as possible in the healthcare industry about the problems providers faced in the exam room. They also researched the kinds of solutions already being explored to address those problems.

“We did quite a few on-site days with Memorial Sloan Kettering,” said Skyler Erickson, Master in Computer Science ’19, “and we were working with physicians there and observing patient encounters and really trying to dig into, ‘Where is the computer workstation a problem and when does that distract from the patient interaction?’”

For Erickson, one of the key moments was “this realization that it’s not so much about what you do see, it’s equally important what you don’t see.” For example, he said, they never saw a physician pull up exam results during an appointment because doing so via their workstation was too difficult, time-consuming, and distracting. The goal, said Erickson, was to allow doctors to order tests “naturally and seamlessly and in one conversation.”

Ultimately the team, which also included Benjamin Yellin, Technion-Cornell Dual Master’s Degrees in Health Tech ’20, and Kriti Shah, Johnson Cornell Tech MBA ’19, developed an efficient new voice assistant, similar to Alexa or Google Home. With the aid of this tool, providers can easily order a test for a patient by saying, “Order a test for Patient X,” rather than interrupting the conversation to navigate various screens on their computer workstations.

The students credit their faculty advisor, Deborah Estrin, Cornell Tech’s Robert V. Tishman ’37 Founder’s Chair, with providing valuable real-world feedback. Toteja described her as “an immense help, constantly offering up the exact criticism and advice we needed to hear.” Her ties to the healthcare field, he said, “kept her current and connected, making her input extremely valuable.”

“We met with her many, many times to make sure we were on the right track and to pivot when we ran into some roadblocks,” said Yellin.

Microsoft asked: How might we use technology to help people with disabilities perform everyday tasks?

As the team from Cornell Tech began thinking about this question, Nabeel Seedat, Master in Electrical and Computer Engineering ’19, said, “We all had an association of people that we knew that had visual disabilities, and we felt that this was something that, as a team, we were all passionate about.” Based on this experience and with assistance from their advisors at Microsoft, the team from Cornell Tech decided to focus on developing a solution for people with visual impairment.

“We wanted to figure out which specific task is a big problem faced by those with vision impairment,” said Raga Kolli, Johnson Cornell Tech MBA ’19. “After doing research online, we realized that a common theme is social engagement—people with vision impairment face a lot of isolation, and a lack of social interaction causes further anxiety. A lot of what we do today is online and there’s a big information gap online for those with visual impairment vs. sighted users.”

That insight led them to tackle the problem of social media engagement, which poses a particular challenge to people with visual disabilities, given the prevalence of image-based content.

Existing solutions address this problem via screen-reader technologies, including programs which describe images simply and literally. Seedat said they wanted to go further, asking themselves, “How could we combine both the image content and the textual content around the image, in order to create a narrative which would allow people to better engage with the content in the posts?’”

They were inspired to develop a web browser extension for people with visual disabilities to use when browsing social media sites. The browser extension “translates” images surrounded by text into a brief description that captures the post’s meaning, not just its literal content. For example, it would translate a photo of a toddler with chocolate smeared all over her face and the caption, “Enjoying her food!” as something like, “Child making a mess with her food” rather than “Child eating food.” This allows people with impaired vision to better understand and meaningfully interact with posts.

Throughout the product development process, the team benefited from feedback from Estrin, their faculty advisor, who encouraged them to develop “not just purely a technological product but also something that can have an impact on people.”

Kolli said Estrin “really helped us figure out our narrative (and) develop an impactful product that has meaning to our users.” Boning Hou, Master in Computer Science ’19, added that Estrin “helped us narrow down the idea and helped us to think about why we are doing what we are doing so that our product will be more helpful to the people that we are trying to help.”

The team has made enormous strides in just one semester. Hou said they have finished a prototype that can parse web content and translate an image into a complete sentence. For now, it works with websites like Instagram and Pinterest; they are in the process of trying to make it compatible with Facebook as well. “Right now, we are planning to make this project open source,” said Hou, “so that more people can contribute to it and make this have a larger impact.”