Aaron Gokaslan, a prominent researcher and PhD student from Cornell Tech, has received the PyTorch Award for 2023. PyTorch’s codebase is a popular deep-learning framework, and the PyTorch Award for Excellence in Code Review is a recognition of Gokaslan’s dedication to maintaining the highest standards in AI technology.

“I am deeply honored to contribute to the PyTorch community and receive the PyTorch Award for Excellence in Code Review. Code review is a vital part of maintaining the quality and reliability of open-source software,” says Gokaslan. “This award reflects the collective efforts of the entire PyTorch community, and I’m grateful for the opportunity to work with such dedicated and talented individuals. I look forward to continuing to ensure that PyTorch remains a trusted and robust platform for AI research and development.”

Gokaslan currently works with Volodymyr Kuleshov, Assistant Professor at the Jacobs Technion-Cornell Institute at Cornell Tech and in the Computer Science Department at Cornell University, and is presently researching open and efficient generative models, specifically looking at how to bring the cost down of training and deploying. He also continues to work on using large language models on DNA sequences and biological data for science, drug discovery, and gene editing.

“The award presented by the PyTorch Foundation is a testament to Aaron’s contributions to the world of machine learning,” says Kuleshov. “Aaron has a significant impact on the PyTorch community, particularly in code review, which plays a critical role in advancing the field of artificial intelligence. His work and research are vital to the growth and strengthening of A.I. infrastructure.”

The post Aaron Gokaslan Receives PyTorch Award for Excellence in Code Review appeared first on Cornell Tech.

Artificial intelligence-powered writing assistants that autocomplete sentences or offer “smart replies” not only put words into people’s mouths, they also put ideas into their heads, according to new research.

Maurice Jakesch, a doctoral student in the field of information science asked more than 1,500 participants to write a paragraph answering the question, “Is social media good for society?” People who used an AI writing assistant that was biased for or against social media were twice as likely to write a paragraph agreeing with the assistant, and significantly more likely to say they held the same opinion, compared with people who wrote without AI’s help.

The study suggests that the biases baked into AI writing tools – whether intentional or unintentional – could have concerning repercussions for culture and politics, researchers said.

“We’re rushing to implement these AI models in all walks of life, but we need to better understand the implications,” said co-author Mor Naaman, professor at the Jacobs Technion-Cornell Institute at Cornell Tech and of information science in the Cornell Ann S. Bowers College of Computing and Information Science. “Apart from increasing efficiency and creativity, there could be other consequences for individuals and also for our society – shifts in language and opinions.”

While others have looked at how large language models such as ChatGPT can create persuasive ads and political messages, this is the first study to show that the process of writing with an AI-powered tool can sway a person’s opinions. Jakesch presented the study, “Co-Writing with Opinionated Language Models Affects Users’ Views,” at the 2023 CHI Conference on Human Factors in Computing Systems in April, where the paper received an honorable mention.

To understand how people interact with AI writing assistants, Jakesch steered a large language model to have either positive or negative opinions of social media. Participants wrote their paragraphs – either alone or with one of the opinionated assistants – on a platform he built that mimics a social media website. The platform collects data from participants as they type, such as which of the AI suggestions they accept and how long they take to compose the paragraph.

People who co-wrote with the pro-social media AI assistant composed more sentences arguing that social media is good, and vice versa, compared to participants without a writing assistant, as determined by independent judges. These participants also were more likely to profess their assistant’s opinion in a follow-up survey.

The researchers explored the possibility that people were simply accepting the AI suggestions to complete the task quicker. But even participants who took several minutes to compose their paragraphs came up with heavily influenced statements. The survey revealed that a majority of the participants did not even notice the AI was biased and didn’t realize they were being influenced.

“The process of co-writing doesn’t really feel like I’m being persuaded,” said Naaman. “It feels like I’m doing something very natural and organic – I’m expressing my own thoughts with some aid.”

When repeating the experiment with a different topic, the research team again saw that participants were swayed by the assistants. Now, the team is looking into how this experience creates the shift, and how long the effects last.

Just as social media has changed the political landscape by facilitating the spread of misinformation and the formation of echo chambers, biased AI writing tools could produce similar shifts in opinion, depending on which tools users choose. For example, some organizations have announced they plan to develop an alternative to ChatGPT, designed to express more conservative viewpoints.

These technologies deserve more public discussion regarding how they could be misused and how they should be monitored and regulated, the researchers said.

“The more powerful these technologies become and the more deeply we embed them in the social fabric of our societies,” Jakesch said, “the more careful we might want to be about how we’re governing the values, priorities and opinions built into them.”

Advait Bhat from Microsoft Research, Daniel Buschek of the University of Bayreuth and Lior Zalmanson of Tel Aviv University contributed to the paper.

Support for the work came from the National Science Foundation, the German National Academic Foundation and the Bavarian State Ministry of Science and the Arts.

Patricia Waldron is a writer for the Cornell Ann S. Bowers College of Computing and Information Science.

This story originally appeared in the Cornell Chronicle.

The post Writing With AI Help Can Shift Your Opinions appeared first on Cornell Tech.

Associate Dean for Impact Deborah Estrin encouraged Michael Byrne to use his dance research to showcase the affordances of the Cornell Tech campus, prompting Byrne to brainstorm with PhD student Fanjun Bu, newly graduated Nialah Wilson-Small, and the Production Glue crew about the ways in which robotics, drones, virtual reality, and choreography could intersect creatively.

The result was Innovación Monumental – a film collaboration with one of New York’s most inventive dance organizations, Ballet Hispánico – in which Artistic Director and CEO Eduardo Vilaro choreographed several vignettes with company members Dandara Veiga and Mariano Zamora, as well as the aforementioned technologies.

“It is a professional thrill to work within a cross-disciplinary institution like Cornell Tech where there is an open invitation to create,” said Michael Byrne, DLI Research Associate and newly appointed Creative Lead for Tech, Arts, and Culture. “This special partnership with Ballet Hispánico allowed us to examine the playful interchange between technology and choreography, revealing a shared institutional passion for innovation – both digital and embodied. The expansive interiors of our campus, as well as the breathtaking surrounds of the nearby memorial, provided Ballet Hispánico’s Eduardo Vilaro with a stage to explore multimodal forms of movement.”

The sublime dancers (and their technological counterparts) can be seen performing Vilaro’s compelling choreography inside Cornell Tech’s Tata Innovation Center and within Four Freedoms Park, both co-located on Roosevelt Island. Projects and partnerships like this highlight Cornell Tech’s ongoing mission to ensure its groundbreaking research and technologies can impact communities and creative industries beyond the campus labs.

“Monumental Innovación also marks the beginning of a longer-term initiative between Cornell Tech, Ballet Hispánico, and other collaborators around history, dance, and digital interventions in civic spaces,” Byrne added excitedly.

Project Credits

Choreography: Eduardo Vilaro, Ballet Hispánico
Dancers: Dandara Veiga and Mariano Zamora, Ballet Hispánico
Creative Direction: Michael Byrne, Cornell Tech
Film Direction: Dom McGee, Production Glue
Producers: Production Glue and Bloomberg Philanthropies

Featured Technologies: Cornell Tech
Drones: Nialah Wilson-Small, Shiri Azenkot, and Kristin H. Petersen
Robotics: Fanjun Bu and Wendy Ju
Mixed Reality: Michael Byrne

Four Freedoms Park Conservancy: Howard Axel and Angela Stangenberg

Music: Santa Maria (Pepe Braddock Mix), La Revancha del Tango (Bonus Track Version), © Gotan Project, 2001 XL Recordings Ltd

A Sample of Behind the Scenes Visuals (10 x photos)

The post Cornell Tech, Ballet Hispánico, and the Four Freedoms Park Conservancy Collaborate to Highlight the Generative Collisions of Dance and Emerging Technology appeared first on Cornell Tech.

A Cornell-led collaboration used machine learning to pinpoint the most accurate means, and timelines, for anticipating the advancement of Alzheimer’s disease in people who are either cognitively normal or experiencing mild cognitive impairment.

The modeling showed that predicting the future decline into dementia for individuals with mild cognitive impairment is easier and more accurate than it is for cognitively normal, or asymptomatic, individuals. At the same time, the researchers found that the predictions for cognitively normal subjects is less accurate for longer time horizons, but for individuals with mild cognitive impairment, the opposite is true.

The modeling also demonstrated that magnetic resonance imaging (MRI) is a useful prognostic tool for people in both stages, whereas tools that track molecular biomarkers, such as positron emission tomography (PET) scans, are more useful for people experiencing mild cognitive impairment.

The team’s paper, “Machine Learning Based Multi-Modal Prediction of Future Decline Toward Alzheimer’s Disease: An Empirical Study,” published Nov. 16 in PLOS ONE. The lead author is Batuhan Karaman, a doctoral student in the field of electrical and computer engineering.

Alzheimer’s disease can take years, sometimes decades, to progress before a person exhibits symptoms. Once diagnosed, some individuals decline rapidly but others can live with mild symptoms for years, which makes forecasting the rate of the disease’s advancement a challenge.

“When we can confidently say someone has dementia, it is too late. A lot of damage has already happened to the brain, and it’s irreversible damage,” said senior author Mert Sabuncu, associate professor of electrical and computer engineering in the College of Engineering and Cornell Tech, and of electrical engineering in radiology at Weill Cornell Medicine.

“We really need to be able to catch Alzheimer’s disease early on,” Sabuncu said, “and be able to tell who’s going to progress fast and who’s going to progress slower, so that we can stratify the different risk groups and be able to deploy whatever treatment options we have.”

Clinicians often focus on a single “time horizon” – usually three or five years – to predict Alzheimer’s progression in a patient. The timeframe can seem arbitrary, according to Sabuncu, whose lab specializes in analysis of biomedical data – particularly imaging data, with an emphasis on neuroscience and neurology.

Sabuncu and Karaman partnered with longtime collaborator and co-author Elizabeth Mormino of Stanford University to use neural-network machine learning that could analyze five years’ worth of data about individuals who were either cognitively normal or had mild cognitive impairment. The data, captured in a study by the Alzheimer’s Disease Neuroimaging Initiative, encompassed everything from an individual’s genetic history to PET and MRI scans.

“What we were really interested in is, can we look at these data and tell whether a person will progress in upcoming years ?” Sabuncu said. “And importantly, can we do a better job in forecasting when we combine all the follow-up datapoints we have on individual subjects?”

The researchers discovered several notable patterns. For example, predicting a person will move from being asymptomatic to exhibiting mild symptoms is much easier for a time horizon of one year, compared to five years. However, predicting if someone will decline from mild cognitive impairment into Alzheimer’s dementia is most accurate on a longer timeline, with the “sweet spot” being about four years.

“This could tell us something about the underlying disease mechanism, and how temporally it is evolving, but that’s something we haven’t probed yet,” Sabuncu said.

Regarding the effectiveness of different types of data, the modeling showed that MRI scans are most informative for asymptomatic cases and are particularly helpful for predicting if someone’s going to develop symptoms over the next three years, but less helpful for forecasting for people with mild cognitive impairment. Once a patient has developed mild cognitive impairment, PET scans, which measure certain molecular markers such as the proteins amyloid and tau, appear to be more effective.

One advantage of the machine learning approach is that neural networks are flexible enough that they can function despite missing data, such as patients who may have skipped an MRI or PET scan.

In future work, Sabuncu plans to modify the modeling further so that it can process complete imaging or genomic data, rather than just summary measurements, to harvest more information that will boost predictive accuracy.

The research was supported by the National Institutes of Health National Library of Medicine and National Institute on Aging, and the National Science Foundation.

Many Weill Cornell Medicine physicians and scientists maintain relationships and collaborate with external organizations to foster scientific innovation and provide expert guidance. The institution makes these disclosures public to ensure transparency. For this information, see profile for Dr. Sabuncu.

This story originally appeared in the Cornell Chronicle.

The post Machine Learning Gives Nuanced View of Alzheimer’s Stages appeared first on Cornell Tech.

Personal sensing data could help monitor and alleviate stress among resident physicians, although privacy concerns over who sees the information and for what purposes must be addressed, according to collaborative research from Cornell Tech.

Burnout in all types of workplaces is on the rise in the U.S., where the “Great Resignation” and “silent quitting” have entered the lexicon in recent years. This is especially true in the health care industry, which has been strained beyond measure due to the COVID-19 pandemic.

Stress is physical as well as mental, and evidence of stress can be measured through the use of smartphones, wearables and personal computers. But data collection and analysis – and the larger questions of who should have access to that information, and for what purpose – raise myriad sociotechnical questions.

“We’ve looked at whether we can measure stress in workplaces using these types of devices, but do these individuals actually want this kind of system? That was the motivation for us to talk to those actual workers,” said Daniel Adler, co-lead author with fellow doctoral student Emily Tseng of “Burnout and the Quantified Workplace: Tensions Around Personal Sensing Interventions for Stress in Resident Physicians,” published Nov. 11 Proceedings of the ACM on Human-Computer Interaction.

The paper is being presented at the ACM Conference on Computer-Supported Cooperative Work (CSCW) and Social Computing, taking place virtually Nov. 8-22.

Adler and Tseng worked with senior author Tanzeem Choudhury, the Roger and Joelle Burnell Professor in Integrated Health and Technology at the Jacobs Technion-Cornell Institute at Cornell Tech. Contributors came from Zucker School of Medicine at Hofstra/Northwell Health and Zucker Hillside Hospital.

The resident physician’s work environment is a bit different from the traditional apprenticeship situation in that their supervisor, the attending physician, is also their mentor. That can blur the lines between the two.

“That’s a new context,” Tseng said. “We don’t really know what the actual boundaries are there, or what it looks like when you introduce these new technologies, either. So you need to try and decide what those norms might be to determine whether this information flow is appropriate in the first place.”

Choudhury and her group addressed these issues through a study involving resident physicians at an urban hospital in New York City. After hourlong interviews with residents on Zoom, the residents and their attendings were given mockups of a Resident Wellbeing Tracker, a dashboard with behavioral data on residents’ sleep, activity and time working; self-reported data on residents’ levels of burnout; and a text box where residents could characterize their well-being.

Tseng said the residents were open to the idea of using technology to enhance well-being. “They were also very interested in the privacy question,” she said, “and how we could use technologies like this to achieve those positive ends while still balancing privacy concerns.”

The study featured two intersecting use cases: self-reflection, in which the residents view their behavioral data, and data sharing, in which the same information is shared with their attendings and program directors for purposes of intervention.

Among the key findings: Residents were hesitant to share their data without the assurance that supervisors would use it to enhance their well-being. There is also a question of anonymity, which was more likely with more participation. But greater participation would hurt the potential usefulness of the program, since supervisors would not be able to identify which residents were struggling.

“This process of sharing personal data is somewhat complicated,” Adler said. “There is a lot of interesting continuing work that we’re involved in that looks at this question of privacy, and how you present yourself through your data in more-traditional mental health care settings. It’s not as simple as, ‘They’re my doctor, therefore I’m comfortable sharing this data.’”

The authors conclude by referring to the “urgent need for further work establishing new norms around data-driven workplace well-being management solutions that better center workers’ needs, and provide protections for the workers they intend to support.”

Other contributors included Emanuel Moss, a postdoctoral researcher at Cornell Tech; David Mohr, a professor in the Feinberg School of Medicine at Northwestern University; as well as Dr. John Kane, Dr. John Young and Dr. Khatiya Moon from Zucker Hillside Hospital.

The research was supported by grants from the National Institute of Mental Health, the National Science Foundation and the Digital Life Initiative at Cornell Tech.

This story originally appeared in the Cornell Chronicle.

The post Personal Sensing at Work: Tracking Burnout, Balancing Privacy appeared first on Cornell Tech.

In a new paper, a team from Cornell Tech in collaboration with the Swiss Federal Institute of Technology Lausanne (EPFL) recommends that, if YouTube wants to truly impact these creators, they develop a shared database of demonetized users in conjunction with Patreon, Twitch, and other alternative monetization sites, to prevent them from using each other’s platforms.

The Cornell Tech team reviewed 71 million videos on YouTube that were published by more than 136,000 popular content creators with more than 10,000 subscribers to understand how creators, including channels that distribute problematic content, employ alternative monetization strategies that could allow them to circumvent the effects of any “demonetization” by YouTube.

In their new paper the researchers found that, when compared to random channels of similar activity, popularity, age, and with similar content fringe content creators are:

• more likely to adopt alternative monetization
• use alternative monetization methods more frequently
• more likely to diversify their alternative monetization efforts

“We found that channels that establish alternative monetization strategies actually become more productive on the platform,” said Cornell Tech doctoral researcher Yiqing Hua, co-lead author on the new paper alongside Cornell Tech professors Thomas Ristenpart and Mor Naaman. A collaboration with Manoel Horta Ribeiro and Robert West of the Swiss Federal Institute of Technology in Lausanne, the paper will be presented this November at the annual ACM Conference on Computer-Supported Cooperative Work And Social Computing.

The researchers learned that creators who produce problematic content thrive from the attention they get from their supporters through alternative monetization. Looking at even just a small sliver of the overall YouTube analytics, Hua found that at least a dozen fringe channels have made more than $100,000 on Patreon alone.

While the problem is not limited to YouTube and Patreon, the two platforms have an outsized influence in this space. The new paper shows that 61 percent of fringe channels use an alternative monetization strategy, compared with 18 percent of channels overall.

The team generally found that the practice of demonetization on YouTube is less effective because of the opportunities to employ alternative monetization strategies, citing Alex Jones’ InfoWars YouTube channel as a high-profile example. Before the channel’s ban in late 2018, the channel featured over 30,000 videos and gathered more than two million subscribers. Despite being demonetized during this period, Jones still managed to amass millions of dollars each year through affiliate links and alternative monetization strategies. This paper shows that many fringe content creators benefit from alternative monetization and are able to maintain an income while producing content.

“We were surprised to discover how much money these creators are making from alternative monetization platforms,” Hua said. “Creators make money on YouTube through engagement, including number of views and minutes watched, but fringe creators that are demonetized focus on ensuring their fans and followers want to support and pay for their work.”

However, the researchers suggested that alternative monetization should not be banned from the platform, as these strategies also empower creators who are often in a vulnerable position when YouTube policies become ambiguous. Alternative monetization also allows for different incentives that may encourage higher-quality content compared to the ad-revenue model.

This research was funded in part by the Siegel Family Endowment and by the National Science Foundation.

The post How YouTube’s Demonetization Strategy Fails — and What They Can Do About It appeared first on Cornell Tech.

This year’s course was taught by Sadik Antwi-Boampong, Johnson Cornell Tech MBA ‘20. With an additional background in the tech industry as a Research & Development Engineer at Intel, Antwi-Boampong is well-versed in both web literacy and internet health issues.

“I’m quite passionate about teaching and education — I taught classes in undergrad and graduate school, and also co-founded an education non-profit in Ghana called Lead For Ghana — so I thought that this was an excellent opportunity to engage with the Roosevelt Island community,” said Antwi-Boampong, who jumped at the opportunity to connect with locals as soon as he discovered the program. “There are many interesting topics in tech that are relevant to seniors so, in addition to increasing their digital awareness, I wanted to include them in the broader conversation in tech.”

The Web Literacy Course focuses on teaching the fundamentals of the internet and giving an overall understanding of the web, in addition to covering topics such as internet health, digital inclusion, and internet safety. The program’s goal is to expand the participants’ online awareness and teach them digital tools they can apply in their daily lives — empowering them to engage with new technology in a safe way.

During the first class, which covered what the internet is and why internet health is important, Antwi-Boampong asked his students what they hoped to learn from this course. The responses ranged from a desire to fine-tune their internet skills to more granular issues, such as personal security, online safety, and managing digital finances. Despite varying motivations per individual, the entire class had one common goal: To learn how to use the internet effectively and safely so they can more comfortably fit into today’s world.

Topics for this year’s course included skills such as navigating the web, using advanced search options, creating and sharing content on blogs, and using collaborative tools like Google Drive. Additionally, useful information regarding online security and privacy, data tracking, avoiding scams, and being a digitally-aware citizen were discussed. The schedule was also flexibly designed to be tailored to the personal needs and interests of those involved in the course.

Now in the third year of the course, past participants have been able to use the skills they learned in their everyday lives.

Jay Jacobson, a participant of the 2017 course taught by Cornell Tech PhD candidate Vibhore Vardhan, found the course to be an excellent “hands-on” primer and extremely helpful in his day-to-day life.

“The Cornell Tech Web Literacy Course did what I hoped it would do,” Jacobson said. “It gave me an elementary understanding of what the web is, how the internet functions as a part of it, and how I could connect myself to the web to do things that I want to do… The course helped me learn how to do things on the web more efficiently and, in almost all cases, much more safely.”

The post Cornell Tech MBA Student Teaches Roosevelt Island Seniors Safe Web Navigation appeared first on Cornell Tech.

The Microsoft Research Ada Lovelace Fellowship is a three-year fellowship that aims to increase the pipeline of diverse talent receiving advanced degrees in computing-related fields by providing a research funding opportunity for doctoral students who are underrepresented in the field of computing.

This year, there were over 600 submissions between Microsoft’s Ada Lovelace Fellowship and PhD Fellowship. Of these submissions, five students were awarded the 2020 Microsoft Research Ada Lovelace Fellowship. Recipients were awarded complete coverage of tuition and fees for three academic years, a $42,000 stipend, an invitation to interviews for a Microsoft internship, and an invitation to the Microsoft PhD Summit.

The post PhD Student Wenqi Xian Receives 2020 Microsoft Research Ada Lovelace Fellowship appeared first on Cornell Tech.

This November, Cornell scholars converged on Austin, Texas, for a showing at CSCW. Cornell’s CSCW contributions this year include 15 refereed papers, a Best Paper, three Honorable Mentions, and much more.

Maggie Jack (Ph.D. student) and Nicola Dell (assistant professor at the Jacobs Technion-Cornell Institute) received the Diversity and Inclusion award for their paper titled “‘Privacy is not a concept, but a way of dealing with life:’ Localization of Transnational Technology Platforms and Liminal Privacy Practices in Cambodia.”

Additionally, Assistant Professor Nicola Dell, Associate Professor Thomas Ristenpart, and Associate Professor Tapan Parikh received Honorable Mention Awards for their respective papers.

Cornell Tech and Jacobs Technion-Cornell Institute faculty and researchers contributed six out of fifteen papers accepted to the conference from Cornell:

*Diversity and Inclusion Award Winner* “Privacy is not a concept, but a way of dealing with life:” Localization of Transnational Technology Platforms and Liminal Privacy Practices in Cambodia

Authors: Margaret C. Jack (Cornell University), Pang Sovannaroth (Independent Researcher), and Nicola Dell (Cornell Tech/Jacobs Institute)

*Honorable Mention Award* “Is my phone hacked?” Analyzing Clinical Computer Security Interventions with Survivors of Intimate Partner Violence

Authors: Diana Freed, Sam Havron, Emily Tseng, Andrea Gallardo, Rahul Chatterjee, Thomas Ristenpart, and Nicola Dell (all of Cornell Tech)

*Honorable Mention Award* Computing Education for Intercultural Learning: Lessons from the Nairobi Play Project

Authors: Ian Arawjo (Cornell University), Ariam Mogos (Nairobi Play Project), Steven Jackson (Cornell University), Tapan Parikh (Cornell Tech), and Kentaro Toyama (University of Michigan) 

Accessible Video Calling: Enabling Non-visual Perception of Visual Conversation Cues

Authors: Lei Shi (Cornell Tech), Brianna J. Tomlinson (Georgia Tech), John Tang (Microsoft), Edward Cutrell (Microsoft Research), Daniel McDuff (Microsoft), Gina Venolia (Microsoft), Paul Johns (Microsoft Research), and Kael Rowan (Microsoft Research)

How Teachers in India Reconfigure their Work Practices around a Teacher-Oriented Technology Intervention

Authors: Rama Adithya Varanasi (Cornell Tech), René F. Kizilcec (Cornell University), and Nicola Dell (Cornell Tech/Jacobs Institute)

Including the Voice of Care Recipients in Community Health Feedback Loops in Rural Kenya

Authors: Fabian Okeke (Cornell Tech), Beatrice Wasunna (Medic Mobile), Mercy Amulele (Medic Mobile), Isaac Holeman (Medic Mobile), and Nicola Dell (Cornell Tech/Jacobs Institute)

Read the full list of Cornell University papers at CSCW.

The post Cornell Tech Faculty and PhD Students Bring 6 Papers to Annual CSCW Conference appeared first on Cornell Tech.

On secondhand marketplaces like eBay, people trust online sellers who post their own high-quality photos of items for sale more than they trust those who use stock images or poor-quality photos, a Cornell Tech study has found.

The findings could help online marketplaces improve trust in their sites by offering guidelines on how to take better photos or introducing augmented reality features that instruct users to change lighting or camera angles, the researchers said.

This is particularly important for new or growing sites that are working to establish trust, said Xiao Ma, lead author of “Understanding Image Quality and Trust in Peer-to-Peer Marketplaces,” to be presented at WACV 2019, Jan. 7-11 in Waikoloa Village, Hawaii.  

“The high-quality product images selected by our model automatically outperform stock images in generating perceptions of trustworthiness,” said Ma, a doctoral student in the field of information science at Cornell Tech. “People believe the user-generated images represent the actual condition of the product better. Stock images present more uncertainty and raise questions such as whether they are too good to be true.”

The study, co-authored with Mor Naaman, associate professor of information science at the Jacobs Technion-Cornell Institute at Cornell Tech, and Serge Belongie, professor of computer science at Cornell Tech, as well as colleagues at École Polytechnique, Google Research and eBay, grew out of previous work on trust in Cornell Tech’s Connected Experiences Lab. Trust is essential for society to function, but research shows levels of trust have been declining in recent years. Establishing and building trust in digital environments is even more complex.

“Without face-to-face interactions there is a lot of uncertainty,” Ma said. “You don’t know what’s going on on the other side of the screen. People could lie; they could post different images. In the beginning of e-commerce there were a lot of studies on trust, but that was limited because our ability to understand images and language computationally has been limited. Now, because of computer vision and natural language processing, we’re able to understand a lot of these online interactions better, and there’s an opportunity to revisit these questions of online trust.”

For this study, the researchers used publicly available data from the mobile classifieds app Letgo.com and private data from eBay. They focused on shoes and handbags because they’re among the most popular goods found on secondhand marketplaces, and because they are visually distinctive enough to pose an interesting computer vision challenge.

Using the images, they developed a deep learning algorithm – a kind of artificial intelligence frequently used for classification tasks – to predict image quality. They found the algorithm to be around 87 percent accurate, but because of the way deep learning works, researchers could not tell how the model arrived at its decisions. To learn more about which elements improve an image, they also analyzed images using classic computer vision methods and linear regression, a kind of statistical modeling.

Among their findings: Images are more likely to be labeled high quality if they are brighter, and less likely if they have a high foreground to background ratio. A good-quality image should have high contrast for the product and low contrast for the background, they found.

Once they had established methods of predicting quality, the researchers investigated its impact on sales and consumer trust. Shoes with higher-quality images were found to be 1.17 times more likely to be sold than those with lower-quality images, and handbags with better photos were 1.25 times more likely to sell. But because sales are also subject to factors such as price, further research is needed, Ma said.

To test trustworthiness, the researchers designed three hypothetical marketplaces and populated one of them with high-quality images, one with low-quality images and one with stock photos. They recruited 300 people to rate the marketplaces from one to five on a series of statements gauging trust.

The site with the better images scored the highest, with participants rating it around 3.8 for the statement “I believe that the products from these sellers will meet my expectations when delivered,” compared with around 3.7 for the site using stock imagery and 3.4 for the site with low-quality images.

The results – which surprised researchers, who did not expect high-quality personal images to perform better than stock imagery – could be especially helpful to new sites, which might introduce features to improve the quality of users’ photos. For example, instead of automatically using the first uploaded image as a thumbnail, apps could use an algorithm to choose the best-quality image. The research could also be applicable to other kinds of sites, such as real estate or dating.

“Digital environments create new challenges and opportunities for different types of trust,” Ma said. “A lot of the challenge in starting online platforms is in gaining the users’ trust to have people adopt it.”

The study was partly funded by a Facebook equipment donation and by Oath, which is part of Verizon, and Yahoo Research through the Connected Experiences Lab.

The post Is seeing believing? Depends on photo quality, study says appeared first on Cornell Tech.