By Allison Linn, Senior Writer at Microsoft
When Yong Rui first began doing computer vision research in 1995, the Internet was still in its infancy and the idea of using technology to automatically search for images seemed almost absurd.
Microsoft Research Podcast
AI Frontiers: Models and Systems with Ece Kamar
Ece Kamar explores short-term mitigation techniques to make these models viable components of the AI systems that give them purpose and shares the long-term research questions that will help maximize their value.
“That [notion] was really quite advanced at the time,” said Rui (opens in new tab), assistant managing director of Microsoft Research Asia.
Rui, who was a doctorate student at the University of Illinois in Champaign-Urbana when he became one of the first researchers to work on the National Science Foundation’s digital libraries project (opens in new tab), an ambitious and forward-thinking attempt to catalog information including visuals electronically. He considers himself lucky to have landed at a university that was chosen for the project—but, it turns out, the computer vision field also was lucky that he fell into this area of research.
On February 22 the Institute of Electrical and Electronics Engineers (IEEE), announced that Rui was among the recipients of the IEEE Computer Society 2016 Technical Achievement Award (opens in new tab). The award recognizes his “pioneering contributions to multimedia analysis and retrieval.”
Hsiao-Wuen Hon, the managing director of Microsoft Research Asia, called the award “a true testament to Yong’s research and development achievements in the fields of multimedia analysis and retrieval over the years.”
“As a brilliant scientist, Yong’s research results in multimedia have inspired a lot of researchers in the academic community,” Hon added.
Rui’s contribution to the field goes back to those days as a Ph.D. student, when he was one of the first researchers to hit on the idea of using a concept called relevance feedback (opens in new tab) in image-based search. Relevance feedback is the idea of refining the results a person gets back from a search engine based on how that person responds to earlier search results. In essence, it’s a way of making search results better based on user behavior.
At the time, lots of people were thinking about using relevance feedback in text search, but he was among the first to introduce it in image search. His early research papers on the topic have since been cited thousands of times (opens in new tab). The work also had a big impact on people outside the research community, because it made it easier for people to search for images and find them.
“That actually turned out to be a paradigm-shifting framework because once we had that [function], the retrieval precision and recall increased significantly,” he said.
In that initial research, Rui used computer vision to try to identify features—such as the shape or color of an object—that would help a search engine identify the image as a vase, a pot, or a bowl. In the years since, his research has expanded into the field of image understanding, which uses computer vision and other technology to both recognize and describe images.
One recent research project created a framework that can automatically describe what is going on in short video clips, using natural language. Rui also has contributed to a number of Microsoft products, including the digital assistant Cortana (opens in new tab), the popular chatbot XiaoIce (opens in new tab) and the set of artificial intelligence-based tools for developers known as Project Oxford (opens in new tab).
Rui also has been actively involved in the research community, serving as editor in chief of the research magazine IEEE MultiMedia, the founding chair of the China chapter of special interest group ACM SIGMultimedia, and an executive member of the China Computing Federation.
Rui, who has worked for Microsoft for nearly 17 years, said one of the advantages of working for the company has been the opportunity to do everything from conducting fundamental scientific research to shipping products to millions of people.
“That’s a huge, huge benefit of working for Microsoft,” he said.
Learn More
Computer vision 
Search and information retrieval