{"id":1081755,"date":"2024-09-26T05:15:00","date_gmt":"2024-09-26T12:15:00","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=1081755"},"modified":"2025-01-14T13:38:56","modified_gmt":"2025-01-14T21:38:56","slug":"microsoft-research-forum-episode-4-the-future-of-multimodal-models-a-new-small-language-model-and-other-ai-updates","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/microsoft-research-forum-episode-4-the-future-of-multimodal-models-a-new-small-language-model-and-other-ai-updates\/","title":{"rendered":"Microsoft Research Forum Episode 4: The future of multimodal models, a new \u201csmall\u201d language model, and other AI updates"},"content":{"rendered":"\n

Microsoft Research Forum is a continuous exchange of ideas about science and technology research in the era of general AI. In the latest episode<\/a>,\u202fresearchers discussed the latest multimodal AI models, advanced benchmarks for AI evaluation and model self-improvement, and an entirely new kind of computer for AI inference and hard optimization. Researchers at Microsoft are working to explore breakthrough technology that can help advance everything from weather prediction to materials design.\u00a0<\/p>\n\n\n\n

Below is a brief recap of the event, including select quotes from the presentations. Register (opens in new tab)<\/span><\/a> to join future Research Forum episodes and view previous sessions (opens in new tab)<\/span><\/a>. Transcripts and additional resources can be found in the Research Forum briefing book<\/a>.<\/p>\n\n\n\n

Keynote<\/h2>\n\n\n\n

Phi-3-Vision: A highly capable and \u201csmall\u201d language vision model<\/a><\/h3>\n\n\n\n
\"Research<\/a><\/figure>\n\n\n\n

Jianfeng Gao introduced Phi-3-Vision, an advanced and economical open-source multimodal model. As a member of the Phi-3 model family, Phi-3-Vision enhances language models by integrating multisensory skills, seamlessly combining language and vision capabilities.<\/p>\n\n\n\n

\n

“Phi-3-Vision is the first multimodal model in the Phi small model family. It matches and sometimes exceeds some of the capabilities of much larger models \u2026 at a much lower cost. And to help everyone build more affordable and accessible AI systems, we have released the model weights into the open-source community.”<\/p>\n\u2014 <\/em>Jianfeng Gao<\/a><\/em><\/strong>, Distinguished Scientist and Vice President, Microsoft Research Redmond<\/em><\/cite><\/blockquote>\n\n\n\n

\n\n\n\n

Panel Discussion<\/h2>\n\n\n\n

Beyond language: The future of multimodal models in healthcare, gaming, and AI<\/a><\/h3>\n\n\n\n
\"Research<\/a><\/figure>\n\n\n\n

This discussion examined the transformative potential and core challenges of multimodal models across various domains, including precision health, game intelligence, and foundation models. Microsoft researchers John Langford<\/a>, Hoifung Poon<\/a>, Katja Hofmann<\/a>, and Jianwei Yang<\/a> shared their thoughts on future directions, bridging gaps, and fostering synergies within the field.\u202f<\/p>\n\n\n\n

\u201cOne of the really cutting-edge treatments for cancer these days is immunotherapy. That works by mobilizing the immune system to fight the cancer. And then one of the blockbuster drugs is a KEYTRUDA, that really can work miracles for some of the late- stage cancers … Unfortunately, only 20 to 30 percent of the patients actually respond. So that’s \u2026 a marquee example of what are the growth opportunity in precision health.\u201d
\u2014 <\/sub><\/em>Hoifung Poon<\/a><\/em><\/sub><\/strong>, General Manager, Microsoft Research Health Futures<\/sub><\/em><\/p>\n\n\n\n

\u201cWe experience the world through vision, touch, and all our other senses before we start to make sense of any of the language that is spoken around us. So, it’s really, really interesting to think through the implications of that, and potentially, as we start to understand more about the different modalities that we can model and the different ways in which we combine them.\u201d
\u2014 <\/sub><\/em>Katja Hofmann<\/a><\/em><\/sub><\/strong>, Senior Principal Researcher, Microsoft Research<\/sub><\/em><\/p>\n\n\n\n

\u201cTo really have a capable multimodal model, we need to encode different information from different modalities, for example, from vision, from language, from even audio, speech, etc. We need to develop a very capable encoder for each of these domains and then \u2026 tokenize each of these raw data.\u201d
\u2014 <\/sub><\/em>Jianwei Yang<\/a><\/em><\/sub><\/strong>, Principal Researcher, Microsoft Research Redmond<\/sub><\/em><\/p>\n\n\n\n

\n\n\n\n

Lightning Talks<\/h2>\n\n\n\n

Analog optical computing for sustainable AI and beyond<\/a><\/h3>\n\n\n\n
\"Research<\/a><\/figure>\n\n\n\n

This talk presented a new kind of computer\u2014an analog optical computer\u2014that has the potential to accelerate AI inference and hard optimization workloads by 100x, leveraging hardware-software co-design to improve the efficiency and sustainability of real-world applications.\u202f<\/p>\n\n\n\n

\u201cMost likely, you or your loved ones have been inside an MRI scan \u2014 <\/em>not really a great place to be in. Imagine if you can reduce that amount of time from 20 to 40 minutes to less than five minutes.\u201d
\u2014 <\/em>Francesca Parmigiani<\/a><\/em><\/strong>, Principal Researcher, Microsoft Research Cambridge<\/em>\u202f<\/sub><\/p>\n\n\n\n

\u201cI’m really excited to share that we have just completed the second generation of [this] computer. It is much smaller in physical size, and this is a world first in that exactly the same computer is simultaneously solving hard optimization problems and accelerating machine learning inference. Looking ahead, we estimate that at scale, this computer can achieve around 450 tera operations per second per watt, which is a 100-times improvement as compared to state-of-the-art GPUs.\u201d
\u2014 <\/sub><\/em>Jiaqi Chu<\/a><\/em><\/sub><\/strong>, Principal Researcher, Microsoft Research Cambridge<\/sub><\/em><\/p>\n\n\n\n

\n\n\n\n

Direct Nash Optimization: Teaching language models to self-improve with general preferences<\/a><\/h3>\n\n\n\n
\"Research<\/a><\/figure>\n\n\n\n

This talk explored teaching language models to self-improve using AI preference feedback, challenging the model to play against itself and a powerful teacher until it arrives at a Nash equilibrium, resulting in state-of-the-art win rates against GPT-4 Turbo on benchmarks such as AlpacaEval and MT-Bench.\u202f<\/p>\n\n\n\n

\u201cThe traditional way to fine-tune an LLM for post-training \u2026 basically tells the model to emulate good behaviors, but it does not target or correct any mistakes or bad behaviors that it makes explicitly. \u2026 Self-improving post-training explicitly identifies and tries to correct bad behaviors or mistakes that the model makes.\u201d
\u2014 Corby Rosset<\/a><\/strong>, Senior Researcher, Microsoft Research AI Frontiers<\/em><\/sub><\/p>\n\n\n\n

\n\n\n\n

Project Aurora: The first large-scale foundation model of the atmosphere<\/a><\/h3>\n\n\n\n
\"Research<\/a><\/figure>\n\n\n\n

This talk presented Aurora, a cutting-edge foundation model that offers a new approach to weather forecasting that could transform our ability to predict and mitigate the impacts of extreme events, air pollution, and the changing climate.<\/p>\n\n\n\n

\u201cIf we look at Aurora’s ability to predict pollutants such as nitrogen dioxide that are strongly related to emissions from human activity, we can see that the model has learned to make these predictions with no emissions data provided. It’s learned the implicit patterns that cause the gas concentrations, which is very impressive.\u201d
\u2014 Megan Stanley<\/a><\/strong>, Senior Researcher, Microsoft Research AI for Science<\/em><\/sub><\/p>\n\n\n\n

\n\n\n\n

A generative model of biology for in-silico experimentation and discovery<\/a><\/h3>\n\n\n\n
\"Research<\/a><\/figure>\n\n\n\n

This talk explored how deep learning enables generation of novel and useful biomolecules, allowing researchers and practitioners to better understand biology.\u202fThis includes EvoDiff, a general-purpose diffusion framework that combines evolutionary-scale data with the distinct conditioning capabilities of diffusion models to generate new proteins, given a protein sequence.<\/p>\n\n\n\n

\u201cOften, protein engineers want proteins that perform a similar function to a natural protein, or they want to produce a protein that performs the same function but has other desirable properties, such as stability. By conditioning EvoDiff with a family of related sequences, we can generate new proteins that are very different in sequence space to the natural proteins but are predicted to fold into similar three-dimensional structures. These may be good starting points for finding new functions or for discovering versions of a protein with desirable properties.\u201d
\u2014 Kevin Yang<\/a><\/strong>, Senior Researcher, Microsoft Research New England<\/em><\/sub><\/p>\n\n\n\n

\n\n\n\n

Fostering appropriate reliance on AI<\/a><\/h3>\n\n\n\n
\"Research<\/a><\/figure>\n\n\n\n

Since AI systems are probabilistic, they can make mistakes. One of the main challenges in human-AI interaction is to avoid overreliance on AI and empower people to determine when to accept or not accept an AI system’s recommendation. This talk explores Microsoft\u2019s work in this area.<\/p>\n\n\n\n

\u201cThis is where I think it is our responsibility as people working in UX disciplines\u2014as people researching UX and human-computer interaction\u2014to really, really step up to the front and see how it is our moment to shine and to address this problem.\u201d
\u2014 Mihaela Vorvoreanu<\/a><\/strong>, Director UX Research and Responsible AI Education, Microsoft AI Ethics and Effects in Engineering and Research (Aether)<\/em><\/sub><\/p>\n","protected":false},"excerpt":{"rendered":"

Explore multimodal & small language models, plus advanced benchmarks for AI evaluation. Microsoft researchers are working on breakthroughs in weather prediction, materials design, even a new kind of computer for AI inference and hard optimization problems.<\/p>\n","protected":false},"author":42735,"featured_media":1079448,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":[{"type":"user_nicename","value":"Jianfeng Gao","user_id":"32246"},{"type":"user_nicename","value":"John Langford","user_id":"32204"},{"type":"user_nicename","value":"Hoifung Poon","user_id":"32016"},{"type":"user_nicename","value":"Katja Hofmann","user_id":"32468"},{"type":"user_nicename","value":"Jianwei Yang","user_id":"40261"},{"type":"user_nicename","value":"Corby Rosset","user_id":"41997"},{"type":"user_nicename","value":"Megan Stanley","user_id":"41482"},{"type":"user_nicename","value":"Kevin Kaichuang Yang","user_id":"39093"},{"type":"user_nicename","value":"Mihaela Vorvoreanu","user_id":"36804"}],"msr_hide_image_in_river":null,"footnotes":""},"categories":[1],"tags":[],"research-area":[13556],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[269148,243984,269142],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-1081755","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-artificial-intelligence","msr-locale-en_us","msr-post-option-approved-for-river","msr-post-option-blog-homepage-featured","msr-post-option-include-in-river"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[199561,199563,199565,849856,851467,992148],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[],"related-events":[],"related-researchers":[{"type":"user_nicename","value":"Jianfeng Gao","user_id":32246,"display_name":"Jianfeng Gao","author_link":"Jianfeng Gao<\/a>","is_active":false,"last_first":"Gao, Jianfeng","people_section":0,"alias":"jfgao"},{"type":"user_nicename","value":"John Langford","user_id":32204,"display_name":"John Langford","author_link":"John Langford<\/a>","is_active":false,"last_first":"Langford, John","people_section":0,"alias":"jcl"},{"type":"user_nicename","value":"Hoifung Poon","user_id":32016,"display_name":"Hoifung Poon","author_link":"Hoifung Poon<\/a>","is_active":false,"last_first":"Poon, Hoifung","people_section":0,"alias":"hoifung"},{"type":"user_nicename","value":"Katja Hofmann","user_id":32468,"display_name":"Katja Hofmann","author_link":"Katja Hofmann<\/a>","is_active":false,"last_first":"Hofmann, Katja","people_section":0,"alias":"kahofman"},{"type":"user_nicename","value":"Corby Rosset","user_id":41997,"display_name":"Corby Rosset","author_link":"Corby Rosset<\/a>","is_active":false,"last_first":"Rosset, Corby","people_section":0,"alias":"corbyrosset"},{"type":"user_nicename","value":"Kevin Kaichuang Yang","user_id":39093,"display_name":"Kevin Kaichuang Yang","author_link":"Kevin Kaichuang Yang<\/a>","is_active":false,"last_first":"Yang, Kevin Kaichuang","people_section":0,"alias":"kevyan"},{"type":"user_nicename","value":"Mihaela Vorvoreanu","user_id":36804,"display_name":"Mihaela Vorvoreanu","author_link":"Mihaela Vorvoreanu<\/a>","is_active":false,"last_first":"Vorvoreanu, Mihaela","people_section":0,"alias":"mivorvor"}],"msr_type":"Post","featured_image_thumbnail":"\"Research","byline":"","formattedDate":"September 26, 2024","formattedExcerpt":"Explore multimodal & small language models, plus advanced benchmarks for AI evaluation. Microsoft researchers are working on breakthroughs in weather prediction, materials design, even a new kind of computer for AI inference and hard optimization problems.","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1081755","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/42735"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=1081755"}],"version-history":[{"count":50,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1081755\/revisions"}],"predecessor-version":[{"id":1120479,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1081755\/revisions\/1120479"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/1079448"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=1081755"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=1081755"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=1081755"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=1081755"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=1081755"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=1081755"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=1081755"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=1081755"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=1081755"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=1081755"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=1081755"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}