{"id":965577,"date":"2023-05-22T22:38:00","date_gmt":"2023-05-23T05:38:00","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-project&p=965577"},"modified":"2023-12-17T10:09:27","modified_gmt":"2023-12-17T18:09:27","slug":"gaming-interaction","status":"publish","type":"msr-project","link":"https:\/\/www.microsoft.com\/en-us\/research\/project\/gaming-interaction\/","title":{"rendered":"Emergent Interaction Agent"},"content":{"rendered":"\n
\n\t
\n\t\t
\n\t\t\t\t\t<\/div>\n\t\t\n\t\t
\n\t\t\t\n\t\t\t
\n\t\t\t\t\n\t\t\t\t
\n\t\t\t\t\t\n\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\n

MindAgent\uff1aEmerging Gaming Interaction<\/h1>\n\n\n\n

We collaborate with X-Box and Mesh team, explored a new gaming infrastructure and designed the dynamic real-time system for human-player and NPCs with GPT-X in the multi-agent platform.<\/p>\n\n\n\n

GitHub: MindAgent (opens in new tab)<\/span><\/a><\/p>\n\n\n\n

ArXiv: https:\/\/arxiv.org\/abs\/2309.09971 (opens in new tab)<\/span><\/a><\/p>\n\n\n\n

Demo: MindAgent.mp4 (opens in new tab)<\/span><\/a><\/p>\n\n\n\n

<\/p>\n\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t<\/div>\n<\/section>\n<\/div>\n\n\n\n\n\n

Gaming Interaction Infrastructure:<\/p>\n\n\n\n

\"MindAgent\"<\/p>\n\n\n\n\n\n

We are very excited to share the good news. Our project \u201cMindAgent: Emergent Gaming Interaction (opens in new tab)<\/span><\/a>\u201d is public recently. We seek to develop a unified interaction infrastructure and architecture that can jointly: understand large language corpora, visual (image and video) inputs, as well as provide meaningful action-based outputs.  Our model on a broad range of gaming video tasks and show agent action stream efficacy across a range of tasks including interactive agent, visual and natural language understanding. In this work, we propose a novel infrastructure – MindAgent<\/strong> – to evaluate planning and coordination emergent capabilities for gaming interaction. In particular, our infrastructure leverages existing gaming framework, to i) require understanding of the coordinator for a multi-agent system, ii) collaborate with human players via un-finetuned proper instructions, and iii) establish an in-context learning on few-shot prompt with feedback. Furthermore, we introduce CuisineWorld<\/strong>, a new gaming scenario and related benchmark that dispatch a multi-agent collaboration efficiency and supervise multiple agents playing the game simultaneously. We conduct comprehensive evaluations with new auto-metric CoS<\/strong> for calculating the collaboration efficiency. Finally, our infrastructure can be deployed into real-world gaming scenarios in a customized VR version of CuisineWorld and adapted in existing broader Minecraft gaming domain. By creating a powerful and general-purpose foundation model with visual, language, and action capabilities, we can have great impact across many industries, both within Microsoft and external.<\/p>\n\n\n\n

minecraft vr demo – YouTube (opens in new tab)<\/span><\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

We collaborate with X-Box and Mesh team, explored a new gaming infrastructure and designed the dynamic real-time system for human-player and NPCs with GPT-X in the multi-agent platform. GitHub: MindAgent (opens in new tab) ArXiv: https:\/\/arxiv.org\/abs\/2309.09971 (opens in new tab) Demo: MindAgent.mp4 (opens in new tab) Gaming Interaction Infrastructure: We are very excited to share […]<\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"research-area":[13556,13562,13554],"msr-locale":[268875],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-965577","msr-project","type-msr-project","status-publish","hentry","msr-research-area-artificial-intelligence","msr-research-area-computer-vision","msr-research-area-human-computer-interaction","msr-locale-en_us","msr-archive-status-active"],"msr_project_start":"","related-publications":[966927],"related-downloads":[],"related-videos":[],"related-groups":[],"related-events":[],"related-opportunities":[],"related-posts":[],"related-articles":[],"tab-content":[],"slides":[],"related-researchers":[{"type":"guest","display_name":"Hoi Vo","user_id":969933,"people_section":"Related people","alias":""},{"type":"guest","display_name":"Steven Gong","user_id":969942,"people_section":"Related people","alias":""},{"type":"guest","display_name":"Zane Durante","user_id":969945,"people_section":"Related people","alias":""},{"type":"guest","display_name":"Yusuke Noda","user_id":969939,"people_section":"Related people","alias":""},{"type":"guest","display_name":"Song-chun Zhu","user_id":969963,"people_section":"Related people","alias":""},{"type":"guest","display_name":"Demetri Terzopoulos","user_id":969951,"people_section":"Related people","alias":""},{"type":"guest","display_name":"Fei-Fei Li","user_id":969957,"people_section":"Related people","alias":""},{"type":"user_nicename","display_name":"Jianfeng Gao","user_id":32246,"people_section":"Related people","alias":"jfgao"}],"msr_research_lab":[199565],"msr_impact_theme":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/965577","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-project"}],"version-history":[{"count":34,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/965577\/revisions"}],"predecessor-version":[{"id":993414,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/965577\/revisions\/993414"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=965577"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=965577"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=965577"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=965577"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=965577"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}