Microsoft Research Blog

Microsoft Research Blog

The Microsoft Research blog provides in-depth views and perspectives from our researchers, scientists and engineers, plus information about noteworthy events and conferences, scholarships, and fellowships designed for academic and scientific communities.

Project Malmo: Reinforcement learning in a complex world

July 27, 2018 | By Noboru Sean Kuno, Research Program Manager

France’s victory over Croatia in the 2018 FIFA World Cup was as thrilling as sports competition gets. If you’re as much a fan of the game as I am, you enjoyed watching 32 national teams vie for the title over a beautiful month across 11 cities in Russia.

The riveting action taking place on the pitch reminded us of another kind of competition. But this one, instead of football teams, involves software agents. Two years ago, a collaborative cross-Microsoft Research team that includes participants from Microsoft Research in Redmond, Washington, New York City and Cambridge, United Kingdom launched Project Malmo – an open-ended platform to advance the state of the art in AI research, especially reinforcement learning in a complex world. The platform is designed to take what’s possible today and push our research toward more ambitious and more difficult tasks.

Last year we held our first competition, the Malmo Collaborative AI Challenge. It focused on human and software agents working together to tackle certain tasks. The competition attracted many students worldwide and the winners were invited to AI Summer School 2017 hosted by Microsoft Research Cambridge. Winning teams received Azure for Research. An interesting discovery in Cambridge was the sheer diversity of approaches from participants. We were delighted to see students showing various creative approaches and well-designed implementations of their agents. Indeed, in the wake of the competition one of the winning teams from Nanyang Technological University published an AAAI paper on their approach titled, “HogRider: Champion Agent of Microsoft Malmo Collaborative AI Challenge” that is absolutely worth a read.

Today we’re happy to share an additional milestone involving Project Malmo. Microsoft is partnering with Queen Mary University of London and CrowdAI to co-host a second competition, Learning to Play: The Multi-Agent Reinforcement Learning in MalmÖ (MARLÖ) Competition. This competition is a brand-new challenge that proposes research on multi-agent reinforcement learning using multiple games. Participants create learning agents able to play multiple 3D games as defined in the Project Malmo platform. The aim of the competition is to encourage AI research on more general approaches through multi-player games. The challenge will consist of not one but several games, each involving tasks of varying difficulty and settings. This represents a very unique approach.

Diego Perez-Liebana, Lecturer in Computer Games and Artificial Intelligence at Queen Mary University of London, United Kingdom talked about the potential impact of the Malmo competitions on AI research, “Our research group has been running AI game-based competitions for many years and we are well aware of the multiple benefits these bring,” said Perez-Liebana. “They provide a common benchmark for multiple researchers across the globe to train their AI agents in a way that is comparable, allowing us to effectively contrast different techniques in a common domain,” he continued. Game AI competitions are a great resource for education, as they can be proposed as assignment or project from undergraduate to PhD level. “Games are fun, and so is AI,” said Perez-Liebana. Indeed, combining the two helps popularize challenges and solutions faster and more broadly than any other methods. Perez-Liebana pointed to the evolution of the Monte Carlo Tree Search methods during successive Go competitions that led to the use of this method in multiple other games and domains as a clear example of this.

Sharada Prasanna Mohanty, a PhD student at EPFL, Switzerland, co-founder of CrowdAI expressed his expectations regarding the competition. “With this challenge, our principal goal is to make available a series of problems for the community of multi-agent reinforcement learning researchers to collaboratively work on,” said Sharada Mohanty. “With Minecraft as the main platform enabling this research, we also hope to inspire many other researchers and engineers from various domains to get involved in reinforcement learning research. The success of this challenge can help establish these tasks as standard benchmark tasks for all multi-agent reinforcement learning researchers to compare their approaches in the future and at the same time can potentially help us better measure our own progress in multi-agent reinforcement learning research as a community over time.”

The competition is open to anyone worldwide. Visit the competition page for more detail about registration and rules. Qualifying rounds last until October 7. The top 32 teams in the qualifying rounds can move forward to the knockout rounds of the final tournament, where team agents compete each other on an exciting set of games and tasks. The tournament will be a live competition in MARLO workshop at the 14th AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE’18) to be held at the University of Alberta in Edmonton, AB, Canada on November 13-17, 2018. We’re also calling for papers at the workshop.

We hope to see as many students, researchers and engineers as possible share their innovative approaches and creative ideas for multi-agent reinforcement learning at the workshop in Edmonton. Let’s kick-off! You are now in possession of the ball!

Up Next

Malmo Collaborative Challenge

Artificial intelligence

Presenting the winners of the Project Malmo Collaborative AI Challenge

By Noburu Kuno, Senior Research Program Manager; Scarlet Schwiderski-Grosche, Principal Research Program Manager As we move from narrow AI to more general AI, it will be important to instill machines with the ability to work together with both other agents and humans. Project Malmo, which is built on the popular multiplayer game Minecraft, is an […]

Microsoft blog editor

Artificial intelligence

The next challenges for reinforcement learning

Recent years have seen great progress for AI. In particular, artificial agents have learned to classify images and recognize speech at near-human level. However, for artificial agents to reach their full potential, they should not only observe, but also act and learn from the consequences of their actions. Learning how to behave is especially important […]

Microsoft blog editor

Malmo Collaborative AI Challenge

Artificial intelligence

Microsoft challenge: Build a collaborative AI in Minecraft

By John Roach, Writer, Microsoft Research A Microsoft Research team has created a competition using Project Malmo, a platform that uses Minecraft as a testing ground for advanced artificial intelligence research. The team is challenging PhD students to develop an AI that learns to collaborate with other randomly assigned players to achieve a high score […]

Microsoft blog editor