{"id":631245,"date":"2020-01-16T11:12:18","date_gmt":"2020-01-16T19:12:18","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=631245"},"modified":"2020-02-10T07:07:00","modified_gmt":"2020-02-10T15:07:00","slug":"by-making-text-based-games-more-accessible-to-rl-agents-jericho-framework-opens-up-exciting-natural-language-challenges","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/by-making-text-based-games-more-accessible-to-rl-agents-jericho-framework-opens-up-exciting-natural-language-challenges\/","title":{"rendered":"By making text-based games more accessible to RL agents, Jericho framework opens up exciting natural language challenges"},"content":{"rendered":"

\"animated<\/p>\n

You\u2019re in a field. In front of you, there\u2019s a white house. The door is boarded shut. The immediate challenge\u2014investigate the house. The game\u2014Zork I: The Great Underground Empire,<\/em> a treasure-seeking adventure in which you\u2019ll encounter monsters, a thief, and other interesting characters along the way.<\/p>\n

As a player of this text-based game, you string together simple commands of only several words, like \u201cwalk to the house.\u201d Once there, you type a series of commands, not all of them fruitful, to circle the house until you find a way in. There is a window ajar. You \u201copen the window\u201d and \u201center the house,\u201d the adventure truly beginning. To continue between rooms within the house and locations beyond and to interact with the objects you find, you rely on your ability to recall prior information like realizing the lantern you need to safely explore underground is in a previous location and commonsense knowledge to know what to do when you find it.<\/p>\n

Interactive fiction (IF) games such as Zork<\/em> provide great environments for reinforcement learning agents to hone their natural language understanding and generation. However, without the commonsense knowledge and effective recall human players possess, they\u2019re faced with a task far more monumental than ours\u2014choosing from billions<\/em> of possible actions. For example, an agent generating a modest four-word command from a provided vocabulary of 700 words is effectively navigating a space of size 7004<\/sup>, or 240 billion possible actions.<\/p>\n

To address this challenge and help researchers take advantage of this valuable testing ground, we introduce Jericho, an open-source environment for agents to interface with IF games (opens in new tab)<\/span><\/a>. Jericho supports template-based action generation in which an agent first selects the template of an action it wishes to execute and then selects words from the game\u2019s vocabulary to fill in the blanks of the template, significantly reducing the action space and making the explore problem significantly more tractable. We\u2019re presenting the paper, \u201cInteractive Fiction Games: A Colossal Adventure,<\/a>\u201d at the 34th AAAI Conference on Artificial Intelligence (opens in new tab)<\/span><\/a>.<\/p>\n

\"text

Above is an excerpt from the interactive fiction (IF) game 9:05. IF games are text-based, providing a great opportunity\u00a0 for reinforcement learning agents to hone such skills as natural language understanding and generation, sequential decision-making, and reasoning. However, given large action spaces and other challenges, they have generally been difficult environments for RL agents to learn in. Jericho is a framework designed to make them more accessible.<\/p><\/div>\n

Making IF games more approachable<\/h3>\n

Generating coherent language-based commands is a challenge for existing RL agents, as the space of possible commands grows combinatorially<\/em> as shown by the example above. Existing agents commonly operate on action spaces with only tens or hundreds of possible actions and are largely unable to tractably explore action spaces presented by IF games without prohibitively long training times. Jericho helps address this challenge by revealing the list of action templates and vocabulary words that are recognized by the game, normally hidden from players in a non-human-readable format, for agents to choose from. The template example \u201ctake ___ from ___\u201d could result in a successful action when combined with the vocabulary words lantern<\/em> and case<\/em>, for instance.<\/p>\n

Providing templates and vocabulary can reduce the size of the action space by several orders of magnitude\u2014from 240 billion possible actions to 98 million. Without game-specific vocabulary and templates, researchers might be inclined to provide their agents with an English dictionary from which to form commands\u2014hence the exponentially large action space\u2014or create a smaller vocabulary list, running the risk of unintentionally leaving out a game-specific key term, like lantern<\/em> in Zork I<\/em>. With Jericho\u2019s vocabulary, you\u2019re guaranteed to not miss crucial words.<\/p>\n

In addition to template-based action generation, Jericho provides other features to make IF games more accessible to existing agents, including the following:<\/p>\n