{"id":750835,"date":"2021-06-07T10:32:29","date_gmt":"2021-06-07T17:32:29","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=750835"},"modified":"2021-06-07T12:37:22","modified_gmt":"2021-06-07T19:37:22","slug":"building-stronger-semantic-understanding-into-text-game-reinforcement-learning-agents","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/building-stronger-semantic-understanding-into-text-game-reinforcement-learning-agents\/","title":{"rendered":"Building stronger semantic understanding into text game reinforcement learning agents"},"content":{"rendered":"\n
\"\"\/<\/figure>\n\n\n\n

AI agents capable of understanding natural language, communicating, and accomplishing tasks hold the promise of revolutionizing the way we interact with computers in our everyday lives. Text-based games, such as the Zork <\/em>series, act as testbeds for development of novel learning agents capable of understanding and interacting exclusively through language. Beyond requiring the use of imagination and myriad concepts of everyday life to solve, these fictional world\u2013based narratives are also a safe sandbox for AI testing<\/a> that avoids the expense of collecting user data and the risk of users having a bad experience interacting with agents that are still learning. <\/p>\n\n\n\n

In this blog post, we share two papers that explore reinforcement learning methods to improve semantic understanding in text agents, a key process by which AI understands and reacts to text-based input. We\u2019re also releasing source code for these agents to encourage the community to continue to improve semantic understanding in text-based games.<\/p>\n\n\n\n

\n
\n
\n\t
\n\t\t
\n\t\t\t\t\t\tPublication<\/span>\n\t\t\tReading and Acting While Blindfolded: The Need for Semantics in Text Game Agents<\/span> <\/span><\/a>\t\t\t\t\t<\/div>\n\t<\/article>\n<\/div>\n\n\n\n
\n\t
\n\t\t
\n\t\t\t\t\t\tCODE<\/span>\n\t\t\tBlindfold Text Game<\/span> <\/span><\/a>\t\t\t\t\t<\/div>\n\t<\/article>\n<\/div>\n<\/div>\n\n\n\n
\n
\n\t
\n\t\t
\n\t\t\t\t\t\tPublication<\/span>\n\t\t\tKeep CALM and Explore: Language Models for Action Generation in Text-based Games<\/span> <\/span><\/a>\t\t\t\t\t<\/div>\n\t<\/article>\n<\/div>\n\n\n\n
\n\t
\n\t\t
\n\t\t\t\t\t\tCode<\/span>\n\t\t\tCALM text game <\/span> <\/span><\/a>\t\t\t\t\t<\/div>\n\t<\/article>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

Ever since text-based games were proposed as a benchmark for language understanding agents, a key challenge in these games has been the enormous action space. Games like Zork 1 (opens in new tab)<\/span><\/a> <\/em>can have up to 98 million possible actions in each state\u2014the majority of which are nonsensical, ungrammatical, or inapplicable. In order to make text-based games more approachable to reinforcement learning (RL) agents, the Jericho framework (opens in new tab)<\/span><\/a>provides several handicaps such as valid-action identification, which uses the game engine to identify a minimal set of 10\u2013100 textual actions applicable in the current game state. Other handicaps involve the ability to extract the recognized vocabulary for a game and to save and restore previously visited states. Certain RL agents depend on the valid-action handicap, like deep reinforcement relevance network (DRRN)<\/a>, which learns to choose the action within the set of valid actions at each timestep that maximizes expected game scores.<\/p>\n\n\n\n

\"Example<\/a>
Figure 1: An example text-based game consists of language-based observations describing the simulated world and actions, taken by the agent, that alter the state of the world and advance the story. Valid Actions, identified through introspection into the game emulator, are a list of actions applicable at the current step that are guaranteed to change the state of the game.<\/figcaption><\/figure>\n\n\n\n

Despite its usefulness from an RL tractability standpoint, the valid-action handicap, through the use of privileged game insights, can expose hidden information about the environment to the agent. For example, In Figure 1, the valid-action take off watch<\/em> inadvertently leaks the existence of a watch that was not revealed in the observation text.<\/p>\n\n\n\n

\n\t
\n\t\t
\n\t\t\t\t\t\tPublication<\/span>\n\t\t\tReading and Acting While Blindfolded: The Need for Semantics in Text Game Agents<\/span> <\/span><\/a>\t\t\t\t\t<\/div>\n\t<\/article>\n<\/div>\n\n\n\n

Our recent paper, \u201cReading and Acting while Blindfolded: The need for semantics in text game agents<\/a>,\u201d shows that when using Jericho-provided handicaps, such as the identification of valid-actions, it\u2019s possible for text agents to achieve competitive scores while using textual representations entirely devoid of semantics. The paper has been accepted to the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2021) (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n

Hash-based representation reveals that handicap agents bypass semantic understanding<\/h2>\n\n\n\n

To probe the extent of semantic understanding of the DRRN agent, we replace the observation text and valid-action texts with consistent, yet semantically meaningless text strings derived by hashing the current state and actions. Figure 2 compares the standard textual representation of observations and actions with two ablations. MIN-OB (b) shortens observation text to only include the name of the current location without description of the objects or characters therein. HASH (c) entirely replaces the text of the observation and actions with the output of a hash-based representation.<\/p>\n\n\n\n

\"Three<\/a>
Figure 2: A standard textual representation (left) and two ablations: MIN-OB (top right) and HASH (bottom right).<\/figcaption><\/figure>\n\n\n\n

Specifically, we use Python\u2019s hash function to map each text string to a unique hash value, which ensures even if one word is changed in text, the representation will be completely different. From the perspective of playing this game, representations (b) and (c) would pose significant challenges to a human due to the lack of semantic information.<\/p>\n\n\n\n

Subsequently, a DRRN text agent was trained from scratch using RL to estimate the quality of each candidate action. Surprisingly, the agent on the hashed-representation was able to achieve comparable scores with the control agents using the original text-based observations and actions (see Figure 3), and it even outperforms the baseline (DRRN-base) in 3 out of the 12 games. This finding suggests that current text agents are likely circumventing the development of semantic understanding, particularly when trained on single games using the valid-action handicap.<\/p>\n\n\n\n

\"Average<\/a>
Figure 3: The average normalized scores for DRRN (HASH), DRRN (MIN-OB), and DRRN-base (left) show that DRRN using the hash representation outperforms DRRN-base trained using textual inputs. Raw scores (right) show average and maximum performance on each game.<\/figcaption><\/figure>\n\n\n\n

In light of this finding, we highlight two recent methods that we believe encourage agents to build stronger semantic understanding throughout the learning process:<\/p>\n\n\n\n

Regularizing semantics via inverse dynamics decoding<\/h2>\n\n\n\n

Standard RL centers around the idea of learning a policy mapping from observations to actions that maximizes cumulative long-term discounted rewards. We contend that such an objective only encourages robust semantic understanding to the extent that\u2019s required to pick out the most rewarding actions from a list of candidates.<\/p>\n\n\n\n

We propose the use of an auxiliary training task that focuses on developing stronger semantic understanding\u2014 the inverse dynamics task challenges the agent to predict the action that spans two adjacent state observations. Consider the following observations: You stand at the foot of a tall tree <\/em>and You sit atop a large branch, feet dangling in the air.<\/em> It\u2019s reasonable to surmise that the intervening action may have been climb tree. Specifically, the agent is presented with a finite set of valid actions that could have been selected from the previous state and must choose which of them was most likely to have been selected given the text of the subsequent state. This auxiliary loss has been investigated in the context of video games like Super Mario Bros.<\/em> (opens in new tab)<\/span><\/a><\/p>\n\n\n\n

By training agents with this inverse dynamics auxiliary task, we show that they have stronger semantic understanding, as evidenced by their ability to generalize between semantically similar states in Figure 4.<\/p>\n\n\n\n

\"DRRN<\/a>
Figure 4: Above are t-SNE visualizations of text observations of the game Zork 1<\/em>. Outlined in black, observations originating from the Living Room are all semantically similar (see text observation 21 in Figure 1). The Inverse-Dynamics agent (inv-dy in middle) minimizes distances between these semantically similar observations, whereas DRRN-base (left) clusters them disparately. Furthermore, when encountering unseen states, DRRN-base is unable to relate these novel observations to any of the previously encountered states, likely leading to poor generalization. On the other hand, Inverse-Dynamics can use its language understanding to relate unseen states to semantically similar, previously seen states.<\/figcaption><\/figure>\n\n\n\n

As an added benefit, we show that it\u2019s possible to use the inverse-dynamics model to incentivize the agent to explore regions of the state space in which model predictions are inaccurate. This improved exploration led the agent to notably high single-episode scores of 54, 94, and 113 on Zork 1<\/em>, compared to previous models that get stuck at a score of 55. Looking at the trajectories, we observe that the inverse-dynamics model uniquely exhibits some challenging exploration behaviors, such as navigating through \u201ca maze of twisty little passages, all alike\u201d; taking a coin at a specific spot of the maze (+10 score); or finding the \u201cCyclops Room\u201d at the exit of the maze and then going up to the \u201cTreasure Room\u201d (+25 score). Thus, we find that by improving semantic understanding, approaches like inverse dynamics can also lead to better exploration in agents.<\/p>\n\n\n\n

Leveraging human priors for action selection<\/h2>\n\n\n\n

Another way to build in semantic language understanding is to leverage it for action generation. Published in the Conference on Empirical Methods in Natural Language Processing (EMNLP 2020) (opens in new tab)<\/span><\/a>, \u201cKeep CALM and Explore: Language Models for Action Generation in Text-based Games,<\/a>\u201d introduces a text game\u2013oriented language model called CALM.<\/p>\n\n\n\n

\n\t
\n\t\t
\n\t\t\t\t\t\tPublication<\/span>\n\t\t\tKeep CALM and Explore: Language Models for Action Generation in Text-based Games<\/span> <\/span><\/a>\t\t\t\t\t<\/div>\n\t<\/article>\n<\/div>\n\n\n\n

CALM (short for Contextual-Action Language Model) avoids the use of the valid-action handicap altogether by using a language model to generate a compact set of candidate actions. CALM starts from a pretrained GPT-2 model  (opens in new tab)<\/span><\/a>and is fine-tuned on a corpus featuring years of human-gameplay transcripts spanning 590 different text-based games. During this training process, CALM is instilled with priors that humans use when playing a large variety of text-based games.<\/p>\n\n\n\n

\"Observation:<\/a>
Figure 5: CALM can pair common verbs with objects from an observation when trained with an n-gram (blue), and it can generate more complex actions when trained with GPT-2 (green).<\/figcaption><\/figure><\/div>\n\n\n\n

As shown in Figure 5, conditioned on the observation text, randomly sampled actions are nonsensical (outlined in red). CALM, trained with an n-gram model, pairs common verbs with objects from the observation (outlined in blue), and CALM trained with GPT-2 (outlined in green) can generate actions that are more complex and highly relevant.<\/p>\n\n\n\n

To quantify this effect, we compared the quality of CALM-generated actions to ground-truth valid actions on walkthrough trajectories from 28 games. We find higher precision and recall with CALM than with a comparably trained model trained using a bag-of-words representation.<\/p>\n\n\n\n

\"The<\/a>
Figure 6: The full CALM architecture. Given the current game context, the CALM language model generates actions candidates such as \u201copen mailbox\u201d. The DRRN reinforcement learning agent conditions on the current observation, which is encoded word-by-word using a GRU, as well as each action candidate which is encoded similarly through a separate GRU. DRRN subsequently estimates a Q-value for each observation-action pair. The final action is sampled from the Q-values using a softmax distribution.<\/figcaption><\/figure><\/div>\n\n\n\n

The figure above shows the full integration of the CALM language model with the DRRN learning agent. CALM is responsible for producing Action Candidates conditioned on the current context, while DRRN attempts to select the best Action Candidate from the produced set. This action is then executed in the environment and the cycle repeats.<\/p>\n\n\n\n

We subsequently deployed CALM in the RL context on held-out games that were excluded from the training set. CALM acts as a drop-in replacement for the valid-action handicap by generating a small set of likely actions conditioned on the current text observation. We find that in eight of 28 game cases, CALM can entirely replace the need for Jericho\u2019s valid-action and agents trained to pick from CALM-generated actions outperform those trained with the handicap.<\/p>\n\n\n\n

We additionally compare against NAIL, a competition-winning general game-playing agent that uses handwritten rules to explore and act. We find that CALM generates an average normalized completion of 9.4% across games, compared to NAIL with 5.6%.<\/p>\n\n\n\n

Valid-action identification and CALM: A springboard for furthering semantic understanding<\/h2>\n\n\n\n

Text-based games remain an exciting and challenging testbed for the development of language agents. Our recent work highlights the need for careful examination of the handicaps as they relate to the development of semantic understanding the text agents trained within. Our findings show that handicaps such as valid-action identification can allow agents to bypass the challenges of developing semantic language understanding. We outlined two different ways of increasing semantic understanding in text agents. First, we show better semantic clustering of text observations using an auxiliary task focused on inverse dynamics prediction. Second, we show compelling performance by using the contextual-action language model to create candidate actions and eliminating the valid-action handicap altogether.<\/p>\n\n\n\n

More broadly, we aim to spark discussion with the community around the new methods and experimental protocols to encourage agents to develop better language understanding, and we are excited about the continued development of text-based agents with strong semantic understanding.<\/p>\n\n\n\n

Read our paper here <\/a>and find the source code here (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n

<\/p>\n","protected":false},"excerpt":{"rendered":"

AI agents capable of understanding natural language, communicating, and accomplishing tasks hold the promise of revolutionizing the way we interact with computers in our everyday lives. Text-based games, such as the Zork series, act as testbeds for development of novel learning agents capable of understanding and interacting exclusively through language. Beyond requiring the use of […]<\/p>\n","protected":false},"author":38838,"featured_media":751741,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":null,"msr_hide_image_in_river":0,"footnotes":""},"categories":[1],"tags":[],"research-area":[13556,13554],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-750835","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-artificial-intelligence","msr-research-area-human-computer-interaction","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[],"related-events":[740920],"related-researchers":[{"type":"guest","value":"karthik-narasimhan","user_id":"751732","display_name":"Karthik Narasimhan","author_link":"Karthik Narasimhan<\/a>","is_active":true,"last_first":"Narasimhan, Karthik ","people_section":0,"alias":"karthik-narasimhan"},{"type":"guest","value":"shunyu-yao","user_id":"751735","display_name":"Shunyu Yao","author_link":"Shunyu Yao","is_active":true,"last_first":"Yao, Shunyu ","people_section":0,"alias":"shunyu-yao"}],"msr_type":"Post","featured_image_thumbnail":"\"New","byline":"Matthew Hausknecht, Karthik Narasimhan<\/a>, and Shunyu Yao","formattedDate":"June 7, 2021","formattedExcerpt":"AI agents capable of understanding natural language, communicating, and accomplishing tasks hold the promise of revolutionizing the way we interact with computers in our everyday lives. Text-based games, such as the Zork series, act as testbeds for development of novel learning agents capable of understanding…","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/750835","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/38838"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=750835"}],"version-history":[{"count":23,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/750835\/revisions"}],"predecessor-version":[{"id":751849,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/750835\/revisions\/751849"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/751741"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=750835"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=750835"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=750835"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=750835"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=750835"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=750835"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=750835"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=750835"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=750835"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=750835"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=750835"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}