{"id":455559,"date":"2017-01-09T00:00:55","date_gmt":"2017-01-09T08:00:55","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=455559"},"modified":"2018-01-23T20:02:19","modified_gmt":"2018-01-24T04:02:19","slug":"creating-curious-machines-building-information-seeking-agents","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/creating-curious-machines-building-information-seeking-agents\/","title":{"rendered":"Creating curious machines: Building information-seeking agents"},"content":{"rendered":"

 <\/p>\n

Humans have an innate desire to know and understand. From a child learning to ride a bike to an adult gaining skills in an online course, we constantly absorb information from our environment through interaction. Motivated by this observation, we\u2019ve developed a suite of tasks that teach artificial agents how to seek information actively, by asking questions. We\u2019ve also designed a deep neural agent that learns to accomplish these tasks through efficient information-seeking behavior. Such behavior is a vital research step towards Artificial General Intelligence.<\/p>\n

Asking the right questions<\/h2>\n

Let\u2019s say you\u2019re at a dinner party with friends and you decide to play 20 Questions. It\u2019s your turn and you choose \u2018cat\u2019 as the thing for others to guess. They begin by asking broad questions: \u201cIs it alive?\u201d, \u201cIs it a person?\u201d, \u201cIs it an animal?\u201d, \u201cDoes it live underwater?\u201d. The person who correctly identifies the item first is the winner, so your friends are not just trying to get the right answer, they\u2019re trying to do so with as few questions as possible. Based on your simple yes-or-no responses, your friends can quickly narrow down the set of viable items until one correctly guesses \u2018cat\u2019.<\/p>\n

This example demonstrates the iterative nature of information seeking: the information currently sought must be intelligently conditioned on the information already acquired. To be effective, an information-seeking agent must in some sense understand the state of its current knowledge. It must know what it knows, and how to bridge the gap between what it knows and what it needs to know.<\/p>\n

The 20 Questions example also highlights how communication necessarily takes place over a restricted channel: each answer is a simple \u2018yes\u2019 or \u2018no\u2019 (conveying just one bit of information), and the number of questions is limited. Real-world information seeking is typically restricted in a similar sense — we communicate via finite languages, over limited amounts of time. Consider searching online to choose a gift for a friend. Perhaps you start broadly — guided loosely by age, gender and budget — then hone in based on specific interests and recommendations.<\/p>\n

Because of its fundamental role in intelligent behavior, information seeking has been studied from a variety of perspectives, including cognitive science, psychology, neuroscience, and machine learning. In neuroscience, for instance, information-seeking strategies are often explained by biases toward novel, surprising, or uncertain events (Ranganath & Rainer, 2003). Information seeking is a key component in formal notions of fun and creativity (Schmidhuber, 2010), and intrinsic motivation (Oudeyer and Kaplan, 2007). It is also closely related to the concept of attention, which improves efficiency by ignoring irrelevant features (Mnih et al., 2014) and may be considered a strategy for information seeking.<\/p>\n

New tasks for exploring information seeking<\/h2>\n

Researchers have used different tools and systems to help train intelligent agents, from datasets through to bespoke learning environments. The use of games like chess, Go, and the Atari suite has been incredibly fruitful in training intelligent agents. Similarly, many of the games that humans enjoy seem expressly designed to train efficient information seeking, or at least to exploit our joy in exercising this skill.<\/p>\n

Thus motivated, we designed a suite of tasks to train and evaluate information-seeking behavior. Three of these tasks are demonstrated here (see our paper for more details):<\/p>\n\t\n\t<\/iframe>\n\t\n

The highlighted tasks were Hangman, Face Challenge, and War Boat.\u00a0Each of these tasks has a distinct mode of play and unique rules and objectives. Each requires the ability to seek information iteratively based on an agent\u2019s current \u201cpicture\u201d of the world. The tasks are illustrated in the gifs below.<\/p>\n

\"Maluuba+-+AGI+-+Hangman\"Hangman:\u00a0The classic game where an agent must identify a phrase within a set number of turns by guessing letters of the alphabet.<\/p>\n

\"Hangman\"<\/p>\n

Face challenge: An agent must determine the answer to questions like \u201cIs this person wearing a hat?\u201d, or \u201cDoes this person have a moustache?\u201d by peeking at small chunks of an occluded portrait.<\/p>\n

\"War<\/p>\n

War Boat: An agent aims to sink an opponent\u2019s naval fleet, which is randomly positioned on a hidden grid. Correctly guessing points where a boat is located means that the ship is \u2018hit\u2019, and with enough hits the boat sinks.<\/p>\n

Training models to seek information<\/h2>\n

The actions agents perform in our tasks can be interpreted as questions asked of the environment, e.g. \u201cDoes this phrase contain the letter \u2018a\u2019?\u201d or \u201cWhat does this block of pixels look like?\u201d. To succeed, an agent must learn to ask useful questions and assimilate the information it obtains.<\/p>\n

We developed a model that can be trained to do just that. At each step in completing a task, the model asks what it believes to be the most useful question, receives a response from the environment, and integrates that response with its existing knowledge. The model is a deep neural network that we trained through a combination of reinforcement learning techniques (specifically: Generalized Advantage Estimation, Schulman et al. 2016) and backpropagation. See the paper for full details.<\/p>\n

\"TopDuring training, the agent maximizes a reward which combines task-specific extrinsic rewards and a task-agnostic intrinsic reward. The extrinsic rewards encourage the agent to achieve its goal using as few questions as possible. The intrinsic reward encourages the model to ask questions which provide the most new information about the environment. Specifically, we reward each question according to how much its answer increases similarity between model\u2019s belief about the world and the actual state of the world. Thus, the agent learns to efficiently form an accurate internal picture of its environment.<\/p>\n

Towards artificial general intelligence<\/h2>\n

As the demo shows, our methods produce agents that succeed across a broad range of tasks. The same approach can be applied to language, image, and strategy domains. In our tasks, the trained agents exhibit interpretable, intelligent information-seeking behavior, often performing at super-human levels.<\/p>\n

We believe that information seeking plays a fundamental role in General Intelligence. Our present work is a small step towards this grander goal.<\/p>\n

Read the paper<\/p>\n

References<\/strong><\/p>\n