{"id":933729,"date":"2023-04-10T18:23:03","date_gmt":"2023-04-11T01:23:03","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-blog-post&p=933729"},"modified":"2024-03-20T08:13:13","modified_gmt":"2024-03-20T15:13:13","slug":"gpt-models-meet-robotic-applications-long-step-robot-control-in-various-environments","status":"publish","type":"msr-blog-post","link":"https:\/\/www.microsoft.com\/en-us\/research\/articles\/gpt-models-meet-robotic-applications-long-step-robot-control-in-various-environments\/","title":{"rendered":"GPT Models Meet Robotic Applications: Long-Step Robot Control in Various Environments"},"content":{"rendered":"\n
\"diagram\"
We have released practical prompts for ChatGPT to generate executable robot action sequences from multi-step human instructions in various environments.<\/figcaption><\/figure>\n\n\n\n
\n
\n
\n
Paper<\/a><\/div>\n\n\n\n
GitHub<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

Introduction<\/h3>\n\n\n\n

Imagine having a humanoid robot in your household that can be instructed and demonstrated household chores without coding\u2014Our team has been developing such a system, which we call Learning-from-Observation<\/em>.<\/p>\n\n\n\n

As part of our effort, we recently released a paper, “ChatGPT Empowered Long-Step Robot Control in Various Environments: A Case Application<\/a>,”<\/strong> where we provide a specific example of how OpenAI’s ChatGPT can be used in a few-shot setting to convert natural language instructions into a sequence of executable robot actions. Our prompts and source code for using them are open-source and publicly available at this GitHub repository (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n

In fact, generating programs for robots from language is an attractive goal and has attracted research interest in the robotics research community; some of them are built on top of large language models such as ChatGPT (opens in new tab)<\/span><\/a>. However, most of them were developed within a limited scope, hardware-dependent, or lack the functionality of human-in-the-loop. Additionally, most of these studies rely on a specific dataset, which requires data recollection and model retraining when transferring or extending them to other robotic scenes. From a practical application standpoint, an ideal robotic solution would be one that can be easily applied to other applications or operational settings without requiring extensive data collection or model retraining<\/strong>.<\/p>\n\n\n\n

In this paper, we provide a specific example of how ChatGPT can be used in a few-shot setting to convert natural language instructions into a sequence of actions that a robot can execute. In designing the prompts, we tried to ensure that they meet the requirements common to many practical applications while also being structured in a way that they can be easily customizable. The requirements we defined for this paper are:<\/p>\n\n\n\n