{"id":953901,"date":"2023-07-10T12:10:56","date_gmt":"2023-07-10T19:10:56","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-project&p=953901"},"modified":"2026-02-05T06:41:47","modified_gmt":"2026-02-05T14:41:47","slug":"project-rumi","status":"publish","type":"msr-project","link":"https:\/\/www.microsoft.com\/en-us\/research\/project\/project-rumi\/","title":{"rendered":"Project Rumi"},"content":{"rendered":"
\n\t
\n\t\t
\n\t\t\t\"Project\t\t<\/div>\n\t\t\n\t\t
\n\t\t\t\n\t\t\t
\n\t\t\t\t\n\t\t\t\t
\n\t\t\t\t\t\n\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\n

Project Rumi<\/h1>\n\n\n\n

Multimodal paralinguistic prompting for large language models<\/p>\n\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t<\/div>\n<\/section>\n\n\n\n\n\n

Multimodal paralinguistic prompting for large language models<\/h2>\n\n\n\n

Large language models (LLMs) are powerful neural networks that generate natural language with demonstrated value across a wide range of domains. They are revolutionizing many aspects of human society and culture including introducing a \u201cwhole new interaction model between humans and computers, turning natural language into the most powerful productivity tool on the planet.\u201d1<\/sup><\/p>\n\n\n\n

However, LLMs also have limitations: they may not always understand the context and nuances of a conversation. Their performance also depends on the quality and specificity of the user\u2019s input, or prompt. The data that the user inputs into the LLM is a lexical entry, which does not comprehensively represent the nuances of human-to-human interaction; it is in fact missing all the paralinguistic<\/strong> information (intonation, gestures, facial expressions, and everything besides the actual words) that contribute to the meaning and intentions of the speaker. This can lead to misinterpretation, misunderstanding, or inappropriate responses from the LLM. Project Rumi incorporates paralinguistic input into prompt-based interactions with LLMs with the objective of improving the quality of communication. Providing this context is critical to enhancing LLMs capabilities in this \u201cAI as a copilot\u201d era.<\/p>\n\n\n\n

Our current system leverages separately trained vision and audio-based models to detect and analyze non-verbal cues extracted from data streams. The models assess sentiment from cognitive and physiological data in real time, generating appropriate paralinguistic tokens to augment standard lexical prompt input to existing LLMs such as GPT4.  This multimodal, muti-step architecture integrates seamlessly with all pretrained text-based LLMs to provide additional information on the user’s sentiment and intention that is not captured by text-based models, augmenting the prompt with the richness and subtlety of human communication to bring human-AI interaction to a new level.<\/p>\n\n\n\n

\"Diagram
Diagram illustrating user interaction with application. Inputs user\u2019s sensor data such as video and audio and uses processed sensor data to generate paralinguistic classification. This classification augments the input to the LLM, generating an augmented output to application.<\/figcaption><\/figure>\n\n\n\n
<\/div>\n\n\n\n

Future explorations include improving performance of existing models and incorporating additional signals like HRV (heart rate variability) derived from standard video, and cognitive and ambient sensing. Conveying unspoken meaning and intention is an essential component in the next generation of AI interaction.<\/p>\n\n\n\n

\n