{"id":1169231,"date":"2026-04-21T18:42:10","date_gmt":"2026-04-22T01:42:10","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-blog-post&p=1169231"},"modified":"2026-04-21T19:36:34","modified_gmt":"2026-04-22T02:36:34","slug":"evaluating-proactive-ai-mediators-in-multi-party-conversation-with-promediate","status":"publish","type":"msr-blog-post","link":"https:\/\/www.microsoft.com\/en-us\/research\/articles\/evaluating-proactive-ai-mediators-in-multi-party-conversation-with-promediate\/","title":{"rendered":"Evaluating Proactive AI Mediators in Multi-Party Conversation with\u00a0ProMediate\u00a0"},"content":{"rendered":"\n

By Ziyi Liu (opens in new tab)<\/span><\/a>, Bahar Sarrafzadeh,<\/a> Pei Zhou,<\/a> Longqi Yang (opens in new tab)<\/span><\/a>, Ashish Sharma<\/a>  <\/p>\n\n\n\n

Imagine you are in a high-stakes group discussion, stuck in a circular argument with no consensus in sight. Now, imagine an AI agent sitting at that table. Unlike traditional tools that wait for a prompt, this agent proactively intervenes at the perfect moment with a breakthrough suggestion. This scenario represents the emerging shift from passive AI assistants to proactive team collaborators.<\/strong> <\/p>\n\n\n\n

As LLMs evolve toward handling multi-party teamwork, they are increasingly expected to navigate complex group dynamics. However, current research often overlooks the nuance of these interactions. While agents are being designed for multi-party settings, we still lack the benchmarks to evaluate the real-time effectiveness<\/strong> of their interventions or their broader socio-cognitive intelligence <\/strong>when working with groups of people.<\/strong> <\/p>\n\n\n\n

In real-world dynamics\u2014like a high-stakes budget negotiation\u2014success is not just about the final outcome; it is about navigating hidden interests, managing “negotiation fatigue,” and knowing exactly when to speak up to break a deadlock. To address these gaps, we introduce ProMediate<\/strong>: a new framework from Microsoft Office of Applied Research designed to evaluate the next generation of proactive agents in multi-party negotiations.<\/p>\n\n\n\n

<\/p>\n\n\n\n

\"graphical<\/figure>\n\n\n\n

<\/p>\n\n\n\n

ProMediate\u00a0moves beyond static benchmarks by introducing a dynamic architecture that evaluates how agents handle the social nuances of human interaction. The framework consists of two integrated parts:<\/p>\n\n\n\n

    \n
  1. Real Case Scenarios:<\/strong> The framework utilizes a collection of high-stakes, multi-issue negotiation cases. These scenarios are built upon asymmetric information<\/strong> and conflicting interests<\/strong>, requiring participants to navigate a complex bargaining space.  <\/li>\n<\/ol>\n\n\n\n
      \n
    1. Proactive Conversation Simulation:<\/strong> ProMediate features a simulation environment with a plug-and-play proactive mediator<\/strong> role. The environment uses LLM-based agents to mimic human negotiators, complete with distinct personas and mental state trajectories. The mediator monitors the dialogue and determines the optimal intervention tempo<\/strong>\u2014deciding not only what<\/em> to say, but exactly when<\/em> to intervene to steer the group toward consensus. <\/li>\n<\/ol>\n\n\n\n

      <\/p>\n\n\n\n

      How does our evaluation framework capture the dynamic of group consensus change?<\/strong> <\/h2>\n\n\n\n
      \"Heatmap\"<\/figure>\n\n\n\n

      <\/p>\n\n\n\n

      Instead of focusing only on the final outcome, it  tracks dynamic group decision-making<\/strong>. It extracts attitudes<\/strong> for each person on each topic and calculates the agreement score<\/strong> among all parties at each step. This provides a clear consensus change trend<\/strong> with enriched signals, as researchers can directly observe where the consensus is going up or down, and if the mediator\u2019s intervention improves the consensus or not.  <\/p>\n\n\n\n

      <\/p>\n\n\n\n

      Socio-cognitive level evaluation<\/strong> <\/h2>\n\n\n\n

      Besides consensus tracking which focuses on the conversation-level evaluation, we also evaluate the mediator\u2019s behavior using socio-cognitive intelligence. Single reliance on the consensus change might not reveal the full capability of the mediator, as it is possible that the mediator tries to facilitate the negotiation but that humans won\u2019t follow. So we evaluate the mediator\u2019s behavior by only focusing on those 4 dimensions: perceptual differences, negative emotions, cognitive challenges and communication breakdown.<\/p>\n\n\n\n

      <\/p>\n\n\n\n

      Socially Intelligent Agent VS Generic Agent<\/strong> <\/h2>\n\n\n\n

      A generic mediator<\/strong> is essentially a general chat-room agent \u2014 it joins the conversation and responds, but without any specialized playbook. A socially intelligent mediator<\/strong>, on the other hand, is equipped with mediation-specific skills in thinking and strategic planning. We compare two mediator agents in different difficulty settings.  <\/p>\n\n\n\n

      The difference shows up clearly in ProMediate’s hardest setting, where participants are actively competing. The socially intelligent mediator produced meaningfully larger gains in consensus than the generic baseline, helping the group close more ground toward agreement. It was also noticeably faster to step in, catching friction points before they hardened into deadlock. In short, it didn’t just talk better \u2014 it acted sooner and hit the right moments. Social intelligence, it turns out, is what separates a chat-room bystander from a mediator that actually moves the needle.<\/strong> <\/p>\n\n\n\n

      <\/p>\n\n\n\n

      \"table\"<\/figure>\n\n\n\n

      <\/p>\n\n\n\n

      Looking ahead<\/strong> <\/h2>\n\n\n\n

      As LLMs continue to expand in capability, the advantages of proactive, socio-cognitive mediation are likely to compound. Our results indicate that while advanced reasoning models show promise, the ability to successfully navigate a multi-party negotiation depends on more than just raw scale; it requires a specialized interaction strategy that can decode the social “pulse” of a room.<\/strong> <\/p>\n\n\n\n

      For teams building intelligent features on collaborative platforms\u2014from AI meeting assistants to automated conflict resolution tools\u2014this work offers a clear message: the architecture of a mediator\u2019s proactivity matters as much as the model\u2019s size. <\/strong>By using the ProMediate<\/strong> metrics as high-signal reward labels, we can transition from simple prompting to sophisticated training. This allows us to develop agents that learn not just what<\/em> to speak, but critically, when<\/em> to speak within the complex rhythm of multi-turn interactions. <\/p>\n\n\n\n

      As the digital environments where we collaborate grow more complex, the need for principled, socially aware intervention will only become more vital. The core insight of our work is that effective mediation is not just about the final deal: it is about the socio-cognitive intelligence required to guide the journey from conflict to consensus. <\/p>\n\n\n\n

      <\/p>\n\n\n\n

      Read the full paper: <\/strong>[2510.25224] ProMediate: A Socio-cognitive framework for evaluating proactive agents in multi-party negotiation (opens in new tab)<\/span><\/a> <\/p>\n","protected":false},"excerpt":{"rendered":"

      By Ziyi Liu (opens in new tab), Bahar Sarrafzadeh, Pei Zhou, Longqi Yang (opens in new tab), Ashish Sharma   Imagine you are in a high-stakes group discussion, stuck in a circular argument with no consensus in sight. Now, imagine an AI agent sitting at that table. Unlike traditional tools that wait for a prompt, this agent proactively intervenes at the […]<\/p>\n","protected":false},"author":43305,"featured_media":1169325,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-content-parent":0,"msr_hide_image_in_river":0,"footnotes":""},"research-area":[13556,13554],"msr-locale":[268875],"msr-post-option":[],"class_list":["post-1169231","msr-blog-post","type-msr-blog-post","status-publish","has-post-thumbnail","hentry","msr-research-area-artificial-intelligence","msr-research-area-human-computer-interaction","msr-locale-en_us"],"msr_assoc_parent":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post\/1169231","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-blog-post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/43305"}],"version-history":[{"count":6,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post\/1169231\/revisions"}],"predecessor-version":[{"id":1169341,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post\/1169231\/revisions\/1169341"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/1169325"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=1169231"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=1169231"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=1169231"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=1169231"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}