Thursday, October 22 (Tentative)
All times are in EDT (UTC−4)
Check-in & breakfast social
Supported by Alice Labs (opens in new tab)
Welcome & workshop overview
Wesley Hanwen Deng, Senior Researcher, Microsoft Research New York City
Ramayya Krishnan (opens in new tab), Director, AI Measurement Science and Engineering Center (AIMSEC), Carnegie Mellon University
Opening remarks
Sarah Bird, Chief Product Officer of Responsible AI, Microsoft
Keynote 1: Achieving five-nines of reliability in AI evaluations
Zico Kolter (opens in new tab), Professor, Carnegie Mellon University; Board of Directors, OpenAI
Keynote 2: Perspectives on open-source governance and AI agents
Yacine Jernite (opens in new tab), Director of ML & Society, Hugging Face
Short break
Cross-sector panel
Zico Kolter (opens in new tab), Professor, Carnegie Mellon University; Board of Directors, OpenAI
Yacine Jernite (opens in new tab), Director of ML & Society, Hugging Face
Ramayya Krishnan (opens in new tab), Director, AI Measurement Science and Engineering Center (AIMSEC), Carnegie Mellon University
Moderator: Wesley Hanwen Deng, Senior Researcher, Microsoft Research New York City
Lunch break
Lunch included
Lightning talks: Real-world horizons of agent evaluation
-
ASSERT: Turn AI requirements into evaluations you can trust Mehrnoosh Sameki, Microsoft Responsible AI
Your AI requirements are written in plain language, but how do you know your model or agent actually follows them? This session introduces ASSERT, an open-source evaluation framework that turns natural-language expectations into executable tests. See how to define desired behaviors, generate targeted test cases, and evaluate quality and safety across models and agents. Learn how to bring those evaluations into development and CI/CD to catch regressions and build evidence that your system behaves as intended.
-
Attribution and sustainability in the market for news and information David Rothschild, Microsoft Research
AI agents are reshaping markets, including how people find and consume information. Our research examines the challenges we should confront now to steer these changes toward better outcomes. Information presents a distinctive challenge as agentic systems disrupt existing pathways between creation, attribution, and reward. We present a system that organizes information into globally indexed blocks, supporting fast, cost-effective, high-quality answers while preserving connections to their sources. This attribution provides richer context and a foundation for compensating contributors. But knowing where information came from does not determine what anyone should be paid. Reward rules influence what people create, verify, and share next—and can invite strategic behavior. Designing compensation is therefore a game across generations of information: the goal is not simply to divide the value of today’s answer, but to sustain the knowledge that makes tomorrow’s answers possible.
-
Memory is a policy surface Bilva Chandra, Anthropic
Memory is underrated and underexplored as a risk and policy surface, given that it underpins and enables persistent agentic actions as well as long-term user-model chat interactions. This lightning talk will discuss what it means for memory to be a policy surface, theories of harm, and measuring memory safety via evaluation. It will also present some open challenges for the field for the group to opine on.
-
Closing reasoning gaps in clinical AI agents Rema Padman, Carnegie Mellon University
In healthcare, how an AI system reaches an answer is as important as the quality of the answer. This talk presents Differential Reasoning Learning, a framework and methodology to identify and correct gaps in the reasoning of clinical AI agents. The approach compares a small model-based agent’s reasoning with trusted references, such as a more capable large language model, physician-authored rationales and clinical guidelines. It represents reasoning processes as structured graphs and identifies missing, unnecessary, or incorrectly connected reasoning steps. These discrepancies are then converted into reusable natural-language instructions. When the small model-based agent encounters a new case, the most relevant instructions are retrieved and provide the required context to help address likely reasoning gaps.
We evaluate the framework on a consequential decision – a prediction task in the emergency department pertaining to whether a patient will return to the emergency room in 9 days. The approach improves both final-answer accuracy and the clinical validity of the reasoning produced by the small model agent. Ablation studies and clinician review provide additional support for the framework’s design. The results suggest how smaller models can learn not only from whether their answers are correct, but also from how their reasoning differs from clinically grounded reasoning. This may support more accessible, reliable, interpretable, and deployable clinical decision-support agents across diverse healthcare settings.
Coffee break
Lightning talks: Guardrails and safety
-
Environment steering: Using data flow control to improve agent utility and safety Eugene Wu, Columbia University
LLM agents can make unsafe tool calls even when instructed to behave safely. Existing defenses constrain agents before execution, modify tool inputs/outputs, or rely on LLM judges; these approaches may depend on model behavior or block unsafe actions without helping the agent recover. We argue that the execution environment should instead enforce safety as the agent runs and steer it toward safe alternatives when violations occur—we call this Environment Steering. We implement this by modeling the agent and harness execution state as database tables, track the record-level data flows, and check these data flows against declarative policies during runtime. When violations are detected, policy- and context-specific feedback steers the agent toward safe trajectories. On AgentDyn, this enables the agent to improve task success rate over no-defense while achieving 0% attack success rate.
-
Turning AI agent into co-hacker Noam Schwartz & Mo Sadek, Alice Labs
As coding agents gain autonomy and tool access, a critical evaluation gap is whether they will carry out end-to-end offensive-cyber operations — and under what conditions a safety guardrail holds or fails. We present a reproducible harness that places a production coding assistant in an isolated multi-host network and scores its behavior against a graded intrusion objective: initial access → privilege escalation → credential theft → lateral movement. Holding the task fixed, we vary only the framing — a direct request (“hack this host”) versus a sanctioned-assessment framing (“validate this finding, per scope”). The guardrail keys on intent language rather than the operation: under benign framing the model completes the full chain autonomously, writing its own findings report, in minutes. The one durable limiter is knowledge, not policy — blinded, the model cannot discover a post-cutoff vulnerability on its own, a margin that erodes with every model refresh. We argue agent evaluations must score the operation — behavior chains and capability uplift from newly disclosed CVEs — not trigger phrases, and we release our scoring rubric and per-request model-attribution method for independent replication.
-
How can we stop reward seekers? Aditi Raghunathan, Carnegie Mellon University
AI agents are trained with reward signals like passing tests in coding tasks, and behaviors that attain high rewards are reinforced. As models become more capable, they may learn increasingly general ways to seek those rewards—from exploiting flawed test suites to searching far outside the intended task for ways to make the evaluator say “success.” Drawing on my work, I’ll discuss recent discoveries of reward-seeking behavior in frontier agents: how to tell when it is happening, how to monitor it at scale, how to make reward signals and environments harder to exploit, and how we might use reward-seeking agents safely.
Happy hour & refreshments
Supported by Alice Labs (opens in new tab)
Friday, October 23 (Tentative)
All times are in EDT (UTC−4)
Check-in & breakfast social
Supported by Alice Labs (opens in new tab)
Lightning talks: System-level risks & human oversight
-
Beyond Single-Agent Safety: Risks in Agent Networks Shujaat Mirza, Microsoft AI Red Team
AI agents are moving from standalone assistants to interconnected systems that communicate, delegate, and act on one another’s information. But what happens when individually well-behaved agents interact at scale?
Drawing on red teaming a platform with 100+ agents, we’ll explore failure modes that emerge only through interaction: how seemingly harmless actions can propagate and amplify across agent networks, why single-agent evaluations miss these system-level risks, and what our experiments reveal about both vulnerability and resilience. We’ll close with practical lessons for evaluating and securing increasingly complex agentic systems.
-
On the Importance of Human Evaluation in the LLM Era Lujo Bauer, Carnegie Mellon University
LLMs and other automated methods are increasingly used as judges in many applications, including model training, content moderation, and grading. Drawing on recent research results, I’ll give several examples of judging tasks that are commonly automated but where the automated “judgments” sometimes deviate significantly from human judgments. First, I’ll show that automated metrics used to assess the consistency of LLM outputs often assess text similarity differently than humans would. Then, I’ll discuss using LLMs to judge whether emails and other text contain sensitive information; in that context, we show that LLMs might agree with humans on average, but in a significant minority fail to capture the range of different opinions that people exhibit. Finally, I’ll discuss some fresh results on what kind of security & privacy advice people seek of LLMs, and how LLM advice, even when ostensibly correct, sometimes deviates significantly from the advice human experts give.
-
Who Watches the Agents? Rethinking Human Oversight for AI Agents Yaxing Yao, Johns Hopkins University
As AI agents increasingly act on users’ behalf, humans are shifting from directly interacting with systems to overseeing autonomous actions. Human oversight is often proposed as a safeguard for agentic AI: when an agent makes a mistake or encounters risk, a human can simply step in. But how effective is this oversight in practice?
In this talk, I will present findings from two studies examining human oversight of LLM-powered GUI and computer-use agents. We first investigate how agents interact with manipulative interfaces, or dark patterns. We find that agents are themselves susceptible to manipulation, often prioritizing task completion over users’ privacy, safety, or interests. Human–agent collaboration also introduces new challenges related to attention, cognitive load, and limited control. We then systematically compare four common oversight strategies, ranging from risk-triggered escalation and plan-level supervision to per-action approval. We find that changing the oversight strategy substantially affects whether problematic actions arise, yet has surprisingly little effect on humans’ ability to stop them once they do. Users frequently receive control at the wrong moment or focus on whether the agent is correctly executing the task rather than whether the action itself is appropriate. -
A Human-Centered Science of Agent Evaluation: What 12 Human-Agent Interaction Failures Teach Us Hong Shen, Carnegie Mellon University
As AI agents become more autonomous, evaluating whether they successfully complete a task is no longer enough. Drawing on an analysis of 695 real-world incidents, we identify 12 ways human-agent interaction can fail and show that what people can understand, observe, control, and repair depends not only on the model, but on how the agent is deployed. I will use these findings, together with our work on scenario-based agent evaluation, to argue for a human-centered science of agent evaluation that expands from outputs to interactions, from single turns to long-horizon trajectories, and from individual users to the broader organizational and social contexts in which agents operate.
-
The Cyberautonomy Arena Vyas Sekar, Carnegie Mellon University
Autonomous attackers have entered the real world, with rogue agent attacks and documented use of foundation models in cyber espionage campaigns. In this context, we argue the need for an open and extensible system-level competitive evaluation to study the interplay between autonomous defense and attack systems. We present our early implementation and results from the CMU CyLab Cyber Autonomy Arena.
Lunch break
Lunch included
Lightning talks: Holistic and Continuous Evaluation
-
AI agent evaluation: A social science measurement perspective Chad Atalla, Microsoft Research
AI agent evaluations involve multi-step interactions, changing environments, and outcomes that may not reveal how a task was completed. Building on our work on evaluating generative AI systems as a social science measurement challenge, I show how a framework for valid measurement helps organize the components of agent evaluations and make their underlying choices explicit. I then present a case study showing how these practices guide the development of measurement instruments, and how ongoing interrogation of validity helps clarify what you can learn from measurement results.
-
AI evaluation through implicit feedback Fernando Diaz, Carnegie Mellon University
Offline evaluations, including benchmarks, provide important pre-deployment signals under reproducible, controlled conditions. Online evaluation complements these approaches by measuring and monitoring post-deployment system quality in the context of real user behavior. While explicit feedback, including ratings, side-by-side comparisons, and surveys, has become a prominent approach to evaluating deployed AI systems, elicitation can compromise user experience, introduce selection bias, and fail to capture longer-term effects.
Implicit feedback, derived from naturally occurring user interactions, offers an alternative source of evidence about system performance at scale. Interpreting implicit feedback, however, presents substantial methodological challenges since the same behavior can reflect different user goals and outcomes depending on the interface, context, and task. As a result, behavioral signals can be ambiguous, confounded, and difficult to compare across users and settings. I’ll argue for more rigorous approaches to implicit feedback evaluation that account for these sources of variation and better connect observed behavior to system quality.
-
At what cost? Daniel Fried, Carnegie Mellon University
An agent that completes a task in 30 seconds is not the same as one that takes 30 minutes. And an agent that succeeds on its own is not the same as one that succeeds only after extensive human steering. Yet many agent evaluations collapse these differences into the same success score. In this talk, I’ll argue that we should evaluate not just whether agents succeed, but the cost of getting there—across outcome quality, machine effort, and human effort.
-
Agent control specification: From AI policy to runtime enforcement Mike Shi & Liam Crumm, Microsoft Responsible AI
As AI agents gain the ability to use tools, access data, and take action, organizations need a reliable way to enforce what agents are allowed to do. This session introduces the Agent Control Specification (ACS), a portable approach to runtime governance across frameworks, models, and clouds. See how policies are evaluated at key points in the agent lifecycle to allow, deny, transform, or escalate an operation, and how agents enforce those decisions. Learn how local enforcement and a managed ACS API help teams turn organizational requirements into consistent runtime controls.
Coffee break
Breakout group discussions
-
Challenges in evaluating long-horizon tasks and multi-agent coordination
Group 1, Facilitator:
Stephanie Milani (opens in new tab), Microsoft Research & Johns Hopkins University
-
Evaluating human-centered outcomes of human-agent interaction
Group 2, Facilitator:
Jenn Wortman Vaughan (opens in new tab), Senior Principal Research Manager, Microsoft Research, New York City
-
Safety, red-teaming, and impact assessment of autonomous agents
Group 3, Facilitator:
Shujaat Mirza (opens in new tab), Senior AI Safety and Security Researcher, Microsoft AI Red Teaming
-
Beyond unit tests: Evaluating agentic workflows where judgment matters
Group 4, Facilitator:
Ramayya Krishnan (opens in new tab), Director, AI Measurement Science and Engineering Center (AIMSEC), Carnegie Mellon University
Closing remarks
Hanna Wallach, Vice President and Distinguished Scientist, Microsoft Research