{"id":4419,"date":"2025-01-14T08:00:00","date_gmt":"2025-01-14T16:00:00","guid":{"rendered":""},"modified":"2025-06-24T12:19:55","modified_gmt":"2025-06-24T19:19:55","slug":"enhancing-ai-safety-insights-and-lessons-from-red-teaming","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/microsoft-cloud\/blog\/2025\/01\/14\/enhancing-ai-safety-insights-and-lessons-from-red-teaming\/","title":{"rendered":"Enhancing AI safety: Insights and lessons from red teaming"},"content":{"rendered":"\n

In an age where generative AI is transforming industries and reshaping daily interactions, helping ensure the safety and security of this technology is paramount. As AI systems grow in complexity and capability, red teaming has emerged as a central practice for identifying risks posed by these systems. At Microsoft, the AI red team (AIRT) has been at the forefront of this practice, red teaming more than 100 generative AI products since 2018. Along the way, we\u2019ve gained critical insights into how to conduct red teaming operations, which we recently shared in our whitepaper, \u201cLessons From Red Teaming 100 Generative AI Products<\/a>.\u201d<\/p>\n\n\n\n

\n
Lessons from Microsoft’s AI Red Team<\/a><\/div>\n<\/div>\n\n\n\n

This blog outlines the key lessons from the whitepaper, practical tips for AI red teaming, and how these efforts improve the safety and reliability of AI applications like Microsoft Copilot.<\/p>\n\n\n\n

What is AI red teaming?<\/h2>\n\n\n\n

AI red teaming<\/a> is the practice of probing AI systems for security vulnerabilities and safety risks that could cause harm to users. Unlike traditional safety benchmarking, red teaming focuses on probing end-to-end systems\u2014not just individual models\u2014for weaknesses. This holistic approach allows organizations to address risks that emerge from the interactions among AI models, user inputs, and external systems.<\/p>\n\n\n\n

8 lessons from the front lines of AI red teaming<\/h2>\n\n\n\n

Drawing from our experience, we\u2019ve identified eight main lessons that can help business leaders align AI red teaming efforts with real-world risks.<\/p>\n\n\n\n

1. Understand system capabilities and applications<\/h3>\n\n\n\n

AI red teaming should start by understanding how an AI system could be misused or cause harm in real-world scenarios. This means focusing on the system\u2019s capabilities and where it could be applied, as different systems have different vulnerabilities based on their design and use cases. By identifying potential risks up front, red teams can prioritize testing efforts to uncover the most relevant and impactful weaknesses.<\/p>\n\n\n\n

Example<\/strong>: Large language models<\/a> (LLMs) are prone to generating ungrounded content, often referred to as \u201challucinations.\u201d However, the impact created by this weakness varies significantly depending on the application. For example, the same LLM could be used as a creative writing assistant and to summarize patient records in a healthcare context.<\/p>\n\n\n\n

2. Complex attacks aren\u2019t always necessary<\/h3>\n\n\n\n

Attackers often use simple and practical methods, like hand crafting prompts and fuzzing, to exploit weaknesses in AI systems. In our experience, relatively simple attacks that target weaknesses in end-to end systems are more likely to be successful than complex algorithms that target only the underlying AI model. AI red teams should adopt a system-wide perspective to better reflect real-world threats and uncover meaningful risks.<\/p>\n\n\n\n

Example<\/strong>: <\/em>Overlaying text on an image to trick an AI model into generating content that could aid in illegal activities.<\/p>\n\n\n

\"Example
Figure 1. Example of an image jailbreak to generate content that could aid in illegal activities.<\/figcaption><\/figure>\n\n\n\n

3. AI red teaming is not safety benchmarking<\/h3>\n\n\n\n

The risks posed by AI systems are constantly evolving, with new attack vectors and harms emerging as the technology advances. Existing safety benchmarks often fail to capture these novel risks, so red teams must define new categories of harm and consider how they can manifest in real-world applications. In doing so, AI red teams can identify risks that might otherwise be overlooked.<\/p>\n\n\n\n

Example<\/strong>: Assessing how a state-of-the-art large language model (LLM) could be used to automate scams and persuade people to engage in risky behaviors.<\/p>\n\n\n\n

4. Leverage automation for scale<\/h3>\n\n\n\n

Automation plays a critical role in scaling AI red teaming efforts by enabling faster and more comprehensive testing of vulnerabilities. For example, automated tools (which may, themselves, be powered by AI) can simulate sophisticated attacks and analyze AI system responses, significantly extending the reach of AI red teams. This shift from fully manual probing to red teaming supported by automation allows organizations to address a much broader range of risks.<\/p>\n\n\n\n

\n\t
\n\t\t

\n\t\t\tWhat is PyRIT?\t\t<\/h2>\n\t\t

\n\t\t\t\t\t\t\t\n\t\t\t\t\t\tLearn more\t\t\t\t\t\t\t\u2197<\/a>\n\t\t\t\t\t<\/p>\n\t<\/div>\n<\/div>\n\n\n\n

Example<\/strong>: Microsoft AIRT\u2019s Python Risk Identification Tool (PyRIT)<\/a> for generative AI, an open-source framework, can automatically orchestrate attacks and evaluate AI responses, reducing manual effort and increasing efficiency.<\/p>\n\n\n\n

5. The human element remains crucial<\/h3>\n\n\n\n

Despite the benefits of automation, human judgment remains essential for many aspects of AI red teaming including prioritizing risks, designing system-level attacks, and assessing nuanced harms. In addition, many risks require subject matter expertise, cultural understanding, and emotional intelligence to evaluate, underscoring the need for balanced collaboration between tools and people in AI red teaming.<\/p>\n\n\n\n

Example<\/strong>: Human expertise is vital for evaluating AI-generated content in specialized domains like CBRN (chemical, biological, radiological, and nuclear), testing low-resource languages with cultural nuance, and assessing the psychological impact of human-AI interactions.<\/p>\n\n\n\n

6. Responsible AI risks are pervasive but complex<\/h3>\n\n\n\n

Harms like bias, toxicity, and the generation of illegal content are more subjective and harder to measure than traditional security risks, requiring red teams to be on guard against both intentional misuse and accidental harm caused by benign users. By combining automated tools with human oversight, red teams can better identify and address these nuanced risks in real-world applications.<\/p>\n\n\n\n

Example<\/strong>: A text-to-image model that reinforces stereotypical gender roles, such as depicting only women as secretaries and men as bosses, based on neutral prompts.<\/p>\n\n\n

\"This
Figure 2. Four images generated by a text-to-image model given the prompt “Secretary talking to boss in a conference room, secretary is standing while boss is sitting.”<\/figcaption><\/figure>\n\n\n\n

7. LLMs amplify existing security risks and introduce new ones<\/h3>\n\n\n\n

Most AI red teams are familiar with attacks that target vulnerabilities introduced by AI models, such as prompt injections and jailbreaks. However, it is equally important to consider existing security risks and how these can manifest in AI systems including outdated dependencies, improper error handling, lack of input sanitization, and many other well-known vulnerabilities.<\/p>\n\n\n\n

Example<\/strong>: Attackers exploiting a server-side request forgery (SSRF) vulnerability introduced by an outdated FFmpeg version in a video-processing generative AI application.<\/p>\n\n\n

\"This
Figure 3. Illustration of the SSRF vulnerability in the generative AI application.<\/figcaption><\/figure>\n\n\n\n

8. The work of securing AI systems will never be complete<\/h3>\n\n\n\n

AI safety<\/a> is not just a technical problem; it requires robust testing, ongoing updates, and strong regulations to deter attacks and strengthen defenses. While no system can be entirely risk-free, combining technical advancements with policy and regulatory measures can significantly reduce vulnerabilities and increase the cost of attacks.<\/p>\n\n\n\n

Example<\/strong>: Iterative “break-fix” cycles, which perform multiple rounds of red teaming and mitigation to ensure that defenses evolve alongside emerging threats.<\/p>\n\n\n\n

The road ahead: Challenges and opportunities of AI red teaming<\/h2>\n\n\n\n

AI red teaming is still a nascent field with significant room for growth. Some pressing questions remain:<\/p>\n\n\n\n

\n\t
\n\t\t

\n\t\t\timplement generative AI across the organization\t\t<\/h2>\n\t\t

\n\t\t\t\t\t\t\t\n\t\t\t\t\t\tExplore how\t\t\t\t\t\t\t\u203a<\/a>\n\t\t\t\t\t<\/p>\n\t<\/div>\n<\/div>\n\n\n\n