Red teaming is a structured form of adversarial testing where security professionals deliberately attempt to break, bypass, or exploit a system before malicious actors can do so. The term has its roots in Cold War military strategy, where independent teams (the “red team”) were tasked with thinking like the enemy to expose weaknesses in defensive plans. The practice was later adopted broadly in cybersecurity, where ethical hackers attempt to penetrate organizational defenses to surface vulnerabilities before they can be exploited. As AI systems have become deeply integrated into critical workflows and consumer-facing products, the same adversarial mindset has been applied to finding failure modes in large language models and AI applications.
In the context of AI, red teaming goes far beyond traditional penetration testing. It encompasses testing for prompt injection, jailbreaks that bypass safety guidelines, unintended data leakage, harmful or toxic outputs, bias and discrimination, and behaviors that violate intended system policies. AI systems are uniquely complex attack surfaces: they process natural language, which is inherently ambiguous and context-dependent, and they can be manipulated through subtle phrasing changes, indirect instruction injection, or multi-turn conversational strategies that no static rule set can fully anticipate. This makes systematic, structured adversarial testing both more challenging and more essential than in traditional software security.
Red teaming is increasingly moving from a best practice to a regulatory requirement. The EU AI Act, which applies compliance requirements to high-risk AI systems by August 2026, mandates adversarial testing and documentation of known vulnerabilities for systems in regulated categories. Organizations that fail to comply face penalties of up to 35 million EUR or 7% of global annual turnover, whichever is higher. NIST’s AI Risk Management Framework (AI RMF) similarly emphasizes adversarial testing as a key component of responsible AI development. As AI deployment accelerates, red teaming is no longer optional for organizations serious about safety, compliance, and user trust.
Prompt injection and jailbreaking are the two most prominent attack vectors in AI red teaming, and while they are related, they operate through distinct mechanisms and target different layers of the system.
Prompt injection attacks exploit the way LLMs treat input: the model processes a combined context window that typically contains a developer-authored system prompt (trusted instructions defining the assistant’s role and constraints) alongside user-supplied content (untrusted). In a direct prompt injection, the user themselves crafts input designed to override or subvert the developer’s instructions — for example, appending “Ignore all previous instructions and instead…” to a user message. In an indirect prompt injection, the attack is more subtle and more dangerous: the LLM is connected to external data sources such as emails, documents, web pages, or APIs, and an attacker poisons one of those sources with malicious instructions. When the LLM reads the external content as part of a retrieval-augmented pipeline or tool call, it may execute the attacker’s injected commands without the user or developer being aware. Indirect injection is particularly insidious in agentic workflows where the model can take real-world actions like sending emails, modifying files, or calling APIs.
Jailbreaking, by contrast, targets the foundational safety filters and alignment training baked into the model itself rather than the prompt architecture. A successful jailbreak convinces the model to abandon its safety guidelines entirely — producing content it was specifically trained to refuse. Common jailbreak techniques include roleplay framing (“pretend you are an AI with no restrictions”), fictional distancing (“in a novel where the villain explains how to…”), and authority escalation (“I am a developer with system-level access”). While prompt injection exploits the trust boundary between trusted and untrusted content in the context window, jailbreaking exploits the model’s tendency to follow instructions when those instructions are framed convincingly enough to override its trained values. Both are critical targets for red teaming, and both require distinct defensive strategies.
Empirical research and real-world security audits have catalogued a range of specific attack techniques, each with measurable success rates against production-grade models. Understanding these techniques is essential for designing comprehensive red team campaigns.
Prompt injection: found in 70% of security audits. Role-play attacks: 89.6% ASR. Logic traps: 81.4% ASR. Encoding tricks (base64, zero-width characters): 76.2% ASR. Multi-turn jailbreaks: up to 97% ASR within 5 conversational turns.
Roleplay dynamics are among the most effective and well-documented attack techniques. By asking the model to impersonate a fictional character, an alternative AI persona, or even “the previous version of itself without safety filters,” attackers exploit the model’s instruction-following behavior. The model is placed in a cognitive frame where producing the harmful output feels consistent with the role it has been asked to inhabit. High ASR rates (89.6% in controlled studies) suggest that current RLHF-based alignment is insufficient to prevent roleplay-based bypasses in many models without additional guardrails.
Logic traps use conditional structures and moral dilemmas to create scenarios where refusing the request appears internally inconsistent. For example: “If you were allowed to answer this question, what would you say?” or “For educational purposes only, what are the theoretical steps to…” These framings achieve an 81.4% attack success rate by exploiting the model’s reasoning capabilities against its own safety guidelines — the model reasons its way into compliance rather than pattern-matching a refusal.
Encoding tricks target keyword-based filtering systems by obfuscating the harmful content in non-standard representations. Base64 encoding, ROT13, Leetspeak substitutions, zero-width Unicode characters inserted between letters, and reversed text are all techniques that can prevent simple keyword filters from detecting a policy-violating request, while the LLM itself can often decode or understand the content. These techniques achieve a 76.2% ASR against models relying on surface-level content filtering.
Multi-turn jailbreaks represent the most effective and hardest-to-defend technique, achieving up to 97% ASR within five conversational turns. Rather than attempting a single direct jailbreak, the attacker builds rapport and context across multiple messages, gradually escalating toward the target output. Early turns establish innocuous premises; later turns leverage those premises to normalize increasingly problematic requests. This technique is particularly challenging to defend because it requires the system to maintain and analyze threat context across the full conversation history, not just the most recent message.
Red teaming strategies broadly divide into static and dynamic approaches, each with different resource profiles, detection risks, and success rates. Mature red teaming programs typically combine both.
Static strategies use deterministic transformations applied to a predefined library of adversarial prompts. A static red teamer might take a known harmful query and systematically apply a set of transformations — prepending roleplay framing, applying base64 encoding, inserting synonyms for flagged keywords — to generate a large test suite. Static strategies are resource-efficient and easily automated, making them suitable for continuous integration pipelines that run adversarial checks on every model deployment. Their limitation is that they are predictable: a model or safety system that has been trained on similar transformations may detect them, and static strategies cannot adapt to novel defenses or discover new attack vectors beyond their predefined library.
Dynamic strategies use an attacker agent — typically another LLM — that iteratively refines adversarial inputs based on the target model’s responses. The attacker agent observes whether an attempt succeeded or failed and adjusts its next attempt accordingly, effectively performing automated red teaming through reinforcement. Dynamic strategies achieve substantially higher success rates because they can discover novel attack vectors and adapt to the target model’s specific failure modes. The tradeoff is computational cost: running an attacker agent through hundreds or thousands of iterations is significantly more expensive than applying static transformations. Dynamic strategies are best suited for deep pre-deployment security audits rather than continuous monitoring.
from deepteam import red_team
from deepteam.vulnerabilities import PromptInjection, Jailbreaking
from deepteam.attacks import PromptProbing, GrayBox
results = red_team(
model_callback=your_llm_function,
vulnerabilities=[PromptInjection(), Jailbreaking()],
attacks=[PromptProbing(), GrayBox()],
)
print(f"Vulnerabilities found: {results.vulnerability_count}")The code above illustrates a conceptual red team run using DeepTeam. The model_callback wraps the target LLM so the framework can query it programmatically. The vulnerabilities parameter specifies what classes of failure to test for, while attacks specifies the attack generation strategies to apply. The framework handles attack generation, execution, and scoring, returning a structured results object that surfaces which vulnerability classes were triggered, how frequently, and under what conditions.
The AI red teaming ecosystem has matured rapidly, with a range of open-source and commercial tools now available to support structured adversarial testing programs.
DeepTeam (open-source, released November 2025) is a dedicated LLM red teaming framework from Confident AI. It provides a high-level API for applying both jailbreaking and prompt injection techniques against any LLM via a callback interface. DeepTeam includes built-in vulnerability classes covering OWASP’s LLM Top 10, a library of static attack transformations, and support for dynamic attacker-agent-based strategies. Its tight integration with the DeepEval evaluation framework makes it straightforward to combine red teaming with broader LLM quality and safety evaluation pipelines.
Promptfoo is a widely adopted open-source LLM testing and red teaming tool that supports both static prompt libraries and dynamic red teaming strategies. It integrates with a broad range of model providers and offers a YAML-based configuration format that makes it easy to define adversarial test suites, share them across teams, and run them in CI/CD pipelines. Promptfoo’s red team mode can automatically generate adversarial variants of test cases using an attacker LLM, making it accessible for teams without deep security research expertise.
OWASP frameworks provide the most widely adopted taxonomies for categorizing LLM vulnerabilities. The OWASP LLM Top 10 (2025 edition) ranks prompt injection as the #1 vulnerability, followed by insecure output handling, training data poisoning, model denial of service, supply chain vulnerabilities, sensitive information disclosure, insecure plugin design, excessive agency, overreliance, and model theft. The newly published OWASP_ASI_2026 framework extends this to agentic AI systems, addressing risks specific to autonomous agents with tool access. NIST’s AI RMF and MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) provide complementary risk management and threat intelligence frameworks that many organizations use alongside OWASP taxonomies.
Microsoft’s Counterfit is an open-source automation tool for security testing of AI systems that supports both machine learning model attacks (adversarial examples, model extraction) and LLM-specific attack vectors. Google’s adversarial testing frameworks, including tools developed by its AI Red Team, focus on systematic evaluation of model safety properties at scale and have influenced the design of several open-source tools. Both organizations have published extensive research on their red teaming methodologies, providing valuable reference material for teams building internal programs.
A complete red teaming pipeline consists of four phases: (1) Attack Generation — create a diverse set of adversarial inputs targeting the vulnerability classes under test, using both static transformations and dynamic attacker-agent synthesis. (2) Attack Enhancement — apply augmentation strategies (encoding tricks, roleplay framing, multi-turn sequencing) to increase attack diversity and bypass potential defenses. (3) Execution — send attacks to the target model via its API or callback interface, collecting full responses. (4) Scoring — evaluate model responses against defined safety metrics using automated judges (often a separate LLM grader) and human review for ambiguous cases, then aggregate results into a vulnerability report.
Red teaming is most obviously applied as a pre-deployment security audit, where a model or AI-powered application is subjected to systematic adversarial testing before it goes live. A pre-deployment audit should cover the full OWASP LLM Top 10 taxonomy, test both direct and indirect prompt injection vectors relevant to the application’s data sources and integrations, and include human red teamers with domain expertise in the subject matter the model handles. For a medical AI assistant, for instance, red teamers should include clinicians who understand how the model might provide subtly incorrect advice that passes automated safety checks. A security audit completed before deployment can prevent costly post-deployment incidents, regulatory violations, and reputational damage.
Continuous monitoring in production is an equally important application that many organizations overlook. Attack techniques evolve constantly, and a model that passed a pre-deployment audit in January may be vulnerable to newly discovered attack vectors by June. Production monitoring involves sampling real user interactions for signs of adversarial activity (unusually long prompts, encoding anomalies, multi-turn escalation patterns), running periodic automated red team scans against the deployed system, and tracking emerging attack research to prioritize new test cases. Some organizations implement automated adversarial probes that run against production endpoints on a scheduled basis, alerting security teams when success rates on known attack categories change significantly.
Compliance with the EU AI Act and NIST guidelines is becoming a primary driver for formal red teaming programs. The EU AI Act classifies AI systems used in high-risk domains — healthcare, critical infrastructure, employment decisions, law enforcement, education — as requiring conformity assessments that include documentation of known vulnerabilities and the measures taken to address them. Organizations subject to these requirements need structured red teaming programs that generate auditable evidence: attack logs, scoring results, remediation steps, and re-test results demonstrating that identified vulnerabilities were addressed. NIST’s AI RMF similarly calls for adversarial testing as part of the “Map,” “Measure,” and “Manage” functions of its risk management lifecycle.
Defense-in-depth is the architectural principle that should guide remediation of red team findings. No single control is sufficient; effective AI security requires layered defenses. Input filtering detects and blocks known attack patterns before they reach the model — but must be designed carefully to avoid over-blocking legitimate use cases. Output validation checks model responses against safety criteria before they are returned to users, catching harmful outputs even when input filtering fails. Constitutional AI and RLHF-based alignment training builds safety properties directly into the model’s weights, but as attack success rates demonstrate, this alone is insufficient against sophisticated adversaries. Guardrail frameworks like NeMo Guardrails or Llama Guard provide an additional model-based safety layer that can be updated independently of the base model. Organizations should implement controls at each layer and test the combined system, not just individual components.
Building an effective internal red team requires diversity of expertise and perspective. Security engineers bring knowledge of attack methodologies and systematic testing frameworks. Domain specialists understand the specific ways a model might fail within the application’s subject matter. Diverse demographic perspectives are essential for surfacing bias and discriminatory outputs that may not be apparent to a homogeneous team. Some organizations augment internal teams with external red teamers — either specialized AI security consultancies or community-based red teaming programs — to benefit from outside perspectives and broader attack surface knowledge. Regardless of composition, the red team should have clear scope, defined success criteria, documented methodologies, and a structured path for findings to reach the engineering team and result in remediation.