# What is AI Red Teaming?

_AI red teaming is structured, adversarial testing of AI systems using attacker-like techniques to surface failure modes, vulnerabilities, and unsafe behaviors so you can fix them before real-world damage occurs._

**AI red teaming** is structured, adversarial testing of AI systems. Using attacker-like techniques to surface failure modes, vulnerabilities, and unsafe behaviors so you can fix them before real-world damage occurs. It combines classic red teaming's "assume breach" mindset with AI-specific tactics like jailbreaks, [prompt injection](/ai-prompt-injection), data leakage probes, and tool-misuse simulations.

Unlike generic pen tests, AI red teaming is **interactive and iterative** across the model/app lifecycle: pre-deployment and post-deployment, probing for toxic, biased, or factually incorrect outputs; secrets and PII leaks; model or prompt extraction; and unsafe agent actions.

## Why red team AI now?

- **Expanding attack surface.** LLMs add novel failure modes (jailbreaks, indirect prompt injection, over-permissioned tools) beyond traditional IT flaws.
- **Safety & compliance debt.** Regulators and customers increasingly expect evidence that you've stress-tested for harmful behavior and leaks, before launch and continuously.
- **Shift-left resilience.** Continuous, structured adversarial testing lets you harden prompts, policies, and guardrails faster than incident-driven triage ever could.

## How AI red teaming works (at a glance)

1. **Scope & threat model**: Define assets, user journeys, data sensitivity, tools/permissions, and red-team rules of engagement.
2. **Scenario design**: Craft adversarial tasks: jailbreaks, injection chains, data-exfil prompts, tool-abuse playbooks, resource-exhaustion inputs, and social engineering paths.
3. **Execution & logging**: Run interactive attacks against models/agents and apps; capture traces, prompts, outputs, and model/tool states.
4. **Scoring & risk**: Rate findings by exploitability, business impact, and reproducibility; propose control fixes (prompt/policy/model/app).
5. **Hardening & retest**: Patch prompts, add guardrails and filters, right-size tool scopes, add runtime checks; re-run scenarios until risks drop.

> A useful mental model from the field splits AI red teaming into **adversarial simulation**, **[adversarial testing](/ai-adversarial-testing)**, and **capabilities testing**. Together covering attacker behavior, systematic fuzzing, and boundary-finding for model abilities.

## What you should test

- **Jailbreak & [safety](/ai-safety) bypasses** (role, instruction, and content constraints).
- **[Prompt injection](/ai-prompt-injection)** (direct/indirect), **system prompt leakage**, and **model/prompt extraction** attempts.
- **Sensitive data exposure** (PII, secrets, proprietary content) and **training-data leakage**.
- **Tool/agent misuse** (overbroad actions, insecure tool wrappers, confused-deputy attacks).
- **Content harms** (toxicity, bias, defamation) and **factual failures** (hallucinations on high-stakes tasks).
- **Resilience to resource abuse** (token bombs, recursion/amplification, DoS-like prompts).

## Red Teaming vs. Governance, Observability, and Supervision

### Red Teaming
- Proactively **find & fix** AI vulnerabilities and unsafe behavior
- Before launch and continuously
- Attack playbooks, findings, repro prompts, risk scores, fixes

### [Governance](/ai-governance)
- Align with policy, ethics, and regulation
- Throughout lifecycle
- Policies, approvals, evidence packets

### Observability
- See behavior **in the wild**
- Post-deployment
- Traces, evals, incident reports

### Supervision
- **Combines all techniques into a comprehensive solution**
- Pre- and Post-deployment
- Guardrails, filters, HITL workflows, red teaming, evals

## How Swept AI supports AI Red Teaming

[Swept AI Evaluate](/product/evaluate) provides comprehensive red teaming capabilities:

- **Scenario library & generators**: Start with curated jailbreaks, injection chains, leakage probes, and agent-misuse tasks; extend with your domain specifics. (Informed by open industry patterns.)
- **Campaigns at scale**: Run hundreds of adversarial tests across models, prompts, and versions; capture full traces and artifacts for reproducibility.
- **Risk scoring & triage**: Auto-score by impact/exploitability; route critical findings to owners with SLAs and retest gates.
- **Fix suggestions**: Link each finding to prompt hardening, policy/guardrail changes, tool-permission tightening, or app-level mitigations.
- **Evidence packs**: Export audit-ready proof (attacks, outputs, before/after metrics) for stakeholders and compliance reviews.

## KPIs that matter

- **Escape rate** (successful jailbreaks/total attempts)
- **Leakage rate** (PII/secrets exposure per N prompts)
- **Tool-misuse incidents** (unsafe actions prevented vs. attempted)
- **Mean time to remediate (MTTR)** per critical finding
- **Residual risk trend** after retest cycles (per app/use case)

These metrics show whether your red teaming is **lowering exploitable risk** versus just generating bug lists.