This post is about AI incident response for the systems you run: what to do when a model, a chatbot, or an agent in production is the thing that caused the harm. If you searched this phrase looking for AI tools that help a SOC triage alerts faster, that is a different topic with a different set of vendors. Here, the model is the incident.
Most companies have an incident response plan. Almost none of them have a line item for "the model did it." The existing plan assumes a human or a piece of malware acted, that the harm looks like a breach, and that the evidence is sitting in a SIEM waiting to be queried. An AI incident has a different shape, and the differences are specific enough to name.
The harm is usually plausible-looking output rather than an intrusion: a policy term the model invented, a refund the agent approved, a summary that dropped the one clause that mattered. The actor may be your own system following instructions, either yours or someone else's, tucked into a document it was asked to read. And the evidence is a trace nobody thought to keep, because at the time the request looked like an ordinary API call.
What counts as an AI incident
An AI incident is an event in which an AI system's output, action, or data exposure caused or nearly caused harm to a person, an organization, or a decision, or in which the system operated outside the boundaries it was authorized for. The near-miss clause is deliberate. A jailbreak that produced nothing harmful still tells you a guardrail failed, and you want that in the log.
We sort AI incidents into six classes. The class determines who owns containment, so assign one at triage.
| Class | What happened | Examples | Usual first signal |
|---|---|---|---|
| A. Harmful or wrong output reached a customer or a decision | False, unsafe, or discriminatory output that someone relied on | Hallucinated policy terms, wrong medical or financial guidance, an adverse outcome that tracks a protected class | User complaint, QA sampling, eval regression |
| B. Data went where it should not | Sensitive data entered a prompt or crossed a boundary | PII, PHI, or MNPI pasted into a public model; a RAG pipeline retrieving across tenant lines | Gateway DLP hit, tenant report |
| C. Unauthorized or unintended agent action | An agent did something real that nobody approved | Sent the email, changed the record, executed the refund, called the right tool with the wrong argument | Downstream alert, reconciliation mismatch |
| D. Prompt injection or jailbreak succeeded | Instructions from outside the system prompt won | Agent followed text in a web page or attachment; user talked the model past its refusal | Output monitor, red-team finding, screenshot on social media |
| E. Model or vendor failure | The model under you changed or stopped | Silent model version change, provider outage, drift past a threshold | Eval regression, error-rate spike, drift monitor |
| F. Shadow deployment discovered | A model is in a workflow nobody reviewed | A team wired an API key into a process and shipped it | Expense report, egress logs, a mention in standup |
One event can span two classes. An injection that leads an agent to change a record is D and C, and it should be logged as both, because the fixes live in different places. The injection fix lives in input handling and supervision, while the action fix is a permissions change. We covered the first in our post on indirect prompt injection and the second in agentic AI governance.
AI incident examples people actually ask about
Three widely reported cases cover most of the pattern, and none of them involved a breach. An airline's customer-service chatbot described a bereavement fare policy that did not exist; the airline argued the bot was responsible for its own statements, and a tribunal held the airline to what it had said. Lawyers in more than one jurisdiction filed briefs citing cases a model fabricated, complete with plausible quotations, and were sanctioned. A car dealership's website chatbot, steered by a visitor, agreed to sell a vehicle for one dollar and called the offer legally binding; nobody took delivery, but the screenshot circulated for weeks.
The first two are class A with the decision already acted on. The third is class D leading to a class A near-miss, and a reminder that reputational blast radius does not require a completed transaction. All three share a root cause: output went straight from the model to a person who treated it as authoritative, and nothing stood between them.
Severity tiers for an AI incident
Generic severity scales measure availability, which says little about what an AI incident cost anyone. Five questions set the tier instead: Was regulated data involved? Was it customer-facing? Has a decision already been acted on? Is it reversible? How many records or people are affected, and do you actually know?
| Tier | Criteria | Response posture |
|---|---|---|
| Sev 1 | Regulated data (PII, PHI, MNPI, cardholder data) left the organization or crossed tenants; or an irreversible action hit customer or financial records; or wrong output drove customer decisions about health, money, or legal rights. Count in the hundreds, or unknown. | Incident commander, legal, and an executive sponsor engaged within the hour. Assume a regulatory clock may be running. |
| Sev 2 | Customer-facing wrong output or agent action that is reversible; or regulated data exposed inside the organization but not outside it. Affected count bounded and known. | Contain within the hour, investigate within the business day, legal informed. |
| Sev 3 | Caught before any decision was acted on; injection that succeeded but produced no harmful action; drift above threshold with no confirmed bad output. | Standard on-call handling, postmortem required. |
| Sev 4 | Shadow deployment with no evidence of harm; vendor outage absorbed by fallback; eval regression in staging. | Fix in the normal change process. Log it anyway. |
One rule sits on top of the table. If you cannot count the affected users because the system did not log them, treat the incident as one tier higher than the evidence suggests. Unknown blast radius is itself a finding, and it usually becomes the first postmortem action item.
The AI incident response runbook
The phases are the ones you already run, with AI-specific work inside each.
Detect
The signals worth wiring up: eval regressions on a fixed test set that runs on every model or prompt change, output monitors (PII detectors, refusal-rate shifts, policy classifiers), gateway logs (tool-call spikes, token anomalies, unexpected identities), vendor status pages, and downstream reconciliation for anything an agent can write to. In practice the most common first signal for class A is still a user report. If that stays true for more than a quarter, your monitoring is behind your exposure.
Triage
Assign a class and a tier in the first fifteen minutes. Two questions do most of the work: has anyone acted on the output, and can we undo it? A third overrides both. Is the system still doing it right now? If yes, contain first and come back.
Contain
This is the phase where AI incidents differ most, and where preparation shows. The options, in rough order of blast radius:
- Kill switch at the gateway. Stop the feature for everyone. Crude, fast, and the right call for any active Sev 1.
- Route to a fallback model. The previous version, or a more restrictive one with tools disabled.
- Disable the specific tool for the agent. Leave it able to read; take away write, send, and execute.
- Revoke the agent's credentials. Its own identity, not a shared service account, so the revocation hits only the agent.
- Pause the workflow. For orchestrated pipelines where a partial run is worse than no run.
Every one of these takes minutes if it was built before the incident and days if it was not. You cannot revoke an agent's credentials when it shares a key with the payments service, and you cannot disable one tool when its permissions are a single all-or-nothing scope.
Investigate
Pull the full trace for every affected request: the identity that made it, the model and version that served it, the system prompt version, the complete prompt including every retrieved chunk, each tool call with its arguments, the output, and the policy decision at each step. You cannot investigate what you did not log. The most common dead end we see in AI postmortems is a trace that ends at "called the model," with the retrieved context gone because the index was re-embedded the following week.
Remediate
Rollback comes first: model version, prompt version, or index snapshot. Then prompt or policy changes, then vendor escalation with the trace attached. Retraining is slow and rarely the right first answer, so do not promise it in the incident channel. Before closing, add a regression test that reproduces the incident and confirm it now fails safely.
Notify
Internal notification follows the tier. External notification (affected customers, regulators, the vendor) is a legal and privacy decision, informed by facts the AI system owner produces. The engineer who knows exactly what the retriever returned should not be the one drafting the customer notice.
Postmortem
Blameless, and organized around one question: which guardrail would have caught this, and where would it have had to sit? The acceptable answers are a change to a policy, a permission, or a log. "A human should have checked" describes the failure without fixing it.
Who does what
The incident commander should not be the AI system owner. The person who built the system is the wrong person to decide whether it is a Sev 1.
| Activity | Incident commander | AI system owner | Security | Legal / privacy | Communications | Vendor |
|---|---|---|---|---|---|---|
| Triage and tiering | A | R | C | C | I | I |
| Containment | A | R | R | I | I | C |
| Investigation | A | R | R | C | I | C |
| Remediation | A | R | C | I | I | R (model or platform fixes) |
| External notification | C | C | C | A / R | R | I |
| Postmortem | A | R | C | C | I | C |
R = responsible, A = accountable, C = consulted, I = informed. The vendor column is easy to forget until a class E incident, at which point you will want a contract clause obligating them to hand over version history and incident details.
Regulatory triggers to check at triage
Four regimes come up most often, and the summaries below are where to start rather than the last word. Confirm applicability with counsel.
EU AI Act, Article 73. Providers of high-risk AI systems must report serious incidents to the market surveillance authority of the member state where the incident occurred. Article 3 defines a serious incident to include death or serious harm to health, serious and irreversible disruption of critical infrastructure, infringement of Union-law obligations protecting fundamental rights, and serious harm to property or the environment. The Act sets a general reporting deadline of no later than 15 days after the provider becomes aware, shortened to 2 days for widespread infringement or critical infrastructure disruption and 10 days in the event of a death. Deployers who identify a serious incident are expected to inform the provider. Whether your system is high-risk is a classification question to settle before an incident, not during one.
NAIC Model Bulletin. For carriers, the bulletin expects the written AIS Program to address how the carrier monitors its AI systems, corrects errors, and handles consumer complaints and inquiries about AI-influenced decisions. Its examination section lists the records a department can request during an investigation or market conduct exam, and an AI incident log with the trace behind it belongs in that file. Our summary of the NAIC bulletin covers the full scope.
State breach-notification laws. A class B incident involving residents' personal data may trigger the state's breach statute on its own timeline, regardless of whether an AI system was involved. Treat PII pasted into a third-party model as a potential disclosure and let privacy counsel decide.
Materiality disclosures. Public companies carry disclosure obligations that an AI incident can trigger like any other. That is counsel's call, but it starts with a severity tier they trust.
Log it before you need it
Look back at the runbook and count how many phases depend on records that exist before the incident starts. Triage needs the affected count, containment needs an agent identity you can revoke and a tool scope you can narrow, investigation needs the full trace, and notification needs to know which tenants and records. AI incident response is mostly a logging and authorization problem, and both can be solved in advance.
Per request, the record you want has six fields: the identity that made the request (a user or a specific agent, never a shared key), the model and version that served it, the full prompt with its retrieved context, every tool call and its arguments, the output, and the policy decision (allowed, blocked, redacted, or routed). That is the AI audit trail regulators are starting to ask for, and it is the same artifact your incident commander needs at two in the morning.
There is one natural place for that record to be produced. A gateway that sits between users, agents, and models sees every request in both directions, so it can log the full trace without asking each application team to instrument itself. It is also the component that can refuse, redact, or reroute a request, which is why the kill switch and the fallback route live there too. Teams that bolt logging onto each application instead end up with six partial traces and an argument about which one is right.
A one-page AI incident response plan
Copy this, fill in the names, and put it next to the plan you already have.
- Scope. Which AI systems this plan covers (if you cannot list them, start there), and a definition of an AI incident that includes near-misses.
- Taxonomy. Six classes: harmful output, data exposure, unauthorized agent action, injection or jailbreak, model or vendor failure, shadow deployment. Every incident gets at least one.
- Severity. Sev 1 through 4, set by regulated data, customer exposure, decisions acted on, reversibility, and affected count. Unknown count bumps the tier.
- Roles. Incident commander rotation (never the system owner), AI system owner per system, security lead, legal and privacy contact, communications contact, vendor escalation path.
- Detection sources. Eval suite on every change, output monitors, gateway logs, user-report intake, vendor status feeds, downstream reconciliation.
- Containment inventory. Per system: the kill switch, the fallback model, which tools can be disabled individually, which credential to revoke, how to pause the workflow. Test quarterly.
- Trace requirements. The six fields above, retained to cover your longest regulatory look-back, with retrieved context preserved as it was at request time.
- Notification matrix. Internal contacts by tier; external triggers (EU AI Act Article 73, NAIC AIS Program expectations, state breach statutes, materiality) with the counsel who owns each decision.
- Postmortem template. Timeline, class and tier, trace excerpt, the guardrail that would have caught it, the change shipped, the regression test.
- Review cadence. After every Sev 1 or 2, and at least twice a year, because the model under you will change whether you plan for it or not.
The plan is short on purpose, because most of its weight sits in the containment inventory and the trace requirements, and those are engineering work rather than documents. The organizations we have seen handle AI incidents well did that work before anything went wrong: they logged every request, scoped what their agents could touch, and built a place to pull the plug. At Swept AI we build the gateway layer that produces that trace and holds the kill switch, so the plan above has something to act on. If you are working out what "when it goes wrong" should look like for your systems, talk to us.