AI Supervision · Updated

How do I monitor AI agents in production at an insurance company?

Most carriers we talk to already have agents in production. A few came out of the data science team with a model review behind them, and many more were built by adjusters, underwriters and service reps in Copilot, Claude or ChatGPT, with no record beyond whatever someone remembered to type into an intake form.

Finding those agents, testing them, and keeping them within the limits your governance committee sets is the work we call AI supervision.

What does monitoring an AI agent in production involve?

Monitoring an AI agent in production means knowing the agent exists, testing it against a baseline before launch, limiting what it can do, and watching its behavior, cost and records for as long as it runs. For a carrier, that adds up to seven pieces of work.

Production agent checklist7 items
  • An inventory of every agent, with its owner, the data it reads and the systems it can write to
  • A baseline evaluation before launch, run against cases your own staff have already worked
  • Guardrails on inputs and outputs, especially anything that touches policyholder data
  • Approval on every write to a claims, policy or billing system
  • Drift detection that compares live behavior with the baseline
  • Cost tracking by agent and by department
  • An audit trail showing who triggered each action and what the agent did

The gaps usually sit around agents that never went through the data science team. Our guide to monitoring a claims intake AI agent walks through the baseline step for one agent in detail.

How do I find every AI agent we already have running?

You find them by routing AI traffic through a single supervised point that every agent has to pass through, because self-reported inventories miss most of what's running.

Carriers that rely on a survey or an intake form tend to find the agents their data scientists built and few others. A claims supervisor who built a Copilot agent to summarize loss notices may not think of it as an agent at all.

Once traffic runs through one point, a new agent shows up the first time it makes a call, along with the data it touched. From there, sort by three risk signals: whether the agent touches sensitive data, whether it affects a business process such as claims handling or underwriting, and whether it writes to a system of record.

Agent typeExampleRisk signalsWhat to supervise
Personal productivity agentLower riskA Copilot agent that drafts broker emailsPolicyholder details in prompts; no named ownerSensitive data in prompts and outputs
Business-user workflow agentMedium riskA Claude or ChatGPT agent that pre-fills underwriting submissionsFeeds a business decision; built outside the data science processAccuracy against a baseline, and drift
Team-built production agentHigher riskA claims intake agent that opens claims from first notice of lossWrites to the claims system; policyholder data at volumeApprovals on writes, drift, cost and audit trail
Vendor agentHigher riskAn AI feature inside your policy or billing softwareChanges when the vendor updates itRe-tests before version changes; outputs through an API or logs
MCP or API connectorHigher riskA connector that lets agents change billing recordsOpens a system of record to any connected agentWhich agents use it, what each call writes, per-tool shutoff
Sort by sensitive data, business process impact and writes to systems of record

Policyholder data typed into personal AI tools is a related problem with its own fixes. See how carriers stop policyholder data leaking into ChatGPT.

What's the difference between monitoring and supervising an AI agent?

Monitoring tells you what an agent did, and supervision adds the work on either side of that: evaluating the agent before launch, then acting on it in production with alerts, blocks and rollbacks.

A dashboard of traces and token counts is useful when someone has time to read it. At most carriers nobody does, and the committee that approved the agent meets once a quarter.

Supervision ties each agent to a baseline and a set of limits, then responds when the agent moves outside them. An agent that starts posting reserve changes it was never approved to make gets blocked before the write lands, and its owner gets an alert.

Gartner uses the term guardian agents for agents that supervise other agents, keeping their actions within set goals and boundaries and stepping in to flag or stop risky ones. For a fuller definition, see what AI supervision is, and for the signals worth capturing, our explainer on AI agent monitoring.

Fig. 1 · The supervision loopThe AI supervision loop: inventory every agent, evaluate and set a baseline, add guardrails and approvals, supervise and alert continuously, and re-test before any version change.
  1. 00 InventoryEvery agent, from every source, in one list.
  2. 01 Evaluate and baselineTest before launch on cases your staff already worked.
  3. 02 Guardrails and approvalsLimits on data, and approval on every write.
  4. 03 Supervise and alertLive behavior compared with the baseline, continuously.
  5. 04 Re-testBack to evaluation before any version change.

What is an AI agent management platform?

An AI agent management platform is Gartner's name for a central tool that manages a company's AI agents wherever they run, covering security, governance and cost in one place.

Gartner describes the category through a set of capabilities. In our paraphrase, they include:

  • Discovering and registering agents in one inventory, regardless of where they were built
  • Managing each agent's lifecycle from testing and launch through monitoring and retirement
  • Setting guardrails that apply whatever platform the agent runs on
  • Evaluating agent quality and decision-making over time
  • Managing agent identity and access
  • Governing which data, APIs, MCP servers and models an agent can connect to
  • Tracking cost and usage

Source: Gartner, AI agent management platforms market definition.

For a carrier, the category matters because agents spread faster than a governance committee can review them, and a platform keeps that review running between meetings.

How do I limit what agents can write to our claims, policy and billing systems?

Require approval on each write before it lands in a system of record, tie every call to a named person, and keep the ability to shut off a single tool without stopping the whole agent.

Agents are reaching claims, policy and billing systems through MCP servers and APIs. Writing to them can change a policyholder's coverage, payment or claim.

The setup we see hold up at carriers has five parts:

  • An approval step, from a person or a policy rule, before any write reaches the system of record
  • A shutoff for each tool, so a failing billing lookup can be turned off while the rest of the agent keeps working
  • Every call tied to the person who triggered it, including calls an agent makes on their behalf
  • Logs of each request, decision and write, kept where compliance can retrieve them
  • Exclusions for sensitive claims, such as litigated or special investigation files

If you have to pick a starting point, start with writes, since a bad write is the action you'll end up explaining to a policyholder.

How do I catch drift before it reaches thousands of policies?

Evaluate the agent before launch to set a baseline, then compare the spread of its live outcomes against that baseline so a change shows up within 10 to 100 actions.

Drift rarely shows up in a single answer, so watch the distribution of outcomes across many cases. Suppose the share of claims an agent handles without an adjuster starts climbing while the claim mix stays the same. Either the agent changed or someone learned how to word a claim so it gets waved through, and both deserve a look.

Alert sensitivity should follow risk. A reserve suggestion that an adjuster always reviews before it posts can run on a looser threshold. A fraud score or underwriting decision that flows through without review needs a tight one, and your governance committee should name who decides how much drift is too much.

What about AI agents built into vendor software?

Supervise vendor agents the same way as your own, through an API or logs, and re-test them before any version change, because the carrier stays accountable for what they do.

Vendors keep updating their agents, and an update can change behavior you already approved. Ask each vendor how they announce version changes and whether you can test a new version on your own cases first.

Swept pins the model version each agent runs on and tests any move to a new version against the baseline first. Where a vendor exposes inputs and outputs through an API or logs, their agent gets the same supervision as yours. Our AI evaluation scorecard guide covers what to ask before you buy.

How do I keep AI agent costs in check?

Set budgets by agent and by department, and track usage against them continuously so a runaway agent shows up before the invoice does.

An agent stuck in a retry loop, or one that pulls a full claim file into every prompt, can run far past its normal usage before anyone notices.

Gartner includes cost tracking in its definition of an AI agent management platform, alongside security and governance. Department budgets also give business owners a reason to retire agents nobody uses.

What do regulators expect for AI agents?

In states that adopted the NAIC model bulletin, regulators expect carriers to govern and monitor every AI system they use, including third-party ones, and to keep records they can produce during an exam.

The NAIC Model Bulletin on the Use of Artificial Intelligence Systems by Insurers asks for a written AI Systems Program that covers ongoing monitoring, and it expects due diligence and oversight for AI acquired from vendors. It also lists the documentation a department may request, including records of your monitoring and audit activities.

Nothing in the bulletin carves out agents, so plan for examiners to ask about them. For the state-by-state picture, see what state insurance regulators expect from carriers using AI.

Who should own agent supervision at a carrier?

Ownership is shared: the governance committee sets policy, the supervision platform enforces it continuously, and business owners, IT and compliance each own a defined piece.

RoleOwns
Governance committeePolicy, risk tiers and drift thresholds
Supervision platformEnforcing that policy on every agent, between committee meetings
Business ownerWhat their agent does, and whether it's still needed
ITGranting and revoking access to systems and data
ComplianceThe audit trail and the answers examiners ask for

A committee that meets quarterly can't review thousands of agents by hand, and it shouldn't have to. People still review high-stakes outputs like coverage denials and large payments, and supervision tells them which outputs to look at first.

Where does Swept fit?

Swept is the AI supervision platform carriers use to evaluate agents before launch and supervise them continuously after, across agents their teams build, agents business users create, vendor AI and the models they run themselves.

It runs on the carrier's data in the carrier's environment, with guardrails, approvals on system-of-record writes, budgets and usage tracking, and an audit trail ready for an exam.

AI supervision for carriers

Get your agents under supervision, starting with the riskiest ones.

Tell us which agents you have running, and we'll show you what supervising them looks like on your own data.

What else do carriers ask about supervising AI agents?

Do we need to monitor agents employees build for personal productivity?

Not all of them with the same intensity. Sort agents by risk and focus supervision on the ones that touch sensitive data or feed a business process. You can only sort what you can see, so every agent belongs in the inventory first.

Can we supervise agents inside Copilot or our core system vendor's software?

Yes, where the tool can route its traffic through a gateway or exposes its inputs and outputs through an API or logs. Ask vendors about both before you sign, and re-test their agents before any version change.

How is this different from an observability tool?

Observability tools show traces and metrics about what an agent did. Supervision adds evaluation before launch, policy enforcement, approvals on writes, and action when an agent drifts, such as an alert, a block or a rollback.

How quickly can we get our agents under supervision?

It depends on how many agents you have and how they connect to your systems. Start with the inventory, then put the highest-risk agents under supervision first, usually the ones that write to claims, policy or billing systems.

Does supervision replace human review?

No. Supervision points human review at the outputs and agents that moved outside their baseline, so reviewers spend their time on the cases most likely to be wrong.

Join our newsletter for AI Insights