See how Forma Health supervises their AI

Keep AI On Spec

Agents drift, models decay, context becomes polluted, and user behavior evolves, all causing incorrect or even dangerous actions. The non-deterministic nature of AI means that quite often, a system that performed well to start can quietly degrade in production, leading to a range of potential problems.

AI is supposed to scale what humans can accomplish. That only works if we trust the output, and trust requires supervision. But humans can’t supervise at scale. As AI use grows, reviewing every output becomes impossible.

Swept Supervision is the active middle layer between your people and your AI agents…the infrastructure that lets you scale AI use without being capped by how much human oversight is available.

Contact us

Swept Governance and Swept Supervision are related but distinct offerings. Governance is a framework, an infrastructure, that enforces your organization’s rules across all AI systems. Supervision monitors specific agents in production and operates within the guidelines of your Governance system.

This page covers Supervision. Learn more about Governance.

If you have AI agents running in production, you likely need both.

Why Doesn’t Traditional Supervision Work?

A classic take on supervision is to have an AI agent or two managed by a human on the IT or Security team, or to have some combination of systems such as Observability, Pre-Production Evals, Documentation, and Orchestration Tooling in place. The trouble with these methods is:

  • Limiting AI use to what can be actively supervised by a human puts a cap on AI that functionally eliminates its scalability and ROI.
  • The older systems/methods are passive rather than active, Each of these is useful, but none of them actively supervise.

Swept Supervision is different. It is active, featuring continuous measurement, outlier detection, and policy enforcement with targeted human oversight.

What We Do

Set a Baseline icon

Set a Baseline

We measure behavior across representative and noisy inputs and record expected ranges for accuracy, escalation rate, token cost and use, etc. This becomes the bespoke standard for your organization, against which we measure all future supervision.

Monitor icon

Monitor

We capture inputs, outputs, plans, and tool calls from live traffic so that internal and external behavior is continuously compared to the baseline.

Detect icon

Detect

Our Supervision layer flags outliers automatically by looking for subtle extraction mistakes, unusual refusal patterns, and slow increases in escalations or cost. The middle layer handles the volume, while targeted reviews and role-based approvals bring humans into the loop only when they need to focus on true anomalies.

Investigate icon

Investigate

We automatically send alerts with a replayable bundle: version, prompt changes, recent data updates, and user context.

Enforce icon

Enforce

Based on your internal guidelines, we apply hard stops and approvals for high-risk actions. Think of this as a circuit breaker for potentially harmful AI behavior: it trips before any disconnect or problematic experience can impact a user.

Improve icon

Improve

We feed confirmed incidents back into evaluations, update prompts and policies, and refresh baselines. Supervision is an iterative process, so we are always circling back.

What You Get

Sampling That Fits Your Risk icon

Sampling That Fits Your Risk

Random sampling for broad coverage, stratified sampling by intent or user segment, and burst sampling during spikes. Swept also sets redaction rules for sensitive fields, encryption in transit, and at rest.

Monitoring at a Glance icon

Monitoring at a Glance

Baseline bands for accuracy and refusal hygiene, safety and hallucination flags by endpoint and role, drift scores for language patterns and output mix, and token cost/usage monitoring, with caps and warnings.

Drift, Bias, and Variance Detection icon

Drift, Bias, and Variance Detection

Our Supervisory AI provides tracking of semantic drift across intent mix and language patterns, outcome drift against ground truth, bias checks across sensitive attributes and cohorts, and variance by prompt, agent/model version, or time of day.

Alerts and Triage Workflows icon

Alerts and Triage Workflows

We work with you to set threshold-based alerts with severity levels, and define incidents (grouped with examples and reproducible steps). Swept offers one-click issue creation with links to failing examples and baseline comparisons that can be sent to Slack,Teams, or your preferred custom destination.

Collaboration and (generalized) Governance icon

Collaboration and (generalized) Governance

Swept sets up roles and permissions for who can change thresholds and approve fixes. We also provide you with comment threads on incidents with mentions and attachments and a full audit log of changes to prompts, models, and thresholds.

FAQs

How much traffic should I sample?
There is no universal answer, but a useful starting point considers your risk tolerance. High-stakes endpoints, such as anything touching sensitive data, financial decisions, or patient-facing outputs, warrant higher sampling rates. Lower-risk, high-volume endpoints can run on lighter random sampling. Swept helps you set appropriate rates at kickoff and adjust them throughout the process as you learn more.
How are baselines set?
Baselines are locked from your last approved evaluation. They capture expected ranges for accuracy, escalation rate, refusal patterns, cost, and latency across representative and intentionally noisy inputs, not just the clean, expected cases.
What counts as drift?
Drift is any meaningful deviation from baseline behavior. That includes semantic drift (changes in intent mix or language patterns), outcome drift (drops in accuracy or rises in hallucinations), bias (score deltas across sensitive attributes or cohorts), and variance (instability by prompt, model version, or time of day). Swept monitors all four.
Can I monitor token cost and usage?
Yes. Tokens are continually tracked with configurable caps and warnings. You set the thresholds; Swept alerts you when behavior approaches or crosses them
How do I keep data private?
Sampling includes redaction rules for sensitive fields and encryption in transit and at rest. Your data does not leave your environment without your authorization. For organizations with strict data sovereignty requirements, Swept operates within your private cloud environment. In fact, setting up a private cloud environment for you is a possibility within our Governance offering.
What is the difference between Supervision and Governance?
Supervision and Governance address related but distinct problems.

Supervision is the AI middle layer between your people and your agents. It samples live traffic, measures behavior against a baseline, detects drift, and enforces policy at the moment an action is about to happen. Critically, it is also what allows AI use to scale because it removes the dependency on direct human review of every output. Without it, the ceiling on AI adoption is equivalent to the ceiling on human oversight.

Governance is broader. It is the continuous practice of enforcing your organization’s rules across all AI systems…not just the agents you built, but the vendor tools your team is using, the access controls that determine who can change what, and the risk-tolerance and use expectations that define what your AI is and isn’t allowed to do.

In practice: Supervision is one of the systems that Governance oversees. If you have sophisticated AI agents in production, you likely need both. If you are primarily managing vendor tools and access controls, Governance is the right starting point.

Note that Swept also offers Evaluation (statistically rigorous tests of agents and models you’re considering, using your data) and Implementation (everything needed to deploy your chosen agents/models securely and in tandem with your other internal systems). One of our most popular entry-points, the Compliance offering, ensures that your organization is perpetually audit-ready.