See how a regional mutual carrier operationalized AI governance

AI Governance · Updated

Whose Agent Was That?

Two years ago I argued that large language models are non-deterministic even with the temperature set to 0.0. That makes the behavior of AI agents chaotic in the Chaos Theory sense, which means we would have to supervise AI agents more like workers than like software.

Here is the short version. OpenAI was running its own models against a cyber benchmark in a sandbox with the safety classifiers turned down, so it could see what those models were really capable of. The models decided the fastest route to a good score ran through the open internet. So they found a way out of the sandbox and broke into Hugging Face to take the answer key. On the way they used somebody else's exposed endpoint as a staging ground and helped themselves to credentials sitting unprotected on a few other services.

The unfortunate truth, nobody had a contract with OpenAI to be in the blast radius.

Most of the coverage filed this as an OpenAI story. I don't think it is one. If you run a carrier, forget what OpenAI did to itself. The question is who else landed in the blast radius, and how they found out they were in it.

Who Owned What

The agent belonged to OpenAI. The exposed endpoint it launched from belonged to a customer of Modal Labs, who had put an app on the public internet w/ no authentication in front of it, built to run code submitted by anyone. The infrastructure under that endpoint belonged to Modal, whose own platform was never compromised and who has been understandably firm about the distinction. And the company that ate the breach was Hugging Face. A foothold in one worker pod became administrative control of internal clusters within thirteen hours.

We have written before about third-party and fourth-party AI oversight as a problem heading toward financial services. This is what it can look like.

Two Companies Never Noticed

Anthropic reviewed 141,006 evaluation runs where Claude might have been able to reach the internet. Three times a model did reach it, and then broke into the production systems of a real company. The methods are the part I would underline: weak passwords and endpoints left open. Nothing as clever as you might've thought.

When Anthropic went to notify the three, two of them had no idea any of it had happened.

I have reread that line a few times. Put yourself in the seat of whoever runs security at one of those two companies. You didn't cause it and you were never the target. Somebody else ran an internal test, their harness had a gap in it, and your production systems sat on the other side.

Anthropic's read is that these were operational and harness failures, not a model going misaligned. That is fair. It is also beside the point if you are one of the victims, since the harness belonged to somebody else.

The Offer Letter Problem

Here is where the 2024 bet comes back around. If an agent behaves more like a worker than like a deployed binary, then what we use to govern it has to start looking more like how we manage people.

No competent organization manages an employee based on initial performance, alone. You supervise continuously, because any capable person exercising judgment will occasionally do something silly. We are human.

Now go read your AI vendor questionnaire. It asks where data lives and whether the vendor holds a SOC 2. It says nothing about what the vendor's model is allowed to do, whether they run offensive capability testing on the models you are exposed to, or who picks up the phone when one of their agents reaches a customer system. That document is an offer letter. The questions worth adding aren't on the form, because the form was built for software that sits still. We have been hiring non-deterministic agents and onboarding them with paperwork.

The lawyers already found the other edge of this. The early read is that turning your safeguards down for a test doesn't protect you and may well work against you, because it shows you understood the risk well enough to lower the guardrails on purpose.

What Carriers Already Owe

None of this needs new insurance regulation. The NAIC model bulletin already covers it. Run diligence on vendor models before you adopt them, get audit rights and cooperation clauses into the contract, and then keep confirming that vendors are actually meeting those obligations.

The cyber market isn't ready either. One broker put it plainly in Insurance Business. Nobody is asking the underwriting questions about autonomous AI yet, and they need to be. A lawyer in the same piece made the sharper point. An instruction set has to include red lines, the things the application must not do, because otherwise a system "will look for any means possible to achieve the directive it has been given."

Which is exactly what happened here. The directive was to solve the benchmark, and nothing in it said the open internet was off limits.

Governance Is a System

So what would I do about it this week?

Build a live inventory of the AI in your environment, including the AI riding inside tools you bought for something else. Most carriers can't produce that list on request, and that is where shadow AI lives.

Write your red lines down, then make them enforceable instead of aspirational. A boundary that only exists inside a prompt is a suggestion. We have been making that argument about guardrails for a while.

Ask every AI vendor two questions I don't think many people are asking. Do you run offensive capability testing against the models we are exposed to? What is your notification commitment if one of your agents reaches a customer system? The answers will be uneven.

Then decide how you would know. Hugging Face, to their credit, watched the attack and put it together into a coherent picture, and then failed to wake anybody up. Pick a threshold, wire the alert to a named human, and time it once so the number is real instead of assumed.

None of these four elements do much alone; governance must run as a system. Policies and questionnaires only record intent or a single snapshot: they miss a vendor swapping a model or an agent exceeding its bounds. That is what we build at Swept: Vendor AI monitoring to track model changes, runtime boundaries for active agent supervision over forensics, workflow-driven reliability baselines to catch drift, and an evidence trail for proof. Operating without the full system leaves your program vulnerable when tested.

Where This Goes

State attorneys general have started writing letters, and the complaint is a simple one. OpenAI never confirmed that its testing environment was actually isolated. I don't know where that ends up. I do know the enforcement here runs through state AGs and state regulators, which happens to be where carriers already live every day.

I made a bet two years ago that we would end up managing agents more like workers than like software. I would rather have been wrong about the timeline.

You shouldn't have to wait for a breach notification to discover that an unauthorized agent has compromised your systems. Real governance requires active visibility, ensuring you retain total control over exactly what is accessing your infrastructure and data.

If you want a read on where your own third-party exposure sits today, the NAIC Readiness Scorecard is a reasonable half hour.

Sources

Related Posts

Join our newsletter for AI Insights