AI Audit Trail: The Evidence Carriers Need for Examiners, Courts, and the Board

An AI audit trail is the record that lets a carrier reconstruct, months or years later, what an AI system decided, which model version decided it, what data it saw, and what a human did with the output. Application logs fall short: they lack version pinning, integrity guarantees, linkage to human review, and the queryability needed to answer an examiner or a subpoena without reading hundreds of thousands of log lines by hand.

Three audiences now ask for this record: state examiners (in financial and market conduct exams and the NAIC AI Systems Evaluation Tool pilot), plaintiffs' counsel in bad-faith suits, who since the Lokken ruling can compel it, and boards, who want to know whether management knows what its models are doing. This piece specifies the artifact set that serves all three: the model inventory, the decision-level audit trail, retention, bias testing, explainability for rate filings, and board reporting.

Why the AI audit trail became an evidentiary requirement

On March 23, 2026, a Minnesota federal court ruled in Estate of Gene B. Lokken v. UnitedHealth Group, Inc., No. 23-CV-3514 (D. Minn.), that plaintiffs could compel discovery into the carrier's use of AI in coverage decisions. Hunton's analysis of the discovery decision lays out the new procedural reality: when an AI system contributes to a coverage decision and the decision is later disputed, plaintiffs can compel production of the records showing how the system reached its conclusion.

The NAIC pilot exposes the same gap. When its questionnaire asks how the carrier reconstructs a specific automated decision, "we have application logs" does not satisfy it. The expected answer is a queryable audit trail with documented integrity safeguards, retention aligned with state statutes of limitations, and a defined process for producing decision histories on demand.

What financial and market conduct examiners ask about AI

Financial examinations follow the NAIC's Financial Condition Examiners Handbook, updated annually for emerging risks. Market conduct examinations follow a parallel handbook focused on policyholder treatment, claims practices, and sales conduct. Both now carry AI-specific inquiries. The NAIC model bulletin, adopted in December 2023, created no new legal requirements. It explained how existing principles (unfair discrimination prohibitions, rate adequacy standards, market conduct obligations) apply to AI, and examiners are using those principles as the basis for their questions. Our NAIC model bulletin explainer covers its structure.

Examiner questions cluster into five categories. Most carriers can answer the first. Few can answer the rest with timestamped evidence.

Why documentation built for the exam reads as weak

Documentation assembled in the 30 days before an exam shows a single snapshot, with no evidence governance happened between exams. It cannot fill real gaps (if nobody measured a model's 2025 fairness metrics, no writing changes that), and teams working under deadline produce inconsistent formats. Most states examine domestic carriers every three to five years, and each sprint gets bigger as model counts rise.

The model inventory examiners request first

The first request in nearly every NAIC pilot engagement is a complete inventory of AI and predictive models across underwriting, pricing, claims, fraud, and customer service. Foley & Lardner's guidance on what to do when an NAIC AI Systems Evaluation Tool pilot request arrives confirms the inventory is the gating artifact.

Most carriers have partial lists scattered across data science, procurement, and actuarial filings. The pilot expects one governed system of record, vendor models included. The schema below aligns with the MAP function of the NIST AI Risk Management Framework.

One inventory feeds every downstream program: pilot responses, state consumer data requests (which models touched which decisions), rate filings (every rate-affecting entry must reconcile to a filed factor, which saves weeks of actuarial reconciliation in exam season), and vendor contracts. It is also the keyspace for the audit trail, since every decision record references a model ID and version. Our AI model governance guide covers the lifecycle around it.

Keeping the inventory current

An inventory built once drifts out of date within a quarter. Four practices keep it honest:

• Accountable ownership. A director-level owner named in the AI governance charter explains any discrepancy an examiner finds.
• Refresh cadence. Monthly for an active program; quarterly only for portfolios under ten models. Each refresh verifies every field.
• Change triggers. Deployments, retirements, material version or scope changes, and vendor changes force an immediate update, built into deployment and procurement workflows.
• Independent verification. At least annually, a function independent of model owners samples production traffic, identifies the models actually serving requests, and reconciles them to the inventory.

The AI audit trail field specification

Every consequential AI decision needs a record with the fields below. "Consequential" means any output that contributes to a coverage determination, claim disposition, pricing decision, customer-facing recommendation, or fraud referral. The set reflects the NIST AI RMF MEASURE function, the NAIC pilot's documentation expectations, and what plaintiffs' counsel now request under Lokken-style theories. Our AI audit trail overview covers the concept.

Timestamp. Timezone-aware, recorded by the system at decision time (never at log ingestion), to the millisecond. Carriers that timestamp at the application server but not the inference layer cannot prove order of operations when several models contribute to one decision.

Model ID and version hash. The identifier from the inventory, with a hash tying to the training code, weights, hyperparameters, and evaluation results that produced this prediction. Without it, the carrier cannot reproduce a decision if the model was retrained between the decision and the discovery request.

Input hash. A cryptographic hash of the inputs at decision time, with the underlying data preserved separately under restricted access. The hash proves what the model actually saw. The preserved data is the material the carrier uses to walk an examiner or a court through the decision.

Output. The output value with confidence or probability estimates. For classifiers, the predicted class and each class probability. For regression, the point estimate and any uncertainty bands. For LLM-based systems, the full generated output plus any retrieved context.

Confidence. A normalized indicator readable across model architectures, which produce different native uncertainty estimates.

Reviewer. The human who validated, modified, or accepted the output, with the review timestamp. For fully automated decisions, an explicit indicator saying so, plus the rule that authorized the automation.

Override flag. Whether the reviewer overrode the recommendation, with a structured reason code. Override patterns are one of the strongest drift signals and one of the first things examiners ask about.

Downstream action. What the carrier did as a result: claim approved, denied, or referred to SIU; quote issued at a specific premium; customer routed to a service tier. This closes the loop between AI output and customer outcome.

These eight fields are the minimum for every carrier regardless of jurisdiction. States with specific AI disclosure or explainability rules will need more.

Retention pinned to bad-faith statutes of limitation

The most common mistake is defaulting to one or two years because that is existing log retention, then receiving a discovery request for a four-year-old claim.

Some states allow bad-faith claims up to four years after the disputed event, and tolling for minors, undiscovered injuries, or continuing breaches extends some claim types further. Retention should match the longest applicable SOL across every state the carrier writes, plus margin for tolling and litigation holds. For most national carriers that comes to seven years.

Retained is different from retrievable. Records on cold archives that take weeks to restore will miss a thirty-day production deadline. Specify tiers: hot storage under ninety days, warm up to two years, cold to the SOL ceiling, with documented retrieval times for each.

A carrier that cannot produce a four-year-old record because it expired is worse off than one that produces an unflattering record. The first invites an adverse-inference instruction. The second gets to argue the facts.

Log vs. audit trail: three structural differences

Replayability. An audit trail lets the carrier reconstruct the exact decision from the recorded inputs and version, which means preserving inputs, model artifacts, and enough environment metadata to rerun the inference. Lokken invites the request: produce the model and inputs, and we will run it ourselves.

Integrity. Cryptographic hashing of records, write-once storage, segregation of duties between the systems that produce records and the systems that retain them, and independent verification of those safeguards. A log without them invites challenges to its evidentiary weight.

Query interface. A discovery request reads like "all AI-influenced denials of claims involving disputed coverage in 2024 where the model's recommendation was not overridden." An audit trail answers that by claim, model, reviewer, date range, override status, and downstream action in minutes. A log turns it into a multi-week project for legal and engineering.

Bias testing methodology examiners accept

An examiner reviewing a personal lines underwriting model in Colorado wants the disparate impact ratio for every protected class the SB21-169 implementing regulations cover, the test data composition, the proxies identified and validated, the remediation history, and the date and method of the latest retest. Programs built around academic fairness frameworks or vendor dashboards often cannot produce that file. The NIST AI RMF treats multi-metric testing as baseline for consequential systems, and examiners expect a reason for each metric.

The four-test minimum

Run all four on every inventoried model that contributes to a consumer-facing decision.

1. Statistical parity. Compare favorable-outcome rates (acceptance, lower tier, claim approval) between each protected group and the reference group. Under the four-fifths rule, a ratio below 0.8 signals potential disparate impact, and it is the threshold most departments use to triage findings. Clearing 0.8 earns no clean bill of health, but falling below it will be flagged.
2. Equal opportunity. Among customers who actually file claims, does the model identify them at the same rate across groups? This catches models accurate on one population and inaccurate on another while overall accuracy looks fine.
3. Calibration. A predicted 20% claim probability should produce filings on about 20% of similar customers in every group. Miscalibration produces predictable drift in pricing or eligibility, which becomes an exam finding.
4. Counterfactual flip. For a sample of decisions, change the protected-class indicator (or the likeliest proxies) and rescore. A high flip rate means the model is using protected-class information directly or through tight proxies. Costly, but it surfaces proxies before an examiner does.

Protected classes vary by state

Colorado's SB21-169 framework covers race, color, national or ethnic origin, religion, sex, sexual orientation, disability, gender identity, and gender expression for the covered lines (currently auto and life, with property under active rulemaking). New York DFS Circular Letter 7 (2024) on AI systems and external consumer data adds documented testing for race, color, creed, national origin, status as a victim of domestic violence, past lawful travel, or sexual orientation, with periodic retesting; the letter sets no cadence, but exam practice suggests annual at minimum plus retests after material model changes. Bulletin-adopting states, now more than half the country, generally expect coverage of the classes in that state's unfair trade practices act. Multistate carriers need a matrix mapping each inventoried model to each state's list, tested against the union.

Proxy validation and the artifact set

Production models rarely include race or gender as features, so the real testing surface is the proxy network: ZIP code, income tier, occupation, vehicle type, education, credit-based variables, behavioral signals. A model that excludes protected inputs but weights variables correlated at 0.7 with race produces roughly the same disparate impact as one that includes race. Proxy validation computes each variable's correlation with each protected class against a reference dataset (Census ACS data, consented customer data, or licensed demographic data). Correlations above 0.3 are commonly flagged and above 0.5 typically scrutinized; flagged variables are removed, transformed, or retained with documented justification such as ablation testing showing unique predictive signal. What examiners look for is evidence that every variable was tested and a decision was made on purpose.

With two to four weeks to respond to an exam request, the carrier should be able to pull:

• The production model card with version history
• A methodology document (tests, data composition, protected-class scope, cadence) written before the results
• Latest results per protected class per metric, with threshold violations flagged
• The proxy validation file covering every input variable
• A remediation log: finding, response, owner, closure date
• Baseline comparison against the prior model version or a deterministic rule set

The remediation log gets overlooked most. Clean results with no remediation history read to an experienced examiner as a model never honestly tested, or a program that never acted on its own findings.

Explainability for rate filings and predictive model regulation

Bias testing answers the population question. Explainability answers why this applicant got this outcome, and examiners increasingly request both in the same exam.

Why rating law strains under predictive models

Every rating statute requires rates that are adequate, not excessive, and not unfairly discriminatory. Those principles were written around generalized linear models, which produce a coefficient per variable that a regulator can isolate and test against demographic distributions. Gradient-boosted trees, random forests, and neural networks work differently. A boosted model may use 200 features whose interactions drive predictions, so two policyholders with the same credit score can get different rate effects depending on dozens of other inputs.

Proxy effects can hide in interactions: ZIP code alone may not correlate with race at a level that triggers concern, while ZIP crossed with vehicle age, credit score, and commute distance can closely track racial demographics, which is why output testing matters. And models retrained on rolling windows can behave differently six months after approval; regulators have not settled whether approval attaches to the architecture, the trained instance, or outputs within a tolerance band.

In one reported case, a top-20 personal auto carrier filed a gradient-boosted territorial model in Colorado that beat its prior GLM on Gini, residuals, and holdout combined ratio. The Division of Insurance rejected it because the carrier could not show its rating variables were not proxies for race or income.

The NAIC Predictive Analytics Working Group has circulated a draft white paper on evaluating predictive models within existing rate-filing frameworks. It contemplates detailed model documentation (training data, feature engineering, architecture, hyperparameters such as tree count and depth, validation method); standardized disparate impact testing that favors testing the rates produced, after adjusting for actuarial risk, over input-by-input correlation; ongoing monitoring of performance and fairness after approval; and materiality thresholds for when retraining requires a new filing. Connecticut, New York, and Illinois have introduced predictive model documentation and bias testing requirements of their own.

What an examiner-grade explanation contains

A SHAP plot ranks features by contribution. It does not tell an examiner why ZIP code carries weight, which proxies were tested, or how attributions map to the factors in the actuarial memorandum. SHAP values also approximate: they show a variable's contribution to this prediction relative to the average, and say nothing about what removing the variable would do, since removal changes every other variable's behavior in a non-linear model. Examiners need a written narrative they can adopt as their working understanding of the model, with three load-bearing parts:

• Data lineage detailed enough to reconstruct the modeling sample: date ranges, inclusion and exclusion rules, missing-value treatment, record counts per stage, rationale for each transformation. "Five years of policy and claims data" will not survive round one.
• Feature attribution mapped to filed factors. If the model uses 47 features and the filing names 12 rating factors, show how the 47 roll up into or modify the 12, and justify any feature that does not map cleanly under state rating law.
• Fairness results, including classes tested, proxies evaluated, statistical tests, thresholds, and remediation. Buchanan Ingersoll & Rooney finds that explainable AI requirements increasingly include affirmative documentation of bias testing methodology, beyond an attestation that testing happened.

It opens with purpose and scope in plain language and closes with the human review process and override conditions. A carrier that submits only a SHAP plot often spends months on deficiency notices and writes the narrative anyway.

The examiner checklist for AI rate filings

1. Is every model contributing to the rate identified, vendor components included?
2. Could an examiner reconstruct the modeling sample from the lineage documentation?
3. Is every feature mapped to a filed factor, with justification where the mapping is not one-to-one?
4. Is bias testing documented with classes, proxies, methods, and thresholds?
5. Are remediation and post-remediation results documented where disparities appeared?
6. Is the human review process described, including override triggers?
7. Does an inventory entry exist with version, training date, validation date, scope, and owner?
8. Does the filing commit to drift and disparate impact monitoring with a cadence and escalation path?
9. Have vendors supplied enough technical detail to meet these requirements?
10. Do retention periods for model artifacts, training data, and decision logs align with the state's bad-faith SOL?

Explanations must be computed and stored at decision time, because the model that set a March rate may have been retrained five times by the time a regulator asks, and regenerated whenever the model or rating plan changes. Before deploying any update, simulate its rate impact across the book, segmented by geography, demographic proxy, and coverage, so disparate patterns surface before the filing does.

Litigation discovery after the Lokken ruling

In Lokken, the court let plaintiffs seek documents on how UnitedHealth's nH Predict algorithm was built, its anticipated benefits, and "whether it was designed to supplant physician decision-making," along with employee training materials, the identities of employees involved in deployment, and government investigations into its use. Internal investigations were largely shielded and financial incentive data was kept out. The ruling is evidentiary, with no finding that the AI did anything wrong, which is why it matters: the carrier must produce the record of what the AI did and what humans did about it before any merits argument.

Hunton and the National Law Review's analysis both read the doctrine past health insurance. Policyholders can seek discovery into whether AI was used to "supplant the adjuster's decision-making authority" in property and liability claims. The "designed to supplant" framework applies just as cleanly to an adjuster's file, a commercial underwriter's classification, or an SIU referral. Expect Lokken-style requests in first-party property, third-party liability, disability, and workers' compensation matters. A claim file holding an AI severity assessment, an adjuster's signature, and a payment record documents the closing of the file and little else.

The privilege trap

In USA v. Heppner, Judge Jed S. Rakoff held that documents created using a public AI tool were protected by neither attorney-client privilege nor work product, even after counsel became involved. Coverage analysts, SIU investigators, and in-house counsel who use general-purpose AI to analyze coverage or draft denial language may be creating subpoenable material. In a bad-faith suit, a carrier could produce the AI outputs behind the claim decision under Lokken and its own team's AI analysis under Heppner. The fix is a written policy on which AI tools may be used for which work, strict limits on public platforms for legal-adjacent tasks, and logs proving what tool was used for what purpose. That is what lets counsel make defensible privilege assertions.

Per-decision human review as a verifiable artifact

Both rulings lead to the same requirement, which the field specification above is built to meet:

• AI output capture: the original output, timestamp, model version, and inputs, reproducible as the adjuster saw it.
• Human review attestation: a recorded action at decision time showing which output the reviewer saw, what they accepted, what they overrode, and why. "Review and approve" with no substance captured looks like a rubber stamp in deposition, and the Lokken plaintiffs argued exactly that.
• Decision finalization with attribution: the final disposition linked to the reviewer, the AI artifacts reviewed, and any overrides. A file showing AI input and human output with nothing connecting them gives plaintiffs evidence of AI involvement and no evidence of meaningful human engagement.

AI board reporting

Boards are the third consumer, and the question to put to them is blunt: if our AI drifts and our reinsurer walks, are we still a company? For a regional or specialty P&C carrier with a meaningful cat XL program and AI anywhere in rating, underwriting, claims, or marketing, that chain is live against the balance sheet.

The chain: a rating model retrains and shifts its risk weights enough to count as a change in underwriting philosophy. The reinsurance panel is not told, because governance does not treat retraining as treaty-disclosable. A cat loss develops larger than the new model priced for, often in a secondary peril the training window underweighted. The reinsurer calls it a disclosure breach and rescinds, non-renews, or reprices, and the next loss flows to capital.

No published case has yet adjudicated that AI-specific chain. The mechanism itself is well documented. FedNat's catastrophe program expired June 30, 2022 without adequate replacement, and the carrier was ordered into liquidation on September 27, 89 days later. Merced Property and Casualty, 113 years old and A- rated by AM Best, went into liquidation on December 3, 2018, 25 days after the Camp Fire. UPC's Hurricane Ian losses grew from an expected $660 million to $1.5 billion, and its February 6, 2023 statements showed negative surplus. State Farm General stopped writing new California business citing, among other things, a challenging reinsurance market, then non-renewed about 30,000 policies. AM Best issued 55 P&C downgrades against 35 upgrades in 2023, up from 30 downgrades in 2022, and Demotech's July 2022 letters warned 17 Florida carriers of downgrades. Cedent-side protection is thin: in Utica Mutual v. Clearwater, 906 F.3d 12 (2d Cir. 2018), the Second Circuit vacated the cedent's claim that the reinsurer was bound by its settlement.

AI-era pricing models face the same tail. Hippo's Q1 2023 net loss ratio hit 273%, and Aon found secondary perils drove 86% of global insured losses in 2023, the category most likely to embarrass a model trained on a too-recent window. Reinsurers are also building their own AI views: Munich Re sells aiSure to cover AI performance errors, and Swiss Re signed an AI risk-transfer MoU with RIQ in December 2025. A cedent that says nothing still gets a view formed about it. More on that in how reinsurers are already underwriting AI.

A regulatory fine is a P&L event; a treaty walking after a bad cat year hits the balance sheet. Five board deliverables answer the question inside one renewal cycle:

1. A model change log written for treaty underwriters: what changed, when, and how it moves the expected loss curve by line and territory, owned by a named risk officer and refreshed with each retraining. The board verifies at least annually that it reached the lead reinsurer.
2. A treaty disclosure register: what was communicated to which reinsurer at which renewal, with correspondence attached. It is far easier to build as renewals close than after a rescission.
3. An ORSA scenario that models rescission, where the layer disappears before the event, alongside the usual exhaustion case.
4. A rating-trigger runbook that treats an AM Best ERM concern letter as a plausible quarter-end outcome, naming the responsible executive, disclosure cadence, panel notification protocol, and contingency capital plan.
5. A written board policy that retraining a production rating model is treaty-disclosable, so the disclosure question does not live only inside data science.

The routine board report draws from the same records as the exam file: inventory counts by risk tier and vendor status, drift status changes and open remediations, override rates by model, bias test results and closures, exam and discovery requests received, and treaty disclosures made. With the inventory and audit trail in place, it is a quarterly query.

One record set for every audience

The inventory, the decision-level audit trail, the bias testing file, and the rate-filing explanation are one compliance system. Examiners sample from the audit trail, monitoring measures against it, litigators defend with it, and the board report summarizes it. Carriers that wire every production model into it at deployment spend exam season on substance. When a claim is disputed, the carrier with a complete record gets to argue from its own contemporaneous evidence.

To see where your program stands against NAIC expectations, take the NAIC readiness scorecard, or see how Swept handles continuous governance and audit-ready logging for every AI interaction.