AI Vendor Risk Assessment: Questions, Testing, and Contract Clauses

A procurement team at a large financial institution once walked us through their AI vendor evaluation process. They had a 200-question security questionnaire covering SOC 2, encryption standards, incident response timelines, and penetration testing cadence. Not one question addressed model drift, output validation, or what the vendor does when the model invents an answer in production.

That is the typical starting point for an AI vendor risk assessment, and for a carrier it is a risky one. The NAIC Model Bulletin puts accountability for vendor AI on the carrier, examiners are already asking for vendor validation records, and a vendor registry is being drafted. This guide covers what to ask, how to test a vendor's claims on your own policies and data before signing, which contract clauses hold up in a market conduct exam, how to handle fourth-party risk, and when building is the better answer.

Why a SaaS questionnaire misses AI vendor risk

Security questionnaires grew up alongside enterprise SaaS. They assume deterministic software: the same input produces the same output, behavior changes only through versioned releases, and security boundaries are well defined.

AI systems behave probabilistically. A language model can return different outputs for identical inputs on consecutive requests. Behavior changes without a software release, sometimes because the vendor updated weights and sometimes because an upstream foundation model provider pushed a change. A SOC 2 report tells you about a vendor's security program. It tells you nothing about whether the vendor's AI hallucinated in 3% of customer interactions last month.

The gaps cluster in five places:

Model behavior. A claims model that performed well in evaluation can degrade within months as input distributions shift. Questionnaires ask about uptime. They rarely ask whether the vendor monitors for model drift, measures hallucination rates, or tracks accuracy over time.
Data handling beyond storage. Training sets, fine-tuning data, retrieval corpora, embedding vectors, prompt logs, and inference outputs each carry distinct privacy exposure.
Supply chain depth. Most AI vendors build on foundation models they did not create. A change to the underlying model ripples through every application on top of it.
Governance maturity. SOC 2 says nothing about whether the vendor keeps a model registry, tracks versions, can roll back, or runs ongoing bias testing.

The carrier owns the outcome, whoever built the model

Regulators have settled where responsibility sits. The NAIC Model Bulletin on Use of Artificial Intelligence Systems by Insurers, now adopted in roughly two dozen states and the District of Columbia, makes the carrier accountable for any AI that touches a regulated decision, including AI it licensed. Section 4 requires a carrier's AI Systems Program to address third-party data and models with the same rigor as internally developed systems. None of that transfers to the vendor. (For what examiners are actually requesting under the bulletin, see NAIC AI bulletin enforcement readiness.)

The industry has not absorbed this. In one 2026 survey of insurance executives, only about 18 percent named third-party model risk as a concern, while roughly 68 percent said outside vendors were a source of their AI. Around 40 percent of the models in use came from more than 70 different vendors.

The exam is where the gap becomes concrete. An examiner asks for the inventory of AI systems in production, then the validation records for a model that helps set rates, then the change history showing who approved the last version and what testing preceded it, then the trail from a specific input to a specific decision. Carriers in the NAIC AI Systems Evaluation Tool pilot have found that they ask their vendor for a model card or bias testing summary and discover the contract never required one. The regulator does not accept "the vendor would not share it."

The trade-secret wall makes this harder. Ask for training data, validation methodology, or feature weights behind a fraud-scoring model and the answer is usually that it is proprietary, which leaves the carrier generating its own evidence about how the model behaves.

The gap is widest at mutuals. A national carrier with an internal data-science group can reconstruct a vendor's validation work. A 300-person regional mutual usually licensed the model precisely because it lacks that bench, and the policyholders affected by a mispriced rate are also the company's owners. A vendor model that discriminates unnoticed breaks the equitable-treatment promise that defines the cooperative, on top of the exam exposure. (More on that in the mutual carrier governance advantage.)

The AI vendor risk assessment questionnaire

Use the categories below on top of your standard security review. Each set is short on purpose: the first five AI-specific questions tend to surface more than the previous 200.

Model governance and lifecycle

Do you maintain a registry of every model in production, with version, training data sources, and performance baselines?
How do model updates reach us? Do we get advance notice, and can we pin to a specific version?
How do you roll back when an update degrades performance?
Do you publish a model card covering training data date range, test data composition, validation methodology, fairness testing protocol, known failure modes, and supported and unsupported uses?
Which governance framework guides your development: NIST AI RMF, ISO 42001, or something comparable?

Monitoring in production

Which performance metrics do you track in production, and how often?
How do you detect drift, and what thresholds trigger alerts or intervention?
Do you log model inputs and outputs? What retention applies, and who has access?
How do you measure hallucination rates, and how do you define accuracy? (A vendor claiming 99% accuracy may define it differently than a regulator would.)
What is your mean time to detect a performance degradation, and will you share your monitoring metrics with us on request?

Data handling for AI workflows

Is our data used for training or fine-tuning? Can we opt out, and how is deletion enforced?
How are prompt logs and inference outputs stored, retained, and permissioned?
How do you handle our data in vector databases or retrieval systems?
What prevents sensitive information in prompts from being logged or persisted?
Where does inference run, and how do you handle residency for AI-specific data such as embeddings?

Output validation and safety

What guardrails prevent harmful, biased, or fabricated output?
How do you validate accuracy for our specific use case?
Do you use output scoring, content filtering, or human review for high-risk decisions?
How do you defend against prompt injection and other adversarial inputs?
What happens, step by step, when an output harms a policyholder?

Bias and fairness

How do you measure bias across protected classes, and which classes do you test for? (Carriers with substantial Colorado, Connecticut, or New York exposure need those states' lists covered.)
What mitigation follows when bias is detected?
For underwriting, pricing, and claims uses, what human review do you require?
Do you commission third-party fairness audits, and how often?
How are prohibited-use policies technically enforced, beyond being documented?

Supply chain and sub-processors

Which foundation model providers do you depend on, and what happens when they change their models?
Who else touches our data or influences outputs: hosting, labeling, fine-tuning partners?
Do you have fallback models from an alternative provider?
How does the contract allocate liability for failures caused by an upstream model change?

Record every answer in the vendor file against a model inventory entry, and treat refusals as findings. A vendor that will not answer the monitoring or fairness questions during a sale will not answer them during an exam.

Testing vendor claims on your own policies and data

A vendor demo runs on the vendor's data. It shows the model can work somewhere. It says nothing about how the model performs on your policyholders, your geographies, your claim types, and your historical mix. Verifying accuracy claims before signing is the single most useful step in the assessment, and it does not require vendor cooperation.

Start from failure modes

Before picking metrics, decide which failures matter. Ask what happens when this system is wrong, who is affected, and what each failure type costs. A first-notice-of-loss assistant that misroutes a claim fails differently from an underwriting score that drifts against a protected class. The answers decide where testing effort goes.

Build a test set from your book

Assemble representative inputs from your own policies, claims, and correspondence, including the awkward ones: endorsements, multi-product households, edge-case coverage questions, inputs in other languages. Run them through the vendor system and score the outputs against thresholds you set. Useful dimensions:

Accuracy and groundedness. Is the answer correct, and is it supported by the policy documents the system retrieved, or did it fill a gap with something plausible?
Completeness and relevance. Partial answers that omit an exclusion are a different failure from wrong answers.
Bias. Does behavior change with names, geographies, or other proxies for protected characteristics?
Refusal accuracy. Does it decline what it should, without refusing legitimate requests?
Information leakage. Can it be induced to reveal system prompts or other policyholders' data?
Cost and latency. Cost per query and response time under realistic load, since a quality gain that doubles cost changes the business case.

Combine automated scoring with domain review; claims and underwriting staff catch errors no classifier will. An LLM-as-a-judge can scale scoring once calibrated against human judgments, though judge models tend to favor longer answers.

Test adversarially

Do not limit testing to expected inputs. Adversarial testing probes for prompt injection, jailbreaks, data extraction, and edge cases outside the vendor's training. A model that scores well on public benchmarks (MMLU, TruthfulQA, HELM, and the like) and fails under adversarial pressure is not ready for a regulated workflow.

Set the baseline you will monitor against

The scores from pre-signing testing become the baseline written into the contract and tracked in production. Without a baseline you cannot answer the basic question an examiner will ask: is this system getting better or worse?

What testing finds in practice

An embedded insurance company handling about 1,500 customer inquiries a month planned to deploy an AI support agent and estimated twelve months to do it safely. Evaluation on its own data surfaced failures no demo had shown. The agent merged information across unrelated insurance products, so a question about one policy returned details from another. It fabricated dollar amounts that looked authoritative. Website data extraction was running at 2.5% accuracy. It generated plausible email addresses that did not exist. Once those were fixed and the agent ran under supervision, the company reached 60–70% automation with zero customer-facing hallucinations, and deployment took six weeks, about 90% less time than the internal estimate. None of those failures would have appeared in a questionnaire.

Swept does this as part of verifying reliability on your own data: evaluation against carrier-specific test cases before go-live, then continued measurement so drift surfaces before a policyholder or a regulator finds it.

Fourth-party and sub-processor risk in AI third-party risk management

A model usually involves more parties than the named vendor: a foundation model API, a hosting provider, a labeling service, a fine-tuning partner. Each one is a sub-processor with access to data or influence over outputs, and the diligence file is incomplete without the chain.

A carrier evaluates a customer service platform in January. In March the platform's foundation model provider ships an update, behavior changes, and the January evaluation is stale. The vendor governs the application layer; the foundation model provider governs the core AI. That is fourth-party risk: your vendor's vendor changes the AI that powers your operations.

The financial services sector has codified how to handle this. The FS AI RMF, published in February 2026 by the Cyber Risk Institute with 108 financial institutions, devotes significant objectives in its Map and Govern functions to third- and fourth-party AI. Its requirements translate directly to carriers:

AI-specific due diligence. Evaluate vendor AI directly for model performance, bias characteristics, hallucination rates, and security posture against AI-specific attacks.
Contracts that address AI behavior. Performance thresholds (accuracy, error rates), bias testing obligations against protected classes, incident notification when AI produces harmful outputs, and audit rights for independent testing.
Downstream dependency documentation. Map the full AI supply chain, not only the vendor in front of you.
Change notification. Vendors disclose when underlying models change.
Independent validation. When the AI changes, validation is refreshed.
Continuous monitoring. Under the Manage function, annual reviews are insufficient for systems that change between reviews. Monitoring covers output-level behavior in your environment, performance against contractual thresholds, and change detection (output distribution shifts, latency changes, and behavioral differences that signal a model swap).

The framework's evidence list is a good checklist for any carrier: model cards, testing results across demographic groups, production monitoring data, and incident histories documenting when and how AI failures occurred. Model risk teams that governed internal models under SR 11-7 now carry the same obligation for AI they did not build.

Concentration risk

Fourth-party risk is concentration risk disguised as vendor diversity. Five AI vendors built on the same foundation model provide less diversification than they appear to. Picture 40% of a sector's customer service AI running on one model: a vulnerability in that model hits 40% of the sector at once.

AI concentration differs from traditional technology concentration in two ways. It is often invisible, because the platforms look independent. And failures propagate differently. A software bug produces consistent, identifiable errors; an AI vulnerability produces inconsistent, hard-to-detect failures across every deployment at once. A biased training set poisons every system built on it, and a prompt injection technique that works on one deployment works on all deployments of the same model.

Contract clauses that survive a market conduct exam

Most AI vendor MSAs signed before the bulletin give the vendor broad latitude to update the model without notice, treat training data as the vendor's confidential property, cap liability at twelve months of fees, and say nothing about bias testing, drift, or sub-processors. The bulletin's obligations sit with the carrier whether or not the carrier has the contractual rights to meet them. Eight clauses close the gaps that pilot reviews and bulletin examinations keep surfacing.

Audit rights. The carrier (or a designated third party under NDA) may inspect, on reasonable notice and cadence, the vendor's documentation on model development, training data provenance, testing results, and incident logs for the model the carrier uses. The audit must be triggerable by a regulatory inquiry, not only on an annual schedule. Vendors typically offer "annual SOC 2 plus model card on request." Hold the line on the right to inspect underlying documentation, even if cadence is constrained.
Model card disclosure. A current model card on signing, on every version change, and on regulator request, with minimum contents specified: training data date range, test data composition, validation methodology, fairness testing protocol, known failure modes, and supported and unsupported uses. The carrier needs it before the regulator does.
Drift-notification SLA. The vendor notifies the carrier within a defined window (typically 72 hours) when its monitoring detects performance degradation, distribution shift, or fairness deterioration above defined thresholds. Separate informational drift (logged, summarized quarterly) from material drift (notified within the SLA). Define material drift in the contract. A starting point: any change of more than 5% in headline accuracy, any disparate impact ratio deterioration that crosses the four-fifths rule threshold, any spike in flagged outputs above the historical baseline.
Training-data attestation. A written attestation, in reasonable detail, of the sources, date ranges, and licensing status of the training data. Vendors resist on trade-secret grounds; categorical disclosure usually settles it. The vendor names the source categories (public claims data, licensed industry data, synthetic augmentation, customer feedback) and confirms licensing for each without exposing specific datasets.
Sub-processor disclosure. A current list of sub-processors that handle covered data or contribute materially to model behavior, with notice of changes within a defined window and a right to object to new sub-processors that present unacceptable risk. The notice creates the diligence trail, even if the objection right is rarely used.
Fairness-test sharing. Fairness results on a defined cadence (quarterly or on every material model change), specified for the protected classes in the carrier's operating states. Reserve the right to have the vendor run a fairness test on a carrier-specified test set at the carrier's cost, since general testing may not reflect your book. A vendor that cannot or will not test against your portfolio is a vendor whose testing you cannot validate.
Exam cooperation. The vendor cooperates with regulator inquiries about its model, providing technical witnesses, documentation, and explanations within timeframes the carrier sets from the regulator's deadline. The vendor bears its own cost up to a cap; the carrier covers effort beyond it.
Exit and escrow. A transition period at the carrier's election, a continued license to the latest model version during transition, and either source-code escrow or a transition data package (model weights, inference code, supporting documentation) deliverable to the carrier or a successor vendor. For a model embedded in pricing or claims, its absence is an existential risk dressed as a contractual gap.

Retrofitting at renewal

Introducing all eight at once will stall. List every AI vendor renewing in the next twelve months and map the calendar against the registry framework's expected effective date: vendors renewing after the framework lands will be more receptive because the registry creates pressure they cannot avoid. Prioritize by exposure. Audit rights, model card disclosure, and fairness-test sharing produce the documentation regulators ask for first, and exam cooperation is the lifeline when an inquiry arrives. Drift notification, training-data attestation, sub-processor disclosure, and exit/escrow can follow on a longer cycle.

Treat the renewal as a procurement evaluation as well as a redline. A vendor that rejects audit rights or fairness-test sharing as policy is telling you how it will behave under regulatory scrutiny, and that model belongs in your inventory flagged for replacement.

The NAIC vendor registry and third-party model law

The NAIC's Third-Party Data and Models (H) Working Group used its March 23, 2026 session to sketch a registration regime for model vendors, with disclosure obligations attached and a clear note that carrier accountability does not transfer to the registered vendor. Registration differs from licensure: licensure would let regulators approve or deny a vendor's right to operate in a state, while registration only requires the vendor to file information and update it on a set cadence. (A later 2026 account noted licensing requirements for vendors could still be part of the eventual model law, so watch the final text.) The working group has also adopted a broad definition of "third party" covering any outside entity supplying data, models, or model outputs for insurance activities.

The group's public materials frame the goal narrowly. Today, when a New York examiner and a Colorado examiner separately review the same third-party underwriting model, they get different documentation packages, scoped differently, with different levels of vendor cooperation. The registry gives examiners one baseline, and that baseline becomes the regulator's expectation of what every carrier using the vendor should already know.

What triggers registration

The draft applies to vendors whose data or models contribute to consumer-facing decisions in regulated lines. The Carlton Fields summary of the spring session lists the categories the group is converging on: pricing, underwriting, claims handling (including coverage decisions and reserve estimation), utilization review, marketing and lead scoring, and fraud detection. Back-office uses with no consumer impact (HR, IT operations, internal analytics) sit outside.

Two scoping questions were unresolved as of spring 2026. One is the contribution threshold: a model that fully automates a decision is in scope, but a recommendation an underwriter overrides 80% of the time is harder. The other is general-purpose foundation models. An LLM accessed by API to summarize claim files arguably falls in scope if the summary influences claim decisions, even though it was never marketed as an insurance product.

Registration as drafted would require a description of the model and intended use, training data sources and date ranges, documented testing methodology including bias testing where applicable, known limitations, change-management practices, and a contact for regulator inquiries.

What the registry does not relieve

A registered vendor's confidential filing answers none of the examiner's questions about your program: what governance you apply to the vendor, how you validated the model before deployment, what you monitor afterward, what your remediation path is, and what contractual rights you hold to the vendor's testing artifacts. The Alston & Bird readout of the spring meeting flagged that state departments will treat the registry as a prerequisite for the carrier's own diligence file, not a substitute.

The two artifacts also have to reconcile. A vendor whose registry filing says one thing while its model card or contract says another will get the carrier in trouble, and catching the mismatch requires the contractual rights to see both. As of the spring meeting, the working group expected public comment in the third quarter of 2026 and consideration for adoption at the November Fall Meeting, with first state implementations in late 2026 or early 2027. Asking for the model card, bias methodology, and sub-processor list now, before vendors are compelled to disclose them, is the stronger negotiating position.

Keep the vendor file current after signing

Vendor evaluation is not a one-time event. Market conduct exams in Colorado, Connecticut, New York, and the bulletin-adopting states routinely look back two to three years, so a file built only at procurement will be stale by the time it is examined. Three habits keep it current:

Tie the vendor file to the model inventory. Each production model gets an entry with business purpose, responsible owner, deployment date, and supervisory cadence. An examiner who asks for "everything that touches underwriting decisions" needs an answer in hours.
Monitor vendor output yourself. Track what the vendor's AI produces in your environment against the contractual thresholds, so a drop shows up in your monitoring before the vendor's quarterly report. Change detection catches the silent model swap.
Treat the file as an evidence pipeline. Every monitoring report, incident notice, and recertification belongs in a queryable system that timestamps it and links it to the inventory entry. Carriers that keep these in shared drives and email threads discover at exam time that they cannot reconstruct what they knew, or when. An AI audit trail built to examiner specification solves most of this.

Build vs buy AI: when to build instead

Sometimes the right answer to a weak vendor is to build. More often it is to buy, and buy carefully. A regional mutual is right to license an underwriting score instead of staffing a data-science team to rebuild it. The discipline is in refusing to license on faith.

The numbers favor buying. 76% of enterprises have stopped building AI in-house entirely, and 60% of AI development time goes to integration work, a cost that exists whether you write the connectors or configure a vendor's. Buying has not fixed outcomes, though. 70-80% of AI projects never deliver their promised value regardless of path, and 65% of total AI costs materialize after deployment. About 41% of business leaders say they are significantly under-resourced in AI talent, and 51% of IT leaders cite governance and compliance as their top barrier to adoption.

Decision criteria

Core or context. If the capability creates competitive advantage for a carrier (a proprietary pricing signal, say), building may be justified. Infrastructure anyone can buy, such as observability or a help desk agent, is context. Engineers building it are not working on what differentiates you.
How unusual the requirement really is. Proprietary data formats or a genuinely unique regulatory constraint can justify building. Most teams overestimate how unusual their needs are.
Scale. Fixed build costs spread poorly over a few use cases. At large volume, per-unit vendor pricing can make building cheaper even after maintenance. Model costs for expected scale, not today's.
Team. Building needs data engineering, ML, and infrastructure skills, at salaries from $100,000 to $300,000. Buying does not remove the talent requirement; it shifts it to people who can configure, monitor, and challenge the vendor's system.
Timeline. Vendor tools run in days or weeks. Internal builds estimated at "a few weeks" routinely take months.
Total cost after launch. Maintenance typically runs 10–20% of the ongoing AI budget, and enterprise systems need $5,000 to $20,000 a month in upkeep. Computing costs rose 89% between 2023 and 2025.
Vendor viability and lock-in. Will the vendor exist in three years, and is this product its focus? How painful is migration? Building trades vendor dependency for dependency on your own team staying put and on open source projects staying maintained.

Hybrid approaches work when the boundary is clear, with the vendor covering a defined scope and custom work filling only what it cannot. Before committing, run a proof of concept on your own integration and data.

When not to train your own model

"Train your own model" is the most expensive version of build. It requires large volumes of clean, labeled data, GPU infrastructure, scarce ML talent, weeks or months of training per iteration, and continuous monitoring and retraining once deployed, plus bias and fairness review of everything you trained. Before going there, check whether a cheaper approach solves the business problem:

Linear regression or simpler machine learning (decision trees, support vector machines, naive Bayes) for forecasting, classification, and pricing problems, which are also easier to explain to an examiner
Rules-based heuristics that codify underwriting or claims expertise
A pre-trained model, fine-tuned on your data if needed
A frontier model accessed through an API or cloud AI service

Define the business problem, check your data, see whether an existing model already solves it, cost the full lifecycle, and if you do build, start small. Either way, the model still needs the same evaluation, monitoring, and evidence file as a vendor model. Building does not exempt a carrier from the bulletin.

Next step

An AI vendor risk assessment comes down to what a carrier can show an examiner: the answers the vendor gave, the testing you ran on your own book, and the contractual rights that let you keep checking.

If you want to see where your third-party oversight stands against the bulletin, the NAIC readiness scorecard takes a few minutes. To see how Swept keeps a live inventory of vendor and internal AI, logs every input and output, and produces the board-ready record examiners ask for, talk to us about governance.

AI Vendor Risk Assessment: What to Ask, What to Test, and When to Build Instead