Test the tool on your own use cases and data, and check its outputs for accuracy, hallucinations and bias. Confirm where your data goes and what the contract allows, make sure you can supervise the tool after go-live, and model the real cost. Then score every vendor on the same scorecard before the go/no-go call.
- 01 · TestTest on your own data, in an isolated environment.
- 02 · ScoreScore each vendor on the same seven areas.
- 03 · DecideMake the go/no-go call from the weighted score.
- 04 · SuperviseKeep checking it continuously after go-live.
A vendor demo shows what a tool does on the vendor's examples. Your evaluation has to show what it does on yours, in a way you can explain to a COO, a board and a regulator.
This guide works as an AI vendor due diligence checklist if you're doing it by hand, with a scorecard for up to three vendors. Swept runs the same evaluation at scale on your own data.
What should an insurance carrier evaluate before buying an AI tool?
Evaluate seven areas: fit to the use case, output quality, bias and fairness, data handling and security, supervision and auditability, contract terms, and cost. Weight them by risk, so output quality and data handling count the most, since a failure in either one is hard to undo.
How do I test an AI tool on our own data without exposing policyholder data?
Test in a private or isolated environment, on real cases from your own work, with personal data redacted wherever the task doesn't need it. Before any data moves, get these from each vendor:
- A private tenant, a dedicated test environment or a deployment inside your own cloud.
- A written commitment that test data won't be used for training and will be deleted after the test.
- A list of every system and subprocessor your test data will touch.
How do I check an AI tool for hallucinations and bias?
Compare the tool's outputs on your own cases with answers your experts would stand behind, and check whether results differ across groups of policyholders or claimants. A hallucination is a confident statement the source documents don't support, so every claim in an output has to trace back to the file.
| The results should show | Why it matters |
|---|---|
| Accuracy on your own cases | A tool can shine on a demo and struggle with your forms and files. |
| No unsupported statements | One invented fact in a coverage letter can become a complaint or an exam finding. |
| Consistent answers | If the same file gets different answers, no single output can be trusted. |
| No unexplained gaps across groups | Differences by region, claim type or policy type need an explanation before go-live. |
How many test cases do we need for a result we can trust?
Enough to reach the confidence level you choose: about 271 test cases per use case for 90% confidence, 385 for 95% and 664 for 99%, each within plus or minus 5 percentage points. A manual test with a few dozen cases is a sensible base level, but it can't tell you whether a 72% pass rate is really 72%.
What should the contract say about our data and the model?
The contract should cover training on your data, retention, subprocessors, notice before model changes, audit rights and how you get your data back when you leave. Ask before the pilot, while you still have leverage.
| Clause | What to ask for |
|---|---|
| Training on your data | No training or fine-tuning on your inputs or outputs without written consent. |
| Retention | A set retention period and written confirmation of deletion. |
| Model version changes | Advance notice, and the right to test the new version first. |
| Audit rights | Audit rights or independent audit reports, and cooperation with regulators. |
| Exit | Your data and logs returned in a usable format, then deleted. |
The NAIC Model Bulletin expects this kind of due diligence on third-party AI systems, including audit rights where appropriate.
How do I know it will stay accurate after we buy it?
Supervise it continuously against the standard you used to buy it, and test every new model version before it reaches production. Before you sign, make sure you'll have:
- Continuous supervision of quality, measured against your evaluation results.
- Notice before any model change, and a test against your standard before it goes live.
- Logs and an audit trail you could hand an examiner in a market conduct exam.
For a claims intake agent, see how carriers monitor a claims intake AI agent.
How do I estimate the real cost?
Estimate cost per use case from expected volume, compare usage pricing with seat pricing, and add the internal time to run and supervise the tool. Ask every vendor to price the same usage scenario in writing so the numbers line up.
Who should be in the evaluation?
Include a business owner who owns the outcome, IT and security, compliance, legal or third-party risk, and a claims or underwriting expert who knows what a right answer looks like. One person can cover more than one role, but every role needs a name next to it.
What goes on an AI evaluation scorecard?
An AI evaluation scorecard lists the seven areas with a weight for each, a score from 1 to 5, the evidence behind each score and a go/no-go rule. Here is ours, with what a 1, a 3 and a 5 look like in each area.
| Area | 1 looks like | 3 looks like | 5 looks like |
|---|---|---|---|
| Fit to the use caseWeight 15 | Doesn't cover the core task, or needs major changes to how the team works. | Covers the core task, with workarounds the team can live with. | Covers the core task inside the existing workflow, with no workarounds. |
| Output qualityWeight 20 | Frequent wrong or unsupported answers on your cases. | Mostly right, with errors on hard cases that reviewers catch. | Accurate on your cases, hard ones included, with no unsupported statements. |
| Bias and fairnessWeight 10 | Unexplained gaps in outcomes across groups, or no way to test for them. | Small gaps the vendor can explain, with a plan to monitor them. | No material gaps, a documented testing method and ongoing checks. |
| Data handling and securityWeight 20 | Data leaves your control, or may be used to train the vendor's models. | Private processing, with open questions on retention or subprocessors. | Runs privately or in your environment, no training on your data, clear retention and subprocessors, current security audits. |
| Supervision and auditabilityWeight 15 | No logs or version history you can get to. | Logs are available, but monitoring and change control are manual. | Full logs, continuous supervision, testing before model changes and an audit trail you can export. |
| Contract and vendor termsWeight 10 | Terms allow training on your data, or give no notice before model changes. | Most terms work, with open items legal is still negotiating. | No training on your data, set retention, notice before model changes, audit rights and a clean exit. |
| Cost and usage controlsWeight 10 | You can't estimate the cost or cap it. | Cost is predictable, but the controls are coarse. | Clear cost per use case, with budgets, limits and usage reporting by team. |
Score each area from 1 to 5 and multiply by its weight. Add the seven weighted scores and divide by 100 to get a weighted average out of 5.
- 4.0 to 5.0GoA weighted average of 4.0 or more is a go.
- 3.0 to 3.9Conditional go3.0 to 3.9 is a conditional go, with a written fix plan and an owner.
- Below 3.0No-goBelow 3.0 is a no-go.
- Any 1 in a critical areaNo-goA score of 1 in Output quality or Data handling is a no-go, whatever the total.
Free download
Get the AI evaluation scorecard
A printable PDF with the seven weighted areas and the go/no-go rule, with room for three vendors side by side.
- Seven areas, weighted to 100
- What a 1, a 3 and a 5 look like in each area
- Evidence and notes columns for your file
- The go/no-go rule for the final call
Can Swept run the AI evaluation for us?
Yes. Swept is the trusted third party carriers use to evaluate, supervise and certify AI. Your team keeps the decision, and we do the testing, the comparison and the supervision after go-live.
| Step | By hand | With Swept |
|---|---|---|
| Test cases | A few dozen, built by your experts. | Sized to the confidence level you choose. |
| Hallucinations and bias | Spot checks, within the time you have. | Tested at scale, including vendor AI. |
| Comparing vendors | Different demos and different numbers. | Every vendor tested the same way. |
| After go-live | Re-checks when someone has time. | Continuous supervision, with model changes tested first. |
| Evidence for examiners | Assembled by hand. | Logs, an audit trail and certification. |
Start with one use case and a shortlist of two or three vendors. Tell us what you're evaluating and we'll show you what Swept's evaluation looks like on your data.
What else do carriers ask about evaluating AI tools?
How long should an AI vendor evaluation take?
Plan for four to eight weeks for one use case and two or three vendors. Contract terms often take the longest, so start them in the first week.
Can we evaluate the AI built into our core system vendor's software?
Yes, because it touches the same policyholder data as any standalone tool, so ask for the same evidence: test access on your data, where the data goes and how model changes are handled. See our vendor AI governance and evaluation guides.
What does the NAIC model bulletin expect for third-party AI systems?
The NAIC Model Bulletin, adopted in December 2023, expects due diligence on AI vendors and their data and models, audit rights or audit reports where appropriate, and vendor cooperation with regulators. It also expects testing and validation of AI systems whether you built them or bought them.
Should we run a pilot before we buy?
Yes, run a short pilot with real users once the tool passes testing on your data. Agree on exit criteria up front, and make sure the contract lets you walk away if it fails.
What score should a tool hit to get a yes?
A weighted average of 4.0 or more out of 5 is a go, 3.0 to 3.9 is a conditional go with a written fix plan and an owner, and below 3.0 is a no-go. A score of 1 in Output quality or Data handling is a no-go whatever the total.