Deploying AI Customer Support in Insurance

Executive Summary

Vertical Insure, a leading provider of embedded insurance solutions for vertical software platforms, sought to modernize their customer support operation by deploying an enterprise AI support agent. Their internal estimate for a safe, validated deployment was twelve months. Swept AI completed it in six weeks.

Before launch, Swept AI audited the AI agent and uncovered critical failure modes that would have reached customers undetected: the system fabricated financial figures, merged unrelated policy types into single answers, and constructed plausible but incorrect contact information. These errors appeared in confident, well-formatted responses that would have been difficult for customers or support staff to identify as wrong.

Through a structured six-week engagement, Swept AI restructured the knowledge base, built a domain-specific test suite of several hundred real customer scenarios, implemented targeted guardrails, and supervised the deployment through production launch.

The results:

• 60-70% automation rate on real customer inquiries, validated against Swept's test suite
• Zero customer-facing hallucinations since launch
• Six-week deployment versus a projected twelve-month internal timeline
• Ongoing monitoring that catches regressions before they impact users

Vertical Insure now operates a production AI support system with confidence, visibility, & continuous oversight.

Company Background

Vertical Insure makes insurance easy by embedding protection directly into software platforms used for sports registration, event management, travel bookings, and more. They handle approximately 1,500 customer inquiries per month. In insurance, incorrect answers create liability and erode trust. There is little margin for error.

When Vertical Insure decided to implement an AI customer support agent, they recognized the deployment required more scrutiny than the vendor's standard onboarding process could provide.

The Problem: AI Systems Fail in Ways That Are Difficult to Detect

Often, Enterprise AI support agents are sold as production-ready out-of-the-box.

And although Vertical Insure's AI agent appeared polished during initial demos, Swept AI's evaluation revealed a different picture.

Finding 1: Cross-Contamination Between Policy Types

The AI merged information from unrelated insurance products into single responses. Answers were confident and well-formatted, but factually incorrect. A customer asking about youth sports coverage might receive guidance that incorporated irrelevant details from event insurance policies.

Finding 2: Fabricated Financial Information

The AI generated dollar amounts that did not exist in any source material. Outdated fees, inferred benefit totals, and invented costs appeared in responses. These figures looked legitimate but were hallucinated.

Finding 3: Poor Website Data Extraction

The vendor marketed the ability to "learn from your website instantly." Testing showed the AI correctly extracted only approximately 2.5% of relevant website content. It could not distinguish authoritative pages from incidental content, and combined website fragments with help articles unpredictably.

Finding 4: Fabricated Contact Information

Left unchecked, the AI constructed plausible but incorrect email addresses. Formats like support@companyname or help@companyname sounded professional but did not route to real inboxes.

Finding 5: Historical Ticket Ingestion Would Degrade Performance

Vertical Insure initially planned to import thousands of historical support conversations to train the AI. Swept AI's analysis showed this approach would overweight edge cases, teach outdated practices, and reduce accuracy on common questions.

The AI looked polished on the surface. Swept AI showed us exactly what was happening using our data.

Swept AI's Approach

Swept AI's engagement followed a structured methodology designed to identify risks before deployment and establish ongoing monitoring afterward.

1. Knowledge Base Restructuring

Vertical Insure's help content was spread across multiple systems with inconsistent formatting. Swept AI consolidated and reorganized this material into a structure the AI could reliably interpret. This eliminated ambiguity that caused the AI to blend information from unrelated sources.

2. Test Suite Development

Swept AI built a comprehensive test suite of several hundred scenarios drawn from real customer inquiries. These tests covered common questions, edge cases, and potential failure modes identified during initial evaluation.

3. Hazard Detection

Swept AI ran the AI agent through the full test suite and documented every failure mode. This included correct answers, incorrect answers, partially correct answers, and cases where the AI expressed inappropriate confidence.

4. Iterative Optimization

Swept AI worked through three to five optimization cycles per week. Each cycle involved adjusting the AI's configuration, grounding data, or guardrails, then re-running the test suite to measure impact. Changes that improved one area were validated against the full suite to ensure they did not create regressions elsewhere.

5. Guardrail Implementation

Based on the hazard detection findings, Swept AI implemented specific protections: financial figures were restricted to approved values only, contact information was limited to verified addresses, website ingestion was disabled to eliminate unpredictable source contamination, and escalation pathways were configured for topics outside the AI's validated scope.

6. Dashboard and Visibility

Swept AI built a custom audit dashboard that showed Vertical Insure exactly how the AI performed across the test suite. This provided visibility that the underlying platform did not offer.

7. Supervised Launch

Vertical Insure went live only after all major risks were addressed and performance reached consistently high accuracy across the test suite. Swept AI validated the final configuration and confirmed that escalation pathways functioned correctly.

8. Ongoing Monitoring

Swept AI continues to monitor Vertical Insure's AI agent in production. This catches performance drift and regressions that the vendor's system does not surface.

That dashboard changed everything. We could finally see how the AI agent behaved, not just hope it would actually work.

Results

Deployment Timeline

Vertical Insure reached production deployment in under six weeks. Without Swept AI's structured approach, the Vertical Insure team estimated this process would have taken them at least a year internally and involved discovering problems through customer complaints.

If we had launched without Swept AI, we would have found these issues the hard way, with real customers and not in time.

Lessons for AI Deployment

Vertical Insure's experience illustrates several principles that apply broadly to AI deployments in customer-facing contexts.

Vendor demos do not represent production behavior

Demo environments are optimized for common cases. Production environments surface edge cases, data conflicts, and failure modes that demos do not reveal. Systematic testing is required to understand how an AI system will actually perform.

AI systems require domain-specific validation

Generic accuracy metrics do not capture whether an AI system handles the specific questions that matter for a particular business. Test suites must be built from real customer behavior, not synthetic benchmarks.

Guardrails must be designed around observed failures

Effective guardrails address specific failure modes discovered through testing. General-purpose safety features do not catch domain-specific errors like fabricated policy details or incorrect contact information.

Ongoing monitoring is necessary

AI systems drift over time as underlying models change, knowledge bases are updated, and edge cases accumulate. Continuous monitoring catches regressions before they impact customers.

The cost of undetected errors is high

In regulated industries and high-stakes contexts, incorrect AI responses create liability, erode trust, and generate support escalations that exceed the cost of proper validation.

Conclusion

Vertical Insure successfully deployed an AI customer support agent by treating the implementation as a validation problem, not just an integration task. Swept AI's audit identified failure modes that would have reached customers, and the subsequent optimization process eliminated those risks.

The engagement produced a system that automates a majority of support inquiries while maintaining the accuracy standards required in insurance. Ongoing monitoring ensures that performance remains stable as the underlying AI continues to evolve.