A support assistant that knew when to stay quiet

Placeholder content. This case study is a structural template. Replace the client details, numbers, and narrative with a real engagement before publishing.

Context

The client's support team was answering the same forty questions repeatedly, drawn from a help center, an internal policy wiki, and a Slack channel where the real answers usually lived. They wanted an assistant that could draft responses for tier-1 tickets.

They had already tried once. The prototype answered everything confidently, including questions about products they did not sell, and support leadership had refused to roll it out. That reaction was correct.

Approach

We built the evaluation set before the system. Working with two senior support agents, we assembled 180 real tickets with agreed-upon correct answers, deliberately including 40 questions the assistant should not attempt - regulatory advice, account-specific balances, anything requiring authentication. Any version that answered those was a failure regardless of how well it did elsewhere.

We fixed the corpus before touching retrieval. Roughly a third of the help center was stale, and the wiki contained three contradictory versions of the refund policy. No retrieval strategy survives contradictory sources. Content cleanup was unglamorous and was the single largest quality improvement in the project.

We made abstention a first-class output. The assistant returns one of three things: a drafted answer with citations, an explicit "insufficient information in the knowledge base," or a routing suggestion to a specialist queue. The second option was treated as a success in evaluation, not a miss.

We kept a human in the loop. Nothing sends automatically. The assistant drafts, an agent reviews and edits, and the edit is captured as a training signal for the next evaluation round.

What made it work

Building the evaluation set first meant every subsequent decision - chunking strategy, reranking, prompt structure - was settled by measurement rather than by argument. It also meant we could tell the client honestly which parts of the ticket volume the system would not help with.

The abstention behavior is what got it approved. Support leadership's objection was never accuracy in the average case; it was the absence of a floor on the worst case. An assistant that reliably declines is one a team can supervise.

Outcome

The assistant now drafts responses for about a third of tier-1 volume, with agents editing roughly a quarter of those drafts before sending. Median first-response time dropped substantially, and the evaluation suite runs in CI so that corpus or model changes cannot silently regress quality.

The remaining two-thirds of tickets still go to humans, which was the honest scope from the start.