Measuring AI answer quality

Why "it seems good" is not a measurement

Most assistants are judged by someone asking it ten questions in a meeting. The ten are usually easy, worded like the documents, and asked by someone who knows the answers. Real users ask in their own words, about edge cases, and about things you never documented.

There is also a subtler problem: answers are not perfectly repeatable. Ask the same question twice and you may get two slightly different answers. A change that seems to fix one question may break two others. Without a fixed set of questions run the same way each time, you cannot tell improvement from noise.

Measure in layers

An answer can fail at several points, and each needs its own test. Mixing them into one "accuracy" score hides what to fix.

• Retrieval - did search find the passage that holds the answer? If not, nothing downstream can fix it.
• Grounding - is everything the answer says supported by the passages it cites?
• Correctness - does the answer contain the fact that matters, stated correctly?
• Refusal - does it decline when the documents do not contain an answer?
• Outcome - did the person get what they needed, and did a ticket follow anyway?

Build a golden question set

A golden set is a small spreadsheet that becomes your most useful quality tool. Each row has a real question, the document (and ideally the section) that should answer it, and one or two words or figures the answer must contain.

Take the questions from tickets, emails and chat logs, in the askers' own words. Cover your top categories in proportion to their volume. Add the hard ones: questions whose answer is in a table, questions that mention an error code, and questions where two documents look relevant but only one is right.

• 30-50 real questions to start, grown over time.
• For each: expected document, expected section, a must-contain fact.
• 5-10 questions your documents deliberately do not answer.
• Every question that ever went wrong in production gets added.
• Re-run after any change to documents, models, chunking or settings.

Retrieval: hit rate and MRR

For each golden question, look at the ranked list of passages search returned. Two numbers summarise it. Hit rate at K is the share of questions where the expected document appears in the top K results - K being roughly the number of passages your system hands the model. Mean reciprocal rank (MRR) rewards finding it near the top: a first-place hit scores 1, second 0.5, third 0.33, and the scores are averaged.

A low hit rate means the answer cannot be right however good the model is; fix search, chunking or the documents' wording. A high hit rate with a low MRR means the right passage is found but ranked below others, which is when assistants start answering from a nearby but wrong passage.

Grounding and citations

A grounded answer says nothing its sources do not. Check it mechanically where you can and by eye where you cannot.

Mechanically: does every citation point to a passage that exists? Does every number in the answer appear in a passage it cites? Did an answer that claims to have found something cite anything at all? By eye, on a sample: does each cited passage actually support the sentence it is attached to?

Citation precision - the share of citations that point at the expected document - is a useful signal of over-citation, where an answer sprinkles references to loosely related passages to look thorough.

Refusals: the off-topic set

Keep a handful of questions no document of yours answers - "What is the capital of France?", a competitor's pricing, a policy you do not have. Run them against every assistant, every time. The target is that all of them are refused.

Then look at the other side: questions that should be answered but were refused. A refusal rate that is too high usually means missing documents or vocabulary that does not match - both fixable - not a gate that is too strict. Loosening the gate to answer more is how invented answers come back.

In production: answered, unanswered, rated

Once real people are asking, track the share of questions that were answered, and read the unanswered ones themselves, grouped so that the same question asked forty times shows as one line with a count. That list is your documentation backlog, already prioritised.

Ratings are useful, used carefully. Few people rate, and they skew towards the annoyed. Treat every thumbs-down as a case to open and diagnose - wrong passage, old document, missing content - rather than as a percentage to report.

Deflection, honestly

Deflection is the number everyone wants and the easiest to inflate. Counting chat conversations that did not end in a ticket counts people who gave up, people who were never going to raise a ticket, and people who emailed instead.

A more honest method: categorise tickets before you start, then compare monthly ticket volume in the categories the assistant covers against the categories it does not, over the same period. If how-to tickets fall while billing and bug tickets hold steady, the difference is a fair estimate of what the assistant absorbed.

• Baseline at least two months of tickets by category before launch.
• Compare covered against uncovered categories, not just before against after.
• Watch for growth in other channels - phone, email - that would hide displacement.
• Read a sample of conversations that did not end in a ticket.

Drafts: acceptance, edits and reopens

When an assistant drafts replies for agents rather than answering customers directly, the agents are the quality check, and their behaviour is the measure. Track, per category, how often a draft is sent unedited, sent after editing, or discarded - and how often the ticket is reopened afterwards.

A category with high unedited sends and few reopens is one where the drafts are trusted. One where drafts are usually discarded points to missing documents for that topic. A rising edit rate after a policy change usually means the old document is still being retrieved.

How SupaCortex helps you measure

Every answer in SupaCortex is recorded with the passages it used, their ranks and trust tiers, the citations it ended up with and whether it passed validation. Each project's Insights show the share of questions answered, the most-asked unanswered questions grouped together, and recent thumbs-down answers with their citations. For help-desk drafts, the desk reports whether each one was sent unedited or edited, and draft metrics show acceptance, edit and reopen rates by category.

You can run your own golden set through the Search and Answer API. We hold ourselves to the same discipline: retrieval and answer changes are tested against fixed question sets with known answers, including off-topic questions every project must refuse, before they ship.