Evaluating RAG the way you would evaluate search.

Search wearing a language model as a hat

Retrieval-augmented generation is a search problem wearing a language model as a hat. Teams that treat it as pure prompting keep re-learning what search engineers established decades ago.

Everything painful about RAG — chunking, ranking, freshness, tenant isolation, evaluation — is a search problem first and a language problem second. The good news: the search field has forty years of tools you can borrow.

Split the eval into three layers

One number cannot tell you whether your RAG stack is working. You need at least three, and they answer different questions.

• Retrieval quality — did the right chunks come back at all?
• Answer faithfulness — does the response cite what was retrieved, without inventing?
• Task success — did the user actually get what they came for?

Retrieval: the boring, essential layer

Build a small gold set — a hundred hand-labeled questions with the passage IDs that should surface. Track recall@k and MRR every release. It is boring, it is uncomfortable, it is the only way to know whether your last chunking change helped or hurt.

Do not skip the labels because they are slow. A hundred well-chosen examples will catch more real regressions than ten thousand synthetic ones generated by another model.

A hundred well-chosen examples will catch more real regressions than ten thousand synthetic ones.

Faithfulness: judge the judge

Use LLM-as-judge with a rubric you actually reviewed, plus periodic human spot-checks. Judges drift. Rubrics rot. Re-calibrate on a schedule — quarterly at minimum, monthly if the model provider ships an update.

Store every judgment with the exact prompt, model version, and rubric hash. When a regression appears, you want to know whether the system got worse or the judge got stricter.

Task success: let the product tell you

For task success, instrument the product. Thumbs, edits, escalations, silent abandonment — every signal your users give you is an eval signal if you capture it.

The strongest signal is usually the quietest one: the user who copied the answer and never came back to correct it. Log the read receipts, not just the reactions.