Evaluation

Retrieval and evaluation before another layer of automation.

Build an evaluation set from real questions

An evaluation set turns a vague impression of quality into repeatable evidence. Include normal work, difficult edge cases and questions that should not receive a confident answer. A collection of only easy examples can hide the failures that matter most.

Evaluation
Choose the errors that matter to the task

Precision and recall describe different aspects of a selection. Precision asks how many selected items are relevant; recall asks how many relevant items were selected. Neither number explains the cost of a particular mistake without context.

Evaluation
Allow a system to say it lacks evidence

For some tasks, a useful refusal is better than a fabricated answer. Abstention gives the system an explicit outcome when available evidence cannot support a response. It should guide the user toward clarification or another channel rather than ending with an opaque failure.

Evaluation
Look beyond a single average score

A strong average can conceal a weak result for a particular language, document type or request category. Slice analysis groups outcomes by meaningful conditions. It helps teams discover where a system needs a narrower scope or better supporting data.

Evaluation
Compare a change before widening its reach

A new prompt, retriever or model can improve one task while making another worse. A change gate compares the proposed version against a known reference. It should consider quality, latency and operational cost rather than one attractive demonstration.

Evaluation

Search the library