The idea
An evaluation set turns a vague impression of quality into repeatable evidence. Include normal work, difficult edge cases and questions that should not receive a confident answer. A collection of only easy examples can hide the failures that matter most.
Design choice
Record why an answer is acceptable and which evidence is required. Keep some examples separate from tuning. Remove identifying details from collected questions unless their use is necessary and authorised.
Worked example
A support assistant is tested on known procedures, ambiguous requests and unsupported products. The scoring guide distinguishes a correct clarification from an incorrect confident answer.
Before you ship
- Include unsupported and ambiguous cases
- Separate tuning and evaluation examples
- Version the scoring guide
Examples are illustrative. Adapt the checks to your own environment.