Applied AI

Build an evaluation set from real questions

Editorial edition · 16 November 2026

The idea

An evaluation set turns a vague impression of quality into repeatable evidence. Include normal work, difficult edge cases and questions that should not receive a confident answer. A collection of only easy examples can hide the failures that matter most.

Design choice

Record why an answer is acceptable and which evidence is required. Keep some examples separate from tuning. Remove identifying details from collected questions unless their use is necessary and authorised.

Worked example

A support assistant is tested on known procedures, ambiguous requests and unsupported products. The scoring guide distinguishes a correct clarification from an incorrect confident answer.

Before you ship

  • Include unsupported and ambiguous cases
  • Separate tuning and evaluation examples
  • Version the scoring guide

Examples are illustrative. Adapt the checks to your own environment.

Further reading

Model evaluation, scikit-learn

Search the library