Applied AI

Retrieval and evaluation before another layer of automation.

Retrieval

Split documents at meaningful boundaries

A retrieval system does not usually pass an entire document to a language model. It selects smaller pieces. The boundary of each piece affects whether a retrieved passage still contains the definitions, qualifications and context needed to answer a question.

Retrieval
Treat an embedding model as part of the index

An embedding is a representation produced by a particular model and preprocessing pipeline. Similar vector dimensions do not guarantee compatible representations. Changing the model can change which passages appear close to a query even when the underlying documents stay the same.

Retrieval
Combine exact terms with semantic similarity

Semantic search can recognise related wording, while lexical search preserves exact names, identifiers and uncommon terms. Many collections contain both kinds of query. Combining the two can be useful, but the combination needs its own evaluation.

Retrieval
Make an answer traceable to its evidence

A citation is useful only when it points to the passage that supports the associated statement. Linking to a document home page is weaker than preserving the specific section and version used. Retrieval and answer generation should share that evidence identity.

Retrieval
Know when a search index is behind

A correct answer to yesterday's document can be wrong for today's process. Index freshness describes the delay between a source change and its availability to search. Deletions and access changes are as important as newly added content.

Retrieval

Evaluation

Build an evaluation set from real questions

An evaluation set turns a vague impression of quality into repeatable evidence. Include normal work, difficult edge cases and questions that should not receive a confident answer. A collection of only easy examples can hide the failures that matter most.

Evaluation
Choose the errors that matter to the task

Precision and recall describe different aspects of a selection. Precision asks how many selected items are relevant; recall asks how many relevant items were selected. Neither number explains the cost of a particular mistake without context.

Evaluation
Allow a system to say it lacks evidence

For some tasks, a useful refusal is better than a fabricated answer. Abstention gives the system an explicit outcome when available evidence cannot support a response. It should guide the user toward clarification or another channel rather than ending with an opaque failure.

Evaluation
Look beyond a single average score

A strong average can conceal a weak result for a particular language, document type or request category. Slice analysis groups outcomes by meaningful conditions. It helps teams discover where a system needs a narrower scope or better supporting data.

Evaluation
Compare a change before widening its reach

A new prompt, retriever or model can improve one task while making another worse. A change gate compares the proposed version against a known reference. It should consider quality, latency and operational cost rather than one attractive demonstration.

Evaluation

Search the library