Data engineering

Isolate invalid records without hiding them

Editorial edition · 16 November 2026

The idea

A single malformed record should not necessarily stop an entire import. Quarantine separates records that fail validation while preserving the reason for failure. The accepted and rejected counts should still reconcile with the received input.

Design choice

Choose which errors require stopping the batch and which permit partial progress. Protect quarantined data with the same access controls as the source. Assign a review owner so that a quarantine folder does not become an invisible backlog.

Worked example

An import receives 1,000 records, accepts 997 and quarantines three with invalid dates. Its report shows all three counts and a link to the restricted review queue.

Before you ship

  • Reconcile received and processed counts
  • Keep rejection reasons actionable
  • Apply a retention policy to quarantine

Examples are illustrative. Adapt the checks to your own environment.

Further reading

Apache Kafka documentation

Search the library