Data engineering

Define what makes two records the same

Editorial edition · 16 November 2026

The idea

Duplicate detection is a business decision as well as a technical operation. Two identical-looking records may represent two legitimate actions. Conversely, the same event can arrive with different delivery metadata. Start by choosing the identity that matters to the downstream use.

Design choice

Prefer a stable source identifier when available. If a composite key is necessary, document its limitations and retention window. Avoid treating a hash of every field as a universal definition of identity.

Worked example

Two identical purchases made seconds apart are separate events. A retransmitted copy of one purchase retains the same event identifier and can be recognised as a duplicate.

Before you ship

  • Define identity with the domain owner
  • Preserve legitimate repetitions
  • Measure duplicates before removal

Examples are illustrative. Adapt the checks to your own environment.

Further reading

Apache Kafka documentation

Search the library