The idea
Duplicate detection is a business decision as well as a technical operation. Two identical-looking records may represent two legitimate actions. Conversely, the same event can arrive with different delivery metadata. Start by choosing the identity that matters to the downstream use.
Design choice
Prefer a stable source identifier when available. If a composite key is necessary, document its limitations and retention window. Avoid treating a hash of every field as a universal definition of identity.
Worked example
Two identical purchases made seconds apart are separate events. A retransmitted copy of one purchase retains the same event identifier and can be recognised as a duplicate.
Before you ship
- Define identity with the domain owner
- Preserve legitimate repetitions
- Measure duplicates before removal
Examples are illustrative. Adapt the checks to your own environment.