Data engineering

Make deletion part of the dataset lifecycle

Editorial edition · 16 November 2026

The idea

A dataset does not become more useful merely because it keeps every historical record. Retention connects storage duration to a purpose. It must account for derived tables, exports, caches and backups as well as the original source.

Design choice

Separate business retention requirements from convenient defaults. Document which copies expire automatically and which require review. A deletion policy is incomplete when data can immediately reappear from an unmanaged export or restore.

Worked example

A temporary analysis workspace expires after its agreed review period. Its owner checks whether any result needs to move into a separately governed long-term collection.

Before you ship

  • Map primary and derived copies
  • Document expiry and restore behaviour
  • Review exceptions with an owner

Examples are illustrative. Adapt the checks to your own environment.

Further reading

Apache Parquet documentation

Search the library