Data engineering

Keep a reproducible view of changing data

Editorial edition · 16 November 2026

The idea

An analysis needs to identify which state of a dataset it used. A snapshot provides a stable reference while later updates continue elsewhere. Without a versioned input, rerunning the same code can produce a different result for reasons unrelated to the code.

Design choice

Record snapshot identity with the analysis output and its transformation version. Decide how long the referenced state remains available. A snapshot should have a clear consistency boundary when it spans several related tables.

Worked example

A monthly report records its input snapshot and calculation version. A later correction creates a new report revision rather than making the original result impossible to reproduce.

Before you ship

  • Record input and transformation versions
  • Define cross-table consistency
  • Align snapshot lifetime with review needs

Examples are illustrative. Adapt the checks to your own environment.

Further reading

Apache Parquet documentation

Search the library