Storage

Reliable paths from a source record to a useful dataset.

Partition for the questions people ask

Partitioning divides a dataset into manageable groups that queries can select. A useful key reduces unnecessary reading. An excessively detailed key can create many tiny fragments and increase metadata and maintenance work instead of improving performance.

Storage
Use columnar storage for selective analysis

Analytical queries often read a few fields across many rows. A columnar layout stores values by column, which can reduce unnecessary reading and improve compression for suitable data. It is a different access pattern from updating one complete record at a time.

Storage
Make deletion part of the dataset lifecycle

A dataset does not become more useful merely because it keeps every historical record. Retention connects storage duration to a purpose. It must account for derived tables, exports, caches and backups as well as the original source.

Storage
Keep a reproducible view of changing data

An analysis needs to identify which state of a dataset it used. A snapshot provides a stable reference while later updates continue elsewhere. Without a versioned input, rerunning the same code can produce a different result for reasons unrelated to the code.

Storage
Connect an output to the work that produced it

Lineage explains how a dataset or result depends on sources and transformations. It helps answer which reports may be affected by a changed field or failed import. Useful lineage is specific enough to support an action rather than merely drawing a large diagram.

Storage

Search the library