Data engineering

Reliable paths from a source record to a useful dataset.

Ingestion

Agree on a source contract before ingestion

A data pipeline depends on promises made by its source: which fields exist, how records are identified and when updates become available. A source contract makes those assumptions visible. It is especially useful when the producer and the analysis team work independently.

Ingestion
Resume an import from a known position

A long-running import needs a record of what has been safely processed. A checkpoint connects progress to durable output. Saving progress before output is committed risks losing records; saving it afterwards can cause repeated processing after a crash.

Ingestion
Define what makes two records the same

Duplicate detection is a business decision as well as a technical operation. Two identical-looking records may represent two legitimate actions. Conversely, the same event can arrive with different delivery metadata. Start by choosing the identity that matters to the downstream use.

Ingestion
Decide how late data changes a result

Streaming results are often published before every event has arrived. A late event can therefore change an already visible total. The pipeline needs a policy for corrections, not just a timer that closes a processing window.

Ingestion
Isolate invalid records without hiding them

A single malformed record should not necessarily stop an entire import. Quarantine separates records that fail validation while preserving the reason for failure. The accepted and rejected counts should still reconcile with the received input.

Ingestion

Storage

Partition for the questions people ask

Partitioning divides a dataset into manageable groups that queries can select. A useful key reduces unnecessary reading. An excessively detailed key can create many tiny fragments and increase metadata and maintenance work instead of improving performance.

Storage
Use columnar storage for selective analysis

Analytical queries often read a few fields across many rows. A columnar layout stores values by column, which can reduce unnecessary reading and improve compression for suitable data. It is a different access pattern from updating one complete record at a time.

Storage
Make deletion part of the dataset lifecycle

A dataset does not become more useful merely because it keeps every historical record. Retention connects storage duration to a purpose. It must account for derived tables, exports, caches and backups as well as the original source.

Storage
Keep a reproducible view of changing data

An analysis needs to identify which state of a dataset it used. A snapshot provides a stable reference while later updates continue elsewhere. Without a versioned input, rerunning the same code can produce a different result for reasons unrelated to the code.

Storage
Connect an output to the work that produced it

Lineage explains how a dataset or result depends on sources and transformations. It helps answer which reports may be affected by a changed field or failed import. Useful lineage is specific enough to support an action rather than merely drawing a large diagram.

Storage

Search the library