Ingestion

Reliable paths from a source record to a useful dataset.

Agree on a source contract before ingestion

A data pipeline depends on promises made by its source: which fields exist, how records are identified and when updates become available. A source contract makes those assumptions visible. It is especially useful when the producer and the analysis team work independently.

Ingestion
Resume an import from a known position

A long-running import needs a record of what has been safely processed. A checkpoint connects progress to durable output. Saving progress before output is committed risks losing records; saving it afterwards can cause repeated processing after a crash.

Ingestion
Define what makes two records the same

Duplicate detection is a business decision as well as a technical operation. Two identical-looking records may represent two legitimate actions. Conversely, the same event can arrive with different delivery metadata. Start by choosing the identity that matters to the downstream use.

Ingestion
Decide how late data changes a result

Streaming results are often published before every event has arrived. A late event can therefore change an already visible total. The pipeline needs a policy for corrections, not just a timer that closes a processing window.

Ingestion
Isolate invalid records without hiding them

A single malformed record should not necessarily stop an entire import. Quarantine separates records that fail validation while preserving the reason for failure. The accepted and rejected counts should still reconcile with the received input.

Ingestion

Search the library