Data engineering

Resume an import from a known position

Editorial edition · 16 November 2026

The idea

A long-running import needs a record of what has been safely processed. A checkpoint connects progress to durable output. Saving progress before output is committed risks losing records; saving it afterwards can cause repeated processing after a crash.

Design choice

Choose a checkpoint boundary together with the duplicate-handling policy. A timestamp alone may be insufficient when several records share the same time. Keep a stable tie-breaker such as a sequence or record identifier.

Worked example

An importer stores the last committed pair of update_time and record_id. After restarting, it resumes from that boundary and safely handles an overlapping batch.

Before you ship

  • Commit progress with durable results
  • Test a crash between writes
  • Handle equal timestamps

Examples are illustrative. Adapt the checks to your own environment.

Further reading

Apache Kafka documentation

Search the library