The idea
A long-running import needs a record of what has been safely processed. A checkpoint connects progress to durable output. Saving progress before output is committed risks losing records; saving it afterwards can cause repeated processing after a crash.
Design choice
Choose a checkpoint boundary together with the duplicate-handling policy. A timestamp alone may be insufficient when several records share the same time. Keep a stable tie-breaker such as a sequence or record identifier.
Worked example
An importer stores the last committed pair of update_time and record_id. After restarting, it resumes from that boundary and safely handles an overlapping batch.
Before you ship
- Commit progress with durable results
- Test a crash between writes
- Handle equal timestamps
Examples are illustrative. Adapt the checks to your own environment.