What is an idempotent pipeline?

An idempotent pipeline gives the same result no matter how many times it runs on the same input. Rerun it after a crash, a timeout, or a manual restart, and the output does not duplicate, corrupt, or drift from what one clean run would have produced. That property is what makes automated retries safe.

How it works generically

Idempotency in a pipeline usually comes from one of a few mechanisms: a stable identifier per row that a merge or upsert step checks before writing, an overwrite-not-append write pattern, or a watermark cursor that tracks exactly how far a run got so a retry resumes from the same point instead of duplicating already-written rows. None of these are automatic. Each has to be built into the write path on purpose.

Why it matters

Pipelines fail. A network blip, a timeout, an out-of-memory worker, a scheduler double-trigger: any of them can cause a step to run twice. Without idempotency, a rerun after a partial failure leaves duplicate rows in a table and forces someone to manually clean up before retrying. With it, retrying is the fix, not a new problem to debug.

How OptimaFlo handles this today

Bronze ingestion is watermark-based: each source tracks a stored cursor, an Iceberg table property, and only pulls rows newer than that cursor on each run, always appending, never overwriting. For REST sources, a client-side dedup filter checks incoming rows against what is already ingested by cursor field before writing. That filter is documented internally as a backstop, not the primary guarantee, and OptimaFlo's own code comments flag one ingestion mode as an incomplete case where duplicates can still occur. Warehouse-level write methods are explicit that they do not enforce source-key uniqueness themselves: the caller is responsible for pre-deduping the source. Idempotency in OptimaFlo today is watermark-driven and best-effort at the row level, not a blanket guarantee across every engine and mode.

Related terms

• Exactly-once delivery: The stronger guarantee idempotency is often confused with: no duplicates and no drops, not just safe-to-rerun.
• Incremental data loading: Idempotency is what makes incremental loads safe to retry without reprocessing everything from scratch.
• Data pipeline: The thing idempotency is a property of, not a pipeline type on its own.