What is incremental data loading?

Incremental data loading pulls only the rows that are new or changed since the last run, instead of reloading an entire table every time. It keeps pipelines fast and cheap as tables grow, since a run only touches what actually changed rather than scanning everything from scratch.

How it works generically

Incremental loads usually key off a cursor: a timestamp column, an auto-incrementing ID, or a change-tracking marker that the pipeline stores after each run and compares against on the next one. Anything with a cursor value past the stored mark gets pulled; everything else is skipped. The pattern applies at every layer: extracting from a source, transforming into a clean model, or refreshing an aggregate table.

Why it matters

A full reload of a large table gets slower and more expensive every time the table grows, and it competes with other traffic on the source system. Incremental loading decouples pipeline runtime from total table size: a table with a billion historical rows and only a thousand new ones today processes in roughly the time it takes to handle a thousand rows.

How OptimaFlo does this

Bronze ingestion is incremental by default: each source stores a watermark, an Iceberg table property tracking the last-synced cursor value, and each run pulls only rows newer than that watermark, always appending, never overwriting, so full history stays intact. Deeper in the platform, a snapshot-diffing layer compares Iceberg table snapshots between medallion runs, so Clean and Ready rebuilds process only the data that changed since the last run instead of the whole table. Gold-incremental writes go through a local-merge path: the pipeline writes locally, then the executor persists it through the standard Iceberg write and read path. It is a real, wired incremental path, just not a server-side MERGE INTO on the warehouse engine itself.

Related terms

• Idempotent pipeline: Incremental loads need idempotent writes, otherwise a retry after a partial run can double-count the same new rows.
• Time travel queries: Both rely on Iceberg snapshots: time travel reads a past snapshot, incremental loading diffs between snapshots to find what changed.
• Data pipeline: Incremental loading is a strategy for how a pipeline moves data, not a pipeline type on its own.