What is change data capture (CDC)?

Change data capture (CDC) is a technique for tracking row-level changes, inserts, updates, and deletes, in a source database as they happen, usually by reading the database's transaction log. Instead of re-reading an entire table, downstream systems consume just the changes, which keeps sync fast and light even on large, frequently updated tables.

How it works

CDC tools tail a database's write-ahead log (Postgres) or binary log (MySQL), rather than issuing repeated SELECT queries. Each change becomes an event: a row inserted, updated, or deleted. A separate process applies those events to a target system in near real time, without touching the source table's read path.

Why it matters

Full-table reads get expensive as a table grows: a nightly re-scan of a billion-row table competes with production traffic and gets slower with every new row. CDC avoids that by shipping only what changed since the last read, which also gives downstream systems a record of exactly when and how a row changed.

How OptimaFlo ingests source changes

OptimaFlo's source connectors, including Postgres and MySQL, read data through table reflection and SELECT queries: either the full table or a query you provide. That's full-table and custom-query ingestion, not log-based CDC. OptimaFlo does not tail a write-ahead log or binary log today, so there's no continuous change stream from source databases.

Inside the platform, after data lands in Raw, a snapshot-diffing layer tracks which Iceberg snapshots changed between medallion runs, so Clean and Ready rebuilds only process new or changed data instead of the whole table. That's incremental processing on data already in the lakehouse, not CDC on the source database itself.

The tradeoff is a real one: full-table and custom-query ingestion needs no replication slot, no binlog access, and no elevated database permissions, so a new source connects in minutes on any Postgres or MySQL instance, managed or self-hosted. What you give up is continuous, sub-second change delivery from the source.

Related terms

• Data ingestion: The step that reads data from a source. CDC is one way to do that; table reads are another.
• Batch processing: The scheduled, full-table or custom-query model OptimaFlo's connectors use today.