What is schema evolution?

Schema evolution is the ability to add, rename, drop, or change the type of a table's columns without rebuilding the table from scratch. In a table format like Apache Iceberg, that change is tracked as new metadata, so old data stays readable under the old schema and new data uses the new one.

Why it matters

Source systems change their shape constantly: a new column shows up, a field gets renamed, a type gets widened. Without schema evolution, a pipeline that assumes a fixed shape breaks the moment the source changes, and someone has to notice, fix the pipeline, and often rebuild the table from history. With it, the table adapts and the pipeline keeps running.

How it works

A table format tracks column identity separately from column name or position, so a rename doesn't look like a drop-and-add, and old files written under an earlier schema stay valid. The engine reconciles what a query asks for against whatever schema version each file was written with.

How OptimaFlo implements this

OptimaFlo runs schema evolution automatically as part of Raw-layer ingestion: when a source's columns change, the pipeline commits the new schema through Iceberg's catalog and keeps writing, instead of failing the run. This happens as a built-in part of ingestion, not as a control you turn on through a dedicated settings panel.

Related terms

• Apache Iceberg: The table format whose metadata layer makes schema evolution possible.
• Medallion architecture: The Raw layer where OptimaFlo's automatic schema evolution runs.