What is Apache Iceberg?

Apache Iceberg is an open table format for data lake storage. It adds database-style guarantees, ACID transactions, schema evolution without rewriting a table, and time travel over historical snapshots, to plain files sitting in object storage like S3 or GCS.

What a table format does

A table format sits between raw files and a query engine. It tracks which files belong to a table, their schema, and their partitioning, in metadata the engine reads before touching data. That metadata is what makes ACID transactions, schema evolution, and time travel possible on top of ordinary files.

Why it matters

Without a table format, a data lake is just a folder of files: no transaction guarantees, no safe way to change a schema, no way to see what a table looked like yesterday. Apache Iceberg brings those warehouse-grade guarantees to open storage, so a table stays correct and queryable even while pipelines write to it concurrently.

How OptimaFlo implements this

OptimaFlo stores every layer, Raw, Clean, and Ready, as Iceberg tables in your own AWS or GCP account. Schema evolution runs automatically as part of ingestion, so a source adding a column doesn't break a running pipeline. Time travel is a live feature: you can pick a past snapshot on a table's detail page and run a query against it. For a longer walkthrough of the tradeoffs between managed and self-managed Iceberg, see Apache Iceberg in your own cloud, without the overhead.

Related terms

• Data lakehouse: The category Apache Iceberg tables build, a data lake with warehouse-style guarantees.
• Schema evolution: One of the guarantees Iceberg's metadata layer makes possible.
• Time travel queries: Querying an Iceberg table as it looked at a past snapshot.
• Medallion architecture: The Raw, Clean, and Ready layers OptimaFlo stores as Iceberg tables.