Data glossary

An AI data team is a group of AI agents that connects sources, builds pipelines, writes SQL, and builds dashboards, with a person approving what ships.

Apache Iceberg is an open table format that adds ACID transactions, schema evolution, and time travel to data lake files, without vendor lock-in.

Batch processing runs data through a pipeline in scheduled groups instead of continuously. See how OptimaFlo schedules its pipeline runs with Airflow.

Bronze, silver, and gold are the traditional names for medallion architecture's three data layers. OptimaFlo calls them Raw, Clean, and Ready.

BYOC (bring your own cloud) means a platform deploys into your own AWS or GCP account, so your data and compute live there, not on the vendor's infrastructure.

A CDC pipeline is built around database change events instead of full-table scans. See the pattern and how OptimaFlo compares with its snapshot-based approach.

Change data capture (CDC) tracks row-level database changes in real time by reading a transaction log. See how it works and how OptimaFlo ingests data today.

A data catalog is a searchable inventory of an organization's data assets, definitions, and ownership. See how it works and what OptimaFlo offers today.

A data fabric is an architecture that connects data across disparate sources and clouds through metadata-driven automation, without necessarily moving it into one place.

Data governance is the policies, roles, and controls that decide who can access, change, and audit an organization data. See how OptimaFlo enforces it today.

Data ingestion is the process of pulling data out of a source system and landing it somewhere it can be processed, as a one-time load, batch, or stream.

A data lakehouse combines a data lake's low-cost, flexible storage with a data warehouse's ACID transactions and schema guarantees, on one set of tables.

Data lineage traces where data came from and what depends on it. See how OptimaFlo tracks pipeline and metric lineage, and where it stops short of column level.

A data mesh is a decentralized architecture where domain teams own their own data as a product, instead of one central team owning every pipeline.

Data orchestration schedules a pipeline's steps and tracks dependencies and retries. See how Airflow DAGs run OptimaFlo's Raw, Clean, and Ready pipelines.

A data pipeline is the automated system that moves data from a source to a destination, usually through extraction, transformation, and loading steps.

Data profiling examines a table to surface null rates, distinct values, min and max, and type. See how it works and how OptimaFlo profiles tables today.

Data quality monitoring checks that data meets defined rules on an ongoing basis, not just once. See how it works and where OptimaFlo stands by default today.

Data transformation reshapes raw data into a usable form: cleaning values, standardizing formats, joining tables, and aggregating rows into metrics.

Data Vault is a modeling pattern of hubs, links, and satellites built for full history and changing sources. See how it works and its fit with OptimaFlo today.

ETL transforms data before loading it into a destination. ELT loads raw data first, then transforms it in place, the pattern most lakehouses use today.

An ETL pipeline extracts data from a source, transforms it before loading, then writes it to a destination, the classic pattern behind most data warehouses.

Exactly-once delivery means every event or row is processed one time: no duplicates, no drops. See what it takes to guarantee and where OptimaFlo stands today.

An idempotent pipeline gives the same result no matter how many times it reruns. Definition, how it works, and how OptimaFlo handles retries today.

Incremental data loading pulls only new or changed rows since the last run instead of reloading a full table. See how it works and how OptimaFlo implements it.

Medallion architecture organizes a data platform into layers: Raw, Clean, and Ready, each one closer to a trusted, business-ready table.

Reverse ETL pushes modeled data from a warehouse back into operational tools like a CRM. See how it works and why OptimaFlo runs the other direction.

Schema evolution lets a table adapt when a source adds, renames, or changes a column type, without rebuilding the table or breaking a running pipeline.

Slowly changing dimensions (SCD) track how a dimension changes over time instead of overwriting it. See the types and how OptimaFlo handles this today.

A star schema is a fact table joined to several dimension tables, built for fast BI queries. See how it works and how OptimaFlo builds it in the Ready layer.

Stream processing handles data continuously as it arrives instead of in batches. See how it works and where OptimaFlo's platform does and doesn't stream today.

A time travel query reads a table as it existed at a past snapshot. OptimaFlo has a live time travel page backed by Apache Iceberg snapshots.