What is batch processing?

Batch processing runs a job on a group of data at once, on a schedule or on demand, rather than processing each record the instant it arrives. A pipeline might read yesterday's orders every night at 2 AM, transform them, and load the results, instead of reacting to each new order in real time.

How it works

A batch is defined by a time window or a trigger: hourly, nightly, or on demand. The job runs on that schedule, usually managed by an orchestrator, and processes the whole dataset or a delta together. Results land all at once at the end of the run, not continuously as data arrives.

Why it matters

Batches are easier to reason about and retry: if a run fails, you rerun the whole batch, not a partial stream of events. That simplicity is a good fit whenever near-real-time freshness isn't the requirement, and it costs less to operate than a system that has to stay running around the clock.

How OptimaFlo runs batches

OptimaFlo's Data Engineer role builds pipelines as Airflow DAGs: template-based jobs that move data through Raw, Clean, and Ready on a schedule you set. That scheduled, whole-batch model is how the platform runs every pipeline today, from ingestion through Clean and Ready.

Related terms

• Stream processing: The alternative model: processing each record as it arrives instead of in scheduled groups.
• Data orchestration: The scheduler, Airflow in OptimaFlo's case, that triggers and sequences batch runs.
• Change data capture: A technique for capturing only what changed, often paired with streaming rather than batch.