Getting Started

Your First Pipeline in 5 Minutes

Connect a data source, build a medallion pipeline, and schedule it all from the browser. No infrastructure to manage, no YAML to write.

Quick Start

Create your organization and workspace through the onboarding wizard

Connect a Source

Link GCS, BigQuery, or a REST API to your workspace

Build a Pipeline

Use the visual canvas or let Concierge AI generate one for you

Run & Schedule

Execute your pipeline and set an Airflow schedule

Sign Up & Set Up Your Workspace

The onboarding wizard walks you through creating your organization and workspace in four steps. Everything is collected up front and submitted together at the end.

Organization: name your organization, set a URL slug, and select your industry and team size

Workspace: create one or more workspaces (up to 5 during onboarding). Separate by environment, team, or project.

BYOC: configure your Bring Your Own Cloud deployment on GCP. Select your infrastructure tier and region, then the platform provisions everything. AWS and Azure support is coming soon.

Each workspace gets its own Apache Iceberg catalog, so data from different workspaces is fully isolated — even within the same organization.

Connect a Data Source

OptimaFlo currently supports Google Cloud Storage, BigQuery, and REST API connectors, with more on the way. The Data Source AI agent guides you through the entire process conversationally; tell it what you want to connect and it handles authentication, browsing, file selection, validation, and schema inference.

Available Today

Google Cloud Storage
BigQuery
REST API (any endpoint)

Coming Soon

Amazon S3
Redshift
Snowflake
PostgreSQL
MySQL
GraphQL

Open Data Sources in the sidebar and click Add Source — this opens the Data Source AI agent

Tell the agent what you want to connect (e.g. "Connect my GCS bucket gs://company-data") — it authenticates via OAuth for cloud sources or asks for credentials for databases

The agent browses your buckets, folders, or tables and lets you select specific files or datasets to ingest

It validates the connection, infers your schema, and creates the data source record — ready for use in a pipeline

For GCS and BigQuery, authentication happens via a Google OAuth popup — no service account keys to manage. The platform handles token refresh automatically.

Build Your Pipeline

Three ways to build: drag-and-drop on the visual canvas, use the Pipeline Generator AI to create a pipeline from a prompt, or let Concierge AI handle the entire workflow end-to-end.

Visual Canvas

Drag nodes onto the canvas and connect them. Configure SQL for each transform node. Preview results before saving.

Add a Ingestion node linked to your data source
Add Cleaning and Aggregation nodes for transformations
Connect them with edges to define data flow
Write SQL or use the SQL Copilot to generate it

Pipeline Generator AI

Describe your transformations and the generator builds the full canvas — Source, Ingestion, Cleaning, and Aggregation nodes with SQL already written.

Open the Generator panel in the canvas sidebar
Describe what you want to transform
Review the generated nodes and SQL
Apply to canvas with one click

Concierge AI

End-to-end orchestration. Concierge connects sources, generates SQL, builds the canvas, and deploys the pipeline — all from one conversation.

"Connect my GCS bucket gs://sales-data, clean the CSVs, deduplicate on order_id, and create a monthly revenue summary."

Run and Schedule

Execute your pipeline manually or set a schedule. OptimaFlo generates an Apache Airflow DAG behind the scenes; you never touch Airflow config directly.

Click Execute to run the pipeline immediately. The platform auto-saves before executing

Monitor progress in the execution panel and watch each layer complete from Ingestion through Aggregation

Open Settings to set a schedule (hourly, daily, weekly, monthly, quarterly, or yearly). The platform converts your selection to an Airflow cron

Use Backfills to re-process historical date ranges when you change transformation logic

Backfills run sequentially by default to avoid Iceberg write conflicts. You can increase parallelism for independent tables.

Core Concepts: Ingestion, Cleaning, Aggregation

The medallion architecture is the backbone of every OptimaFlo pipeline.

Ingestion

Raw Data

Your source data lands here untouched. Every record is preserved in Apache Iceberg tables with full history, ACID transactions, and time-travel.

Zero transformations — exact copy of the source
Schema detected automatically on ingestion
Full history retention for compliance and replay
Partitioned for query performance

Cleaning

Cleaned Data

Cleaned, deduplicated, and type-cast. SQL Copilot generates transformations from plain English, then you review and approve before anything runs.

LLM-generated SQL from natural language
Preview results before committing
Deduplication, null handling, and type casting
Validated and schema-enforced before execution

Aggregation

Business Metrics

Aggregated, business-ready metrics and star schemas. Feed dashboards, exports, and analyst queries from a single source of truth.

Aggregations and business KPIs
Star schema for analytics
Incremental updates to minimize compute
Direct connection to BI dashboards

What's Next

Deepen your knowledge with these guides.

Platform Architecture

Understand the 8-layer architecture and engine selection

AI Agents

Use Concierge AI to build pipelines from a single prompt

Pipeline Builder

Deep dive into the visual canvas and node types

BYOC Deployment

Deploy in your own GCP project for full data residency

Ready to build?

Create your first pipeline in minutes.

Start Building Contact Us

Connect a source, transform with SQL, and schedule with Airflow — all from the browser.

OptimaFlo

AI-native data platform. From raw data to business dashboards powered by Apache open standards, visual pipeline building, and AI agents that handle the heavy lifting.