MySQL to BigQuery: sync MySQL and MariaDB data to BigQuery

MySQL and MariaDB share one connector in OptimaFlo, and both land in BigQuery the same way: through an Iceberg-backed Raw layer in your own cloud, not a direct copy. Setup is 15 to 20 minutes for the connection and table selection; the schema reflects automatically once a table is chosen.

What is actually different about MySQL

The connector mirrors the Postgres one almost exactly (same SQLAlchemy reflection approach, same type mapping to integer/float/string/boolean/timestamp/date, same fail-loud behavior when no table or query is set), with one structural difference: in MySQL, the database is the schema namespace. Postgres has a separate schema concept inside a database; MySQL does not, so table reflection is scoped straight to the database you connect to. It accepts a mysql:// connection string or discrete host, port, database, and credential fields, plus a raw SQL query as the table-less alternative.

Getting from MySQL into an Iceberg table

The first pipeline run pulls the target table (or query result) in full and writes it as an Apache Iceberg table in the Raw layer, inside your own GCP or AWS account under OptimaFlo's BYOC model. From there the AI data team builds Clean and Ready models, the same layered process every source goes through, whether it started as a database, a file in cloud storage, or an API response.

Full refresh, not change data capture

There is no binlog tailing here. MySQL supports row-based binary log replication for true CDC, but this connector does not use it: every run is a plain SELECT, either against the whole table or against a query you write yourself. For most reporting workloads, an hourly or daily run against an indexed, append-heavy table (orders, events, logs) is enough. If a table gets frequent in-place updates and you need to track exactly which rows changed since last run, add your own filter (WHERE updated_at > ...) to a custom query; the pipeline does not track a watermark for you automatically today.

Exporting to BigQuery

Once a Ready-layer table exists, the destination export writes to your BigQuery project, dataset, and table in one of three modes: append, overwrite, or upsert. This is a separate concern from BigQuery's other role in OptimaFlo as a warehouse compute engine for large-scale SQL transforms; here it is simply where the finished Gold table lands.

Honest limits

No CDC, no schema-from-query (a query-based source returns an empty schema rather than guessing one), and no automatic incremental watermarking. If sub-minute freshness between MySQL and BigQuery is the requirement, this pipeline is not built for that; it is built for the modeling, versioning, and orchestration around a scheduled batch load.