Does GCS ingestion use my Google login or a service account?

Either. OptimaFlo supports OAuth2 user consent for a conversational connect flow, a direct service account key, an impersonated service account, or Application Default Credentials, checked in that priority order.

What if my GCS prefix has more than one file in it?

Same guard as the S3 connector: unless you pin specific files or opt into "all files in this prefix," a prefix with more than one candidate file fails the connection with a clear error, instead of silently ingesting whichever one the bucket listing returned first.

Is GCS-to-BigQuery ingestion incremental?

No, not automatically. Each run reads the configured file or files in full. There is no modification-date watermark tracked between runs today, so plan your pipeline schedule around full reads of the current file set.

Data integration

GCS to BigQuery: land Google Cloud Storage files as Iceberg, export to BigQuery

Google Cloud StorageBigQuery

Last updated Sunday, Aug 2, 2026

How it works

1
Choose an auth method
OAuth2 user consent, a service account key, an impersonated service account, Application Default Credentials, or a stored workload identity, checked in that order.
2
Point at a bucket and prefix
Give OptimaFlo a gs:// URI. The wizard can browse projects, buckets, and folders before the source is fully configured.
3
Pick a file format and resolve ambiguity
Declare csv, parquet, json, or avro. If a prefix matches more than one file and none is pinned, OptimaFlo fails loud rather than guessing which one to read.
4
Infer schema
Parquet schema comes from the file's footer metadata; other formats are parsed and mapped to the shared integer/float/string/boolean/date/timestamp vocabulary.
5
Land in Raw, model Clean and Ready, export to BigQuery
Matching files land as an Iceberg table in your own GCP project first, then the AI data team builds Clean and Ready layers, and the Gold table exports to your BigQuery dataset.

GCS to BigQuery is a two-hop path through OptimaFlo: files land as Apache Iceberg tables in your own GCP project first (the Raw layer), then modeled Gold output exports to a BigQuery dataset. Setup is about 10 to 15 minutes once you know the bucket, prefix, and file format.

Authentication options

GCS supports more auth paths than most connectors, in priority order: OAuth2 user credentials from a conversational connect flow, a directly supplied service account key, an impersonated service account, Application Default Credentials when explicitly requested, a stored Workload Identity lookup for a previously saved source, and a default-credentials fallback. In practice this means you are not required to hand over a long-lived service account key just to try the connector; OAuth consent covers the common case.

File formats and the ambiguous-prefix guard

CSV, Parquet, JSON, and Avro are supported. The same rule S3 enforces applies here: if a prefix matches more than one file and you have not pinned specific object paths or explicitly opted into "all files in this prefix," the connector refuses to guess and fails with a clear error naming every candidate it found. Parquet schema reads from the file's footer metadata; the other formats are parsed and their Arrow schema mapped into the same type vocabulary every connector shares, so a Postgres-sourced table and a GCS-sourced table look identical to anything downstream.

Getting to BigQuery

Once Ready-layer models exist, the Gold table exports to your BigQuery project and dataset in append, overwrite, or upsert mode. Because both the source (GCS) and the destination (BigQuery) sit in GCP, this is often the shortest network path of any integration on this list: no cross-cloud egress, assuming your OptimaFlo BYOC deployment runs in the same GCP project or a peered one.

What is not supported

There is no modification-date watermark or "only ingest new and changed files" behavior tracked between runs; each run reads the currently configured file set in full. Partitioning of the resulting Iceberg table is not a GCS-connector setting, it happens during Clean and Ready modeling. And there is no event-driven trigger (a Pub/Sub notification on new object creation, for example) wired up yet; ingestion runs on your pipeline's schedule.

Frequently asked questions

Set up by an AI data team, not a config file

Connect this source and see it land in your own cloud.

Start your pilot Book a walkthrough

Now in early beta. One flat plan, no per-query tax. Runs in your cloud. Your data never leaves.