BYOC Documentation
GCP BYOC Setup Guide
BYOC (Bring Your Own Cloud) on GCP deploys OptimaFlo's agent and Apache Iceberg storage inside your own Google Cloud project, provisioned by Terraform. Your data is stored in your infrastructure, never ours.
New to BYOC? Start with the BYOC overview to see how it works and how it compares to a fully-hosted platform.
Quick Start Overview
Step 1: Enable Required GCP APIs
The BYOC agent requires several GCP APIs to be enabled in your project. These APIs provide the foundation for Cloud Run deployment, networking, data services, and monitoring.
| API | Purpose | Status |
|---|---|---|
compute.googleapis.comCompute Engine API | VPC networks, firewall rules, networking | Required |
run.googleapis.comCloud Run Admin API | Deploy and manage Cloud Run services | Required |
vpcaccess.googleapis.comServerless VPC Access API | VPC connectors for private networking | Required |
iam.googleapis.comIAM API | Service accounts and IAM bindings | Required |
secretmanager.googleapis.comSecret Manager API | Store agent tokens securely | Required |
cloudresourcemanager.googleapis.comCloud Resource Manager API | Project-level IAM policies | Required |
storage.googleapis.comCloud Storage API | GCS bucket operations | Required |
bigquery.googleapis.comBigQuery API | BigQuery dataset and table operations | Required |
composer.googleapis.comCloud Composer API | Managed Apache Airflow for pipeline orchestration | Required |
container.googleapis.comKubernetes Engine API | GKE clusters (Composer dependency) | Required |
sqladmin.googleapis.comCloud SQL Admin API | Composer metadata database | Required |
pubsub.googleapis.comPub/Sub API | Composer internal messaging | Required |
logging.googleapis.comCloud Logging API | Application logging | Required |
monitoring.googleapis.comCloud Monitoring API | Metrics and alerting | Required |
cloudtrace.googleapis.comCloud Trace API | Distributed tracing | Required |
iamcredentials.googleapis.comIAM Service Account Credentials API | Service account impersonation tokens | Required |
artifactregistry.googleapis.comArtifact Registry API | Container image storage | Required |
cloudbuild.googleapis.comCloud Build API | Infrastructure provisioning builds | Required |
servicenetworking.googleapis.comService Networking API | VPC peering for Cloud SQL | Required |
sql-component.googleapis.comCloud SQL Component API | Cloud SQL internal components | Required |
bigqueryconnection.googleapis.comBigQuery Connection API | BigLake connections for Iceberg | Required |
bigquerystorage.googleapis.comBigQuery Storage API | High-throughput BigQuery reads/writes | Required |
Enable All APIs at Once In The Command Line
gcloud services enable \
compute.googleapis.com \
run.googleapis.com \
vpcaccess.googleapis.com \
servicenetworking.googleapis.com \
iam.googleapis.com \
iamcredentials.googleapis.com \
secretmanager.googleapis.com \
cloudresourcemanager.googleapis.com \
storage.googleapis.com \
bigquery.googleapis.com \
bigqueryconnection.googleapis.com \
bigquerystorage.googleapis.com \
composer.googleapis.com \
container.googleapis.com \
sqladmin.googleapis.com \
sql-component.googleapis.com \
pubsub.googleapis.com \
artifactregistry.googleapis.com \
cloudbuild.googleapis.com \
dataproc.googleapis.com \
logging.googleapis.com \
monitoring.googleapis.com \
cloudtrace.googleapis.com \
--project=YOUR_PROJECT_IDNote: APIs take 30-60 seconds to propagate after enabling. Wait before running BYOC.
Step 2: Configure IAM Permissions
The OptimaFlo platform service account needs access to browse, preview, and write processed data (Bronze/Silver/Gold layers) to your BigQuery and GCS. Permissions follow the principle of least privilege.
The platform service account in the commands below is optimaflo-sa@production-optimaflo.iam.gserviceaccount.com. This is OptimaFlo's fixed identity, not a placeholder. Use it as-is when granting roles in your project. Authentication from the platform to your project uses Workload Identity Federation under the hood, so no key files are exchanged.
| IAM Role | Purpose |
|---|---|
roles/bigquery.admin | Create and manage BigQuery datasets, tables, and query jobs (Bronze/Silver/Gold layers) |
roles/cloudbuild.builds.editor | Submit Cloud Build jobs during deploy and unattended teardown |
roles/cloudsql.admin | Provision and manage the Cloud SQL instance backing Airflow |
roles/composer.admin | Create and manage the Cloud Composer environment and run Airflow commands |
roles/compute.admin | Provision the VPC, subnets, and compute resources the deployment needs |
roles/container.admin | Create and manage the GKE Autopilot cluster that runs Sail queries |
roles/iam.serviceAccountAdmin | Create the deployment's own service accounts |
roles/iam.serviceAccountTokenCreator | Impersonate the Agent service account to mint short-lived tokens |
roles/iam.serviceAccountUser | Act as a service account when submitting jobs (e.g. Cloud Build) |
roles/iam.workloadIdentityPoolAdmin | Set up Workload Identity Federation for keyless authentication |
roles/logging.logWriter | Write deployment and pipeline logs to Cloud Logging |
roles/monitoring.admin | Configure monitoring and alerting for the deployment |
roles/resourcemanager.projectIamAdmin | Grant itself the rest of this list in one pass (the one-time bootstrap grant) |
roles/run.admin | Deploy and invoke the Polaris catalog and Agent API Cloud Run services |
roles/secretmanager.admin | Create and rotate secrets (database credentials, API keys) |
roles/secretmanager.secretAccessor | Read secrets at runtime |
roles/serviceusage.serviceUsageAdmin | Enable required GCP APIs on later unattended deploys |
roles/servicenetworking.networksAdmin | Set up private VPC peering for Cloud SQL |
roles/storage.admin | Manage GCS buckets and their IAM policies |
roles/storage.objectAdmin | Read and write your data buckets (Bronze/Silver/Gold layers) |
roles/vpcaccess.admin | Create the Serverless VPC Access connector |
Granting Access
OptimaFlo grants itself the roles above. You don't run a command per role. The first time you deploy to a project, the deploy stops and shows two commands, pre-filled with your project ID and org's label value: one that proves you control the project, one that lets OptimaFlo configure IAM there. Run both, then deploy again.
1. Prove you control the project:
# Set your project ID and the expected label value shown in the app
export PROJECT_ID="your-project-id"
export EXPECTED_LABEL_VALUE="value-shown-in-app"
gcloud projects update $PROJECT_ID \
--update-labels optimaflo-org=$EXPECTED_LABEL_VALUE2. Let OptimaFlo configure IAM:
export PROJECT_ID="your-project-id"
export PLATFORM_SA="optimaflo-sa@production-optimaflo.iam.gserviceaccount.com"
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member="serviceAccount:$PLATFORM_SA" \
--role="roles/resourcemanager.projectIamAdmin"With that role, OptimaFlo grants itself the roles listed above the first time it deploys to the project. Older deployments, from before that automatic grant existed, may still need a few of the individual commands below.
Grant Permissions via gcloud CLI
Step 3: Deploy via Dashboard
Once APIs are enabled and permissions are granted, deploy the BYOC agent using the OptimaFlo dashboard. The guided wizard walks you through the deployment process.
- 1Connect GCP Project, enter your project ID and authenticate with OAuth
- 2Configure Agent, select region, network settings, and data sources
- 3Deploy, click deploy and watch real-time progress as your agent provisions
- 4Verify, the dashboard shows agent health and connectivity status