BYOC Documentation
GCP BYOC Setup Guide
BYOC (Bring Your Own Cloud) on GCP deploys OptimaFlo's agent and Apache Iceberg storage inside your own Google Cloud project, provisioned by Terraform. Your data is stored in your infrastructure, never ours.
New to BYOC? Start with the BYOC overview to see how it works and how it compares to a fully-hosted platform.
Quick Start Overview
1
Enable APIs
Enable required GCP APIs in your project
2
Grant Permissions
Configure IAM roles for the platform service account
3
Deploy via Dashboard
Use the OptimaFlo BYOC dashboard to deploy
Step 1: Enable Required GCP APIs
The BYOC agent requires several GCP APIs to be enabled in your project. These APIs provide the foundation for Cloud Run deployment, networking, data services, and monitoring.
| API | Purpose | Status |
|---|---|---|
compute.googleapis.comCompute Engine API | VPC networks, firewall rules, networking | Required |
run.googleapis.comCloud Run Admin API | Deploy and manage Cloud Run services | Required |
vpcaccess.googleapis.comServerless VPC Access API | VPC connectors for private networking | Required |
iam.googleapis.comIAM API | Service accounts and IAM bindings | Required |
secretmanager.googleapis.comSecret Manager API | Store agent tokens securely | Required |
cloudresourcemanager.googleapis.comCloud Resource Manager API | Project-level IAM policies | Required |
storage.googleapis.comCloud Storage API | GCS bucket operations | Required |
bigquery.googleapis.comBigQuery API | BigQuery dataset and table operations | Required |
composer.googleapis.comCloud Composer API | Managed Apache Airflow for pipeline orchestration | Required |
container.googleapis.comKubernetes Engine API | GKE clusters (Composer dependency) | Required |
sqladmin.googleapis.comCloud SQL Admin API | Composer metadata database | Required |
pubsub.googleapis.comPub/Sub API | Composer internal messaging | Required |
logging.googleapis.comCloud Logging API | Application logging | Required |
monitoring.googleapis.comCloud Monitoring API | Metrics and alerting | Required |
cloudtrace.googleapis.comCloud Trace API | Distributed tracing | Required |
iamcredentials.googleapis.comIAM Service Account Credentials API | Service account impersonation tokens | Required |
artifactregistry.googleapis.comArtifact Registry API | Container image storage | Required |
cloudbuild.googleapis.comCloud Build API | Infrastructure provisioning builds | Required |
servicenetworking.googleapis.comService Networking API | VPC peering for Cloud SQL | Required |
sql-component.googleapis.comCloud SQL Component API | Cloud SQL internal components | Required |
bigqueryconnection.googleapis.comBigQuery Connection API | BigLake connections for Iceberg | Required |
bigquerystorage.googleapis.comBigQuery Storage API | High-throughput BigQuery reads/writes | Required |
Enable All APIs at Once In The Command Line
bash
gcloud services enable \
compute.googleapis.com \
run.googleapis.com \
vpcaccess.googleapis.com \
servicenetworking.googleapis.com \
iam.googleapis.com \
iamcredentials.googleapis.com \
secretmanager.googleapis.com \
cloudresourcemanager.googleapis.com \
storage.googleapis.com \
bigquery.googleapis.com \
bigqueryconnection.googleapis.com \
bigquerystorage.googleapis.com \
composer.googleapis.com \
container.googleapis.com \
sqladmin.googleapis.com \
sql-component.googleapis.com \
pubsub.googleapis.com \
artifactregistry.googleapis.com \
cloudbuild.googleapis.com \
dataproc.googleapis.com \
logging.googleapis.com \
monitoring.googleapis.com \
cloudtrace.googleapis.com \
--project=YOUR_PROJECT_IDNote: APIs take 30-60 seconds to propagate after enabling. Wait before running BYOC.
Step 2: Configure IAM Permissions
The OptimaFlo platform service account needs access to browse, preview, and write processed data (Bronze/Silver/Gold layers) to your BigQuery and GCS. Permissions follow the principle of least privilege.
The platform service account in the commands below is optimaflo-sa@production-optimaflo.iam.gserviceaccount.com. This is OptimaFlo's fixed identity, not a placeholder. Use it as-is when granting roles in your project. Authentication from the platform to your project uses Workload Identity Federation under the hood, so no key files are exchanged.
| IAM Role | Scope | Purpose |
|---|---|---|
roles/bigquery.dataEditor | Project | Read and write datasets, tables, and table data (Bronze/Silver/Gold layers) bigquery.datasets.getbigquery.tables.getbigquery.tables.listbigquery.tables.getDatabigquery.tables.createbigquery.tables.updateData |
roles/bigquery.jobUser | Project | Execute queries for data preview and transformation bigquery.jobs.create |
roles/storage.objectViewer | Bucket | Read files and list objects in your data buckets storage.objects.getstorage.objects.list |
roles/run.invoker | Service | Invoke BYOC Cloud Run services (Polaris, Agent API) run.routes.invoke |
roles/iam.serviceAccountTokenCreator | Service Account | Generate access tokens for impersonation (on Agent SA) iam.serviceAccounts.getAccessTokeniam.serviceAccounts.signJwt |
roles/composer.worker | Project | Execute Airflow commands via Cloud Composer API (Agent SA) composer.environments.executeAirflowCommandcomposer.environments.get |
Grant Permissions via gcloud CLI
Step 3: Deploy via Dashboard
Once APIs are enabled and permissions are granted, deploy the BYOC agent using the OptimaFlo dashboard. The guided wizard walks you through the deployment process.
BYOC Agent Deployment
The BYOC dashboard provides a guided wizard to deploy your agent in minutes
Automatic infrastructure provisioning
Secure credential management
Real-time deployment progress
Health monitoring built-in
Automatic IAM configuration
No Terraform knowledge required
How It Works
- 1Connect GCP Project, enter your project ID and authenticate with OAuth
- 2Configure Agent, select region, network settings, and data sources
- 3Deploy, click deploy and watch real-time progress as your agent provisions
- 4Verify, the dashboard shows agent health and connectivity status