Why "S3 to Iceberg" and not "S3 to BigQuery" or another destination?

Apache Iceberg is OptimaFlo's native format for the Raw, Clean, and Ready layers, not an export target you configure. Once data lands as Iceberg tables in your AWS account, you can still export a Gold table to BigQuery, Snowflake, GCS, Postgres, or MySQL, that is a separate step layered on top of this one.

What happens if my S3 prefix has more than one file?

OptimaFlo does not silently pick the first key it lists. Unless you pin specific files or explicitly opt into "select all files in this prefix," a prefix with multiple candidate files fails the connection with a clear error listing what it found, so a folder with an unrelated leftover file cannot get ingested by accident.

Does OptimaFlo need my AWS access keys?

Not necessarily. The recommended pattern for production is a cross-account IAM role with a trust policy scoped to OptimaFlo's account and an external ID, so no long-lived secret ever leaves your AWS account. Static access keys are supported but are the fallback, not the default.

Data integration

S3 to Iceberg: land Amazon S3 files as Apache Iceberg tables

Amazon S3Apache Iceberg

Last updated Sunday, Aug 2, 2026

How it works

1
Choose an auth method
Explicit access keys, cross-account STS AssumeRole with an external ID, an IAM role attached to the running instance, or the default AWS credential chain, in that priority order.
2
Point at a bucket and prefix
Give OptimaFlo an s3:// or s3a:// URI. The wizard can also list buckets and browse top-level prefixes for a credential set before the source is fully configured.
3
Pick a file format and resolve ambiguity
Declare csv, parquet, json, or avro. If more than one file matches the prefix and none is pinned, OptimaFlo fails loud and asks you to select specific files or opt into "all files in this prefix," rather than silently reading the first one it finds.
4
Infer schema
Parquet schema comes from the file's own footer metadata; other formats are parsed and their Arrow schema mapped to the same integer/float/string/boolean/date/timestamp vocabulary every connector shares.
5
Land in Raw (Iceberg)
Matching files land as an Apache Iceberg table in your own AWS account, with ACID snapshots and full history from the first load.

S3 to Iceberg is the most literal integration on this list: OptimaFlo reads files from a bucket and prefix you choose, and writes them as Apache Iceberg tables inside your own AWS account. There is no separate "destination" step for this one, Iceberg is the format everything downstream (Clean, Ready, and any BI tool that speaks Iceberg or SQL) reads from. Setup is about 10 to 15 minutes once you know the bucket, prefix, and file format.

Authentication, in priority order

The connector tries, in order: explicit access_key_id/secret_access_key from config, STS AssumeRole for cross-account access (with an external ID to prevent the confused-deputy problem), an IAM role attached to the infrastructure OptimaFlo is running on, environment variables, then the default AWS credential chain. For production, cross-account role assumption is the pattern: you create an IAM role in your account with a trust policy naming OptimaFlo's account and your external ID, OptimaFlo generates that trust policy and a matching bucket policy for you, and no static secret ever has to leave your account.

File formats and the ambiguous-prefix guard

CSV, Parquet, JSON, and Avro are supported, with automatic gzip, bz2, and snappy decompression on read. The detail worth knowing before you connect: if your prefix contains more than one object and you have not pinned specific files or explicitly opted into "include all files here," the connector refuses to guess. It raises an error naming every candidate key it found. This exists because silently reading whichever file S3's listing API happened to return first is exactly how a stale or unrelated file in the same folder ends up in your pipeline without anyone noticing.

Schema inference

For Parquet, schema comes from the file's own footer metadata, which describes column names, types, and nullability precisely. For CSV, JSON, and Avro, OptimaFlo parses the file and maps its Arrow schema to the same integer, float, string, boolean, date, and timestamp vocabulary that every other connector (GCS, BigQuery, Postgres, MySQL, Redshift) uses, so a downstream pipeline step does not need to know or care which connector a table came from.

What is not supported

Parquet schema inference today downloads the whole object rather than range-requesting just the footer, which is fine for typical file sizes and a known cost on very large single files (a planned optimization, not a blocker). There is no built-in partition-key configuration in the wizard: files land as a single Iceberg table per configured source, and partitioning decisions for that table happen as part of the Clean and Ready modeling, not as an S3-connector setting. And there is no continuous "watch this prefix for new files" trigger yet; ingestion runs on your pipeline's schedule.

Frequently asked questions

Set up by an AI data team, not a config file

Connect this source and see it land in your own cloud.

Start your pilot Book a walkthrough

Now in early beta. One flat plan, no per-query tax. Runs in your cloud. Your data never leaves.