Redshift to Iceberg: land Amazon Redshift tables as open Iceberg tables

Redshift to Iceberg moves table or query results out of a Redshift cluster and into open Apache Iceberg tables in your own AWS account. Setup runs about 20 to 30 minutes, most of it spent deciding on an auth model (password versus IAM, same-account versus cross-account) rather than the connection itself.

Authentication

Three modes, in the order you would reach for them: explicit username and password, IAM database authentication (Redshift issues temporary credentials for a named database user through its own API, no long-lived password stored), and cross-account STS AssumeRole followed by IAM authentication, for a Redshift cluster that lives in a different AWS account than OptimaFlo's BYOC deployment. IAM auth is the credential-hygiene option: nothing static to rotate or leak.

How data actually moves

This is the detail worth knowing before you connect a large table: the connector uses the redshift_connector Python driver directly over Redshift's wire protocol. It opens a cursor, executes your SELECT (either a full table read or your custom query, with a LIMIT appended automatically if you did not specify one), and fetches the result set. There is no UNLOAD to S3 and re-read step, the pattern Redshift itself recommends for genuinely large bulk exports. For typical analytics tables this is not a practical problem; for a table in the billions of rows, expect the direct-cursor path to be the bottleneck, not compute.

Schema and identifiers

Table and schema names are validated against a safe identifier pattern before they are interpolated into the generated SELECT, defense in depth on top of using a parameterized cursor for everything else. If no table is set and no query is given, schema inference fails loud with a clear message rather than returning nothing; a query-based source (no single table) returns an empty schema today, since inferring columns from arbitrary SQL is not implemented yet.

Landing in Iceberg

The result lands as an Apache Iceberg table in the Raw layer, inside your own AWS account. From there it goes through the same Clean and Ready modeling every source does. Redshift itself can also be a destination-side warehouse concept in other setups, but for this connector Redshift is strictly the source: OptimaFlo does not write back to it.

What is not supported

No UNLOAD-based bulk export path, no schema-from-query inference, and no automatic incremental extraction, each run pulls the full table or the full result of your query. If a table is large enough that a single cursor fetch is impractical, splitting it with your own query-level filtering (a date range per pipeline run, for instance) is the current workaround, not something the connector does for you.