What pagination styles does OptimaFlo handle?

Cursor-based (a token in the response points to the next page), offset-based, page-number-based, and next-URL-based, where the response itself contains the full URL to call next. You configure which one your API uses; OptimaFlo follows it until a page returns fewer rows than requested.

How does OptimaFlo flatten nested JSON?

A five-phase algorithm: flatten nested dicts with underscore-joined keys, unwrap lists-of-lists, separate scalars from arrays, transpose parallel arrays into one row per index, then re-flatten anything left over. It is the same flatten used for schema inference, preview, and the real ingest run, so what you see in preview is what actually lands.

What stops OptimaFlo from being pointed at an internal service by mistake or by an attacker-supplied URL?

Every REST and GraphQL connector goes through an SSRF guard that resolves the target host and blocks loopback, link-local, private (RFC1918), carrier-grade NAT, and cloud-metadata IP ranges, both on the initial request and on every redirect hop, so a 302 cannot bounce a clean-looking URL to an internal target.

Data integration

REST API to your warehouse: ingest any JSON or CSV endpoint

REST APIBigQuery, Snowflake, S3, GCS, Postgres, or MySQL

Last updated Sunday, Aug 2, 2026

How it works

1
Point at a base URL and endpoint
Give OptimaFlo the API's base URL and endpoint path. Every request goes through an SSRF guard that resolves and checks the target host before it is ever called.
2
Configure authentication
API key (header or query param), bearer token, basic auth, OAuth2 client credentials, or custom headers, matched to whatever the API actually requires.
3
Set the pagination strategy
Cursor, offset, page number, next-URL, or none, whichever the API uses. OptimaFlo follows it automatically until the response signals there is nothing left.
4
Extract and flatten the response
Point at the JSON path holding your records, then a five-phase flatten turns nested objects and parallel arrays into flat rows with underscore-joined column names.
5
Land in Raw, model Clean and Ready, export to your warehouse
Flattened records land as an Iceberg table, the AI data team builds Clean and Ready layers, and the Gold table exports to BigQuery, Snowflake, S3, GCS, Postgres, or MySQL.

REST API is the most flexible source on this list because it is not one API, it is any JSON or CSV endpoint you point it at. Setup time depends entirely on the API: a simple bearer-token, offset-paginated endpoint is a 10-minute connect; an OAuth2, cursor-paginated, deeply nested one is closer to 30 to 40 minutes of configuration.

Authentication

Five auth types are supported: API key (sent as a header or a query parameter), bearer token, basic auth (username and password), OAuth2 client credentials for machine-to-machine access, and custom headers for anything proprietary. You configure which one matches the API; OptimaFlo does not try to auto-detect it.

Pagination, handled for you

Four pagination strategies, plus "none" for single-response endpoints: cursor (a token in the response body points to the next page), offset, page number, and next-URL (the response contains a full URL to call next). Whichever style your API uses, OptimaFlo follows it automatically, stopping when a page returns fewer rows than requested, the standard termination signal for cursor-style pagination.

Nested JSON, flattened consistently

Most APIs do not return flat rows, they return nested objects and arrays. OptimaFlo runs a five-phase flatten: recursively flatten nested dicts into underscore-joined column names, unwrap lists-of-lists, separate the now-flat record into scalars and arrays, transpose parallel arrays into one output row per index, then flatten anything still left over. This is not a one-off preview transform, it is the same function schema inference, the live preview, and the actual Bronze ingest all call, so the columns you see before connecting match what actually lands.

Many APIs also report failures as HTTP 200 with an errors or error field and an empty data array (a known pattern from several sports and finance data APIs). OptimaFlo checks for those body-level error fields explicitly, so a rate limit or a missing parameter shows up as a clear failure instead of looking like "the endpoint just has no data."

Security: SSRF protection by construction

A REST connector is, by definition, a URL you configure OptimaFlo to call. Before OPT-668, that check only lived in one wizard code path; every connector's actual request path now goes through a shared SSRF guard that resolves the target hostname and blocks loopback, link-local, RFC1918 private ranges, carrier-grade NAT, and cloud-metadata IP addresses, both on the initial request and on every redirect hop. Redirects are still followed (some APIs legitimately 302 to a CDN), just revalidated at each hop rather than trusted blindly.

What is not supported

There is no automatic schema-drift handling if an API silently renames a field between runs (the new field just shows up as a new column), no built-in rate-limit backoff configuration beyond what you set explicitly, and GraphQL APIs use a separate connector with its own query-based configuration rather than this REST one.

Frequently asked questions

Set up by an AI data team, not a config file

Connect this source and see it land in your own cloud.

Start your pilot Book a walkthrough

Now in early beta. One flat plan, no per-query tax. Runs in your cloud. Your data never leaves.