HTTP(s) Data Connector

The HTTP(s) Data Connector enables federated SQL query across supported file formats stored at an HTTP(s) endpoint. The connector supports dynamic query and data refresh through SQL-based filtering.

datasets:
  - from: http://static_username@localhost:3001/report.csv
    name: local_report
    params:
      http_password: ${env:MY_HTTP_PASS}

Examples

Basic Example

datasets:
  - from: https://github.com/LAION-AI/audio-dataset/raw/7fd6ae3cfd7cde619f6bed817da7aa2202a5bc28/metadata/freesound/parquet/freesound_parquet.parquet
    name: laion_freesound

Using Basic Authentication

datasets:
  - from: http://static_username@localhost:3001/report.csv
    name: local_report
    params:
      http_password: ${env:MY_HTTP_PASS}

The username is taken from the user-info section of the from URL (user@host) or from the http_username parameter. The password comes from the http_password parameter. The connector then sends a standard RFC 7617 Basic authentication header on every request:

Authorization: Basic <base64(username:password)>

For example, static_username with password s3cret produces Authorization: Basic c3RhdGljX3VzZXJuYW1lOnMzY3JldA==. Only one of http_password or user info in the URL can provide the password: setting both is not supported.

Prefer http_password with a secret store over a password in the URL. For a dynamic JSON API endpoint, query errors and endpoint-validation warnings name only the origin of the request URL (scheme, host, and port), for example HTTP request to http://127.0.0.1:8080 failed: error sending request, so its user info, path, and query string stay out of them. The dataset registration log line, and errors from a structured HTTP file dataset, can print the configured from URL with its user info.

Using Custom Headers

Custom HTTP headers can be specified for authentication, API keys, or other requirements. Headers are treated as sensitive data and will not be logged.

http_headers applies to dynamic JSON API endpoints only. A structured HTTP file dataset (csv, tsv, parquet, arrow, avro, jsonl/ndjson/ldjson, soda, socrata, vortex, or a static json file) is served by the object-store listing path, which cannot carry request headers, so the headers are ignored and a warning is logged naming the dataset. To authenticate a structured file download, use Basic authentication (http_username / http_password, or user info in the URL).

datasets:
  - from: https://api.example.com
    name: api_data
    params:
      http_headers: 'Authorization:Bearer ${secrets:api_token},Accept:application/json'

Headers can also be separated by semicolons:

datasets:
  - from: https://api.example.com
    name: api_data
    params:
      http_headers: 'Authorization: Bearer ${secrets:api_token}; X-API-Key: ${secrets:api_key}'

Using OAuth2 Refresh-Token Authentication

For JSON APIs protected by OAuth2, the connector can acquire short-lived access tokens from a token endpoint and keep them fresh automatically. The refresh-token grant (RFC 6749 §6) exchanges a long-lived refresh token; the client-credentials grant (RFC 6749 §4.4) authenticates with a client_id/client_secret for machine-to-machine APIs. On startup Spice hits the configured token endpoint once, then stamps Authorization: Bearer <access_token> on every data request (or a custom header: see Custom Token Header) and refreshes the token in the background before it expires.

datasets:
  - from: https://api.example.com
    name: secure_data
    params:
      file_format: json
      allowed_request_paths: '/v1/**'
      auth_token_url: https://auth.example.com/oauth/token
      http_auth_refresh_token: ${secrets:my_refresh_token}
      http_auth_client_id: ${secrets:my_client_id}
      http_auth_client_secret: ${secrets:my_client_secret}
      auth_scopes: 'read:data offline_access'

The http_auth_refresh_token, http_auth_client_id, and http_auth_client_secret parameters can be loaded from any supported secret store (environment variables, Kubernetes Secrets, AWS Secrets Manager, HashiCorp Vault, the OS keychain, etc.) using the ${secrets:...} replacement syntax.

Applies to dynamic JSON API endpoints only (e.g. file_format: json with allowed_request_paths). A structured HTTP file dataset (csv/parquet/etc.) goes through the object-store listing path, which cannot attach an access token, so setting any OAuth2 parameter on one is rejected at registration: the dataset fails to load with a configuration error naming the parameters to remove, rather than quietly sending unauthenticated requests. http_headers is not applied on that path either; use Basic authentication to authenticate a structured file download.

See OAuth2 Authentication for the full parameter reference and behavior notes.

Configuration

from

The from field specifies the HTTP(s) endpoint and can be configured in two ways:

1. Direct URL to a file: A complete URL pointing to a specific supported file.

   from: https://example.com/data/report.csv

2. Base domain/path: A base URL that will be combined with special metadata fields to construct the complete request.

   from: https://api.example.com/v1

The connector supports templated URLs with query parameters that can be dynamically populated using refresh_sql filters and special metadata fields.

name

The dataset name. This will be used as the table name within Spice.

Example:

datasets:
  - from: http://static_username@localhost:3001/report.csv
    name: cool_dataset
    params: ...

SELECT COUNT(*) FROM cool_dataset;

+----------+
| count(*) |
+----------+
| 6001215  |
+----------+

The dataset name cannot be a reserved keyword.

params

The connector supports authentication, timeout, connection pooling, and retry configuration via params.

Mutual TLS (mTLS) Client Authentication

For upstream servers that require mutual TLS, the connector can present a client certificate during the TLS handshake. Provide the certificate and key either as file paths or inline PEM: the file-path and inline forms are mutually exclusive, and the certificate and key must always be set together. mTLS client identity applies to dynamic JSON API endpoints only; structured HTTP file datasets reject these parameters.

Rate Control Parameters

HTTP-based connectors share a rate control system that limits concurrency and request rate per upstream origin. These parameters can be set per-dataset (in params) or globally (in runtime.params). Dataset-level settings override the global defaults.

Multiple datasets targeting the same origin share the same rate controller, ensuring the limits apply across all datasets for that origin. Every dataset that shares an origin must resolve to the same values for these parameters; a dataset whose values differ fails to load. See HTTP Rate Control for adaptive throttling, bounded waits, and limits across instances.

runtime:
  params:
    http_max_concurrent_requests: 10
    http_requests_per_second_limit: 5

datasets:
  - from: https://api.example.com/v1
    name: api_data
    params:
      file_format: json
      allowed_request_paths: '/data/**'
      max_concurrent_requests: 3        # Override: this dataset uses at most 3 concurrent requests

Pagination Parameters

Caching Mode Parameters

When using refresh_mode: caching, cache freshness is controlled by additional parameters placed under acceleration.params: not under the top-level params block.

To prefer the origin and fall back to the accelerator only when the origin fails, set caching_ttl: 0s, caching_stale_while_revalidate_ttl: 0s, and caching_stale_if_error to enabled or a finite duration. While the origin is healthy, Spice does not serve the accelerator. A successful response is still stored and is consulted only after a later origin failure. A zero TTL does not mean every read is served from the accelerator. Pair unbounded enabled with caching_max_size, caching_max_items, or retention. See Prefer the origin, fall back on failure.

datasets:
  - from: https://api.example.com/v1
    name: api_cache
    params:
      file_format: json
      allowed_request_paths: '/data/**'
      request_query_filters: enabled
    acceleration:
      enabled: true
      refresh_mode: caching
      refresh_check_interval: 15m
      on_zero_results: use_source
      params:
        caching_ttl: 15m                        # explicit: default is only 30s
        caching_stale_while_revalidate_ttl: 5m  # serve stale data while refreshing

See Caching Refresh Mode for full TTL semantics, stale-while-revalidate behaviour, and cache persistence options.

HTTP Response Headers

When querying HTTP(s) datasets, Spice respects standard HTTP caching headers in responses. The connector supports the following cache-related response headers:

Cache-Control

The Cache-Control response header from the HTTP(s) endpoint is passed through to clients querying Spice. When the HTTP(s) server returns a Cache-Control header with the stale-while-revalidate directive, clients can use this value to determine appropriate caching behavior.

For example, if the HTTP(s) endpoint returns:

Cache-Control: max-age=10, stale-while-revalidate=10

Clients querying Spice will receive this header and can:

1. Serve fresh data for 10 seconds after fetching.
2. Between 10-20 seconds, serve stale data while fetching fresh data in the background.
3. After 20 seconds, fetch fresh data before serving the next request.

The stale-while-revalidate behavior in Spice is controlled by the stale_while_revalidate_ttl parameter in the caching configuration. When stale_while_revalidate_ttl is set to 0 (default), stale data will not be served. When set to a non-zero value, Spice serves stale cache entries while revalidating in the background.

Response Cache

For dynamic JSON API endpoints, the connector keeps origin responses in an in-memory cache keyed by the request shape (path, query, body, and request headers) so a repeat of the same request inside the response's own freshness window is served without another origin call. This cache is the connector's own; it is not one of the caches configured under runtime.caching, and it is unrelated to refresh_mode: caching, which caches query results rather than HTTP responses. Structured HTTP file datasets (parquet, csv, and the other listing-table formats) do not use it.

Size budget

response_cache_max_size_bytes bounds what one dataset's cache retains, counting each response body together with the request key it is held under. It defaults to 67108864 (64 MiB) and is applied per dataset, so the runtime's total exposure grows with the number of HTTP datasets: raise it only for a dataset that earns it. Once the budget is reached, entries are evicted to stay inside it. Set 0 to disable the cache entirely.

The value must be a whole number of bytes; suffixed forms such as 64MiB are rejected at load with an error naming the parameter, rather than being replaced by the default.

A single entry (one response plus the request key it is held under) of 4 GiB or more is never admitted, whatever the budget is set to, because it cannot be charged what it holds.

What is retained, and for how long

The origin decides first, from the Cache-Control header on its response:

A Vary response header additionally refuses retention when it is *, when it names the header the connector's HTTP authentication uses, or when its bytes cannot be read as text. Any other named field is safe, because everything a query can vary (path, query, body, and pushed-down request headers) is already part of the cache key.

Caching an origin that sends no Cache-Control

response_cache_fallback_ttl is how long to retain a response whose origin sent no Cache-Control header at all, for example:

datasets:
  - from: https://api.example.com/v1
    name: api_data
    params:
      file_format: json
      response_cache_fallback_ttl: 5m
      response_cache_max_size_bytes: 134217728 # 128 MiB

It is unset by default, so responses from a header-less origin stay uncached until an operator asks for them to be retained. It never overrides an origin that did send Cache-Control, including that origin's refusals, and unlike an origin-supplied max-age it is not reduced by the response's Age: it is how long you asked Spice to keep a response the origin said nothing about.

Occupancy of the cache is reported per dataset: see Metrics.

Advanced Features

The HTTP connector provides advanced capabilities for working with dynamic APIs and RESTful services, including built-in pagination and special metadata fields.

Pagination

The HTTP connector supports automatic pagination for REST APIs that return data across multiple pages. Pagination is configured via params and works transparently with acceleration (caching, append, and full refresh modes): each page is streamed as a separate batch without buffering entire result sets in memory.

Pagination Modes

There are three pagination modes:

URL mode: The next page URL is extracted from the response body (via pagination_next_pointer) or from the HTTP Link header with rel="next".

datasets:
  - from: https://api.example.com/v1/items
    name: items
    params:
      pagination: enabled
      pagination_next_pointer: /links/next
      pagination_data_pointer: /data
      pagination_max_pages: 50

Token mode: A cursor/token is extracted from the response body (via pagination_next_pointer) and passed as a query parameter (specified by pagination_token_param) in subsequent requests.

datasets:
  - from: https://api.example.com/v1/items
    name: items
    params:
      pagination: enabled
      pagination_next_pointer: /pagination/cursor
      pagination_token_param: cursor
      pagination_data_pointer: /results

Query-parameter mode: The client drives pagination by expanding a template (pagination_query_params) with {offset}, {limit}, and {page} variables. Pagination stops when a page returns fewer rows than pagination_page_size.

datasets:
  - from: https://api.example.com/v1/widgets
    name: widgets
    params:
      pagination: enabled
      pagination_query_params: "offset={offset}&limit={limit}"
      pagination_page_size: "100"
      pagination_max_pages: "50"

Map-to-Array Conversion

Some APIs return data as a JSON object/map (e.g., {"1": {...}, "2": {...}}) instead of an array. Set pagination_data_map_to_array: enabled to extract the map values as individual rows.

datasets:
  - from: https://api.example.com/v1/records
    name: records
    params:
      pagination: enabled
      pagination_data_map_to_array: enabled
      pagination_query_params: "offset={offset}&limit={limit}"
      pagination_page_size: "100"

Auto Mode

By default, pagination is set to auto, which automatically follows HTTP Link headers with rel="next" if present in responses. Set pagination: disabled to turn off all pagination behavior, or pagination: enabled to explicitly configure pagination with the parameters above.

Validation Rules

• pagination_query_params requires pagination_page_size (and vice versa)
• pagination_query_params is mutually exclusive with pagination_next_pointer and pagination_token_param
• pagination_query_params must contain {offset} or {page} to ensure pages advance
• pagination_token_param requires pagination_next_pointer
• pagination_next_pointer and pagination_data_pointer must be valid JSON pointers starting with /

SSRF Protection

When using URL mode, next-page URLs extracted from response bodies are validated to share the same origin as the base URL configured in from. Cross-origin redirects are rejected.

Special Metadata Fields

The HTTP connector supports special metadata fields that provide fine-grained control over HTTP requests. These fields can be included in your dataset schema to dynamically construct request URLs and payloads.