title: 'HTTP(s) Data Connector Deployment Guide' sidebar_label: 'Deployment Guide' description: 'Operating guide for the HTTP(s) data connector in production: authentication, rate control, retries, and observability.' sidebar_position: 10 pagination_prev: null pagination_next: null tags:

• data-connectors
• https
• observability

Production operating guide for the HTTP(s) data connector covering authentication, rate control, retry tuning, and observability.

Authentication & Secrets

The connector supports HTTP Basic, custom-header, and OAuth2 (refresh-token and client-credentials grants) authentication. Secrets must be sourced from a secret store (opens in a new tab) in production.

For OAuth2-protected APIs, prefer refresh-token flow over storing long-lived bearer tokens. The connector exchanges the refresh token for short-lived access tokens at startup and refreshes them before expiry.

TLS

Use HTTPS endpoints in production. auth_token_url must use HTTPS (loopback addresses are allowed for local testing only). Self-signed certificates require a trusted CA bundle in the container or host OS trust store.

For upstream servers that require mutual TLS (mTLS), the connector can present a client certificate during the TLS handshake. Supply the certificate and key as file paths or inline PEM — the two forms are mutually exclusive, and the certificate and key must be set together. mTLS client identity applies to dynamic JSON API endpoints only.

Resilience Controls

Rate Control

The HTTP connector participates in the shared HTTP rate control system. Concurrency and per-second/per-minute request limits can be configured per-dataset (in params) or globally (in runtime.params). Dataset-level settings override the global defaults. Multiple datasets targeting the same upstream origin share a single rate controller.

The runtime equivalents (http_max_concurrent_requests, http_requests_per_second_limit, http_requests_per_minute_limit, http_rate_control_jitter_min, http_rate_control_jitter_max, http_rate_control_acquire_timeout, http_rate_control_failure_threshold, http_rate_control_window) set defaults that apply to every HTTP-based connector unless overridden per dataset. Datasets that share an origin must use the same values, or the later dataset fails to load.

runtime:
  params:
    http_max_concurrent_requests: 10
    http_requests_per_second_limit: 5

datasets:
  - from: https://api.example.com/v1
    name: api_data
    params:
      file_format: json
      allowed_request_paths: '/data/**'
      max_concurrent_requests: 3 # Override for this dataset
      requests_per_minute_limit: 60

runtime:
  params:
    http_max_concurrent_requests: 10
    http_requests_per_second_limit: 5

datasets:
  - from: https://api.example.com/v1
    name: api_data
    params:
      file_format: json
      allowed_request_paths: '/data/**'
      max_concurrent_requests: 3 # Override for this dataset
      requests_per_minute_limit: 60

Use rate control when the upstream API enforces request quotas, when many datasets share a single origin, or when running large IN-list refreshes that would otherwise burst hundreds of concurrent requests.

Rate control is adaptive. While the origin's error rate, counting 408, 429, and 5xx responses, connection failures, and timeouts, stays at or below rate_control_failure_threshold, the configured limits apply unchanged. Above it, Spice admits fewer requests than the configured limits and logs a warning naming the origin; as the origin recovers, Spice returns to the full limits and logs that the origin recovered. The rate_control_adaptive_admission_ratio metric shows the share of the configured limits currently admitted. A request that cannot get capacity within rate_control_acquire_timeout fails with Timed out after ... waiting for rate-control capacity. See HTTP Rate Control (opens in a new tab) for details.

Retry Behavior

For a dynamic JSON API endpoint, 5xx and 429 responses and transient network errors are retried. A 408 is not retried, but adaptive rate control counts it as a failure. A request is retried at most max_retries times after the first attempt, so max_retries: 2 sends at most three requests. The connector respects Retry-After, retry-after-ms, and x-retry-after-ms headers.

Retries are independent of rate control. If a retry would exceed the configured per-second or per-minute rate, it waits for the rate window to open before issuing the request, for at most rate_control_acquire_timeout.

Timeouts and Connection Pool

Increase client_timeout for endpoints with large response bodies or expensive server-side computation. Reduce pool_max_idle_per_host when running many small datasets against the same host to keep the runtime's open file descriptors bounded.

Caching Mode

When using refresh_mode: caching, a 5xx or 429 that is still failing after retries fails the fetch and nothing is written to the cache. The error propagates to clients unless caching_stale_if_error serves the expired cached entry instead. See Error Responses (opens in a new tab). Set caching_stale_if_error to serve expired cached data on upstream failure — prefer a duration (caching_stale_if_error: 600s), which bounds both how stale a served entry may be and how long it is retained; enabled is the unbounded form. Always set caching_ttl explicitly — the default of 30s is rarely the desired window.

Set caching_max_size (a byte budget, e.g. 512MiB) or caching_max_items (a row budget) to bound the acceleration. A TTL alone does not: a workload that keeps fetching new request paths grows it indefinitely, and with caching_stale_if_error: enabled expired entries are deliberately kept as fallback material and are never expired away (a duration value instead derives an eviction deadline, so it does not have this effect). The runtime warns at startup, naming the dataset, when a caching accelerator has nothing bounding it. The eviction sweep runs at caching_ttl, clamped to 30s–5m, so the acceleration may overshoot its budget by whatever the workload writes between sweeps. See Cache Size and Item Limits (opens in a new tab).

Capacity & Sizing

• Throughput: Bounded by the upstream rate limit, then by max_concurrent_requests and connect_timeout. Plan limits to stay within the API quota.
• Memory: Response bodies are streamed; memory footprint is bounded by max_request_body_bytes (filter inputs) and DataFusion's record-batch size for response rows.
• Response cache: Each dynamic JSON API dataset holds its own response cache (opens in a new tab), bounded by response_cache_max_size_bytes — 67108864 (64 MiB) by default, per dataset. The runtime's total exposure therefore scales with the number of HTTP datasets, not with the budget alone: budget it as datasets × response_cache_max_size_bytes and raise the value only for the datasets that earn it. Set 0 on a dataset whose responses are never repeated. This cache is not one of the caches under runtime.caching, so its memory is not counted against those limits.
• Connection setup: TLS handshake adds latency. The connection pool keeps pool_max_idle_per_host warm connections to absorb burst traffic.
• Partitioned refreshes: When using IN-list filters or cross-product partitioning, the runtime issues one HTTP request per partition. Use max_request_partitions to cap the request count for unbounded filter combinations, and max_concurrent_requests to throttle their fan-out.

Metrics

The connector reports two metric families: response-cache occupancy, and per-origin rate control. Both are registered automatically — no metrics configuration is required.

Response cache

Occupancy of the dataset's response cache (opens in a new tab). These are reported for every HTTP dataset, because the memory the cache holds is otherwise attributable to nothing; structured file-format datasets do not use the cache and report 0.

Both are refreshed when a request consults the cache, so an idle dataset reports its last observed occupancy — which is the same figure, since nothing enters or leaves the cache except on a request.

Rate control

Per-origin rate-control metrics, exposed for dynamic JSON API datasets. The limit gauges report 0 when the corresponding limit is not configured. Structured file-format datasets (parquet, csv, and the other listing-table formats) do not expose them:

These metrics are auto-registered — no configuration is required to export them. To turn one off for a dataset, set enabled: false in the dataset's metrics section:

datasets:
  - from: https://api.example.com/v1
    name: api_data
    params:
      file_format: json
    metrics:
      - name: rate_control_wait_duration_ms
        enabled: false

datasets:
  - from: https://api.example.com/v1
    name: api_data
    params:
      file_format: json
    metrics:
      - name: rate_control_wait_duration_ms
        enabled: false

Instruments from both families are exposed with the prefix dataset_http_ — the HTTP connector's component name is http, not https — so the exported names are dataset_http_response_cache_size_bytes, dataset_http_rate_control_wait_duration_ms, and so on. The two families are attributed differently: the rate-control instruments carry an origin attribute (scheme://host:port) identifying the upstream origin instead of a dataset name, because datasets sharing an origin share one rate controller, while the response-cache gauges carry the dataset name, because each dataset has its own cache. See Component Metrics (opens in a new tab) for general configuration.

For broader observability, also monitor:

• Spice query execution metrics (query_duration_ms, query_returned_rows, query_failures) from runtime.metrics.

Task History

HTTP requests participate in task history (opens in a new tab) through the HTTP client's span. Each partitioned request and each pagination page is a child of the enclosing sql_query or acceleration_refresh task.

Known Limitations

• Read-only: The connector is read-only. Only GET and POST (via request_body filters) are supported.
• Filter pushdown is opt-in: request_path, request_query, request_body, and request_headers filters require explicit allowlists or _filters: enabled parameters.
• OAuth2 OOS scope: The refresh-token and client-credentials grants are supported. The authorization-code and device-code flows are not exposed.
• OR across virtual filter columns: WHERE request_path = '/a' OR request_query = 'b=1' is rejected. Use separate datasets or UNION ALL for cross-column alternatives. Single-column OR (and IN-lists) is supported.

Troubleshooting