# HTTP(s) Data Connector Deployment Guide

Operating guide for the HTTP(s) data connector in production: authentication, rate control, retries, and observability.

Production operating guide for the HTTP(s) data connector covering authentication, rate control, retry tuning, and observability.

## Authentication & Secrets

The connector supports HTTP Basic, custom-header, and OAuth2 (refresh-token and client-credentials grants) authentication. Secrets must be sourced from a [secret store](https://spiceai.org/docs/components/secret-stores) in production.

| Parameter | Description |
| - | - |
| `http_username` | Username for HTTP Basic authentication. |
| `http_password` | Password for HTTP Basic authentication. Use `${secrets:...}` to resolve from a secret store. |
| `http_headers` | Custom headers (e.g. `Authorization:Bearer ${secrets:api_token}`). Treated as sensitive: not logged. Dynamic JSON API endpoints only; structured HTTP file datasets ignore these headers. |
| `auth_token_url` | OAuth2 token endpoint URL (must be HTTPS in production). |
| `auth_grant_type` | OAuth2 grant: `refresh_token` (default) or `client_credentials`. |
| `http_auth_refresh_token` | OAuth2 refresh token. Required for the (default) refresh-token grant; unused by `client_credentials`. |
| `http_auth_client_id` | OAuth2 client ID (required for confidential clients and for `client_credentials`). |
| `http_auth_client_secret` | OAuth2 client secret (required for confidential clients and for `client_credentials`). Use `${secrets:...}`. |
| `auth_header_name` | Header carrying the access token. Default `Authorization` (`Bearer <token>`); any other name sends the bare token. |

For OAuth2-protected APIs, prefer refresh-token flow over storing long-lived bearer tokens. The connector exchanges the refresh token for short-lived access tokens at startup and refreshes them before expiry.

### TLS

Use HTTPS endpoints in production. `auth_token_url` must use HTTPS (loopback addresses are allowed for local testing only). Self-signed certificates require a trusted CA bundle in the container or host OS trust store.

For upstream servers that require mutual TLS (mTLS), the connector can present a client certificate during the TLS handshake. Supply the certificate and key as file paths or inline PEM: the two forms are mutually exclusive, and the certificate and key must be set together. mTLS client identity applies to dynamic JSON API endpoints only.

| Parameter | Description |
| - | - |
| `http_tls_client_certificate_file` | Path to a PEM client certificate chain. Pair with `http_tls_client_key_file`. |
| `http_tls_client_key_file` | Path to the PEM private key matching the client certificate file. |
| `http_tls_client_certificate` | Inline PEM client certificate chain. Use `${secrets:...}`. Pair with `http_tls_client_key`. |
| `http_tls_client_key` | Inline PEM private key matching the inline certificate. Use `${secrets:...}`. |

## Resilience Controls

### Rate Control

The HTTP connector participates in the shared HTTP rate control system. Concurrency and per-second/per-minute request limits can be configured per-dataset (in `params`) or globally (in `runtime.params`). Dataset-level settings override the global defaults. Multiple datasets targeting the same upstream origin share a single rate controller.

| Parameter | Description |
| - | - |
| `max_concurrent_requests` | Maximum concurrent HTTP requests to the same origin. Disabled when unset. |
| `requests_per_second_limit` | Maximum HTTP requests per second to the same origin. Disabled when unset. |
| `requests_per_minute_limit` | Maximum HTTP requests per minute to the same origin. Disabled when unset. |
| `rate_control_jitter_min` | Minimum random delay before requests when rate control is active. Defaults to `5ms`. |
| `rate_control_jitter_max` | Maximum random delay before requests when rate control is active. Defaults to `10ms`. |

The runtime equivalents (`http_max_concurrent_requests`, `http_requests_per_second_limit`, `http_requests_per_minute_limit`, `http_rate_control_jitter_min`, `http_rate_control_jitter_max`) set defaults that apply to every HTTP-based connector unless overridden per dataset.

```yaml
runtime:
  params:
    http_max_concurrent_requests: 10
    http_requests_per_second_limit: 5

datasets:
  - from: https://api.example.com/v1
    name: api_data
    params:
      file_format: json
      allowed_request_paths: '/data/**'
      max_concurrent_requests: 3 # Override for this dataset
      requests_per_minute_limit: 60
```

Use rate control when the upstream API enforces request quotas, when many datasets share a single origin, or when running large `IN`-list refreshes that would otherwise burst hundreds of concurrent requests.

### Retry Behavior

HTTP-level retries follow the shared `resilient_http` policy: 408, 429, and 5xx responses plus transient network errors are retried. The connector respects `Retry-After`, `retry-after-ms`, and `x-retry-after-ms` headers.

| Parameter | Default | Description |
| - | - | - |
| `max_retries` | `3` | Maximum retry attempts per request. |
| `retry_backoff_method` | `fibonacci` | Backoff strategy. Options: `fibonacci`, `linear`, `exponential`. |
| `retry_max_duration` | unset | Maximum total duration across all retries (e.g. `30s`, `5m`). When set, retries stop after this elapsed time. |
| `retry_jitter` | `0.3` | Randomization factor (`0.0` to `1.0`) applied to retry delays. Set to `0` to disable jitter. |

Retries are independent of rate control. If a retry would exceed the configured per-second or per-minute rate, it waits for the rate window to open before issuing the request.

### Timeouts and Connection Pool

| Parameter | Default | Description |
| - | - | - |
| `client_timeout` | `30` | Maximum time (seconds) to wait for the entire request-response cycle. |
| `connect_timeout` | `10` | Maximum time (seconds) to establish a TCP/TLS connection. |
| `pool_max_idle_per_host` | `10` | Maximum idle connections held per upstream host. |
| `pool_idle_timeout` | `90` | Idle connection lifetime (seconds) before the pool closes them. |

Increase `client_timeout` for endpoints with large response bodies or expensive server-side computation. Reduce `pool_max_idle_per_host` when running many small datasets against the same host to keep the runtime's open file descriptors bounded.

### Caching Mode

When using `refresh_mode: caching`, transient HTTP errors (5xx, 429) are excluded from the cache and propagated to clients. Set `caching_stale_if_error: enabled` to serve expired cached data on upstream failure. Always set `caching_ttl` explicitly: the default of `30s` is rarely the desired window.

Set `caching_max_size` (a byte budget, e.g. `512MiB`) or `caching_max_items` (a row budget) to bound the acceleration. A TTL alone does not: a workload that keeps fetching new request paths grows it indefinitely, and with `caching_stale_if_error: enabled` expired entries are deliberately kept as fallback material and are never expired away. The runtime warns at startup, naming the dataset, when a caching accelerator has nothing bounding it. The eviction sweep runs at `caching_ttl`, clamped to 30s: 5m, so the acceleration may overshoot its budget by whatever the workload writes between sweeps. See [Cache Size and Item Limits](https://spiceai.org/docs/features/data-acceleration/refresh-modes/caching#cache-size-and-item-limits).

## Capacity & Sizing

- **Throughput**: Bounded by the upstream rate limit, then by `max_concurrent_requests` and `connect_timeout`. Plan limits to stay within the API quota.
- **Memory**: Response bodies are streamed; memory footprint is bounded by `max_request_body_bytes` (filter inputs) and DataFusion's record-batch size for response rows.
- **Response cache**: Each dynamic JSON API dataset holds its own [response cache](https://spice.ai/docs/building-blocks/data-connectors/https#response-cache), bounded by `response_cache_max_size_bytes`: `67108864` (64 MiB) by default, **per dataset**. The runtime's total exposure therefore scales with the number of HTTP datasets, not with the budget alone: budget it as `datasets × response_cache_max_size_bytes` and raise the value only for the datasets that earn it. Set `0` on a dataset whose responses are never repeated. This cache is not one of the caches under `runtime.caching`, so its memory is not counted against those limits.
- **Connection setup**: TLS handshake adds latency. The connection pool keeps `pool_max_idle_per_host` warm connections to absorb burst traffic.
- **Partitioned refreshes**: When using `IN`-list filters or cross-product partitioning, the runtime issues one HTTP request per partition. Use `max_request_partitions` to cap the request count for unbounded filter combinations, and `max_concurrent_requests` to throttle their fan-out.

## Metrics

The connector reports two metric families: response-cache occupancy, and per-origin rate control. Both are registered automatically: no `metrics` configuration is required.

### Response cache

Occupancy of the dataset's [response cache](https://spice.ai/docs/building-blocks/data-connectors/https#response-cache). These are reported for every HTTP dataset, because the memory the cache holds is otherwise attributable to nothing; structured file-format datasets do not use the cache and report `0`.

| Metric Name | Type | Description |
| - | - | - |
| `response_cache_size_bytes` | Gauge | Bytes retained by the response cache, counting response bodies, their headers, and the request keys they are held under. Excludes the cache's own per-entry bookkeeping. Compare against `response_cache_max_size_bytes` to see how close a dataset is to its budget. |
| `response_cache_items_count` | Gauge | Number of responses held by the response cache. Read beside the byte figure, this separates a cache holding a few large responses from one holding very many small ones. |

Both are refreshed when a request consults the cache, so an idle dataset reports its last observed occupancy, which is the same figure, since nothing enters or leaves the cache except on a request.

### Rate control

Per-origin rate-control metrics, exposed for dynamic JSON API datasets. The limit gauges report `0` when the corresponding limit is not configured. Structured file-format datasets (`parquet`, `csv`, and the other listing-table formats) do not expose them:

| Metric Name | Type | Description |
| - | - | - |
| `inflight_operations` | Gauge | Current number of HTTP requests holding a rate-control permit. |
| `rate_control_max_concurrent_requests` | Gauge | Configured maximum concurrent HTTP requests for this upstream origin; `0` means disabled. |
| `rate_control_requests_per_second_limit` | Gauge | Configured HTTP request-per-second limit for this upstream origin; `0` means disabled. |
| `rate_control_requests_per_minute_limit` | Gauge | Configured HTTP request-per-minute limit for this upstream origin; `0` means disabled. |
| `rate_control_jitter_min_ms` | Gauge | Configured minimum rate-control jitter (ms) before HTTP requests. |
| `rate_control_jitter_max_ms` | Gauge | Configured maximum rate-control jitter (ms) before HTTP requests. |
| `rate_control_available_permits` | Gauge | Current available permits in the HTTP request concurrency semaphore; `0` when concurrency is disabled. |
| `rate_control_acquisitions_total` | Counter | Total HTTP request rate-control permits acquired. |
| `rate_control_acquire_errors_total` | Counter | Total HTTP request rate-control permit acquisition errors. |
| `rate_control_wait_duration_ms` | Counter | Cumulative time (ms) spent waiting for HTTP rate-control permits, quotas, and jitter. |
| `rate_limit_retry_after_updates_total` | Counter | Total upstream cooldown hints accepted from `Retry-After` or `RateLimit` reset headers. |
| `rate_limit_retry_after_waits_total` | Counter | Total waits caused by `Retry-After` or `RateLimit` reset headers. |
| `rate_limit_retry_after_wait_duration_ms` | Counter | Cumulative time (ms) spent waiting because of `Retry-After` or `RateLimit` reset headers. |
| `rate_limit_retry_after_remaining_ms` | Gauge | Current remaining `Retry-After` / `RateLimit` cooldown (ms) for this upstream origin. |

These metrics are auto-registered: no configuration is required to export them. To turn one off for a dataset, set `enabled: false` in the dataset's `metrics` section:

```yaml
datasets:
  - from: https://api.example.com/v1
    name: api_data
    params:
      file_format: json
    metrics:
      - name: rate_control_wait_duration_ms
        enabled: false
```

Instruments from both families are exposed with the prefix `dataset_http_` (the HTTP connector's component name is `http`, not `https`) so the exported names are `dataset_http_response_cache_size_bytes`, `dataset_http_rate_control_wait_duration_ms`, and so on. The two families are attributed differently: the rate-control instruments carry an `origin` attribute (`scheme://host:port`) identifying the upstream origin instead of a dataset `name`, because datasets sharing an origin share one rate controller, while the response-cache gauges carry the dataset `name`, because each dataset has its own cache. See [Component Metrics](https://spiceai.org/docs/features/observability/component_metrics) for general configuration.

For broader observability, also monitor:

- Spice query execution metrics (`query_duration_ms`, `query_returned_rows`, `query_failures`) from `runtime.metrics`.

## Task History

HTTP requests participate in [task history](https://spiceai.org/docs/reference/task_history) through the HTTP client's span. Each partitioned request and each pagination page is a child of the enclosing `sql_query` or `acceleration_refresh` task.

## Known Limitations

- **Read-only**: The connector is read-only. Only `GET` and `POST` (via `request_body` filters) are supported.
- **Filter pushdown is opt-in**: `request_path`, `request_query`, `request_body`, and `request_headers` filters require explicit allowlists or `_filters: enabled` parameters.
- **OAuth2 OOS scope**: The refresh-token and client-credentials grants are supported. The authorization-code and device-code flows are not exposed.
- **OR across virtual filter columns**: `WHERE request_path = '/a' OR request_query = 'b=1'` is rejected. Use separate datasets or `UNION ALL` for cross-column alternatives. Single-column `OR` (and `IN`-lists) is supported.

## Troubleshooting

| Symptom | Likely cause | Resolution |
| - | - | - |
| `401 Unauthorized` | Wrong/expired token or password. | Rotate the credential in the secret store. |
| `429 Too Many Requests` (frequent) | Upstream rate limit hit; concurrency too high. | Set `requests_per_second_limit` / `requests_per_minute_limit`; reduce `max_concurrent_requests`. |
| Refresh blocked / queue building up | `max_concurrent_requests` set too low for the workload. | Raise the dataset-level limit or move heavy datasets to their own origin. |
| OAuth2 token refresh fails | `auth_token_url` not HTTPS, or wrong client credentials. | Verify the token endpoint URL; check `http_auth_client_id`/`secret` and required scopes. |
| Request rejected: "OR across HTTP filter columns" | `WHERE request_path = '...' OR request_query = '...'`. | Split into separate refreshes or `UNION ALL`. |
| Many partitions created from cross-product | Multiple `IN`-list filters multiplied into many requests. | Set `max_request_partitions` to cap; tighten filters. |
| Slow first refresh | Cold connection pool + TLS handshake per request. | Raise `pool_max_idle_per_host`; ensure `pool_idle_timeout` is long enough to keep connections warm. |
| Runtime memory grows with HTTP traffic | Response caches are held per dataset, 64 MiB each by default. | Check `dataset_http_response_cache_size_bytes` per dataset; lower `response_cache_max_size_bytes`, or set it to `0` where responses are never repeated. |
| Repeat queries still hit the origin | The origin refuses retention (`no-store`, `no-cache`, `private`, `Vary: *`), or sends no `Cache-Control` at all. | Confirm with `dataset_http_response_cache_items_count` staying at `0`. For a header-less origin, set `response_cache_fallback_ttl`; an origin that refuses explicitly is always honored. |