spiceai/docs

title: 'HTTP(s) Data Connector' sidebar_label: 'HTTP(s) Data Connector' description: 'HTTP(s) Data Connector Documentation' pagination_prev: null

The HTTP(s) Data Connector enables federated SQL query across supported file formats (opens in a new tab) stored at an HTTP(s) endpoint.

datasets:
  - from: http://static_username@localhost:3001/report.csv
    name: local_report
    params:
      http_password: ${env:MY_HTTP_PASS}

datasets:
  - from: http://static_username@localhost:3001/report.csv
    name: local_report
    params:
      http_password: ${env:MY_HTTP_PASS}

Configuration

`from`

The from field must contain a valid URI to the location of a supported file (opens in a new tab). For example, http://static_username@localhost:3001/report.csv.

`name`

The dataset name. This will be used as the table name within Spice.

Example:

datasets:
  - from: http://static_username@localhost:3001/report.csv
    name: cool_dataset
    params: ...

datasets:
  - from: http://static_username@localhost:3001/report.csv
    name: cool_dataset
    params: ...

SELECT COUNT(*) FROM cool_dataset;

SELECT COUNT(*) FROM cool_dataset;

+----------+
| count(*) |
+----------+
| 6001215  |
+----------+

+----------+
| count(*) |
+----------+
| 6001215  |
+----------+

The dataset name cannot be a reserved keyword (opens in a new tab).

`params`

The connector supports Basic HTTP authentication via param values.

Parameter Name	Description
`http_port`	Optional. Port to create HTTP(s) connection over. Default: 80 and 443 for HTTP and HTTPS respectively.
`http_username`	Optional. Username to provide connection for HTTP basic authentication. Default: None.
`http_password`	Optional. Password to provide connection for HTTP basic authentication. Default: None. Use the secret replacement syntax (opens in a new tab) to load the password from a secret store, e.g. `${secrets:my_http_pass}`.
`client_timeout`	Optional. Specifies timeout for HTTP operations. Default value is `30s` E.g. `client_timeout: 60s`

Examples

Basic example

datasets:
  - from: https://github.com/LAION-AI/audio-dataset/raw/7fd6ae3cfd7cde619f6bed817da7aa2202a5bc28/metadata/freesound/parquet/freesound_parquet.parquet
    name: laion_freesound

datasets:
  - from: https://github.com/LAION-AI/audio-dataset/raw/7fd6ae3cfd7cde619f6bed817da7aa2202a5bc28/metadata/freesound/parquet/freesound_parquet.parquet
    name: laion_freesound

Using Basic Authentication

datasets:
  - from: http://static_username@localhost:3001/report.csv
    name: local_report
    params:
      http_password: ${env:MY_HTTP_PASS}

datasets:
  - from: http://static_username@localhost:3001/report.csv
    name: local_report
    params:
      http_password: ${env:MY_HTTP_PASS}

Secrets

Spice integrates with multiple secret stores to help manage sensitive data securely. For detailed information on supported secret stores, refer to the secret stores documentation (opens in a new tab). Additionally, learn how to use referenced secrets in component parameters by visiting the using referenced secrets guide (opens in a new tab).

spiceai/docs/README.md