Connecting Data

Overview

Spice.ai supports federated SQL queries across a wide range of data sources. Data connectors let you query external data directly or replicate it into Spice for accelerated access.

For the full list of connectors and configuration options, see Data Connectors.

Supported Data Sources

Adding a Data Source

1. Store credentials as secrets

Data source credentials should be stored as secrets in your project. In the Portal, go to your project → Secrets and create entries for connection strings, access keys, or tokens.

2. Configure the dataset in your Spicepod

Add a dataset entry to your project's spicepod.yaml referencing the connector and secrets:

datasets:
  - from: postgres:my_schema.my_table
    name: my_table
    params:
      pg_host: my-database.example.com
      pg_port: "5432"
      pg_db: my_database
      pg_user: "${secrets:PG_USER}"
      pg_pass: "${secrets:PG_PASS}"

3. Deploy

Commit and deploy your project to make the dataset available for querying.

Data Acceleration

For faster queries, enable data acceleration to cache datasets in a local engine:

datasets:
  - from: postgres:my_schema.my_table
    name: my_table
    acceleration:
      engine: arrow  # or duckdb, sqlite, postgres
      refresh_interval: 10m

Available acceleration engines:

• Arrow — In-memory, fastest performance.
• DuckDB — Embedded analytical engine.
• SQLite — Lightweight, persistent.
• PostgreSQL — Full-featured relational engine.

Common Issues

Table not found

Spice normalizes unquoted SQL identifiers to lowercase (PostgreSQL-style). If your source uses mixed-case table or column names, double-quote them:

SELECT "ColumnName" FROM "MySchema"."MyTable"

Object store file format errors

When connecting to object stores (S3, ABFS):

• Folder paths require an explicit file_format (e.g., parquet, csv).
• Single file paths auto-detect the format from the file extension.

Schema changes break data refresh

Spice infers the schema at deployment time. If the source schema changes, refresh will fail. Redeploy the project to pick up the new schema.

Connection failures

• Confirm the data source is network-accessible from Spice.ai Cloud (firewalls, IP allowlists).
• Verify secrets are correct and referenced with ${secrets:SECRET_NAME} syntax.
• Check connector-specific docs for required parameters.

Further Reading

• Data Connectors reference
• Data Acceleration
• Federated SQL Query
• Catalogs