Query across sources
without ETL

Spice joins data from operational databases, warehouses, data lakes, and APIs in one SQL query. Connect a new source without building another pipeline.

Diagram of apps, AI agents, and dashboards querying Spice federation, with an accelerator, over a data store of Databricks, PostgreSQL, Snowflake, and Amazon S3

Enterprise AI apps need data across many systems

With traditional data systems, every new app, feature, or dashboard requires another ETL pipeline to build and maintain. Each answer is only as current as the last pipeline run. Agents and latency-sensitive apps need current results from every system, fast.

Query every source through a single SQL endpoint

Spice queries each source where it lives, and materializes the hot-path data next to the app.

Federated SQL

Query and join data where it lives

Join data from databases, warehouses, data lakes, and APIs in one SQL query. Spice plans each query with Apache DataFusion, sending filters and other supported operations to each source, and then joins the results.

Read the query federation docs (opens in a new tab)

Query and join data where it lives: an agent sends one SQL query to Spice. Spice reads MySQL and Databricks over SQL, Parquet files in Amazon S3, and an HTTP API, then joins the rows into one answer, with no data copied ahead of timeAgentquery{ }{ }MySQLDatabaseDatabricksWarehouseAmazon S3Data lakeHTTP APIWeb serviceSQLSQLParquetHTTPOne SQL query reaches every kind of source.Spice reads each source through its own interface.Each answer becomes rows in one SQL result.One joined answer, with no data copied ahead of time.
Local acceleration

Fast answers on the hot path

Accelerate frequently queried datasets locally with Spice Cayenne. Queries read the local acceleration, so they do not wait on the source.

Read the data acceleration docs (opens in a new tab)

Infrastructure simplification

Replace separate engines, ETL jobs, and caches

Replace separate query engines, ETL jobs, and custom caches with one runtime. The same runtime handles federation, acceleration, and hybrid search.

Read the features docs (opens in a new tab)

CDC and refresh

Keep local accelerations current

Reload a small table on a schedule, append new rows to a log table, or stream each change from an orders table over CDC. Each local acceleration stays current with its source.

Read the data refresh docs (opens in a new tab)

Query across sources in minutes

Join Parquet in S3 with a PostgreSQL table in one federated query.

# spicepod.yamldatasets:  - from: s3://spiceai-demo-datasets/taxi_trips/2024/    name: taxi_trips    params: { file_format: parquet, s3_auth: public }  - from: postgres:taxi_zones    name: taxi_zones    params:      pg_host: localhost      pg_port: 15432      pg_db: taxi      pg_user: postgres      pg_pass: ${secrets:SPICE_PG_PASS}      pg_sslmode: disablesql> SELECT z.zone, COUNT(*) AS trips     FROM taxi_trips t                 -- S3     JOIN taxi_zones z                 -- PostgreSQL       ON t."PULocationID" = z.location_id     GROUP BY z.zone ORDER BY trips DESC LIMIT 3;JFK Airport                 145240Midtown Center              143471Upper East Side South       142708Time: 2.705 seconds
  1. Connect your sources

    In a spicepod.yaml, the Spice configuration file, list each source table. Spice queries each source where it lives.

  2. Accelerate the tables you read most

    Enable acceleration on a dataset, with no ETL job. Set refresh_check_interval to refresh it on a schedule. Without an interval, Spice runs no scheduled refresh.

  3. Deploy and query

    Start Spice with spice run, or deploy it as a sidecar, at the edge, in your cluster, or on Spice Cloud. Then query from agents, dashboards, or your own apps.

  1. /plugin marketplace add spiceai/skills
  2. /plugin install spiceai@spiceai

Proven in production

Teams that build agents and apps on data from many systems run Spice as one SQL layer.

Ramachandra Ramarathinam

“Spice is the data plane behind Summation. Every connector we ship, from Snowflake to a generic REST-as-a-table, collapses into one SQL surface, which is what makes our AI agents portable across a customer's stack.”

Ramachandra Ramarathinam

CTO, Summation

Peter Janovsky

“Spice opened the door to take these critical control-plane datasets and move them next to our services in the runtime path.”

Peter Janovsky

Software Architect, Twilio

Darin Douglass

100x

Faster queries

“It just spins up and works, which is really nice. The responsiveness is amazing, which is a huge gain for the customer.”

FAQs

Answers for teams that query data across many systems.

Define each table as a dataset in spicepod.yaml, and then join the datasets by name in one SQL query. Spice plans the query with Apache DataFusion and sends each source the filters that its connector supports. It then joins the results in the runtime. For more, read SQL federation and acceleration.

Accelerate the datasets that agents and apps read often. Also accelerate sources that are slow or that charge per query. Leave large tables that queries read rarely as federated. A federated dataset reads its source at query time, apart from a short-lived results cache.

Each dataset in the query is as current as its own path. A federated dataset reads the source at query time, or returns a recent result from the SQL results cache. An accelerated dataset is as current as its last refresh. With change data capture (refresh_mode: changes), an accelerated dataset applies each change from PostgreSQL, MySQL, MongoDB, or DynamoDB as it occurs.

A query on a federated dataset fails when its source is unavailable, unless the results cache holds its answer. After its first load, an accelerated dataset continues to serve its data while a refresh fails. Spice retries a failed refresh by default. For restarts during an outage, read serving an existing acceleration.

An ETL pipeline fits when you reshape data heavily before anyone queries it. It also fits a warehouse that keeps years of history, or a report that runs once a day. Spice fits queries that need current data from many systems. Many teams use both.