Distributed SQL query execution across a cluster
Distributed query in Spice splits one SQL query into stages and runs them in parallel across executor nodes. A scheduler built on Apache Ballista and DataFusion plans the work. It is designed for scans of large, partitioned data lake tables that exceed the memory or compute of a single node, and a failed task is retried without restarting the whole query.
Scale query compute independently of storage
Add executors to a cluster to increase query capacity, while the data stays in object storage and applications keep the same SQL endpoint.
Up to 100x
up to 100x faster queries
Up to 80%
up to 80% cost savings on data lakehouse spend
2x
increase in data reliability for critical workloads
Declarative cluster configuration
Define the cluster in one Spicepod on the scheduler. Executors start with the scheduler address and their node certificates, then fetch the configuration, catalog, UDFs, and secrets from it. Each executor reads its own share of the data from the source. No external orchestrator, etcd, or ZooKeeper is required.
View the docs
Parallel execution engine
Built on Apache Ballista and DataFusion, Spice splits a query into stages and dispatches them as tasks across executors for concurrent reads, joins, and aggregations. Each executor processes its partitions independently, and results stream back through the scheduler as Arrow record batches.
View the docs
Automatic dependency management
Executors receive the catalog and UDF registry from the scheduler when they join. Each executor also loads the models from the Spicepod, so a query can call embed() on any executor. You define models once, on the scheduler.
View the docs
Optimized query planning
A custom optimizer decomposes large scans into shuffle partitions and pushes projections closer to the data source. Executors move less data, so queries finish faster and use less network.
View the docs
Integrations across all your data sources
Distributed query scans partitioned Parquet, Delta Lake, and Iceberg tables in object storage, plus HTTP datasets. In the same Spicepod, 40+ prebuilt connectors federate databases, warehouses, and file stores, from Databricks and S3 to MySQL and PostgreSQL.
Deployed in production
Run data-intensive workloads on a high-performance engine trusted by teams building real-time systems at scale.
“Spice opened the door to take these critical control-plane datasets and move them next to our services in the runtime path.”
Peter Janovsky
Software Architect, Twilio
100x
Faster queries
“It just spins up and works, which is really nice. The responsiveness is amazing, which is a huge gain for the customer.”
Darin Douglass
Principal Software Engineer, Barracuda
Learn how Barracuda gained 100x faster query responses and reduced costs 50% with Spice
“Partnering with Spice AI has transformed how NRC Health delivers AI-driven insights. By unifying siloed data across systems, we accelerated AI feature development, reducing time-to-market from months to weeks - and sometimes days. With predictable costs and faster innovation, Spice isn't just solving some of our data and AI challenges - it's helping us redefine personalized healthcare.”
Tim Ottersburg
VP of Technology, NRC Health
FAQs
Answers to common questions about running distributed SQL queries in Spice.
Yes. The open-source Spice runtime includes distributed query as a preview feature, introduced in v1.9.0. In open source, a cluster distributes scans of partitioned data lake tables and HTTP datasets. Spice.ai Enterprise adds multi-active schedulers and distributed accelerations, generally available since v2.0.
No. You start one scheduler with a Spicepod, then start executors with the scheduler address. Executors register with the scheduler and get the configuration from it. In Spice.ai Enterprise, multiple active schedulers share state through an object store, such as Amazon S3, with no etcd or ZooKeeper. Kubernetes is optional.
In open source, distributed query reads partitioned data lake tables, such as Parquet, Delta Lake, and Iceberg in object storage. It also reads HTTP and HTTPS datasets without acceleration. Each executor scans its share of the data with its own client and credentials. Other connectors work in the same Spicepod, but their queries do not spread across executors.
Run EXPLAIN on the query against the scheduler endpoint. When distributed planning is active, the output has a distributed_plan section. That section shows how the scheduler splits the query into stages across executors.
Only an unauthenticated async query resumes on another scheduler, from its saved execution graph. This needs a shared state location in runtime.state.location. An async job from an authenticated principal fails with SCHEDULER_UNAVAILABLE, and the client resubmits it. A synchronous query fails, and the client retries it on another scheduler.
A distributed Spice cluster runs the same spiced binary in two roles. A scheduler accepts SQL, plans the query, and assigns work. Executors run the physical plan and exchange intermediate data through a shuffle. Spice builds both roles on Apache Ballista, the distributed execution layer for Apache DataFusion. A query moves through five steps:
- Submit: a client sends SQL to a scheduler over HTTP, Arrow Flight, or Flight SQL. These are the same endpoints that a single Spice instance exposes.
- Plan: the scheduler builds a DataFusion logical plan. It then builds a distributed physical plan that splits the query into stages at each shuffle. The optimizer pushes projections to the scan, so executors read only the columns and partitions that the query needs.
- Dispatch: the scheduler sends stage tasks to executors. Each executor registers with the scheduler and stays connected over a control stream.
- Execute: each executor scans its partitions and writes intermediate shuffle data to local disk, memory, or object storage. The next stage reads that data. The scheduler retries a failed task without a restart of the whole query.
- Return: results stream back through the scheduler to the client as Arrow record batches.
Only the scheduler needs a spicepod.yaml. Executors start with the scheduler address and their node certificates. They get the configuration, catalog, and secrets from the scheduler, without its runtime.auth and runtime.tls settings. Each executor reads data from the source with its own client. Each executor also loads the Spicepod models when it joins, so a query can call embed() on any executor.
For local development, create the certificates first:
spice cluster tls init
spice cluster tls add scheduler1 --host scheduler1
spice cluster tls add executor1 --host executor1Then start the scheduler and an executor. Each node must set --node-advertise-address, the address that other nodes use to reach it:
# Scheduler: reads spicepod.yaml from the current directory
spiced --role scheduler \
--flight 0.0.0.0:50051 \
--node-advertise-address scheduler1 \
--node-mtls-ca-certificate-file ~/.spice/pki/ca.crt \
--node-mtls-certificate-file ~/.spice/pki/scheduler1.crt \
--node-mtls-key-file ~/.spice/pki/scheduler1.key
# Executor: needs no spicepod.yaml
spiced --role executor \
--scheduler-address https://scheduler1:50052 \
--node-advertise-address executor1 \
--node-mtls-ca-certificate-file ~/.spice/pki/ca.crt \
--node-mtls-certificate-file ~/.spice/pki/executor1.crt \
--node-mtls-key-file ~/.spice/pki/executor1.keyRead the distributed query documentation for the full flag and configuration reference.
Distributed mode is built for throughput on large analytical queries that take seconds to minutes. Sub-second application queries stay on single-node Spice with SQL federation and acceleration. The three deployment options compare as follows:
| Deployment | Where query compute runs | Data it scales | Best fit | Availability |
|---|---|---|---|---|
| Single-node Spice | One spiced process | Federated sources and local accelerations that fit on one node | Low-latency application and agent queries, sidecar deployments | Open source |
| Distributed query | A scheduler plus an executor fleet | Partitioned Parquet, Delta Lake, and Iceberg tables, and HTTP datasets | Large data lake scans, joins, aggregations, and batch embed() calls | Open source (preview), and Spice.ai Enterprise |
| Distributed accelerations | Executors that each own a set of partitions | Accelerated datasets sharded by partition_by across executors | Accelerated datasets larger than one node's memory or disk | Spice.ai Enterprise |
The two modes combine in a cluster-sidecar architecture. A cluster processes the large scans, and sidecars next to each application serve a smaller accelerated working set. For a partitioned lakehouse workload end to end, see the data lake accelerator.
Spice ran an early preview test in October 2025. A 48-core Spice cluster with 24 GB of RAM ran three queries. Apache Spark on Databricks ran the same queries with 48 cores and 192 GB of RAM. The Spice cluster finished each query faster:
| Query | Spice cluster | Spark on Databricks | Difference |
|---|---|---|---|
| Wildcard filter, no matches | 6.1 s | 47 s | 7.7x faster |
| Wildcard filter, with matches | 6.6 s | 25 s | 3.8x faster |
embed() over about 100,000 rows | 9.8 s | 17 s | 1.7x faster |
The cluster used one-eighth of the RAM. Spice is written in Rust and moves data as Apache Arrow over Arrow Flight, so it has no JVM garbage collection. These are three queries on one configuration, not a general benchmark. The Apache Ballista engineering post describes the setup and the extensions that Spice added to Ballista.
A cluster adds operational requirements that a single node does not have. Plan for these before you deploy one:
- Mutual TLS between nodes: scheduler-executor traffic uses port 50052, and every cluster node requires mTLS by default. This traffic carries the Spicepod, expanded secrets, and shuffle data. Each node needs a CA certificate, a node certificate with its advertise address in the SANs, and a private key. Use
spice cluster tlscertificates for development only. Production clusters use certificates from the org's PKI. - One Spice version across the cluster: schedulers and executors must run the same version, so upgrade all nodes together.
- Shared state for async queries: the async queries API (
/v1/queries) persists job state toruntime.state.location. This location is an object store or filesystem that supports conditional writes, such as Amazon S3, Azure Blob Storage, or Google Cloud Storage. Clients submit a job, poll its status, and fetch paginated results. - High availability: a synchronous query does not survive the loss of its scheduler. Multiple active schedulers behind a load balancer are a Spice.ai Enterprise feature.
- Preview status: open-source distributed query is a preview feature. Clusters can hit stability or performance issues that a single node does not.
To run a managed cluster or distributed accelerations, compare Spice Cloud and Enterprise plans.
See Spice in action
Walk through your use case with an engineer and see how Spice handles federation, acceleration, and AI integration for production workloads.
Talk to an engineer (opens in a new tab)