Distributed SQL query execution across a cluster
Distributed query in Spice splits one SQL query into stages and runs them in parallel across executor nodes. A scheduler built on Apache Ballista and DataFusion plans the work. It is designed for scans of large, partitioned data lake tables that exceed the memory or compute of a single node, and a failed task is retried without restarting the whole query.
Scale query compute independently of storage
Add executors to a cluster to increase query capacity, while the data stays in object storage and applications keep the same SQL endpoint.
100x
up to 100x faster queries
80%
up to 80% cost savings on data lakehouse spend
2x
increase in data reliability for critical workloads
Declarative cluster configuration
Define the cluster in one Spicepod on the scheduler. Executors start with the scheduler address and their node certificates, then fetch the configuration, catalog, UDFs, and secrets from it. Each executor reads its own share of the data from the source. No external orchestrator, etcd, or ZooKeeper is required.
View the docs
Parallel execution engine
Built on Apache Ballista and DataFusion, Spice splits a query into stages and dispatches them as tasks across executors for concurrent reads, joins, and aggregations. Each executor processes its partitions independently, and results stream back through the scheduler as Arrow record batches.
View the docs
Automatic dependency management
Executors receive the catalog and UDF registry from the scheduler when they join. When a query calls a model, such as embed(), the planner tells each executor to load that model on demand, so models are defined once in the Spicepod.
View the docs
Optimized query planning
A custom optimizer decomposes large scans into shuffle partitions and pushes projections closer to the data source. Less data movement, faster results, and lower network cost.
View the docs
Integrations across all your data sources
Distributed query scans partitioned Parquet, Delta Lake, and Iceberg tables in object storage, plus HTTP datasets. In the same Spicepod, 40+ prebuilt connectors federate databases, warehouses, and file stores, from Databricks and S3 to MySQL and PostgreSQL.
How does distributed query work in Spice?
A distributed Spice cluster runs the same spiced binary in two roles. A scheduler accepts SQL, plans the query, and assigns work. Executors run the physical plan and exchange intermediate data through a shuffle. Spice builds both roles on Apache Ballista, the distributed execution layer for Apache DataFusion. A query moves through five steps:
- Submit: a client sends SQL to a scheduler over HTTP, Arrow Flight, or Flight SQL, the same endpoints a single Spice instance exposes.
- Plan: the scheduler builds a DataFusion logical plan, then a distributed physical plan that splits the query into stages at each shuffle.
- Dispatch: the scheduler pushes stage tasks to executors, which register with it and stay connected over a control stream.
- Execute: each executor scans its partitions and writes intermediate shuffle data to local disk, memory, or object storage. The next stage reads that data. A failed task is retried without restarting the whole query. If an executor is lost along with shuffle data on its disk or in its memory, the upstream stage reruns to rebuild that data. Shuffle data in object storage survives the loss of an executor.
- Return: results stream back through the scheduler to the client as Arrow record batches.
Only the scheduler needs a spicepod.yaml. Executors start with the scheduler's address and their node certificates, and fetch the configuration, catalog, and UDFs from it. Executors read data from the source with their own clients, and load a model only when a query calls it. For local development, create the certificates first:
spice cluster tls init
spice cluster tls add scheduler1
spice cluster tls add executor1Then start the scheduler and an executor:
# Scheduler: reads spicepod.yaml from the current directory
spiced --role scheduler \
--flight 0.0.0.0:50051 \
--node-mtls-ca-certificate-file ~/.spice/pki/ca.crt \
--node-mtls-certificate-file ~/.spice/pki/scheduler1.crt \
--node-mtls-key-file ~/.spice/pki/scheduler1.key
# Executor: needs no spicepod.yaml
spiced --role executor \
--scheduler-address https://scheduler1:50052 \
--node-mtls-ca-certificate-file ~/.spice/pki/ca.crt \
--node-mtls-certificate-file ~/.spice/pki/executor1.crt \
--node-mtls-key-file ~/.spice/pki/executor1.keyThe full flag and configuration reference is in the distributed query documentation.
When should a workload run distributed instead of on a single node?
Distributed mode is built for throughput on large analytical queries that take seconds to minutes. Sub-second application queries stay on single-node Spice with SQL federation and acceleration. The three deployment options compare as follows:
| Deployment | Where query compute runs | Data it scales | Best fit | Availability |
|---|---|---|---|---|
| Single-node Spice | One spiced process | Federated sources and local accelerations that fit on one node | Low-latency application and agent queries, sidecar deployments | Open source |
| Distributed query | A scheduler plus an executor fleet | Partitioned Parquet, Delta Lake, and Iceberg tables, and HTTP datasets | Large data lake scans, joins, aggregations, and batch embed() calls | Open source (preview) |
| Distributed accelerations | Executors that each own a set of partitions | Accelerated datasets sharded by partition_by across executors | Accelerated datasets larger than one node's memory or disk | Spice.ai Enterprise |
The two modes combine in a cluster-sidecar architecture. A cluster processes the large scans, and sidecars next to each application serve a smaller accelerated working set. For a partitioned lakehouse workload end to end, see the data lake accelerator.
How does a Spice cluster compare with Apache Spark?
In an early preview run in October 2025, a 48-core Spice cluster with 24 GB of RAM finished each test query faster than Apache Spark on Databricks with 48 cores and 192 GB of RAM:
| Query | Spice cluster | Spark on Databricks | Difference |
|---|---|---|---|
| Wildcard filter, no matches | 6.1 s | 47 s | 7.7x faster |
| Wildcard filter, with matches | 6.6 s | 25 s | 3.8x faster |
embed() over about 100,000 rows | 9.8 s | 17 s | 1.7x faster |
The cluster used one-eighth of the RAM. Spice is written in Rust and moves data as Apache Arrow over Arrow Flight, so it has no JVM garbage collection. These are three queries on one configuration, not a general benchmark. The Apache Ballista engineering post describes the setup and the extensions Spice added to Ballista.
What does a distributed query cluster require?
A cluster adds operational requirements that a single node does not have. Plan for these before deploying one:
- Mutual TLS between nodes: scheduler-executor traffic runs on port 50052 with mTLS enforced by default, because it carries the Spicepod and expanded secrets. Each node needs a CA certificate, a node certificate with its address in the SANs, and a private key.
spice cluster tls initandspice cluster tls add <node>generate development certificates; production clusters use certificates from the organization's PKI. - One Spice version across the cluster: a scheduler rejects executors built for a different cluster protocol version, so upgrade all nodes together.
- Shared state for high availability: multiple active schedulers coordinate through shared state storage that supports conditional writes, such as Amazon S3, Azure Blob Storage, or a shared filesystem, with no etcd or ZooKeeper. A load balancer spreads client queries across them.
- Async queries for long jobs: a synchronous query does not survive the loss of its scheduler mid-execution. The async queries API (
/v1/queries) persists job state to the scheduler state location. Clients submit a job, poll its status, and fetch paginated results, and another scheduler can resume an unauthenticated job if the first one stops. - Preview status: open-source distributed query is a preview feature, introduced in Spice v1.9.0, and clusters can hit stability or performance issues that a single node does not.
To run a managed cluster or distributed accelerations, compare Spice Cloud and Enterprise plans.
Deployed in production
Run data-intensive workloads on a high-performance engine trusted by teams building real-time systems at scale.
“Spice opened the door to take these critical control-plane datasets and move them next to our services in the runtime path.”
Peter Janovsky
Software Architect, Twilio
100x
Faster queries
“It just spins up and works, which is really nice. The responsiveness is amazing, which is a huge gain for the customer.”
Darin Douglass
Principal Software Engineer, Barracuda
Learn how Barracuda gained 100x faster query responses and reduced costs 50% with Spice
“Partnering with Spice AI has transformed how NRC Health delivers AI-driven insights. By unifying siloed data across systems, we accelerated AI feature development, reducing time-to-market from months to weeks - and sometimes days. With predictable costs and faster innovation, Spice isn't just solving some of our data and AI challenges - it's helping us redefine personalized healthcare.”
Tim Ottersburg
VP of Technology, NRC Health
FAQs
Answers to common questions about running distributed SQL queries in Spice
Yes. Distributed query is part of the open-source Spice runtime as a preview feature, introduced in v1.9.0. The open-source cluster distributes scans of partitioned data lake tables and HTTP datasets. Distributing accelerated datasets across executors is a Spice.ai Enterprise feature called distributed accelerations.
No. You start one scheduler with a Spicepod, then start executors with the scheduler address. Executors register with the scheduler and fetch the configuration from it. For high availability, schedulers share state through an object store that supports conditional writes, such as Amazon S3, so no etcd or ZooKeeper is needed. Kubernetes is optional.
A custom optimizer splits large scans into shuffle partitions and pushes projections closer to the data source. Executors read only the columns and partitions a query needs. Executors also get UDFs and model definitions from the scheduler, and they load a model only when a query calls it.
In open source, distributed query reads partitioned data lake tables, such as Parquet, Delta Lake, and Iceberg in object storage. It also reads HTTP and HTTPS datasets without acceleration. Each executor scans its share of the data with its own client and credentials. Other connectors still work in the same Spicepod, but their queries do not spread across executors.
Run EXPLAIN on the query against the scheduler endpoint. When distributed planning is active, the output has a distributed_plan section. That section shows how the scheduler splits the query into stages across executors.
Spice uses mutual TLS (mTLS) on the internal cluster port, 50052, and enforces it by default. This traffic carries the Spicepod and expanded secrets, so only nodes with a trusted certificate can join. The spice cluster tls init and spice cluster tls add commands create development certificates. Production clusters use certificates from your own PKI.
Only an unauthenticated async query with a scheduler state location resumes on another scheduler, from its saved execution graph. An async job from an authenticated principal fails with SCHEDULER_UNAVAILABLE, and the client resubmits it. A synchronous query fails, and the client retries it on another scheduler.
See Spice in action
Walk through your use case with an engineer and see how Spice handles federation, acceleration, and AI integration for production workloads.
Talk to an engineer (opens in a new tab)