spiceai/docs

title: 'Data Accelerators' sidebar_label: 'Data Accelerators' description: 'Data acceleration engines for local materialization and query acceleration in Spice' image: /img/og/data-accelerators.png sidebar_position: 2 pagination_prev: null pagination_next: null

Data sourced by Data Connectors can be locally materialized and accelerated using a Data Accelerator.

A Data Accelerator queries/fetches data from a connected data source and stores/updates it locally in an embedded acceleration engine, such as Spice Cayenne, DuckDB, or SQLite. To set data refresh behavior, such as refreshing data on an interval, see Data Refresh (opens in a new tab).

Dataset acceleration is enabled by setting the acceleration configuration:

For the complete reference specification, see datasets (opens in a new tab).

By default, datasets are locally materialized using in-memory Arrow records.

Supported Data Accelerators

Name	Description	Status	Engine Modes
`cayenne`	Spice Cayenne (opens in a new tab)	Stable	`memory`, `file`, `file_create`, `file_update`
`arrow`	In-Memory Arrow Records	Stable	`memory`
`duckdb`	Embedded DuckDB (opens in a new tab)	Stable	`memory`, `file`, `file_create`, `file_update`
`postgres`	Attached PostgreSQL (opens in a new tab) (Spice.ai Enterprise)	Release Candidate

Choosing an Accelerator

Select the appropriate accelerator based on dataset size, query patterns, and resource constraints:

Use Case	Recommended Accelerator	Rationale
Small datasets (under 1 GB), maximum speed	`arrow`	In-memory storage provides lowest latency
Small datasets (1-10 GB), complex SQL	`duckdb`	Mature SQL support with memory management
Datasets 10 GB and above (up to 1+ TB)	`cayenne`	Needs 1/3 to 1/2 the memory of `duckdb`; Vortex columnar format scales beyond single-file limits
Point lookups on large datasets	`cayenne`	Vortex provides 100x faster random access vs Parquet
Simple queries, low resource usage	`sqlite`	Lightweight, minimal overhead
Async operations, concurrent workloads	`turso`	Native async support, modern connection pooling
External database integration	`postgres`	Use existing PostgreSQL infrastructure

Spice Cayenne vs DuckDB

Both Spice Cayenne (opens in a new tab) and DuckDB (opens in a new tab) support file-based acceleration, but differ in architecture and performance characteristics. Spice Cayenne is recommended for any dataset of 10 GB or larger, because of DuckDB's memory requirements: Cayenne typically needs one-third to one-half the memory of the DuckDB accelerator for the same dataset.

Choose Spice Cayenne when:

Datasets are 10 GB or larger
Memory headroom is constrained, or the deployment must run in a smaller container
Multi-file data ingestion is required (e.g., partitioned S3 data)
Workloads benefit from Vortex's 10-20x faster scans (opens in a new tab)
Point lookups and random access patterns are common (100x faster than Parquet (opens in a new tab))

Choose DuckDB when:

Datasets are under 10 GB
Complex SQL features are required (window functions, CTEs)
Existing DuckDB tooling integration is beneficial
Database-enforced index semantics are required (a unique index that rejects duplicate writes; Cayenne's indexes narrow reads but do not constrain writes)

Data Types

Data Accelerators may not support all possible Apache Arrow data types. For complete compatibility, see specifications (opens in a new tab).

:::warning[Memory Considerations]

When accelerating a dataset using mode: memory (the default), some or all of the dataset is loaded into memory. Ensure sufficient memory is available, including overhead for queries and the runtime, especially with concurrent queries.

In-memory limitations can be mitigated by storing acceleration data on disk, which is supported by duckdb (opens in a new tab), sqlite (opens in a new tab), and turso (opens in a new tab) accelerators by specifying mode: file.

:::

Schema Handling

Data accelerators store the schema that Spice infers from the data source at startup. This schema is fixed for the lifetime of the runtime process and defines the column names, data types, and nullability of the accelerated table.

If the source schema changes while the runtime is running (for example, new columns are added or data types change), subsequent data refreshes into the accelerator will fail because the incoming data no longer matches the schema of the accelerated table. Restart the runtime to re-infer the schema and re-initialize the accelerated table.

For details on how schema inference works per connector and recommendations for managing schema drift, see Schema Inference (opens in a new tab).

Data Accelerator Docs

import DocCardList from '@theme/DocCardList';