Data sourced by Data Connectors can be locally materialized and accelerated using a Data Accelerator.
A Data Accelerator queries/fetches data from a connected data source and stores/updates it locally in an embedded acceleration engine, such as Spice Cayenne, DuckDB, or SQLite. To set data refresh behavior, such as refreshing data on an interval, see Data Refresh (opens in a new tab).
Dataset acceleration is enabled by setting the acceleration configuration:
For the complete reference specification, see datasets (opens in a new tab).
An acceleration that sets no engine uses Spice Cayenne (opens in a new tab). With no mode, the acceleration is in mode: memory: Cayenne holds the data in RAM and reloads it from the source on restart. On Windows, where Cayenne is not available, the default engine is arrow. Spice v2.3 and earlier use arrow as the default on every platform.
Some configurations load on Arrow but not on Cayenne. On Linux and macOS, with no engine, each of these fails to load until the acceleration sets engine: arrow or the configuration changes:
partition_by (opens in a new tab) in mode: memory, the default mode. Cayenne partitions only in a file mode, such as mode: file.Interval or Duration. Set unsupported_type_action (opens in a new tab) to convert or skip it.primary_key.retention_period (opens in a new tab) on a numeric time_column. Cayenne retention needs a Timestamp, Date, or ISO 8601 string column.| Name | Description | Status | Engine Modes |
|---|---|---|---|
cayenne | Spice Cayenne (opens in a new tab) | Stable | memory, file, file_create, file_update |
arrow | In-Memory Arrow Records | Deprecated | memory |
duckdb | Embedded DuckDB (opens in a new tab) | Deprecated | memory, file, file_create, file_update |
postgres | Attached PostgreSQL (opens in a new tab) (Spice.ai Enterprise) | Deprecated |
Every engine other than cayenne is deprecated in v2.4 and will be removed in v2.5. See Migrating to Spice Cayenne (opens in a new tab).
Select the appropriate accelerator based on dataset size, query patterns, and resource constraints:
| Use Case | Recommended Accelerator | Rationale |
|---|---|---|
| Small datasets (under 1 GB), maximum speed | arrow | In-memory storage provides lowest latency |
| Small datasets (1-10 GB), complex SQL | duckdb | Mature SQL support with memory management |
| Datasets 10 GB and above (up to 1+ TB) | cayenne | Needs 1/3 to 1/2 the memory of duckdb; Vortex columnar format scales beyond single-file limits |
| Point lookups on large datasets | cayenne | Vortex provides 100x faster random access vs Parquet |
| Simple queries, low resource usage | sqlite | Lightweight, minimal overhead |
| Async operations, concurrent workloads | turso | Native async support, modern connection pooling |
| External database integration | postgres | Use existing PostgreSQL infrastructure |
Both Spice Cayenne (opens in a new tab) and DuckDB (opens in a new tab) support file-based acceleration, but differ in architecture and performance characteristics. Spice Cayenne is recommended for any dataset of 10 GB or larger, because of DuckDB's memory requirements: Cayenne typically needs one-third to one-half the memory of the DuckDB accelerator for the same dataset.
Choose Spice Cayenne when:
refresh_mode: changes)refresh_mode: full would be repeated often enough to grow a DuckDB file — see Sizing the volume (opens in a new tab)Choose DuckDB when:
unique index that rejects duplicate writes; Cayenne's indexes narrow reads but do not constrain writes)Set acceleration.engine to cayenne and remove the DuckDB-only parameters, which Cayenne does not read: duckdb_file, duckdb_memory_limit, duckdb_preserve_insertion_order, on_refresh_sort_columns, and on_full_refresh. To keep the data clustered, replace on_refresh_sort_columns (opens in a new tab) with Cayenne sort_columns (opens in a new tab):
Unlike on_refresh_sort_columns, which drops primary_key, indexes, and on_conflict, sort_columns works with a primary key. For CDC datasets with a primary_key, the default cayenne_deletion_mode: auto (opens in a new tab) applies deletes by key.
Leave cayenne_file_path (opens in a new tab) and cayenne_metadata_dir (opens in a new tab) unset so every Cayenne dataset shares the default data root and catalog. If a deployment sets cayenne_file_path, use the same path on every Cayenne dataset; see Metastore location (opens in a new tab) for why.
Data Accelerators may not support all possible Apache Arrow data types. For complete compatibility, see specifications (opens in a new tab).
:::warning[Memory Considerations]
When accelerating a dataset using mode: memory (the default), some or all of the dataset is loaded into memory. Ensure sufficient memory is available, including overhead for queries and the runtime, especially with concurrent queries.
In-memory limitations can be mitigated by storing acceleration data on disk, which is supported by duckdb (opens in a new tab), sqlite (opens in a new tab), and turso (opens in a new tab) accelerators by specifying mode: file.
:::
Data accelerators store the schema that Spice infers from the data source at startup. This schema is fixed for the lifetime of the runtime process and defines the column names, data types, and nullability of the accelerated table.
If the source schema changes while the runtime is running (for example, new columns are added or data types change), subsequent data refreshes into the accelerator will fail because the incoming data no longer matches the schema of the accelerated table. Restart the runtime to re-infer the schema and re-initialize the accelerated table.
For details on how schema inference works per connector and recommendations for managing schema drift, see Schema Inference (opens in a new tab).
import DocCardList from '@theme/DocCardList';
maintained_aggregatesMINMAX| N/A |
sqlite | Embedded SQLite (opens in a new tab) | Deprecated | memory, file, file_create, file_update |
turso | Embedded Turso (opens in a new tab) | Deprecated | memory, file, file_create, file_update |