title: 'Data Accelerators' sidebar_label: 'Data Accelerators' description: '' image: /img/og/data-accelerators.png sidebar_position: 2 pagination_prev: null pagination_next: null

Data sourced by Data Connectors can be locally materialized and accelerated using a Data Accelerator.

A Data Accelerator will query/fetch data from a connected data source and store/update it locally in an embedded acceleration engine, such as DuckDB or SQLite. To set data refresh behavior, such as refreshing data on an interval see Data Refresh (opens in a new tab).

Dataset acceleration is enabled by setting the acceleration configuration. E.g.

datasets:
  - name: accelerated_dataset
    acceleration:
      enabled: true

datasets:
  - name: accelerated_dataset
    acceleration:
      enabled: true

For the complete reference specification, see datasets (opens in a new tab).

By default, datasets will be locally materialized using in-memory Arrow records.

A choice of DuckDB, SQLite, or PostgreSQL engines can be used to materialize data, in-memory, on disk, or in attached databases.

Supported Data Accelerators include:

Choosing an Accelerator

Select the appropriate accelerator based on dataset size, query patterns, and resource constraints:

Spice Cayenne vs DuckDB

Both Spice Cayenne (opens in a new tab) and DuckDB (opens in a new tab) support file-based acceleration, but differ in architecture and performance characteristics:

Choose Spice Cayenne when:

• Datasets exceed ~1 TB
• Multi-file data ingestion is required (e.g., partitioned S3 data)
• Lower memory overhead is preferred
• Workloads benefit from Vortex's 10-20x faster scans (opens in a new tab)
• Point lookups and random access patterns are common (100x faster than Parquet (opens in a new tab))

Choose DuckDB when:

• Datasets are under ~1 TB
• Complex SQL features are required (window functions, CTEs)
• Existing DuckDB tooling integration is beneficial
• Explicit index control is required

Data Types

Data Accelerators may not support all possible Apache Arrow data types. For complete compatibility, see specifications (opens in a new tab).

:::warning[Memory Considerations]

When accelerating a dataset using mode: memory (the default), some or all of the dataset is loaded into memory. Ensure sufficient memory is available, including overhead for queries and the runtime, especially with concurrent queries.

In-memory limitations can be mitigated by storing acceleration data on disk, which is supported by duckdb (opens in a new tab) and sqlite (opens in a new tab) accelerators by specifying mode: file.

:::

Data Accelerator Docs

import DocCardList from '@theme/DocCardList';

Related Documentation

• Managing Memory Usage (opens in a new tab) - Memory configuration reference
• Data Refresh (opens in a new tab) - Refresh mode configuration
• Indexes (opens in a new tab) - Index configuration for DuckDB and SQLite