title: 'Data Accelerators' sidebar_label: 'Data Accelerators' description: '' image: /img/og/data-accelerators.png sidebar_position: 2 pagination_prev: null pagination_next: null

Data sourced by Data Connectors can be locally materialized and accelerated using a Data Accelerator.

A Data Accelerator will query/fetch data from a connected data source and store/update it locally in an embedded acceleration engine, such as DuckDB or SQLite. To set data refresh behavior, such as refreshing data on an interval see Data Refresh (opens in a new tab).

Dataset acceleration is enabled by setting the acceleration configuration. E.g.

datasets:
  - name: accelerated_dataset
    acceleration:
      enabled: true

datasets:
  - name: accelerated_dataset
    acceleration:
      enabled: true

For the complete reference specification see datasets (opens in a new tab).

By default, datasets will be locally materialized using in-memory Arrow records.

A choice of DuckDB, SQLite, or PostgreSQL engines can be used to materialize data, in-memory, on disk, or in attached databases.

Supported Data Accelerators include:

Data Types

Data Accelerators may not support all possible Apache Arrow data types. For complete compatibility, see specifications (opens in a new tab).

:::warning[Memory Considerations]

When accelerating a dataset using mode: memory (the default), some or all of the dataset is loaded into memory. Ensure sufficient memory is available, including overhead for queries and the runtime, especially with concurrent queries.

In-memory limitations can be mitigated by storing acceleration data on disk, which is supported by duckdb (opens in a new tab) and sqlite (opens in a new tab) accelerators by specifying mode: file.

:::

Data Accelerator Docs

import DocCardList from '@theme/DocCardList';