:::warning[Deprecated]
The arrow acceleration engine is deprecated in v2.4 and will be removed in v2.5. A Spicepod with a dataset or view that uses it logs one warning at startup, and one on each hot reload that adds or changes such a component, naming those datasets and views. A cluster executor does not log it. Runtimes up to and including v2.4.0-rc.3 do not log this warning. Move these accelerations to Spice Cayenne (opens in a new tab) before upgrading to v2.5. On Windows, where arrow is the default engine, an acceleration that sets no engine uses arrow and is included in the warning; Cayenne is not available on native Windows, so run Spice in Docker or WSL to use it; see Migrating to Spice Cayenne (opens in a new tab).
:::
The In-Memory Arrow Data Accelerator uses Apache Arrow to store data in-memory for fast access and query performance. It is the default data accelerator on Windows. On Linux and macOS, the default is Spice Cayenne (opens in a new tab); see Default Engine (opens in a new tab).
To use the In-Memory Arrow Data Accelerator, set arrow as the engine for acceleration:
Set under a dataset's acceleration.params:
| Parameter | Description |
|---|---|
arrow_sort_columns | Optional. Comma-separated list of columns to sort the data by when it is inserted, for example timestamp,user_id. |
:::warning[Experimental] Hash index is an experimental feature available in Spice v1.11.0-rc.2 and later. :::
The In-Memory Arrow Data Accelerator supports an optional hash index (opens in a new tab) for O(1) point lookups on primary key columns. Hash indexing activates automatically when a primary_key (or a secondary indexes (opens in a new tab) entry) is configured — no additional parameter is required:
The legacy hash_index: enabled parameter is accepted but no longer activates indexing on its own; when set, the runtime logs a warning and falls back to the automatic rules above.
See Hash Index (opens in a new tab) for configuration details, supported data types, and performance characteristics.
Decimal256 (76 digits), as it exceeds Arrow's maximum Decimal width of 38 digits.unique constraints.value1 / value2 are treated as integer division if the values are integers. For example, 1 / 2 will result in 0 instead of the expected 0.5. Use casting to FLOAT to ensure conversion to a floating-point value: CAST(1 AS FLOAT) / CAST(2 AS FLOAT) (or CAST(1 AS FLOAT) / 2).:::warning[Memory Considerations]
When accelerating a dataset using the In-Memory Arrow Data Accelerator, some or all of the dataset is loaded into memory. Ensure sufficient memory is available, including overhead for queries and the runtime, especially with concurrent queries.
In-memory limitations can be mitigated by storing acceleration data on disk, which is supported by duckdb (opens in a new tab) and sqlite (opens in a new tab) accelerators by specifying mode: file.
:::