title: 'Arrow Data Accelerator Deployment Guide' sidebar_label: 'Deployment Guide' description: 'Operating guide for the Arrow (in-memory) data accelerator in production: memory sizing, indexes, and observability.' sidebar_position: 10 pagination_prev: null pagination_next: null tags:
Production operating guide for the Arrow in-memory data accelerator covering memory sizing, optional hash indexes, and observability.
The Arrow accelerator is an in-process, in-memory engine. There is no external storage and no authentication or secret management required.
The Arrow accelerator is not durable. Data is held in RAM and is lost on process restart; every restart re-materializes the dataset from the source connector.
primary_key (or secondary indexes entry) is configured, building a hash map over the indexed columns. Build time scales linearly with rows. The index stores no key bytes — only a 16-byte slot per entry (8-byte hash + packed 8-byte row location) plus roughly 1.25 bytes of bloom filter — so budget from the entry count, not the key width. Reported usage tracks allocated slot capacity, which exceeds the live entry count, so plan above the ~17 bytes/row floor.Generic acceleration metrics are available with the dataset_acceleration_ prefix. Hash-index operations emit dedicated metrics when the index is enabled:
| Metric | Type | Description |
|---|---|---|
hash_index_builds | Counter | Total hash-index builds (one per refresh). |
hash_index_build_duration_ms | Histogram | Time to build the hash index. |
hash_index_entries | Histogram | Number of entries in the index. |
hash_index_memory_bytes | Histogram | Approximate memory footprint of the index. |
hash_index_lookups | Counter | Total hash-index lookups performed by queries. |
hash_index_lookup_rows | Counter | Total rows returned via hash-index lookups. |
See Component Metrics (opens in a new tab) for enabling and exporting metrics. Refresh metrics are described in Acceleration (opens in a new tab).
Arrow acceleration operations (refresh, query) participate in task history (opens in a new tab) through the shared acceleration spans (acceleration_refresh, sql_query). No Arrow-specific spans are emitted — the accelerator is a thin wrapper over Arrow memory.
primary_key or a secondary indexes (opens in a new tab) entry is configured; the hash_index acceleration parameter is stripped with a warning and enables nothing on its own. A primary_key alone does not build an index under refresh_mode: caching, which drops the primary-key constraint before the table is created — configure indexes instead.partition_by partitions in memory only: Setting partition_by on engine: arrow switches the dataset to the partitioned Arrow accelerator, which holds one MemTable per partition value — see Partitioning (opens in a new tab). Every partition still lives in RAM, so partitioning prunes scans but does not relieve the memory ceiling above.| Symptom | Likely cause | Resolution |
|---|---|---|
| OOM on refresh | Source dataset larger than RAM. | Switch to a durable accelerator (DuckDB / SQLite / Cayenne) that supports spill to disk. |
| Long startup time | Full-dataset refresh runs on boot. | Switch to a durable accelerator so refresh is incremental, not full, on restart. |
The hash_index acceleration parameter is ignored for Arrow acceleration warning | hash_index no longer activates indexing on its own. | Remove hash_index from params: and set primary_key: or indexes: instead. |
No hash index despite primary_key being set | refresh_mode: caching drops the primary-key constraint. | Configure indexes: on the looked-up columns, or use a non-caching refresh mode. |
| Query slow for point lookups | No primary key/index, or wrong key column. | Add a primary_key: (or secondary indexes: entry); ensure the query filter matches the indexed columns. |
| Accelerator refuses to start with file mode | Arrow rejects file-mode acceleration. | Switch engine: to duckdb, sqlite, postgres, or cayenne. |