title: 'Cayenne Data Accelerator' sidebar_label: 'Cayenne Data Accelerator' description: 'Cayenne Data Accelerator (Vortex) Documentation' sidebar_position: 1 tags:
:::info Alpha The Cayenne Data Accelerator is in Alpha. Features and configuration may change. Available in Spice v1.9.0-rc.1 and later. :::
Cayenne is a Spice data acceleration engine designed for high-performance, scalable query on large-scale datasets. Built on Vortex (opens in a new tab), a next-generation columnar file format, Cayenne combines columnar storage with in-process metadata management to provide fast query performance to scale to datasets beyond 1TB.
Cayenne uses Vortex as its storage format, providing significant performance advantages:
Vortex is a Linux Foundation (LF AI & Data) project under Apache-2.0 license with neutral governance.
While DuckDB (opens in a new tab) excels for datasets up to approximately 1TB, Cayenne with Vortex is designed to scale beyond these limits.
For detailed Vortex performance benchmarks, visit bench.vortex.dev (opens in a new tab).
To use Cayenne as the data accelerator, specify cayenne as the engine for acceleration. Cayenne only supports mode: file and stores data on disk.
Cayenne uses Vortex's advanced columnar format, which provides:
Consider the following limitations when using Cayenne acceleration:
mode: file and does not support in-memory (mode: memory) acceleration.on_conflict Support: Cayenne does not yet support the on_conflict (opens in a new tab) configuration for handling duplicate keys during data refresh.retention_sql: Data deletion and cleanup operations require configuring retention_sql (opens in a new tab) to define retention policies. Manual DELETE statements can also be executed directly.refresh_append_overlap: A dataset accelerated by Cayenne that sets fails to load, with . itself is supported — only the overlap window is not, so late-arriving rows behind the high-water mark are missed rather than re-read.:::warning[Alpha Software]
As an Alpha feature, Cayenne should be thoroughly tested in development environments before production deployment. Monitor release notes for updates, breaking changes, and new capabilities.
:::
Resource requirements for Cayenne depend on dataset size, query patterns, and metastore configuration.
Cayenne manages memory efficiently through columnar storage and selective caching. Allocate sufficient memory based on:
Cayenne stores data in a columnar format optimized for analytical queries. Ensure adequate disk space for:
Complete example configuration using Cayenne:
acceleration.refresh_append_overlapCayenne data accelerator does not yet support refresh_append_overlap. Please remove this configuration