title: 'Spice Cayenne Data Accelerator' sidebar_label: 'Spice Cayenne Data Accelerator' description: 'Spice Cayenne Data Accelerator (Vortex) Documentation' sidebar_position: 1 tags:
Spice Cayenne is a data acceleration engine designed for high-performance, scalable query on large-scale datasets. Built on Vortex (opens in a new tab), a high-performance columnar file format, Spice Cayenne combines columnar storage with in-process metadata management to provide fast query performance to scale to datasets beyond 1TB.
Spice Cayenne uses Vortex as its storage format, providing significant performance advantages:
Vortex is a Linux Foundation (LF AI & Data) project under Apache-2.0 license with neutral governance. For performance benchmarks, see bench.vortex.dev (opens in a new tab).
Spice Cayenne is recommended for any dataset of 10 GB or larger. DuckDB (opens in a new tab) suits smaller datasets, but its memory requirements grow faster: Cayenne typically needs one-third to one-half the memory of the DuckDB accelerator for the same dataset, and scales beyond the limits at which a single DuckDB file becomes impractical.
Spice Cayenne follows a lakehouse architecture inspired by DuckLake (opens in a new tab), separating metadata management from data storage:
Key Design Principles:
ListingTable at a unique directory, enabling append operations and parallel readsFor optimal performance, store Cayenne data files and the metastore on local NVMe storage, and point runtime.query.temp_directory at the same fast volume. What matters most is per-I/O latency rather than IOPS: Vortex's random-access reads are a chain of dependent segment reads and the metastore commits with fsync on every write, so the tens of microseconds an NVMe operation takes — against a millisecond or more on network storage — is multiplied along every query and every commit. Local NVMe on cloud instances is ephemeral, so pair it with acceleration snapshots (opens in a new tab) for fast cold starts.
Network block storage (Amazon EBS, Azure Managed Disks, GCP Persistent Disk) works as a durable fallback: Cayenne detects it as the network-attached storage tier and adapts — larger inline flushes, an O_DIRECT compaction writer, and a write-concurrency cap derived from the volume's bandwidth — but every cache miss still pays the volume's per-read latency, so prefer a sub-millisecond tier such as io2 Block Express. Network file systems (NFS, SMB, EFS, Azure Files) are not recommended: the metastore is a SQLite database, and SQLite locking is unreliable on them. See Storage (opens in a new tab) in the Cayenne performance guide and Storage (opens in a new tab) in the Performance Tuning guide.
Use S3 Express One Zone (opens in a new tab) when persistence of accelerations across restarts is required. S3 Express One Zone adds network latency compared to local NVMe but provides durability. Sharing accelerated data across multiple Spice instances is planned for a future release. Express One Zone is the Cayenne data tier (cayenne_file_path, cayenne_s3_*) only. Acceleration snapshots (opens in a new tab) use snapshots.location on standard S3, GCS, or ADLS, and the cold tier (opens in a new tab) uses cayenne_datalake_location on a general-purpose S3 or S3-compatible bucket. An Express directory bucket does not substitute for the snapshot bucket.
Spice Cayenne is the default data accelerator on Linux and macOS: an acceleration that sets no engine uses Cayenne, in mode: memory unless it sets another mode. Cayenne is not available on Windows, where the default is Arrow (opens in a new tab). See Default Engine (opens in a new tab). To select Spice Cayenne explicitly, specify cayenne as the engine for acceleration. Spice Cayenne supports two storage modes:
mode: file (durable) — data is written as Vortex files on local disk or S3 Express One Zone, with a SQLite/Turso metastore, and the acceleration survives restarts. This is the recommended mode for Cayenne and is used in the examples throughout this page. The mode: file_create and mode: file_update variants control how an existing on-disk acceleration is reused or rebuilt on startup.mode: memory (ephemeral) — all data lives fully in RAM with an in-memory metastore; nothing is written to disk. The dataset is ephemeral and reloads from its source on restart (like the Arrow (opens in a new tab) accelerator). Memory mode works for all refresh modes (full/append/changes) and for both keyed and no-primary-key datasets, but does not support partitioned tables (partition_by), and it enforces a hard per-table RAM bound rather than spilling to disk (see cayenne_cdc_mem_tier_max_bytes (opens in a new tab)). INSERT, UPDATE and DELETE apply to the in-RAM tier the same way they apply to a mode: file acceleration — see Writes in memory mode (opens in a new tab).Spice Cayenne is configured through two distinct parameter scopes:
acceleration.params and control how that dataset's accelerated data is stored, compressed, written, and compacted.runtime.params and control engine-global behavior — caches, optimizer rules, and dedicated memory pools — shared by every Cayenne-accelerated dataset.The two scopes are not interchangeable: setting a runtime parameter under acceleration.params (or a per-dataset parameter under runtime.params) has no effect — the value is ignored.
acceleration.params)Set under a dataset's acceleration.params:
| Parameter | Description |
|---|---|
cayenne_compression_strategy | Compression algorithm for accelerated data. Defaults to btrblocks. Supports btrblocks or zstd. |
cayenne_delta_encoding | Encoding effort applied to delta (incremental) writes such as appends and inline-memtable flushes. Accepts auto (default) or a fixed level 0–10. Higher levels search more encoding schemes for a better compression ratio at the cost of more write-time CPU; auto encodes every delta write at a light level regardless of size, because deltas are transient staged streams that compaction later re-encodes with the full cascade. Levels 7–10 all apply the full default cascade, so set 7 to encode every delta with the full cascade instead. Applies at write time only — changing it never re-encodes existing data or forces a table re-create. Invalid values fall back to auto with a warning. |
cayenne_unsupported_type_action | Action when an unsupported data type is encountered. Defaults to error. See Data Type Support (opens in a new tab). |
cayenne_segment_cache_mb | Ignored here. The Vortex segment cache is one budget shared by every Cayenne table, so a per-dataset size has nothing to size. A value set under acceleration.params is reported at startup and otherwise has no effect — set runtime.params.cayenne_segment_cache_mb (opens in a new tab) instead. |
cayenne_force_view_types | Whether scans emit Arrow view types (Utf8View/BinaryView) instead of native Utf8/Binary. Accepts true or false; defaults to false. View types are not compacted across RepartitionExec, so wide strings fanned out through a partitioned join can inflate query-pool memory reservations — the native types avoid that. Set to true to re-enable view types for a table. Any value other than false (case-insensitive) enables view types. |
cayenne_file_path | Custom path for storing Cayenne data files. Supports local paths or S3 Express One Zone URLs (e.g., s3://bucket--usw2-az1--x-s3/prefix/). |
cayenne_target_file_size_mb | Target size for individual Vortex files in MB. When writes exceed this size, a new Vortex file is created. Accepts auto (default) or an explicit MB value. auto is storage-aware: 256 MB on EBS-class network storage, 64 MB on RAM-backed (tmpfs) mounts, 512 MB on S3 Express (large immutable objects cut object count and per-request cost), and 256 MB on local SSD or unknown storage. Smaller files enable better parallelism and predicate pushdown. |
cayenne_metadata_dir | Directory for the Cayenne SQLite catalog (cayenne.db). When unset, a local cayenne_file_path selects {cayenne_file_path}/metadata; otherwise the directory is {spice_data_path}/metadata, including when cayenne_file_path is an object-store URL. An explicit path overrides both. Must resolve outside the dataset's data directory — see Metastore location (opens in a new tab). |
cayenne_metastore | Metastore backend type. Supports sqlite (default) or turso (requires turso feature flag). |
cayenne_upload_concurrency | Maximum number of concurrent file uploads when writing multiple Vortex files to S3 Express One Zone. Accepts auto (default) or an explicit value; auto uses the runtime's CPU entitlement (opens in a new tab) in whole cores. The aggregate encode concurrency across all Cayenne tables is separately bounded by a process-global budget sized from that same entitlement, less a query reserve of a quarter of the cores (at least 2). |
cayenne_write_concurrency | Writer partition override for unsorted ingests, controlling how many Vortex files are encoded in parallel during a write. Accepts auto (default) or an explicit value. auto encodes up to min(4, session target_partitions) files in parallel per write — an intentionally small per-table default (not the full CPU entitlement), so many independently-writing tables do not oversubscribe CPU under concurrent CDC. An explicitly-set value is capped at the session target_partitions, which defaults to the runtime's CPU entitlement (opens in a new tab) in whole cores; the aggregate encode concurrency across all Cayenne tables is separately bounded by a process-global budget sized from that entitlement. Values below 1 are clamped to 1. The sort-and-rewrite compaction path always writes serially regardless of this setting. |
cayenne_scan_concurrency | How many splits a single Vortex file scan decodes concurrently. auto (default) derives it from the query fan-out and the number of files the scan plans, so a table held in few files still decodes in parallel. off (also disabled, none, or 0) decodes each file serially, and a positive integer pins the count. Each concurrent split holds a decoded batch charged to runtime.query.memory_limit, so lowering this value reduces a scan's decode memory without changing runtime.query.target_partitions. Any other value fails the dataset's load. |
cayenne_pk_keyset_cache_mb | Memory budget in MB for the in-memory primary-key index used to detect upsert conflicts during CDC ingestion. Within the budget, an exact keyset is kept. Over it, Cayenne falls back to a bounded Bloom filter instead of rebuilding the index for each batch. When unset or auto, the budget is derived from available memory, as described in Memory (opens in a new tab). 0 is raised to 1 with a warning. |
cayenne_deletion_mode | How primary-key deletions are recorded and applied. Accepts auto, key, or position; defaults to auto, which chooses per table: a small primary-key table on local disk uses position deletes on Linux with at least 8 physical CPU cores, other primary-key tables use key deletes, and a table without a primary key always uses position deletes. See Deletion Strategies (opens in a new tab). |
cayenne_pk_conflict_detection | Controls primary-key conflict detection on insert. Accepts auto or none; defaults to auto, which detects existing primary keys and resolves them as merge-on-read upserts. Set to none to skip conflict detection (blind append), so repeated keys are stored as written. Registration refuses none on a dataset that rewrites stored keys: refresh_mode: changes, durable write-back, or writes kept in the accelerator. See Turning off conflict detection (opens in a new tab). |
cayenne_bake_deletion_index_trigger | Number of live primary-key tombstones in the deletion index at or above which Cayenne consolidates the settled older snapshots so their tombstones leave the index. A larger value consolidates less often and writes less; a smaller value keeps the index smaller and queries cheaper to probe. Applies to tables that record key deletions. Default: 50000. |
cayenne_compaction_trigger_files | Minimum number of small Vortex files in the current snapshot before tiered compaction runs. A "small" file is one whose size is below cayenne_target_file_size_mb / 4. Defaults to 4 for refresh_mode: caching / changes, or append with refresh_check_interval ≤ 5m; 8 otherwise. A value of 1 is clamped to a minimum of 2. |
cayenne_compaction_trigger_protected_snapshots | Number of protected snapshots before snapshot-maintenance compaction runs. Separate from cayenne_compaction_trigger_files so small-file tuning does not silently change scan amplification behavior. Defaults to 4 for refresh_mode: caching / changes, or append with refresh_check_interval ≤ 5m; 8 otherwise. A value of 1 is clamped to a minimum of 2. |
cayenne_compaction_trigger_snapshot_age_ms | Maximum age in milliseconds of the oldest protected snapshot before snapshot-maintenance compaction runs. Set to 0 to disable the age trigger. Defaults to 60000 for refresh_mode: caching / changes, or append with refresh_check_interval ≤ 5m; 300000 otherwise. |
cayenne_compaction_max_levels | Maximum number of consecutive compaction passes per trigger. Bounds write amplification when promotion keeps producing new candidates. Defaults to 3. |
cayenne_compaction_max_files_per_pick | Maximum number of eligible file paths retained in one compaction candidate. For a table eligible for subset compaction — key-based deletes (cayenne_deletion_mode: key, including append-only tables) or a table without a primary key that holds no position deletes, no configured sort_columns or cayenne_cluster_by, and no protected snapshots — the runner rewrites only the picked candidate and carries the unpicked settled files into the new snapshot by hardlink (local filesystem) or copy (S3 or cross-device), so this value does bound rewrite IO and memory on a key-delete table. A table without a primary key extends the candidate with its tier's next-smallest files up to one sixteenth of cayenne_target_file_size_mb (16 MiB at a 256 MB target), so its merge is bounded by bytes rather than by this value, and its small-file tier also qualifies once it holds this many small files (or cayenne_compaction_trigger_files, if larger), whatever their total size. Such a table holds writers off only to commit the pass, not while it encodes. A table that holds position deletes, has a configured sort_columns or cayenne_cluster_by (whose subset hardlinks would break the global sort attestation), or carries protected snapshots (which the rewrite has to fold) still rewrites the whole current snapshot, and this value then only affects trigger selection and observability. Defaults to 32. |
cayenne_compaction_background_interval_ms | Background compaction interval in milliseconds. The accelerator runs a per-table background task at this interval. Set to 0 to disable the background task — inline compaction on writes still runs. Defaults to 10000 for refresh_mode: caching / changes, or append with refresh_check_interval ≤ 5m; 0 for refresh_mode: full, where each refresh replaces the whole table and leaves nothing for the background task to consolidate (set the parameter explicitly to turn it back on for a table that also takes INSERTs); 30000 otherwise. When refresh_mode is left unset the connector fills it in — debezium and cdc resolve to changes, sink to disabled, every other connector to full — so an unannotated dataset takes the full default here. |
cayenne_cdc_durability | Durability mode for the inline CDC write path. Accepts memory (default) or file. memory appends batches to an in-RAM tier and defers the source-slot acknowledgement to a periodic or cap-triggered checkpoint, collapsing per-batch durability cost; on crash the un-checkpointed tail is replayed from the source slot, and because the apply is primary-key-idempotent this remains exactly-once. The in-memory path is eligibility-gated — it applies only to the small-write / CDC profile on non-partitioned tables, and other profiles always use file. file persists each CDC batch durably before advancing the source slot and is the conservative opt-out. |
cayenne_cdc_mem_tier_max_bytes | Per-table RAM-tier byte cap that forces a spill (checkpoint) and source-slot advance, in cayenne_cdc_durability: memory mode only. Auto-derived from host memory (~1/64 of RAM, clamped to 256 MiB–1 GiB; 256 MiB on hosts at or under 16 GiB). A process-global byte budget also bounds aggregate resident memory across all tables; whichever cap is breached first triggers the spill. Set to 0 to disable the per-table cap (the global budget still applies). |
cayenne_cdc_mem_tier_max_age_ms | Maximum wall-clock milliseconds a RAM-tier epoch may age before a forced checkpoint, in cayenne_cdc_durability: memory mode only. Bounds the crash-replay window and the deferred source-slot acknowledgement for tables that never reach a byte threshold. Defaults to 10000 (10 s). Set to 0 to disable the age trigger. |
cayenne_cdc_mem_tier_min_flush_bytes | Minimum resident RAM-tier bytes before the periodic background checkpoint tick durably checkpoints, in cayenne_cdc_durability: memory mode only. Bounds snapshot / delete-file churn — below this size a tick is skipped unless the tier has reached cayenne_cdc_mem_tier_max_age_ms. Query freshness is unaffected (RAM rows are visible immediately); only the deferred slot acknowledgement waits. The write-path byte-cap spill is not gated by this value. Auto-derived as 1/8 of the resolved cayenne_cdc_mem_tier_max_bytes (clamped to 32–128 MiB; 32 MiB on hosts at or under 16 GiB). Set to 0 to flush on every tick. |
cayenne_cdc_mem_tier_checkpoint_interval_ms | Periodic background mem-tier checkpoint interval in milliseconds, in cayenne_cdc_durability: memory mode only. The accelerator spawns a per-table background task that checkpoints the RAM tier every interval, advancing the deferred source-slot acknowledgement on an idle or pure-upsert stream that never trips a write-path cap or event trigger. Defaults to 1000 (1 s). Set to 0 to disable the periodic task. |
cayenne_cdc_mem_tier_shards | Number of primary-key hash shards the in-RAM CDC tier is split into, in cayenne_cdc_durability: memory mode only, on tables that are not partitioned and use key-based merge-on-read (cayenne_deletion_mode: key; see Deletion Strategies (opens in a new tab)). Disjoint keys are validated and appended in parallel, while every version of one key, including deletes, stays in one shard, so the last write for a key still wins. Default: 1, the serial path. Raise it, for example to 4, on CDC tables with many updates or inserts. |
sort_columns | Comma-separated list of columns to sort data by on refresh operations. Improves segment pruning for frequently filtered columns. The order is established by the compaction rewrite and, on a refresh_mode: full table (which never compacts, because each refresh replaces the whole table), by the whole-table replace itself. A sorted write goes through a single writer, so it trades refresh throughput for scan pruning. Sorting on a full refresh requires an operator-configured value — an inference-derived order (see cayenne_sort_columns_origin (opens in a new tab)) does not trigger it. Cannot be combined with cayenne_cluster_by; registration fails until one is removed. |
cayenne_shard_key_columns | Comma-separated columns that decide which output file a row is written to when a write encodes several files in parallel. Defaults to the primary key. A CDC write hashes this key across the files; a compaction rewrite and a refresh_mode: full whole-table replace instead split it into ascending, disjoint ranges, one per file, so each file's statistics prune (see Sorted data and segment pruning (opens in a new tab)). It selects a layout, not an order: no sort is advertised to the planner and query results are unaffected. |
cayenne_sort_columns_origin | Provenance of cayenne_sort_columns, which decides whether that sort order outranks the filter columns observed on scans. Accepts user (the default when absent) — the sort order is an explicit operator choice and is authoritative — or inferred, meaning schema inference filled it from the source's declared order (for most CDC datasets, the primary key). An inferred order is treated as a guess and ranks below the observed filter columns, so the default-on adaptive layout can cluster for the workload actually being queried. Schema inference sets this automatically whenever it populates cayenne_sort_columns; set it by hand only to reproduce an inferred configuration (for example in a benchmark or test). |
unsupported_type_action | Action when encountering unsupported data types. Options: error (default), string, warn, ignore. |
These are acceleration parameters (set under acceleration.params) used when storing Cayenne data files in S3 Express One Zone:
| Parameter | Description |
|---|---|
cayenne_s3_zone_ids | Comma-separated availability zone IDs (e.g., usw2-az1,usw2-az2). Auto-generates bucket names in format spice-{app}-{dataset}--{zone}--x-s3. |
cayenne_s3_region | AWS region (e.g., us-west-2). Auto-derived from zone ID if not specified. |
cayenne_s3_auth | Authentication method: iam_role (default) or key. |
cayenne_s3_key | AWS access key ID (required when cayenne_s3_auth: key). |
cayenne_s3_secret | AWS secret access key (required when cayenne_s3_auth: key). |
cayenne_s3_session_token | AWS session token (optional, for temporary credentials). |
cayenne_s3_endpoint | Custom S3 endpoint URL (optional, overrides auto-generated endpoint). |
cayenne_s3_client_timeout | Request timeout duration (e.g., 30s, 5m). Defaults to 120s. |
cayenne_s3_unsigned_payload | Use unsigned payload for S3 Express One Zone requests. Defaults to true. |
cayenne_s3_allow_http | Set to true for testing with local S3-compatible storage. Defaults to false. |
These acceleration parameters (set under acceleration.params) configure the optional cold object-store tier (opens in a new tab). Setting cayenne_datalake_location enables the tier; the rest tune the clustering key, cold file size, the warm→cold promotion trigger, and the cold store's credentials. When cayenne_datalake_location is unset (the default), the cold tier is dormant and behaves byte-identically to a warm-only table.
| Parameter | Description |
|---|---|
cayenne_datalake_location | Object-store URL prefix for the cold tier (the bottom tier of the storage cascade). Must be an s3:// URL (e.g. s3://bucket/prefix) — a general-purpose S3 or S3-compatible bucket, distinct from the warm S3 Express One Zone store. When set, a background promotion stage graduates the warm local-disk tier to read-optimized, Z-order-clustered Vortex files on this store, and queries span the warm and cold tiers with per-tier pushdown. Unset (default) disables the cold tier. Requires key-based deletes (cayenne_deletion_mode auto-resolves to key; an explicit position is rejected at registration) and refresh_mode: changes or append. A primary_key is required to activate the tier — without one the tier registers but stays inactive. v1 constraints: local file:// locations are not supported; partitioned and position-delete tables are not supported. |
cayenne_cluster_by | Comma-separated columns to Hilbert-cluster both the warm and the datalake tier by, e.g. tenant_id,ts. Clustering tightens each file's per-column zone maps so selective queries on any clustering dimension prune at the storage layer. It is a complete layout instruction: while it is set, automatic and inferred sort columns are not applied, and it cannot be combined with cayenne_sort_columns — registration fails until one is removed. Every column must exist and be a Boolean, numeric (except Decimal256), temporal, string, or binary column; a missing or unsupported column is rejected at registration. When unset, layout falls back to an operator-configured cayenne_sort_columns; when that is also unset, to the hottest columns observed in query pushdown filters (default-on adaptive layout), then to an inference-derived cayenne_sort_columns (see cayenne_sort_columns_origin (opens in a new tab)), and finally to the primary key when no usable observed column exists. Replaces the cold-tier-only cayenne_datalake_clustering_columns, which is no longer accepted. |
cayenne_datalake_target_file_size_mb | Target size for cold-tier Vortex files in MB. Larger than the warm cayenne_target_file_size_mb because object stores favor fewer, larger objects and cold scans are range reads. Accepts auto or an explicit MB value. Defaults to 512. |
cayenne_datalake_warm_max_bytes | The warm tier graduates to cold once its total Vortex bytes reach this threshold. 0 (default) disables the byte trigger; set alongside cayenne_datalake_warm_max_files to bound warm-tier size. When the tier is enabled and neither trigger is set, Cayenne applies a default byte trigger (16× cayenne_datalake_target_file_size_mb) so promotion is not silently disabled. |
cayenne_datalake_warm_max_files | The warm tier graduates to cold once its Vortex file count reaches this threshold. 0 (default) disables the file-count trigger. |
cayenne_datalake_tiering_check_interval_ms | How often the background loop checks the warm→datalake tiering trigger, in milliseconds. Datalake tiering is not latency-critical, so this is coarser than compaction. Defaults to 60000 (60s). 0 disables the tiering and garbage-collection loop entirely and is rejected at registration when the datalake tier is enabled. |
cayenne_datalake_gc_interval_ms | Physical-GC cadence (and orphan grace) for superseded cold-tier objects: the background sweep runs about this often and deletes an object no longer referenced by the manifest only after it has been observed orphaned for at least one interval. Defaults to 300000 (5m). 0 collapses the grace period to zero (a superseded object could be deleted while a running query still reads it) and is rejected at registration when the cold tier is enabled. |
The cold store is authenticated independently from the warm tier (which uses the cayenne_s3_* parameters) via the following acceleration.params:
| Parameter | Description |
|---|---|
cayenne_datalake_s3_auth | Authentication method for the cold S3 store: iam_role (default, uses environment/SDK credentials) or key. |
cayenne_datalake_s3_key | AWS access key ID for the cold store (required when cayenne_datalake_s3_auth: key). |
cayenne_datalake_s3_secret | AWS secret access key for the cold store (required when cayenne_datalake_s3_auth: key). |
cayenne_datalake_s3_session_token | AWS session token for the cold store (optional, for temporary credentials). |
cayenne_datalake_s3_region | AWS region of the cold bucket. Defaults to the environment region (AWS_REGION / AWS_DEFAULT_REGION), then us-east-1; inert for S3-compatible endpoints. |
cayenne_datalake_s3_endpoint | Custom S3 endpoint URL for the cold store (e.g. an S3-compatible store such as MinIO). http:// endpoints implicitly allow HTTP. |
cayenne_datalake_s3_allow_http | Allow plain-HTTP connections to the cold S3 endpoint. Defaults to false. |
cayenne_datalake_s3_client_timeout | HTTP client timeout for cold-store requests, as a duration (e.g. 2m). Defaults to 2m. |
cayenne_datalake_s3_unsigned_payload | Use unsigned payloads for cold S3 uploads. Defaults to true. |
runtime.params)Set once under the top-level runtime.params and applied to every Cayenne-accelerated dataset in the instance. These are not valid under a dataset's acceleration.params.
Adaptive tuning is turned on for every Cayenne dataset and catalog with runtime.adaptive_tuning (opens in a new tab), and its service-level targets are the runtime-wide runtime.params.target_* (opens in a new tab) parameters. See Self-Tuning (opens in a new tab).
| Parameter | Description |
|---|---|
cayenne_segment_cache_mb | Total size in megabytes of the in-memory Vortex segment cache, which holds decompressed data segments for faster reads. One cache serves every Cayenne table in the process — dataset accelerations and from: cayenne catalogs alike — so adding a table divides this pool instead of reserving another cache of its own. Takes a whole number of megabytes; 0 disables segment caching. When unset, derived as ~1/64 of the process's memory entitlement, clamped to 256 MB–2 GB. A non-numeric value (including auto) is warned about and falls back to the derived budget. |
cayenne_footer_cache_mb | Size of the engine-wide in-memory Vortex footer cache in megabytes. The footer cache stores Vortex file metadata (schemas, statistics, encoding information) and is shared across all Cayenne datasets. Larger values improve query performance for repeated scans. Optional; when unset, no explicit limit is applied and DataFusion's default file-metadata-cache limit of 50 MB applies (there is no fixed 128 MB default). |
cayenne_filter_propagation | Enables Cayenne's filter-propagation optimizer rules. Accepts enabled or disabled; defaults to disabled. |
cayenne_optimizer_rules | Selects which Cayenne optimizer rules run. Accepts auto (default — enables the recommended set, gated by cayenne_filter_propagation), all, none / disabled, or a comma-separated list of individual rule names. |
cayenne_compaction_memory_fraction | Fraction of the query memory pool carved out for a dedicated Cayenne compaction memory pool. Defaults to 0.2 and is clamped to a supported range. Only applied when at least one enabled Cayenne dataset can accumulate files to compact — a file acceleration mode on a refresh mode that is not a whole-table replace — and dedicated thread pools are not disabled. A pod whose Cayenne datasets are all refresh_mode: full, or all mode: memory, carves no compaction pool and brings up no compaction runtime. |
cayenne_sort_merge_min_rows | Advanced anti-join tuning: row-count threshold above which the filter-propagation optimizer switches to a sort-merge strategy. Defaults to an internally tuned value; override only when profiling indicates a need. |
cayenne_sort_merge_memory_pool_fraction | Advanced anti-join tuning: fraction of the memory pool the sort-merge anti-join strategy may use. Defaults to an internally tuned value. |
cayenne_metastore_cache_mb | SQLite metastore page-cache size in megabytes. Applies to the default sqlite metastore backend (see cayenne_metastore); the turso backend uses MVCC and ignores the cayenne_metastore_* family. Defaults to 256. |
cayenne_metastore_mmap_mb | SQLite metastore memory-mapped I/O size in megabytes. Defaults to 1024 (1 GiB). |
cayenne_metastore_busy_timeout_ms | SQLite metastore busy_timeout in milliseconds — how long a blocked connection waits for a lock before erroring. Defaults to 30000. |
cayenne_metastore_wal_autocheckpoint_pages | SQLite metastore WAL auto-checkpoint threshold in pages. 0 disables the inline auto-checkpoint so the WAL is drained off the hot commit path by a dedicated background checkpoint instead. Defaults to 0. |
cayenne_metastore_wal_truncate_threshold_mb | WAL size in megabytes above which the background checkpoint escalates to a TRUNCATE checkpoint to reclaim file space. Defaults to 160. |
cayenne_metastore_auto_vacuum | SQLite metastore auto_vacuum mode: none, incremental, or full. Defaults to incremental, so freed pages are marked reclaimable and reclaimed in bounded batches by the background maintenance pass — see cayenne_metastore_incremental_vacuum_pages. An unrecognized value logs a warning and keeps the incremental default. SQLite fixes the mode at file creation, so a metastore created while the default was none stays on none — the runtime warns at startup when it finds one. To move it to incremental, stop the runtime, run PRAGMA auto_vacuum = INCREMENTAL; VACUUM; against the metastore file, and restart: the pragma alone is a no-op on a file created with auto_vacuum = none, and a bare VACUUM keeps it on none. Switching between full and incremental needs no VACUUM. See the SQLite auto_vacuum documentation (opens in a new tab). |
cayenne_metastore_incremental_vacuum_pages | Freelist pages the background maintenance pass reclaims per tick when the metastore is in incremental auto_vacuum mode; ignored in every other mode. Defaults to 256, which is 1 MiB at SQLite's 4 KiB default page size. Reclamation holds the write lock while it relocates pages, so the cap is what keeps each pause short; raise it to drain a large freelist faster at the cost of longer write-lock holds, or set 0 to stop reclaiming without changing the database's auto_vacuum mode. |
A mode: file table records the settings it was built with in the metastore. When a dataset's settings change between restarts, Cayenne compares them with the recorded ones on the next start. A setting that only changes how new rows are written is applied to the existing table. A setting that changes what the stored rows mean is applied by rebuilding the acceleration from the source, but only when a reload brings back every row the acceleration holds: refresh_mode: full with write_mode: write_through (the default), from a source other than sink.
| Setting changed between restarts | refresh_mode: full, write_mode: write_through | Any other refresh mode or write mode |
|---|---|---|
cayenne_sort_columns, cayenne_cluster_by, cayenne_compression_strategy, cayenne_delta_encoding, cayenne_shard_key_columns, or cayenne_pk_conflict_detection from auto to none | Applied in place | Applied in place |
primary_key, or the data location (cayenne_file_path) | Rebuilt from the source | Kept with the recorded settings, with a warning |
partition_by, or cayenne_pk_conflict_detection from none to auto on a table with a primary_key | Rebuilt from the source | Fails to load (see Changing partition_by after refresh (opens in a new tab) and Turning off conflict detection (opens in a new tab)) |
| Source schema | Governed by on_schema_change (opens in a new tab) | Governed by on_schema_change |
A setting applied in place is recorded in the metastore, and the table keeps serving its existing files. Rows already stored keep their current layout until a refresh or compaction rewrites them. The runtime logs the change:
A rebuild drops the table and loads it from the source as on a first start, and the runtime logs which setting caused it:
A rebuild happens only after the source answers a read on the schema the dataset registered with. While the source is unreachable, or while on_schema_change: block holds back a change to its schema, the dataset keeps serving its existing acceleration with the recorded settings and logs a warning. Restart once the source can be read to rebuild. A changed partition_by, or cayenne_pk_conflict_detection changed from none to auto, cannot be served from the stored rows, so in that case the dataset fails to load with an error that names both fixes: revert the setting, or restart once the source can be read.
When the data location changes, the rebuild deletes the local directory the old table was written to and never the newly configured one. Files stored in S3 Express One Zone are not deleted, and a warning lists the prefixes they remain under. A data location change is detected only when the metastore stays in the same directory, such as with an explicit cayenne_metadata_dir (opens in a new tab). Without it, moving a local cayenne_file_path also moves the metastore, so the dataset starts a new acceleration.
In any other refresh mode or write mode, the acceleration can hold rows a reload does not bring back, such as an append window, a change stream that resumes from a recorded position, or writes that exist only in the acceleration. A layout-only change is still applied in place and needs no rebuild. When a change to primary_key or the data location is among the changed settings, Cayenne keeps every recorded setting, including layout settings changed in the same restart, and logs a warning that names the changed settings. To apply those settings, restart once with mode: file_create, and with write_mode: write_through if the dataset uses write_mode: write_back, then set both back. Rows that exist only in the acceleration are lost.
In earlier releases, including v2.4.0-rc.1, a changed primary_key or cayenne_sort_columns on a mode: file Cayenne table was not applied: the table kept its recorded settings and logged a warning that the existing acceleration was not recreated. Now a changed sort order is applied in place, and a refresh_mode: full dataset whose primary_key changes is rebuilt from the source on the next start.
Spice Cayenne is self-tuning by default and rarely needs manual tuning. For the full tuning reference — self-tuning and goal-driven SLOs (opens in a new tab), cache sizing (opens in a new tab), compression strategy (opens in a new tab), and file-size tuning (opens in a new tab) — see the dedicated Performance Tuning (opens in a new tab) page.
Spice Cayenne is DataFusion query-native, meaning all query execution uses Apache DataFusion (opens in a new tab) and adheres to the runtime.query.memory_limit setting. This provides:
DataFusion's GreedyMemoryPool (opens in a new tab) allows memory reservations on a first-come, first-served basis, improving throughput for high-concurrency queries with many partitions.
Spice Cayenne uses Vortex's advanced columnar format, which provides:
Vortex delivers 100x faster random access reads (opens in a new tab) compared to Apache Parquet through several architectural features:
Segment Statistics (Zone-Map Equivalent):
Vortex's ChunkedLayout (opens in a new tab) maintains per-segment statistics for each column, enabling segment pruning during query execution. Statistics include:
| Statistic | Description | Use Case |
|---|---|---|
min | Minimum value in segment | Range predicate pruning |
max | Maximum value in segment | Range predicate pruning |
null_count | Count of null values | IS NULL/IS NOT NULL optimization |
is_sorted | Whether segment is sorted | Binary search for point lookups |
is_constant | Whether all values are identical | Immediate value return |
When a query includes a WHERE clause, Spice Cayenne evaluates whether each segment could contain matching rows. Segments that cannot match based on min/max statistics are skipped entirely, similar to DuckDB's zone-maps (opens in a new tab) without requiring explicit index creation.
Example - Segment Pruning:
For a table with segments containing timestamp ranges [2024-01-01, 2024-01-15], [2024-01-16, 2024-01-31], [2024-02-01, 2024-02-15], a query:
Prunes the first segment (max < 2024-01-20) and reads only the second and third segments.
Fast Random Access Encodings:
Vortex encodings support direct random access to compressed data:
Compute Push-Down:
Vortex supports executing filter and compute operations directly on compressed data, avoiding full decompression for predicate evaluation. This compute push-down reduces CPU and memory overhead by processing data in its compressed form:
| Encoding | Data Type | Operations |
|---|---|---|
| FSST | Strings | Equality, prefix matching on compressed symbols |
| FastLanes | Integers | SIMD-accelerated comparison on bit-packed data |
| ALP | Floats | Range comparisons with minimal decompression |
| Dictionary | Any | Lookup predicates evaluated on dictionary indices |
| RLE | Any | Constant runs evaluated once per run |
Array-level statistics (is_sorted, is_constant, min, max) enable additional optimizations beyond filtering. For example, is_sorted enables binary search for point lookups, and is_constant returns values immediately without scanning.
Performance Characteristics:
For point lookups and selective queries, Spice Cayenne with Vortex often matches or exceeds the performance of traditional B-tree indexes while consuming no additional memory for index structures. Performance scales with:
refresh_mode: changes)Spice Cayenne is the recommended accelerator for Change Data Capture (CDC) (opens in a new tab). A dataset configured with refresh_mode: changes is bootstrapped from a source snapshot and then kept continuously in sync by applying the source's row-level change stream — inserts, updates, and deletes — so the acceleration reflects the current state of the source row-for-row.
CDC into Cayenne is available for every Spice CDC connector:
See Change Data Capture (opens in a new tab) for source-side setup and the Changes refresh mode (opens in a new tab) reference for the acceleration-side configuration.
Cayenne is not the only engine that can apply a change stream — arrow, duckdb, and sqlite are also valid changes sinks — but Cayenne adds CDC-specific capabilities built for large-scale, continuously-updated datasets:
maintained_aggregates are updated in place as each change event is applied, rather than recomputed, so GROUP BY rollups stay fresh at CDC speed. See Maintained Aggregates (opens in a new tab).cayenne_deletion_mode: auto resolves to key-based deletion vectors so source DELETE and UPDATE events apply efficiently. See Deletion Vectors (opens in a new tab).cayenne_cdc_durability and the cayenne_cdc_mem_tier_* parameters) for low apply latency, with exactly-once semantics via primary-key-idempotent replay. The deployment guide (opens in a new tab) documents the CDC apply metrics for monitoring lag and throughput.runtime.params.target_replication_lag, runtime.params.target_freshness) and adjust its apply behavior to meet it. See Goal-driven tuning (opens in a new tab).Cayenne caches the scan input a query reads — the merged view of files, in-memory rows, and deletions — and shares one build across concurrent scans. How long a cached view is reused is decided per dataset:
refresh_mode: full, append, snapshot or caching, and a changes dataset that also takes INSERT/UPDATE/DELETE — reuses its cached view until a write invalidates it. Every write, delete, refresh and snapshot flip invalidates, so a query issued after a write always observes it; there is no staleness window. A capture that races a write is discarded and retaken rather than published.refresh_mode: changes dataset — including the implicit changes mode of cdc: and debezium: sources — reuses its view for up to 1 second (CAYENNE_SCAN_VIEW_FRESHNESS_MS), so a burst of CDC applies shares one build instead of forcing a rebuild per apply. A query may therefore see the table as it stood up to that long ago. Set CAYENNE_SCAN_VIEW_FRESHNESS_MS=0 to recapture on every scan.Schema evolution and retention invalidate cached views in both modes, so neither can serve a view built against a schema or a file set that no longer exists.
changes mode Cayenne always applies an inferred or declared primary key and routes source updates through an upsert. Declare primary_key on the dataset; change streams on Cayenne do not need on_conflict. The per-connector CDC pages document each connector's key requirement.mode: file. mode: memory is supported for ephemeral, in-RAM CDC where the accelerator is rebuilt from the source on restart.Spice Cayenne can incrementally maintain aggregate views over a CDC-accelerated dataset so that GROUP BY aggregate queries are answered from continuously-updated summary state instead of re-scanning the full table. As change events are applied (refresh_mode: changes), each maintained view is updated with the incoming deltas; a matching query is then rewritten by the physical optimizer to read the maintained state directly. When a query does not match a declared view — or the view is not currently fresh — Cayenne transparently falls back to a full scan, so results are always correct.
Maintained aggregates are declared per dataset under acceleration.maintained_aggregates:
Each entry declares one maintained view:
| Field | Required | Description |
|---|---|---|
group_by | Optional | Columns used as the GROUP BY key, in query output order. Omit for a single-group (grand total) view. |
aggregates | Required | Aggregate expressions maintained for each group, in query output order. |
filter_sql | Optional | A SQL row predicate (a WHERE expression over the dataset's columns, e.g. ol_delivery_d > '2007-01-02') restricting which rows contribute to the view. |
Each item in aggregates has:
| Field | Required | Description |
|---|---|---|
function | Required | One of count, sum, avg, min, or max. |
column | Conditional | The column to aggregate. Omit for count (COUNT(*)); required for sum, avg, min, and max. |
Supported input column types by function:
count — any column (or COUNT(*) when column is omitted).sum and avg — signed integers (Int8–Int64), unsigned integers (UInt8–UInt64), floats (Float32/Float64), and Decimal128. For integer and float inputs, sum widens to BIGINT/Float64 and avg returns Float64. For a Decimal128(p, s) input, sum returns Decimal128(min(38, p + 10), s) and avg returns Decimal128(min(38, p + 4), min(38, s + 4)), matching DataFusion's exact decimal return types (avg requires a non-negative scale). Decimal256 is not supported.min and max — signed/unsigned integers, Date32/Date64, Timestamp, and Decimal128, preserving the input type (MIN(Int32) -> Int32). Float min/max is not yet supported.A query is served from a maintained view only when it matches the view exactly:
GROUP BY keys match the view's group_by, in order. A query without GROUP BY, such as SELECT COUNT(*), SUM(amount) FROM orders, matches a view that omits group_by.SUM(amount * 2), is not served. COUNT(*) and COUNT of a non-NULL literal such as COUNT(1) count every row and match a count declared without a column. An aggregate call with DISTINCT, a FILTER (WHERE ...) clause, or an ORDER BY inside it is not served, and neither is a query that uses GROUPING SETS, ROLLUP, or CUBE.WHERE predicate matches the view's filter_sql. An unfiltered view (no filter_sql) answers only unfiltered queries, and a filtered view answers only a query whose predicate has the same conjuncts. A view whose filter_sql mirrors a common dashboard filter lets that filtered analytical query be served incrementally instead of by a full re-scan.Predicates are compared as sets of AND-ed conjuncts, so the order of the conjuncts does not matter. A query with an extra conjunct, or with one fewer, does not match. The comparison holds whether the query planner keeps the WHERE above the scan or pushes it into the scan. Before comparison, filter_sql is simplified the same way the planner simplifies a query's WHERE clause: a string compared with a timestamp column becomes a timestamp literal, and BETWEEN becomes two comparisons. Writing filter_sql exactly as the query writes its WHERE clause gives the most reliable match. A query is not served when its predicate calls a volatile function such as random(), when a join pushes a runtime filter into the dataset's scan, or when a LIMIT in a subquery limits the rows the aggregate reads.
A view also answers only when its maintained state is at the same table snapshot the query's scan reads, so a served result never includes changes that other scans in the same query do not see. Views are updated in the background after each write is published. A query planned before that update completes, or a query on a read-only refresh_mode: changes dataset that reuses a cached scan view captured before the latest update (see Visibility of applied changes (opens in a new tab)), reads the base table instead.
Any query that does not match a declared view (or that reaches the view while it is stale) falls back to the base-table scan — correct, but not accelerated.
The matching rules are implemented in MaintainedAggregateView::matches_query (opens in a new tab) and plan_relation_predicate (opens in a new tab).
refresh_mode: changes).min and max are retraction-hard: they require a primary_key on the acceleration so that UPDATE and DELETE changes can retract a prior extremum. Set acceleration.primary_key, enable extended schema inference for a source primary key, or omit min/max. count, sum, and avg do not require a primary key.partition_by).On a dataset with a primary key, Cayenne keeps a retraction index: for each row that at least one view selects (every row, when a view has no filter_sql), the group the row joined in each view and the values it contributed, so an UPDATE or DELETE can subtract exactly what the row added. One index serves every view on the dataset. The distinct-value state that min and max keep counts toward the same budget.
Each view also keeps one entry per group: the group's key and its running aggregates. This group state is allocated outside the query pool and is not counted toward the budget below, so its memory grows with the number of distinct groups the view holds.
Each dataset's budget is 10% of the query memory pool set by runtime.query.memory_limit (opens in a new tab), with a floor of 8 MiB that never exceeds the pool itself. When the query pool has no limit, the budget is 512 MiB. The index is allocated outside the query pool, so it adds to the memory described in What the Memory Limit Does Not Cover (opens in a new tab), and each dataset with maintained aggregates has a budget of its own (see maintained_aggregate_max_index_bytes (opens in a new tab)).
When the retained state, the retraction index plus the min and max distinct-value state, grows past its budget, the dataset's views go stale, queries run on base-table scans, and the runtime logs a warning. For a dataset named orders on a 4 GiB query pool, the line looks like this:
The entry count, sizes, and epoch are the dataset's own. To make the retained state fit, narrow the view with filter_sql, or give the query pool more memory. The retained state lives outside the query pool, so a larger budget also raises the process's total memory. When runtime.query.memory_limit is unset, give the container more memory and let the runtime derive a larger pool. Raising an explicit limit, as the log line suggests, needs matching container headroom, or the process can be killed for running out of memory (see Tuning the Memory Limit Safely (opens in a new tab)).
Cayenne rebuilds stale views from a scan of the table when the dataset opens and after in-memory CDC tier checkpoints: at the first checkpoint after the views go stale, then at every 32nd. Without in-memory tier checkpoints, stale views are rebuilt only when the dataset next opens, such as after a runtime restart.
Writes that land while a rebuild reads the table are held and applied once it finishes, so a rebuild completes on a dataset that is written continuously. A rebuild is abandoned and retried later when the held writes exceed a quarter of the budget.
A rebuild whose state still does not fit the budget is retried at the next interval. A rebuild that fails with an error, such as an error reading the table's files during its scan, is attempted three times and then stops until the runtime restarts. A filter_sql that cannot be planned against the table, or that is not a Boolean predicate, is rejected when the dataset loads, and the dataset fails to load.
When a rebuild succeeds after the views went stale, the runtime logs Maintained aggregate state rebuilt after staleness; queries are served from maintained state again (see try_rearm_maintained_aggregates (opens in a new tab)).
To keep a view's declaration in configuration without materializing or maintaining it, use the policy form with mode: disabled:
When any view is declared with the list form above, maintenance is enabled by default; the policy form's mode: enabled is equivalent.
Spice Cayenne implements efficient deletes without rewriting data files using deletion vectors. Deletion vectors track which rows have been logically deleted, and the information is applied transparently during query execution.
How deletions are recorded and applied is controlled by the cayenne_deletion_mode parameter:
| Mode | How deletes are applied |
|---|---|
auto (default) | Chooses per table. A primary-key table uses key deletes, except that a small table on local disk uses position deletes while it stays within a size bound. A table without a primary key uses position, since it has no key to record. See How auto chooses a mode (opens in a new tab). |
position | Per-file row-position RoaringBitmaps are pushed into the Vortex scan, skipping deleted rows at the storage layer with no per-row CPU cost. |
key | Deletes are applied above the Vortex scan via a per-row probe on the byte representation of the primary key columns. This is the recommended mode for primary-key tables under continuous writes, because compaction can keep running while writes continue. |
Key-delete compaction runs concurrently with writers, and the in-memory key deletion index is pruned in the background so it stays bounded. Setting cayenne_deletion_mode: position on a primary-key table opts into merge-on-read position deletes at any table size. With that explicit setting, compaction must take the table's write lock, so a continuous writer can block it on every attempt, and the table's file count grows until writes pause. The runtime then logs a warning that begins Protected-snapshot compaction is being starved.
auto chooses a modeFor a table with a primary_key, auto uses key deletes unless the table is small enough that rewriting it whole is cheaper than tracking its deletes. A primary-key table starts on position deletes only when all of the following hold:
runtime.cpu.cores (opens in a new tab), a cgroup CPU limit, or the CPUs the process may run on), in physical cores: with two SMT threads per core, 8 vCPUs count as 4. When the physical core count cannot be determined, auto uses key deletes.mode: memory table, a partitioned table, a table on object storage, a refresh_mode: changes dataset, and a cold-tier (opens in a new tab) table always use key deletes.cayenne_target_file_size_mb, and never more than 8 MiB. With the default 256 MB target the bound is 8 MiB; with a 64 MB target it is 2 MiB.While a table is on position deletes, Cayenne reclaims its deletes by rewriting the whole table alongside writers, so scans read one compact snapshot with no deletion filter. Each auto table spends at most a tenth of its time on whole-table rewrites, and a process-wide budget limits how many tables rewrite at once. Neither limit applies when a table's deletion index is over its memory ceiling: the rewrite then runs right away, without waiting for the pace or the budget, because the rewrite is what bounds the index's memory.
Once the table grows past its bound, it moves to key deletes until it next loads. Queries see every write before and after the change. When whole-table rewrites had kept the table compacted before the move, the runtime logs an INFO message that begins Dataset '<name>' grew past and names the bound. A table already past its bound when it loads starts on key deletes, as does one whose size on disk cannot be read at load, which the runtime reports with a warning that begins Failed to read the size on disk of dataset.
No parameter sets the bound directly. A smaller cayenne_target_file_size_mb lowers it, and a larger one does not raise it past 8 MiB. To use one mode at any size, set cayenne_deletion_mode: key or cayenne_deletion_mode: position. The Cayenne auto-tuned config line the runtime logs for each table reports the choice and its reason as deletion_mode, for example auto (position deletes up to 8 MiB, then key deletes), auto (key deletes: 4 physical cores, fewer than 8), or key (auto: CDC dataset).
:::note[Change from v2.4.0-rc.1 and earlier]
In v2.4.0-rc.1 and earlier releases, auto resolved to key only for a CDC dataset (refresh_mode: changes) or a cold-tier (opens in a new tab) table that has a primary key, and to position for every other table. A primary-key table that relied on the default, such as an append or full table, now uses key deletes unless it qualifies for position deletes as described in How auto chooses a mode (opens in a new tab). Position deletion vectors already on disk continue to load and apply after the mode changes. To keep position deletes at any size, set cayenne_deletion_mode: position explicitly.
:::
Under position deletes (a table without a primary key, a small primary-key table under auto, or an explicit cayenne_deletion_mode: position):
RoaringBitmap for memory-efficient storage of deleted row IDs, providing 50-90% memory savings compared to HashSet for sparse deletions.row_idx() read-back after each write, with a key-based fallback for any row whose position is not yet known. Pushing the deletes into the scan eliminates the per-row RowConverter deletion tax above it.Key-based deletion (cayenne_deletion_mode: key) uses the byte representation of primary key columns and applies deletes above the scan. This approach is position-independent and survives data reorganization.
On a primary-key upsert table, a deletion vector becomes orphaned once time-based retention empties the snapshot(s) it shadowed — the surviving-sequence floor rises above the DV's delete sequence, so the DV is a query-time no-op that still occupies disk and the in-memory deletion cache. Cayenne runs a background sweep that reclaims these orphaned .arrow files and their catalog rows.
The sweep is always on and requires no configuration:
retention_period + time_column) empties the snapshots it shadowed. Without a retention policy nothing is ever orphaned and the sweep is a no-op, at zero ingest cost.The sweep bounds only the orphaned tail. The live (not-yet-orphaned) DV set is bounded separately by compaction.
For tables with a single-column Int64 primary key, Cayenne uses an optimized direct lookup strategy that avoids serialization overhead:
Segment statistics prune a selective query well, but they are not a row address: when a query pins an exact key the table is not clustered on, most files remain candidates and the scan opens them. A dataset's indexes (opens in a new tab) — the same acceleration field DuckDB, SQLite, Turso and PostgreSQL accept — gives a Cayenne table one secondary index per entry, in both mode: file and mode: memory:
There are no Cayenne-specific parameters for this. The indexes entries are read on every registration, so adding, changing or removing one takes effect the next time the dataset is registered. In mode: file the index itself is persisted with the table, so a restart does not rebuild it (see Persistence in mode: file below).
When an index is used. For a lookup whose filters pin every column of one index to one or more literals, compared against the bare column: an equality, an IN list, or an OR of equalities. WHERE tenant_id = 7 AND service_id IN ('a', 'b') probes two key tuples. The index takes every combination of the columns' values, up to 2,048 tuples. A cast on the column side (CAST(score AS BIGINT) = 5, which also holds for 5.2), a range the query writes itself (BETWEEN, >= and <=), a negated list, or more than 2,048 tuples scans as before. A cast on the value side is evaluated, not stripped.
Join lookups in mode: file. A hash join whose build side is collected in full (CollectLeft) can also probe an index on the other table with the build side's join keys, once the build side is complete. A single-column join probes its scalar keys, and a composite join probes key tuples that pin every column of one index. The table scans normally instead when the hash join is partitioned, when the build side has more than 2,048 distinct keys, or when the keys resolve to more candidate rows than a bounded budget. mode: memory does not use indexes for joins.
What it changes. Only what is read. The index returns candidate rows; every original predicate and join still runs on them and LIMIT still applies above the scan, so an index can never add or hide a row. Postings are non-unique — uniqueness is never assumed — and in mode: file a candidate row selection is intersected with the table's deletion vectors, so a deleted row is never selected.
Index column types. Integer, decimal, floating-point, string, binary, boolean, date, time, timestamp, duration and interval columns can be index columns, and a dictionary-encoded column is indexed by its value type. A float is indexed by its bits after -0.0 is given the bits of 0.0 and every NaN the bits of one NaN, so values that compare equal share an index entry. A column of another type, such as an array, is rejected at registration when an indexes entry names it. When a dataset declares no indexes, schema inference copies the source's secondary indexes into the acceleration; an inferred index on a column Cayenne cannot index is skipped instead, and the dataset loads with the remaining inferred indexes. The runtime logs one warning per skipped index, naming the dataset, the index columns, and the column's type. Equality lookups on those columns scan the table.
unique builds an index but constrains nothing. Both enabled and unique build the same structure. A unique entry does not reject duplicate rows, and registration warns:
Dataset '<name>' (cayenne): a `unique` entry in `indexes` speeds up lookups but does not constrain writes, so duplicate rows are not rejected. Set `primary_key` with `on_conflict` to deduplicate on a column set.
Freshness in mode: file. Each write indexes the rows it writes, so the index follows appends, compactions, overwrites and full refreshes without a rebuild. The index covers the files of the current snapshot and of every protected snapshot, so an upsert table's recent rows are found through it. A file no index run covers is read in full. That happens for an append of more than 2^20 rows until its index finishes in the background, and for an append whose indexing fell more than 256 MiB behind the write. A single background build indexes those files; builds run one at a time and are paced (at least max(1s, 10 × the previous build) apart, backing off after a build that publishes nothing, capped at one hour). A compaction, overwrite or full refresh publishes its index before its files become visible, so it never leaves files uncovered. A schema change that alters a key column's type or nullability in place, such as on_schema_change: sync_all_columns widening an integer key to a float, resets that index, and lookups read every file until the background build covers them again. In mode: memory each memory-tier segment carries its own index, so no rebuild is needed.
Persistence in mode: file. Each write's index is also written in the background to the table's directory, under _lookup_index/, and registered in the Cayenne metastore. When the dataset is registered again after a restart, or is restored from an acceleration snapshot (opens in a new tab), the table loads the persisted index instead of reading its files back to rebuild it, and logs:
Dataset '<name>' (cayenne): loaded its secondary index from disk (<size> MiB), covering all <n> of its files
When some files have no persisted index, such as files written just before the runtime stopped, the message ends with covering <c> of its <n> files; the other <m> are indexed in the background. The table indexes those files before it serves queries, so when that build succeeds its first lookups read no file in full. This applies at startup and when refresh_mode: snapshot reopens a table before swapping it in, while the previous table keeps serving. A table whose persisted index covers every file starts no build and is not delayed. A persisted index file that is missing or unreadable is discarded, and its files are indexed again before the table is served. Files that build cannot index, for example when the query memory pool refuses it or a read fails, are read in full and indexed in the background.
An acceleration snapshot (opens in a new tab) first persists every published index run, so a snapshot taken immediately after a refresh, such as with snapshots_trigger: refresh_complete, carries the index for that refresh's files. If a run cannot be written, the snapshot keeps the runs already persisted, and the restored table indexes the rest when it opens. A snapshot does not wait for an index run still being finished in the background, such as a large append's, so a dataset restored from it indexes those files when it opens and reads in full any file that build cannot index. If the query memory pool cannot fit the persisted index when the table opens, the table reads its files in full and keeps the persisted index for a later start. Removing every indexes entry deletes the persisted index the next time the dataset is registered. cayenne_data_dir_bytes{kind="lookup_index"} reports the index's size on disk. mode: memory persists nothing, so its index is built again when the dataset reloads.
Memory. Index bytes are admitted against the query memory pool (runtime.query.memory_limit (opens in a new tab)) and reported on cayenne_memory_account_bytes{kind="lookup_index"}. When the pool cannot fit an index, mode: file reads the files that index would cover in full and mode: memory reads that batch whole; results never change.
Observing it. EXPLAIN reports the decision on CayenneAccelerationExec. lookup_index names the index that served the lookup, or is none. Beside an index, candidate_files is the number of files the lookup reads, uncovered_files is how many of them are read in full because no index run covers them yet (0 when the index covers everything the lookup reads), and candidate_rows is the number of candidate rows. For example, lookup_index=(tenant_id, service_id), candidate_files=3, uncovered_files=2, candidate_rows=4. mode: memory reports candidate_batches and uncovered_batches over its in-memory batches instead. EXPLAIN shows only the planning-time decision, so a scan that a join lookup narrows at run time reports lookup_index=none. When an indexed table's lookup scans, lookup_index_reason says why:
lookup_index_reason | Why the lookup scanned | What to do |
|---|---|---|
no_key_pinned | The filters do not give every column of any index a value. | Filter on every column of an index, or add an index on the columns the query filters by. |
too_many_keys | The values combine into more than 2,048 key tuples. | Shorten the IN lists. |
too_many_candidates | A lookup of several key tuples matches more than 1,000,000 rows, so a scan is cheaper. | Add a more selective column to the index. |
value_not_indexable | A value cannot be cast to its key column's type. | Cast the literal to the column's type. |
Each lookup an index serves, including a join lookup, is counted on cayenne_lookup_index_probe_total{table, shape}. In mode: file, cayenne_lookup_index_files{table, shape, coverage} reports how many of the table's current data files each index covers (coverage="covered") and how many are still read in full (coverage="uncovered"). A fully built index reports uncovered as 0. The gauge has no series until a scan has listed the table's files.
A Cayenne table with a primary_key upserts on that key, with no on_conflict setting. Cayenne does not use on_conflict to resolve keys (see on_conflict is deprecated (opens in a new tab)). Upserts use sequence numbers (Iceberg-style ordering):
When a primary key is deleted and then re-inserted:
A Cayenne table with a primary_key keeps one row per key. Because Cayenne is the default engine (opens in a new tab), this also applies to an acceleration that sets no engine on Linux and macOS. When a write carries the same key more than once, or carries a key the table already stores, the dataset's configuration decides which version is kept:
| Dataset | Version kept for a repeated key in a refresh |
|---|---|
primary_key and a time_column that is not part of the primary_key, with refresh_mode: full or append | The newest by time_column. On equal times, the version that arrived last. A NULL time is older than any time. |
refresh_mode: changes | Each source change is applied in order, so a later change of a key supersedes an earlier one. |
Any other dataset with a primary_key, such as one with no time_column or with the time_column in its primary_key | The version that arrived last. |
Ordering by time_column applies to refreshes. A SQL INSERT, including one inside a BEGIN … COMMIT transaction, keeps the version that arrived last and replaces a stored row with the same key. An INSERT reports the rows it inserted or replaced. An UPDATE does not replace another row: one that would give a row a key another row keeps fails and changes nothing, as described below. Time ordering also does not apply when the time_column is part of the primary_key; that table keeps the version that arrived last.
Each write is resolved as a whole, not one record batch at a time, so a key repeated across record batches of one refresh is resolved like any other repeat. A refresh is one write, and so is each INSERT and each write statement inside a BEGIN … COMMIT transaction. Statements are resolved separately, not across the transaction.
Arrival order is the order rows reach the table. It follows the source order when the source is read as one partition, and it is not defined when the source is read in parallel, so the version kept without a time_column can differ between refreshes. Set a time_column for a reproducible result. The rules are implemented in key_conflicts.rs (opens in a new tab) (Cayenne) and latest_by_time.rs (opens in a new tab) (time ordering for a refresh).
At load, each Cayenne dataset with a primary_key logs the rule it applies. For example:
A refresh that orders versions by time_column must read that column. When the rows a refresh reads do not include it, for example because a refresh_sql (opens in a new tab) leaves it out, the refresh fails instead of falling back to the version that arrived last, and the acceleration keeps its previous data. The refresh log line names the column:
An append refresh reads only rows newer than the latest stored time, so a late update to an older row is not loaded unless refresh_append_overlap (opens in a new tab) re-reads it. A day-granular time_column (opens in a new tab) (Date32 or Date64) is the exception: the refresh re-reads the whole latest stored day, so a late row on that day is loaded without an overlap, and a late row on an earlier day still needs one. Within that window, an append compares each row it reads with the version already stored, so a late row never replaces a newer stored version. A dataset that keeps the newest version by time_column with refresh_mode: append and no refresh_append_overlap logs a warning at load, including for a day-granular column. An Iceberg source (opens in a new tab) follows its table snapshots and loads late rows without an overlap, so it does not log the warning unless the dataset computes columns, such as embeddings, over the source:
Rows a refresh or statement received but did not keep are counted in the dataset_acceleration_rows_superseded (opens in a new tab) metric. The reason label is older when a version with a later time_column was kept, and arrival when a version that arrived later was kept. A row that an append re-reads from the overlap window unchanged writes nothing and is counted as arrival.
An UPDATE that would give a row a key another row keeps fails and changes nothing, including in a partitioned acceleration. This covers a new key that matches a stored row the statement does not update, a key two updated rows would share, and a NULL key:
A partitioned acceleration (partition_by) stores each partition as its own table, so a write resolves repeated keys within each partition. When a partition column is not part of the primary_key, a key that lands in two partitions in one write is stored once in each. Include every partition column in the primary_key to avoid this.
cayenne_pk_conflict_detection: none (opens in a new tab) turns off conflict detection, so repeated keys are not resolved. A write to a stored key then adds a second row for that key instead of replacing it, and a durable write-back commit is not recorded for delivery to the source.
Registration refuses none on any Cayenne dataset that rewrites a stored key:
refresh_mode: changes, set explicitly or by the connector's default.write_mode: write_back (durable write-back (opens in a new tab)).write_mode: acceleration, which keeps writes in the accelerator.sink source, whose writes also stay in the accelerator.The unprefixed pk_conflict_detection parameter is checked too; when both are set, cayenne_pk_conflict_detection takes precedence. A read-only dataset takes no writes, so a read-only dataset with a full or append refresh still loads with none. Neither refresh mode is guaranteed to keep one row per key under none. Use it with a full refresh only when the source guarantees primary-key uniqueness. An append refresh also must not read a stored key again: it filters the source from the stored time_column high-water mark inclusively and drops only rows identical to stored ones, so a new version of a stored key, or a refresh without a time_column that re-reads the source, adds a second row for that key. Use none with an append refresh only when each key arrives once, with a time_column whose values only increase. The refusal names the parameter and what would go wrong for the dataset:
The setting is stored with a mode: file table, and a table created with none can hold more than one row per key. Changing it from auto to none is applied to the existing table in place. When a keyed table created with none is reopened with auto, a refresh_mode: full dataset with write_mode: write_through is rebuilt from the source on that start (see Changing settings on an existing acceleration (opens in a new tab)). Any other keyed table fails to load. A table without a primary key has no key to repeat, so it reopens with auto. The failure for a keyed table:
When a refresh_mode: full dataset's source cannot be read at that start, the dataset fails with a different error that says to set the parameter back to none or to restart once the source can be read.
To rebuild any other table, restart once with acceleration.mode: file_create, then set mode: file again. A write-back dataset also restarts with write_mode: write_through for that run, because write-back refuses file_create. Deleting the table's data directory is not enough, because the table's entry stays in the shared metastore. Rows that exist only in the acceleration are lost.
on_conflict is deprecatedacceleration.on_conflict is deprecated. Cayenne does not use it to resolve keys: the values drop, upsert, upsert_dedup, and upsert_dedup_by_row_id all behave as described above. A Cayenne dataset that sets on_conflict logs a warning at load, which names the change in the version kept when there is one:
The following behaviors differ from Spice v2.3 and earlier:
on_conflict: drop kept the first version of a key. Cayenne now keeps the newest or last-arriving version, and an INSERT of a stored key replaces the stored row instead of being ignored.on_conflict: upsert_dedup failed a write whose rows repeated a key with different values. Cayenne now keeps one version.UPDATE that moved a row onto another row's key, or set a key to NULL, could remove a row. It now fails and changes nothing.on_conflict on a read-write dataset kept its writes in the acceleration when the source accepts only reads. Set acceleration.write_mode: acceleration (opens in a new tab) instead.Durable write-back (opens in a new tab) and PostgreSQL (opens in a new tab), MySQL (opens in a new tab), and MongoDB (opens in a new tab) change streams need only primary_key, so on_conflict can be removed from them. Spice v2.4.0-rc.1 refused a write-back dataset without an on_conflict upsert on the primary key.
A mode: memory acceleration accepts the same DML as mode: file. The RAM mem-tier is the permanent store — nothing is ever checkpointed to Vortex — so writes and deletes are applied to that tier directly:
DELETE evaluates its predicate against the mem-tier and rebuilds it without the matching rows. An unfiltered DELETE FROM <table> purges the tier.INSERT appends to the tier. Where the acceleration declares a primary_key, the table keeps one row per key: an incoming row replaces a stored row with the same key, as described in Duplicate primary keys in one write (opens in a new tab).UPDATE combines the two, but it does not replace another row by key: an UPDATE that would give a row a key another row keeps fails and changes nothing.retention_sql (opens in a new tab) applies to the tier the same way. Each write queues a retention pass, and the pass deletes the rows its predicate matches from the tier, as it does in mode: file.
:::note
Memory-mode DML is available from v2.3.1. On v2.3.0, use mode: file for a Cayenne acceleration that is written to with INSERT, UPDATE, or a filtered DELETE.
:::
Spice Cayenne supports storing data files in AWS S3 Express One Zone (opens in a new tab) for single-digit millisecond latency, ideal for latency-sensitive query workloads that require persistence. Metadata remains on local disk for fast catalog operations while data files are stored in S3 Express One Zone.
S3 Express One Zone directory buckets provide:
Example 1 - Explicit bucket:
Example 2 - Auto-generated bucket with IAM role:
Example 3 - Explicit credentials:
S3 Express One Zone buckets use a specific naming format:
{base-name}--{zone-id}--x-s3{region-code}-az{number} (e.g., usw2-az1, use1-az4)spice-{app-name}-{dataset-name}--{zone-id}--x-s3The zone ID is automatically extracted from the bucket name to configure the correct endpoint.
S3 Express One Zone is available in select regions. Spice derives the region from the prefix of each zone ID and recognizes these prefixes:
| Zone ID Prefix | Region |
|---|---|
use1 | us-east-1 |
use2 | us-east-2 |
usw1 | us-west-1 |
usw2 | us-west-2 |
euw1 | eu-west-1 |
euw2 | eu-west-2 |
euw3 | eu-west-3 |
euc1 | eu-central-1 |
euc2 | eu-central-2 |
eun1 | eu-north-1 |
eus1 | eu-south-1 |
eus2 | eu-south-2 |
apne1 | ap-northeast-1 |
apne2 | ap-northeast-2 |
apne3 | ap-northeast-3 |
apse1 | ap-southeast-1 |
apse2 | ap-southeast-2 |
apse3 | ap-southeast-3 |
apse4 | ap-southeast-4 |
apse5 | ap-southeast-5 |
aps1 | ap-south-1 |
aps2 | ap-south-2 |
ape1 | ap-east-1 |
sae1 | sa-east-1 |
cac1 | ca-central-1 |
caw1 | ca-west-1 |
afs1 | af-south-1 |
mes1 | me-south-1 |
mec1 | me-central-1 |
ilc1 | il-central-1 |
A cayenne_s3_region value takes precedence over the derived region. For a zone ID whose prefix is not listed, set cayenne_s3_region; otherwise Spice reports an error that asks for it.
See AWS documentation for the complete list of S3 Express One Zone availability zones (opens in a new tab).
cayenne_file_path and the cayenne_s3_* parameters store Cayenne data files on an S3 Express One Zone directory bucket. A general-purpose S3 bucket is not accepted as cayenne_file_path.snapshots.location on standard S3, GCS, or ADLS so bucket or object replication and readers in another region can use the prefix. An Express directory bucket, and the cayenne_file_path / cayenne_s3_* settings, are the data tier only. See Snapshot location and the Cayenne data tier (opens in a new tab).cayenne_datalake_location (opens in a new tab) is a general-purpose S3 or S3-compatible prefix, separate from both the Express data tier and snapshots.location.cayenne_s3_client_timeout) and retries.cayenne_s3_zone_ids, Spice automatically creates the S3 Express directory bucket if it doesn't exist (requires appropriate IAM permissions).Cayenne can cascade data across three storage tiers — an in-RAM mem-tier, a local-disk warm tier, and an object-store cold tier — with each row living in exactly one tier. The cold tier is optional and disabled by default; it is enabled by setting cayenne_datalake_location (opens in a new tab). When unset, a table is warm-only and behaves byte-identically to before.
The cold tier lets a table grow beyond local NVMe capacity while keeping recent, hot data on fast local storage and graduating older data to cheaper, durable object storage — without sacrificing pushdown on the cold data. Promotion is incremental: unchanged cold files are carried forward. The cold prefix is standard object storage, separate from acceleration snapshots (opens in a new tab) (a full copy of the acceleration file on each write) and from the S3 Express One Zone data tier.
Promotion (write path). A dedicated background worker graduates the warm tier once the size or file-count threshold (cayenne_datalake_warm_max_bytes / cayenne_datalake_warm_max_files) is crossed, evaluated every cayenne_datalake_tiering_check_interval_ms. Promotion is incremental carry-forward: rather than re-materializing the whole table each cycle, it classifies the existing cold manifest into dirty files that may host a tombstoned key — using per-PK-column min/max rectangles from each file's persisted statistics, refined by a per-file PK bloom filter, and conservative in the safe direction (a false positive costs an extra rewrite; a missed tombstone is impossible) — and clean files that are provably untouched. Only the warm delta plus the dirty files are re-read (all deletes applied, one version per key), Z-order clustered for tight multi-column zone maps, and written as read-optimized Vortex at the larger cayenne_datalake_target_file_size_mb under a per-promotion prefix; clean files are carried forward by manifest reference and never re-read. The new files are atomically registered while the promoted warm files are cleared in a single transaction, so promotion cost tracks the changed data, not total table size.
Cross-tier scan (read path). Queries span all tiers and push filters, projection, and limits down to each. Cold files are pruned from statistics held in the metastore, so pruning requires no object-store round-trip on the query path. Each cold file's PK bloom lets an upsert's keyset rebuild answer cold-tier key existence without scanning the object store. A DELETE after promotion correctly hides a cold-resident row.
Physical GC. A periodic mark-and-sweep, rooted at the manifest, reclaims cold objects orphaned by carry-forward rewrites. It runs every cayenne_datalake_gc_interval_ms (default 5m), which doubles as the orphan grace period: an object no longer referenced by the manifest is deleted only after it has been observed orphaned for at least one interval.
Clustering. cayenne_cluster_by names the clustering key, and applies it to both warm-tier rewrites and datalake promotion, so the two tiers share one layout contract. When it is unset, clustering falls back to an operator-configured cayenne_sort_columns; when that is also unset, promotion clusters by the hottest columns observed in query pushdown filters (default-on adaptive layout), then by an inference-derived cayenne_sort_columns, and finally the primary key. Clustering on more than one dimension prunes far better than a single-column sort for selective queries on any clustering column.
The curve is a Hilbert interleave, not Z-order/Morton: consecutive points on a Hilbert curve are always spatially adjacent, whereas Morton jumps a whole quadrant at each boundary and every jump is a zone map stretched over everything it skipped. Each column's values are rescaled onto a common coordinate space from the table's maintained min/max statistics before interleaving — without that, a wide column (a microsecond timestamp) consumes every high-order round before a narrow one (a tenant id) contributes anything, and the key degenerates into a single-column sort. Clustering is a layout-quality property, never a correctness one.
A Cayenne table created through SQL uses the equivalent CREATE TABLE ... CLUSTER BY (column, ...) clause in place of the parameter. It accepts bare column names only — any other expression is a planning error naming the clause.
Only an operator-configured sort order shadows the observed filter columns. A sort order that schema inference (opens in a new tab) supplied — which for most CDC datasets resolves to the primary key — is tagged cayenne_sort_columns_origin: inferred and ranks below the observations, so the adaptive layout still clusters cold files for the queries the table actually receives rather than for inference's guess. It is used as the pre-observation fallback, before the primary key.
cayenne_deletion_mode: key. When cayenne_deletion_mode is unset (or auto), Cayenne auto-resolves it to key for a datalake-enabled table. An explicit cayenne_deletion_mode: position conflicts with the cold tier and is rejected at registration — position deletes are file-path scoped and cannot survive the warm→cold rewrite. Set cayenne_deletion_mode: key (or remove it), or remove cayenne_datalake_location.cayenne_datalake_location but declares no primary_key registers successfully with the datalake tier inactive — data is never promoted and remains in the warm tier (Cayenne logs a loud warning). Add a primary_key to activate the tier, or remove cayenne_datalake_location. Leaving PK-less tables inactive (rather than failing registration) lets a fleet-wide datalake location coexist with datasets that have no primary key.cayenne_datalake_tiering_check_interval_ms: 0 (which would disable the datalake tiering and garbage-collection loop) and cayenne_datalake_gc_interval_ms: 0 (which would collapse the GC grace period, risking deletion of an object a running query still reads) are rejected at registration. Leave them at their defaults or set a positive value.s3:// URL — a general-purpose S3 or S3-compatible bucket (for example MinIO via cayenne_datalake_s3_endpoint), authenticated independently through the cayenne_datalake_s3_* parameters. It is a separate store from the warm S3 Express One Zone tier and does not need to share its bucket. Local file:// cold locations are not supported in v1.refresh_mode: changes or refresh_mode: append; a full refresh re-materializes the whole table each cycle and is rejected.cayenne_datalake_location set does not create acceleration snapshots (opens in a new tab) and does not bootstrap from one. It loads from its source instead, and Spice logs a warning naming the dataset. A snapshot does not cover the cold tier: it refers to cold files that the instance that created it later deletes, and a copy restored from it would share that instance's cold prefix, where each instance's garbage collection deletes the other's files.Cayenne supports most Arrow data types. The accelerator data type table (opens in a new tab) lists storage representations and engine comparisons.
Int8, Int16, Int32, Int64, UInt*)Float32, Float64)| Original Type | Converted To | Notes |
|---|---|---|
Float16 | Float32 | Automatic conversion for Vortex compatibility |
:::note Tables created before v2.2.0
These tables continue to normalize timestamps to microseconds. Preserving the source unit requires
recreating the table with mode: file_create in an empty directory. The
on_schema_change (opens in a new tab) setting does not migrate existing timestamps.
:::
The following types require the unsupported_type_action parameter:
Interval typesDuration typesFixedSizeBinaryUnion typesRunEndEncodedunsupported_type_action options:
| Value | Behavior |
|---|---|
error | Fail with error (default) |
string | Convert to Utf8 string |
warn | Include as-is with warning (may fail on insert) |
ignore | Skip the column entirely |
Resource requirements for Spice Cayenne depend on dataset size, query patterns, and cache configuration.
Spice Cayenne manages memory efficiently through columnar storage and selective caching. Memory allocation should account for:
| Component | Default | Notes |
|---|---|---|
| Runtime overhead | ~500 MB | Fixed baseline for the Spice runtime |
| Footer cache | 50 MB | Unset by default (DataFusion file-metadata-cache default); increase for datasets with many files (1-10 KB per file) |
| Segment cache | 256 MB | Increase based on hot data volume |
| Query execution | Variable | Depends on query complexity and concurrency |
Example - Memory-constrained environment:
Spice Cayenne stores data in a columnar format optimized for analytical queries. Storage requirements include:
runtime.query.temp_directory; the in-memory CDC tier checkpoints to the data directory under memory pressureThe runtime warns at startup when the data or metastore volume has under 10% or under 2 GiB free, because a full data volume fails CDC ingestion. Store all three on local NVMe — see Storage Recommendations (opens in a new tab).
One metastore holds the catalog — manifests, snapshot pointers, and partition rows — for every Cayenne dataset sharing a cayenne_metadata_dir. Recreating a dataset deletes its data directory, so a metastore directory that resolves on or beneath that data directory would be deleted along with it, taking the catalog for unrelated datasets.
A file-mode Cayenne dataset whose resolved metastore directory falls inside its resolved data directory therefore fails to load, naming the dataset and both paths. With neither path set, the data directory is {spice_data_path}/{dataset_name}/ and the metastore is {spice_data_path}/metadata, so a dataset named metadata collides. An explicit cayenne_metadata_dir that resolves on or beneath a dataset's data directory collides the same way. Resolve it by pointing cayenne_metadata_dir outside the data directory, or by renaming the dataset.
An unset cayenne_metadata_dir is resolved per dataset, in this order:
cayenne_metadata_dir, when set.{cayenne_file_path}/metadata, when cayenne_file_path is a local filesystem path. A file:// URL counts as local. An s3:// URL does not.{spice_data_path}/metadata otherwise — no cayenne_file_path, or an object-store cayenne_file_path. The SQLite catalog cannot live on object storage, so that case keeps the metastore on local disk.A local cayenne_file_path therefore does not use the {spice_data_path}/metadata default. Its data directory is {cayenne_file_path}/{dataset_name}/ and its unset metastore is {cayenne_file_path}/metadata, which is the same directory when the dataset is named metadata.
File-mode Cayenne keeps one shared catalog in the process. Whichever Cayenne dataset initializes that catalog first supplies the directory, and every later Cayenne dataset uses that same catalog even when its own resolution names a different directory. A restart that opens the other directory finds an empty catalog: the Vortex files are still on disk, and the acceleration looks empty. Set the same explicit cayenne_metadata_dir on every Cayenne dataset so the directory does not depend on which dataset initializes first.
Datasets that set different cayenne_file_path values are refused at load unless every one of them sets that same cayenne_metadata_dir. Each local data root would otherwise resolve its own {cayenne_file_path}/metadata, and only the catalog opened first would be used.
Pointing cayenne_metadata_dir at an existing metadata location adopts that catalog in place. Pointing it elsewhere opens a new empty catalog without erroring — prior Vortex files remain on disk but no table resolves them, so the acceleration appears empty or from scratch. Stop Spice with a graceful shutdown (SIGTERM) before moving the catalog. Shutdown checkpoints the SQLite WAL (Metastore durability (opens in a new tab)), and copying or moving cayenne.db or its -wal/-shm sidecars while Cayenne is still writing can leave those files inconsistent. After shutdown, move cayenne.db together with the sidecars. Changing only the parameter leaves the files behind.
Paths are compared after ./.. are collapsed and symlinks are resolved, so neither hides an overlap, and a sibling that merely shares a name prefix (…/meta next to …/metadata) is not affected. Datasets whose data lives on object storage (for example an S3 Express cayenne_file_path) are exempt — the metastore is always local, so it cannot sit inside an object-store data path.
Several file-mode Cayenne datasets that set different cayenne_file_path values must also set the same cayenne_metadata_dir on every one of them. Without it, which metastore a restart opens depends on which data root is resolved first, and Cayenne can come up with an empty catalog while the data files are still on disk — the acceleration looks lost. The spicepod is rejected at startup, naming every dataset and the path it configured:
Datasets that all share one cayenne_file_path, or that leave it unset, are unaffected; a trailing / does not make two paths differ.
Before recreating or deleting a data directory, Spice checks for cayenne.db and its SQLite sidecars, including those belonging to other datasets. Deletion is refused if the directory contains a metastore, links directly to one, or has unreadable entries. Metastore files must reside outside data directories.
Query performance scales with available CPU cores. Vortex's columnar format supports parallel decompression and scanning across multiple threads. Allocate sufficient CPU for:
Cayenne supports serializable, gated transactions on accelerator-only Cayenne tables (a read-write dataset with acceleration.write_mode: acceleration (opens in a new tab), or a sink dataset) and on Cayenne tables configured for durable write-back. A client submits a single BEGIN … COMMIT SQL body — over the HTTP /v1/sql endpoint or FlightSQL — and every statement in the body commits atomically, or not at all:
How it works:
BEGIN (or START TRANSACTION) and whose last statement is COMMIT is executed as a transaction — there is no configuration flag to enable it. Every statement runs through the standard query path, so authorization, column masking, logging, and tracing apply to each.COMMIT. Each participant table's version is captured at BEGIN and re-validated at COMMIT using per-key optimistic concurrency control (OCC). If a participant changed since the transaction started, COMMIT fails with a retryable conflict (HTTP 409 on /v1/sql) — re-run the transaction against the latest committed state. Any statement error (including a failed gate) rolls back every staged write.assert(). The assert(<boolean expression>) function evaluates its argument at execution time. If the expression is false or NULL, the transaction aborts with assertion failed: gate expression was false or NULL. Use it to enforce invariants (for example, a sufficient balance) as part of the transaction. Comparison gates such as … >= cap are NULL-safe./v1/sql returns the final statement's result (for the canonical gate-plus-write shape, the last write's row-count summary), or COMMIT when the body produces no rows.Durable federated write-back: A Cayenne dataset configured with write_mode: write_back (opens in a new tab) and refresh_mode: changes (CDC) stages its committed writes to the accelerator and then reconciles them asynchronously back to the federated source by a per-table write-back worker. write_mode: write_back requires replication.enabled: true as an explicit opt-in to asynchronous source durability.
:::warning PostgreSQL is the only supported source
Reconciling a row to the source has to happen in one atomic step. Delivered as a separate DELETE followed by an INSERT, the standalone DELETE echoes back over the changes stream — the accelerator is CDC-fed from that same source — and removes the committed row from the accelerator; if the follow-up INSERT then fails, the write is silently gone from both sides.
The PostgreSQL connector (opens in a new tab) delivers each row as a single INSERT ... ON CONFLICT (<primary key>) DO UPDATE, so it is the only source durable write-back can be configured against today. Any other source is rejected at registration rather than accepted as a configuration that can lose a committed write:
Failed to register dataset
<name>(<connector>): durable write-back needs a source that can apply a delivered row in one atomic step, and the<connector>connector cannot yet.
Choose a different acceleration.write_mode (opens in a new tab). The full message suggests write_mode: write_through, which sends writes to the source directly, together with access: read if the source does not accept writes. Atomic delivery for other connectors is planned.
:::
Durable write-back requirements:
acceleration.primary_key (opens in a new tab) is required.acceleration.mode (opens in a new tab) must be file.cayenne_pk_conflict_detection (opens in a new tab) must stay at its default, auto.on_conflict is not required.Supported writes:
INSERT and UPDATE must run inside a transaction (opens in a new tab).DELETE and MERGE are not supported.Delivery is asynchronous. Failed deliveries and temporarily unreadable rows remain pending for retry; an unreadable row does not trigger a source deletion. A persistent dataset_acceleration_write_back_pending_keys (opens in a new tab) backlog indicates a delivery problem.
Before a source deletion, writes must stop and pending keys must reach zero while write-back is enabled. Write-back can then be disabled and the source rows deleted, with CDC refreshing the accelerator. Disabling write-back resets the gauge without delivering pending rows.
Requirements and v1 limitations:
INSERT and UPDATE writes are supported inside a transaction. DELETE and MERGE are rejected.IS NOT NULL, IS NULL, COALESCE) are not yet reliable — prefer comparison gates.Consider the following limitations when using Spice Cayenne acceleration:
mode: memory (fully in-RAM, ephemeral) is supported alongside mode: file, but it does not persist any data (the dataset reloads from its source on restart), does not support partitioned tables (partition_by), and enforces a hard per-table RAM bound instead of spilling to disk — a breach returns an error rather than growing without limit. Use mode: file when persistence across restarts is required. DML and retention_sql (opens in a new tab) are not among the constraints — see Writes in memory mode (opens in a new tab).partition_by rebuilds a refresh_mode: full, write_mode: write_through dataset from its source on the next start. Any other mode: file dataset whose stored data was written with different partition columns fails to load, and changing its partition_by requires rebuilding the acceleration by hand, which can lose rows that exist only in Cayenne. See Changing partition_by after refresh (opens in a new tab).Interval, Duration, FixedSizeBinary, Union, and RunEndEncoded types require unsupported_type_action configuration.indexes builds a read-path secondary index (opens in a new tab), not a database index — a unique entry does not constrain writes, and registration logs a warning saying so. Deduplication requires only primary_key, with cayenne_pk_conflict_detection at its default auto. See Duplicate primary keys in one write (opens in a new tab).retention_sql (opens in a new tab) runs during maintenance after writes, full refreshes, and CDC checkpoints, independently of periodic retention settings. In mode: memory, it runs after writes, because that mode has no checkpoints.retention_period (opens in a new tab) hides expired rows in either mode. Scheduled deletion requires both retention_check_enabled: true (opens in a new tab) and retention_check_interval (opens in a new tab), which has no default. Without both, a warning is logged and storage is reclaimed only when compaction rewrites affected files.INSERT/UPDATE on non-partitioned, accelerator-only or durable write-back datasets, with one write per table. DELETE and MERGE are not supported.Complete example configuration using Spice Cayenne with performance tuning:
Spice Documentation:
External References: