title: 'Spice Cayenne Data Accelerator' sidebar_label: 'Spice Cayenne Data Accelerator' description: 'Spice Cayenne Data Accelerator (Vortex) Documentation' sidebar_position: 1 tags:
Spice Cayenne is a data acceleration engine designed for high-performance, scalable query on large-scale datasets. Built on Vortex (opens in a new tab), a high-performance columnar file format, Spice Cayenne combines columnar storage with in-process metadata management to provide fast query performance to scale to datasets beyond 1TB.
Spice Cayenne uses Vortex as its storage format, providing significant performance advantages:
Vortex is a Linux Foundation (LF AI & Data) project under Apache-2.0 license with neutral governance. For performance benchmarks, see bench.vortex.dev (opens in a new tab).
Spice Cayenne is recommended for any dataset of 10 GB or larger. DuckDB (opens in a new tab) suits smaller datasets, but its memory requirements grow faster: Cayenne typically needs one-third to one-half the memory of the DuckDB accelerator for the same dataset, and scales beyond the limits at which a single DuckDB file becomes impractical.
Spice Cayenne follows a lakehouse architecture inspired by DuckLake (opens in a new tab), separating metadata management from data storage:
Key Design Principles:
ListingTable at a unique directory, enabling append operations and parallel readsFor optimal performance, store Cayenne data files and the metastore on local NVMe storage, and point runtime.query.temp_directory at the same fast volume. What matters most is per-I/O latency rather than IOPS: Vortex's random-access reads are a chain of dependent segment reads and the metastore commits with fsync on every write, so the tens of microseconds an NVMe operation takes — against a millisecond or more on network storage — is multiplied along every query and every commit. Local NVMe on cloud instances is ephemeral, so pair it with acceleration snapshots (opens in a new tab) for fast cold starts.
Network block storage (Amazon EBS, Azure Managed Disks, GCP Persistent Disk) works as a durable fallback: Cayenne detects it as the network-attached storage tier and adapts — larger inline flushes, an O_DIRECT compaction writer, and a write-concurrency cap derived from the volume's bandwidth — but every cache miss still pays the volume's per-read latency, so prefer a sub-millisecond tier such as io2 Block Express. Network file systems (NFS, SMB, EFS, Azure Files) are not recommended: the metastore is a SQLite database, and SQLite locking is unreliable on them. See Storage (opens in a new tab) in the Cayenne performance guide and Storage (opens in a new tab) in the Performance Tuning guide.
Use S3 Express One Zone (opens in a new tab) when persistence of accelerations across restarts is required. S3 Express One Zone adds network latency compared to local NVMe but provides durability. Sharing accelerated data across multiple Spice instances is planned for a future release. Express One Zone is the Cayenne data tier (cayenne_file_path, cayenne_s3_*) only. Acceleration snapshots (opens in a new tab) use snapshots.location on standard S3, GCS, or ADLS, and the cold tier (opens in a new tab) uses cayenne_datalake_location on a general-purpose S3 or S3-compatible bucket. An Express directory bucket does not substitute for the snapshot bucket.
To use Spice Cayenne as the data accelerator, specify cayenne as the engine for acceleration. Spice Cayenne supports two storage modes:
mode: file (durable) — data is written as Vortex files on local disk or S3 Express One Zone, with a SQLite/Turso metastore, and the acceleration survives restarts. This is the recommended mode for Cayenne and is used in the examples throughout this page. The mode: file_create and mode: file_update variants control how an existing on-disk acceleration is reused or rebuilt on startup.mode: memory (ephemeral) — all data lives fully in RAM with an in-memory metastore; nothing is written to disk. The dataset is ephemeral and reloads from its source on restart (like the Arrow (opens in a new tab) accelerator). Memory mode works for all refresh modes (full/append/changes) and for both keyed and no-primary-key datasets, but does not support partitioned tables (partition_by), and it enforces a hard per-table RAM bound rather than spilling to disk (see cayenne_cdc_mem_tier_max_bytes (opens in a new tab)). INSERT, UPDATE and DELETE apply to the in-RAM tier the same way they apply to a mode: file acceleration — see .Spice Cayenne is configured through two distinct parameter scopes:
acceleration.params and control how that dataset's accelerated data is stored, compressed, written, and compacted.runtime.params and control engine-global behavior — caches, optimizer rules, and dedicated memory pools — shared by every Cayenne-accelerated dataset.The two scopes are not interchangeable: setting a runtime parameter under acceleration.params (or a per-dataset parameter under runtime.params) has no effect — the value is ignored.
acceleration.params)Set under a dataset's acceleration.params:
| Parameter | Description |
|---|---|
cayenne_tuning | Auto-tuning mode. Accepts auto or adaptive (preview). auto derives the memory-, CPU-, and storage-sensitive knobs statically from the detected environment (cgroup-aware cores and memory, storage class) and the inferred schema (cardinality, row width, primary key) — no feedback loop. adaptive additionally runs a per-table closed-feedback controller that measures the live CDC ingest rate (and delete fraction and arrival burstiness) and the runtime's response (apply latency vs. offered load, read amplification, memory pressure) and adjusts the inline-memtable flush caps, the in-memory CDC tier byte cap, compaction cadence/trigger, and write concurrency over time, within the environment-derived bounds. adaptive is reached only by setting it here — nothing else turns the closed loop on. An unset value, an unrecognized value (warned about, then treated as auto), inferred schema, and a configured cayenne_goal_* SLO all resolve to auto. adaptive needs a non-zero cayenne_compaction_background_interval_ms (the controller runs on the background compaction tick); if it is 0, Cayenne falls back to auto. Schema inference (always on) sharpens the warm-start but is not required — without inferred metadata the controller relearns the observed row width from live ingest. In both modes an explicit per-knob value overrides the derived value; under an explicitly-set knob is pinned (the loop will not move it). See . |
These are acceleration parameters (set under acceleration.params) used when storing Cayenne data files in S3 Express One Zone:
| Parameter | Description |
|---|---|
cayenne_s3_zone_ids | Comma-separated availability zone IDs (e.g., usw2-az1,usw2-az2). Auto-generates bucket names in format spice-{app}-{dataset}--{zone}--x-s3. |
cayenne_s3_region | AWS region (e.g., us-west-2). Auto-derived from zone ID if not specified. |
cayenne_s3_auth | Authentication method: iam_role (default) or key. |
cayenne_s3_key | AWS access key ID (required when cayenne_s3_auth: key). |
cayenne_s3_secret | AWS secret access key (required when cayenne_s3_auth: key). |
cayenne_s3_session_token | AWS session token (optional, for temporary credentials). |
cayenne_s3_endpoint | Custom S3 endpoint URL (optional, overrides auto-generated endpoint). |
These acceleration parameters (set under acceleration.params) configure the optional cold object-store tier (opens in a new tab). Setting cayenne_datalake_location enables the tier; the rest tune the clustering key, cold file size, the warm→cold promotion trigger, and the cold store's credentials. When cayenne_datalake_location is unset (the default), the cold tier is dormant and behaves byte-identically to a warm-only table.
| Parameter | Description |
|---|---|
cayenne_datalake_location | Object-store URL prefix for the cold tier (the bottom tier of the storage cascade). Must be an s3:// URL (e.g. s3://bucket/prefix) — a general-purpose S3 or S3-compatible bucket, distinct from the warm S3 Express One Zone store. When set, a background promotion stage graduates the warm local-disk tier to read-optimized, Z-order-clustered Vortex files on this store, and queries span the warm and cold tiers with per-tier pushdown. Unset (default) disables the cold tier. Requires key-based deletes (cayenne_deletion_mode auto-resolves to key; an explicit position is rejected at registration) and refresh_mode: changes or append. A primary_key is required to activate the tier — without one the tier registers but stays inactive. v1 constraints: local file:// locations are not supported; partitioned and position-delete tables are not supported. |
cayenne_cluster_by | Comma-separated columns to Hilbert-cluster both the warm and the datalake tier by, e.g. tenant_id,ts. Clustering tightens each file's per-column zone maps so selective queries on any clustering dimension prune at the storage layer. It is a complete layout instruction: while it is set, automatic and inferred sort columns are not applied, and it — registration fails until one is removed. Every column must exist and be a Boolean, numeric (except ), temporal, string, or binary column; a missing or unsupported column is rejected at registration. When unset, layout falls back to an operator-configured ; when that is also unset, to the hottest columns observed in query pushdown filters (default-on adaptive layout), then to an inference-derived (see ), and finally to the primary key when no usable observed column exists. Replaces the cold-tier-only , which is no longer accepted. |
The cold store is authenticated independently from the warm tier (which uses the cayenne_s3_* parameters) via the following acceleration.params:
| Parameter | Description |
|---|---|
cayenne_datalake_s3_auth | Authentication method for the cold S3 store: iam_role (default, uses environment/SDK credentials) or key. |
cayenne_datalake_s3_key | AWS access key ID for the cold store (required when cayenne_datalake_s3_auth: key). |
cayenne_datalake_s3_secret | AWS secret access key for the cold store (required when cayenne_datalake_s3_auth: key). |
cayenne_datalake_s3_session_token | AWS session token for the cold store (optional, for temporary credentials). |
cayenne_datalake_s3_region | AWS region of the cold bucket. Defaults to the environment region (AWS_REGION / AWS_DEFAULT_REGION), then us-east-1; inert for S3-compatible endpoints. |
cayenne_datalake_s3_endpoint | Custom S3 endpoint URL for the cold store (e.g. an S3-compatible store such as MinIO). http:// endpoints implicitly allow HTTP. |
runtime.params)Set once under the top-level runtime.params and applied to every Cayenne-accelerated dataset in the instance. With the exception of the goal-driven SLO setpoints (cayenne_goal_*) — which set a global default that a dataset can override under its own acceleration.params (except cayenne_goal_qph, which is global-only) — these are not valid under a dataset's acceleration.params.
A global cayenne_goal_* setpoint steers only the datasets that run the closed loop: it takes effect on a dataset that sets cayenne_tuning: adaptive and is ignored (with a warning) on every dataset left at the auto default.
| Parameter | Description |
|---|---|
cayenne_segment_cache_mb | Total size in megabytes of the in-memory Vortex segment cache, which holds decompressed data segments for faster reads. One cache serves every Cayenne table in the process — dataset accelerations and from: cayenne catalogs alike — so adding a table divides this pool instead of reserving another cache of its own. Takes a whole number of megabytes; 0 disables segment caching. When unset, derived as ~1/64 of the process's memory entitlement, clamped to 256 MB–2 GB. A non-numeric value (including auto) is warned about and falls back to the derived budget. |
cayenne_footer_cache_mb | Size of the engine-wide in-memory Vortex footer cache in megabytes. The footer cache stores Vortex file metadata (schemas, statistics, encoding information) and is shared across all Cayenne datasets. Larger values improve query performance for repeated scans. Optional; when unset, no explicit limit is applied and DataFusion's default file-metadata-cache limit of 50 MB applies (there is no fixed 128 MB default). |
cayenne_filter_propagation | Enables Cayenne's filter-propagation optimizer rules. Accepts enabled or disabled; defaults to disabled. |
cayenne_optimizer_rules | Selects which Cayenne optimizer rules run. Accepts auto (default — enables the recommended set, gated by ), , / , or a comma-separated list of individual rule names. |
Spice Cayenne is self-tuning by default and rarely needs manual tuning. For the full tuning reference — self-tuning and goal-driven SLOs (opens in a new tab), cache sizing (opens in a new tab), compression strategy (opens in a new tab), and file-size tuning (opens in a new tab) — see the dedicated Performance Tuning (opens in a new tab) page.
Spice Cayenne is DataFusion query-native, meaning all query execution uses Apache DataFusion (opens in a new tab) and adheres to the runtime.query.memory_limit setting. This provides:
DataFusion's GreedyMemoryPool (opens in a new tab) allows memory reservations on a first-come, first-served basis, improving throughput for high-concurrency queries with many partitions.
Spice Cayenne uses Vortex's advanced columnar format, which provides:
Vortex delivers 100x faster random access reads (opens in a new tab) compared to Apache Parquet through several architectural features:
Segment Statistics (Zone-Map Equivalent):
Vortex's ChunkedLayout (opens in a new tab) maintains per-segment statistics for each column, enabling segment pruning during query execution. Statistics include:
| Statistic | Description | Use Case |
|---|---|---|
min | Minimum value in segment | Range predicate pruning |
max | Maximum value in segment | Range predicate pruning |
null_count | Count of null values | IS NULL/IS NOT NULL optimization |
is_sorted | Whether segment is sorted | Binary search for point lookups |
is_constant | Whether all values are identical | Immediate value return |
When a query includes a WHERE clause, Spice Cayenne evaluates whether each segment could contain matching rows. Segments that cannot match based on min/max statistics are skipped entirely, similar to DuckDB's zone-maps (opens in a new tab) without requiring explicit index creation.
Example - Segment Pruning:
For a table with segments containing timestamp ranges [2024-01-01, 2024-01-15], [2024-01-16, 2024-01-31], [2024-02-01, 2024-02-15], a query:
Prunes the first segment (max < 2024-01-20) and reads only the second and third segments.
Fast Random Access Encodings:
Vortex encodings support direct random access to compressed data:
Compute Push-Down:
Vortex supports executing filter and compute operations directly on compressed data, avoiding full decompression for predicate evaluation. This compute push-down reduces CPU and memory overhead by processing data in its compressed form:
| Encoding | Data Type | Operations |
|---|---|---|
| FSST | Strings | Equality, prefix matching on compressed symbols |
| FastLanes | Integers | SIMD-accelerated comparison on bit-packed data |
| ALP | Floats | Range comparisons with minimal decompression |
| Dictionary | Any | Lookup predicates evaluated on dictionary indices |
| RLE | Any | Constant runs evaluated once per run |
Array-level statistics (is_sorted, is_constant, min, max) enable additional optimizations beyond filtering. For example, is_sorted enables binary search for point lookups, and is_constant returns values immediately without scanning.
Performance Characteristics:
For point lookups and selective queries, Spice Cayenne with Vortex often matches or exceeds the performance of traditional B-tree indexes while consuming no additional memory for index structures. Performance scales with:
refresh_mode: changes)Spice Cayenne is the recommended accelerator for Change Data Capture (CDC) (opens in a new tab). A dataset configured with refresh_mode: changes is bootstrapped from a source snapshot and then kept continuously in sync by applying the source's row-level change stream — inserts, updates, and deletes — so the acceleration reflects the current state of the source row-for-row.
CDC into Cayenne is available for every Spice CDC connector:
See Change Data Capture (opens in a new tab) for source-side setup and the Changes refresh mode (opens in a new tab) reference for the acceleration-side configuration.
Cayenne is not the only engine that can apply a change stream — arrow, duckdb, and sqlite are also valid changes sinks — but Cayenne adds CDC-specific capabilities built for large-scale, continuously-updated datasets:
maintained_aggregates are updated in place as each change event is applied, rather than recomputed, so GROUP BY rollups stay fresh at CDC speed. See Maintained Aggregates (opens in a new tab).cayenne_deletion_mode: auto resolves to key-based deletion vectors so source DELETE and UPDATE events apply efficiently. See Deletion Vectors (opens in a new tab).cayenne_cdc_durability and the cayenne_cdc_mem_tier_* parameters) for low apply latency, with exactly-once semantics via primary-key-idempotent replay. The deployment guide (opens in a new tab) documents the CDC apply metrics for monitoring lag and throughput.cayenne_goal_replication_lag, cayenne_goal_freshness) and adjust its apply behavior to meet it. See Goal-driven tuning (opens in a new tab).Cayenne caches the scan input a query reads — the merged view of files, in-memory rows, and deletions — and shares one build across concurrent scans. How long a cached view is reused is decided per dataset:
refresh_mode: full, append, snapshot or caching, and a changes dataset that also takes INSERT/UPDATE/DELETE — reuses its cached view until a write invalidates it. Every write, delete, refresh and snapshot flip invalidates, so a query issued after a write always observes it; there is no staleness window. A capture that races a write is discarded and retaken rather than published.refresh_mode: changes dataset — including the implicit changes mode of cdc: and debezium: sources — reuses its view for up to 1 second (CAYENNE_SCAN_VIEW_FRESHNESS_MS), so a burst of CDC applies shares one build instead of forcing a rebuild per apply. A query may therefore see the table as it stood up to that long ago. Set CAYENNE_SCAN_VIEW_FRESHNESS_MS=0 to recapture on every scan.Schema evolution and retention invalidate cached views in both modes, so neither can serve a view built against a schema or a file set that no longer exists.
changes mode Cayenne always applies an inferred or declared primary key and routes source updates through an upsert. Declare primary_key on the dataset (and, where the connector requires it, on_conflict: upsert); the per-connector CDC pages document the exact requirement.mode: file. mode: memory is supported for ephemeral, in-RAM CDC where the accelerator is rebuilt from the source on restart.Spice Cayenne can incrementally maintain aggregate views over a CDC-accelerated dataset so that GROUP BY aggregate queries are answered from continuously-updated summary state instead of re-scanning the full table. As change events are applied (refresh_mode: changes), each maintained view is updated with the incoming deltas; a matching query is then rewritten by the physical optimizer to read the maintained state directly. When a query does not match a declared view — or the view is not currently fresh — Cayenne transparently falls back to a full scan, so results are always correct.
Maintained aggregates are declared per dataset under acceleration.maintained_aggregates:
Each entry declares one maintained view:
| Field | Required | Description |
|---|---|---|
group_by | Optional | Columns used as the GROUP BY key, in query output order. Omit for a single-group (grand total) view. |
aggregates | Required | Aggregate expressions maintained for each group, in query output order. |
filter_sql | Optional | A SQL row predicate (a WHERE expression over the dataset's columns, e.g. ol_delivery_d > '2007-01-02') restricting which rows contribute to the view. |
Each item in aggregates has:
| Field | Required | Description |
|---|---|---|
function | Required | One of count, sum, avg, min, or max. |
column | Conditional | The column to aggregate. Omit for count (COUNT(*)); required for sum, avg, min, and max. |
Supported input column types by function:
count — any column (or COUNT(*) when column is omitted).sum and avg — signed integers (Int8–Int64), unsigned integers (UInt8–UInt64), floats (Float32/Float64), and Decimal128. For integer and float inputs, sum widens to BIGINT/Float64 and avg returns Float64. For a Decimal128(p, s) input, sum returns Decimal128(min(38, p + 10), s) and returns , matching DataFusion's exact decimal return types ( requires a non-negative scale). is not supported.A query is served from a maintained view only when it matches the view exactly:
GROUP BY keys match the view's group_by, in order.WHERE predicate matches the view's filter_sql exactly — an unfiltered view (no filter_sql) answers only unfiltered queries, and a filtered view answers only a query carrying the identical predicate. A view whose filter_sql mirrors a common dashboard filter lets that filtered analytical query be served incrementally instead of by a full re-scan.Any query that does not match a declared view (or that reaches the view while it is stale) falls back to the base-table scan — correct, but not accelerated.
refresh_mode: changes).min and max are retraction-hard: they require a primary_key on the acceleration so that UPDATE and DELETE changes can retract a prior extremum. Set acceleration.primary_key, enable extended schema inference for a source primary key, or omit min/max. count, sum, and avg do not require a primary key.partition_by).To keep a view's declaration in configuration without materializing or maintaining it, use the policy form with mode: disabled:
When any view is declared with the list form above, maintenance is enabled by default; the policy form's mode: enabled is equivalent.
Spice Cayenne implements efficient deletes without rewriting data files using deletion vectors. Deletion vectors track which rows have been logically deleted, and the information is applied transparently during query execution.
How deletions are recorded and applied is controlled by the cayenne_deletion_mode parameter:
| Mode | How deletes are applied |
|---|---|
auto (default) | Resolves to position (merge-on-read) for most tables. For CDC datasets (refresh_mode: changes) that declare a primary_key, auto resolves to key instead, so deletes compact concurrently with the continuous writer. |
position | Per-file row-position RoaringBitmaps are pushed into the Vortex scan, skipping deleted rows at the storage layer with no per-row CPU cost. |
key | Deletes are applied above the Vortex scan via a per-row probe on the byte representation of the primary key columns. The explicit opt-out from merge-on-read for primary-key tables. |
Under position mode (the auto resolution for all tables except CDC datasets with a primary key):
RoaringBitmap for memory-efficient storage of deleted row IDs, providing 50-90% memory savings compared to HashSet for sparse deletions.row_idx() read-back after each write, with a key-based fallback for any row whose position is not yet known. Pushing the deletes into the scan eliminates the per-row RowConverter deletion tax above it.Key-based deletion (cayenne_deletion_mode: key) uses the byte representation of primary key columns and applies deletes above the scan. This approach is position-independent and survives data reorganization.
On a primary-key upsert table, a deletion vector becomes orphaned once time-based retention empties the snapshot(s) it shadowed — the surviving-sequence floor rises above the DV's delete sequence, so the DV is a query-time no-op that still occupies disk and the in-memory deletion cache. Cayenne runs a background sweep that reclaims these orphaned .arrow files and their catalog rows.
The sweep is always on and requires no configuration:
retention_period + time_column) empties the snapshots it shadowed. Without a retention policy nothing is ever orphaned and the sweep is a no-op, at zero ingest cost.The sweep bounds only the orphaned tail. The live (not-yet-orphaned) DV set is bounded separately by compaction.
For tables with a single-column Int64 primary key, Cayenne uses an optimized direct lookup strategy that avoids serialization overhead:
Segment statistics prune a selective query well, but they are not a row address: when a query pins an exact key the table is not clustered on, most files remain candidates and the scan opens them. A dataset's indexes (opens in a new tab) — the same acceleration field DuckDB, SQLite, Turso and PostgreSQL accept — gives a Cayenne table one secondary index per entry, in both mode: file and mode: memory:
There are no Cayenne-specific parameters for this. Nothing about an index is persisted with the table: the entries are read on every registration, so adding, changing or removing one takes effect the next time the dataset is registered.
When an index is used. Only for a lookup whose filters pin every column of one index to a literal, compared against the bare column — WHERE tenant_id = 7 AND service_id = 'a'. A cast on the column side (CAST(score AS BIGINT) = 5, which also holds for 5.2), a range, an IN list or an OR scans as before. A cast on the value side is evaluated, not stripped.
What it changes. Only what is read. The index returns candidate rows; every original predicate still runs on them and LIMIT still applies above the scan, so an index can never add or hide a row. Postings are non-unique — uniqueness is never assumed — and in mode: file a candidate row selection is intersected with the table's deletion vectors, so a deleted row is never selected.
Index columns must have exact equality. Float16, Float32 and Float64 columns are rejected at registration (values such as signed zero have no single byte representation); use an integer, decimal, string, or another exact-equality column.
unique builds an index but constrains nothing. Both enabled and unique build the same structure. A unique entry does not reject duplicate rows, and registration warns:
Dataset '<name>' (cayenne): a `unique` entry in `indexes` speeds up lookups but does not constrain writes, so duplicate rows are not rejected. Set `primary_key` with `on_conflict` to deduplicate on a column set.
Staleness in mode: file. The index covers one snapshot. A full refresh publishes its index in the same fenced flip that makes the snapshot visible, so no lookup between the two falls back to a scan. Anything else that changes the files — an append, a compaction, a restart — leaves the index stale; it is dropped and the next lookup-shaped query claims a single background rebuild. Rebuilds run one at a time and are paced (at least max(1s, 10 × the previous build) apart, backing off after a build that publishes nothing, capped at one hour), so a table whose index cannot be built is not re-read on every lookup. In mode: memory each memory-tier segment carries its own index, so no rebuild is needed.
Memory. Index bytes are admitted against the query memory pool (runtime.query.memory_limit (opens in a new tab)) and reported on cayenne_memory_account_bytes{kind="lookup_index"}. When the pool cannot fit an index, mode: file publishes none and mode: memory reads that batch whole; results never change.
Observing it. EXPLAIN reports the decision on CayenneAccelerationExec as lookup_index, lookup_index_outcome, and — where they apply — candidate_files and candidate_rows. An indexed table whose predicate does not pin a complete key reports lookup_index=none, lookup_index_outcome=not_applicable. Each probe is also counted on cayenne_lookup_index_probe_total{table, shape, outcome}, with outcome one of selected, empty (no candidate for the key), unbuilt (no index covers the rows read) or snapshot_mismatch (the index no longer matches the table's files).
When on_conflict is configured, Cayenne supports upsert semantics using sequence numbers (Iceberg-style ordering):
When a primary key is deleted and then re-inserted:
With a primary_key, an append refresh checks keys as each batch of the write arrives. Duplicate keys inside one batch are collapsed. on_conflict: upsert keeps the last row, and on_conflict: drop keeps the first row and drops the later copies. upsert_dedup and upsert_dedup_by_row_id are also applied per batch. The same key in a later batch of that write is rejected, with or without on_conflict:
A key already stored in the acceleration is a separate case. upsert replaces the stored row, and drop drops the incoming row.
An append refresh hits the error when one poll's incoming rows contain two versions of a key in different batches. A cold load of a history or log does this, and so does a refresh_append_overlap (opens in a new tab) window that itself holds two source rows for the key. Append refresh drops a re-read that matches a stored row on every column before the write, so that re-read is not a second incoming copy. With on_conflict: upsert, a single changed version of a stored key replaces the stored row.
DuckDB (opens in a new tab) applies on_conflict: upsert across batches of one write. The end-to-end incremental ingestion example (opens in a new tab) uses DuckDB.
When the source is a log of versions, append without a primary_key and read the latest version from a view:
See QUALIFY (opens in a new tab) and Views (opens in a new tab). Each changed version is appended, and the view returns the row with the latest event_time for that key.
A mode: memory acceleration accepts the same DML as mode: file. The RAM mem-tier is the permanent store — nothing is ever checkpointed to Vortex — so writes and deletes are applied to that tier directly:
DELETE evaluates its predicate against the mem-tier and rebuilds it without the matching rows. An unfiltered DELETE FROM <table> purges the tier.INSERT appends to the tier. Where the acceleration declares a primary_key, incoming rows are validated against it as the input streams in, and conflicts resolve through on_conflict (opens in a new tab): upsert supersedes the existing row, while a primary_key with no on_conflict configured drops the conflicting incoming row instead.UPDATE combines the two, so both rules above apply.retention_sql (opens in a new tab) is the exception: it does not reach a mode: memory tier, and matching rows stay queryable (see Limitations (opens in a new tab)).
:::note
Memory-mode DML is available from v2.3.1. On v2.3.0, use mode: file for a Cayenne acceleration that is written to with INSERT, UPDATE, or a filtered DELETE.
:::
Spice Cayenne supports storing data files in AWS S3 Express One Zone (opens in a new tab) for single-digit millisecond latency, ideal for latency-sensitive query workloads that require persistence. Metadata remains on local disk for fast catalog operations while data files are stored in S3 Express One Zone.
S3 Express One Zone directory buckets provide:
Example 1 - Explicit bucket:
Example 2 - Auto-generated bucket with IAM role:
Example 3 - Explicit credentials:
S3 Express One Zone buckets use a specific naming format:
{base-name}--{zone-id}--x-s3{region-code}-az{number} (e.g., usw2-az1, use1-az4)spice-{app-name}-{dataset-name}--{zone-id}--x-s3The zone ID is automatically extracted from the bucket name to configure the correct endpoint.
S3 Express One Zone is available in select regions. Spice automatically derives the region from zone IDs:
| Zone ID Prefix | Region |
|---|---|
use1 | us-east-1 |
use2 | us-east-2 |
usw1 | us-west-1 |
usw2 | us-west-2 |
euw1 | eu-west-1 |
euw2 | eu-west-2 |
euw3 | eu-west-3 |
euc1 | eu-central-1 |
eun1 | eu-north-1 |
eus1 | eu-south-1 |
apne1 | ap-northeast-1 |
apne2 | ap-northeast-2 |
apse1 | ap-southeast-1 |
apse2 | ap-southeast-2 |
See AWS documentation for the complete list of S3 Express One Zone availability zones (opens in a new tab).
cayenne_file_path and the cayenne_s3_* parameters store Cayenne data files on an S3 Express One Zone directory bucket. A general-purpose S3 bucket is not accepted as cayenne_file_path.snapshots.location on standard S3, GCS, or ADLS so bucket or object replication and readers in another region can use the prefix. An Express directory bucket, and the cayenne_file_path / cayenne_s3_* settings, are the data tier only. See Snapshot location and the Cayenne data tier (opens in a new tab).cayenne_datalake_location (opens in a new tab) is a general-purpose S3 or S3-compatible prefix, separate from both the Express data tier and snapshots.location.cayenne_s3_client_timeout) and retries.Cayenne can cascade data across three storage tiers — an in-RAM mem-tier, a local-disk warm tier, and an object-store cold tier — with each row living in exactly one tier. The cold tier is optional and disabled by default; it is enabled by setting cayenne_datalake_location (opens in a new tab). When unset, a table is warm-only and behaves byte-identically to before.
The cold tier lets a table grow beyond local NVMe capacity while keeping recent, hot data on fast local storage and graduating older data to cheaper, durable object storage — without sacrificing pushdown on the cold data. Promotion is incremental: unchanged cold files are carried forward. The cold prefix is standard object storage, separate from acceleration snapshots (opens in a new tab) (a full copy of the acceleration file on each write) and from the S3 Express One Zone data tier.
Promotion (write path). A dedicated background worker graduates the warm tier once the size or file-count threshold (cayenne_datalake_warm_max_bytes / cayenne_datalake_warm_max_files) is crossed, evaluated every cayenne_datalake_tiering_check_interval_ms. Promotion is incremental carry-forward: rather than re-materializing the whole table each cycle, it classifies the existing cold manifest into dirty files that may host a tombstoned key — using per-PK-column min/max rectangles from each file's persisted statistics, refined by a per-file PK bloom filter, and conservative in the safe direction (a false positive costs an extra rewrite; a missed tombstone is impossible) — and clean files that are provably untouched. Only the warm delta plus the dirty files are re-read (all deletes applied, one version per key), Z-order clustered for tight multi-column zone maps, and written as read-optimized Vortex at the larger cayenne_datalake_target_file_size_mb under a per-promotion prefix; clean files are carried forward by manifest reference and never re-read. The new files are atomically registered while the promoted warm files are cleared in a single transaction, so promotion cost tracks the changed data, not total table size.
Cross-tier scan (read path). Queries span all tiers and push filters, projection, and limits down to each. Cold files are pruned from statistics held in the metastore, so pruning requires no object-store round-trip on the query path. Each cold file's PK bloom lets an upsert's keyset rebuild answer cold-tier key existence without scanning the object store. A DELETE after promotion correctly hides a cold-resident row.
Physical GC. A periodic mark-and-sweep, rooted at the manifest, reclaims cold objects orphaned by carry-forward rewrites. It runs every cayenne_datalake_gc_interval_ms (default 5m), which doubles as the orphan grace period: an object no longer referenced by the manifest is deleted only after it has been observed orphaned for at least one interval.
cayenne_deletion_mode: key. When cayenne_deletion_mode is unset (or auto), Cayenne auto-resolves it to key for a datalake-enabled table. An explicit cayenne_deletion_mode: position conflicts with the cold tier and is rejected at registration — position deletes are file-path scoped and cannot survive the warm→cold rewrite. Set cayenne_deletion_mode: key (or remove it), or remove cayenne_datalake_location.cayenne_datalake_location but declares no primary_key registers successfully with the datalake tier inactive — data is never promoted and remains in the warm tier (Cayenne logs a loud warning). Add a primary_key to activate the tier, or remove cayenne_datalake_location. Leaving PK-less tables inactive (rather than failing registration) lets a fleet-wide datalake location coexist with datasets that have no primary key.cayenne_datalake_tiering_check_interval_ms: 0 (which would disable the datalake tiering and garbage-collection loop) and cayenne_datalake_gc_interval_ms: 0 (which would collapse the GC grace period, risking deletion of an object a running query still reads) are . Leave them at their defaults or set a positive value.Cayenne supports most Arrow data types. The accelerator data type table (opens in a new tab) lists storage representations and engine comparisons.
Int8, Int16, Int32, Int64, UInt*)Float32, Float64)| Original Type | Converted To | Notes |
|---|---|---|
Float16 | Float32 | Automatic conversion for Vortex compatibility |
:::note Tables created before v2.2.0
These tables continue to normalize timestamps to microseconds. Preserving the source unit requires
recreating the table with mode: file_create in an empty directory. The
on_schema_change (opens in a new tab) setting does not migrate existing timestamps.
:::
The following types require the unsupported_type_action parameter:
Interval typesDuration typesFixedSizeBinaryUnion typesRunEndEncodedunsupported_type_action options:
| Value | Behavior |
|---|---|
error | Fail with error (default) |
string | Convert to Utf8 string |
warn | Include as-is with warning (may fail on insert) |
ignore | Skip the column entirely |
Resource requirements for Spice Cayenne depend on dataset size, query patterns, and cache configuration.
Spice Cayenne manages memory efficiently through columnar storage and selective caching. Memory allocation should account for:
| Component | Default | Notes |
|---|---|---|
| Runtime overhead | ~500 MB | Fixed baseline for the Spice runtime |
| Footer cache | 50 MB | Unset by default (DataFusion file-metadata-cache default); increase for datasets with many files (1-10 KB per file) |
| Segment cache | 256 MB | Increase based on hot data volume |
| Query execution | Variable | Depends on query complexity and concurrency |
Example - Memory-constrained environment:
Spice Cayenne stores data in a columnar format optimized for analytical queries. Storage requirements include:
runtime.query.temp_directory; the in-memory CDC tier checkpoints to the data directory under memory pressureThe runtime warns at startup when the data or metastore volume has under 10% or under 2 GiB free, because a full data volume fails CDC ingestion. Store all three on local NVMe — see Storage Recommendations (opens in a new tab).
One metastore holds the catalog — manifests, snapshot pointers, and partition rows — for every Cayenne dataset sharing a cayenne_metadata_dir. Recreating a dataset deletes its data directory, so a metastore directory that resolves on or beneath that data directory would be deleted along with it, taking the catalog for unrelated datasets.
A file-mode Cayenne dataset whose resolved metastore directory falls inside its resolved data directory therefore fails to load, naming the dataset and both paths. The stock defaults reach this without any operator error: the data directory defaults to {spice_data_path}/{dataset_name}/ and the metastore to {spice_data_path}/metadata, so a dataset named metadata collides. An explicit cayenne_metadata_dir set beneath a dataset's data directory collides the same way. Resolve it by pointing cayenne_metadata_dir outside the data directory, or by renaming the dataset.
Paths are compared after ./.. are collapsed and symlinks are resolved, so neither hides an overlap, and a sibling that merely shares a name prefix (…/meta next to …/metadata) is not affected. Datasets whose data lives on object storage (for example an S3 Express cayenne_file_path) are exempt — the metastore is always local, so it cannot sit inside an object-store data path.
Several file-mode Cayenne datasets that set different cayenne_file_path values must also set the same cayenne_metadata_dir on every one of them. Without it, which metastore a restart opens depends on which data root is resolved first, and Cayenne can come up with an empty catalog while the data files are still on disk — the acceleration looks lost. The spicepod is rejected at startup, naming every dataset and the path it configured:
Datasets that all share one cayenne_file_path, or that leave it unset, are unaffected; a trailing / does not make two paths differ.
Before recreating or deleting a data directory, Spice checks for cayenne.db and its SQLite sidecars, including those belonging to other datasets. Deletion is refused if the directory contains a metastore, links directly to one, or has unreadable entries. Metastore files must reside outside data directories.
Query performance scales with available CPU cores. Vortex's columnar format supports parallel decompression and scanning across multiple threads. Allocate sufficient CPU for:
Cayenne supports serializable, gated transactions on accelerator-only Cayenne tables and on Cayenne tables configured for durable write-back. A client submits a single BEGIN … COMMIT SQL body — over the HTTP /v1/sql endpoint or FlightSQL — and every statement in the body commits atomically, or not at all:
How it works:
BEGIN (or START TRANSACTION) and whose last statement is COMMIT is executed as a transaction — there is no configuration flag to enable it. Every statement runs through the standard query path, so authorization, column masking, logging, and tracing apply to each.COMMIT. Each participant table's version is captured at BEGIN and re-validated at COMMIT using per-key optimistic concurrency control (OCC). If a participant changed since the transaction started, COMMIT fails with a retryable conflict (HTTP 409 on /v1/sql) — re-run the transaction against the latest committed state. Any statement error (including a failed gate) rolls back every staged write.assert(). The assert(<boolean expression>) function evaluates its argument at execution time. If the expression is false or NULL, the transaction aborts with assertion failed: gate expression was false or NULL. Use it to enforce invariants (for example, a sufficient balance) as part of the transaction. Comparison gates such as are NULL-safe.Durable federated write-back: A Cayenne dataset configured with write_mode: write_back (opens in a new tab), on_conflict, and refresh_mode: changes (CDC) stages its committed writes to the accelerator and then reconciles them asynchronously back to the federated source by a per-table write-back worker. write_mode: write_back requires replication.enabled: true as an explicit opt-in to asynchronous source durability.
:::warning PostgreSQL is the only supported source
Reconciling a row to the source has to happen in one atomic step. Delivered as a separate DELETE followed by an INSERT, the standalone DELETE echoes back over the changes stream — the accelerator is CDC-fed from that same source — and removes the committed row from the accelerator; if the follow-up INSERT then fails, the write is silently gone from both sides.
The PostgreSQL connector (opens in a new tab) delivers each row as a single INSERT ... ON CONFLICT (<primary key>) DO UPDATE, so it is the only source durable write-back can be configured against today. Any other source is rejected at registration rather than accepted as a configuration that can lose a committed write:
Failed to register dataset
<name>(<connector>): durable write-back needs a source that can apply a delivered row in one atomic step, and the<connector>connector cannot yet.
Remove on_conflict to keep writes on the accelerator, or choose a different acceleration.write_mode (opens in a new tab). Atomic delivery for other connectors is planned.
:::
Durable write-back requirements:
acceleration.primary_key (opens in a new tab) is required.acceleration.mode (opens in a new tab) must be file.Supported writes:
INSERT and UPDATE must run inside a transaction (opens in a new tab).DELETE and MERGE are not supported.Delivery is asynchronous. Failed deliveries and temporarily unreadable rows remain pending for retry; an unreadable row does not trigger a source deletion. A persistent dataset_acceleration_write_back_pending_keys (opens in a new tab) backlog indicates a delivery problem.
Before a source deletion, writes must stop and pending keys must reach zero while write-back is enabled. Write-back can then be disabled and the source rows deleted, with CDC refreshing the accelerator. Disabling write-back resets the gauge without delivering pending rows.
Requirements and v1 limitations:
INSERT and UPDATE writes are supported inside a transaction. DELETE and MERGE are rejected.IS NOT NULL, IS NULL, COALESCE) are not yet reliable — prefer comparison gates.Consider the following limitations when using Spice Cayenne acceleration:
mode: memory (fully in-RAM, ephemeral) is supported alongside mode: file, but it does not persist any data (the dataset reloads from its source on restart), does not support partitioned tables (partition_by), and enforces a hard per-table RAM bound instead of spilling to disk — a breach returns an error rather than growing without limit. Use mode: file when persistence across restarts is required. DML is not among the constraints — see Writes in memory mode (opens in a new tab) — but retention_sql (opens in a new tab) is, as the next-but-one entry notes.Interval, Duration, FixedSizeBinary, Union, and RunEndEncoded types require unsupported_type_action configuration.indexes builds a read-path , not a database index — a entry does not constrain writes, and registration logs a warning saying so. Deduplication requires with .Complete example configuration using Spice Cayenne with performance tuning:
Spice Documentation:
External References:
adaptiveadaptivecayenne_goal_replication_lag | Goal-driven adaptive tuning (preview): target end-to-end CDC replication lag, as a duration (e.g. 5s). Best set globally under runtime.params; a value here overrides the global setpoint for this dataset. When set, the closed-loop controller converges toward this SLO in small, bounded steps within cayenne_goal_convergence_window. Requires cayenne_tuning: adaptive on the dataset — a goal never enables the loop by itself; on a table left at the auto default it is ignored, with a warning. Goal-seeking also requires a non-zero cayenne_compaction_background_interval_ms. See Goal-driven tuning (opens in a new tab). |
cayenne_goal_freshness | Goal-driven adaptive tuning (preview): target data freshness — the age of the newest queryable data — as a duration (e.g. 30s). Best set globally under runtime.params; a value here overrides the global setpoint for this dataset. |
cayenne_goal_query_latency | Goal-driven adaptive tuning (preview): target p99 query latency on this table, as a duration (e.g. 250ms or 10s). Best set globally under runtime.params; a value here overrides the global setpoint for this dataset. |
cayenne_goal_convergence_window | Goal-driven adaptive tuning (preview): the time budget to converge toward the configured cayenne_goal_* SLOs, as a duration (e.g. 1m). Defaults to 60s. A per-dataset control-cadence knob — it paces how fast the loop chases the goals rather than declaring an outcome, so it is not part of the global SLO surface and has no runtime.params form. |
cayenne_compression_strategy | Compression algorithm for accelerated data. Defaults to btrblocks. Supports btrblocks or zstd. |
cayenne_delta_encoding | Encoding effort applied to delta (incremental) writes such as appends and inline-memtable flushes. Accepts auto (default) or a fixed level 0–10. Higher levels search more encoding schemes for a better compression ratio at the cost of more write-time CPU; auto encodes every delta write at a light level regardless of size, because deltas are transient staged streams that compaction later re-encodes with the full cascade. Levels 7–10 all apply the full default cascade, so set 7 to encode every delta with the full cascade instead. Applies at write time only — changing it never re-encodes existing data or forces a table re-create. Invalid values fall back to auto with a warning. |
cayenne_unsupported_type_action | Action when an unsupported data type is encountered. Defaults to error. See Data Type Support (opens in a new tab). |
cayenne_segment_cache_mb | Ignored here. The Vortex segment cache is one budget shared by every Cayenne table, so a per-dataset size has nothing to size. A value set under acceleration.params is reported at startup and otherwise has no effect — set runtime.params.cayenne_segment_cache_mb (opens in a new tab) instead. |
cayenne_force_view_types | Whether scans emit Arrow view types (Utf8View/BinaryView) instead of native Utf8/Binary. Accepts true or false; defaults to false. View types are not compacted across RepartitionExec, so wide strings fanned out through a partitioned join can inflate query-pool memory reservations — the native types avoid that. Set to true to re-enable view types for a table. Any value other than false (case-insensitive) enables view types. |
cayenne_file_path | Custom path for storing Cayenne data files. Supports local paths or S3 Express One Zone URLs (e.g., s3://bucket--usw2-az1--x-s3/prefix/). |
cayenne_target_file_size_mb | Target size for individual Vortex files in MB. When writes exceed this size, a new Vortex file is created. Accepts auto (default) or an explicit MB value. auto is storage-aware: 256 MB on EBS-class network storage, 64 MB on RAM-backed (tmpfs) mounts, 512 MB on S3 Express (large immutable objects cut object count and per-request cost), and 256 MB on local SSD or unknown storage. Smaller files enable better parallelism and predicate pushdown. |
cayenne_metadata_dir | Custom directory for storing Cayenne metadata (SQLite catalog). When unset, a local cayenne_file_path selects {cayenne_file_path}/metadata; otherwise the directory is {spice_data_path}/metadata, including when cayenne_file_path is an object-store URL. Must resolve outside the dataset's data directory — see Metastore location (opens in a new tab). |
cayenne_metastore | Metastore backend type. Supports sqlite (default) or turso (requires turso feature flag). |
cayenne_upload_concurrency | Maximum number of concurrent file uploads when writing multiple Vortex files to S3 Express One Zone. Accepts auto (default) or an explicit value; auto uses the runtime's CPU entitlement (opens in a new tab) in whole cores. The aggregate encode concurrency across all Cayenne tables is separately bounded by a process-global budget sized from that same entitlement, less a query reserve of a quarter of the cores (at least 2). |
cayenne_write_concurrency | Writer partition override for unsorted ingests, controlling how many Vortex files are encoded in parallel during a write. Accepts auto (default) or an explicit value. auto encodes up to min(4, session target_partitions) files in parallel per write — an intentionally small per-table default (not the full CPU entitlement), so many independently-writing tables do not oversubscribe CPU under concurrent CDC. An explicitly-set value is capped at the session target_partitions, which defaults to the runtime's CPU entitlement (opens in a new tab) in whole cores; the aggregate encode concurrency across all Cayenne tables is separately bounded by a process-global budget sized from that entitlement. Values below 1 are clamped to 1. The sort-and-rewrite compaction path always writes serially regardless of this setting. |
cayenne_deletion_mode | How primary-key deletions are recorded and applied. Accepts auto, key, or position; defaults to auto, which resolves to position (merge-on-read) for most tables, or to key for CDC datasets (refresh_mode: changes) that declare a primary_key. See Deletion Strategies (opens in a new tab). |
cayenne_pk_conflict_detection | Controls primary-key conflict detection on insert. Accepts auto or none; defaults to auto, which detects existing primary keys and resolves them as merge-on-read upserts. Set to none to skip conflict detection (blind append) for append-only CDC workloads where the source guarantees primary-key uniqueness. |
cayenne_compaction_trigger_files | Minimum number of small Vortex files in the current snapshot before tiered compaction runs. A "small" file is one whose size is below cayenne_target_file_size_mb / 4. Defaults to 4 for refresh_mode: caching / changes, or append with refresh_check_interval ≤ 5m; 8 otherwise. A value of 1 is clamped to a minimum of 2. |
cayenne_compaction_trigger_protected_snapshots | Number of protected snapshots before snapshot-maintenance compaction runs. Separate from cayenne_compaction_trigger_files so small-file tuning does not silently change scan amplification behavior. Defaults to 4 for refresh_mode: caching / changes, or append with refresh_check_interval ≤ 5m; 8 otherwise. A value of 1 is clamped to a minimum of 2. |
cayenne_compaction_trigger_snapshot_age_ms | Maximum age in milliseconds of the oldest protected snapshot before snapshot-maintenance compaction runs. Set to 0 to disable the age trigger. Defaults to 60000 for refresh_mode: caching / changes, or append with refresh_check_interval ≤ 5m; 300000 otherwise. |
cayenne_compaction_max_levels | Maximum number of consecutive compaction passes per trigger. Bounds write amplification when promotion keeps producing new candidates. Defaults to 3. |
cayenne_compaction_max_files_per_pick | Maximum number of eligible file paths retained in one compaction candidate. For a table eligible for subset compaction — key-based deletes (cayenne_deletion_mode: key, including append-only tables), no configured sort_columns, and no protected snapshots — the runner rewrites only the picked candidate and carries the unpicked settled files into the new snapshot by hardlink (local filesystem) or copy (S3 or cross-device), so this value does bound rewrite IO and memory. A table using position deletes, a configured sort_columns (whose subset hardlinks would break the global sort attestation), or carrying protected snapshots (which the rewrite has to fold) still rewrites the whole current snapshot, and this value then only affects trigger selection and observability. Defaults to 32. |
cayenne_compaction_background_interval_ms | Background compaction interval in milliseconds. The accelerator runs a per-table background task at this interval. Set to 0 to disable the background task — inline compaction on writes still runs. Defaults to 10000 for refresh_mode: caching / changes, or append with refresh_check_interval ≤ 5m; 0 for refresh_mode: full, where each refresh replaces the whole table and leaves nothing for the background task to consolidate (set the parameter explicitly to turn it back on for a table that also takes INSERTs); 30000 otherwise. When refresh_mode is left unset the connector fills it in — debezium and cdc resolve to changes, sink to disabled, every other connector to full — so an unannotated dataset takes the full default here. |
cayenne_cdc_durability | Durability mode for the inline CDC write path. Accepts memory (default) or file. memory appends batches to an in-RAM tier and defers the source-slot acknowledgement to a periodic or cap-triggered checkpoint, collapsing per-batch durability cost; on crash the un-checkpointed tail is replayed from the source slot, and because the apply is primary-key-idempotent this remains exactly-once. The in-memory path is eligibility-gated — it applies only to the small-write / CDC profile on non-partitioned tables, and other profiles always use file. file persists each CDC batch durably before advancing the source slot and is the conservative opt-out. |
cayenne_cdc_mem_tier_max_bytes | Per-table RAM-tier byte cap that forces a spill (checkpoint) and source-slot advance, in cayenne_cdc_durability: memory mode only. Auto-derived from host memory (~1/64 of RAM, clamped to 256 MiB–1 GiB; 256 MiB on hosts at or under 16 GiB). A process-global byte budget also bounds aggregate resident memory across all tables; whichever cap is breached first triggers the spill. Set to 0 to disable the per-table cap (the global budget still applies). |
cayenne_cdc_mem_tier_max_age_ms | Maximum wall-clock milliseconds a RAM-tier epoch may age before a forced checkpoint, in cayenne_cdc_durability: memory mode only. Bounds the crash-replay window and the deferred source-slot acknowledgement for tables that never reach a byte threshold. Defaults to 10000 (10 s). Set to 0 to disable the age trigger. |
cayenne_cdc_mem_tier_min_flush_bytes | Minimum resident RAM-tier bytes before the periodic background checkpoint tick durably checkpoints, in cayenne_cdc_durability: memory mode only. Bounds snapshot / delete-file churn — below this size a tick is skipped unless the tier has reached cayenne_cdc_mem_tier_max_age_ms. Query freshness is unaffected (RAM rows are visible immediately); only the deferred slot acknowledgement waits. The write-path byte-cap spill is not gated by this value. Auto-derived as 1/8 of the resolved cayenne_cdc_mem_tier_max_bytes (clamped to 32–128 MiB; 32 MiB on hosts at or under 16 GiB). Set to 0 to flush on every tick. |
cayenne_cdc_mem_tier_checkpoint_interval_ms | Periodic background mem-tier checkpoint interval in milliseconds, in cayenne_cdc_durability: memory mode only. The accelerator spawns a per-table background task that checkpoints the RAM tier every interval, advancing the deferred source-slot acknowledgement on an idle or pure-upsert stream that never trips a write-path cap or event trigger. Defaults to 1000 (1 s). Set to 0 to disable the periodic task. |
sort_columns | Comma-separated list of columns to sort data by. Improves segment pruning for frequently filtered columns. The order is established by the compaction rewrite, so it applies to a table that accumulates files to consolidate — see cayenne_compaction_background_interval_ms for when such a table compacts at all. A refresh_mode: full refresh replaces the whole table rather than adding to it, so it never compacts: on v2.3.0 its replacement data is written in arrival order and sort_columns has no effect on it; from v2.3.1 the whole-table replace itself orders the replacement stream, which is the only write such a table makes and so the only chance to establish the order. Sorting a full refresh goes through a single writer, and requires an operator-configured value — an inference-derived order (see cayenne_sort_columns_origin (opens in a new tab)) does not trigger it. Cannot be combined with cayenne_cluster_by; registration fails until one is removed. |
cayenne_shard_key_columns | Comma-separated columns that decide which output file a row is written to when a write encodes several files in parallel. Defaults to the primary key. A CDC write hashes this key across the files; a compaction rewrite and a refresh_mode: full whole-table replace instead split it into ascending, disjoint ranges, one per file, so each file's statistics prune (see Sorted data and segment pruning (opens in a new tab)). It selects a layout, not an order: no sort is advertised to the planner and query results are unaffected. |
cayenne_sort_columns_origin | Provenance of cayenne_sort_columns, which decides whether that sort order outranks the filter columns observed on scans. Accepts user (the default when absent) — the sort order is an explicit operator choice and is authoritative — or inferred, meaning schema inference filled it from the source's declared order (for most CDC datasets, the primary key). An inferred order is treated as a guess and ranks below the observed filter columns, so the default-on adaptive layout can cluster for the workload actually being queried. Schema inference sets this automatically whenever it populates cayenne_sort_columns; set it by hand only to reproduce an inferred configuration (for example in a benchmark or test). |
unsupported_type_action | Action when encountering unsupported data types. Options: error (default), string, warn, ignore. |
cayenne_s3_client_timeout | Request timeout duration (e.g., 30s, 5m). Defaults to 120s. |
cayenne_s3_unsigned_payload | Use unsigned payload for S3 Express One Zone requests. Defaults to true. |
cayenne_s3_allow_http | Set to true for testing with local S3-compatible storage. Defaults to false. |
cayenne_sort_columnsDecimal256cayenne_sort_columnscayenne_sort_columnscayenne_datalake_clustering_columnscayenne_datalake_target_file_size_mb | Target size for cold-tier Vortex files in MB. Larger than the warm cayenne_target_file_size_mb because object stores favor fewer, larger objects and cold scans are range reads. Accepts auto or an explicit MB value. Defaults to 512. |
cayenne_datalake_warm_max_bytes | The warm tier graduates to cold once its total Vortex bytes reach this threshold. 0 (default) disables the byte trigger; set alongside cayenne_datalake_warm_max_files to bound warm-tier size. When the tier is enabled and neither trigger is set, Cayenne applies a default byte trigger (16× cayenne_datalake_target_file_size_mb) so promotion is not silently disabled. |
cayenne_datalake_warm_max_files | The warm tier graduates to cold once its Vortex file count reaches this threshold. 0 (default) disables the file-count trigger. |
cayenne_datalake_tiering_check_interval_ms | How often the background loop checks the warm→datalake tiering trigger, in milliseconds. Datalake tiering is not latency-critical, so this is coarser than compaction. Defaults to 60000 (60s). 0 disables the tiering and garbage-collection loop entirely and is rejected at registration when the datalake tier is enabled. |
cayenne_datalake_gc_interval_ms | Physical-GC cadence (and orphan grace) for superseded cold-tier objects: the background sweep runs about this often and deletes an object no longer referenced by the manifest only after it has been observed orphaned for at least one interval. Defaults to 300000 (5m). 0 collapses the grace period to zero (a superseded object could be deleted while a running query still reads it) and is rejected at registration when the cold tier is enabled. |
cayenne_datalake_s3_allow_http | Allow plain-HTTP connections to the cold S3 endpoint. Defaults to false. |
cayenne_datalake_s3_client_timeout | HTTP client timeout for cold-store requests, as a duration (e.g. 2m). Defaults to 2m. |
cayenne_datalake_s3_unsigned_payload | Use unsigned payloads for cold S3 uploads. Defaults to true. |
cayenne_filter_propagationallnonedisabledcayenne_compaction_memory_fraction | Fraction of the query memory pool carved out for a dedicated Cayenne compaction memory pool. Defaults to 0.2 and is clamped to a supported range. Only applied when at least one enabled Cayenne dataset can accumulate files to compact — a file acceleration mode on a refresh mode that is not a whole-table replace — and dedicated thread pools are not disabled. A pod whose Cayenne datasets are all refresh_mode: full, or all mode: memory, carves no compaction pool and brings up no compaction runtime. |
cayenne_sort_merge_min_rows | Advanced anti-join tuning: row-count threshold above which the filter-propagation optimizer switches to a sort-merge strategy. Defaults to an internally tuned value; override only when profiling indicates a need. |
cayenne_sort_merge_memory_pool_fraction | Advanced anti-join tuning: fraction of the memory pool the sort-merge anti-join strategy may use. Defaults to an internally tuned value. |
cayenne_goal_replication_lag | Goal-driven adaptive tuning (preview): global target end-to-end CDC replication lag, as a duration (e.g. 5s), applied to every Cayenne dataset. Override per-dataset under a dataset's acceleration.params. See Goal-driven tuning (opens in a new tab). |
cayenne_goal_freshness | Goal-driven adaptive tuning (preview): global target data freshness — the age of the newest queryable data — as a duration (e.g. 30s). Override per-dataset under a dataset's acceleration.params. |
cayenne_goal_query_latency | Goal-driven adaptive tuning (preview): global target p99 query latency, as a duration (e.g. 250ms or 10s). Override per-dataset under a dataset's acceleration.params. |
cayenne_goal_qph | Goal-driven adaptive tuning (preview): target query throughput in queries per hour (higher is better), e.g. 5000. Must be a positive number. Global-only — query throughput is measured system-wide (a query spanning multiple datasets, such as a join, is counted once), so it has no per-dataset form; a value set under acceleration.params is ignored. |
cayenne_metastore_cache_mb | SQLite metastore page-cache size in megabytes. Applies to the default sqlite metastore backend (see cayenne_metastore); the turso backend uses MVCC and ignores the cayenne_metastore_* family. Defaults to 256. |
cayenne_metastore_mmap_mb | SQLite metastore memory-mapped I/O size in megabytes. Defaults to 1024 (1 GiB). |
cayenne_metastore_busy_timeout_ms | SQLite metastore busy_timeout in milliseconds — how long a blocked connection waits for a lock before erroring. Defaults to 30000. |
cayenne_metastore_wal_autocheckpoint_pages | SQLite metastore WAL auto-checkpoint threshold in pages. 0 disables the inline auto-checkpoint so the WAL is drained off the hot commit path by a dedicated background checkpoint instead. Defaults to 0. |
cayenne_metastore_wal_truncate_threshold_mb | WAL size in megabytes above which the background checkpoint escalates to a TRUNCATE checkpoint to reclaim file space. Defaults to 160. |
cayenne_metastore_auto_vacuum | SQLite metastore auto_vacuum mode: none, incremental, or full. Defaults to incremental, so freed pages are marked reclaimable and reclaimed in bounded batches by the background maintenance pass — see cayenne_metastore_incremental_vacuum_pages. An unrecognized value logs a warning and keeps the incremental default. SQLite fixes the mode at file creation, so a metastore created while the default was none stays on none — the runtime warns at startup when it finds one. To move it to incremental, stop the runtime, run PRAGMA auto_vacuum = INCREMENTAL; VACUUM; against the metastore file, and restart: the pragma alone is a no-op on a file created with auto_vacuum = none, and a bare VACUUM keeps it on none. Switching between full and incremental needs no VACUUM. See the SQLite auto_vacuum documentation (opens in a new tab). |
cayenne_metastore_incremental_vacuum_pages | Freelist pages the background maintenance pass reclaims per tick when the metastore is in incremental auto_vacuum mode; ignored in every other mode. Defaults to 256, which is 1 MiB at SQLite's 4 KiB default page size. Reclamation holds the write lock while it relocates pages, so the cap is what keeps each pause short; raise it to drain a large freelist faster at the cost of longer write-lock holds, or set 0 to stop reclaiming without changing the database's auto_vacuum mode. |
avgDecimal128(min(38, p + 4), min(38, s + 4))avgDecimal256min and max — signed/unsigned integers, Date32/Date64, Timestamp, and Decimal128, preserving the input type (MIN(Int32) -> Int32). Float min/max is not yet supported.aps1 |
| ap-south-1 |
sae1 | sa-east-1 |
cac1 | ca-central-1 |
afs1 | af-south-1 |
mes1 | me-south-1 |
cayenne_s3_zone_idsClustering. cayenne_cluster_by names the clustering key, and applies it to both warm-tier rewrites and datalake promotion, so the two tiers share one layout contract. When it is unset, clustering falls back to an operator-configured cayenne_sort_columns; when that is also unset, promotion clusters by the hottest columns observed in query pushdown filters (default-on adaptive layout), then by an inference-derived cayenne_sort_columns, and finally the primary key. Clustering on more than one dimension prunes far better than a single-column sort for selective queries on any clustering column.
The curve is a Hilbert interleave, not Z-order/Morton: consecutive points on a Hilbert curve are always spatially adjacent, whereas Morton jumps a whole quadrant at each boundary and every jump is a zone map stretched over everything it skipped. Each column's values are rescaled onto a common coordinate space from the table's maintained min/max statistics before interleaving — without that, a wide column (a microsecond timestamp) consumes every high-order round before a narrow one (a tenant id) contributes anything, and the key degenerates into a single-column sort. Clustering is a layout-quality property, never a correctness one.
A Cayenne table created through SQL uses the equivalent CREATE TABLE ... CLUSTER BY (column, ...) clause in place of the parameter. It accepts bare column names only — any other expression is a planning error naming the clause.
Only an operator-configured sort order shadows the observed filter columns. A sort order that schema inference (opens in a new tab) supplied — which for most CDC datasets resolves to the primary key — is tagged cayenne_sort_columns_origin: inferred and ranks below the observations, so the adaptive layout still clusters cold files for the queries the table actually receives rather than for inference's guess. It is used as the pre-observation fallback, before the primary key.
s3:// URL — a general-purpose S3 or S3-compatible bucket (for example MinIO via cayenne_datalake_s3_endpoint), authenticated independently through the cayenne_datalake_s3_* parameters. It is a separate store from the warm S3 Express One Zone tier and does not need to share its bucket. Local file:// cold locations are not supported in v1.refresh_mode: changes or refresh_mode: append; a full refresh re-materializes the whole table each cycle and is rejected.… >= cap/v1/sql returns the final statement's result (for the canonical gate-plus-write shape, the last write's row-count summary), or COMMIT when the body produces no rows.uniqueprimary_keyretention_sql (opens in a new tab) runs during maintenance after writes, full refreshes, and CDC checkpoints, independently of periodic retention settings. It is ignored in mode: memory, with a warning; matching rows remain queryable.retention_period (opens in a new tab) hides expired rows in either mode. Scheduled deletion requires both retention_check_enabled: true (opens in a new tab) and retention_check_interval (opens in a new tab), which has no default. Without both, a warning is logged and storage is reclaimed only when compaction rewrites affected files.INSERT/UPDATE on non-partitioned, accelerator-only or durable write-back datasets, with one write per table. DELETE and MERGE are not supported.