title: 'Spice Cayenne Data Accelerator' sidebar_label: 'Spice Cayenne Data Accelerator' description: 'Spice Cayenne Data Accelerator (Vortex) Documentation' sidebar_position: 1 tags:

• cayenne
• vortex
• data-accelerators
• performance
• s3-express

Spice Cayenne is a data acceleration engine designed for high-performance, scalable query on large-scale datasets. Built on Vortex (opens in a new tab), a high-performance columnar file format, Spice Cayenne combines columnar storage with in-process metadata management to provide fast query performance to scale to datasets beyond 1TB.

Why Vortex?

Spice Cayenne uses Vortex as its storage format, providing significant performance advantages:

• 100x faster random access reads compared to modern Apache Parquet
• 10-20x faster scans for analytical queries
• 5x faster writes with similar compression ratios
• Zero-copy compatibility with Apache Arrow for efficient data processing
• Extensible architecture with pluggable encoding, compression, and layout strategies

Vortex is a Linux Foundation (LF AI & Data) project under Apache-2.0 license with neutral governance. For performance benchmarks, see bench.vortex.dev (opens in a new tab).

Spice Cayenne is recommended for any dataset of 10 GB or larger. DuckDB (opens in a new tab) suits smaller datasets, but its memory requirements grow faster: Cayenne typically needs one-third to one-half the memory of the DuckDB accelerator for the same dataset, and scales beyond the limits at which a single DuckDB file becomes impractical.

Architecture

Spice Cayenne follows a lakehouse architecture inspired by DuckLake (opens in a new tab), separating metadata management from data storage:

Key Design Principles:

• Virtual Files: Each "file" is a Vortex ListingTable at a unique directory, enabling append operations and parallel reads
• Lazy Statistics: Summary statistics are loaded on-demand for query optimization
• Sequence-based Ordering: Iceberg-style sequence numbers enable upsert semantics without requiring separate tracking of "undeleted" records (rows that were deleted and then re-inserted)
• Pluggable Storage: Data files can be stored locally or in S3 Express One Zone while metadata remains local

Storage Recommendations

For optimal performance, store Cayenne data files on NVMe storage. NVMe provides the lowest latency and highest throughput for the random access patterns that Vortex files require.

Use S3 Express One Zone (opens in a new tab) when persistence of accelerations across restarts is required. S3 Express One Zone adds network latency compared to local NVMe but provides durability. Sharing accelerated data across multiple Spice instances is planned for a future release.

Configuration

To use Spice Cayenne as the data accelerator, specify cayenne as the engine for acceleration. Spice Cayenne supports two storage modes:

• mode: file (durable) — data is written as Vortex files on local disk or S3 Express One Zone, with a SQLite/Turso metastore, and the acceleration survives restarts. This is the recommended mode for Cayenne and is used in the examples throughout this page. The mode: file_create and mode: file_update variants control how an existing on-disk acceleration is reused or rebuilt on startup.
• mode: memory (ephemeral) — all data lives fully in RAM with an in-memory metastore; nothing is written to disk. The dataset is ephemeral and reloads from its source on restart (like the Arrow (opens in a new tab) accelerator). Memory mode works for all refresh modes (full/append/changes) and for both keyed and no-primary-key datasets, but does not support partitioned tables (partition_by), and it enforces a hard per-table RAM bound rather than spilling to disk (see cayenne_cdc_mem_tier_max_bytes (opens in a new tab)).

datasets:
  - from: spice.ai:path.to.my_dataset
    name: my_dataset
    acceleration:
      engine: cayenne
      mode: file

datasets:
  - from: spice.ai:path.to.my_dataset
    name: my_dataset
    acceleration:
      engine: cayenne
      mode: file

Parameters

Spice Cayenne is configured through two distinct parameter scopes:

• Acceleration parameters are set per dataset under acceleration.params and control how that dataset's accelerated data is stored, compressed, written, and compacted.
• Runtime parameters are set once per instance under runtime.params and control engine-global behavior — caches, optimizer rules, and dedicated memory pools — shared by every Cayenne-accelerated dataset.

The two scopes are not interchangeable: setting a runtime parameter under acceleration.params (or a per-dataset parameter under runtime.params) has no effect — the value is ignored.

Acceleration parameters (acceleration.params)

Set under a dataset's acceleration.params:

S3 Express One Zone parameters

These are acceleration parameters (set under acceleration.params) used when storing Cayenne data files in S3 Express One Zone:

Cold object-store tier parameters

These acceleration parameters (set under acceleration.params) configure the optional cold object-store tier (opens in a new tab). Setting cayenne_datalake_location enables the tier; the rest tune the clustering key, cold file size, the warm→cold promotion trigger, and the cold store's credentials. When cayenne_datalake_location is unset (the default), the cold tier is dormant and behaves byte-identically to a warm-only table.

The cold store is authenticated independently from the warm tier (which uses the cayenne_s3_* parameters) via the following acceleration.params:

Runtime parameters (runtime.params)

Set once under the top-level runtime.params and applied to every Cayenne-accelerated dataset in the instance. With the exception of the goal-driven SLO setpoints (cayenne_goal_*) — which set a global default that a dataset can override under its own acceleration.params (except cayenne_goal_qph, which is global-only) — these are not valid under a dataset's acceleration.params.

A global cayenne_goal_* setpoint steers only the datasets that run the closed loop: it takes effect on a dataset that sets cayenne_tuning: adaptive and is ignored (with a warning) on every dataset left at the auto default.

runtime:
  params:
    # Engine-global Cayenne tuning, shared by every Cayenne-accelerated dataset
    cayenne_footer_cache_mb: 512
    cayenne_segment_cache_mb: 1024
    cayenne_filter_propagation: enabled

datasets:
  - from: s3://analytics-bucket/events/
    name: events
    acceleration:
      engine: cayenne
      mode: file
      params:
        # Per-dataset Cayenne tuning
        cayenne_target_file_size_mb: 256

runtime:
  params:
    # Engine-global Cayenne tuning, shared by every Cayenne-accelerated dataset
    cayenne_footer_cache_mb: 512
    cayenne_segment_cache_mb: 1024
    cayenne_filter_propagation: enabled

datasets:
  - from: s3://analytics-bucket/events/
    name: events
    acceleration:
      engine: cayenne
      mode: file
      params:
        # Per-dataset Cayenne tuning
        cayenne_target_file_size_mb: 256

Performance Tuning

Spice Cayenne is self-tuning by default and rarely needs manual tuning. For the full tuning reference — self-tuning and goal-driven SLOs (opens in a new tab), cache sizing (opens in a new tab), compression strategy (opens in a new tab), and file-size tuning (opens in a new tab) — see the dedicated Performance Tuning (opens in a new tab) page.

Features

DataFusion Query-Native Execution

Spice Cayenne is DataFusion query-native, meaning all query execution uses Apache DataFusion (opens in a new tab) and adheres to the runtime.query.memory_limit setting. This provides:

• Vectorized execution: Multi-threaded, SIMD-optimized query processing
• Automatic memory management: Query memory is tracked and spilled to disk when limits are exceeded
• Dynamic filter pushdown: Filters from TopK, Join, and Aggregate operators push down to file scans

DataFusion's GreedyMemoryPool (opens in a new tab) allows memory reservations on a first-come, first-served basis, improving throughput for high-concurrency queries with many partitions.

High-Performance Columnar Storage

Spice Cayenne uses Vortex's advanced columnar format, which provides:

• Efficient Compression: Cascading compression with nested encoding schemes including RLE, dictionary encoding, FastLanes, FSST, and ALP
• Rich Statistics: Lazy-loaded summary statistics for query optimization
• Extensible Encodings: Pluggable physical layouts optimized for different data patterns
• Wide Table Support: Efficient handling of tables with many columns through zero-copy metadata access

Point Lookups and Random Access

Vortex delivers 100x faster random access reads (opens in a new tab) compared to Apache Parquet through several architectural features:

Segment Statistics (Zone-Map Equivalent):

Vortex's ChunkedLayout (opens in a new tab) maintains per-segment statistics for each column, enabling segment pruning during query execution. Statistics include:

When a query includes a WHERE clause, Spice Cayenne evaluates whether each segment could contain matching rows. Segments that cannot match based on min/max statistics are skipped entirely, similar to DuckDB's zone-maps (opens in a new tab) without requiring explicit index creation.

Example - Segment Pruning:

For a table with segments containing timestamp ranges [2024-01-01, 2024-01-15], [2024-01-16, 2024-01-31], [2024-02-01, 2024-02-15], a query:

SELECT * FROM events WHERE timestamp > '2024-01-20';

SELECT * FROM events WHERE timestamp > '2024-01-20';

Prunes the first segment (max < 2024-01-20) and reads only the second and third segments.

Fast Random Access Encodings:

Vortex encodings support direct random access to compressed data:

• FSST (opens in a new tab) (Fast Static Symbol Table): String compression with O(1) random access
• FastLanes (opens in a new tab): High-performance integer encoding with vectorized decoding
• ALP (opens in a new tab): Adaptive lossless floating-point compression with random access

Compute Push-Down:

Vortex supports executing filter and compute operations directly on compressed data, avoiding full decompression for predicate evaluation. This compute push-down reduces CPU and memory overhead by processing data in its compressed form:

Array-level statistics (is_sorted, is_constant, min, max) enable additional optimizations beyond filtering. For example, is_sorted enables binary search for point lookups, and is_constant returns values immediately without scanning.

Performance Characteristics:

For point lookups and selective queries, Spice Cayenne with Vortex often matches or exceeds the performance of traditional B-tree indexes while consuming no additional memory for index structures. Performance scales with:

• Data sorting (sorted columns benefit most from segment pruning)
• Segment cache hit rate (hot data patterns)
• Compression encoding match to data characteristics

Change Data Capture (refresh_mode: changes)

Spice Cayenne is the recommended accelerator for Change Data Capture (CDC) (opens in a new tab). A dataset configured with refresh_mode: changes is bootstrapped from a source snapshot and then kept continuously in sync by applying the source's row-level change stream — inserts, updates, and deletes — so the acceleration reflects the current state of the source row-for-row.

CDC into Cayenne is available for every Spice CDC connector:

• PostgreSQL Logical Replication (opens in a new tab)
• MySQL Binlog Replication (opens in a new tab)
• MongoDB Change Streams (opens in a new tab)
• DynamoDB Streams (opens in a new tab)
• Debezium (opens in a new tab) (over Kafka)

See Change Data Capture (opens in a new tab) for source-side setup and the Changes refresh mode (opens in a new tab) reference for the acceleration-side configuration.

datasets:
  - from: postgres:public.orders
    name: orders
    acceleration:
      engine: cayenne
      mode: file
      refresh_mode: changes

datasets:
  - from: postgres:public.orders
    name: orders
    acceleration:
      engine: cayenne
      mode: file
      refresh_mode: changes

Cayenne is not the only engine that can apply a change stream — arrow, duckdb, and sqlite are also valid changes sinks — but Cayenne adds CDC-specific capabilities built for large-scale, continuously-updated datasets:

• Incrementally maintained aggregate views (IVM). Aggregate views declared with maintained_aggregates are updated in place as each change event is applied, rather than recomputed, so GROUP BY rollups stay fresh at CDC speed. See Maintained Aggregates (opens in a new tab).
• Key-based incremental deletes. With a primary key, cayenne_deletion_mode: auto resolves to key-based deletion vectors so source DELETE and UPDATE events apply efficiently. See Deletion Vectors (opens in a new tab).
• In-memory CDC tier. Change events can be staged through an in-memory tier (cayenne_cdc_durability and the cayenne_cdc_mem_tier_* parameters) for low apply latency, with exactly-once semantics via primary-key-idempotent replay. The deployment guide (opens in a new tab) documents the CDC apply metrics for monitoring lag and throughput.
• Replication-lag and freshness SLOs. The adaptive self-tuner can target an end-to-end replication-lag or data-freshness goal (cayenne_goal_replication_lag, cayenne_goal_freshness) and adjust its apply behavior to meet it. See Goal-driven tuning (opens in a new tab).

CDC Requirements

• A primary key is required. In changes mode Cayenne always applies an inferred or declared primary key and routes source updates through an upsert. Declare primary_key on the dataset (and, where the connector requires it, on_conflict: upsert); the per-connector CDC pages document the exact requirement.
• File or memory mode. Durable resume across restarts requires mode: file. mode: memory is supported for ephemeral, in-RAM CDC where the accelerator is rebuilt from the source on restart.

Maintained Aggregates

Spice Cayenne can incrementally maintain aggregate views over a CDC-accelerated dataset so that GROUP BY aggregate queries are answered from continuously-updated summary state instead of re-scanning the full table. As change events are applied (refresh_mode: changes), each maintained view is updated with the incoming deltas; a matching query is then rewritten by the physical optimizer to read the maintained state directly. When a query does not match a declared view — or the view is not currently fresh — Cayenne transparently falls back to a full scan, so results are always correct.

Maintained aggregates are declared per dataset under acceleration.maintained_aggregates:

datasets:
  - from: postgres:orders
    name: orders
    acceleration:
      enabled: true
      engine: cayenne
      refresh_mode: changes
      primary_key: id
      maintained_aggregates:
        - group_by: [customer_id]
          aggregates:
            - function: count
            - function: sum
              column: amount
            - function: avg
              column: amount
            - function: min
              column: amount
            - function: max
              column: amount

datasets:
  - from: postgres:orders
    name: orders
    acceleration:
      enabled: true
      engine: cayenne
      refresh_mode: changes
      primary_key: id
      maintained_aggregates:
        - group_by: [customer_id]
          aggregates:
            - function: count
            - function: sum
              column: amount
            - function: avg
              column: amount
            - function: min
              column: amount
            - function: max
              column: amount

Each entry declares one maintained view:

Each item in aggregates has:

Supported input column types by function:

• count — any column (or COUNT(*) when column is omitted).
• sum and avg — signed integers (Int8–Int64), unsigned integers (UInt8–UInt64), floats (Float32/Float64), and Decimal128. For integer and float inputs, sum widens to BIGINT/Float64 and avg returns Float64. For a Decimal128(p, s) input, sum returns Decimal128(min(38, p + 10), s) and avg returns Decimal128(min(38, p + 4), min(38, s + 4)), matching DataFusion's exact decimal return types (avg requires a non-negative scale). Decimal256 is not supported.
• min and max — signed/unsigned integers, Date32/Date64, Timestamp, and Decimal128, preserving the input type (MIN(Int32) -> Int32). Float min/max is not yet supported.

Query matching

A query is served from a maintained view only when it matches the view exactly:

• The query's GROUP BY keys match the view's group_by, in order.
• The query's WHERE predicate matches the view's filter_sql exactly — an unfiltered view (no filter_sql) answers only unfiltered queries, and a filtered view answers only a query carrying the identical predicate. A view whose filter_sql mirrors a common dashboard filter lets that filtered analytical query be served incrementally instead of by a full re-scan.

Any query that does not match a declared view (or that reaches the view while it is stale) falls back to the base-table scan — correct, but not accelerated.

Constraints

• Maintained aggregates are a Spice Cayenne feature designed for CDC-accelerated datasets (refresh_mode: changes).
• min and max are retraction-hard: they require a primary_key on the acceleration so that UPDATE and DELETE changes can retract a prior extremum. Set acceleration.primary_key, enable extended schema inference for a source primary key, or omit min/max. count, sum, and avg do not require a primary key.
• Maintained aggregates are not supported on partitioned tables (partition_by).

Retaining specs without maintaining them

To keep a view's declaration in configuration without materializing or maintaining it, use the policy form with mode: disabled:

maintained_aggregates:
        mode: disabled
        views:
          - group_by: [customer_id]
            aggregates:
              - function: sum
                column: amount

maintained_aggregates:
        mode: disabled
        views:
          - group_by: [customer_id]
            aggregates:
              - function: sum
                column: amount

When any view is declared with the list form above, maintenance is enabled by default; the policy form's mode: enabled is equivalent.

Deletion Vectors

Spice Cayenne implements efficient deletes without rewriting data files using deletion vectors. Deletion vectors track which rows have been logically deleted, and the information is applied transparently during query execution.

Deletion Strategies

How deletions are recorded and applied is controlled by the cayenne_deletion_mode parameter:

datasets:
  - from: s3://bucket/events/
    name: events
    acceleration:
      engine: cayenne
      mode: file
      primary_key: event_id
      params:
        cayenne_deletion_mode: auto # default; set to `key` to opt out of merge-on-read

datasets:
  - from: s3://bucket/events/
    name: events
    acceleration:
      engine: cayenne
      mode: file
      primary_key: event_id
      params:
        cayenne_deletion_mode: auto # default; set to `key` to opt out of merge-on-read

Under position mode (the auto resolution for all tables except CDC datasets with a primary key):

• Tables without a primary key record deletions by row position. Cayenne uses RoaringBitmap for memory-efficient storage of deleted row IDs, providing 50-90% memory savings compared to HashSet for sparse deletions.
• Tables with a primary key capture row positions via a row_idx() read-back after each write, with a key-based fallback for any row whose position is not yet known. Pushing the deletes into the scan eliminates the per-row RowConverter deletion tax above it.

Key-based deletion (cayenne_deletion_mode: key) uses the byte representation of primary key columns and applies deletes above the scan. This approach is position-independent and survives data reorganization.

Orphaned deletion-vector cleanup

On a primary-key upsert table, a deletion vector becomes orphaned once time-based retention empties the snapshot(s) it shadowed — the surviving-sequence floor rises above the DV's delete sequence, so the DV is a query-time no-op that still occupies disk and the in-memory deletion cache. Cayenne runs a background sweep that reclaims these orphaned .arrow files and their catalog rows.

The sweep is always on and requires no configuration:

• Fixed per-table threshold — the sweep runs per-table once at least 20 orphaned DVs have accumulated. This threshold is a fixed internal constant, not a tunable parameter.
• Retention-only — a DV is orphaned only after retention (retention_period + time_column) empties the snapshots it shadowed. Without a retention policy nothing is ever orphaned and the sweep is a no-op, at zero ingest cost.
• No correctness impact — orphaned DVs are already query-time no-ops; the benefit is reclaimed disk plus a smaller in-memory deletion cache and less file I/O at restart.
• Off the write path — the sweep is lock-free and throttled, and runs on the compaction runtime; it never extends the write-lock window.

The sweep bounds only the orphaned tail. The live (not-yet-orphaned) DV set is bounded separately by compaction.

Primary Key Optimization

For tables with a single-column Int64 primary key, Cayenne uses an optimized direct lookup strategy that avoids serialization overhead:

datasets:
  - from: s3://bucket/events/
    name: events
    acceleration:
      engine: cayenne
      mode: file
      primary_key: event_id  # Int64 column - uses optimized deletion

datasets:
  - from: s3://bucket/events/
    name: events
    acceleration:
      engine: cayenne
      mode: file
      primary_key: event_id  # Int64 column - uses optimized deletion

Upsert Support

When on_conflict is configured, Cayenne supports upsert semantics using sequence numbers (Iceberg-style ordering):

datasets:
  - from: kafka:events
    name: events
    acceleration:
      engine: cayenne
      mode: file
      primary_key: id
      on_conflict:
        id: upsert

datasets:
  - from: kafka:events
    name: events
    acceleration:
      engine: cayenne
      mode: file
      primary_key: id
      on_conflict:
        id: upsert

When a primary key is deleted and then re-inserted:

1. The new insert gets a higher sequence number than the delete
2. During scan, the delete doesn't apply to data with higher sequence numbers
3. The new data is visible without requiring separate tracking of "undeleted" records

AWS S3 Express One Zone Storage

Spice Cayenne supports storing data files in AWS S3 Express One Zone (opens in a new tab) for single-digit millisecond latency, ideal for latency-sensitive query workloads that require persistence. Metadata remains on local disk for fast catalog operations while data files are stored in S3 Express One Zone.

Why S3 Express One Zone?

S3 Express One Zone directory buckets provide:

• Single-digit millisecond latency: 10x faster than S3 Standard for first-byte latency
• High request throughput: Up to 10x higher request rates than S3 Standard
• Cost efficiency: Lower per-request costs for high-frequency access patterns
• Durability: Same 99.999999999% (11 9s) durability as S3 Standard

S3 Express Examples

Example 1 - Explicit bucket:

datasets:
  - from: s3://source-bucket/events/
    name: analytics_events
    acceleration:
      engine: cayenne
      enabled: true
      mode: file
      params:
        # Store data in S3 Express One Zone bucket
        cayenne_file_path: s3://my-bucket--usw2-az1--x-s3/cayenne/
        cayenne_s3_region: us-west-2

datasets:
  - from: s3://source-bucket/events/
    name: analytics_events
    acceleration:
      engine: cayenne
      enabled: true
      mode: file
      params:
        # Store data in S3 Express One Zone bucket
        cayenne_file_path: s3://my-bucket--usw2-az1--x-s3/cayenne/
        cayenne_s3_region: us-west-2

Example 2 - Auto-generated bucket with IAM role:

datasets:
  - from: postgresql://db/events
    name: fast_events
    acceleration:
      engine: cayenne
      enabled: true
      mode: file
      params:
        # Auto-generates bucket: spice-{spicepod-name}-fast_events--usw2-az1--x-s3
        cayenne_s3_zone_ids: usw2-az1

datasets:
  - from: postgresql://db/events
    name: fast_events
    acceleration:
      engine: cayenne
      enabled: true
      mode: file
      params:
        # Auto-generates bucket: spice-{spicepod-name}-fast_events--usw2-az1--x-s3
        cayenne_s3_zone_ids: usw2-az1

Example 3 - Explicit credentials:

datasets:
  - from: kafka:events
    name: realtime
    acceleration:
      engine: cayenne
      enabled: true
      mode: file
      params:
        cayenne_s3_zone_ids: use1-az4
        cayenne_s3_region: us-east-1
        cayenne_s3_auth: key
        cayenne_s3_key: ${secrets:AWS_ACCESS_KEY_ID}
        cayenne_s3_secret: ${secrets:AWS_SECRET_ACCESS_KEY}

datasets:
  - from: kafka:events
    name: realtime
    acceleration:
      engine: cayenne
      enabled: true
      mode: file
      params:
        cayenne_s3_zone_ids: use1-az4
        cayenne_s3_region: us-east-1
        cayenne_s3_auth: key
        cayenne_s3_key: ${secrets:AWS_ACCESS_KEY_ID}
        cayenne_s3_secret: ${secrets:AWS_SECRET_ACCESS_KEY}

Bucket Naming Conventions

S3 Express One Zone buckets use a specific naming format:

• Format: {base-name}--{zone-id}--x-s3
• Zone ID format: {region-code}-az{number} (e.g., usw2-az1, use1-az4)
• Auto-generated names: spice-{app-name}-{dataset-name}--{zone-id}--x-s3

The zone ID is automatically extracted from the bucket name to configure the correct endpoint.

Supported AWS Regions

S3 Express One Zone is available in select regions. Spice automatically derives the region from zone IDs:

See AWS documentation for the complete list of S3 Express One Zone availability zones (opens in a new tab).

Important Considerations

• Standard S3 not supported: Cayenne currently only supports S3 Express One Zone, not standard S3 buckets.
• Same-AZ optimization: S3 Express One Zone is optimized for same-availability-zone access. For external access, Cayenne uses extended timeouts (a 2-minute per-request timeout by default, configurable via cayenne_s3_client_timeout) and retries.
• Bucket auto-creation: When using cayenne_s3_zone_ids, Spice automatically creates the S3 Express directory bucket if it doesn't exist (requires appropriate IAM permissions).
• Metadata locality: Cayenne metadata (SQLite catalog) remains on local disk. Only data files are stored in S3 Express.

Cold Object-Store Tier

Cayenne can cascade data across three storage tiers — an in-RAM mem-tier, a local-disk warm tier, and an object-store cold tier — with each row living in exactly one tier. The cold tier is optional and disabled by default; it is enabled by setting cayenne_datalake_location (opens in a new tab). When unset, a table is warm-only and behaves byte-identically to before.

The cold tier lets a table grow beyond local NVMe capacity while keeping recent, hot data on fast local storage and graduating older data to cheaper, durable object storage — without sacrificing pushdown on the cold data.

How it works

• Promotion (write path). A dedicated background worker graduates the warm tier once the size or file-count threshold (cayenne_datalake_warm_max_bytes / cayenne_datalake_warm_max_files) is crossed, evaluated every cayenne_datalake_tiering_check_interval_ms. Promotion is incremental carry-forward: rather than re-materializing the whole table each cycle, it classifies the existing cold manifest into dirty files that may host a tombstoned key — using per-PK-column min/max rectangles from each file's persisted statistics, refined by a per-file PK bloom filter, and conservative in the safe direction (a false positive costs an extra rewrite; a missed tombstone is impossible) — and clean files that are provably untouched. Only the warm delta plus the dirty files are re-read (all deletes applied, one version per key), Z-order clustered for tight multi-column zone maps, and written as read-optimized Vortex at the larger cayenne_datalake_target_file_size_mb under a per-promotion prefix; clean files are carried forward by manifest reference and never re-read. The new files are atomically registered while the promoted warm files are cleared in a single transaction, so promotion cost tracks the changed data, not total table size.

• Cross-tier scan (read path). Queries span all tiers and push filters, projection, and limits down to each. Cold files are pruned from statistics held in the metastore, so pruning requires no object-store round-trip on the query path. Each cold file's PK bloom lets an upsert's keyset rebuild answer cold-tier key existence without scanning the object store. A DELETE after promotion correctly hides a cold-resident row.

• Physical GC. A periodic mark-and-sweep, rooted at the manifest, reclaims cold objects orphaned by carry-forward rewrites. It runs every cayenne_datalake_gc_interval_ms (default 5m), which doubles as the orphan grace period: an object no longer referenced by the manifest is deleted only after it has been observed orphaned for at least one interval.

• Clustering. Cold files are clustered by cayenne_datalake_clustering_columns (multi-column Z-order / Morton order). When it is unset, clustering falls back to an operator-configured cayenne_sort_columns; when that is also unset, cold-tier promotion clusters by the hottest columns observed in query pushdown filters (default-on adaptive layout), then by an inference-derived cayenne_sort_columns, and finally the primary key. Clustering on more than one dimension prunes far better than a single-column sort for selective queries on any clustering column.

  Only an operator-configured sort order shadows the observed filter columns. A sort order that schema inference (opens in a new tab) supplied — which for most CDC datasets resolves to the primary key — is tagged cayenne_sort_columns_origin: inferred and ranks below the observations, so the adaptive layout still clusters cold files for the queries the table actually receives rather than for inference's guess. It is used as the pre-observation fallback, before the primary key.

Requirements and v1 constraints

• Key-based deletes required. The cold tier requires cayenne_deletion_mode: key. When cayenne_deletion_mode is unset (or auto), Cayenne auto-resolves it to key for a datalake-enabled table. An explicit cayenne_deletion_mode: position conflicts with the cold tier and is rejected at registration — position deletes are file-path scoped and cannot survive the warm→cold rewrite. Set cayenne_deletion_mode: key (or remove it), or remove cayenne_datalake_location.
• Primary key required to activate. Promotion classifies and rewrites cold files by primary key, so a table that sets cayenne_datalake_location but declares no primary_key registers successfully with the datalake tier inactive — data is never promoted and remains in the warm tier (Cayenne logs a loud warning). Add a primary_key to activate the tier, or remove cayenne_datalake_location. Leaving PK-less tables inactive (rather than failing registration) lets a fleet-wide datalake location coexist with datasets that have no primary key.
• Non-zero background intervals. When the cold tier is enabled, cayenne_datalake_tiering_check_interval_ms: 0 (which would disable the datalake tiering and garbage-collection loop) and cayenne_datalake_gc_interval_ms: 0 (which would collapse the GC grace period, risking deletion of an object a running query still reads) are rejected at registration. Leave them at their defaults or set a positive value.
• S3-only location. The cold location must be an s3:// URL — a general-purpose S3 or S3-compatible bucket (for example MinIO via cayenne_datalake_s3_endpoint), authenticated independently through the cayenne_datalake_s3_* parameters. It is a separate store from the warm S3 Express One Zone tier and does not need to share its bucket. Local file:// cold locations are not supported in v1.
• Continuous refresh only. The cold tier requires refresh_mode: changes or refresh_mode: append; a full refresh re-materializes the whole table each cycle and is rejected.
• Unsupported in v1: partitioned tables and position-delete tables.

datasets:
  - from: postgres:public.events
    name: events
    acceleration:
      engine: cayenne
      mode: file
      refresh_mode: changes
      primary_key: event_id
      params:
        # Enable the cold object-store tier (general-purpose S3, not S3 Express)
        cayenne_datalake_location: s3://my-cold-bucket/events/
        cayenne_datalake_s3_region: us-west-2
        cayenne_datalake_clustering_columns: tenant_id,created_at
        # Graduate the warm tier to cold once it reaches 8 GiB
        cayenne_datalake_warm_max_bytes: 8589934592

datasets:
  - from: postgres:public.events
    name: events
    acceleration:
      engine: cayenne
      mode: file
      refresh_mode: changes
      primary_key: event_id
      params:
        # Enable the cold object-store tier (general-purpose S3, not S3 Express)
        cayenne_datalake_location: s3://my-cold-bucket/events/
        cayenne_datalake_s3_region: us-west-2
        cayenne_datalake_clustering_columns: tenant_id,created_at
        # Graduate the warm tier to cold once it reaches 8 GiB
        cayenne_datalake_warm_max_bytes: 8589934592

Data Type Support

Cayenne (via Vortex) supports most Arrow data types with the following considerations:

Fully Supported Types

• All integer types (Int8, Int16, Int32, Int64, UInt*)
• Floating point (Float32, Float64)
• Boolean
• Utf8 and LargeUtf8 strings
• Binary and LargeBinary
• Timestamps, preserving the source unit and timezone
• Date32 and Date64
• Lists and FixedSizeLists
• Maps
• Structs

Automatically Converted Types

:::note Tables created before v2.2.0

These tables continue to normalize timestamps to microseconds. Preserving the source unit requires recreating the table with mode: file_create in an empty directory. The on_schema_change (opens in a new tab) setting does not migrate existing timestamps.

:::

Unsupported Types

The following types require the unsupported_type_action parameter:

• Interval types
• Duration types
• FixedSizeBinary

unsupported_type_action options:

acceleration:
  engine: cayenne
  mode: file
  params:
    unsupported_type_action: string  # Convert unsupported types to strings

acceleration:
  engine: cayenne
  mode: file
  params:
    unsupported_type_action: string  # Convert unsupported types to strings

Resource Considerations

Resource requirements for Spice Cayenne depend on dataset size, query patterns, and cache configuration.

Memory

Spice Cayenne manages memory efficiently through columnar storage and selective caching. Memory allocation should account for:

Example - Memory-constrained environment:

runtime:
  params:
    cayenne_footer_cache_mb: 64
    # One segment-cache budget for the whole process, not per dataset
    cayenne_segment_cache_mb: 128

datasets:
  - from: s3://my-bucket/data/
    name: constrained_data
    acceleration:
      engine: cayenne
      mode: file

runtime:
  params:
    cayenne_footer_cache_mb: 64
    # One segment-cache budget for the whole process, not per dataset
    cayenne_segment_cache_mb: 128

datasets:
  - from: s3://my-bucket/data/
    name: constrained_data
    acceleration:
      engine: cayenne
      mode: file

Storage

Spice Cayenne stores data in a columnar format optimized for analytical queries. Storage requirements include:

• Acceleration data: Compressed Vortex files (typically 30-50% of raw data size with btrblocks)
• Metadata: SQLite database for catalog and statistics (~10 MB per 1000 files)
• Temporary files: Query spill files during complex operations

Metastore location

One metastore holds the catalog — manifests, snapshot pointers, and partition rows — for every Cayenne dataset sharing a cayenne_metadata_dir. Recreating a dataset deletes its data directory, so a metastore directory that resolves on or beneath that data directory would be deleted along with it, taking the catalog for unrelated datasets.

A file-mode Cayenne dataset whose resolved metastore directory falls inside its resolved data directory therefore fails to load, naming the dataset and both paths. The stock defaults reach this without any operator error: the data directory defaults to {spice_data_path}/{dataset_name}/ and the metastore to {spice_data_path}/metadata, so a dataset named metadata collides. An explicit cayenne_metadata_dir set beneath a dataset's data directory collides the same way. Resolve it by pointing cayenne_metadata_dir outside the data directory, or by renaming the dataset.

Paths are compared after ./.. are collapsed and symlinks are resolved, so neither hides an overlap, and a sibling that merely shares a name prefix (…/meta next to …/metadata) is not affected. Datasets whose data lives on object storage (for example an S3 Express cayenne_file_path) are exempt — the metastore is always local, so it cannot sit inside an object-store data path.

CPU

Query performance scales with available CPU cores. Vortex's columnar format supports parallel decompression and scanning across multiple threads. Allocate sufficient CPU for:

• Query execution parallelism
• Data refresh and compression operations
• Concurrent query workloads

Transactions

Cayenne supports serializable, gated transactions on accelerator-only Cayenne tables. A client submits a single BEGIN … COMMIT SQL body — over the HTTP /v1/sql endpoint or FlightSQL — and every statement in the body commits atomically, or not at all:

BEGIN;
SELECT assert((SELECT balance FROM accounts WHERE id = 1) >= 100);  -- optional gate
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
UPDATE accounts SET balance = balance + 100 WHERE id = 2;
COMMIT;

BEGIN;
SELECT assert((SELECT balance FROM accounts WHERE id = 1) >= 100);  -- optional gate
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
UPDATE accounts SET balance = balance + 100 WHERE id = 2;
COMMIT;

How it works:

• Detection is automatic. A multi-statement body whose first statement is BEGIN (or START TRANSACTION) and whose last statement is COMMIT is executed as a transaction — there is no configuration flag to enable it. Every statement runs through the standard query path, so authorization, column masking, logging, and tracing apply to each.
• Atomicity and isolation. Writes are staged off-lock and published together in a single metastore transaction at COMMIT. Each participant table's version is captured at BEGIN and re-validated at COMMIT using per-key optimistic concurrency control (OCC). If a participant changed since the transaction started, COMMIT fails with a retryable conflict (HTTP 409 on /v1/sql) — re-run the transaction against the latest committed state. Any statement error (including a failed gate) rolls back every staged write.
• Preconditions with assert(). The assert(<boolean expression>) function evaluates its argument at execution time. If the expression is false or NULL, the transaction aborts with assertion failed: gate expression was false or NULL. Use it to enforce invariants (for example, a sufficient balance) as part of the transaction. Comparison gates such as … >= cap are NULL-safe.
• Return value. On success, /v1/sql returns the final statement's result (for the canonical gate-plus-write shape, the last write's row-count summary), or COMMIT when the body produces no rows.

Durable federated write-back: A Cayenne dataset configured with write_mode: write_back (opens in a new tab), on_conflict, and refresh_mode: changes (CDC) stages its committed writes to the accelerator and then reconciles them asynchronously back to the federated source by a per-table write-back worker. write_mode: write_back requires replication.enabled: true as an explicit opt-in to asynchronous source durability.

:::warning Durable write-back is not currently available Reconciling a row to the source is delivered as a separate DELETE followed by an INSERT. Because the accelerator is CDC-fed from that same source, the standalone DELETE echoes back over the changes stream and removes the committed row from the accelerator; if the follow-up INSERT then fails, the write is silently gone from both sides. Closing that window needs a source that can apply a delivered row in one atomic step — a single transaction covering both legs, or a native conditional upsert.

No connector advertises that capability yet, so rather than accept a configuration that can lose a committed write, Spice now rejects the dataset at registration:

Failed to register dataset <name> (<connector>): durable write-back needs a source that can apply a delivered row in one atomic step, and the <connector> connector cannot yet.

Remove on_conflict to keep writes on the accelerator, or choose a different acceleration.write_mode (opens in a new tab). Per-connector atomic delivery is planned; until it lands, this combination cannot be loaded. :::

Requirements and v1 limitations:

• Write targets must be accelerator-only, non-partitioned Cayenne datasets. Other dataset modes route writes to the federated source — where the gate cannot govern them — and are rejected. Durable write-back datasets are not currently loadable (see the warning above).
• Only INSERT and UPDATE writes are supported inside a transaction. DELETE and MERGE are rejected.
• At most one write per table per transaction. Multiple tables may be written in the same transaction and are committed atomically together.
• Reading a Cayenne table that is not a registered participant (for example, a partitioned table) fails the transaction closed.
• Nullability-predicate gates (IS NOT NULL, IS NULL, COALESCE) are not yet reliable — prefer comparison gates.
• A gated write whose superseded row is still in the in-memory/inline tier (recently CDC-streamed rows) is rejected; file-backed rows are supported.

Limitations

Consider the following limitations when using Spice Cayenne acceleration:

• Memory Mode Constraints: mode: memory (fully in-RAM, ephemeral) is supported alongside mode: file, but it does not persist any data (the dataset reloads from its source on restart), does not support partitioned tables (partition_by), and enforces a hard per-table RAM bound instead of spilling to disk — a breach returns an error rather than growing without limit. Use mode: file when persistence across restarts is required.
• S3 Express Only: Standard S3 buckets are not supported for remote storage. Only S3 Express One Zone directory buckets are supported.
• Unsupported Data Types: Interval, Duration, and FixedSizeBinary types require unsupported_type_action configuration.
• No Traditional Indexes: Spice Cayenne does not support explicit index creation via the indexes configuration. Vortex's segment statistics and fast random access encodings provide equivalent or better performance for most point lookup workloads.
• No MVCC: Multi-version concurrency control is not yet implemented. Snapshots and time-travel queries are planned for future releases.
• Transaction Constraints: Transactions (opens in a new tab) support gated INSERT/UPDATE writes on accelerator-only, non-partitioned Cayenne tables only (no DELETE/MERGE, one write per table). See Transactions (opens in a new tab) for the full list.
• No refresh_append_overlap in v2.2.0: In v2.2.0, a dataset accelerated by Spice Cayenne that sets acceleration.refresh_append_overlap (opens in a new tab) fails to load, with Cayenne data accelerator does not yet support refresh_append_overlap. Please remove this configuration. refresh_mode: append (opens in a new tab) itself is supported — only the overlap window is not, so late-arriving rows behind the high-water mark are missed rather than re-read. The check runs during file-mode initialization, so a mode: memory dataset is not rejected. v2.2.1 supports the overlap window: the high-water mark is moved back by the configured duration and the re-fetched rows are deduplicated before the write.

Example Spicepod

Complete example configuration using Spice Cayenne with performance tuning:

version: v1
kind: Spicepod
name: cayenne-example

runtime:
  query:
    memory_limit: 4GiB
    temp_directory: /tmp/spice
  params:
    # Engine-global Cayenne runtime tuning (shared by all Cayenne datasets)
    cayenne_footer_cache_mb: 256
    cayenne_segment_cache_mb: 512

datasets:
  # Local file storage example with upsert
  - from: s3://source-bucket/analytics/
    name: analytics_data
    params:
      file_format: parquet
    time_column: created_at
    acceleration:
      engine: cayenne
      enabled: true
      mode: file
      primary_key: id
      on_conflict:
        id: upsert
      refresh_mode: append
      refresh_check_interval: 1h
      params:
        cayenne_compression_strategy: btrblocks
        cayenne_target_file_size_mb: 64
        sort_columns: created_at,id
      retention_sql: DELETE FROM analytics_data WHERE created_at < NOW() - INTERVAL '30 days'

  # S3 Express One Zone storage example
  - from: kafka:realtime-events
    name: realtime_events
    acceleration:
      engine: cayenne
      enabled: true
      mode: file
      primary_key: event_id
      refresh_mode: append
      params:
        # S3 Express One Zone for low-latency persistence
        cayenne_s3_zone_ids: usw2-az1
        cayenne_s3_region: us-west-2
        cayenne_compression_strategy: zstd  # Fast writes for streaming
        cayenne_target_file_size_mb: 32     # Smaller files for faster ingestion

version: v1
kind: Spicepod
name: cayenne-example

runtime:
  query:
    memory_limit: 4GiB
    temp_directory: /tmp/spice
  params:
    # Engine-global Cayenne runtime tuning (shared by all Cayenne datasets)
    cayenne_footer_cache_mb: 256
    cayenne_segment_cache_mb: 512

datasets:
  # Local file storage example with upsert
  - from: s3://source-bucket/analytics/
    name: analytics_data
    params:
      file_format: parquet
    time_column: created_at
    acceleration:
      engine: cayenne
      enabled: true
      mode: file
      primary_key: id
      on_conflict:
        id: upsert
      refresh_mode: append
      refresh_check_interval: 1h
      params:
        cayenne_compression_strategy: btrblocks
        cayenne_target_file_size_mb: 64
        sort_columns: created_at,id
      retention_sql: DELETE FROM analytics_data WHERE created_at < NOW() - INTERVAL '30 days'

  # S3 Express One Zone storage example
  - from: kafka:realtime-events
    name: realtime_events
    acceleration:
      engine: cayenne
      enabled: true
      mode: file
      primary_key: event_id
      refresh_mode: append
      params:
        # S3 Express One Zone for low-latency persistence
        cayenne_s3_zone_ids: usw2-az1
        cayenne_s3_region: us-west-2
        cayenne_compression_strategy: zstd  # Fast writes for streaming
        cayenne_target_file_size_mb: 32     # Smaller files for faster ingestion

Cookbook

• A cookbook recipe to configure Cayenne as a data accelerator in Spice. Cayenne Data Accelerator (opens in a new tab)

Related Documentation

Spice Documentation:

• Performance Tuning (opens in a new tab) - Comprehensive performance optimization guide
• Managing Memory Usage (opens in a new tab) - Memory configuration reference
• Data Acceleration (opens in a new tab) - Data acceleration overview

External References:

• Apache DataFusion (opens in a new tab) - Query execution engine
• DataFusion Configuration (opens in a new tab) - DataFusion settings and tuning
• Vortex Project (opens in a new tab) - Columnar file format
• Vortex Benchmarks (opens in a new tab) - Performance benchmarks
• FSST Paper (opens in a new tab) - Fast Static Symbol Table compression
• FastLanes Paper (opens in a new tab) - High-performance integer encoding
• ALP Paper (opens in a new tab) - Adaptive floating-point compression
• BtrBlocks Paper (opens in a new tab) - Compression algorithm
• AWS S3 Express One Zone (opens in a new tab) - Low-latency object storage