title: 'Spice Cayenne Data Accelerator' sidebar_label: 'Spice Cayenne Data Accelerator' description: 'Spice Cayenne Data Accelerator (Vortex) Documentation' sidebar_position: 1 tags:

• cayenne
• vortex
• data-accelerators
• performance
• s3-express

Spice Cayenne is a data acceleration engine designed for high-performance, scalable query on large-scale datasets. Built on Vortex (opens in a new tab), a high-performance columnar file format, Spice Cayenne combines columnar storage with in-process metadata management to provide fast query performance to scale to datasets beyond 1TB.

Why Vortex?

Spice Cayenne uses Vortex as its storage format, providing significant performance advantages:

• 100x faster random access reads compared to modern Apache Parquet
• 10-20x faster scans for analytical queries
• 5x faster writes with similar compression ratios
• Zero-copy compatibility with Apache Arrow for efficient data processing
• Extensible architecture with pluggable encoding, compression, and layout strategies

Vortex is a Linux Foundation (LF AI & Data) project under Apache-2.0 license with neutral governance. For performance benchmarks, see bench.vortex.dev (opens in a new tab).

While DuckDB (opens in a new tab) excels for datasets up to approximately 1TB, Spice Cayenne with Vortex is designed to scale beyond these limits.

Architecture

Spice Cayenne follows a lakehouse architecture inspired by DuckLake (opens in a new tab), separating metadata management from data storage:

Key Design Principles:

• Virtual Files: Each "file" is a Vortex ListingTable at a unique directory, enabling append operations and parallel reads
• Lazy Statistics: Summary statistics are loaded on-demand for query optimization
• Sequence-based Ordering: Iceberg-style sequence numbers enable upsert semantics without requiring separate tracking of "undeleted" records (rows that were deleted and then re-inserted)
• Pluggable Storage: Data files can be stored locally or in S3 Express One Zone while metadata remains local

Storage Recommendations

For optimal performance, store Cayenne data files on NVMe storage. NVMe provides the lowest latency and highest throughput for the random access patterns that Vortex files require.

Use S3 Express One Zone (opens in a new tab) when persistence of accelerations across restarts is required. S3 Express One Zone adds network latency compared to local NVMe but provides durability. Sharing accelerated data across multiple Spice instances is planned for a future release.

Configuration

To use Spice Cayenne as the data accelerator, specify cayenne as the engine for acceleration. Spice Cayenne supports mode: file, mode: file_create, and mode: file_update and stores data on disk.

datasets:
  - from: spice.ai:path.to.my_dataset
    name: my_dataset
    acceleration:
      engine: cayenne
      mode: file

datasets:
  - from: spice.ai:path.to.my_dataset
    name: my_dataset
    acceleration:
      engine: cayenne
      mode: file

Parameters

Spice Cayenne is configured through two distinct parameter scopes:

• Acceleration parameters are set per dataset under acceleration.params and control how that dataset's accelerated data is stored, compressed, written, and compacted.
• Runtime parameters are set once per instance under runtime.params and control engine-global behavior — caches, optimizer rules, and dedicated memory pools — shared by every Cayenne-accelerated dataset.

The two scopes are not interchangeable: setting a runtime parameter under acceleration.params (or a per-dataset parameter under runtime.params) has no effect — the value is ignored.

Acceleration parameters (acceleration.params)

Set under a dataset's acceleration.params:

S3 Express One Zone parameters

These are acceleration parameters (set under acceleration.params) used when storing Cayenne data files in S3 Express One Zone:

Runtime parameters (runtime.params)

Set once under the top-level runtime.params and applied to every Cayenne-accelerated dataset in the instance. With the exception of the goal-driven SLO setpoints (cayenne_goal_*) — which set a global default that a dataset can override under its own acceleration.params (except cayenne_goal_qph, which is global-only) — these are not valid under a dataset's acceleration.params:

runtime:
  params:
    # Engine-global Cayenne tuning, shared by every Cayenne-accelerated dataset
    cayenne_footer_cache_mb: 512
    cayenne_filter_propagation: enabled

datasets:
  - from: s3://analytics-bucket/events/
    name: events
    acceleration:
      engine: cayenne
      mode: file
      params:
        # Per-dataset Cayenne tuning
        cayenne_segment_cache_mb: 1024

runtime:
  params:
    # Engine-global Cayenne tuning, shared by every Cayenne-accelerated dataset
    cayenne_footer_cache_mb: 512
    cayenne_filter_propagation: enabled

datasets:
  - from: s3://analytics-bucket/events/
    name: events
    acceleration:
      engine: cayenne
      mode: file
      params:
        # Per-dataset Cayenne tuning
        cayenne_segment_cache_mb: 1024

Performance Tuning

Spice Cayenne performance can be optimized through cache configuration, compression strategy selection, and resource allocation. By default Cayenne is self-tuning — see Self-Tuning (opens in a new tab) — so manual tuning is only needed to override a specific knob.

Self-Tuning

Cayenne sizes its memory-, CPU-, and storage-sensitive knobs automatically so it runs well on any host without hand-tuning, from a small container to a large multi-core box across different storage classes. Every numeric cayenne_* knob also accepts the literal auto, which lets the runtime derive that knob's value while leaving the rest of the configuration untouched.

The mode is controlled by the cayenne_tuning acceleration parameter:

• auto — derive the correct configuration values statically from the detected environment (cgroup-aware cores and memory, storage class) and the inferred schema (cardinality, row width, primary key). No feedback loop runs.
• adaptive (preview) — in addition to the static derivation, run a per-table closed-feedback controller that measures the live CDC ingest rate and the runtime's response (apply latency vs. offered load, read amplification, memory pressure) and adjusts the inline-memtable flush caps, compaction cadence/trigger, and write concurrency over time. Adjustments are bounded by the same environment-derived [floor, ceiling] the static tier uses, so the loop can only ever pick a value the static tier could have picked.

Environment detection feeds both modes. On AWS EC2, the runtime additionally probes the Instance Metadata Service (IMDSv2) (opens in a new tab) once at first table registration to detect T-family burstable CPU and the instance's EBS baseline bandwidth, and measures real storage write throughput with a cloud-agnostic calibration probe — both refine the environment-derived [floor, ceiling] bounds. The IMDS probe is non-blocking and fail-open: it uses a tight timeout and a single attempt, so off-AWS (or when the metadata service is unreachable) it is a fast no-op and detection falls back to the calibration probe alone. To skip the IMDS probe entirely, set SPICE_DISABLE_IMDS=1 (the standard AWS AWS_EC2_METADATA_DISABLED environment variable is also honored).

When cayenne_tuning is left unset, the mode defaults to adaptive if the dataset opted into schema_inference: extended and the source emitted the inferred metadata; otherwise it defaults to auto. Opting into extended schema is treated as the signal that you want Cayenne to self-tune, and that metadata also seeds the controller's warm-start. To keep the static auto mode even with extended schema enabled, set cayenne_tuning: auto explicitly.

acceleration:
  engine: cayenne
  params:
    cayenne_tuning: adaptive

acceleration:
  engine: cayenne
  params:
    cayenne_tuning: adaptive

:::warning[Preview]

cayenne_tuning: adaptive is in preview. Cayenne logs a startup warning when it is enabled; verify query correctness and performance before using it for production workloads. The static auto mode is recommended for production.

:::

adaptive needs a non-zero cayenne_compaction_background_interval_ms, since the controller runs on the background compaction tick; if it is 0, Cayenne logs a warning and falls back to auto. schema_inference: extended sharpens the adaptive warm-start — the loop's data-aware warm-start uses the inferred cardinality and size — but is not required for an explicit cayenne_tuning: adaptive; without it the controller relearns the observed row width from live ingest and converges from the hardware-derived warm-start.

In both modes, setting any cayenne_* knob to an explicit value overrides the derived value. Under adaptive, an explicitly-set knob is pinned — the controller will not move it.

Goal-driven tuning

Under adaptive, you can give the controller high-level service-level objectives (SLOs) instead of leaving it to optimize from its built-in signals alone. Set one or more cayenne_goal_* SLOs — globally under runtime.params, where they apply to every Cayenne-accelerated dataset — and the closed loop converges toward each target in small, bounded steps:

The time-based goals accept a duration string (e.g. 5s, 1m, 250ms); cayenne_goal_qph is a positive number.

Scope each goal where the runtime reads it:

• Time-based goals (cayenne_goal_replication_lag, cayenne_goal_freshness, cayenne_goal_query_latency) are set globally under runtime.params and may be overridden per-dataset under a dataset's acceleration.params.
• cayenne_goal_qph is global-only: query throughput is measured system-wide — a query spanning multiple datasets (such as a join) is counted once — so it is read only from runtime.params, and a value set under acceleration.params is ignored.
• cayenne_goal_convergence_window (duration, default 60s) sets the time budget the controller targets for convergence. It is a per-dataset control-cadence knob (set under acceleration.params) and is not part of the global SLO surface.

Setting any cayenne_goal_* parameter implies the closed loop — a goal is inert without it — so the goals also enable adaptive unless you explicitly set cayenne_tuning: auto (in which case the goals are ignored and Cayenne logs a warning). Goal-seeking also requires a non-zero cayenne_compaction_background_interval_ms, like adaptive itself.

runtime:
  params:
    # Global SLOs — apply to every Cayenne-accelerated dataset
    cayenne_goal_replication_lag: 5s
    cayenne_goal_query_latency: 250ms
    cayenne_goal_qph: 5000 # global-only — no per-dataset form

datasets:
  - from: postgres:public.orders
    name: orders
    acceleration:
      engine: cayenne
      params:
        # Per-dataset override of the global query-latency SLO
        cayenne_goal_query_latency: 100ms
        cayenne_goal_convergence_window: 1m

runtime:
  params:
    # Global SLOs — apply to every Cayenne-accelerated dataset
    cayenne_goal_replication_lag: 5s
    cayenne_goal_query_latency: 250ms
    cayenne_goal_qph: 5000 # global-only — no per-dataset form

datasets:
  - from: postgres:public.orders
    name: orders
    acceleration:
      engine: cayenne
      params:
        # Per-dataset override of the global query-latency SLO
        cayenne_goal_query_latency: 100ms
        cayenne_goal_convergence_window: 1m

Cache Tuning

Spice Cayenne uses two in-memory caches to accelerate query performance:

Footer Cache (cayenne_footer_cache_mb) — runtime parameter:

The footer cache stores Vortex file metadata, including schemas, statistics, and encoding information. It is engine-global and shared across every Cayenne-accelerated dataset, so it is set under runtime.params, not per dataset. Larger cache sizes benefit workloads with many files.

• Default: unset — when omitted, DataFusion's default file-metadata-cache limit of 50 MB applies
• Increase for datasets with many small files
• Each file requires approximately 1-10 KB of footer cache

Segment Cache (cayenne_segment_cache_mb) — acceleration parameter:

The segment cache stores decompressed data segments. It is configured per dataset under acceleration.params. Larger cache sizes benefit workloads with repeated queries on the same data.

• Default: 256 MB
• Increase for workloads with hot data patterns
• Size based on frequently accessed data volume

Example - High-throughput configuration:

runtime:
  params:
    # Engine-global footer cache, shared by all Cayenne datasets
    cayenne_footer_cache_mb: 512

datasets:
  - from: s3://analytics-bucket/events/
    name: events
    acceleration:
      engine: cayenne
      mode: file
      params:
        # Per-dataset segment cache
        cayenne_segment_cache_mb: 1024

runtime:
  params:
    # Engine-global footer cache, shared by all Cayenne datasets
    cayenne_footer_cache_mb: 512

datasets:
  - from: s3://analytics-bucket/events/
    name: events
    acceleration:
      engine: cayenne
      mode: file
      params:
        # Per-dataset segment cache
        cayenne_segment_cache_mb: 1024

Compression Strategy

Spice Cayenne supports two compression strategies, each with different performance characteristics. The BtrBlocks (opens in a new tab) compression algorithm is designed for fast analytical queries, while zstd (opens in a new tab) provides fast write performance. Additionally, zstd achieves better compression ratios when data contains large chunks of binary or text.

Example - Write-optimized configuration:

datasets:
  - from: kafka:events
    name: realtime_events
    acceleration:
      engine: cayenne
      mode: file
      refresh_mode: append
      params:
        cayenne_compression_strategy: zstd

datasets:
  - from: kafka:events
    name: realtime_events
    acceleration:
      engine: cayenne
      mode: file
      refresh_mode: append
      params:
        cayenne_compression_strategy: zstd

File Size Tuning

The cayenne_target_file_size_mb parameter controls when new Vortex files are created during writes:

• Smaller files (32-64 MB): Better parallelism, finer-grained statistics, faster ingestion
• Larger files (128-256 MB): Fewer files to manage, reduced metadata overhead

params:
  cayenne_target_file_size_mb: 64  # More parallelism for high-concurrency workloads

params:
  cayenne_target_file_size_mb: 64  # More parallelism for high-concurrency workloads

Features

DataFusion Query-Native Execution

Spice Cayenne is DataFusion query-native, meaning all query execution uses Apache DataFusion (opens in a new tab) and adheres to the runtime.query.memory_limit setting. This provides:

• Vectorized execution: Multi-threaded, SIMD-optimized query processing
• Automatic memory management: Query memory is tracked and spilled to disk when limits are exceeded
• Dynamic filter pushdown: Filters from TopK, Join, and Aggregate operators push down to file scans

DataFusion's GreedyMemoryPool (opens in a new tab) allows memory reservations on a first-come, first-served basis, improving throughput for high-concurrency queries with many partitions.

High-Performance Columnar Storage

Spice Cayenne uses Vortex's advanced columnar format, which provides:

• Efficient Compression: Cascading compression with nested encoding schemes including RLE, dictionary encoding, FastLanes, FSST, and ALP
• Rich Statistics: Lazy-loaded summary statistics for query optimization
• Extensible Encodings: Pluggable physical layouts optimized for different data patterns
• Wide Table Support: Efficient handling of tables with many columns through zero-copy metadata access

Point Lookups and Random Access

Vortex delivers 100x faster random access reads (opens in a new tab) compared to Apache Parquet through several architectural features:

Segment Statistics (Zone-Map Equivalent):

Vortex's ChunkedLayout (opens in a new tab) maintains per-segment statistics for each column, enabling segment pruning during query execution. Statistics include:

When a query includes a WHERE clause, Spice Cayenne evaluates whether each segment could contain matching rows. Segments that cannot match based on min/max statistics are skipped entirely, similar to DuckDB's zone-maps (opens in a new tab) without requiring explicit index creation.

Example - Segment Pruning:

For a table with segments containing timestamp ranges [2024-01-01, 2024-01-15], [2024-01-16, 2024-01-31], [2024-02-01, 2024-02-15], a query:

SELECT * FROM events WHERE timestamp > '2024-01-20';

SELECT * FROM events WHERE timestamp > '2024-01-20';

Prunes the first segment (max < 2024-01-20) and reads only the second and third segments.

Fast Random Access Encodings:

Vortex encodings support direct random access to compressed data:

• FSST (opens in a new tab) (Fast Static Symbol Table): String compression with O(1) random access
• FastLanes (opens in a new tab): High-performance integer encoding with vectorized decoding
• ALP (opens in a new tab): Adaptive lossless floating-point compression with random access

Compute Push-Down:

Vortex supports executing filter and compute operations directly on compressed data, avoiding full decompression for predicate evaluation. This compute push-down reduces CPU and memory overhead by processing data in its compressed form:

Array-level statistics (is_sorted, is_constant, min, max) enable additional optimizations beyond filtering. For example, is_sorted enables binary search for point lookups, and is_constant returns values immediately without scanning.

Performance Characteristics:

For point lookups and selective queries, Spice Cayenne with Vortex often matches or exceeds the performance of traditional B-tree indexes while consuming no additional memory for index structures. Performance scales with:

• Data sorting (sorted columns benefit most from segment pruning)
• Segment cache hit rate (hot data patterns)
• Compression encoding match to data characteristics

Maintained Aggregates

Spice Cayenne can incrementally maintain aggregate views over a CDC-accelerated dataset so that GROUP BY aggregate queries are answered from continuously-updated summary state instead of re-scanning the full table. As change events are applied (refresh_mode: changes), each maintained view is updated with the incoming deltas; a matching query is then rewritten by the physical optimizer to read the maintained state directly. When a query does not match a declared view — or the view is not currently fresh — Cayenne transparently falls back to a full scan, so results are always correct.

Maintained aggregates are declared per dataset under acceleration.maintained_aggregates:

datasets:
  - from: postgres:orders
    name: orders
    acceleration:
      enabled: true
      engine: cayenne
      refresh_mode: changes
      maintained_aggregates:
        - group_by: [customer_id]
          aggregates:
            - function: count
            - function: sum
              column: amount
            - function: avg
              column: latency_ms

datasets:
  - from: postgres:orders
    name: orders
    acceleration:
      enabled: true
      engine: cayenne
      refresh_mode: changes
      maintained_aggregates:
        - group_by: [customer_id]
          aggregates:
            - function: count
            - function: sum
              column: amount
            - function: avg
              column: latency_ms

Each entry declares one maintained view:

Each item in aggregates has:

Supported input column types by function:

• count — any column (or COUNT(*) when column is omitted).
• sum — signed integers (Int8–Int64), unsigned integers (UInt8–UInt64), and floats (Float32/Float64), widening to BIGINT/Float64.
• avg — floats (Float32/Float64), always returning Float64.

Query matching

A query is served from a maintained view only when it matches the view exactly:

• The query's GROUP BY keys match the view's group_by, in order.
• The query's WHERE predicate matches the view's filter_sql exactly — an unfiltered view (no filter_sql) answers only unfiltered queries, and a filtered view answers only a query carrying the identical predicate. A view whose filter_sql mirrors a common dashboard filter lets that filtered analytical query be served incrementally instead of by a full re-scan.

Any query that does not match a declared view (or that reaches the view while it is stale) falls back to the base-table scan — correct, but not accelerated.

Constraints

• Maintained aggregates are a Spice Cayenne feature designed for CDC-accelerated datasets (refresh_mode: changes).
• Maintained aggregates are not supported on partitioned tables (partition_by).

Retaining specs without maintaining them

To keep a view's declaration in configuration without materializing or maintaining it, use the policy form with mode: disabled:

maintained_aggregates:
        mode: disabled
        views:
          - group_by: [customer_id]
            aggregates:
              - function: sum
                column: amount

maintained_aggregates:
        mode: disabled
        views:
          - group_by: [customer_id]
            aggregates:
              - function: sum
                column: amount

When any view is declared with the list form above, maintenance is enabled by default; the policy form's mode: enabled is equivalent.

Deletion Vectors

Spice Cayenne implements efficient deletes without rewriting data files using deletion vectors. Deletion vectors track which rows have been logically deleted, and the information is applied transparently during query execution.

Deletion Strategies

How deletions are recorded and applied is controlled by the cayenne_deletion_mode parameter:

datasets:
  - from: s3://bucket/events/
    name: events
    acceleration:
      engine: cayenne
      mode: file
      primary_key: event_id
      params:
        cayenne_deletion_mode: auto # default; set to `key` to opt out of merge-on-read

datasets:
  - from: s3://bucket/events/
    name: events
    acceleration:
      engine: cayenne
      mode: file
      primary_key: event_id
      params:
        cayenne_deletion_mode: auto # default; set to `key` to opt out of merge-on-read

Under position mode (the auto resolution for all tables except CDC datasets with a primary key):

• Tables without a primary key record deletions by row position. Cayenne uses RoaringBitmap for memory-efficient storage of deleted row IDs, providing 50-90% memory savings compared to HashSet for sparse deletions.
• Tables with a primary key capture row positions via a row_idx() read-back after each write, with a key-based fallback for any row whose position is not yet known. Pushing the deletes into the scan eliminates the per-row RowConverter deletion tax above it.

Key-based deletion (cayenne_deletion_mode: key) uses the byte representation of primary key columns and applies deletes above the scan. This approach is position-independent and survives data reorganization.

Orphaned deletion-vector cleanup

On a primary-key upsert table, a deletion vector becomes orphaned once time-based retention empties the snapshot(s) it shadowed — the surviving-sequence floor rises above the DV's delete sequence, so the DV is a query-time no-op that still occupies disk and the in-memory deletion cache. Cayenne runs a background sweep that reclaims these orphaned .arrow files and their catalog rows.

The sweep is controlled by the cayenne_orphaned_dv_cleanup_min_files acceleration parameter:

• Default 20 — the sweep runs per-table once at least 20 orphaned DVs have accumulated. Set to 0 to disable it.
• Retention-only — a DV is orphaned only after retention (retention_period + time_column) empties the snapshots it shadowed. Without a retention policy nothing is ever orphaned and the sweep is a no-op, at zero ingest cost.
• No correctness impact — orphaned DVs are already query-time no-ops; the benefit is reclaimed disk plus a smaller in-memory deletion cache and less file I/O at restart.
• Off the write path — the sweep is lock-free and throttled, and runs on the compaction runtime; it never extends the write-lock window.

This knob bounds only the orphaned tail. The live (not-yet-orphaned) DV set is bounded separately by compaction.

acceleration:
  engine: cayenne
  mode: file
  primary_key: id
  time_column: created_at
  retention_period: 30d
  params:
    cayenne_orphaned_dv_cleanup_min_files: 20 # default; set 0 to disable

acceleration:
  engine: cayenne
  mode: file
  primary_key: id
  time_column: created_at
  retention_period: 30d
  params:
    cayenne_orphaned_dv_cleanup_min_files: 20 # default; set 0 to disable

Primary Key Optimization

For tables with a single-column Int64 primary key, Cayenne uses an optimized direct lookup strategy that avoids serialization overhead:

datasets:
  - from: s3://bucket/events/
    name: events
    acceleration:
      engine: cayenne
      mode: file
      primary_key: event_id  # Int64 column - uses optimized deletion

datasets:
  - from: s3://bucket/events/
    name: events
    acceleration:
      engine: cayenne
      mode: file
      primary_key: event_id  # Int64 column - uses optimized deletion

Upsert Support

When on_conflict is configured, Cayenne supports upsert semantics using sequence numbers (Iceberg-style ordering):

datasets:
  - from: kafka:events
    name: events
    acceleration:
      engine: cayenne
      mode: file
      primary_key: id
      on_conflict:
        id: upsert

datasets:
  - from: kafka:events
    name: events
    acceleration:
      engine: cayenne
      mode: file
      primary_key: id
      on_conflict:
        id: upsert

When a primary key is deleted and then re-inserted:

1. The new insert gets a higher sequence number than the delete
2. During scan, the delete doesn't apply to data with higher sequence numbers
3. The new data is visible without requiring separate tracking of "undeleted" records

AWS S3 Express One Zone Storage

Spice Cayenne supports storing data files in AWS S3 Express One Zone (opens in a new tab) for single-digit millisecond latency, ideal for latency-sensitive query workloads that require persistence. Metadata remains on local disk for fast catalog operations while data files are stored in S3 Express One Zone.

Why S3 Express One Zone?

S3 Express One Zone directory buckets provide:

• Single-digit millisecond latency: 10x faster than S3 Standard for first-byte latency
• High request throughput: Up to 10x higher request rates than S3 Standard
• Cost efficiency: Lower per-request costs for high-frequency access patterns
• Durability: Same 99.999999999% (11 9s) durability as S3 Standard

S3 Express Examples

Example 1 - Explicit bucket:

datasets:
  - from: s3://source-bucket/events/
    name: analytics_events
    acceleration:
      engine: cayenne
      enabled: true
      mode: file
      params:
        # Store data in S3 Express One Zone bucket
        cayenne_file_path: s3://my-bucket--usw2-az1--x-s3/cayenne/
        cayenne_s3_region: us-west-2

datasets:
  - from: s3://source-bucket/events/
    name: analytics_events
    acceleration:
      engine: cayenne
      enabled: true
      mode: file
      params:
        # Store data in S3 Express One Zone bucket
        cayenne_file_path: s3://my-bucket--usw2-az1--x-s3/cayenne/
        cayenne_s3_region: us-west-2

Example 2 - Auto-generated bucket with IAM role:

datasets:
  - from: postgresql://db/events
    name: fast_events
    acceleration:
      engine: cayenne
      enabled: true
      mode: file
      params:
        # Auto-generates bucket: spice-{spicepod-name}-fast_events--usw2-az1--x-s3
        cayenne_s3_zone_ids: usw2-az1

datasets:
  - from: postgresql://db/events
    name: fast_events
    acceleration:
      engine: cayenne
      enabled: true
      mode: file
      params:
        # Auto-generates bucket: spice-{spicepod-name}-fast_events--usw2-az1--x-s3
        cayenne_s3_zone_ids: usw2-az1

Example 3 - Explicit credentials:

datasets:
  - from: kafka:events
    name: realtime
    acceleration:
      engine: cayenne
      enabled: true
      mode: file
      params:
        cayenne_s3_zone_ids: use1-az4
        cayenne_s3_region: us-east-1
        cayenne_s3_auth: key
        cayenne_s3_key: ${secrets:AWS_ACCESS_KEY_ID}
        cayenne_s3_secret: ${secrets:AWS_SECRET_ACCESS_KEY}

datasets:
  - from: kafka:events
    name: realtime
    acceleration:
      engine: cayenne
      enabled: true
      mode: file
      params:
        cayenne_s3_zone_ids: use1-az4
        cayenne_s3_region: us-east-1
        cayenne_s3_auth: key
        cayenne_s3_key: ${secrets:AWS_ACCESS_KEY_ID}
        cayenne_s3_secret: ${secrets:AWS_SECRET_ACCESS_KEY}

Bucket Naming Conventions

S3 Express One Zone buckets use a specific naming format:

• Format: {base-name}--{zone-id}--x-s3
• Zone ID format: {region-code}-az{number} (e.g., usw2-az1, use1-az4)
• Auto-generated names: spice-{app-name}-{dataset-name}--{zone-id}--x-s3

The zone ID is automatically extracted from the bucket name to configure the correct endpoint.

Supported AWS Regions

S3 Express One Zone is available in select regions. Spice automatically derives the region from zone IDs:

See AWS documentation for the complete list of S3 Express One Zone availability zones (opens in a new tab).

Important Considerations

• Standard S3 not supported: Cayenne currently only supports S3 Express One Zone, not standard S3 buckets.
• Same-AZ optimization: S3 Express One Zone is optimized for same-availability-zone access. For external access, Cayenne uses extended timeouts (a 2-minute per-request timeout by default, configurable via cayenne_s3_client_timeout) and retries.
• Bucket auto-creation: When using cayenne_s3_zone_ids, Spice automatically creates the S3 Express directory bucket if it doesn't exist (requires appropriate IAM permissions).
• Metadata locality: Cayenne metadata (SQLite catalog) remains on local disk. Only data files are stored in S3 Express.

Data Type Support

Cayenne (via Vortex) supports most Arrow data types with the following considerations:

Fully Supported Types

• All integer types (Int8, Int16, Int32, Int64, UInt*)
• Floating point (Float32, Float64)
• Boolean
• Utf8 and LargeUtf8 strings
• Binary and LargeBinary
• Timestamps (normalized to Microsecond precision)
• Date32 and Date64
• Lists and FixedSizeLists
• Maps
• Structs

Automatically Converted Types

Unsupported Types

The following types require the unsupported_type_action parameter:

• Interval types
• Duration types
• FixedSizeBinary

unsupported_type_action options:

acceleration:
  engine: cayenne
  mode: file
  params:
    unsupported_type_action: string  # Convert unsupported types to strings

acceleration:
  engine: cayenne
  mode: file
  params:
    unsupported_type_action: string  # Convert unsupported types to strings

Resource Considerations

Resource requirements for Spice Cayenne depend on dataset size, query patterns, and cache configuration.

Memory

Spice Cayenne manages memory efficiently through columnar storage and selective caching. Memory allocation should account for:

Example - Memory-constrained environment:

runtime:
  params:
    cayenne_footer_cache_mb: 64

datasets:
  - from: s3://my-bucket/data/
    name: constrained_data
    acceleration:
      engine: cayenne
      mode: file
      params:
        cayenne_segment_cache_mb: 128

runtime:
  params:
    cayenne_footer_cache_mb: 64

datasets:
  - from: s3://my-bucket/data/
    name: constrained_data
    acceleration:
      engine: cayenne
      mode: file
      params:
        cayenne_segment_cache_mb: 128

Storage

Spice Cayenne stores data in a columnar format optimized for analytical queries. Storage requirements include:

• Acceleration data: Compressed Vortex files (typically 30-50% of raw data size with btrblocks)
• Metadata: SQLite database for catalog and statistics (~10 MB per 1000 files)
• Temporary files: Query spill files during complex operations

CPU

Query performance scales with available CPU cores. Vortex's columnar format supports parallel decompression and scanning across multiple threads. Allocate sufficient CPU for:

• Query execution parallelism
• Data refresh and compression operations
• Concurrent query workloads

Limitations

Consider the following limitations when using Spice Cayenne acceleration:

• File Mode Only: Spice Cayenne only supports mode: file and does not support in-memory (mode: memory) acceleration.
• S3 Express Only: Standard S3 buckets are not supported for remote storage. Only S3 Express One Zone directory buckets are supported.
• Unsupported Data Types: Interval, Duration, and FixedSizeBinary types require unsupported_type_action configuration.
• No Traditional Indexes: Spice Cayenne does not support explicit index creation via the indexes configuration. Vortex's segment statistics and fast random access encodings provide equivalent or better performance for most point lookup workloads.
• No MVCC: Multi-version concurrency control is not yet implemented. Snapshots and time-travel queries are planned for future releases.
• No refresh_append_overlap: A dataset accelerated by Spice Cayenne that sets acceleration.refresh_append_overlap (opens in a new tab) fails to load, with Cayenne data accelerator does not yet support refresh_append_overlap. Please remove this configuration. refresh_mode: append (opens in a new tab) itself is supported — only the overlap window is not, so late-arriving rows behind the high-water mark are missed rather than re-read.

Example Spicepod

Complete example configuration using Spice Cayenne with performance tuning:

version: v1
kind: Spicepod
name: cayenne-example

runtime:
  query:
    memory_limit: 4GiB
    temp_directory: /tmp/spice
  params:
    # Engine-global Cayenne runtime tuning (shared by all Cayenne datasets)
    cayenne_footer_cache_mb: 256

datasets:
  # Local file storage example with upsert
  - from: s3://source-bucket/analytics/
    name: analytics_data
    params:
      file_format: parquet
    time_column: created_at
    acceleration:
      engine: cayenne
      enabled: true
      mode: file
      primary_key: id
      on_conflict:
        id: upsert
      refresh_mode: append
      refresh_check_interval: 1h
      params:
        cayenne_compression_strategy: btrblocks
        cayenne_segment_cache_mb: 512
        cayenne_target_file_size_mb: 64
        sort_columns: created_at,id
      retention_sql: DELETE FROM analytics_data WHERE created_at < NOW() - INTERVAL '30 days'

  # S3 Express One Zone storage example
  - from: kafka:realtime-events
    name: realtime_events
    acceleration:
      engine: cayenne
      enabled: true
      mode: file
      primary_key: event_id
      refresh_mode: append
      params:
        # S3 Express One Zone for low-latency persistence
        cayenne_s3_zone_ids: usw2-az1
        cayenne_s3_region: us-west-2
        cayenne_compression_strategy: zstd  # Fast writes for streaming
        cayenne_target_file_size_mb: 32     # Smaller files for faster ingestion

version: v1
kind: Spicepod
name: cayenne-example

runtime:
  query:
    memory_limit: 4GiB
    temp_directory: /tmp/spice
  params:
    # Engine-global Cayenne runtime tuning (shared by all Cayenne datasets)
    cayenne_footer_cache_mb: 256

datasets:
  # Local file storage example with upsert
  - from: s3://source-bucket/analytics/
    name: analytics_data
    params:
      file_format: parquet
    time_column: created_at
    acceleration:
      engine: cayenne
      enabled: true
      mode: file
      primary_key: id
      on_conflict:
        id: upsert
      refresh_mode: append
      refresh_check_interval: 1h
      params:
        cayenne_compression_strategy: btrblocks
        cayenne_segment_cache_mb: 512
        cayenne_target_file_size_mb: 64
        sort_columns: created_at,id
      retention_sql: DELETE FROM analytics_data WHERE created_at < NOW() - INTERVAL '30 days'

  # S3 Express One Zone storage example
  - from: kafka:realtime-events
    name: realtime_events
    acceleration:
      engine: cayenne
      enabled: true
      mode: file
      primary_key: event_id
      refresh_mode: append
      params:
        # S3 Express One Zone for low-latency persistence
        cayenne_s3_zone_ids: usw2-az1
        cayenne_s3_region: us-west-2
        cayenne_compression_strategy: zstd  # Fast writes for streaming
        cayenne_target_file_size_mb: 32     # Smaller files for faster ingestion

Cookbook

• A cookbook recipe to configure Cayenne as a data accelerator in Spice. Cayenne Data Accelerator (opens in a new tab)

Related Documentation

Spice Documentation:

• Performance Tuning (opens in a new tab) - Comprehensive performance optimization guide
• Managing Memory Usage (opens in a new tab) - Memory configuration reference
• Data Acceleration (opens in a new tab) - Data acceleration overview

External References:

• Apache DataFusion (opens in a new tab) - Query execution engine
• DataFusion Configuration (opens in a new tab) - DataFusion settings and tuning
• Vortex Project (opens in a new tab) - Columnar file format
• Vortex Benchmarks (opens in a new tab) - Performance benchmarks
• FSST Paper (opens in a new tab) - Fast Static Symbol Table compression
• FastLanes Paper (opens in a new tab) - High-performance integer encoding
• ALP Paper (opens in a new tab) - Adaptive floating-point compression
• BtrBlocks Paper (opens in a new tab) - Compression algorithm
• AWS S3 Express One Zone (opens in a new tab) - Low-latency object storage