date: 2026-07-09 title: 'Spice v2.1.0 (Jul 9, 2026)' type: blog authors: [jeadie] tags: [release, cayenne, cdc, postgres, distributed-query, datafusion, iceberg, ballista, ai, search, security, kafka, duckdb, arrow]

Spice v2.1.0 is now available! 🎉

Spice v2.1.0 is the next minor release of Spice, headlined by high-throughput Cayenne CDC, scaling and resilience improvements to PostgreSQL logical replication, expanded distributed query with Iceberg catalog scans and broadcast joins, and the upgrade to DataFusion v54 (including v53), Arrow v58.3, and Vortex v0.74. The release also adds experimental adaptive self-tuning for the Cayenne accelerator, distributed GLM inference, and a range of security, search, and connector improvements.

Highlights in v2.1.0 include:

• High-Throughput Cayenne CDC (opens in a new tab) — in-memory CDC tier, dedicated compaction runtime, and write-path optimizations that cut replication lag on high-volume CDC workloads
• PostgreSQL Replication at Scale (opens in a new tab) — multiple changes-mode datasets share a single replication slot, unchanged-TOAST recovery, and resilient reconnects across rolling deploys
• Distributed Query (opens in a new tab) — distributed Ballista scans of Iceberg catalog tables, broadcast joins for small dimension tables, and shared scheduler job state with failover
• DataFusion v54 (opens in a new tab) — upgrade to DataFusion v54 (folding in v53), Arrow v58.3, and Vortex v0.74, bringing faster joins, scans, and planning
• Adaptive Self-Tuning (Experimental) (opens in a new tab) — opt-in closed-loop tuning and maintained aggregates that adapt Cayenne to hardware, schema, and live workload

What's New in v2.1.0

High-Throughput Cayenne CDC

A major focus of v2.1 is Spice Cayenne (opens in a new tab) write-path throughput for change-data-capture (HTAP) workloads:

• In-Memory CDC Tier: A new in-memory CDC tier and follow-ups cut replication lag on hot upsert tables, with bounded mem-tier checkpointing and O(1) per-scan deletion views, plus a two-phase off-fence checkpoint on the ingest path.
• Dedicated Compaction Runtime: A dedicated compaction runtime with CDC pipelining and protected snapshots isolates compaction from query and ingest paths, with parallelized deletion-vector writes, per-batch directory-barrier coalescing, and size-aware parallel encode for protected-snapshot compaction.
• Incremental Protected Snapshot Compaction: Incremental compaction of protected snapshots (used in Cayenne's merge-on-read deletion index) reduces disk usage and improves query performance.
• Smaller WAL & Metadata-Only Publish: cayenne_insert_record table IDs are stored as 16-byte raw-UUID BLOBs, cutting CDC WAL volume ~34%; upsert commits publish metadata-only, dropping per-key insert records; transient staged CDC deltas are light-encoded.
• Delta-Write Encoding Levels: A new cayenne_delta_encoding setting (default auto) selects delta-write encoding, and cayenne_compression_strategy: zstd is now fully wired.
• In-Memory CDC Sharding: PK-hash intra-apply sharding parallelizes in-memory CDC apply.
• Scan Safety Under Write: In-flight scans are ref-counted so snapshot GC can't delete Vortex files mid-read; in-RAM scan parallelism, query admission control, and sound scan output ordering improve read behavior under sustained CDC.

Delta-write encoding effort and Vortex compression are tunable per accelerator. cayenne_delta_encoding: auto (the default) size-gates fresh CDC/append writes — small deltas use a light scheme and are re-encoded during compaction — or pin an explicit level 0..10 (7 is the full default cascade); cayenne_compression_strategy selects the Vortex compression:

acceleration:
  engine: cayenne
  refresh_mode: changes
  params:
    cayenne_delta_encoding: auto # 'auto' (default), or pin a level 0..10 (7 = full cascade)
    cayenne_compression_strategy: zstd # 'btrblocks' (default) or 'zstd'

acceleration:
  engine: cayenne
  refresh_mode: changes
  params:
    cayenne_delta_encoding: auto # 'auto' (default), or pin a level 0..10 (7 = full cascade)
    cayenne_compression_strategy: zstd # 'btrblocks' (default) or 'zstd'

Change Data Capture & HTAP

PostgreSQL logical replication (CDC, refresh_mode: changes, introduced in v2.0 (opens in a new tab)) gets significant scaling and resilience work in v2.1:

• Shared Replication Slot: Multiple refresh_mode: changes PostgreSQL (opens in a new tab) datasets on the same connection can name the same pg_replication_slot to share a single replication slot, walsender decoder, and publication, with decoded changes multiplexed by (schema, table) to each dataset. This collapses the slot count from one-per-dataset to one — staying well under Postgres's default max_replication_slots = 10.

datasets:
  - from: postgres:public.orders
    name: orders
    params:
      pg_db: mydb
      pg_replication_slot: spice_cdc # shared slot name
    acceleration:
      refresh_mode: changes
  - from: postgres:public.customers
    name: customers
    params:
      pg_db: mydb
      pg_replication_slot: spice_cdc # same name -> one slot, walsender & publication
    acceleration:
      refresh_mode: changes

datasets:
  - from: postgres:public.orders
    name: orders
    params:
      pg_db: mydb
      pg_replication_slot: spice_cdc # shared slot name
    acceleration:
      refresh_mode: changes
  - from: postgres:public.customers
    name: customers
    params:
      pg_db: mydb
      pg_replication_slot: spice_cdc # same name -> one slot, walsender & publication
    acceleration:
      refresh_mode: changes

• Unchanged-TOAST Recovery: Under REPLICA IDENTITY FULL, when an UPDATE leaves a large TOASTed column unchanged, pgoutput sends an "unchanged" marker; Spice now fills that value from the old tuple — its old value is its current value — so updates no longer error or drop columns. Without an old tuple, the error persists with a hint to enable REPLICA IDENTITY FULL.
• Transient Walsender Contention: Slot-contention errors during rolling deploys — SQLSTATE 55006 ("replication slot is active for PID") and 53300 ("requested standby connections exceeds max_wal_senders") — are now classified as transient and retried with backoff instead of fatally terminating the stream. Replication connections are also released at shutdown start (not process exit), freeing walsender seats for replacement instances.
• Strict CDC Param Validation: PostgreSQL CDC parameters are strictly validated rather than silently defaulted.
• Debezium Schema Evolution: Fixes for Debezium schema-evolution support, including tombstone-message handling and sign-extension of minimal-width base64 decimals.

Distributed Query

Distributed Query (opens in a new tab) gains:

• Distributed Iceberg Catalog Scans: Ballista distributes scans of Iceberg catalog tables across executors.
• Broadcast Joins: Small dimension tables are broadcast to executors for distributed joins.
• Shared Scheduler Job State with Failover: Ballista job state is shared so the scheduler can fail over without losing in-flight work.

Performance & Query Engine

Apache DataFusion (opens in a new tab) is upgraded to v54, folding in v53, alongside Arrow v58.3 and Vortex v0.74 (with a pin bump adding intra-file decode split and a per-execution kernel cache). Two DataFusion releases land in this upgrade:

• DataFusion v54 (release notes (opens in a new tab)): adds LATERAL joins, SQL lambda functions (x -> expr with array_transform/array_filter/array_any_match), spilling nested-loop joins, and a faster arrow-avro reader. Performance work includes morsel-driven Parquet scans (up to ~2x faster for skewed scans), 20-50x faster sort-merge semi/anti/mark joins, redundant-sort-key pruning, NDV-based cardinality estimation, and inner_product/cosine_distance functions.
• DataFusion v53 (release notes (opens in a new tab)): adds LIMIT-aware Parquet row-group pruning, broader filter pushdown through joins and UNION, nested-field pushdown (get_field into the scan), faster query planning (some plans dropping from ~4-5ms to ~100us), and 42 faster built-in functions.

Federation deny-list enforcement and catalog DDL are restored after the DataFusion upgrades, and a cost-based left-deep join reordering rule is added for Cayenne acceleration.

AI & LLM

• Native GLM Support with Distributed Inference: Native GLM model support with surfaced reasoning_content, including tensor-parallel GLM inference. Load a GLM model with model_type: glm4 (glm4moe and glm4moelite are also supported):

models:
  - name: glm
    from: huggingface:huggingface.co/THUDM/glm-4-9b-chat
    params:
      model_type: glm4

models:
  - name: glm
    from: huggingface:huggingface.co/THUDM/glm-4-9b-chat
    params:
      model_type: glm4

For large models, GLM inference can be distributed across nodes (tensor parallelism) via the mistral.rs pure-TCP ring all-reduce backend — no NCCL/system dependency. This is a Spice.ai Enterprise (opens in a new tab) feature requiring the distributed build. Run the same model on each node, changing only node_rank:

models:
  - name: glm
    from: huggingface:huggingface.co/THUDM/glm-4-9b-chat
    params:
      model_type: glm4
      distributed_backend: ring
      nodes: 10.0.4.21,10.0.4.22 # ordered host/IP per rank; the ring backend currently requires exactly 2
      node_rank: 0 # rank of THIS node in [0, world_size); rank 0 serves the API. Set node_rank: 1 on 10.0.4.22

models:
  - name: glm
    from: huggingface:huggingface.co/THUDM/glm-4-9b-chat
    params:
      model_type: glm4
      distributed_backend: ring
      nodes: 10.0.4.21,10.0.4.22 # ordered host/IP per rank; the ring backend currently requires exactly 2
      node_rank: 0 # rank of THIS node in [0, world_size); rank 0 serves the API. Set node_rank: 1 on 10.0.4.22

• NSQL Context Endpoint: A new GET /v1/nsql/context endpoint returns the SQL dialect, dataset schemas (with optional sample rows), and registered functions that Spice injects into natural-language-to-SQL (POST /v1/nsql) requests — useful for inspecting or caching exactly what the model sees:

# Inspect the context injected into /v1/nsql requests (examples_limit default 3, max 100)
curl "http://localhost:8090/v1/nsql/context?include_examples=true&examples_limit=3"

# Inspect the context injected into /v1/nsql requests (examples_limit default 3, max 100)
curl "http://localhost:8090/v1/nsql/context?include_examples=true&examples_limit=3"

Returns the dialect, per-dataset schema (keys, indexes, searchable columns), the registered function inventory, and sample rows (abbreviated):

{
  "context": "# Spice.ai NSQL Context",
  "instructions": [
    "Write SQL for the Spice runtime, which uses Apache DataFusion with the SQL parser configured for the PostgreSQL dialect.",
    "Use table and column descriptions, primary keys, foreign keys, unique constraints, and indexes when choosing joins and filters."
  ],
  "sql": {
    "engine": "Apache DataFusion",
    "version": "54.0.0",
    "dialect": "PostgreSQL",
    "parser": "DataFusion SQL parser configured with PostgreSQL dialect"
  },
  "datasets": [
    {
      "name": "sales.orders",
      "table": "orders",
      "description": "Customer orders",
      "columns": [
        { "name": "order_id", "data_type": "Int64", "nullable": false, "primary_key": true, "indexed": true },
        { "name": "customer_id", "data_type": "Utf8", "nullable": false, "vector_search": true, "full_text_search": true }
      ],
      "primary_key": ["order_id"],
      "foreign_keys": [
        { "columns": ["customer_id"], "foreign_table": "spice.sales.customers", "foreign_columns": ["id"] }
      ]
    }
  ],
  "functions": {
    "summary": "Spice SQL runs on Apache DataFusion ... Run SELECT * FROM list_udfs() to inspect the full registered function inventory",
    "search": [
      { "name": "vector_search", "syntax": "vector_search(dataset, 'query text'[, column])" },
      { "name": "text_search", "syntax": "text_search(dataset, 'query text'[, column])" }
    ]
  },
  "samples": [
    { "title": "Example rows for `sales.orders`", "content": "| order_id | customer_id |\n| --- | --- |\n| 42 | CUST-1 |" }
  ]
}

{
  "context": "# Spice.ai NSQL Context",
  "instructions": [
    "Write SQL for the Spice runtime, which uses Apache DataFusion with the SQL parser configured for the PostgreSQL dialect.",
    "Use table and column descriptions, primary keys, foreign keys, unique constraints, and indexes when choosing joins and filters."
  ],
  "sql": {
    "engine": "Apache DataFusion",
    "version": "54.0.0",
    "dialect": "PostgreSQL",
    "parser": "DataFusion SQL parser configured with PostgreSQL dialect"
  },
  "datasets": [
    {
      "name": "sales.orders",
      "table": "orders",
      "description": "Customer orders",
      "columns": [
        { "name": "order_id", "data_type": "Int64", "nullable": false, "primary_key": true, "indexed": true },
        { "name": "customer_id", "data_type": "Utf8", "nullable": false, "vector_search": true, "full_text_search": true }
      ],
      "primary_key": ["order_id"],
      "foreign_keys": [
        { "columns": ["customer_id"], "foreign_table": "spice.sales.customers", "foreign_columns": ["id"] }
      ]
    }
  ],
  "functions": {
    "summary": "Spice SQL runs on Apache DataFusion ... Run SELECT * FROM list_udfs() to inspect the full registered function inventory",
    "search": [
      { "name": "vector_search", "syntax": "vector_search(dataset, 'query text'[, column])" },
      { "name": "text_search", "syntax": "text_search(dataset, 'query text'[, column])" }
    ]
  },
  "samples": [
    { "title": "Example rows for `sales.orders`", "content": "| order_id | customer_id |\n| --- | --- |\n| 42 | CUST-1 |" }
  ]
}

Search & Vectors

• S3 Vectors Pagination: QueryVectors paginates for top-K up to 10,000.
• Elasticsearch kNN Candidate Pool: The default kNN candidate pool is raised from 10 to 1000 for better recall.

SQL & Query Engine

• FlightSQL Substrait Plans: CommandStatementSubstraitPlan support.
• Large Result Streaming: Flight streaming is optimized for large result sets.
• Write Authorization: The SQL tool allows writes for ReadWrite API keys.
• Schema Evolution Policies: on_schema_change supports widening-only evolution and a drop_and_recreate policy.

Security & Connectors

• Kafka mTLS: Mutual TLS configuration is surfaced in the Kafka data connector.
• Secret Resolution at Startup: Secret references are checked and reported at startup.
• DuckDB HNSW: Upgrade to DuckDB v1.5.3 with the statically linked VSS (HNSW) vector extension.

Adaptive Self-Tuning (Experimental)

The Spice Cayenne accelerator gains experimental opt-in self-tuning. cayenne_tuning: auto derives configuration from the detected hardware and inferred schema, while adaptive additionally runs a per-table closed-feedback controller that adapts flush caps, the in-memory CDC tier, compaction cadence, and write concurrency toward operator SLOs (replication lag, freshness, query latency, queries-per-hour). Cayenne can also maintain aggregates incrementally — with predicate-aware delta serving and incremental retraction — and fold whole-table SUM/AVG/COUNT/MIN/MAX from statistics. These features are experimental and disabled by default.

datasets:
  - from: postgres:public.orders
    name: orders
    acceleration:
      engine: cayenne
      refresh_mode: changes
      params:
        cayenne_tuning: adaptive # 'auto' (static, env- + schema-derived) or 'adaptive' (closed-loop)

datasets:
  - from: postgres:public.orders
    name: orders
    acceleration:
      engine: cayenne
      refresh_mode: changes
      params:
        cayenne_tuning: adaptive # 'auto' (static, env- + schema-derived) or 'adaptive' (closed-loop)

Observability

• Per-Dataset Query Attribution: The query_executions metric gains a datasets dimension.
• HTAP Diagnostics: Improved HTAP replication diagnostics on non-convergence.
• Cayenne Write Observability: Write-phase observability for the in-memory CDC tier.

Notable Bug Fixes

• Cayenne Utf8View: The Utf8View read schema avoids a hash-join offset overflow.
• Cayenne metastore: cayenne_metastore: turso is honored for partitioned tables and the dataset checkpoint.
• Dual-write detection: Dual-write accelerated tables are detected behind the metadata-enrichment wrapper.
• digest_many collisions: Values are length-prefixed so column boundaries can't collide.
• Turso WAL checkpoint: WAL checkpoints route through the native Turso connection.
• TLS status probe: The status check probes the metrics endpoint over HTTPS when TLS is enabled.
• Search snippet offsets: Character chunk offsets persist so search snippets aren't shifted or garbled.
• Async query chunk offsets: /v1/queries chunk row_offset uses the cumulative offset rather than chunk_index * chunk_size.

Dependency Updates

Contributors

• @bjchambers (opens in a new tab)
• @ewgenius (opens in a new tab)
• @Jeadie (opens in a new tab)
• @krinart (opens in a new tab)
• @lukekim (opens in a new tab)
• @melks (opens in a new tab)
• @mvanhorn (opens in a new tab)
• @Oxygen56 (opens in a new tab)
• @peasee (opens in a new tab)
• @phillipleblanc (opens in a new tab)
• @sgrebnov (opens in a new tab)
• @v1gnesh (opens in a new tab)

Breaking Changes

No breaking changes.

Cookbook Updates

No new cookbook recipes.

The Spice Cookbook (opens in a new tab) includes more than 100 recipes to help you get started with Spice quickly and easily.

Upgrading

To upgrade to v2.1.0, use one of the following methods:

CLI:

spice upgrade

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.1.0 image:

docker pull spiceai/spiceai:2.1.0

docker pull spiceai/spiceai:2.1.0

For available tags, see DockerHub (opens in a new tab).

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.1.0

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.1.0

AWS Marketplace:

Spice is available in the AWS Marketplace (opens in a new tab).

What's Changed

Changelog

• Make DuckDB schema cast logic more robust by @sgrebnov (opens in a new tab) in #10991 (opens in a new tab)
• perf(cayenne): reduce allocation overheads in hot paths by @lukekim (opens in a new tab) in #10950 (opens in a new tab)
• improve error message for 'params.spiceai_region' by @Jeadie (opens in a new tab) in #10954 (opens in a new tab)
• fix(acceleration): rename WriteMode variants to fix #10960 by @phillipleblanc (opens in a new tab) in #10974 (opens in a new tab)
• add Scylla, bigquery and turso to throughput benchmarking by @Jeadie (opens in a new tab) in #11006 (opens in a new tab)
• deps(ballista): pull in shuffle-on-object-store correctness fixes (datafusion-ballista PRs #42 + #43) by @phillipleblanc (opens in a new tab) in #10919 (opens in a new tab)
• fix(kafka): seek to sidecar offsets via post_rebalance callback on restart by @ewgenius (opens in a new tab) in #11007 (opens in a new tab)
• Propagate source comments into schema metadata by @lukekim (opens in a new tab) in #10944 (opens in a new tab)
• feat(chbench-driver): better alignment to BenchBase by @sgrebnov (opens in a new tab) in #11012 (opens in a new tab)
• fix: handle EXISTS/NOT EXISTS subqueries in federation analyzer by @sgrebnov (opens in a new tab) in #10996 (opens in a new tab)
• Enable filter pushdown in spicepod defined UDTFs by @Jeadie (opens in a new tab) in #11004 (opens in a new tab)
• Refactor spice dataset configuration command by @Jeadie (opens in a new tab) in #10999 (opens in a new tab)
• fix: ensure HashJoinExec partition counts match when join input is statically empty by @phillipleblanc (opens in a new tab) in #11025 (opens in a new tab)
• feat(chbench): Improve OLTP throughput and reduce PostgreSQL CDC overhead by @sgrebnov (opens in a new tab) in #11018 (opens in a new tab)
• feat(cluster): add distributed query observability metrics by @phillipleblanc (opens in a new tab) in #10990 (opens in a new tab)
• fix: delegate truncate in PolyTableProvider to inner write provider by @claudespice (opens in a new tab) in #11036 (opens in a new tab)
• Remove default runtime features - enable explicitly in spiced by @phillipleblanc (opens in a new tab) in #11037 (opens in a new tab)
• fix: preserve field and schema metadata in Vortex physical schema calculation by @claudespice (opens in a new tab) in #11013 (opens in a new tab)
• Expose metadata descriptions via PostgreSQL UDFs by @lukekim (opens in a new tab) in #11032 (opens in a new tab)
• fix: route Turso WAL checkpoint through native turso connection (fixes #10657) by @claudespice (opens in a new tab) in #11048 (opens in a new tab)
• fix: add missing truncate delegation and guards for wrapper TableProviders by @claudespice (opens in a new tab) in #11014 (opens in a new tab)
• Update DataConnector statuses by @krinart (opens in a new tab) in #11052 (opens in a new tab)
• remove redundant readonly checks by @Jeadie (opens in a new tab) in #10975 (opens in a new tab)
• Fix Unity Catalog connector compatibility with OSS Unity Catalog by @ewgenius (opens in a new tab) in #11026 (opens in a new tab)
• refactor(cdc): reduce CDC sub-batch splits for interleaved upsert/delete workloads by @sgrebnov (opens in a new tab) in #11051 (opens in a new tab)
• feat(cayenne): allow inline writes with pending deletions (deletes/upserts) by @sgrebnov (opens in a new tab) in #11031 (opens in a new tab)
• fix(sql-tool): defer read-only gate to caller's API key role by @phillipleblanc (opens in a new tab) in #11040 (opens in a new tab)
• fix: map Gemini Recitation finish reason to ContentFilter by @claudespice (opens in a new tab) in #11046 (opens in a new tab)
• feat(cayenne): fast-path CDC deletes by extracting PK values from filters by @sgrebnov (opens in a new tab) in #11049 (opens in a new tab)
• fix(cluster): gate scheduler readiness on executor partition loads by @phillipleblanc (opens in a new tab) in #10992 (opens in a new tab)
• fix(snowflake): enforce function deny-list in federation pushdown by @claudespice (opens in a new tab) in #11057 (opens in a new tab)
• perf(cdc): Last-write-wins dedup in group_into_sub_batches to reduce sub-batch splits by @sgrebnov (opens in a new tab) in #11059 (opens in a new tab)
• [Bug] Timing between reconnect and AllocateInitialPartitions leaves connection without flight_sql_client by @Jeadie (opens in a new tab) in #10805 (opens in a new tab)
• Define 'trait QueryEngine' to refactor runtime crate by @Jeadie (opens in a new tab) in #11028 (opens in a new tab)
• fix(snowflake): apply Spice function deny-list in extracted connector crate by @claudespice (opens in a new tab) in #11071 (opens in a new tab)
• perf(cayenne): keep CDC upsert PK keysets resident to avoid per-batch full-table rebuilds by @lukekim (opens in a new tab) in #11074 (opens in a new tab)
• fix(postgres-replication): emit recovery log + reduce reconnect-warn volume by @claudespice (opens in a new tab) in #11084 (opens in a new tab)
• fix metadata on search indexing by @Jeadie (opens in a new tab) in #11080 (opens in a new tab)
• perf(cayenne): scale CDC inline flush caps with memory + storage class by @lukekim (opens in a new tab) in #11087 (opens in a new tab)
• feat(cayenne): merge-on-read position deletes for PK upsert tables + memory-pool accounting by @lukekim (opens in a new tab) in #11085 (opens in a new tab)
• Support tuple-IN composite PK extraction in Cayenne delete fast-path by @sgrebnov (opens in a new tab) in #11093 (opens in a new tab)
• feat(cluster): report per-executor table statistics so distributed JoinSelection can size joins by @phillipleblanc (opens in a new tab) in #11089 (opens in a new tab)
• feat(cluster): NDV-aware executor stats so CDC q18 join swap fires by @phillipleblanc (opens in a new tab) in #11098 (opens in a new tab)
• Improve HTAP replication diagnostics on non-convergence by @sgrebnov (opens in a new tab) in #11100 (opens in a new tab)
• Normalize DataType::Null to Int32 in acceleration schema for duckdb by @krinart (opens in a new tab) in #11062 (opens in a new tab)
• Fix debezium schema evolution support by @ewgenius (opens in a new tab) in #11095 (opens in a new tab)
• feat(cayenne): incremental write-path executor statistics for distributed join sizing by @phillipleblanc (opens in a new tab) in #11104 (opens in a new tab)
• fix(cache): periodic moka maintenance to drain invalidation predicates (#11077) by @phillipleblanc (opens in a new tab) in #11106 (opens in a new tab)
• fix: validate Snowflake account identifiers and auth config by @Jeadie (opens in a new tab) in #11024 (opens in a new tab)
• fix: trace external mcp server tool calls by @ewgenius (opens in a new tab) in #11058 (opens in a new tab)
• Remove unwrap from test code; drop clippy::unwrap_used suppressions by @phillipleblanc (opens in a new tab) in #11108 (opens in a new tab)
• Upgrade to DuckDB 1.5.3 + statically link the VSS (HNSW) extension by @sgrebnov (opens in a new tab) in #11107 (opens in a new tab)
• Fix fetched_at for HTTP connector by @Jeadie (opens in a new tab) in #11116 (opens in a new tab)
• fix(search): propagate LIMIT to base TableScan in VectorScanTableProvider (fixes #8368) by @claudespice (opens in a new tab) in #11124 (opens in a new tab)
• feat(runtime): add spicebench feature to register the Cayenne catalog connector by @phillipleblanc (opens in a new tab) in #11122 (opens in a new tab)
• fix(cayenne): tombstone inline-checkpointed rows on upsert to prevent duplicate PKs by @sgrebnov (opens in a new tab) in #11129 (opens in a new tab)
• Remove possibility of a deadlock in RuntimeStatus by @krinart (opens in a new tab) in #11114 (opens in a new tab)
• Fix Windows build: vendored-vss duckdb-rs + adapt to table-providers mongodb API by @phillipleblanc (opens in a new tab) in #11140 (opens in a new tab)
• localpod: synchronize child refreshes when parent uses in-memory (arrow) accelerator by @phillipleblanc (opens in a new tab) in #11139 (opens in a new tab)
• Add datasets dimension to the query_executions metric by @phillipleblanc (opens in a new tab) in #11138 (opens in a new tab)
• fix(spiceai): keep correlated subqueries out of JOIN ON for Spice Cloud federation by @phillipleblanc (opens in a new tab) in #11143 (opens in a new tab)
• fix(duckdb): normalize timestamp columns to microsecond precision (fixes #10627) by @claudespice (opens in a new tab) in #11145 (opens in a new tab)
• feat(cayenne): sharded parallel Vortex encode with key/time clustering by @lukekim (opens in a new tab) in #11144 (opens in a new tab)
• fix(cluster): prevent DoPut write pipeline self-deadlock under ingest backpressure by @phillipleblanc (opens in a new tab) in #11160 (opens in a new tab)
• feat(chbench): configurable HTAP concurrency, DuckDB query overrides, and OLTP rate control by @sgrebnov (opens in a new tab) in #11162 (opens in a new tab)
• fix(http): preserve non-JSON response rows instead of crashing nested decomposition (fixes #11155) by @claudespice (opens in a new tab) in #11161 (opens in a new tab)
• Use declared schema in DynamoDB/MongoDB/Debezium by @krinart (opens in a new tab) in #11066 (opens in a new tab)
• fix(cluster): prevent partitioned datasets from staying Refreshing by @phillipleblanc (opens in a new tab) in #11157 (opens in a new tab)
• fix(runtime): don't list postgres as a valid accelerator engine when postgres-accel is disabled by @sgrebnov (opens in a new tab) in #11169 (opens in a new tab)
• fix(spark): recover stale or broken Spark Connect sessions on failure by @lukekim (opens in a new tab) in #11171 (opens in a new tab)
• fix(secrets): don't abort secret lookup precedence walk on a failing store by @phillipleblanc (opens in a new tab) in #11175 (opens in a new tab)
• feat(cayenne): bound aggregate write concurrency, conservative defaults, and write/read observability by @lukekim (opens in a new tab) in #11170 (opens in a new tab)
• feat(unity_catalog): support Unity Catalog credential vending for Delta Lake tables by @phillipleblanc (opens in a new tab) in #11180 (opens in a new tab)
• fix(secrets): keep failed secret stores registered so lookups report the init root cause by @phillipleblanc (opens in a new tab) in #11181 (opens in a new tab)
• fix(debezium): sign-extend minimal-width base64 decimals instead of zero-padding by @claudespice (opens in a new tab) in #11184 (opens in a new tab)
• fix(deps): update hickory-resolver to 0.26 (evicts hickory-proto 0.25.x) by @phillipleblanc (opens in a new tab) in #11183 (opens in a new tab)
• refactor(secrets): derive secret store metadata from a single registry table by @phillipleblanc (opens in a new tab) in #11188 (opens in a new tab)
• perf(cayenne): cut CDC replication lag on hot upsert tables by @lukekim (opens in a new tab) in #11191 (opens in a new tab)
• feat(cayenne): async inline-fallback (per-tombstone published flag) + 64c/256GB tuning by @lukekim (opens in a new tab) in #11194 (opens in a new tab)
• Add HTTP connector mTLS support by @lukekim (opens in a new tab) in #11127 (opens in a new tab)
• feat(snowflake): push AT TIME ZONE as CONVERT_TIMEZONE and pin session to UTC by @lukekim (opens in a new tab) in #11190 (opens in a new tab)
• feat(cdc): make cdc_max_coalesce_age_ms a real apply-loop linger by @sgrebnov (opens in a new tab) in #11196 (opens in a new tab)
• feat(cayenne): delta-write encoding levels (cayenne_delta_encoding, default auto) + make compression_strategy=zstd real by @lukekim (opens in a new tab) in #11199 (opens in a new tab)
• Add NSQL context endpoint by @lukekim (opens in a new tab) in #11075 (opens in a new tab)
• fix(federation): respect the Spice function deny-list across all SQL connectors; dialect-aware DuckDB pushdown by @claudespice (opens in a new tab) in #11186 (opens in a new tab)
• fix: surface unknown/applied cayenne_* runtime.params at startup (fixes #10970) by @claudespice (opens in a new tab) in #11133 (opens in a new tab)
• perf(cayenne): plain-fsync ordering tier on the staged-commit hot path by @lukekim (opens in a new tab) in #11198 (opens in a new tab)
• fix(kafka): decode JSON payloads to Arrow directly — fixes lossy Decimal128 + removes double-parse (#11192) by @claudespice (opens in a new tab) in #11207 (opens in a new tab)
• feat(cayenne): self-tuning accelerator — hardware + schema + closed-loop adaptive (auto/adaptive modes) by @lukekim (opens in a new tab) in #11213 (opens in a new tab)
• perf(cayenne): CDC throughput — SF-100 @10K txn/s toward <5s lag + 5K QPH by @lukekim (opens in a new tab) in #11206 (opens in a new tab)
• fix(cluster): distribute accelerated tables wrapped by metadata/index providers by @phillipleblanc (opens in a new tab) in #11226 (opens in a new tab)
• Improve Cayenne adaptive tuning and schema safety by @lukekim (opens in a new tab) in #11237 (opens in a new tab)
• feat: Add cayenne_file_pruning param by @peasee (opens in a new tab) in #11239 (opens in a new tab)
• Debezium connector - handle tombstone messages in kafka topic, with schema evolution enabled by @ewgenius (opens in a new tab) in #11197 (opens in a new tab)
• feat(cayenne): broadcast small-dimension joins to executors by @phillipleblanc (opens in a new tab) in #11245 (opens in a new tab)
• fix: scope request context across the managed query runtime by @phillipleblanc (opens in a new tab) in #11253 (opens in a new tab)
• fix: Strip inference columns from table schema on query by @peasee (opens in a new tab) in #11251 (opens in a new tab)
• fix(flightsql): don't drop un-pushed FilterExec predicates in distributed pushdown rules by @claudespice (opens in a new tab) in #11256 (opens in a new tab)
• feat(postgres): share one replication slot across multiple changes-mode datasets by @phillipleblanc (opens in a new tab) in #11255 (opens in a new tab)
• Upgrade to DataFusion v53.1, Arrow v58.3, Vortex v0.74, and dependencies by @lukekim (opens in a new tab) in #11118 (opens in a new tab)
• feat(connectors): support file_format: vortex everywhere parquet is supported by @lukekim (opens in a new tab) in #11282 (opens in a new tab)
• perf(cayenne): metadata-only publish — drop per-key insert records on upsert commit by @lukekim (opens in a new tab) in #11260 (opens in a new tab)
• fix(kafka): harden fetch_latest_message for multi-partition topics by @ewgenius (opens in a new tab) in #11285 (opens in a new tab)
• perf(cayenne): bound mem-tier checkpoint churn + O(1) per-scan deletion view by @lukekim (opens in a new tab) in #11249 (opens in a new tab)
• fix(udfs): length-prefix digest_many values so column boundaries can't collide (fixes #11272) by @claudespice (opens in a new tab) in #11288 (opens in a new tab)
• fix: restore federation deny-list enforcement regressed by the DataFusion 53 upgrade by @claudespice (opens in a new tab) in #11294 (opens in a new tab)
• fix(postgres): recover unchanged-TOAST columns from the old tuple; classify walsender contention as transient by @phillipleblanc (opens in a new tab) in #11293 (opens in a new tab)
• feat: deepen extended schema inference and wire it into cayenne compaction sharding/sorting by @lukekim (opens in a new tab) in #11284 (opens in a new tab)
• perf(cayenne): in-memory CDC tier follow-ups + write-phase observability by @lukekim (opens in a new tab) in #11278 (opens in a new tab)
• Support per-dataset CDC tunable overrides by @sgrebnov (opens in a new tab) in #11295 (opens in a new tab)
• feat(cayenne): harden adaptive auto-tuner (controller hygiene, mem-tier actuator, delete/burst signals, single opt-in) by @lukekim (opens in a new tab) in #11302 (opens in a new tab)
• fix(cayenne): scan inlined-view capture starvation under sustained CDC (analytical QPH) by @lukekim (opens in a new tab) in #11299 (opens in a new tab)
• fix(vortex): don't row-evaluate hash-join dynamic filters in the scan by @sgrebnov (opens in a new tab) in #11307 (opens in a new tab)
• fix(deps): bump rust-postgres crates (RUSTSEC-2026-0178/0179) by @lukekim (opens in a new tab) in #11313 (opens in a new tab)
• fix: strict validation of Postgres CDC params instead of silent defaults (fixes #11274) by @claudespice (opens in a new tab) in #11304 (opens in a new tab)
• fix(duckdb): always quote on-refresh sort columns so reserved-word names don't break refresh by @claudespice (opens in a new tab) in #11305 (opens in a new tab)
• feat(flightsql): fall back to original connection when endpoint location is unreachable by @melks (opens in a new tab) in #11287 (opens in a new tab)
• perf(cayenne): light-encode transient staged CDC deltas by @lukekim (opens in a new tab) in #11311 (opens in a new tab)
• feat(acceleration): widening-only schema evolution via on_schema_change by @lukekim (opens in a new tab) in #11261 (opens in a new tab)
• deps(vortex): bump pin to spiceai-53 HEAD — intra-file decode split + per-execution kernel cache by @lukekim (opens in a new tab) in #11314 (opens in a new tab)
• feat(github): enhance GitHub component validation and error handling by @lukekim (opens in a new tab) in #11259 (opens in a new tab)
• feat(cayenne): goal-driven adaptive tuning toward operator SLOs (lag, freshness, query latency, QPH) by @lukekim (opens in a new tab) in #11310 (opens in a new tab)
• fix(cayenne): broadcast-join rewrite must bail on ambiguous columns, NULL-equal joins, and residual filters by @claudespice (opens in a new tab) in #11252 (opens in a new tab)
• Add Cayenne maintained aggregates by @lukekim (opens in a new tab) in #11235 (opens in a new tab)
• perf(cayenne): single-hash composite deletion filter via KeyDeletionIndex::get_batch by @phillipleblanc (opens in a new tab) in #11325 (opens in a new tab)
• fix(Vortex): decline only the InList membership conjunct of hash-join dynamic filters by @sgrebnov (opens in a new tab) in #11335 (opens in a new tab)
• fix: scope SQL UDF arg inlining to args-table columns (fixes #11273) by @claudespice (opens in a new tab) in #11337 (opens in a new tab)
• fix(refresh): restore S3 ETag/Version refresh-skip behind provider wrappers by @phillipleblanc (opens in a new tab) in #11339 (opens in a new tab)
• fix(cayenne): ref-count in-flight scans so GC can't delete Vortex files mid-read by @phillipleblanc (opens in a new tab) in #11321 (opens in a new tab)
• fix(runtime): retry object-store dataset load when source files are not yet available by @phillipleblanc (opens in a new tab) in #11342 (opens in a new tab)
• feat(s3): default to path-style for dotted bucket names on standard AWS by @phillipleblanc (opens in a new tab) in #11347 (opens in a new tab)
• fix(runtime): resolve accelerated table through metadata-enrichment wrapper by @phillipleblanc (opens in a new tab) in #11345 (opens in a new tab)
• fix: detect dual-write accelerated tables behind the metadata-enrichment wrapper by @claudespice (opens in a new tab) in #11351 (opens in a new tab)
• feat(cayenne): incremental seq-prefix bake — shrink the merge-on-read deletion index by @lukekim (opens in a new tab) in #11326 (opens in a new tab)
• fix(adbc): prevent Spice-specific UDFs from being pushed down to ADBC sources by @krinart (opens in a new tab) in #11297 (opens in a new tab)
• fix: Query Redshift schema details from svv_redshift tables by @peasee (opens in a new tab) in #11362 (opens in a new tab)
• perf(cayenne): tune Turso connection PRAGMAs + jitter metastore retries by @lukekim (opens in a new tab) in #11359 (opens in a new tab)
• Upgrade to DataFusion 54 by @sgrebnov (opens in a new tab) in #11360 (opens in a new tab)
• feat(runtime): dedicated CDC-apply tokio runtime + per-runtime tokio metrics by @lukekim (opens in a new tab) in #11370 (opens in a new tab)
• fix(cayenne): spill oversized hash joins via sort-merge to avoid OOM by @lukekim (opens in a new tab) in #11371 (opens in a new tab)
• fix(cayenne): honor cayenne_metastore: turso for partitioned tables and the dataset checkpoint by @phillipleblanc (opens in a new tab) in #11365 (opens in a new tab)
• perf(cayenne): in-RAM scan parallelism, query admission control, skip no-op deletion encode, sound scan output_ordering by @lukekim (opens in a new tab) in #11332 (opens in a new tab)
• fix(catalog): restore DDL after DataFusion 54 broke transparent catalog-provider downcasts by @phillipleblanc (opens in a new tab) in #11375 (opens in a new tab)
• feat(flightsql): infer schema via SELECT * LIMIT 1 when GetTables is unimplemented by @melks (opens in a new tab) in #11286 (opens in a new tab)
• fix(cayenne): Utf8View read schema avoids hash-join offset overflow by @lukekim (opens in a new tab) in #11379 (opens in a new tab)
• feat(cluster): support distributed (Ballista) scans of Iceberg tables by @phillipleblanc (opens in a new tab) in #11378 (opens in a new tab)
• feat(optimizer): cost-based left-deep join reordering for Cayenne acceleration by @sgrebnov (opens in a new tab) in #11377 (opens in a new tab)
• cli - fix service-account auth in spice cloud * commands by @ewgenius (opens in a new tab) in #11316 (opens in a new tab)
• fix: Support external Redshift table schema inference and Hive external type parsing by @peasee (opens in a new tab) in #11399 (opens in a new tab)
• feat(llms): native GLM support — opt-in flash-attn + surface reasoning_content by @lukekim (opens in a new tab) in #11400 (opens in a new tab)
• feat(s3_vectors): paginate QueryVectors for topK up to 10,000 by @bjchambers (opens in a new tab) in #11405 (opens in a new tab)
• fix: surface .env parse errors with line numbers instead of silently skipping by @Oxygen56 (opens in a new tab) in #11306 (opens in a new tab)
• fix(status): probe metrics endpoint over https when TLS is enabled by @phillipleblanc (opens in a new tab) in #11393 (opens in a new tab)
• fix(mcp): record task_history spans for tool calls proxied through /v1/mcp by @phillipleblanc (opens in a new tab) in #11397 (opens in a new tab)
• Properly handle date_trunc in BigQueryDialect by @krinart (opens in a new tab) in #11416 (opens in a new tab)
• fix(snowflake): honor column scale in Int64 timestamp arm and cast TIME by @claudespice (opens in a new tab) in #11418 (opens in a new tab)
• fix: /v1/queries chunk row_offset uses cumulative offset, not chunk_index * chunk_size (fixes #11271) by @claudespice (opens in a new tab) in #11398 (opens in a new tab)
• feat(cayenne): incremental retraction for maintained aggregates + anchor bench by @lukekim (opens in a new tab) in #11389 (opens in a new tab)
• feat(cluster): support distributed (Ballista) scans of Iceberg catalog tables by @phillipleblanc (opens in a new tab) in #11419 (opens in a new tab)
• Simplify chat/responses models by @Jeadie (opens in a new tab) in #10997 (opens in a new tab)
• feat(llms): distributed tensor-parallel GLM inference via mistral.rs ring backend by @lukekim (opens in a new tab) in #11406 (opens in a new tab)
• Optimize Flight streaming for large result sets by @lukekim (opens in a new tab) in #11420 (opens in a new tab)
• fix(deps): evict rustls 0.21 / rustls-webpki 0.101.7 (GHSA-82j2-j2ch-gfr8) by @phillipleblanc (opens in a new tab) in #11428 (opens in a new tab)
• feat(cayenne): in-memory CDC intra-apply sharding (PK-hash shards) by @lukekim (opens in a new tab) in #11421 (opens in a new tab)
• fix(cayenne): shard CDC upserts with pending deletions so the N>1 slot-ack advances by @lukekim (opens in a new tab) in #11445 (opens in a new tab)
• fix(cluster): keep built-in avg over Spark avg (distributed aggregate state schema mismatch) by @phillipleblanc (opens in a new tab) in #11434 (opens in a new tab)
• feat(views): support params.file_format for embedding chunking by @Jeadie (opens in a new tab) in #11424 (opens in a new tab)
• fix: offload blocking sync calls off the primary async runtime by @phillipleblanc (opens in a new tab) in #11435 (opens in a new tab)
• fix(udfs): rebind dot_product alias to Spice's inner_product on DataFusion 54 by @lukekim (opens in a new tab) in #11443 (opens in a new tab)
• feat(secrets): add full-fidelity reference iteration and a resolution-status API by @phillipleblanc (opens in a new tab) in #11195 (opens in a new tab)
• fix: Deny unsupported array functions for Postgres pushdown by @peasee (opens in a new tab) in #11450 (opens in a new tab)
• fix(cli): spice query honors --http-endpoint instead of failing on a Flight connect by @phillipleblanc (opens in a new tab) in #11452 (opens in a new tab)
• fix(cayenne): coordinate query-pool + in-memory CDC tier budgets to prevent adaptive OOM by @lukekim (opens in a new tab) in #11449 (opens in a new tab)
• Surface mTLS config in Kafka data connector by @v1gnesh (opens in a new tab) in #11372 (opens in a new tab)
• feat(secrets): check and report secret references at startup by @phillipleblanc (opens in a new tab) in #11457 (opens in a new tab)
• Default cayenne_force_view_types to false by @sgrebnov (opens in a new tab) in #11459 (opens in a new tab)
• fix: resolve table-reference qualification in results-cache invalidation (fixes #11266) by @claudespice (opens in a new tab) in #11460 (opens in a new tab)
• feat(cayenne): metadata aggregate pushdown — fold whole-table SUM/AVG/COUNT/MIN/MAX from statistics by @bjchambers (opens in a new tab) in #11414 (opens in a new tab)
• fix(search): restore numeric trunc and fix SortPreservingMergeExec planning error (DF54) by @Jeadie (opens in a new tab) in #11415 (opens in a new tab)
• fix(cluster): route Ballista shuffle/temp to the data PVC by @phillipleblanc (opens in a new tab) in #11454 (opens in a new tab)
• fix(search): default Elasticsearch kNN candidate pool to 1000 instead of 10 (fixes #11264) by @claudespice (opens in a new tab) in #11467 (opens in a new tab)
• feat(acceleration): add on_schema_change drop_and_recreate policy by @lukekim (opens in a new tab) in #11462 (opens in a new tab)
• feat(cayenne): predicate-aware maintained aggregates serve filtered analytical queries from the CDC delta by @lukekim (opens in a new tab) in #11458 (opens in a new tab)
• feat(cluster): shared Ballista job state with scheduler failover by @phillipleblanc (opens in a new tab) in #11436 (opens in a new tab)
• feat(cayenne): extend HLL NDV sketching to string and date columns by @bjchambers (opens in a new tab) in #11468 (opens in a new tab)
• fix(search): persist character chunk offsets so search snippets aren't shifted/garbled (fixes #11269) by @claudespice (opens in a new tab) in #11479 (opens in a new tab)
• feat(cayenne): storage-aware adaptive CDC tuning — calibration probe, IMDS, I/O-cliff fast path, infeasible-SLO feedback by @lukekim (opens in a new tab) in #11463 (opens in a new tab)
• fix(cayenne): LIMIT N under-delivers on key-deletion tables by @lukekim (opens in a new tab) in #11490 (opens in a new tab)
• fix(cayenne): live/tier-accurate join build-side stats (merge-on-read deletes + never-shrink NDV) by @lukekim (opens in a new tab) in #11496 (opens in a new tab)
• perf(cayenne): kernel-space I/O hygiene — compaction fadvise + staged-commit barrier reduction by @lukekim (opens in a new tab) in #11495 (opens in a new tab)
• feat(cayenne): feed maintained-aggregate IVM from the staged-disk CDC path by @lukekim (opens in a new tab) in #11491 (opens in a new tab)
• feat(cayenne): global adaptive-tuning SLOs with per-dataset overrides; QPH global-only by @lukekim (opens in a new tab) in #11497 (opens in a new tab)
• Update search snapshots by @sgrebnov (opens in a new tab) in #11473 (opens in a new tab)
• fix: Support reading column types longer than 128 chars in Redshift by @peasee (opens in a new tab) in #11500 (opens in a new tab)
• fix(cluster): distributed (Ballista) query-execution config + scheduled SF10 bench by @phillipleblanc (opens in a new tab) in #11478 (opens in a new tab)
• fix(http): retry transient response-body read failures; de-flake backoff test by @claudespice (opens in a new tab) in #11482 (opens in a new tab)
• Re-land orphaned deletion-vector cleanup during retention deletes by @lukekim (opens in a new tab) in #11501 (opens in a new tab)
• fix(cayenne): re-upsert over a pending delete tombstone records an insert-record (overwrite resurrection) by @bjchambers (opens in a new tab) in #11469 (opens in a new tab)
• fix: Flight DoPut silently dropped client batches on early sink completion by @claudespice (opens in a new tab) in #11507 (opens in a new tab)
• Upgrade OpenTelemetry to 0.32 and reqwest to 0.13 by @phillipleblanc (opens in a new tab) in #11506 (opens in a new tab)
• fix(queries): run async /v1/queries jobs under the submitting request context by @phillipleblanc (opens in a new tab) in #11505 (opens in a new tab)
• fix(cayenne): restore append-only current-snapshot compaction by @Jeadie (opens in a new tab) in #11439 (opens in a new tab)
• perf(cayenne): orphaned deletion-vector cleanup off the write path, behind a knob by @bjchambers (opens in a new tab) in #11517 (opens in a new tab)
• fix(datafusion): accurate projected scan byte size so hash joins build the smaller side by @sgrebnov (opens in a new tab) in #11503 (opens in a new tab)
• fix(cayenne): user-visible DELETE WHERE pk IN (...) reports the real row count by @lukekim (opens in a new tab) in #11514 (opens in a new tab)
• fix(cayenne): seed persisted num_rows for hash-join sizing by @sgrebnov (opens in a new tab) in #11515 (opens in a new tab)
• fix(cayenne): correctness & memory_limit fixes from perf audit (2 P0, 3 P1) by @lukekim (opens in a new tab) in #11516 (opens in a new tab)
• feat(cayenne): wire orphaned-DV cleanup knob to spicepod params + doc sync by @bjchambers (opens in a new tab) in #11523 (opens in a new tab)
• Fix s3 vectors API by @krinart (opens in a new tab) in #11536 (opens in a new tab)
• fix: Placeholder table initialization lock swap by @peasee (opens in a new tab) in #11540 (opens in a new tab)
• chore(cluster): bump ballista pin for the null-aware anti-join fix by @phillipleblanc (opens in a new tab) in #11544 (opens in a new tab)
• fix(runtime-tools): fix memory table identifier validation rejecting valid names by @Jeadie (opens in a new tab) in #11546 (opens in a new tab)
• Bump datafusion to include spiceai/datafusion#181 by @Jeadie (opens in a new tab) in #11563 (opens in a new tab)
• Update deny.toml by @krinart (opens in a new tab) in #11571 (opens in a new tab)
• Bump datafusion (spiceai/datafusion#182) and datafusion-table-providers (#27): fix q16 CollectLeft planning error and SQLite q6 wrong revenue by @Jeadie (opens in a new tab) in #11598 (opens in a new tab)

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.0.0...v2.1.0 (opens in a new tab)