date: 2026-10-27 title: 'Spice v2.4.0 (Oct 27, 2026)' type: blog authors: [jeadie] tags: [release, cayenne, datafusion, ai, s3, caching, snapshots, iceberg, huggingface]

Spice v2.4.0 is a feature release that expands data ingestion, caching, and AI decisions, and makes Spice Cayenne the default accelerator. It also upgrades the query engine to Apache DataFusion v55 and Apache Arrow v59.3. Datasets and views that do not set acceleration.engine now use Cayenne instead of Arrow, except on native Windows. Other accelerators are deprecated and will be removed in v2.5. In tests, Cayenne served uncached primary-key lookups about twice as fast, and query latency stayed steady during full refreshes.

SQL queries and the new OpenAI-compatible /v1/decisions API can now ask TypeSafe or chat models for decisions. With refresh_mode: changes, accelerated S3 datasets read new objects from S3 when SQS delivers S3 event notifications. This release also adds SQL results-cache warmup, Enterprise acceleration snapshots, and new data connectors and formats.

Read Breaking Changes (opens in a new tab) before you upgrade. SQL results-cache entries now expire only by TTL, not when a dataset refreshes. Cayenne ignores on_conflict and keeps one row for each primary key. Some datasets that loaded on Arrow fail to load on Cayenne unless they set engine: arrow.

Highlights in v2.4.0 include:

• Spice Cayenne by Default (opens in a new tab): Datasets without acceleration.engine now use Cayenne instead of Arrow, except on native Windows. Cayenne adds automatic primary-key handling, persistent secondary indexes, and adaptive tuning (preview).
• DataFusion v55 (opens in a new tab): the query engine upgrades to DataFusion v55 and Arrow v59.3. In DataFusion benchmarks, the TopK query suite ran about 43% faster. Ballista v55, Vortex v0.86, and Iceberg v0.11 also ship in this release.
• Faster Cayenne Lookups and Refreshes (opens in a new tab): in tests, uncached Cayenne primary-key lookups ran about twice as fast. P99 query latency during full refreshes fell from 130–690 ms to 44–50 ms.
• AI Decisions in SQL and HTTP (opens in a new tab): five SQL decision functions and an OpenAI-compatible POST /v1/decisions API, with TypeSafe models or any chat model.
• S3 Event-Driven Refresh (opens in a new tab): with refresh_mode: changes, accelerated S3 datasets read new objects when SQS delivers S3 event notifications. A periodic listing loads objects whose notification did not arrive.
• Acceleration Snapshots (opens in a new tab) (Enterprise): serve a dataset from published snapshots without its source connector. Snapshots use the shared state location and expire automatically.
• SQL Results-Cache Warmup (opens in a new tab): record query shapes, then replay them after a restart so that the first queries use a warm cache.
• New Data Connectors and Formats (opens in a new tab): query Hugging Face datasets, ORC files, and Iceberg tables without a catalog.

What's New in v2.4.0

Spice Cayenne by Default

Spice Cayenne (opens in a new tab) now accelerates datasets and views that omit acceleration.engine. Before this release, they used Arrow. Native Windows keeps Arrow. On native Windows, run Spice in Docker or WSL to use Cayenne. Storage still defaults to mode: memory. Set mode: file for persistent acceleration. To move an existing acceleration, read the Cayenne migration guide (opens in a new tab).

Automatic ON CONFLICT handling for Primary Keys

A Cayenne dataset with a primary_key now keeps one row for each key in every refresh mode, without an on_conflict setting. This table shows which version of a repeated key Cayenne keeps:

With a time_column, a late row with an older time does not replace a newer stored row. Sources such as Iceberg, where writers can commit out of time order, keep the newest version of each key.

datasets:
  - from: s3://my-bucket/orders/
    name: orders
    time_column: updated_at
    params:
      file_format: parquet
    acceleration:
      enabled: true
      mode: file
      primary_key: id

datasets:
  - from: s3://my-bucket/orders/
    name: orders
    time_column: updated_at
    params:
      file_format: parquet
    acceleration:
      enabled: true
      mode: file
      primary_key: id

Cayenne no longer uses on_conflict to select its conflict policy. For read-write datasets whose writes must stay in the acceleration, set access: read_write and acceleration.write_mode: acceleration. The source need not support writes. This setting cannot accompany a change-stream refresh (refresh_mode: changes, or a CDC source whose default refresh mode is changes).

An UPDATE that creates a primary-key collision fails before it changes rows. See duplicate primary keys (opens in a new tab) and the migration guidance (opens in a new tab).

Runtime-Wide Adaptive Tuning (Preview)

Cayenne can now change its own settings to meet runtime-wide targets for freshness, replication lag, query latency, and throughput. Set runtime.adaptive_tuning (opens in a new tab) to enable it for Cayenne datasets and catalogs. Adaptive tuning is in preview.

runtime:
  adaptive_tuning: enabled
  params:
    target_freshness: 30s

datasets:
  - from: postgres:public.orders
    name: orders
    acceleration:
      enabled: true
      mode: file
      refresh_mode: changes
      primary_key: id

runtime:
  adaptive_tuning: enabled
  params:
    target_freshness: 30s

datasets:
  - from: postgres:public.orders
    name: orders
    acceleration:
      enabled: true
      mode: file
      refresh_mode: changes
      primary_key: id

Adaptive tuning accepts four target objectives and one pacing setting in runtime.params. Cayenne changes its settings in small steps to move each measured value toward its target:

• target_replication_lag: Desired maximum age of the newest applied source commit. Applies only to change-stream (refresh_mode: changes) datasets.
• target_freshness: Desired maximum delay from a source commit until its data becomes queryable. Applies only to change-stream datasets.
• target_query_latency: Desired P99 latency of the queries that read each Cayenne dataset.
• target_qph: Desired throughput, in queries per hour (QPH), of queries that read Cayenne datasets.
• target_convergence_window: Time budget for tuning to approach its targets. Default: 60s.

Adaptive tuning defaults to disabled. Targets apply only when adaptive tuning is enabled. A refresh_mode: full dataset does not use adaptive tuning. Each refresh replaces the whole table, so Cayenne turns off background compaction (cayenne_compaction_background_interval_ms: 0). The tuner runs on the compaction task, so it does not run for that dataset. The runtime logs a warning for each such dataset. See Cayenne self-tuning (opens in a new tab) and Cayenne Tuning Settings (opens in a new tab).

Indexes, Initial Loads, and Compaction

• Persistent secondary indexes (opens in a new tab): File-mode Cayenne stores index runs with its acceleration. Writes maintain the indexes, and acceleration snapshots retain them. Dynamic join filters can probe secondary indexes.
• Keyed initial loads: Cayenne can build the primary-key filter during the first load within its memory budget. Otherwise, it retains the existing warmup path. Bulk loads avoid per-row routing and a subsequent whole-table fold.
• Maintained aggregates: Views share a compact index of per-key contributions. Rebuilds capture concurrent writes and apply them after the scan. A bounded query pool assigns 10% to aggregates without the former 512 MiB cap. An unbounded pool retains that cap.
• Append-only compaction: For unkeyed datasets without position deletes, Cayenne encodes merged files outside the write lock. Publication still takes the lock. Datasets with position deletes retain the full-rewrite path.
• Configuration updates: A restart applies changed write settings to an existing acceleration. Identity changes can rebuild eligible full-refresh accelerations from their source. See the Cayenne documentation (opens in a new tab).

Apache DataFusion v55 and Query Engine Upgrades

Spice now runs on Apache DataFusion v55 and Apache Arrow v59.3. This release also upgrades Apache Ballista to v55, Vortex to v0.86.1, and Apache Iceberg to v0.11.0. The numbers in this section come from each project's own benchmarks, not from Spice tests.

Apache DataFusion v55

The DataFusion v55 release (opens in a new tab) adds these changes:

• TopK pruning: As the threshold of an ORDER BY ... LIMIT query tightens, Parquet scans check the unread row groups again. They skip groups that cannot change the result. Compound ORDER BY keys are also supported. The topk_tpch suite ran about 43% faster in total.
• Join planning: The optimizer converts eligible inner joins to semi joins. It removes redundant sides of LEFT and RIGHT joins and orders filter predicates by estimated cost. ClickBench Q21 ran 10–13% faster.
• Planning speed: The optimizer collapses chained projections and rewrites subqueries in place. One planning case fell from 623 ms to 155 ms, and TPC-DS optimization fell from 220 ms to 170 ms.
• Functions: IN lists with few primitive values evaluate up to 14.7× faster in microbenchmarks. approx_distinct over 50,000 groups fell from 1,723 ms to 17 ms. More string functions keep dictionary encoding.
• Aggregation: Multi-column GROUP BY keeps column-oriented storage for all key types, such as fixed-size binary UUIDs. Grouping 200 million UUIDs ran about 5% faster and used less memory.
• Parquet scans: Scans skip nested fields that the table schema does not declare. They also skip the page index when a file has none. ClickBench Q1 on a single file ran 38% faster.
• Spills: A sort under memory pressure spills its largest stream again in smaller batches instead of failing. Merges also limit their number of input streams.
• SQL: EXPLAIN accepts PostgreSQL-style options and FORMAT pgjson. New array functions add, subtract, scale, sum, and average array elements.

Spice carries these changes through its query plans and keeps statistics across its plan wrappers. Cayenne does not split Vortex scans below 10 MiB, to avoid repeated footer reads.

Apache Ballista v55

Spice.ai Enterprise (opens in a new tab) feature. See Distributed Query (opens in a new tab).

Spice distributed query runs on a Spice fork of Apache Ballista v55. The Ballista v55 release (opens in a new tab) adds these upstream changes:

• Adaptive query execution by default: The scheduler replans between stages with row counts from completed stages. All 99 TPC-DS queries complete and match single-process DataFusion in the Ballista test suite.
• Fewer tasks: One task can cover several input partitions, up to the available virtual cores of its executor. This change reduces planning, serialization, and dispatch work.
• Shuffle: Remote shuffle reads receive blocks without a copy. Sort shuffle writes one file for each task.
• Protocol version: Clients receive a clear error when their major version differs from the version of the scheduler.
• Benchmark: On TPC-H at scale factor 1,000, Ballista ran the 22 queries in 502.4 seconds. Apache Spark 4.1.3 took 898.2 seconds on the same cluster.

Schedulers and executors must run the same version, so upgrade all cluster components together.

Vortex v0.79.0 to v0.86.1

Spice v2.3.3 used the Vortex v0.79.0 fork. This upgrade covers every upstream release from v0.79.0 through v0.86.1:

• Types and arithmetic: Vortex adds native Map arrays, union arrays, and decimal addition, subtraction, multiplication, and division.
• Row selection: Piecewise-sequence indices represent contiguous selections without an index entry for every row.
• Scans and expressions: Layout scans get a physical plan and expression optimization. Filters can pass through scalar functions with more than one argument. Filters on wide lists read only the selected range of child elements.
• Compression: Binary arrays support FSST compression with variable-length offsets. OnPair becomes a stable encoding with a dictionary in storage.
• File metadata: Files can store custom metadata. Readers cache decoded type descriptions.
• Correctness and validation: Vortex corrects NULL handling for dictionary predicates, BETWEEN bounds, and empty arrays. Readers validate footer offsets, compression metadata, and array indices.

See the Vortex release notes (opens in a new tab) for v0.80.0 through v0.86.0 (opens in a new tab) and v0.86.1 (opens in a new tab).

Apache Iceberg v0.11.0

Spice updates its Iceberg reader and catalog integrations from v0.10.1 to the v0.11.0 fork for DataFusion v55 and Arrow v59. The upstream v0.11 changes (opens in a new tab) extend reads and catalog compatibility:

• Iceberg v3 reads: The reader applies deletion vectors from Puffin files. It carries row identifiers and sequence numbers through scans.
• Metadata-only scans: Projections of metadata only do not read data columns. Positional-delete processing buffers runs instead of one key for each row.
• REST catalogs: Clients negotiate the endpoints that the server advertises. They use OAuth2 authentication for each session.

The upgrade also aligns Iceberg storage with OpenDAL v0.58.

Performance

Spice Cayenne serves lookups, loads, and refreshes faster in this release. Each number below comes from the test in the linked pull request. The tests ran on different machines, so do not compare numbers between them.

Spice Cayenne

• Steady query latency during refreshes: Full and append refreshes now write on dedicated refresh threads by default. Before, a large refresh write ran on the threads that serve queries. In a test, a client sent 500 point-lookup joins per second to a Cayenne table of 2.2 million rows and about 8 GB. P99 latency during each full-refresh commit fell from 130–690 ms to the steady 44–50 ms. With runtime.params.dedicated_thread_pool: disabled, refresh writes run as before. See #15262 (opens in a new tab).
• Primary-key point lookups: Cayenne serves uncached primary-key point lookups about twice as fast. On a file-mode table of 10 million rows, P50 latency improved 2.11–2.41× at 1 to 128 clients. See #14433 (opens in a new tab).
• Lookups after the first load: Uncached primary-key lookups right after the first full refresh are faster. On a table of 10 million rows, P50 latency at one client fell from 6.0 ms to 1.1 ms. At 16 clients, P99 latency fell from 16.3 ms to 3.9 ms. See #14314 (opens in a new tab).
• First refresh after a load: Cayenne builds the primary-key filter during the first load, so the first append refresh does not rebuild it. On a keyed table of 40.2 million rows, the first append refresh fell from a median of 4.2 s to 0.11 s. See #15099 (opens in a new tab).
• Keyed bulk loads: A first load into an empty keyed table no longer routes each row separately. On an Iceberg table of 40.2 million rows, the slowest first load fell from 21.0 s to 13.5 s. Iceberg scans also decode at the session batch size instead of 1,024 rows. See #15076 (opens in a new tab) and #15072 (opens in a new tab).
• Append visibility on clustered tables: Cayenne orders each scan partition of a whole-table rewrite separately, then merges them. On an append table of 20 million rows with cayenne_cluster_by, P99 time from append to query visibility fell from 17.2 s to 3.0 s. See #14488 (opens in a new tab).
• Append-only ingestion: Small-file compaction no longer stalls writers on tables without a primary key or position deletes. In an OTLP log-ingestion test, export P99 latency fell from about 1.2 s to 0.12 s. Throughput rose from about 210,000 to 775,000 records per second. See #15227 (opens in a new tab).
• Metastore writes: Cayenne batches manifest rewrites in its SQLite metastore and runs every write on one writer connection. At 128 clients, P99 latency of a 1,000-file manifest rewrite fell from 5.39 s to 374 ms. See #14369 (opens in a new tab).
• Filtered key deletes: A filtered delete on a keyed table now reads only the key and filter columns. In a test, P99 latency of a key-only delete fell from 27.7 ms to 8.9 ms. See #14374 (opens in a new tab).

Caching

• Caching mode under sustained misses: Cayenne datasets with refresh_mode: caching now batch cache replacements. In a test with 500 lookups per second over 2.5 million random keys, request P99 latency fell from 363 ms to 9 ms. Dropped cache fills fell from 122,508 to 0. See #15253 (opens in a new tab).
• Caching mode without a primary key: Lookups stay fast when the cache replaces entries continuously on a dataset without a primary key. In a two-core test at 300 lookups per second, P99 latency fell from 1,377 ms to 9.3 ms. See #15362 (opens in a new tab).
• SQL results-cache expiry: Expiry work now grows with the number of expired entries, not with the size of the cache. With 3 million entries, insert P99.9 latency fell from 10.8–15.7 ms to 70–84 µs. See #14919 (opens in a new tab).

Object-Store Listings

• Metadata-only queries: GROUP BY, DISTINCT, MIN, and MAX over partition or metadata columns no longer scan whole files. Spice answers them from the file listing, or from the first record of each file. count, sum, and avg still scan the files. See #14116 (opens in a new tab).
• Listing pruning: Filters on _last_modified and other metadata columns now prune the file listing, so Spice does not open the excluded objects. Timestamp bounds joined by AND prune the listing. Filters with OR or NOT read the full listing. See #14265 (opens in a new tab) and #14303 (opens in a new tab).

AI Decisions in SQL and HTTP

SQL queries and HTTP clients can now ask a model for yes-or-no answers, probabilities, labels, and scores. This release adds five SQL decision functions (opens in a new tab) and an OpenAI-compatible POST /v1/decisions (opens in a new tab) API. TypeSafe (opens in a new tab) decision models return calibrated probabilities. Any chat model can also answer decisions, but its probabilities are uncalibrated estimates.

Add a TypeSafe model to the Spicepod:

models:
  - from: typesafe:jev
    name: support_model
    params:
      typesafe_api_key: ${secrets:TYPESAFE_API_KEY}

models:
  - from: typesafe:jev
    name: support_model
    params:
      typesafe_api_key: ${secrets:TYPESAFE_API_KEY}

Then call a decision function in SQL. Each function selects its model with the model argument:

SELECT
    id,
    ai_classify(body, ['billing', 'technical', 'account'],
                model => 'support_model') AS team,
    ai_probability(body, 'The customer requests a refund',
                   model => 'support_model') AS refund_probability
FROM tickets
WHERE status = 'open';

SELECT
    id,
    ai_classify(body, ['billing', 'technical', 'account'],
                model => 'support_model') AS team,
    ai_probability(body, 'The customer requests a refund',
                   model => 'support_model') AS refund_probability
FROM tickets
WHERE status = 'open';

See AI decisions (opens in a new tab).

S3 Event-Driven Refresh

With refresh_mode: changes, accelerated S3 (opens in a new tab) datasets read new objects from S3 when SQS delivers S3 event notifications. Set refresh_mode: changes and s3_changes_queue_url together. The queue must serve only this dataset, and from must be a prefix, not a single object or a glob:

datasets:
  - from: s3://my-bucket/events/
    name: events
    params:
      file_format: parquet
      s3_region: us-east-1
      s3_changes_queue_url: ${secrets:events_queue_url}
    acceleration:
      enabled: true
      mode: file
      refresh_mode: changes

datasets:
  - from: s3://my-bucket/events/
    name: events
    params:
      file_format: parquet
      s3_region: us-east-1
      s3_changes_queue_url: ${secrets:events_queue_url}
    acceleration:
      enabled: true
      mode: file
      refresh_mode: changes

A periodic listing of the prefix (s3_changes_backfill_interval, default 1h) loads objects whose notification did not arrive. Spice loads each object key one time. By default, rows from deleted objects stay in the acceleration (s3_on_object_removed: ignore).

Acceleration Snapshots

Spice.ai Enterprise (opens in a new tab) feature. See Acceleration Snapshots (opens in a new tab).

A dataset can now serve published acceleration snapshots without its original source connector. Set file_format: snapshot, and set from to the snapshot location:

datasets:
  - from: s3://my-bucket/spice/snapshots/
    name: orders
    params:
      file_format: snapshot
      s3_region: us-east-1

datasets:
  - from: s3://my-bucket/spice/snapshots/
    name: orders
    params:
      file_format: snapshot
      s3_region: us-east-1

Snapshots now expire after 28 days, except for the current and newest snapshot of each dataset. A reader that cannot load a newer snapshot keeps serving its loaded snapshot and tries again on the next poll. Each failure increments dataset_acceleration_refresh_errors{mode="snapshot", reason="snapshot_download_failed"}. OSS builds refuse snapshot datasets and do not publish snapshots. See Serve a dataset from published snapshots (opens in a new tab) and Shared State and Snapshot Migration (opens in a new tab).

One Location for Shared Runtime State

runtime.state (opens in a new tab) sets one object-store location for the state that the runtime keeps across restarts and instances. Each of these features stores its state in its own subdirectory:

• Cluster state
• SQL results-cache warmup
• Shared HTTP rate control (Spice.ai Enterprise)
• Acceleration snapshots (Spice.ai Enterprise)

runtime:
  state:
    location: s3://my-bucket/spice/
    params:
      s3_region: us-east-1

datasets:
  - from: postgres:orders
    name: orders
    acceleration:
      enabled: true
      mode: file
      snapshots: enabled

runtime:
  state:
    location: s3://my-bucket/spice/
    params:
      s3_region: us-east-1

datasets:
  - from: postgres:orders
    name: orders
    acceleration:
      enabled: true
      mode: file
      snapshots: enabled

Earlier per-feature locations, such as runtime.scheduler.state_location, still work but are deprecated. See Shared State and Snapshot Migration (opens in a new tab).

SQL Results Cache and HTTP Caching

SQL Results-Cache Warmup

After a restart, the first queries can now read from a warm SQL results cache (opens in a new tab). The cache records query shapes and stores them. When Spice starts again, it replays the stored shapes after the first full or append refresh of each accelerated dataset:

runtime:
  caching:
    sql_results:
      enabled: true
      warmup: on_first_refresh

runtime:
  caching:
    sql_results:
      enabled: true
      warmup: on_first_refresh

Spice records the first 10 distinct shapes of cacheable, unauthenticated queries. Queries that differ only in equality-filter values share one shape. During replay, warmup reads up to 1,024 distinct combinations of filter-column values for each shape. It reruns the shape with each combination until the cache is full. Datasets do not report ready until warmup completes. The runtime does not start when a Spicepod sets both warmup and cache_key_type: sql.

Cache Backend

The SQL results, search, and embedding caches now use the Spice sharded cache. The runtime accepts engine: moka and engine: pingora for compatibility, but these values no longer select a backend.

Caching Refresh Mode

These changes apply to datasets with refresh_mode: caching (opens in a new tab), such as HTTP datasets:

• Shared cache-miss fetches: Concurrent cache misses for the same key now share one origin fetch.
• stale-if-error: Cached HTTP responses now follow the RFC 5861 stale-if-error rules.

New Data Connectors and Formats

Hugging Face Datasets

The Hugging Face data connector (opens in a new tab) queries and accelerates Parquet, CSV, TSV, JSON, and ORC files from the Hub:

datasets:
  - from: hf://datasets/stanfordnlp/imdb/plain_text/
    name: imdb
    acceleration:
      enabled: true

datasets:
  - from: hf://datasets/stanfordnlp/imdb/plain_text/
    name: imdb
    acceleration:
      enabled: true

ORC and Object Metadata

Listing connectors now support file_format: orc (opens in a new tab). Enable metadata columns (opens in a new tab) to query the location and modification time of each row's source object:

datasets:
  - from: s3://my-bucket/events/
    name: events
    params:
      file_format: orc
      s3_region: us-east-1
    metadata:
      _location: enabled
      _last_modified: enabled

datasets:
  - from: s3://my-bucket/events/
    name: events
    params:
      file_format: orc
      s3_region: us-east-1
    metadata:
      _location: enabled
      _last_modified: enabled

For example, query objects modified since October 1:

SELECT DISTINCT _location, _last_modified
FROM events
WHERE _last_modified >= TIMESTAMP '2026-10-01 00:00:00';

SELECT DISTINCT _location, _last_modified
FROM events
WHERE _last_modified >= TIMESTAMP '2026-10-01 00:00:00';

Apache Iceberg Without a Catalog

Apache Iceberg (opens in a new tab) datasets can read a table directly from its root directory on local or object storage. Spice finds the current metadata file on every query and every refresh, so it reads new commits without a restart:

datasets:
  - from: iceberg:s3://my-bucket/warehouse/db/taxis
    name: taxis
    params:
      iceberg_s3_region: us-east-1

datasets:
  - from: iceberg:s3://my-bucket/warehouse/db/taxis
    name: taxis
    params:
      iceberg_s3_region: us-east-1

To read one fixed table version, set iceberg_metadata_path to a metadata file. A pinned dataset is read-only, so it fails to load with access: read_write:

datasets:
  - from: iceberg:s3://my-bucket/warehouse/db/taxis
    name: taxis_v3
    params:
      iceberg_s3_region: us-east-1
      iceberg_metadata_path: /metadata/00003-8f2c.metadata.json

datasets:
  - from: iceberg:s3://my-bucket/warehouse/db/taxis
    name: taxis_v3
    params:
      iceberg_s3_region: us-east-1
      iceberg_metadata_path: /metadata/00003-8f2c.metadata.json

Iceberg Append Refresh Follows Table Snapshots

An Iceberg dataset with refresh_mode: append and a time_column now loads the rows that each new commit adds, whatever their time values. This applies to Cayenne datasets and to datasets without a primary key. Before this release, an append refresh read only rows later than the stored maximum time minus refresh_append_overlap. A writer that committed out of time order could put rows outside that window. The refresh now saves the last Iceberg snapshot that it loaded, and the next refresh reads the data files that later snapshots added. See append refresh (opens in a new tab).

Operations and Observability

The runtime can now send structured logs through the existing OTLP exporter (opens in a new tab):

runtime:
  telemetry:
    otel_exporter:
      endpoint: otel-collector:4317
      signals: [metrics, logs]
  query:
    log_errors: execution

runtime:
  telemetry:
    otel_exporter:
      endpoint: otel-collector:4317
      signals: [metrics, logs]
  query:
    log_errors: execution

Log records carry severity, typed attributes, and the query trace ID. The runtime batches delivery on a dedicated export runtime. signals defaults to [metrics].

Failure records contain error_code and datasets. They do not contain SQL text. See runtime.query.log_errors (opens in a new tab) and observability (opens in a new tab).

Other changes:

• Adaptive HTTP rate control: HTTP rate control now reduces admission after upstream failures and stays within the configured request limits. In Spice.ai Enterprise, instances that share runtime.state also share per-second and per-minute limits. See HTTP rate control (opens in a new tab).
• Open-file limits on Unix: The runtime raises its soft open-file limit to the hard limit at startup. On macOS, kern.maxfilesperproc also bounds the result. If the hard limit remains too low, the Cayenne error names the effective limit and the remedy.
• Shared-state metrics: state_store_request_duration_ms{feature, operation}, state_store_conflicts_total{feature}, and state_store_cleanup_deleted_total{feature} report requests, conditional-write conflicts, and cleanup.
• Prometheus scrapes: Derived summary families emit one HELP and TYPE block per family. Strict parsers no longer reject the scrape for repeated metadata.
• Datadog dashboard: New panels cover snapshot freshness, Cayenne segment caches, and pod health.

Security Improvements

• Brotli decoder (CVE-2026-111497): The DuckDB that Spice bundles no longer writes out of bounds when it decodes a malformed Brotli-compressed Parquet page. Most Parquet reads in Spice do not go through DuckDB. The fix matters where DuckDB decodes a Parquet file that an untrusted party can supply.
• Flight (opens in a new tab) sessions: Authentication precedes session creation. Sessions remain bound to their creating principal. TLS handshakes have a 10-second deadline, and Flight SQL statement updates count against the Flight write limit.
• HTTP paths: The HTTP connector (opens in a new tab) rejects request_path values that URL parsing would rewrite outside their allowed paths. This covers encoded dot segments and control-character rewrites.

Bug Fixes

The fixes with the widest effect are below. The changelog (opens in a new tab) lists every fix.

• Wrong rows from Cayenne queries: Cayenne queries no longer return wrong rows from maintained views, Vortex IN lists, or shared dynamic filters. See #14427 (opens in a new tab).
• NOT IN and NULL values: Queries with NOT IN no longer lose their NULL semantics after a join reorder or the Cayenne sort-merge rewrite. See #14429 (opens in a new tab).
• Wrong results from DataFusion: The DataFusion upgrade includes upstream fixes for wrong results from filter pushdown, expression simplification, and planning. See #14430 (opens in a new tab).
• Federated pushdown: Federated queries no longer send some casts and filters to a source that evaluates them differently from Spice. Affected sources include DuckDB, PostgreSQL, MySQL, BigQuery, DynamoDB, Cosmos DB, and MongoDB. See #14484 (opens in a new tab), #14601 (opens in a new tab), and #14419 (opens in a new tab).
• Accelerated HTTP datasets: An HTTP error response no longer replaces the rows of an accelerated HTTP dataset. See #13538 (opens in a new tab).
• Unavailable sources at startup: A dataset whose source is unreachable at startup now retries. An existing acceleration keeps serving queries while its source is unavailable. See #14623 (opens in a new tab) and #14624 (opens in a new tab).
• Stale plans and results after a reload: Queries no longer use cached plans or results after a hot reload replaces a catalog or unloads a dataset. See #13914 (opens in a new tab) and #14365 (opens in a new tab).
• CDC positions at shutdown: Change-data-capture sources now record their positions before shutdown closes the accelerations. See #14702 (opens in a new tab).
• Tool calls: Tool-use streams no longer end on a stale tool_calls finish. /v1/responses runs Spice tools and honors tool_recursion_limit when it streams. See #14548 (opens in a new tab) and #14930 (opens in a new tab).
• Vector search on chunked columns: vector_search returns its top k rows by score on chunked and multi-vector columns. See #15056 (opens in a new tab).

SDK Updates

Updated SDKs release alongside v2.4.0:

• spice.js v3.3.0 (opens in a new tab): adds query cancellation with an AbortSignal, runtimeStatus(), and refreshDataset(). refreshAcceleration() is deprecated. Fixes intermittent HTTP query failures on Node, API keys on plaintext gRPC, and typed Flight parameters. If you set only one of flightUrl or httpUrl, the client now uses the matching address for the other. An empty URL or a Flight authentication failure now throws.
• spicepy v4.0.1 (opens in a new tab): sends API keys as a Bearer token on Flight and ADBC calls to self-managed runtimes, and binds numpy and pandas scalars as query parameters. refresh_dataset() and runtime_status() now raise SpiceAIError with the runtime's error message instead of requests.HTTPError. Requires certifi 2026.10.19 or later.
• spice-rs v5.0.0 (opens in a new tab): upgrades to Arrow 59 and binds chrono and std::time values as query parameters. Parameterized queries now work with API keys, and queries retry after a connection reset. Applications that also depend on arrow must move to Arrow 59.
• spice-java v0.8.1 (opens in a new tab): accepts any non-empty API key, and sends it on refreshDataset. Fixes the NoClassDefFoundError that v0.8.0 raised on every TLS connection, including Spice.ai Cloud.
• spice-dotnet v0.5.0 (opens in a new tab): adds GetRuntimeStatusAsync() for /v1/status, which all six SDKs now support. Accepts any non-empty API key, and supports grpc:// Flight addresses.
• gospice v9.1.0 (opens in a new tab): binds time.Time, time.Duration, pointers, sql.Null* types, and named types as query parameters. Accepts any non-empty API key, and uses one Flight session for each client. Requires Go 1.26 or later. Decimal128Param and Decimal256Param now read Arrow little-endian bytes, so check those calls before you upgrade.

Agent Skills Updates

Updated Agent Skills release alongside v2.4.0:

• spiceai/skills v2.4.0 (opens in a new tab):
  • New launch skill: deploys a Spice.ai Cloud project, validates that it serves data and model responses, and sets up monitors and alerts.
  • New sdk skill: covers the six client SDKs.
  • cloud skill (previously spice-cloud-management): now works through the management MCP server and can fork projects.
  • All skills are updated for v2.4.0, including Cayenne as the default accelerator and the deprecation of other accelerators.

Dependency Updates

Breaking Changes

Accelerator Selection and Primary-Key Policies

Spice Cayenne is now the default accelerator on supported platforms. Datasets and views without acceleration.engine switch from Arrow to Cayenne. Native Windows keeps Arrow. If datasets rely on another accelerator's behavior, review the Cayenne migration guide (opens in a new tab) before the upgrade.

Other accelerators are deprecated for removal in v2.5. This applies to Arrow, DuckDB, SQLite, Turso, and PostgreSQL acceleration, and to the DuckDB vector engine. They remain available in v2.4. A startup warning groups affected components by accelerator.

On native Windows, run Spice in Docker or WSL to use Cayenne before v2.5.

Some datasets that loaded on Arrow fail to load on Cayenne. These configurations loaded on Arrow when they did not set engine. They now fail to load unless the dataset sets engine: arrow:

• partition_by in memory mode, or with no mode
• A column of type Interval, Duration, FixedSizeBinary, Union, or RunEndEncoded, such as a PostgreSQL interval column
• A Debezium or cdc: source with no key
• A numeric time_column with retention_period
• maintained_aggregates with MIN or MAX and no primary key

To retain Arrow temporarily in v2.4, name it explicitly:

acceleration:
  enabled: true
  engine: arrow

acceleration:
  enabled: true
  engine: arrow

Plan acceleration migration before v2.5. Do not delete acceleration-only data to change accelerators. Keep those datasets on v2.4 until a supported migration can preserve their data. The migration guide describes other limitations and vector-score changes.

on_conflict is deprecated for removal in v3.0. Cayenne accepts the field but no longer applies its policy. A primary key selects automatic version resolution. Before an upgrade, review drop policies and policies that reject conflicts. Other accelerators retain their existing policies in v2.4.

on_conflict no longer routes writes to the acceleration, for all accelerators. In v2.3, a read-write dataset that set on_conflict sent its writes to the acceleration, so its source did not need to accept writes. In v2.4, only write_mode selects where writes go. Such a dataset does not load if its source cannot accept writes. To keep writes in the acceleration, set write_mode: acceleration:

datasets:
  - from: s3://my-bucket/orders/
    name: orders
    access: read_write
    params:
      file_format: parquet
    acceleration:
      enabled: true
      mode: file
      primary_key: id
      write_mode: acceleration

datasets:
  - from: s3://my-bucket/orders/
    name: orders
    access: read_write
    params:
      file_format: parquet
    acceleration:
      enabled: true
      mode: file
      primary_key: id
      write_mode: acceleration

SQL Results-Cache TTL and Freshness

SQL results-cache entries now expire only by TTL. Full and append refreshes, CDC batches, and retention passes no longer invalidate them. Queries can return data from before the latest refresh until item_ttl expires. The stale_while_revalidate_ttl setting can extend this period. Review both settings before the upgrade. Reduce them if results must reflect refreshed data sooner. See Cached Results and Dataset Refresh (opens in a new tab).

HTTP Rate-Control Wait Timeouts

If rate_control_acquire_timeout is unset, an HTTP request now waits up to the connector's client_timeout for rate-control capacity. Requests previously waited without a limit. Set rate_control_acquire_timeout: 0 under the dataset's params to retain unbounded waits. Otherwise, choose a duration that matches the request budget, such as 30s.

Datasets that share an HTTP origin share one rate controller. They must use the same max_concurrent_requests, requests_per_second_limit, requests_per_minute_limit, rate_control_jitter_min, rate_control_jitter_max, rate_control_failure_threshold, and rate_control_window values. If the values differ, the dataset fails to load with an error that names these settings. See rate control parameters (opens in a new tab).

Iceberg REST Compatibility

GET /v1/namespaces/{namespace}/tables/{table} now returns HTTP 400 with BadRequestException for accelerated datasets, views, and other unsupported datasets. The endpoint no longer synthesizes metadata for those datasets. For schema discovery, use SQL DESCRIBE or information_schema.columns. For queries, use /v1/sql or Arrow Flight SQL. Eligible unaccelerated Iceberg datasets expose their source metadata, and clients need their own storage credentials.

Instance Identity and Cluster Upgrades

Instance identity resolves in this order: nonempty SPICE_INSTANCE_ID, HOSTNAME, COMPUTERNAME, then the host name. A scheduler's default changes from host:port to the host name. Executors also register under the instance identity.

If several schedulers or executors share a host, give each a distinct SPICE_INSTANCE_ID. Upgrade schedulers before executors. See distributed query (opens in a new tab). Older schedulers cannot fan out task-history requests to newer identity-based registrations. All cluster components must use compatible Ballista protocol versions.

PostgreSQL replication slots keep their names when a nonempty identity environment variable already selected the instance name. Without those variables, the host-name fallback creates a new slot. An empty SPICE_INSTANCE_ID also changes the previous slot name. After the new slot works, drop the unused old slot to stop its WAL retention. See PostgreSQL replication (opens in a new tab):

SELECT pg_drop_replication_slot('<old slot name>');

SELECT pg_drop_replication_slot('<old slot name>');

Shared State and Snapshot Migration

The top-level snapshots section and runtime.scheduler.state_location and params are deprecated. They continue to work, and when set they take priority over runtime.state for their feature. Prefer runtime.state, with per-dataset overrides (acceleration.snapshots_location, acceleration.snapshots_params) when necessary.

For upgrades from v2.4.0-rc.1, move state from the shared root into these paths:

An explicit legacy scheduler location retains its earlier layout until migrated. In rc.1, Azure state ignored the URL path and used the container root. Move that state into the configured path and feature subdirectory.

The configured store must support writes and conditional writes. The runtime refuses newer state formats and stores that ignore write conditions. Check storage permissions before the upgrade.

Snapshots now expire automatically after 28 days, except for each dataset's current and newest snapshot. Rollback cannot select a pruned snapshot. Review external retention requirements before the upgrade.

Upgrade snapshot readers before, or alongside, their writers. Cayenne writes format 2 of the metastore slice when a snapshot contains partition children or pending write-back markers. Older readers reject that format. Other slices retain format 1.

Cayenne cannot publish acceleration snapshots for datasets with cayenne_datalake_location. Review snapshot settings for those datasets.

For a serving snapshot reader, bootstrap_on_failure_behavior: retry no longer retries a newer snapshot indefinitely inside one refresh. The reader reports the failure and retries at the next poll. Startup behavior does not change.

Cayenne Tuning Settings

Replace retired per-dataset and per-catalog tuning settings with runtime settings:

The runtime no longer applies the old names. It warns with their replacements. Targets are runtime-wide. runtime.adaptive_tuning accepts only enabled or disabled. A Boolean such as true fails Spicepod loading.

Upgrading

Review Breaking Changes (opens in a new tab) before upgrading. Upgrade all distributed cluster components together, with schedulers before executors.

Upgrade snapshot readers before, or alongside, their writers so they can read the newer Cayenne metastore slice format.

To upgrade to v2.4.0, use one of these methods:

CLI:

spice upgrade v2.4.0

spice upgrade v2.4.0

Homebrew:

brew upgrade spiceai/spiceai/spice

brew upgrade spiceai/spiceai/spice

Homebrew follows the version published in the tap.

Docker:

docker pull spiceai/spiceai:2.4.0

docker pull spiceai/spiceai:2.4.0

See Docker Hub (opens in a new tab) for available tags.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.4.0

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.4.0

AWS Marketplace:

Spice is available in the AWS Marketplace (opens in a new tab). Marketplace availability follows its published versions.

Cookbook Updates

• New Decisions API recipe (opens in a new tab): call /v1/decisions with TypeSafe Jev and GPT-6 Luna.

The Spice Cookbook (opens in a new tab) provides recipes to help you get started with Spice.

Contributors

• @bjchambers (opens in a new tab)
• @ewgenius (opens in a new tab)
• @Jeadie (opens in a new tab)
• @krinart (opens in a new tab)
• @lukekim (opens in a new tab)
• @peasee (opens in a new tab)
• @phillipleblanc (opens in a new tab)
• @sgrebnov (opens in a new tab)

What's Changed

Changelog

• fix(runtime): discard cached logical plans when a hot reload replaces a catalog (fixes #13910) by @claudespice (opens in a new tab) in #13914 (opens in a new tab)
• fix(acceleration): let a schema repair correct a checkpoint without resetting the freshness clock (fixes #13817) by @claudespice (opens in a new tab) in #13894 (opens in a new tab)
• fix(search): filter a chunked Elasticsearch delete on a field that can match the key (fixes #13714) by @claudespice (opens in a new tab) in #13926 (opens in a new tab)
• fix(search): classify a partially non-finite embedding as unindexable on every backend (fixes #13872) by @claudespice (opens in a new tab) in #13902 (opens in a new tab)
• fix: stabilize GitHub tests and bound GraphQL registration (fixes #13762) by @lukekim (opens in a new tab) in #13939 (opens in a new tab)
• fix(postgres): decode versioned JSONB binary replication values by @phillipleblanc (opens in a new tab) in #13962 (opens in a new tab)
• fix(cayenne): release the keyset bytes an abandoned PK checkout accounted (fixes #13668) by @grokspice (opens in a new tab) in #13925 (opens in a new tab)
• fix(duckdb): deny the regexp built-ins DuckDB cannot answer faithfully (fixes #13809) by @claudespice (opens in a new tab) in #13871 (opens in a new tab)
• fix(search): drop the chunks a shortened row no longer produces from a chunked index (refs #13717) by @claudespice (opens in a new tab) in #13960 (opens in a new tab)
• Reduce Cayenne allocations during primary-key validation and filtering by @lukekim (opens in a new tab) in #14009 (opens in a new tab)
• fix: clarify OpenDAL S3 retry warnings by @lukekim (opens in a new tab) in #14040 (opens in a new tab)
• feat(cayenne): materialize multi-reference CTEs on the query path by @lukekim (opens in a new tab) in #13918 (opens in a new tab)
• fix(smb): pad an empty CREATE buffer so Samba lists the share root (fixes #13293) by @grokspice (opens in a new tab) in #14050 (opens in a new tab)
• feat(mcp): support MCP specification 2026-07-28 (dual-era) by @lukekim (opens in a new tab) in #14043 (opens in a new tab)
• feat(connectors): add ORC listing format via in-repo FileFormat by @lukekim (opens in a new tab) in #14075 (opens in a new tab)
• feat(cloud-connect): add a GetDatasets command that answers the /v1/datasets document (refs #13369) by @grokspice (opens in a new tab) in #14051 (opens in a new tab)
• Suppress Cayenne startup logs when no Cayenne dataset is configured by @Jeadie (opens in a new tab) in #14042 (opens in a new tab)
• fix(cache): re-bind parameter values when revalidating a stale result (fixes #14099) by @bjchambers (opens in a new tab) in #14100 (opens in a new tab)
• fix(deps): remediate open Dependabot alerts by @phillipleblanc (opens in a new tab) in #14111 (opens in a new tab)
• feat(caching): Make caching_stale_if_error RFC-5861 compliant (with stale-if-error header) by @Jeadie (opens in a new tab) in #14134 (opens in a new tab)
• perf(runtime-table): defer cache-eviction key extraction to entries a delete actually names by @Jeadie (opens in a new tab) in #14138 (opens in a new tab)
• fix(runtime): report the acceleration.ready_state deprecation once per component (fixes #13749) by @claudespice (opens in a new tab) in #14006 (opens in a new tab)
• fix(runtime): write the inferred Arrow sort order under the prefixed key its validation accepts (fixes #14023) by @claudespice (opens in a new tab) in #14032 (opens in a new tab)
• fix(connectors): Fix JSON/Orca files using metadata columns by @Jeadie (opens in a new tab) in #14115 (opens in a new tab)
• fix(turso): build a dictionary column directly so a dictionary over a list, map or boolean value reads back (fixes #13033) by @grokspice (opens in a new tab) in #14181 (opens in a new tab)
• fix(github): paginate nested GraphQL connections and pace to GitHub's rate limits by @lukekim (opens in a new tab) in #14179 (opens in a new tab)
• fix(vortex): stop an IN list holding a NULL from panicking the scan by @krinart (opens in a new tab) in #14163 (opens in a new tab)
• fix(runtime-table): stop rebuilding SessionContext on every cache fetch by @Jeadie (opens in a new tab) in #14141 (opens in a new tab)
• fix(runtime): count a first load as still loading in the Dataset load summary (fixes #13974) by @claudespice (opens in a new tab) in #14020 (opens in a new tab)
• fix(duckdb): push regexp_count down again at a rendering that counts as the kernel does (fixes #13870) by @claudespice (opens in a new tab) in #14153 (opens in a new tab)
• fix(cayenne): build the DELETE sink inside the execution-time write lock (fixes #13828) by @claudespice (opens in a new tab) in #14218 (opens in a new tab)
• fix(cluster): decide what a Flight message carries by its IPC header, not its body length (refs #13737) by @claudespice (opens in a new tab) in #14212 (opens in a new tab)
• fix(runtime): install the Spice function deny-list on the PostgreSQL catalog connector (refs #13664) by @claudespice (opens in a new tab) in #14225 (opens in a new tab)
• perf(cayenne): share inline-cache view entries by Arc instead of cloning them per scan by @krinart (opens in a new tab) in #14191 (opens in a new tab)
• feat(cache): Spice sharded cache as the sole LruCache engine by @lukekim (opens in a new tab) in #14206 (opens in a new tab)
• feat: TypeSafe Jev System One evaluation provider by @lukekim (opens in a new tab) in #14215 (opens in a new tab)
• fix(cayenne): tag file statistics bounds as the column's Arrow type (fixes #14280) by @phillipleblanc (opens in a new tab) in #14283 (opens in a new tab)
• fix(cayenne): coalesce inline writes into one batch by @sgrebnov (opens in a new tab) in #14279 (opens in a new tab)
• fix(cayenne): count each Arrow allocation once in the inline-cache gauge by @krinart (opens in a new tab) in #14272 (opens in a new tab)
• feat(s3): SQS event-driven changes for refresh_mode: changes by @lukekim (opens in a new tab) in #14121 (opens in a new tab)
• feat(caching): single-flight coalesce concurrent cache-miss fetches by @Jeadie (opens in a new tab) in #14142 (opens in a new tab)
• fix(caching): make caching_stale_if_error detect transient HTTP failures on real schemas by @krinart (opens in a new tab) in #14161 (opens in a new tab)
• Use Cayenne secondary indexes for dynamic join filters by @phillipleblanc (opens in a new tab) in #14282 (opens in a new tab)
• fix(cli): surface the full error chain for spice chat connection failures by @krinart (opens in a new tab) in #14289 (opens in a new tab)
• fix(runtime-table): serialize a direct write against acceleration snapshot creation (fixes #13548) by @claudespice (opens in a new tab) in #14310 (opens in a new tab)
• fix(duckdb): screen regexp_like and regexp_replace as regexp_count is screened (fixes #14148) by @claudespice (opens in a new tab) in #14321 (opens in a new tab)
• fix(http): honor retry budget without an extra origin request by @phillipleblanc (opens in a new tab) in #14313 (opens in a new tab)
• Fix SchemaCastScanExec's schema conversion in fn partition_statistics by @Jeadie (opens in a new tab) in #14258 (opens in a new tab)
• fix(cluster): recognise a Flight keepalive by its empty envelope, not by what its header declares (fixes #13737) by @claudespice (opens in a new tab) in #14327 (opens in a new tab)
• perf(http): defer zero-TTL acceleration lookup until origin failure by @phillipleblanc (opens in a new tab) in #14302 (opens in a new tab)
• fix(cayenne): keep a key visible when it is re-inserted over a stale-insert tombstone by @sgrebnov (opens in a new tab) in #14312 (opens in a new tab)
• perf(cayenne): read only key and filter columns in filtered key deletes by @sgrebnov (opens in a new tab) in #14374 (opens in a new tab)
• fix(cayenne): let small protected-snapshot merges run during a long large-tier merge by @sgrebnov (opens in a new tab) in #14296 (opens in a new tab)
• perf(cayenne): serve primary-key lookups on a freshly loaded table in ~1 ms by @lukekim (opens in a new tab) in #14314 (opens in a new tab)
• fix(cayenne): skip min/max statistics for nested columns (fixes #14368) by @sgrebnov (opens in a new tab) in #14392 (opens in a new tab)
• fix(turso): keep a quantified comparison out of Turso SQL (fixes #14041) by @grokspice (opens in a new tab) in #14393 (opens in a new tab)
• perf(cayenne): batch metastore manifest rewrites, upsert in place, and run every write on one writer connection by @lukekim (opens in a new tab) in #14369 (opens in a new tab)
• fix(runtime-table): ignore zero-row stale fallback batches by @phillipleblanc (opens in a new tab) in #14331 (opens in a new tab)
• Prune object-store file listing by _last_modified predicates by @Jeadie (opens in a new tab) in #14265 (opens in a new tab)
• Reading only partition or metadata columns needlessly scans all file contents by @Jeadie (opens in a new tab) in #14116 (opens in a new tab)
• fix(duckdb): keep a concat over a binary operand out of the federated plan (fixes #13915) by @claudespice (opens in a new tab) in #14333 (opens in a new tab)
• fix(federation): keep a correlated subquery predicate above a join of two sources (refs #8220) by @claudespice (opens in a new tab) in #14372 (opens in a new tab)
• fix(caching): snapshot staleness before the origin fetch for stale_if_error by @Jeadie (opens in a new tab) in #14263 (opens in a new tab)
• perf(snapshots): skip unchanged snapshot metadata with a conditional GET by @sgrebnov (opens in a new tab) in #14409 (opens in a new tab)
• fix(search): prune the rest of a key group from an Elasticsearch chunked index (refs #13717) by @claudespice (opens in a new tab) in #14320 (opens in a new tab)
• fix(graphql): debit a LIMIT by the rows a page returned, not the declared page size (fixes #14308) by @claudespice (opens in a new tab) in #14353 (opens in a new tab)
• fix(snapshots): resolve snapshot entries relative to the metadata location (#14425) by @sgrebnov (opens in a new tab) in #14426 (opens in a new tab)
• Lower GitHub connector default concurrency limit to 4 by @lukekim (opens in a new tab) in #14431 (opens in a new tab)
• fix(cache): don't report an invalidation cancelled by runtime shutdown as a failure by @grokspice (opens in a new tab) in #14417 (opens in a new tab)
• fix(runtime): keep the built-in date_part so both weekday spellings agree (fixes #13920) by @claudespice (opens in a new tab) in #14154 (opens in a new tab)
• fix(cayenne): include the in-memory CDC tier when an overwrite or a retention pass covers the whole table by @lukekim (opens in a new tab) in #14428 (opens in a new tab)
• fix: keep NOT IN null-aware through the join reorder and the Cayenne sort-merge rewrite by @lukekim (opens in a new tab) in #14429 (opens in a new tab)
• fix(cayenne): discard a compaction whose snapshot an overwrite replaced mid-pass by @lukekim (opens in a new tab) in #14432 (opens in a new tab)
• fix(cayenne): stop maintained views, Vortex IN lists and dynamic-filter sharing from returning wrong rows by @lukekim (opens in a new tab) in #14427 (opens in a new tab)
• fix(cayenne): hide spilled rows on CDC upsert fallback by @bjchambers (opens in a new tab) in #14416 (opens in a new tab)
• fix(cayenne): judge a filtered key delete's captured sources by the index captured with them (refs #13913) by @claudespice (opens in a new tab) in #14455 (opens in a new tab)
• fix(cayenne): correctness fixes for the goal-driven adaptive controller, with a closed-loop simulation harness by @lukekim (opens in a new tab) in #14443 (opens in a new tab)
• fix(cdc): keep the newest source commit timestamp on a coalesced change batch by @lukekim (opens in a new tab) in #14463 (opens in a new tab)
• fix(cayenne): draw a sequence for a current-snapshot append so per-key OCC can order it (fixes #13685) by @claudespice (opens in a new tab) in #14360 (opens in a new tab)
• fix(http): name a configured model's load failure instead of reporting it not found (fixes #13303) by @claudespice (opens in a new tab) in #14395 (opens in a new tab)
• fix(duckdb): keep inferred source indexes off change-stream accelerations so upserts commit under concurrent reads (refs #13929) by @claudespice (opens in a new tab) in #14396 (opens in a new tab)
• fix(duckdb): keep a text cast over a binary operand out of the federated plan (fixes #14355) by @claudespice (opens in a new tab) in #14448 (opens in a new tab)
• fix(llms): honor tool_choice required and allowed_tools on mistral.rs-hosted models instead of panicking (fixes #14230) by @claudespice (opens in a new tab) in #14460 (opens in a new tab)
• fix(runtime): stop a replaced dataset configuration's load from registering over the new one (fixes #1458) by @claudespice (opens in a new tab) in #14367 (opens in a new tab)
• fix(install): stop asking for sudo on a first install into a fresh HOME (fixes #14445) by @claudespice (opens in a new tab) in #14495 (opens in a new tab)
• perf(cayenne): serve primary-key point lookups in half the time by @lukekim (opens in a new tab) in #14433 (opens in a new tab)
• fix(deps): bump DataFusion for upstream fixes to wrong results from filter pushdown, simplification and planning by @lukekim (opens in a new tab) in #14430 (opens in a new tab)
• fix(runtime): size every internal DataFusion session from the CPU budget by @bjchambers (opens in a new tab) in #14412 (opens in a new tab)
• fix(runtime): serve nested, zoned and half-float columns from the Iceberg catalog API (fixes #4815) by @claudespice (opens in a new tab) in #14480 (opens in a new tab)
• fix(acceleration): keep cached results when a snapshot refresh finds no newer snapshot by @sgrebnov (opens in a new tab) in #14497 (opens in a new tab)
• fix(cayenne): stop the small-file compaction of a position-mode PK table from deadlocking on its own write lock (fixes #14420) by @claudespice (opens in a new tab) in #14481 (opens in a new tab)
• fix(runtime-table): report refresh bytes for the rows each batch holds by @Jeadie (opens in a new tab) in #14469 (opens in a new tab)
• fix(spark,databricks): keep Spice-only functions out of SQL sent to Spark Connect and Databricks SQL Warehouse (refs #13664) by @claudespice (opens in a new tab) in #14498 (opens in a new tab)
• fix(vortex): size a cached footer by what it retains, not its serialized bytes (fixes #12917) by @claudespice (opens in a new tab) in #14502 (opens in a new tab)
• fix(cayenne,telemetry): Register accelerated sink dataset immediately from existing acceleration by @peasee (opens in a new tab) in #13955 (opens in a new tab)
• fix(cayenne): keep a rewrite's count inexact when it retains a late protected snapshot (fixes #14383) by @claudespice (opens in a new tab) in #14385 (opens in a new tab)
• fix(postgres): release a shared slot's hold on a table its publication cannot drop (fixes #13032) by @claudespice (opens in a new tab) in #14527 (opens in a new tab)
• fix(cayenne): size the build side before sort-merging a non-Cayenne outer join by @krinart (opens in a new tab) in #14520 (opens in a new tab)
• fix(deps): move the DataFusion pin past the four unparser fixes, and guard each of them (fixes #13022) by @grokspice (opens in a new tab) in #14570 (opens in a new tab)
• fix(dynamodb, cosmosdb, mongodb): push filters down only where the source evaluates them as SQL does by @lukekim (opens in a new tab) in #14419 (opens in a new tab)
• feat(snapshots): serve a dataset from published snapshots with from: s3://… and file_format: snapshot by @lukekim (opens in a new tab) in #14529 (opens in a new tab)
• perf(cayenne): keep the per-shard PK index across checkpoint flushes and back off futile bakes; fix q17 and two wrong-results races (fixes #14235) by @lukekim (opens in a new tab) in #14555 (opens in a new tab)
• fix(graphql): show the parse-failure bytes in JSON decode previews by @Jeadie (opens in a new tab) in #14530 (opens in a new tab)
• Defer source-first cache fallback planning by @phillipleblanc (opens in a new tab) in #14556 (opens in a new tab)
• perf(cayenne): order whole-table rewrites per scan partition, then merge by @bjchambers (opens in a new tab) in #14488 (opens in a new tab)
• fix(graphql): retry an inferred credential refusal instead of failing the refresh by @Jeadie (opens in a new tab) in #14539 (opens in a new tab)
• fix(cayenne): serve a widened table's files from their persisted statistics (refs #13829) by @claudespice (opens in a new tab) in #14316 (opens in a new tab)
• fix(acceleration): checkpoint DuckDB's write-ahead log before a snapshot copies its file (fixes #13912) by @claudespice (opens in a new tab) in #14325 (opens in a new tab)
• fix(models): apply a tool_choice that forces a call to one round of the tool-use loop (fixes #14459) by @claudespice (opens in a new tab) in #14635 (opens in a new tab)
• fix(runtime): don't end a tool-use stream on a stale tool_calls finish (fixes #13309) by @claudespice (opens in a new tab) in #14548 (opens in a new tab)
• fix(smb): name listed locations under the share so directory datasets load (fixes #14060) by @claudespice (opens in a new tab) in #14549 (opens in a new tab)
• fix(models): name a configured model's load failure in ai() and streaming nsql (fixes #14394) by @claudespice (opens in a new tab) in #14632 (opens in a new tab)
• fix(cayenne): record sharded CDC keys in the table-wide PK index by @lukekim (opens in a new tab) in #14603 (opens in a new tab)
• fix(sqlite): keep TRY_CAST and every cast SQLite answers differently out of the federated plan (fixes #14398) by @claudespice (opens in a new tab) in #14496 (opens in a new tab)
• fix(snapshots): retry a failed snapshot attempt with backoff by @sgrebnov (opens in a new tab) in #14676 (opens in a new tab)
• fix(snapshots): large snapshots no longer fail bootstrap on slow connection by @sgrebnov (opens in a new tab) in #14571 (opens in a new tab)
• fix(refresh): start a refresh's source scan only when the sink reads it, so the full-text index keeps its rows (fixes #14619) by @claudespice (opens in a new tab) in #14663 (opens in a new tab)
• Evaluate any chat model via /v1/evaluate by @lukekim (opens in a new tab) in #14568 (opens in a new tab)
• fix(cayenne): stop snapshotting datasets with a datalake tier, whose restored copies deleted each other's files by @lukekim (opens in a new tab) in #14583 (opens in a new tab)
• fix(cayenne): count each Arrow allocation once for the mem-tier limit and checkpoint write sizing by @krinart (opens in a new tab) in #14675 (opens in a new tab)
• fix(runtime): retry a dataset whose source is unreachable at startup (fixes #14609) by @bjchambers (opens in a new tab) in #14623 (opens in a new tab)
• fix(cayenne): archive only the snapshot directories a dataset snapshot references (fixes #14605) by @sgrebnov (opens in a new tab) in #14627 (opens in a new tab)
• fix(smb): serve every share on a host from the one store registered for it (fixes #14550) by @grokspice (opens in a new tab) in #14689 (opens in a new tab)
• fix(runtime): discard cached plans and results when a dataset is unloaded (refs #14251) by @claudespice (opens in a new tab) in #14365 (opens in a new tab)
• feat(snapshots): reload snapshot-mode datasets from S3 event notifications (SQS) by @lukekim (opens in a new tab) in #14335 (opens in a new tab)
• fix(cayenne): report runs per size tier when protected-snapshot compaction declines (fixes #13622) by @claudespice (opens in a new tab) in #14476 (opens in a new tab)
• fix(cayenne): re-run a post-write compaction pass a concurrent append asked for (fixes #13906) by @claudespice (opens in a new tab) in #14479 (opens in a new tab)
• fix(http): keep the request URL out of HTTP connector errors (fixes #13534) by @claudespice (opens in a new tab) in #14490 (opens in a new tab)
• fix(runtime): load every localpod dataset that reads from one parent at startup (fixes #13087) by @claudespice (opens in a new tab) in #14532 (opens in a new tab)
• fix(duckdb): push regexp_replace down again for a one-digit group reference, and refresh the Postgres ClickBench q35 plan (refs #13966) by @claudespice (opens in a new tab) in #14552 (opens in a new tab)
• fix(duckdb): keep a cast into binary out of the federated plan (refs #14397) by @claudespice (opens in a new tab) in #14633 (opens in a new tab)
• fix(runtime): keep finished async query jobs finished, and delete expired ones by @lukekim (opens in a new tab) in #14585 (opens in a new tab)
• fix(federation): keep DataFusion's cast built-ins out of every federated plan (fixes #14444) by @claudespice (opens in a new tab) in #14484 (opens in a new tab)
• fix(runtime): decide every Spark/built-in function collision by name, and refuse an undecided one (fixes #14361) by @grokspice (opens in a new tab) in #14400 (opens in a new tab)
• fix(runtime-component): skip an inferred Cayenne index on a floating-point column instead of failing the load (fixes #14590) by @grokspice (opens in a new tab) in #14599 (opens in a new tab)
• fix(cache): serve stale cached results during frequent table updates (stale_while_revalidate_ttl) by @sgrebnov (opens in a new tab) in #14708 (opens in a new tab)
• Update Turso to 0.8.1 and retry write conflicts a metastore statement raises by @lukekim (opens in a new tab) in #14680 (opens in a new tab)
• fix(snapshots): stop a snapshot dataset's load when a reload replaces it by @lukekim (opens in a new tab) in #14673 (opens in a new tab)
• fix(cayenne): hand non-partition filters to every partition scan (fixes #12959) by @claudespice (opens in a new tab) in #14370 (opens in a new tab)
• fix(postgres-accel): resolve secret references in the sidecar connection parameters (fixes #13296) by @claudespice (opens in a new tab) in #14544 (opens in a new tab)
• fix(cayenne): drain the in-memory CDC tier before a full rewrite scans it (fixes #14450) by @claudespice (opens in a new tab) in #14486 (opens in a new tab)
• feat(caching): warm SQL results cache on first refresh from persisted plan shapes by @lukekim (opens in a new tab) in #14178 (opens in a new tab)
• fix(federation): keep a fractional-to-integer cast out of the plans pushed to DuckDB, PostgreSQL, MySQL and BigQuery (fixes #14482) by @grokspice (opens in a new tab) in #14601 (opens in a new tab)
• fix(federation): keep arrow_typeof and its plan-introspection siblings local on every backend (fixes #14334) by @grokspice (opens in a new tab) in #14695 (opens in a new tab)
• fix(cayenne): back off futile bakes under a violated query goal too by @lukekim (opens in a new tab) in #14736 (opens in a new tab)
• fix(snapshots): bootstrap readers from late snapshot publications by @phillipleblanc (opens in a new tab) in #14468 (opens in a new tab)
• fix(cayenne): apply retention_sql to a mode: memory acceleration (fixes #14045) by @claudespice (opens in a new tab) in #14344 (opens in a new tab)
• fix(duckdb, sqlite): keep the first copy of a key a write repeats under on_conflict: drop (fixes #14629) by @claudespice (opens in a new tab) in #14748 (opens in a new tab)
• fix(cache): release a removed dataset's memory when the plan cache discards its plans (fixes #14251) by @claudespice (opens in a new tab) in #14760 (opens in a new tab)
• fix(llms): explain an Anthropic model's refusal of a sampling control instead of passing the bare 400 through (fixes #13564) by @grokspice (opens in a new tab) in #14700 (opens in a new tab)
• fix(snapshots): allow cayenne_file_path and cayenne_metadata_dir on file_format: snapshot datasets (fixes #14696) by @sgrebnov (opens in a new tab) in #14698 (opens in a new tab)
• feat(cayenne): keep the secondary index current across every write by @bjchambers (opens in a new tab) in #14593 (opens in a new tab)
• fix(http): stop a non-2xx response body replacing an accelerated table's rows (fixes #13515) by @claudespice (opens in a new tab) in #13538 (opens in a new tab)
• fix(chat-api): answer a tool-call-only assistant turn instead of panicking (fixes #13207) by @grokspice (opens in a new tab) in #14232 (opens in a new tab)
• fix(sqlite, duckdb): keep a decimal AVG, and on SQLite a decimal SUM, out of the federated plan (fixes #14492) by @claudespice (opens in a new tab) in #14670 (opens in a new tab)
• fix(spiceai, duckdb, sharepoint): name the Spicepod key a missing-parameter error asks for (refs #14446) by @claudespice (opens in a new tab) in #14769 (opens in a new tab)
• fix(cayenne): keep the per-shard PK index when the table-wide index is discarded by @lukekim (opens in a new tab) in #14604 (opens in a new tab)
• Upgrade to DataFusion 55.1 and Arrow 59.3 by @krinart (opens in a new tab) in #14612 (opens in a new tab)
• Upgrade to iceberg-rust to 0.11.0 and DF 55.2 by @sgrebnov (opens in a new tab) in #14771 (opens in a new tab)
• fix(cayenne): write an append into an empty unkeyed table as one load by @phillipleblanc (opens in a new tab) in #14784 (opens in a new tab)
• fix(mysql_replication): don't re-send a live member's delivered commits after a reconnect by @lukekim (opens in a new tab) in #14751 (opens in a new tab)
• fix(acceleration): accept time_format iso8601 when the time_column is already a timestamp by @lukekim (opens in a new tab) in #14777 (opens in a new tab)
• fix(cache): store SQL results that go stale mid-query by @lukekim (opens in a new tab) in #14710 (opens in a new tab)
• fix(cayenne): serve filtered and global maintained aggregates by @lukekim (opens in a new tab) in #14761 (opens in a new tab)
• perf(cayenne): build the checkpoint's tombstone union after releasing the capture locks by @lukekim (opens in a new tab) in #14759 (opens in a new tab)
• fix(cayenne): keep maintenance from deleting files a snapshot is archiving by @sgrebnov (opens in a new tab) in #14789 (opens in a new tab)
• fix(snapshots): never overwrite shared snapshot metadata, and publish more than once to a file:// location by @lukekim (opens in a new tab) in #14582 (opens in a new tab)
• perf(cdc): build deferred change rows in the reader while the apply runs by @lukekim (opens in a new tab) in #14746 (opens in a new tab)
• fix(runtime,data_components): wait for change-data-capture sources to record their positions before shutdown closes the accelerations (fixes #14523) by @grokspice (opens in a new tab) in #14702 (opens in a new tab)
• fix(graphql): detect non-JSON responses and treat gateway errors as retryable by @lukekim (opens in a new tab) in #14781 (opens in a new tab)
• fix(runtime): require keys for s3_auth: key, honor gs:// state location params, and leave newer rate-control state alone by @lukekim (opens in a new tab) in #14584 (opens in a new tab)
• fix(listing): skip zero-byte objects (S3 folder markers) in format-selected listings by @sgrebnov (opens in a new tab) in #14822 (opens in a new tab)
• Avoid unnecessary repartitioning in indexed dynamic-filter joins by @bjchambers (opens in a new tab) in #14799 (opens in a new tab)
• feat(cayenne): keep one row per primary key automatically and deprecate on_conflict by @bjchambers (opens in a new tab) in #14726 (opens in a new tab)
• fix(object-store): stop reading a response body at its first error by @phillipleblanc (opens in a new tab) in #14831 (opens in a new tab)
• fix(listing): keep a partition predicate as a residual filter on the _location fast path by @grokspice (opens in a new tab) in #14790 (opens in a new tab)
• fix(federation): keep date, timestamp and interval values local on SQLite reached through ADBC or ODBC (fixes #14753) by @claudespice (opens in a new tab) in #14840 (opens in a new tab)
• perf(runtime): skip EnsureRequirements while planning a point lookup by @lukekim (opens in a new tab) in #14807 (opens in a new tab)
• feat(acceleration): make Cayenne the default acceleration engine by @phillipleblanc (opens in a new tab) in #14837 (opens in a new tab)
• fix(cayenne): keep a join's LIMIT when the oversized-join rewrite makes it a sort-merge join by @lukekim (opens in a new tab) in #14805 (opens in a new tab)
• fix(cayenne): fail contradictory write settings once, and order NULL times below every time by @bjchambers (opens in a new tab) in #14847 (opens in a new tab)
• fix(runtime): serve an existing acceleration while its source is unavailable (fixes #14610) by @bjchambers (opens in a new tab) in #14624 (opens in a new tab)
• Prune object-store file listing by metadata columns (#14264) by @Jeadie (opens in a new tab) in #14303 (opens in a new tab)
• fix(vortex): fold partition values into the file-pruning predicate; test null-equal joins under mode:file by @lukekim (opens in a new tab) in #14797 (opens in a new tab)
• feat(rate-control): adaptive, bounded and cluster-coordinated HTTP rate controls by @Jeadie (opens in a new tab) in #14143 (opens in a new tab)
• fix(sql): align date_part('dow') with EXTRACT(dow) Sunday=0 by @lukekim (opens in a new tab) in #14796 (opens in a new tab)
• fix(runtime): carry API-key principal into MCP tools/call by @lukekim (opens in a new tab) in #14828 (opens in a new tab)
• fix(llms): un-mute the LLM integration suite and make it pass (fixes #6399; Bedrock deferred to #13558) by @Jeadie (opens in a new tab) in #13656 (opens in a new tab)
• perf(cayenne): hold maintained-aggregate retraction state in a compact shared index by @lukekim (opens in a new tab) in #14762 (opens in a new tab)
• fix(deps): keep a hash join's order once it has a dynamic filter, so vector_search plans again by @Jeadie (opens in a new tab) in #14857 (opens in a new tab)
• feat(cayenne): persist secondary index runs, recorded in the metastore by @bjchambers (opens in a new tab) in #14594 (opens in a new tab)
• fix(cayenne): fence schema statistics and control maintenance tests by @bjchambers (opens in a new tab) in #14856 (opens in a new tab)
• fix(github): retry nested GraphQL pages in place and fail incomplete nested connections by @sgrebnov (opens in a new tab) in #14862 (opens in a new tab)
• fix(cayenne): load an append dataset that has retention_sql and orders versions by time by @phillipleblanc (opens in a new tab) in #14878 (opens in a new tab)
• fix(cdc): Cayenne replication needs only a primary key, not on_conflict by @phillipleblanc (opens in a new tab) in #14880 (opens in a new tab)
• feat(connector-huggingface): Hugging Face datasets data connector (hf://datasets/...) by @lukekim (opens in a new tab) in #14877 (opens in a new tab)
• fix(iceberg): Iceberg REST clients read the real table or get an error, never an empty one by @lukekim (opens in a new tab) in #14588 (opens in a new tab)
• fix(federation): ask the executor for the SELECT 1 placeholder of an empty projection by @krinart (opens in a new tab) in #14851 (opens in a new tab)
• fix(cayenne): float statistics must not leave NaN out of pruning or metadata answers (fixes #14719) by @claudespice (opens in a new tab) in #14792 (opens in a new tab)
• fix(caching): a GET and a POST to one HTTP path no longer share a cache entry (fixes #14768) by @claudespice (opens in a new tab) in #14884 (opens in a new tab)
• fix(anthropic): drop empty text blocks from assistant messages by @Jeadie (opens in a new tab) in #14834 (opens in a new tab)
• fix(cayenne): ask the table whether it takes a versioned append, instead of predicting it by @lukekim (opens in a new tab) in #14882 (opens in a new tab)
• fix(cayenne): a hash join over a secondary index reads the indexed table's files once (fixes #14933) by @lukekim (opens in a new tab) in #14934 (opens in a new tab)
• fix(sharepoint): list drive items when the listing contains a folder (fixes #2797) by @phillipleblanc (opens in a new tab) in #14946 (opens in a new tab)
• fix(runtime): log each failing dataset's load failure (fixes #14918) by @phillipleblanc (opens in a new tab) in #14939 (opens in a new tab)
• fix(deps): move the DataFusion pin past spiceai/datafusion#256, so FlightSQL Substrait plans read string columns (fixes #14935) by @lukekim (opens in a new tab) in #14944 (opens in a new tab)
• fix(cayenne): publish a keyed partition's append as a protected overlay (fixes #14947) by @lukekim (opens in a new tab) in #14955 (opens in a new tab)
• fix(acceleration): a failed acceleration build no longer keeps the dataset down until restart (fixes #14917) by @phillipleblanc (opens in a new tab) in #14942 (opens in a new tab)
• fix(postgres): reject a changes dataset without a primary key when it loads by @phillipleblanc (opens in a new tab) in #14960 (opens in a new tab)
• fix(cluster): treat secrets the scheduler declines as not found on executors by @phillipleblanc (opens in a new tab) in #14957 (opens in a new tab)
• fix(flight): return query and data errors as InvalidArgument, not Internal by @phillipleblanc (opens in a new tab) in #14956 (opens in a new tab)
• fix(duckdb,sqlite): correct pushdown for string_agg ORDER BY, concat NULLs, and unsupported fns by @lukekim (opens in a new tab) in #14798 (opens in a new tab)
• fix(llms): stream Anthropic responses that open with a thinking block by @krinart (opens in a new tab) in #14920 (opens in a new tab)
• fix(arrow): serve lookups from enabled indexes; a repeated key disables the index with a warning by @sgrebnov (opens in a new tab) in #14968 (opens in a new tab)
• feat(nsql): accept reasoning_effort on POST /v1/nsql by @lukekim (opens in a new tab) in #14852 (opens in a new tab)
• fix(listing): scan file_extension '*' tables through ListingTable, so Glue joins broadcast again (fixes #14890) by @krinart (opens in a new tab) in #14972 (opens in a new tab)
• fix(llms): load and serve an OpenAI model when a reply names an unknown service tier (fixes #14916) by @krinart (opens in a new tab) in #14927 (opens in a new tab)
• feat(cayenne): always persist secondary index runs by @sgrebnov (opens in a new tab) in #14979 (opens in a new tab)
• fix(cache): expire results-cache entries in insertion order instead of walking every LRU list by @bjchambers (opens in a new tab) in #14919 (opens in a new tab)
• feat: decisions in SQL (ai_if, ai_probability, ai_classify, ai_score, ai_decide) and an OpenAI-compatible /v1/decisions by @lukekim (opens in a new tab) in #14838 (opens in a new tab)
• fix(search): keep named arguments on search calls nested in rrf() (fixes #14923) by @phillipleblanc (opens in a new tab) in #14975 (opens in a new tab)
• fix(rate-control): hold instances that set different limits for one origin to the lowest (fixes #14913) by @krinart (opens in a new tab) in #14932 (opens in a new tab)
• fix(vortex): don't push down temporal casts Vortex can't evaluate by @sgrebnov (opens in a new tab) in #14985 (opens in a new tab)
• fix(acceleration): recommend mode: file for CDC on_conflict migrations to Cayenne by @bjchambers (opens in a new tab) in #14928 (opens in a new tab)
• fix(http): reject a request_path that URL parsing would rewrite by @bjchambers (opens in a new tab) in #14965 (opens in a new tab)
• fix(clickhouse): load LowCardinality, DateTime64, Array, Map, Tuple, Enum, 128-bit integer and IP columns by @phillipleblanc (opens in a new tab) in #14953 (opens in a new tab)
• Add anthropic_workspace_id param for Anthropic models by @Jeadie (opens in a new tab) in #14829 (opens in a new tab)
• fix(acceleration): keep retention-evicted rows out of the on_zero_results source fallback by @lukekim (opens in a new tab) in #14783 (opens in a new tab)
• fix(drasi): forward CDC creates and snapshots as upserts by @ewgenius (opens in a new tab) in #14791 (opens in a new tab)
• fix(datafusion-optimizer-rules): answer only plain partition columns from the listing by @lukekim (opens in a new tab) in #14806 (opens in a new tab)
• fix(connector-huggingface): retry Hub API responses that break off part way, and harden the live-Hub check by @lukekim (opens in a new tab) in #15009 (opens in a new tab)
• fix(models): /v1/responses runs Spice tools and honors tool_recursion_limit when streaming (fixes #14905, #14631) by @krinart (opens in a new tab) in #14930 (opens in a new tab)
• fix(cayenne): refuse acceleration data stored under a different partition_by by @phillipleblanc (opens in a new tab) in #15019 (opens in a new tab)
• fix: Cayenne deletion-index bounds and reload fencing, scan-statistics single-flight, UDF filter pushdown, Server-Timing, results-cache determinism, and view loading by @lukekim (opens in a new tab) in #14993 (opens in a new tab)
• fix(cayenne): load write-back without on_conflict, and refuse pk_conflict_detection: none where it loses data (fixes #14886, #14889) by @bjchambers (opens in a new tab) in #14891 (opens in a new tab)
• feat(cayenne): runtime-wide adaptive_tuning and target_* settings by @peasee (opens in a new tab) in #14844 (opens in a new tab)
• fix(snapshots): give each snapshot publish its own object name, so publishes in the same second stop overwriting each other by @phillipleblanc (opens in a new tab) in #15028 (opens in a new tab)
• fix(glue): load Iceberg tables from S3 buckets with dots in the name by @phillipleblanc (opens in a new tab) in #15053 (opens in a new tab)
• fix(runtime): give each runtime.state feature its own folder by @phillipleblanc (opens in a new tab) in #15036 (opens in a new tab)
• fix(json): reject incomplete arrays and avoid repeated parsing by @phillipleblanc (opens in a new tab) in #15054 (opens in a new tab)
• fix(cayenne): keep secondary indexes complete across acceleration snapshots by @sgrebnov (opens in a new tab) in #15068 (opens in a new tab)
• feat(rerank): use evaluation models (TypeSafe) in rerank(), with per-query strategy by @Jeadie (opens in a new tab) in #14782 (opens in a new tab)
• fix(cayenne): reclaim the snapshot directories a snapshot reader's reload supersedes by @sgrebnov (opens in a new tab) in #15069 (opens in a new tab)
• feat(iceberg): read a table from a pinned metadata file without a catalog by @sgrebnov (opens in a new tab) in #15081 (opens in a new tab)
• Fixes for MCP server by @Jeadie (opens in a new tab) in #15041 (opens in a new tab)
• fix(search): vector_search returns its top k rows by score on chunked and multi-vector columns (fixes #14819) by @claudespice (opens in a new tab) in #15056 (opens in a new tab)
• fix(postgres): keep applying CDC changes after a source column type change, and never store a rounded value (refs #14718) by @claudespice (opens in a new tab) in #15037 (opens in a new tab)
• perf(iceberg): decode Iceberg scans at the session batch_size by @bjchambers (opens in a new tab) in #15072 (opens in a new tab)
• fix(rate-control): admit waiting requests in arrival order at the charge in force (fixes #14912) by @krinart (opens in a new tab) in #14931 (opens in a new tab)
• fix(rate-control): compare shared-origin settings as they resolve (fixes #14914) by @krinart (opens in a new tab) in #14937 (opens in a new tab)
• fix(decisions): Fixes for /v1/decisions by @Jeadie (opens in a new tab) in #15040 (opens in a new tab)
• perf(cayenne): bulk-load a keyed table without per-row routing or a whole-table fold by @bjchambers (opens in a new tab) in #15076 (opens in a new tab)
• fix(cayenne): list a new snapshot's files once, when it is published by @bjchambers (opens in a new tab) in #15077 (opens in a new tab)
• fix(duckdb): unparse string literals with backslash-quote exactly by @lukekim (opens in a new tab) in #15083 (opens in a new tab)
• fix(vortex): keep casts Vortex cannot evaluate out of the scan by @phillipleblanc (opens in a new tab) in #15096 (opens in a new tab)
• feat(index): operation-aware CDC index maintenance via IndexChangeSet by @Jeadie (opens in a new tab) in #14340 (opens in a new tab)
• feat(iceberg): read a table root without a catalog, finding its current metadata on every query by @sgrebnov (opens in a new tab) in #15095 (opens in a new tab)
• fix(runtime): fail recovery of an unsaved running query by @phillipleblanc (opens in a new tab) in #15092 (opens in a new tab)
• perf(cayenne): build a keyed table's primary-key filter during its first load by @phillipleblanc (opens in a new tab) in #15099 (opens in a new tab)
• feat(cache): age SQL results on TTL instead of invalidating them on refresh by @bjchambers (opens in a new tab) in #15033 (opens in a new tab)
• fix(deps): estimate selective IN lists and OR-of-equalities from NDV by @bjchambers (opens in a new tab) in #14140 (opens in a new tab)
• fix(flightsql): prepared-statement GetFlightInfo schema matches the stream when parameters are bound by @lukekim (opens in a new tab) in #15106 (opens in a new tab)
• fix(telemetry): report the configured CPU count as host.cpu.count by @lukekim (opens in a new tab) in #15108 (opens in a new tab)
• fix(runtime-table): prepare every partition of a partitioned Cayenne table after a refresh by @bjchambers (opens in a new tab) in #15111 (opens in a new tab)
• feat(snapshots): store snapshots under runtime.state, built by the shared state-location builder by @sgrebnov (opens in a new tab) in #15117 (opens in a new tab)
• fix(federation): match NULL rows in pushed-down INTERSECT and EXCEPT (fixes #14941) by @sgrebnov (opens in a new tab) in #15110 (opens in a new tab)
• fix(models): allowed_tools shape, provider error statuses, decision response model (fixes #14910) by @krinart (opens in a new tab) in #14977 (opens in a new tab)
• feat(cayenne): let deletion_mode auto pick position deletes for small primary-key tables by @lukekim (opens in a new tab) in #15042 (opens in a new tab)
• fix(iceberg): keep nested child namespaces when loading an Iceberg catalog by @lukekim (opens in a new tab) in #15105 (opens in a new tab)
• fix(flight): authenticate before binding sessions, and bound TLS handshakes by @phillipleblanc (opens in a new tab) in #15128 (opens in a new tab)
• fix(models): return Spice tool calls in /v1/responses output so clients can replay them by @krinart (opens in a new tab) in #15126 (opens in a new tab)
• feat(runtime): check runtime.state at startup, and record its state format by @phillipleblanc (opens in a new tab) in #15138 (opens in a new tab)
• feat(acceleration): apply changed acceleration settings on restart by @lukekim (opens in a new tab) in #15094 (opens in a new tab)
• fix(models): enable Luna chat and NSQL support by @krinart (opens in a new tab) in #15166 (opens in a new tab)
• fix(acceleration): read every accelerator partition in a caching point lookup by @phillipleblanc (opens in a new tab) in #15177 (opens in a new tab)
• fix(cayenne): run snapshot-directory sweeps when a snapshot releases its file-deletion hold by @sgrebnov (opens in a new tab) in #15179 (opens in a new tab)
• feat(telemetry): export runtime logs over OTLP and log failed queries by level by @lukekim (opens in a new tab) in #15176 (opens in a new tab)
• fix(iceberg): find the committed Hadoop metadata version by @phillipleblanc (opens in a new tab) in #15142 (opens in a new tab)
• feat(iceberg): append refresh follows Iceberg snapshots so late-arriving rows are not missed by @sgrebnov (opens in a new tab) in #15147 (opens in a new tab)
• Drive accelerator refresh writes on the refresh runtime by @sgrebnov (opens in a new tab) in #15262 (opens in a new tab)
• fix(models): answer ai_* decisions from Bedrock models instead of failing them as refusals by @krinart (opens in a new tab) in #15270 (opens in a new tab)
• fix(cache): coalesce Cayenne cache replacements so caching mode keeps up with sustained misses by @krinart (opens in a new tab) in #15253 (opens in a new tab)
• feat(cluster): lend a distributed task's slot while it waits on an HTTP source by @peasee (opens in a new tab) in #15123 (opens in a new tab)
• fix(cayenne): evict a folded snapshot's cached listing once its superseded files are removed by @bjchambers (opens in a new tab) in #15303 (opens in a new tab)
• fix(cache): delete replaced Cayenne cache entries by primary key, not by scanning the table by @krinart (opens in a new tab) in #15312 (opens in a new tab)
• fix(cayenne): list snapshot directories under the listing fence by @bjchambers (opens in a new tab) in #15304 (opens in a new tab)
• fix(cayenne): checkpoint before a durable CDC write only when a source callback is installed by @phillipleblanc (opens in a new tab) in #15315 (opens in a new tab)
• fix(iceberg): read tables whose last commit is schema-only (ADD/RENAME COLUMN) by @lukekim (opens in a new tab) in #15334 (opens in a new tab)
• fix(runtime): skip the refresh_append_overlap warning for Iceberg append datasets by @sgrebnov (opens in a new tab) in #15340 (opens in a new tab)
• fix(spiced): raise the soft open file limit to the hard limit at startup by @bjchambers (opens in a new tab) in #15339 (opens in a new tab)
• fix(iceberg): keyed DuckDB/SQLite append refreshes read by time so late rows don't overwrite newer versions by @sgrebnov (opens in a new tab) in #15346 (opens in a new tab)
• fix(acceleration): restore the strict high-water mark in append dedupe by @sgrebnov (opens in a new tab) in #15358 (opens in a new tab)
• fix(observability): export a pinned query's execution log records under its trace id by @krinart (opens in a new tab) in #15363 (opens in a new tab)
• fix(caching): let an empty origin response replace a cached HTTP entry by @krinart (opens in a new tab) in #15365 (opens in a new tab)
• feat: one instance identity for every shared-state feature by @phillipleblanc (opens in a new tab) in #15135 (opens in a new tab)
• feat(snapshots): per-dataset snapshot settings; deprecate top-level snapshots and runtime.scheduler state settings by @phillipleblanc (opens in a new tab) in #15136 (opens in a new tab)
• fix(state): make every shared-state write in object storage conditional by @phillipleblanc (opens in a new tab) in #15137 (opens in a new tab)
• fix(mysql): read the shared binlog on a thread of its own by @lukekim (opens in a new tab) in #15181 (opens in a new tab)
• fix(datafusion): keep placeholder type inference linear over deep expressions by @lukekim (opens in a new tab) in #15187 (opens in a new tab)
• fix(sql): cast float literals to decimal through their exact spelling by @lukekim (opens in a new tab) in #15082 (opens in a new tab)
• fix(federation): keep a Spice-only function local without unfederating the plan around it (fixes #13770) by @krinart (opens in a new tab) in #15078 (opens in a new tab)
• fix(snapshots): count every failed snapshot in dataset_acceleration_snapshot_failure_count by @sgrebnov (opens in a new tab) in #15212 (opens in a new tab)
• feat(state): clean up unused shared state and keep 14 days of snapshots by @phillipleblanc (opens in a new tab) in #15199 (opens in a new tab)
• feat(state): add state-store request, conflict and cleanup-delete metrics by @phillipleblanc (opens in a new tab) in #15203 (opens in a new tab)
• fix(sql): keep float token spellings when planning refresh_sql by @lukekim (opens in a new tab) in #15235 (opens in a new tab)
• feat(state): one ownership primitive for scheduler membership, job ownership and snapshot writers by @phillipleblanc (opens in a new tab) in #15196 (opens in a new tab)
• fix(search): reject empty or whitespace-only text on /v1/search by @lukekim (opens in a new tab) in #15119 (opens in a new tab)
• feat(snapshots): keep a dataset's snapshots for 28 days by @phillipleblanc (opens in a new tab) in #15241 (opens in a new tab)
• fix(snapshots): release the snapshot writer lease on graceful shutdown by @sgrebnov (opens in a new tab) in #15256 (opens in a new tab)
• feat(monitoring): snapshot freshness, Cayenne segment cache and pod health panels on the Datadog dashboard by @sgrebnov (opens in a new tab) in #15309 (opens in a new tab)
• fix(observability): report snapshot reader refresh failures in metrics and label them mode=snapshot by @sgrebnov (opens in a new tab) in #15310 (opens in a new tab)
• fix(cayenne): leave the Turso metastore log out of snapshots by @phillipleblanc (opens in a new tab) in #15318 (opens in a new tab)
• fix(object_store_occ): never let a local version repeat an earlier version's ETag by @claudespice (opens in a new tab) in #15437 (opens in a new tab)
• fix(cayenne): complete, version, and validate a dataset's metastore slice (fixes #13241) by @claudespice (opens in a new tab) in #14238 (opens in a new tab)
• fix(acceleration): fail and retry a refresh whose source goes silent mid-stream by @sgrebnov (opens in a new tab) in #15457 (opens in a new tab)
• fix: emit one HELP/TYPE per derived Prometheus summary family by @lukekim (opens in a new tab) in #15416 (opens in a new tab)
• fix: serve append accelerations with snapshots enabled while the source is down by @sgrebnov (opens in a new tab) in #15463 (opens in a new tab)
• perf(cayenne): compact an append-only table's small files without stalling its writers by @lukekim (opens in a new tab) in #15227 (opens in a new tab)
• fix(cayenne): keep lookups fast under sustained replacement on tables without a primary key by @krinart (opens in a new tab) in #15362 (opens in a new tab)
• fix(snapshots): report a snapshot reader that cannot load a newer snapshot by @sgrebnov (opens in a new tab) in #15472 (opens in a new tab)
• fix(cayenne): rematerialize autocommit UPDATE inside write_lock by @lukekim (opens in a new tab) in #15490 (opens in a new tab)
• feat: deprecate non-Cayenne acceleration engines and the DuckDB vector engine by @bjchambers (opens in a new tab) in #15496 (opens in a new tab)
• fix(cayenne): count logical nulls so IS NULL keeps rows in inline and RAM-tier data by @lukekim (opens in a new tab) in #15186 (opens in a new tab)
• fix(cayenne): filter a scan view's rows against the deletions it captured by @lukekim (opens in a new tab) in #15169 (opens in a new tab)
• fix(snapshots): stop a restarted snapshot reader from serving a partial table by @sgrebnov (opens in a new tab) in #15536 (opens in a new tab)
• fix(cluster): keep executor shuffle fetches on cluster mTLS by @phillipleblanc (opens in a new tab) in #15484 (opens in a new tab)
• Add table-bound ChangeSink ownership and backends by @phillipleblanc (opens in a new tab) in #14704 (opens in a new tab)
• chore: Update Turso crate to 0.8.2 by @spiceemma (opens in a new tab) in #15022 (opens in a new tab)
• chore(deps): raise the datafusion-table-providers pin to include the NUMERIC result-column fix by @phillipleblanc (opens in a new tab) in #14254 (opens in a new tab)
• feat(key-index): immutable secondary index runs over compound Arrow keys by @bjchambers (opens in a new tab) in #14592 (opens in a new tab)
• feat(object_store_occ): add transactional WAL and MVCC snapshots by @lukekim (opens in a new tab) in #14732 (opens in a new tab)
• Route CDC refresh and Cayenne cache writes through the table-bound ChangeSink by @phillipleblanc (opens in a new tab) in #14788 (opens in a new tab)
• Reapply "fix(sqlite, duckdb): keep a decimal AVG, and on SQLite a decimal SUM, out of the federated plan (fixes #14492)" by @krinart (opens in a new tab) in #14888 (opens in a new tab)
• fix(duckdb): pull in the bundled Brotli decoder fix (CVE-2026-111497) by @phillipleblanc (opens in a new tab) in #15580 (opens in a new tab)

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.3.3...v2.4.0 (opens in a new tab)