date: 2026-10-22 title: 'Spice v2.4.0-rc.3 (Oct 22, 2026)' type: blog authors: [sgrebnov] tags: [release, performance, iceberg, cayenne]

Spice v2.4.0-rc.3 is now available! 🛠️

Spice v2.4.0-rc.3 is the third release candidate for v2.4.0, with mostly fixes since rc.2. By default, full and append refreshes now write on their own threads, not on the threads that serve queries. In a Cayenne test, P99 query latency at each refresh commit fell from 130–690 ms to the steady P99 of 44–50 ms. On Unix, the runtime also raises its soft open file limit at startup, up to the hard limit. Before, a few hundred client connections in a container could make Cayenne files fail to open.

Twelve other changes apply to Iceberg append refresh and table reads, Spice Cayenne, Bedrock decisions, OTLP logs, distributed queries, and TLS verification. Some apply only in specific setups. Two fix regressions from earlier v2.4.0 release candidates: Failed to open Vortex file errors after a Cayenne load, and failed MySQL CDC snapshots into Cayenne.

Testers of rc.2 should retest Iceberg append refresh, Cayenne refresh_mode: caching, and ai_* functions on Bedrock models. If you are new to v2.4.0, start with the rc.1 (opens in a new tab) and rc.2 (opens in a new tab) notes for its main features and breaking changes.

Highlights in this release candidate include:

• Lower query latency during refreshes (opens in a new tab): By default, full and append refreshes write to the accelerator on dedicated refresh threads. In a Cayenne test, P99 latency at each refresh commit fell from 130–690 ms to the steady P99 of 44–50 ms.
• Higher open file limit on Unix (opens in a new tab): The runtime raises its soft open file limit at startup, up to the hard limit. In a container with a soft limit of 1024 and a hard limit of 524288, the runtime raised its soft limit to 524288.

What's New in v2.4.0-rc.3

Lower Query Latency During Refreshes

Full and append refreshes now write to the accelerator on the dedicated refresh threads. Before this release, the write ran on the threads that also serve HTTP requests. For Cayenne, the write sorts, encodes, checks for duplicate keys, and publishes the data. A large write held those threads, and query latency rose until the write ended.

The change applies to these refresh writes, for all accelerators:

• refresh_mode: full, and the initial load
• refresh_mode: append
• refresh_mode: changes, only when it rebuilds the table from a snapshot

Steady CDC apply, the caching and snapshot refresh modes, and retention deletes do not change. With runtime.params.dedicated_thread_pool: disabled, there are no refresh threads, and refresh writes run as before.

In a test, Cayenne ran a full refresh every 5 minutes on a table of 2.2 million rows and about 8 GB. A client sent 500 point-lookup joins per second over HTTP. Before the change, P99 latency in the 15-second window of each refresh commit was 130–690 ms. After the change, it matched the steady P99 of 44–50 ms. The test did not cover append refreshes or other accelerators. See #15262 (opens in a new tab).

Higher Open File Limit on Unix

On Unix systems such as Linux and macOS, the runtime now raises its soft open file limit (RLIMIT_NOFILE) to the hard limit at startup. Client connections, acceleration files, and pipes all count against this one limit. Containers often start the runtime with a soft limit of 1024 and a hard limit of 524288. Before this release, a few hundred client connections could make Cayenne files fail to open. Queries then failed with Unable to open Cayenne acceleration file, or the client saw Connection reset by peer.

• Measured: On macOS with a soft limit of 256, v2.3.1 failed queries with 176 to 220 idle client connections open. A build with the fix raised the limit to 245760, the kern.maxfilesperproc value of that Mac. It answered queries with 600 connections open.
• Containers: In a Docker container with --ulimit nofile=1024:524288, the runtime raised its soft limit from 1024 to 524288.
• macOS: The runtime raises the limit to at most kern.maxfilesperproc.
• Logs: The runtime logs the result at DEBUG, for example Raised the open file limit from 1024 to 524288. The runtime also logs a failed raise only at DEBUG. Start the runtime with -v or set SPICED_LOG to see it.
• Windows: Nothing changes.

If the hard limit is also low, the runtime cannot raise the soft limit above it. For this case, the Cayenne error now names the effective limit and says that client connections count against it. It also keeps the Too many open files (os error 24) text from the operating system:

Failed to open Cayenne acceleration file '…/01a15b2f5c697c01b2a8acdede2d637d_p000_00011.vortex' because the process reached its open file limit of 256, which client connections and all other open files also count against. Raise the limit where spiced is started (`ulimit -n`, or the container's `nofile` limit), or write fewer, larger acceleration files with `cayenne_target_file_size_mb`. See: https://spiceai.org/docs/components/data-accelerators/cayenne#params. Cause: … Too many open files (os error 24)

Failed to open Cayenne acceleration file '…/01a15b2f5c697c01b2a8acdede2d637d_p000_00011.vortex' because the process reached its open file limit of 256, which client connections and all other open files also count against. Raise the limit where spiced is started (`ulimit -n`, or the container's `nofile` limit), or write fewer, larger acceleration files with `cayenne_target_file_size_mb`. See: https://spiceai.org/docs/components/data-accelerators/cayenne#params. Cause: … Too many open files (os error 24)

See #15339 (opens in a new tab).

Iceberg Append Refresh Follows Table Snapshots

An Iceberg dataset with refresh_mode: append and a time_column now loads the rows that each new commit adds, whatever their time values. This applies to Cayenne datasets and to datasets without a primary key. It covers Iceberg tables read through the Iceberg (opens in a new tab) or Glue (opens in a new tab) data connectors. Glue tables in Parquet, CSV, or ORC format still read by time.

Before this release, an append refresh read only the rows with a time_column value later than the stored maximum minus refresh_append_overlap. A writer that commits out of time order can put rows outside that window. A sink with several shards, one of them behind, is an example. Those rows did not load until a full refresh.

The refresh now saves the Iceberg snapshot that it loaded last. The next refresh reads the data files that later snapshots added. An unchanged table scans no data. This change adds no setting:

datasets:
  - from: iceberg:https://iceberg-catalog.example.com/v1/namespaces/sales/tables/orders
    name: orders
    time_column: updated_at
    acceleration:
      enabled: true
      engine: cayenne
      mode: file
      primary_key: order_id
      refresh_mode: append

datasets:
  - from: iceberg:https://iceberg-catalog.example.com/v1/namespaces/sales/tables/orders
    name: orders
    time_column: updated_at
    acceleration:
      enabled: true
      engine: cayenne
      mode: file
      primary_key: order_id
      refresh_mode: append

• Newest version by time: With a primary key, Cayenne keeps the newest version of each key by time_column. A late row with an older time does not replace a newer stored row.
• Skipped commits: The refresh skips compactions and commits that add no data file. An append refresh never removes a row.
• Fallback to reads by time: If the runtime cannot follow the snapshots, it reads by time for one refresh. This happens after an overwrite on a dataset without a primary key, or after an expired snapshot in the interval. A rollback, a branch switch, or a recreated table has the same effect.
• After a fallback: A refresh that reads by time does not load rows older than refresh_append_overlap. A full refresh loads them. The runtime logs a warning when the reason changes. The new dataset_acceleration_refresh_fallbacks metric counts each fallback by reason.
• Saved position: File-mode accelerations store the last loaded snapshot with the acceleration data. Other accelerations keep it in memory. A full refresh saves the current snapshot, so the next append refresh continues from it.
• Computed columns: A dataset that computes columns over the Iceberg table, such as embedding columns, still reads by time.
• Overlap: refresh_append_overlap now applies only to the first load and to refreshes that read by time. Datasets that follow snapshots no longer log the startup warning that asks for refresh_append_overlap.

With a primary key, a dataset on DuckDB, SQLite, or another accelerator that keeps the last row to arrive still reads by time. This matches rc.2. A primary key that includes the time_column is the exception: such a dataset follows snapshots. On those accelerators, a late row with an older time would replace the newer stored row. If such a dataset has no refresh_append_overlap, the runtime logs this warning when it loads the dataset:

Dataset 'acc' (`engine: duckdb`) has no `refresh_append_overlap`, so a row that arrives after a newer one is never loaded. Use `engine: cayenne` to load every new row, or set `refresh_append_overlap` to how late rows can arrive. See: https://spiceai.org/docs/features/data-acceleration/data-refresh

Dataset 'acc' (`engine: duckdb`) has no `refresh_append_overlap`, so a row that arrives after a newer one is never loaded. Use `engine: cayenne` to load every new row, or set `refresh_append_overlap` to how late rows can arrive. See: https://spiceai.org/docs/features/data-acceleration/data-refresh

To load every late row on such a dataset, use engine: cayenne, or set refresh_append_overlap to how late rows can arrive. Append refresh on other data sources does not change. See append refresh (opens in a new tab), #15147 (opens in a new tab), and #15346 (opens in a new tab).

Spice Cayenne Bug Fixes

• Stale file lists (rare): In rare cases, Cayenne kept a stale file list for a dataset in mode: file. Queries then missed the rows of an appended file with no error. After retention removed a file, every query failed with Failed to open Vortex file … not found instead. Later appends did not repair the list, but a restart did.

  The fault needed a secondary-index build or some queries to list the files while retention or an append changed them. Tests reproduced it only with a delayed file listing. No spiced run has shown it so far. Cayenne now lists files under the same lock that its writers take. See #15304 (opens in a new tab).

• Vortex file errors after a load: Some keyed Cayenne datasets on network block storage no longer fail every query and refresh with Failed to open Vortex file … not found. This was a regression in earlier v2.4.0 release candidates. The fault needed all of these conditions:

  • mode: file, a primary_key, and key deletion mode (cayenne_deletion_mode: key, or auto when it uses key deletes)
  • a first load or a full refresh with more than one version of some keys
  • network block storage: storage_profile: ebs, or auto when it resolves to EBS or NAS

  Cayenne rewrote the files that held older versions and deleted the originals, but a cached file list still named the deleted files. The data on disk was correct, and a restart cleared the error. Tests reproduced the bug on rc.2. Cayenne now clears the cached list after it deletes the files. See #15303 (opens in a new tab).

• MySQL CDC snapshots (rc.1 regression): A MySQL dataset with refresh_mode: changes no longer fails its initial snapshot on large tables when a Cayenne file acceleration has maintained aggregates. The error was Bootstrap snapshot error: snapshot row read: … bytes remaining on stream, and the dataset stayed in the Error state.

  Since rc.1, Cayenne ran a checkpoint before each snapshot batch. The checkpoint could rebuild the maintained aggregates with a scan of the whole table. The MySQL reader then waited longer than net_write_timeout, and MySQL closed the connection. Cayenne now runs this checkpoint only when it can hold CDC rows in memory, as before rc.1. In a test with 40 million rows and net_write_timeout=2, the snapshot completed in 36–44 seconds. See #15315 (opens in a new tab).

Cayenne Caching Mode Fixes

These fixes apply to Cayenne datasets with refresh_mode: caching (opens in a new tab).

• Removed rows served from the cache: An empty origin response now replaces the cached entry of a GET request. Before this release, Cayenne refused that write when the origin started to answer [] for a cached request. Each refusal logged An empty cache replacement requires the complete logical request key. The query that received [] still returned no rows. Later queries could return the rows that the origin had removed:

  • With caching_stale_if_error: enabled, the next origin failure served the removed rows.
  • With caching_stale_while_revalidate_ttl, lookups served the removed rows for the rest of that window. Each lookup started another refresh that Cayenne refused.

  An empty response now replaces only the entry of its own request. Cayenne still refuses an empty response when the lookup filters do not name one request, and it logs An empty cache replacement requires filters that name one request. Entries for a POST request or another query string under the same path stay. DuckDB, SQLite, and Arrow caching datasets still have this fault, because their writer drops empty responses. To get this fix, use engine: cayenne. See #15365 (opens in a new tab).

• Dropped cache writes under load: Cayenne no longer drops most cache writes above about 100 cache misses per second. Before this release, Cayenne applied each replacement of a cached entry as a separate write. Under load, Cayenne fell behind. The runtime then dropped most writes with Cache write admission timed out. The cache held only part of what it served, so caching_stale_if_error had little data to fall back on. Cayenne now combines replacements and new entries into one write when their requests cannot overlap.

  In a test, a client sent 500 lookups per second over 2.5 million random keys for 300 seconds. Cayenne used mode: file, caching_ttl: 60s, and caching_stale_if_error: enabled. Two runs of each build gave these results:

  Measure	Before	After
  Cache writes dropped	121,957–122,508	0
  Keys stored	15,915–16,624	143,356–143,488
  Request P99	361–363 ms	9 ms

  In a second test, the client cached 60,000 keys, and then the origin stopped. Before the change, 2,932 of 5,001 repeated lookups failed. After the change, 0 of 5,000 failed.

  Cayenne also deletes a replaced entry by its primary key when the request names every primary-key column and the new response has rows. Before, each delete read every stored row, so its cost grew with the cache. In a 60-minute test, the share of CPU for these deletes during origin recovery fell from 3.6% to 0.2%.

  In that phase, P99 latency rose from 303 ms to 433 ms, and throughput fell from 487 to 459 requests per second. The extra writes added compaction work. The test limited the runtime to 8 CPUs and 4 GiB. See #15253 (opens in a new tab) and #15312 (opens in a new tab).

Iceberg Bug Fixes

• Old or uncommitted versions of Hadoop tables: Spice now reads the committed version of a Hadoop-layout Iceberg table. Before this release, a stale version-hint.text kept SQL queries and the Iceberg REST API on an older snapshot. Without a hint, Spice could read the uncommitted metadata file of a writer as the current table. The fix applies to a table read from its root directory, a dataset pinned to version-hint.text, and an Iceberg catalog over a Hadoop warehouse.

  A Hadoop writer commits a version when it renames <uuid>.metadata.json to v<N>.metadata.json. It updates version-hint.text after that, on a best-effort basis. Spice now treats the hint as a starting point and checks v<N+1>, v<N+2>, and later versions, as Iceberg Java does. A metadata file that a bare UUID names no longer counts as a version. See #15142 (opens in a new tab).

• Tables whose last commit changed only the schema: Queries on an Iceberg table no longer fail when its last commit was ADD COLUMN or RENAME COLUMN with no write after it. Such queries failed with DataInvalid => Column value not found in table. A renamed column did not resolve, an added column was hidden, and a dropped column stayed visible. An Arrow acceleration of the table never loaded, so /v1/ready returned 503.

  A schema-only commit creates no snapshot, and the scan used the schema of the last snapshot. Scans now use the current schema of the table. A scan pinned with snapshot_id keeps the schema of its snapshot. v2.3.2, rc.1, and rc.2 had this bug. A column that someone adds while the runtime runs stays hidden until a restart. Distributed scans pinned to a snapshot still use the schema of that snapshot. See #15334 (opens in a new tab).

AI and Observability Bug Fixes

• Decisions on Bedrock models: ai_if, ai_probability, ai_classify, ai_score, and /v1/decisions now return answers from Bedrock (opens in a new tab) models. Before this release, each such request failed with Evaluation model '<name>' failed: the model refused to answer: . rc.1 and rc.2 had this bug.

  The Bedrock adapter set an empty refusal on every non-streaming reply, and the evaluator read it as a refusal. The adapter no longer sets it, so /v1/chat/completions replies from Bedrock models no longer carry "refusal": "". The evaluator now ignores an empty refusal beside an answer.

  This also covers an OpenAI-compatible model whose upstream is an rc.1 or rc.2 runtime that serves Bedrock. A refusal with text still fails the request. ai_decide uses the same evaluator, but the tests for this fix did not cover it. See #15270 (opens in a new tab).

• OTLP log trace IDs for pinned queries: A query that a client pins to a trace ID with spice-trace-id or traceparent now exports all of its OTLP log records under that ID. Before this release, records that the runtime logged while the query plan ran carried another trace ID. That ID matched neither the spice-trace-id response header nor the runtime.task_history row. A backend filter on the trace ID missed those records.

  Chat, NSQL, Responses, and decisions requests with a pinned trace ID use the same code path. The tests for this fix covered SQL queries only. This bug did not affect queries without a pinned trace ID. OTLP log export first shipped in rc.2. See observability (opens in a new tab) and #15363 (opens in a new tab).

Distributed Queries Lend Task Slots During HTTP Waits

In a distributed query (opens in a new tab), a task that waits on a slow HTTP response no longer holds its executor task slot for the whole wait. The task lends the slot to other queries until the response arrives. It then takes a slot back before it continues. Before this release, a few HTTP requests to a slow origin could hold every slot, and other queries waited behind them.

This change needs no configuration. Other data sources and non-distributed queries do not change. In a test, slow HTTP requests filled every executor slot, and a fast query still finished while those requests waited. See #15123 (opens in a new tab).

TLS Certificate Verification

The runtime now uses rustls-webpki 0.103.16 and version 1.0.10 of the webpki-roots and webpki-root-certs root certificate bundles. The bundles follow the current root store of Mozilla. rustls-webpki 0.103.16 rejects a certificate chain that passes through a CA key that upstream no longer trusts. This also applies to cross-signed copies of that key.

The check applies wherever rustls verifies a chain: the bundled roots, rustls-native-certs, and rustls-platform-verifier on Linux. A connection to a server with such a chain now fails certificate verification. Connections through OpenSSL or native-tls use the trust store of the operating system and do not change. See #15374 (opens in a new tab).

Dependency Updates

The upstream versions of DataFusion, Arrow, Iceberg, and Ballista do not change.

Upgrading

An upgrade from rc.2 needs no configuration change. TLS connections that rustls verifies now fail if the server's certificate chain passes through a CA that Mozilla no longer trusts. See TLS Certificate Verification (opens in a new tab).

To upgrade to v2.4.0-rc.3 once the release artifacts are available, use one of the following methods:

CLI:

spice upgrade v2.4.0-rc.3

spice upgrade v2.4.0-rc.3

Homebrew:

brew upgrade spiceai/spiceai/spice

brew upgrade spiceai/spiceai/spice

Homebrew follows the version published in the tap. Use the explicit CLI version or Docker tag to select this release candidate.

Docker:

Pull the spiceai/spiceai:2.4.0-rc.3 image:

docker pull spiceai/spiceai:2.4.0-rc.3

docker pull spiceai/spiceai:2.4.0-rc.3

For available tags, see DockerHub (opens in a new tab).

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.4.0-rc.3

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.4.0-rc.3

AWS Marketplace:

Spice is available in the AWS Marketplace (opens in a new tab). Marketplace availability follows its published versions.

Cookbook Updates

• New Decisions API recipe (opens in a new tab): call /v1/decisions with TypeSafe Jev and GPT-6 Luna.
• The PostgreSQL CDC recipe (opens in a new tab) now accelerates with Cayenne.

The Spice Cookbook (opens in a new tab) provides recipes to help you get started with Spice.

Contributors

• @bjchambers (opens in a new tab)
• @Jeadie (opens in a new tab)
• @krinart (opens in a new tab)
• @lukekim (opens in a new tab)
• @peasee (opens in a new tab)
• @phillipleblanc (opens in a new tab)
• @sgrebnov (opens in a new tab)

What's Changed

Changelog

• fix(iceberg): find the committed Hadoop metadata version by @phillipleblanc (opens in a new tab) in #15142 (opens in a new tab)
• feat(iceberg): append refresh follows Iceberg snapshots so late-arriving rows are not missed by @sgrebnov (opens in a new tab) in #15147 (opens in a new tab)
• Drive accelerator refresh writes on the refresh runtime by @sgrebnov (opens in a new tab) in #15262 (opens in a new tab)
• fix(models): answer ai_* decisions from Bedrock models instead of failing them as refusals by @krinart (opens in a new tab) in #15270 (opens in a new tab)
• fix(cache): coalesce Cayenne cache replacements so caching mode keeps up with sustained misses by @krinart (opens in a new tab) in #15253 (opens in a new tab)
• feat(cluster): lend a distributed task's slot while it waits on an HTTP source by @peasee (opens in a new tab) in #15123 (opens in a new tab)
• fix(cayenne): evict a folded snapshot's cached listing once its superseded files are removed by @bjchambers (opens in a new tab) in #15303 (opens in a new tab)
• fix(cache): delete replaced Cayenne cache entries by primary key, not by scanning the table by @krinart (opens in a new tab) in #15312 (opens in a new tab)
• fix(cayenne): list snapshot directories under the listing fence by @bjchambers (opens in a new tab) in #15304 (opens in a new tab)
• fix(cayenne): checkpoint before a durable CDC write only when a source callback is installed by @phillipleblanc (opens in a new tab) in #15315 (opens in a new tab)
• fix(iceberg): read tables whose last commit is schema-only (ADD/RENAME COLUMN) by @lukekim (opens in a new tab) in #15334 (opens in a new tab)
• fix(runtime): skip the refresh_append_overlap warning for Iceberg append datasets by @sgrebnov (opens in a new tab) in #15340 (opens in a new tab)
• fix(spiced): raise the soft open file limit to the hard limit at startup by @bjchambers (opens in a new tab) in #15339 (opens in a new tab)
• fix(iceberg): keyed DuckDB/SQLite append refreshes read by time so late rows don't overwrite newer versions by @sgrebnov (opens in a new tab) in #15346 (opens in a new tab)
• fix(acceleration): restore the strict high-water mark in append dedupe by @sgrebnov (opens in a new tab) in #15358 (opens in a new tab)
• fix(deps): bump rustls-webpki to 0.103.16 and webpki-roots to 1.0.10 by @phillipleblanc (opens in a new tab) in #15374 (opens in a new tab)
• fix(observability): export a pinned query's execution log records under its trace id by @krinart (opens in a new tab) in #15363 (opens in a new tab)
• fix(caching): let an empty origin response replace a cached HTTP entry by @krinart (opens in a new tab) in #15365 (opens in a new tab)

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.4.0-rc.2...v2.4.0-rc.3 (opens in a new tab)