How Spice AI responded to the eMudhra certificate compromise

TL;DR: The private keys of eight eMudhra intermediate CA certificates can be recovered from the certificates themselves, so anyone can issue a trusted TLS certificate for any domain under them. Spice v2.3.3, Spice v2.4.0-rc.3, and Spice.ai Enterprise v2.2.2 and v1.11.7 stop trusting those intermediates, in the spiced binary and in the Docker images. The Spice Cloud Stable channel now runs v2.3.3. Upgrade Spice and update your operating system's CA certificates. Our tests show the OS update is the half you cannot skip.

What this post covers:

• What happened, and why server-side TLS clients were exposed when browsers were not
• How Spice decides which certificates to trust, and what our tests found
• What we shipped, and when
• What to upgrade on Spice Cloud, Spice.ai OSS, Spice.ai Enterprise, and the SDKs
• How to verify a deployment

What happened

On October 17, 2026, a researcher showed the browser root programs that the private keys of eight eMudhra intermediate CA certificates were compromised. The proof was a message signed with each of the eight keys. eMudhra generated the keys incorrectly, so anyone can recover them from the public certificates. The flaw dates back to when the intermediates were created in April 2025.

An attacker with one of those keys can issue a certificate for any hostname, and any client that trusts the intermediate accepts it. Cloudflare's write-up covers the incident and its own response. A crowdsourced tracker lists patched packages and runs test hosts that serve certificates under the compromised intermediates.

Browsers were not the exposed population. Firefox, Chrome, Edge, and Safari enforce Certificate Transparency (CT), and the intermediates have no CT log entries for that period, so a forged certificate would not have passed. The root programs also pushed the block through their revocation channels (OneCRL, CRLite, and CRLSets): Firefox and Chrome on October 17, Apple on October 19, and Microsoft on October 20.

Server-side TLS clients are a different story. Most do not enforce CT or consume browser revocation lists: they trust whatever their trust store contains until a software update changes it. Spice is one of those clients.

Why it matters for Spice

Spice is a TLS client for most of the work it does. It federates queries to databases, warehouses, object stores, and HTTP sources, and it calls model providers. Those connections carry credentials and query results.

An attacker who can intercept traffic between Spice and one of those endpoints can present a certificate for the endpoint's hostname, issued with one of the compromised keys. An unpatched Spice accepts it, and the attacker can then read or change everything on the connection, credentials included. The key half of that attack is now available to anyone. The network position is the only remaining barrier, and it is not one to rely on.

We confirmed the exposure directly. Spice v2.3.2, in the published spiceai/spiceai:2.3.2 image, connected to the tracker's test hosts and registered datasets from them. eMudhra has revoked the intermediates, and that made no difference: Spice does not check certificate revocation lists (CRLs) today. CRL support is on the roadmap for Spice v2.5.0.

How Spice decides what to trust

Spice verifies a server certificate against one of two trust stores. Which one applies depends on the TLS library behind the connection:

• Built-in roots. Connections that rustls verifies against the bundled webpki-roots use root certificates compiled into spiced. Changing them takes a new release.
• The operating system's trust store. Connections through OpenSSL or native-tls, and rustls connections that load the system certificates, use the OS CA bundle (ca-certificates on Debian and Ubuntu). Changing it takes an OS package update or an image rebuild.

The Spice Docker image is built FROM scratch, with its CA bundle copied from a debian:trixie-slim build stage. The image's trust store is whatever Debian shipped on the day of the build.

To see which store protects a real connection, we pointed the HTTPS data connector at the tracker's test hosts on Debian 13. We crossed the old and new spiced binaries with the old and new ca-certificates package:

The OS trust store decides for this connector. The new binary rejects any chain through the eight compromised keys, but only on connections that rustls verifies. The HTTPS connector verifies against the OS trust store instead, so v2.4.0-rc.3 on an unpatched image still connected.

Spice's dependency tree also includes rustls clients built with the bundled roots, so a complete fix has to update both stores. That is why every release below changes the binary and the image together.

What we shipped

Each release makes the same three changes:

• rustls-webpki 0.103.13 to 0.103.16. Rejects any certificate chain that passes through one of the eight compromised intermediate keys, including the Sectigo cross-signed copies of the same keys. (v2.4.0-rc.3 moves from 0.103.15.)
• webpki-roots and webpki-root-certs 1.0.7 to 1.0.10. Removes the emSign ECC Root CA - G3 root from the built-in root bundle.
• ca-certificates 20250419 to 20250419+deb13u1 in the Docker images. Debian's security update, which distrusts the emSign ECC Root CA - G3 and Comodo AAA Services root certificates.

The patch releases need no configuration changes and have no breaking changes.

Spice Cloud runs the same fix, at two levels:

• The runtimes Spice Cloud deploys. The Stable channel, the default for Spice Cloud projects, now runs Spice v2.3.3.
• The Spice Cloud Platform itself. We updated the whole platform, including the Spice Cloud portal and the API service. The API service moved to rustls-webpki 0.103.16, so it rejects any chain through the compromised intermediates, whatever its CA bundle contains. That rollout reached every production region and dedicated cluster by October 22 at 03:22 UTC. Our infrastructure deployment pipelines also moved to Node.js 24.22.1, which carries the refreshed CA store.

Timeline

All dates are in 2026, UTC:

• October 17: The researcher proves the compromise to the root programs. Firefox and Chrome block the intermediates.
• October 19: Downstream trust stores receive the disclosure. Apple blocks the intermediates. webpki-roots 1.0.10 ships without emSign ECC Root CA - G3.
• October 20: rustls-webpki 0.103.16 ships with the distrusted keys. Debian publishes ca-certificates 20250419+deb13u1. Microsoft blocks the intermediates, and Cloudflare publishes its write-up.
• October 21: The fix merges to Spice trunk.
• October 21 to 22: Spice v2.3.3, Spice.ai Enterprise v2.2.2 and v1.11.7, and Spice v2.4.0-rc.3 ship.
• October 22: The API service fix reaches every Spice Cloud production region and dedicated cluster. The Spice Cloud Stable channel moves to Spice v2.3.3.

What to do

Spice Cloud

Each deployment uses the latest runtime version on the project's update channel. A project on the Stable channel picks up v2.3.3 on its next deployment. To move now, open the project's Deployments tab and select Create Deployment.

Spice.ai OSS

Upgrade to v2.3.3 with the method you installed with:

spice upgrade
docker pull spiceai/spiceai:2.3.3
helm repo update && helm upgrade spiceai spiceai/spiceai --version 2.3.3

On the v2.4 pre-release line, move to v2.4.0-rc.3 or later.

Spice.ai Enterprise

Pull the patched image for your release line from the AWS Marketplace registry:

• v2.3: 2.3.3-enterprise or 2.3.3-enterprise-jemalloc
• v2.2: 2.2.2-enterprise
• v1.11: 1.11.7-enterprise or 1.11.7-enterprise-models

v2.3.3 keeps the default allocator of the v2.3 line, the system allocator. To run jemalloc, the default in v2.4, use the -jemalloc image. The AWS Marketplace guide has the registry paths and Helm charts.

Your own images and hosts

• A Docker image you build yourself: rebuild it so it picks up ca-certificates 20250419+deb13u1 (or your distribution's equivalent), and build it on a patched spiced. The rebuild fixes connections that use the OS trust store. Only the new binary fixes the built-in roots.
• Spice outside Docker: upgrade spiced and update the OS ca-certificates package. The tracker lists the patched package version for each major distribution.

SDKs

The SDKs are TLS clients too. What to update depends on the trust store each SDK uses:

• Rust (spice-rs): the SDK verifies with the bundled webpki-roots. Its trunk now requires 1.0.10, and the next release includes that requirement. Until then, run cargo update -p webpki-roots -p rustls-webpki in your application.
• Python (spicepy): the SDK verifies Flight connections against certifi unless you pass tls_root_cert. Upgrade certifi to 2026.10.19 or later with pip install --upgrade "certifi>=2026.10.19".
• Go, Java, .NET, and JavaScript: update the language runtime and the OS CA certificates. The tracker lists the patched versions of Go, Java, and Node.js.

The trade-off is deliberate: a server that still uses one of these intermediates fails verification until it gets a new certificate. Fix the server's certificate, not the client. Disabling certificate verification to restore the connection removes the protection for every certificate, not only eMudhra's.

Verify a deployment

Add a probe dataset that points at the tracker's test hosts. Run it on a test runtime with the same image and host as production:

version: v1
kind: Spicepod
name: emudhra-check
datasets:
  - from: https://dv-g3b-emudhra.we-have-a-problem.com/probe.csv
    name: emudhra_probe
    params:
      file_format: csv
  - from: https://dv-g3b-aaa-emudhra.we-have-a-problem.com/probe.csv
    name: emudhra_aaa_probe
    params:
      file_format: csv

The first host chains the EM DV TLS CA - G3B intermediate to the emSign roots. The second chains the same intermediate to the AAA Certificate Services root through its cross-sign.

A patched runtime logs Cannot connect to the dataset emudhra_probe and never registers the dataset. That log line ends in error sending request and does not mention certificates. An unpatched runtime logs Dataset emudhra_probe registered: it trusted the compromised chain.

This check covers the HTTPS connector, which verifies against the OS trust store. It does not exercise the built-in roots, which only the binary upgrade changes.