date: 2025-12-09 title: 'Spice v1.10.0 (Dec 9, 2025)' type: blog authors: [peasee] tags: [release, caching, acceleration, dynamodb, cayenne, opentelemetry, data-connector, distributed-query, performance]

Announcing the release of Spice v1.10.0! ⚡

Spice v1.10.0 introduces a new Caching Acceleration Mode with stale-while-revalidate (SWR) semantics for disk-persisted, low-latency queries with background refresh. This release also adds the TinyLFU eviction policy for the SQL results cache, a preview of the DynamoDB Streams connector for real-time CDC, S3 location predicate pruning for faster partitioned queries, improved distributed query execution, and multiple security hardening improvements.

What's New in v1.10.0

Caching Acceleration Mode

Low-Latency Queries with Background Refresh: This release introduces a new caching acceleration mode (opens in a new tab) that implements the stale-while-revalidate (SWR) pattern. Queries return cached results immediately while data refreshes asynchronously in the background, eliminating query latency spikes during refresh cycles. Cached data persists to disk using DuckDB, SQLite, or Cayenne file modes.

Key Features:

• Stale-While-Revalidate (SWR): Returns cached data immediately while refreshing in the background, reducing query latency
• Disk Persistence: Cached results persist across restarts using DuckDB, SQLite, or Cayenne file modes
• Configurable Refresh: Control refresh intervals with refresh_check_interval to balance freshness and source load

Recommendation: Use retention configuration (opens in a new tab) with caching acceleration to ensure stale data is cleaned up over time.

Example spicepod.yaml configuration:

datasets:
  - from: http://localhost:7400
    name: cached_data
    time_column: fetched_at
    acceleration:
      enabled: true
      engine: duckdb
      mode: file # Persist cache to disk
      refresh_mode: caching
      refresh_check_interval: 10m
      retention_check_enabled: true
      retention_period: 24h
      retention_check_interval: 1h

datasets:
  - from: http://localhost:7400
    name: cached_data
    time_column: fetched_at
    acceleration:
      enabled: true
      engine: duckdb
      mode: file # Persist cache to disk
      refresh_mode: caching
      refresh_check_interval: 10m
      retention_check_enabled: true
      retention_period: 24h
      retention_check_interval: 1h

For more details, refer to the Data Acceleration Documentation (opens in a new tab).

TinyLFU Cache Eviction Policy

Higher Cache Hit Rates for SQL Results Cache: A new TinyLFU cache eviction policy (opens in a new tab) is now available for the SQL results cache. TinyLFU is a probabilistic cache admission policy that maintains higher hit rates than LRU while keeping memory usage predictable, making it ideal for workloads with varying query frequency patterns.

Example spicepod.yaml configuration:

runtime:
  caching:
    sql_results:
      enabled: true
      eviction_policy: tiny_lfu # default: lru

runtime:
  caching:
    sql_results:
      enabled: true
      eviction_policy: tiny_lfu # default: lru

For more details, refer to the Caching Documentation (opens in a new tab) and the Moka TinyLFU Documentation (opens in a new tab) for details of the algorithm.

DynamoDB Streams Data Connector (Preview)

Real-Time Change Data Capture for DynamoDB: The DynamoDB connector now integrates with DynamoDB Streams for real-time change data capture (CDC). This enables continuous synchronization of DynamoDB table changes into Spice for real-time query, search, and LLM-inference.

Key Features:

• Real-Time CDC: Automatically captures inserts, updates, and deletes from DynamoDB tables as they occur
• Table Bootstrapping: Performs an initial full table scan before streaming changes, ensuring complete data consistency
• Acceleration Integration: Works with refresh_mode: changes to incrementally update accelerated datasets

Note: DynamoDB Streams must be enabled on your DynamoDB table. This feature is in preview.

Example spicepod.yaml configuration:

datasets:
  - from: dynamodb:my_table
    name: orders_stream
    acceleration:
      enabled: true
      refresh_mode: changes # Enable Streams capture

datasets:
  - from: dynamodb:my_table
    name: orders_stream
    acceleration:
      enabled: true
      refresh_mode: changes # Enable Streams capture

For more details, refer to the DynamoDB Connector Documentation (opens in a new tab).

OpenTelemetry Metrics Exporter

Spice can now push metrics to an OpenTelemetry (opens in a new tab) collector, enabling integration with platforms such as Jaeger (opens in a new tab), New Relic (opens in a new tab), Honeycomb (opens in a new tab), and other OpenTelemetry-compatible backends.

Key Features:

• Protocol Support: Supports the gRPC (default port 4317) protocol
• Configurable Push Interval: Control how frequently metrics are pushed to the collector

Example spicepod.yaml configuration for gRPC:

runtime:
  telemetry:
    enabled: true
    otel_exporter:
      endpoint: 'localhost:4317'
      push_interval: '30s'

runtime:
  telemetry:
    enabled: true
    otel_exporter:
      endpoint: 'localhost:4317'
      push_interval: '30s'

For more details, refer to the Observability & Monitoring Documentation (opens in a new tab).

S3 Connector Improvements

S3 Location Predicate Pruning: The S3 data connector (opens in a new tab) now supports location-based predicate pruning, dramatically reducing data scanned by pushing down location filter predicates to S3 listing operations. For partitioned datasets (e.g., year=2025/month=12/), Spice now skips listing irrelevant partitions entirely, significantly reducing query latency and S3 API costs.

AWS S3 Tables Write Support: Full read/write capability for AWS S3 Tables (opens in a new tab), enabling direct integration with AWS's managed table format for S3. Use standard SQL INSERT INTO to write data.

For more details, refer to the S3 Data Connector Documentation (opens in a new tab) and Glue Data Connector Documentation (opens in a new tab).

Faster Distributed Query Execution

Distributed query (opens in a new tab) planning and execution have been significantly improved:

• Fixed executor registration in cluster mode for more reliable distributed deployments
• Improved hostname resolution for Flight server binding, enabling better executor discovery
• Distributed accelerator registration: Data accelerators now properly register in distributed mode
• Optimized query planning: DistributeFileScanOptimizer improvements for faster planning with large datasets

For more details, refer to the Distributed Query Documentation (opens in a new tab).

Search Improvements

Search (opens in a new tab) capabilities have been improved with several performance and reliability enhancements:

• Fixed FTS query blocking: Full-text search queries no longer block unnecessarily, improving query responsiveness
• Optimized vector index operations: Eliminated unnecessary list_vectors calls for better performance
• Improved limit pushdown: IndexerExec now properly handles limit pushdown for more efficient searches

For more details, refer to the Search Documentation (opens in a new tab).

Security Hardening

Multiple security improvements have been implemented:

• SQL Identifier Quoting: Hardened SQL identifier quoting across all database connectors (PostgreSQL, MySQL, DuckDB, etc.) to prevent SQL injection attacks through table or column names
• Token Redaction: Sensitive authentication tokens are now fully redacted in debug and error output, preventing accidental credential exposure in logs
• Path Traversal Prevention: Fixed tar extraction operations to prevent directory traversal vulnerabilities when processing archived files
• Input Sanitization: Added strict validation for top_n_sample order_by clause parsing to prevent injection attacks
• Glue Credential Handling: Prevented automatic loading of AWS credentials from environment in Glue connector, ensuring explicit credential configuration

Developer Experience Improvements

• Health probe metrics: Added health probe latency metrics for better observability
• CLI improvements: Fixed .clear history command in the REPL to fully clear persisted history

Contributors

• @lukekim (opens in a new tab)
• @sgrebnov (opens in a new tab)
• @peasee (opens in a new tab)
• @krinart (opens in a new tab)
• @Jeadie (opens in a new tab)
• @mach-kernel (opens in a new tab)
• @phillipleblanc (opens in a new tab)
• @ewgenius (opens in a new tab)

Breaking Changes

No breaking changes.

Cookbook Updates

No major cookbook updates.

The Spice Cookbook (opens in a new tab) includes 82 recipes to help you get started with Spice quickly and easily.

Upgrading

To upgrade to v1.10.0, use one of the following methods:

CLI:

spice upgrade

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.10.0 image:

docker pull spiceai/spiceai:1.10.0

docker pull spiceai/spiceai:1.10.0

For available tags, see DockerHub (opens in a new tab).

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

helm repo update
helm upgrade spiceai spiceai/spiceai

AWS Marketplace:

🎉 Spice is now available in the AWS Marketplace (opens in a new tab)!

What's Changed

Changelog

• Test-operator: Add tpcds_q8 to the default row-count validation skip list by @sgrebnov (opens in a new tab) in #8185 (opens in a new tab)
• fix: Remove unwrap_used from test by @peasee (opens in a new tab) in #8212 (opens in a new tab)
• Run glue_iceberg_integration_test_catalog as part of main integration tests by @sgrebnov (opens in a new tab) in #8222 (opens in a new tab)
• Add TPCH sf100 testoperator spicepods with dispatch by @Jeadie (opens in a new tab) in #8192 (opens in a new tab)
• Build with CPU native flags by @lukekim (opens in a new tab) in #8224 (opens in a new tab)
• fix: Apply assertion clippy in CI/Makefile only by @peasee (opens in a new tab) in #8229 (opens in a new tab)
• feat: Support running queries only in testoperator by @peasee (opens in a new tab) in #8211 (opens in a new tab)
• DuckDB query planning: aggregate pushdown by @mach-kernel (opens in a new tab) in #8174 (opens in a new tab)
• install.sh improvements by @lukekim (opens in a new tab) in #8252 (opens in a new tab)
• Fix .clear history by @lukekim (opens in a new tab) in #8254 (opens in a new tab)
• Improve the output of dataset loading by @lukekim (opens in a new tab) in #8256 (opens in a new tab)
• Refactor view validation by @lukekim (opens in a new tab) in #8258 (opens in a new tab)
• Upgrade AWS crates by @lukekim (opens in a new tab) in #8259 (opens in a new tab)
• fix: Pushdown dynamic filters to partition scans by @peasee (opens in a new tab) in #8240 (opens in a new tab)
• Harden SQL identifier quoting in connectors by @phillipleblanc (opens in a new tab) in #8276 (opens in a new tab)
• Cayenne sort_columns on insert by @lukekim (opens in a new tab) in #8091 (opens in a new tab)
• Redact token debug output by @phillipleblanc (opens in a new tab) in #8280 (opens in a new tab)
• fix: Cayenne configuration options by @lukekim (opens in a new tab) in #8281 (opens in a new tab)
• Prevent path traversal in untar by @phillipleblanc (opens in a new tab) in #8284 (opens in a new tab)
• Fix cluster mode executor registration by @mach-kernel (opens in a new tab) in #8292 (opens in a new tab)
• Unignore s3_vectors_kafka_stream test by @Jeadie (opens in a new tab) in #8289 (opens in a new tab)
• Post-release house keeping by @krinart (opens in a new tab) in #8293 (opens in a new tab)
• Improve generate_changelog script by @krinart (opens in a new tab) in #8273 (opens in a new tab)
• Acceleration mode caching by @lukekim (opens in a new tab) in #8237 (opens in a new tab)
• Sanitization and security checks by @lukekim (opens in a new tab) in #7854 (opens in a new tab)
• Add health probe latency metric by @phillipleblanc (opens in a new tab) in #8300 (opens in a new tab)
• Add distributed registration for data accelerators by @phillipleblanc (opens in a new tab) in #8299 (opens in a new tab)
• Pass IndexedTableProvider down in 'changes_stream' and 'append_stream' by @Jeadie (opens in a new tab) in #8295 (opens in a new tab)
• Add dynamodb-streams crate by @krinart (opens in a new tab) in #8283 (opens in a new tab)
• Distributed query: resolve executor hostname when determining Flight server binding by @mach-kernel (opens in a new tab) in #8304 (opens in a new tab)
• Return computed embeddings from index for partitioned S3Vectors by @Jeadie (opens in a new tab) in #8306 (opens in a new tab)
• [DDB Streams] Skeleton for DynamoDB Streams by @krinart (opens in a new tab) in #8296 (opens in a new tab)
• DistributeFileScanOptimizer: Improve planning performance by @mach-kernel (opens in a new tab) in #8305 (opens in a new tab)
• feat: Add an ExactLeftAccumulator implementation by @peasee (opens in a new tab) in #8302 (opens in a new tab)
• deps: Upgrade Vortex to 0.56 by @peasee (opens in a new tab) in #8311 (opens in a new tab)
• DynamoDB table bootstrapping + streaming by @krinart (opens in a new tab) in #8312 (opens in a new tab)
• Avoid calling S3Vector list_vectors (or equivalent) when indexing into VectorIndexs by @Jeadie (opens in a new tab) in #8282 (opens in a new tab)
• Add on_conflict testing support to append benchmark by @sgrebnov (opens in a new tab) in #8314 (opens in a new tab)
• docker: Add valid home directory to fix duckdb extension loading issue by @phillipleblanc (opens in a new tab) in #8318 (opens in a new tab)
• Add GH Workflow to run Append benchmark test by @sgrebnov (opens in a new tab) in #8321 (opens in a new tab)
• Exclude MySQL SF100 from test-operator dispatch by @sgrebnov (opens in a new tab) in #8320 (opens in a new tab)
• feat: Update clippy lints by @peasee (opens in a new tab) in #8317 (opens in a new tab)
• Add S3 location predicate pruning to listing connector by @phillipleblanc (opens in a new tab) in #8319 (opens in a new tab)
• Review feedback for caching mode accelerator by @phillipleblanc (opens in a new tab) in #8326 (opens in a new tab)
• Also include Dockerfile home changes for release build by @phillipleblanc (opens in a new tab) in #8327 (opens in a new tab)
• Change communication channel from Discord to Slack by @Jeadie (opens in a new tab) in #8330 (opens in a new tab)
• Replace Discord link with Slack link in README by @Jeadie (opens in a new tab) in #8331 (opens in a new tab)
• fix(glue): Prevent OpenDAL from automatic loading of AWS credentials from environment by @sgrebnov (opens in a new tab) in #8337 (opens in a new tab)
• Block on index read for FTS queries by @Jeadie (opens in a new tab) in #8339 (opens in a new tab)
• Fix search query provider by @Jeadie (opens in a new tab) in #8343 (opens in a new tab)
• Support for writing into AWS S3 Tables by @sgrebnov (opens in a new tab) in #8344 (opens in a new tab)
• Acceleration file_create mode by @lukekim (opens in a new tab) in #8347 (opens in a new tab)
• Don't block on lock in FTS query path by @Jeadie (opens in a new tab) in #8348 (opens in a new tab)
• feat: Add an optimizer rule to replace join accumulator for Cayenne by @peasee (opens in a new tab) in #8316 (opens in a new tab)
• S3 Vectors limit updates by @lukekim (opens in a new tab) in #8352 (opens in a new tab)
• Sanitize top_n_sample order_by parsing by @phillipleblanc (opens in a new tab) in #8356 (opens in a new tab)
• Add distributed registration for data connectors by @phillipleblanc (opens in a new tab) in #8354 (opens in a new tab)
• Improve IndexerExec to properly handle limit pushdown by @sgrebnov (opens in a new tab) in #8366 (opens in a new tab)
• Fix Cayenne partition_by metadata flaky integration test by @phillipleblanc (opens in a new tab) in #8367 (opens in a new tab)
• Rework caching accelerator to use the stale-while-revalidate pattern. by @phillipleblanc (opens in a new tab) in #8365 (opens in a new tab)
• Add TinyLFU caching policy by @lukekim (opens in a new tab) in #8370 (opens in a new tab)
• Make Arrow acceleration on_conflict verification more robust by @sgrebnov (opens in a new tab) in #8375 (opens in a new tab)
• Add additional test for verify_on_conflict_matches_primary_key (Arrow acceleration) by @sgrebnov (opens in a new tab) in #8376 (opens in a new tab)
• Add v1.10.0-rc1 release notes by @mach-kernel (opens in a new tab) in #8373 (opens in a new tab)
• docs: Remove DuckDB agg pushdown from release notes by @peasee (opens in a new tab) in #8383 (opens in a new tab)
• Testoperator dispatch: add Append support and test configurations by @sgrebnov (opens in a new tab) in #8360 (opens in a new tab)
• fix: Increase TPCDS DuckDB connection pool size by @peasee (opens in a new tab) in #8386 (opens in a new tab)
• fix: Update benchmark snapshots by @app/github-actions (opens in a new tab) in #8385 (opens in a new tab)
• fix: Update benchmark snapshots by @app/github-actions (opens in a new tab) in #8389 (opens in a new tab)
• Fix Windows build by @phillipleblanc (opens in a new tab) in #8391 (opens in a new tab)
• DuckDB aggregate pushdown: fix partitioning and schema rewrite bugs by @mach-kernel (opens in a new tab) in #8397 (opens in a new tab)
• Delta table: Store current snapshot ref with table instance by @mach-kernel (opens in a new tab) in #8358 (opens in a new tab)
• GetAppDefinition: Check if executor is part of cluster by @mach-kernel (opens in a new tab) in #8396 (opens in a new tab)
• 1.10.0-rc1 housekeeping by @mach-kernel (opens in a new tab) in #8394 (opens in a new tab)
• Change debug log to warning for vector engine config by @Jeadie (opens in a new tab) in #8378 (opens in a new tab)
• Clarify /v1/nsql datasets sampling hint by @phillipleblanc (opens in a new tab) in #8395 (opens in a new tab)
• Use bmi1 target feature for x86_64 by @phillipleblanc (opens in a new tab) in #8401 (opens in a new tab)
• benchmarks: Default to update snapshots when run on a non-release branch by @phillipleblanc (opens in a new tab) in #8402 (opens in a new tab)
• Update threat model for v1.9.2 by @phillipleblanc (opens in a new tab) in #8400 (opens in a new tab)
• Fix iceberg tables metadata - assign ids to all fields, including nested by @ewgenius (opens in a new tab) in #8351 (opens in a new tab)
• Fix databricks_spark_connect_m2m_integration_test_catalog snapshot by @ewgenius (opens in a new tab) in #8403 (opens in a new tab)
• Move all GitHub Actions workflows to use exact commit sha by @phillipleblanc (opens in a new tab) in #8409 (opens in a new tab)
• Upgrade datafusion-tableproviders (df v50) by @lukekim (opens in a new tab) in #8261 (opens in a new tab)
• Batching for CDC by @krinart (opens in a new tab) in #8359 (opens in a new tab)
• fix: make DuckDB attachments logic more robust by @sgrebnov (opens in a new tab) in #8411 (opens in a new tab)
• Persistent checkpoints for DynamoDB Streams by @krinart (opens in a new tab) in #8345 (opens in a new tab)
• Distributed query: Support AsyncFuncExec and Spice UDFs in Ballista by @mach-kernel (opens in a new tab) in #8414 (opens in a new tab)
• Pin GitHub Actions to fix Testoperator build action by @sgrebnov (opens in a new tab) in #8416 (opens in a new tab)
• Watermarks support for DynamoDB by @krinart (opens in a new tab) in #8417 (opens in a new tab)
• Fix typo in .vscode/launch.json by @sgrebnov (opens in a new tab) in #8415 (opens in a new tab)
• DuckSqlExec: Update equivalence properties when rewriting schema by @phillipleblanc (opens in a new tab) in #8420 (opens in a new tab)
• Validate that the commit for datafusion-table-providers exists on the spiceai branch by @phillipleblanc (opens in a new tab) in #8421 (opens in a new tab)
• Append Tests: add support for retention testing by @sgrebnov (opens in a new tab) in #8419 (opens in a new tab)
• New crate google-genai by @Jeadie (opens in a new tab) in #8390 (opens in a new tab)
• federation: Improve error message and add debug logging for cast failures by @phillipleblanc (opens in a new tab) in #8422 (opens in a new tab)
• DynamoDB Streams Error Handling by @krinart (opens in a new tab) in #8418 (opens in a new tab)
• Append tests: add support for with_retention_data to dispatch by @sgrebnov (opens in a new tab) in #8430 (opens in a new tab)
• Append tests: add support for test metrics reporting by @sgrebnov (opens in a new tab) in #8432 (opens in a new tab)
• test-operator: fix metrics reporting by @sgrebnov (opens in a new tab) in #8435 (opens in a new tab)
• Follow-up improvements and bug fixes by @krinart (opens in a new tab) in #8433 (opens in a new tab)
• Periodic snapshots for append/changes streams by @krinart (opens in a new tab) in #8407 (opens in a new tab)
• Add support for caching_stale_if_error to caching accelerator; fix multiple upstream requests during SWR; fix Arrow accelerator by @phillipleblanc (opens in a new tab) in #8425 (opens in a new tab)
• Disable dataset health monitor for dynamic HTTP connector by @phillipleblanc (opens in a new tab) in #8441 (opens in a new tab)
• Metrics + snapshots_trigger_threshold for DynamoDB Streams by @krinart (opens in a new tab) in #8437 (opens in a new tab)
• Clear the in-flight revalidations cache after a revalidation has completed. by @phillipleblanc (opens in a new tab) in #8443 (opens in a new tab)
• dont build 'spicepod-validator' on 'make install' by @Jeadie (opens in a new tab) in #8426 (opens in a new tab)
• fix: Update benchmark snapshots by @app/github-actions (opens in a new tab) in #8436 (opens in a new tab)
• fix: Disable Cayenne HashJoin rewriter optimizer by @peasee (opens in a new tab) in #8439 (opens in a new tab)
• Testoperator: add duckdb-partitioned query override by @sgrebnov (opens in a new tab) in #8446 (opens in a new tab)
• Add a check to validate that results cache SWR and caching accelerator SWR are not both set. by @phillipleblanc (opens in a new tab) in #8445 (opens in a new tab)
• OTel exporter for push metrics by @lukekim (opens in a new tab) in #8442 (opens in a new tab)
• fix: Update benchmark snapshots by @app/github-actions (opens in a new tab) in #8448 (opens in a new tab)
• Add snapshot creation logging by @krinart (opens in a new tab) in #8469 (opens in a new tab)
• Fix PeriodicReader panic by @krinart (opens in a new tab) in #8471 (opens in a new tab)
• fix: Pin CUDA build actions to commits by @peasee (opens in a new tab) in #8477 (opens in a new tab)
• DuckDB agg pushdown: gate behind accelerator parameter by @mach-kernel (opens in a new tab) in #8474 (opens in a new tab)
• Rename aggregate_pushdown_optimization -> optimizer_duckdb_aggregate_pushdown by @ewgenius (opens in a new tab) in #8485 (opens in a new tab)
• Handle throttling exception for DynamoDB streams by @phillipleblanc (opens in a new tab) in #8492 (opens in a new tab)