date: 2025-03-27
title: 'Spice v1.0.7 (Mar 26, 2025)'
type: blog
authors: [phillipleblanc]
tags: [release, duckdb, arrow, dremio]
Announcing the release of Spice v1.0.7 🏎️
Spice v1.0.7 improves memory usage when using DuckDB, improves schema inference performance when using object-store based data connectors, and fixes a bug in Dremio schema inference.
Highlights in v1.0.7
-
DuckDB Memory Usage: Memory usage when using DuckDB has been significantly improved for data loads and refreshes through expanded use of zero-copy Arrow and multi-threading for data loads. When a duckdb_memory_limit is specified, disk spilling has been improved for greater-than-memory workloads. In addition, a new temp_directory runtime parameter supports storing temporary files to alternative location than the DuckDB data file for higher throughput. For example, temp_directory could be set to a different high-IOPs IO2 EBS volume that is separate from the duckdb_file_path.
Automated end-to-end tests for the DuckDB Accelerator coverage has been significantly expanded (opens in a new tab).
For configuration details, see the documentation for runtime parameters (opens in a new tab) and the DuckDB Data Accelerator (opens in a new tab).
-
Schema Inference Performance for Object-Store Data Connectors: Schema inference performance has been improved, especially for large numbers of objects (1M+ objects) when using object-store based data connectors by making the object-listing and selection more efficient.
Performance
When compared to previous versions, Spice v1.0.7 loads DuckDB accelerated datasets significantly faster. When using the TPCH lineitem dataset at Scale Factor 100 (600M rows):
Without Indexes
5x faster, 28% less memory usage.
| Version | Load Time | Peak Memory Usage |
|---|
| v1.0.6 | 16m 3s | 32GB |
| v1.0.7 | 3m 149ms | 24.4GB |
With Indexes
2.5x faster. Higher memory usage in v1.0.7 is due to better resource utilization to achieve faster load times. Use the duckdb_memory_limit parameter to control memory usage.
| Version | Load Time | Peak Memory Usage |
|---|
| v1.0.6 | 27m 9s | 50GB |
| v1.0.7 | 11m 30s | 77GB |
Documentation
Contributors
- @phillipleblanc
- @sgrebnov
- @peasee
- @Sevenannn
Breaking Changes
No breaking changes.
Upgrading
To upgrade to v1.0.7, use one of the following methods:
CLI:
Homebrew:
brew upgrade spiceai/spiceai/spice
brew upgrade spiceai/spiceai/spice
Docker:
Pull the spiceai/spiceai:1.0.7 image:
docker pull spiceai/spiceai:1.0.7
docker pull spiceai/spiceai:1.0.7
For available tags, see DockerHub (opens in a new tab).
Helm:
helm repo update
helm upgrade spiceai spiceai/spiceai
helm repo update
helm upgrade spiceai spiceai/spiceai
What's Changed
Dependencies
Changelog
- fix: Remove on zero results arguments from benchmarks by @peasee in https://github.com/spiceai/spiceai/pull/4533
- Run benchmark tests w/o uploading test results (pending improvements) by @sgrebnov in https://github.com/spiceai/spiceai/pull/4843
- fix: Return BAD_REQUEST when not embeddings are configured by @peasee in https://github.com/spiceai/spiceai/pull/4804
- Fix Dremio schema inference by @phillipleblanc in https://github.com/spiceai/spiceai/pull/5114
- Improve performance of schema inference for object-store data connectors by @sgrebnov in https://github.com/spiceai/spiceai/pull/5124
- Always download spice runtime version matched with spice cli version by @Sevenannn in https://github.com/spiceai/spiceai/pull/4761
- Fix go lint errors by @sgrebnov in https://github.com/spiceai/spiceai/pull/5147
- Make DuckDB acceleration E2E tests more comprehensive by @sgrebnov in https://github.com/spiceai/spiceai/pull/5146
- Enable Spice to load larger than memory datasets into DuckDB accelerations by @phillipleblanc in https://github.com/spiceai/spiceai/pull/5149
- Add `temp_directory` runtime parameter and insert it for DuckDB accelerations by @phillipleblanc in https://github.com/spiceai/spiceai/pull/5152
- Fix Postgres and MySQL installation on macos14-runner (E2E CI) by @sgrebnov in https://github.com/spiceai/spiceai/pull/5155
- Enable E2E for DuckDB full mode acceleration with indexes only in CI by @sgrebnov in https://github.com/spiceai/spiceai/pull/5154
- fix: Remove on zero results arguments from benchmarks by @peasee in https://github.com/spiceai/spiceai/pull/4533
- Run benchmark tests w/o uploading test results (pending improvements) by @sgrebnov in https://github.com/spiceai/spiceai/pull/4843
- fix: Return BAD_REQUEST when not embeddings are configured by @peasee in https://github.com/spiceai/spiceai/pull/4804
- Fix Dremio schema inference by @phillipleblanc in https://github.com/spiceai/spiceai/pull/5114
- Improve performance of schema inference for object-store data connectors by @sgrebnov in https://github.com/spiceai/spiceai/pull/5124
- Always download spice runtime version matched with spice cli version by @Sevenannn in https://github.com/spiceai/spiceai/pull/4761
- Fix go lint errors by @sgrebnov in https://github.com/spiceai/spiceai/pull/5147
- Make DuckDB acceleration E2E tests more comprehensive by @sgrebnov in https://github.com/spiceai/spiceai/pull/5146
- Enable Spice to load larger than memory datasets into DuckDB accelerations by @phillipleblanc in https://github.com/spiceai/spiceai/pull/5149
- Add `temp_directory` runtime parameter and insert it for DuckDB accelerations by @phillipleblanc in https://github.com/spiceai/spiceai/pull/5152
- Fix Postgres and MySQL installation on macos14-runner (E2E CI) by @sgrebnov in https://github.com/spiceai/spiceai/pull/5155
- Enable E2E for DuckDB full mode acceleration with indexes only in CI by @sgrebnov in https://github.com/spiceai/spiceai/pull/5154
Full Changelog: github.com/spiceai/spiceai/compare/v1.0.6...v1.0.7 (opens in a new tab)
