date: 2025-03-27 title: 'Spice v1.0.7 (Mar 26, 2025)' type: blog authors: [phillipleblanc] tags: [release, duckdb, arrow, dremio]
Announcing the release of Spice v1.0.7 🏎️
Spice v1.0.7 improves memory usage when using DuckDB, improves schema inference performance when using object-store based data connectors, and fixes a bug in Dremio schema inference.
Highlights in v1.0.7
-
DuckDB Memory Usage: Memory usage when using DuckDB has been significantly improved for data loads and refreshes through expanded use of zero-copy Arrow and multi-threading for data loads. When a
duckdb_memory_limitis specified, disk spilling has been improved for greater-than-memory workloads. In addition, a newtemp_directoryruntime parameter supports storing temporary files to alternative location than the DuckDB data file for higher throughput. For example,temp_directorycould be set to a different high-IOPs IO2 EBS volume that is separate from theduckdb_file_path.Automated end-to-end tests for the DuckDB Accelerator coverage has been significantly expanded.
For configuration details, see the documentation for runtime parameters and the DuckDB Data Accelerator.
-
Schema Inference Performance for Object-Store Data Connectors: Schema inference performance has been improved, especially for large numbers of objects (1M+ objects) when using object-store based data connectors by making the object-listing and selection more efficient.
Performance
When compared to previous versions, Spice v1.0.7 loads DuckDB accelerated datasets significantly faster. When using the TPCH lineitem dataset at Scale Factor 100 (600M rows):
Without Indexes
5x faster, 28% less memory usage.
| Version | Load Time | Peak Memory Usage |
|---|---|---|
| v1.0.6 | 16m 3s | 32GB |
| v1.0.7 | 3m 149ms | 24.4GB |
With Indexes
2.5x faster. Higher memory usage in v1.0.7 is due to better resource utilization to achieve faster load times. Use the duckdb_memory_limit parameter to control memory usage.
| Version | Load Time | Peak Memory Usage |
|---|---|---|
| v1.0.6 | 27m 9s | 50GB |
| v1.0.7 | 11m 30s | 77GB |
Documentation
- DuckDB Data Accelerator: Has been expanded with additional resource usage guidance.
- Memory: A new section for memory considerations has been added to the Reference section.
Contributors
- @phillipleblanc
- @sgrebnov
- @peasee
- @Sevenannn
Breaking Changes
No breaking changes.
Upgrading
To upgrade to v1.0.7, use one of the following methods:
CLI:
Homebrew:
Docker:
Pull the spiceai/spiceai:1.0.7 image:
For available tags, see DockerHub.
Helm:
What's Changed
Dependencies
- DataFusion Table Providers: Upgraded from
760ece6ac52b7d180d697f347642af403c2e711cto9ba9dce19a1fdbd5e22cc2e445c5b3ea731944b4.
Changelog
Full Changelog: github.com/spiceai/spiceai/compare/v1.0.6...v1.0.7