date: 2026-04-10
title: 'Spice v2.0-rc.2 (Apr 10, 2026)'
type: blog
authors: [ewgenius]
tags: [release, distributed-query, cayenne, datafusion, delta-lake, databricks, adbc, data-connector]
Announcing the release of Spice v2.0-rc.2! 🔥
v2.0.0-rc.2 is the second release candidate for advanced testing of v2.0, building on v2.0.0-rc.1 (opens in a new tab).
Highlights in this release candidate include:
- Distributed Spice Cayenne Query and Write Improvements with data-local query routing and partition-aware write-through
- DataFusion v52.4.0 Upgrade with aligned
arrow-rs, datafusion-federation, and datafusion-table-providers
- MERGE INTO for Spice Cayenne catalog tables with distributed support across executors
PARTITION BY Support for Cayenne enabling SQL-defined partitioning in CREATE TABLE statements- ADBC Data Connector & Catalog with full query federation, BigQuery support, and schema/table discovery
- Databricks Lakehouse Federation Improvements with improved reliability, resilience, DESCRIBE TABLE fallback, and source-native type parsing
- Delta Lake Column Mapping supporting Name and Id mapping modes
- HTTP Pagination support for paginated API endpoints in the HTTP data connector
- New Catalog Connectors for PostgreSQL, MySQL, MSSQL, and Snowflake
- JSON Ingestion Improvements with single-object support,
soda (Socrata Open Data) format support, json_pointer extraction, and auto-detection
- Per-Model Rate-Limited AI UDF Execution for controlling concurrent AI function invocations
- Dependency upgrades including Turso v0.5.3, iceberg-rust v0.9, and Vortex improvements
What's New in v2.0.0-rc.2
Distributed Cayenne Query and Write Improvements
Distributed query (opens in a new tab) for Cayenne (opens in a new tab)-backed tables now has better partition awareness for both reads and writes.
Key improvements:
- Data-Local Query Routing: Cayenne catalog queries can now be routed to executors that hold the relevant partitions, improving distributed query efficiency.
- Partition-Aware Write Through: Scheduler-side Flight
DoPut ingestion now splits partitioned Cayenne writes and forwards them to the responsible executors instead of routing through a single raw-forward path.
- Dynamic Partition Assignment: Newly observed partitions can be added and assigned atomically as data arrives, with persisted partition metadata for future routing.
- Better Cluster Coordination: Partition management is now separated for accelerated and federated tables, improving routing behavior for distributed Cayenne catalog workloads.
- Distributed UPDATE/DELETE DML: UPDATE and DELETE statements for Cayenne catalog tables are now forwarded to all executors in distributed mode, with all executors required to succeed.
- Distributed
runtime.task_history: Task history is now replicated across the distributed cluster for observability.
- RefreshDataset Control Stream: Dataset refresh operations are now distributed via the control stream to executors.
- Executor DDL Sync: When an executor connects, it receives DDL for all existing tables, ensuring late-joining executors have full table state.
MERGE INTO for Spice Cayenne
Spice now supports MERGE INTO statements for Cayenne (opens in a new tab) catalog tables, enabling upsert-style data operations with full distributed support.
Key improvements:
- MERGE INTO Support: Execute
MERGE INTO statements against Cayenne catalog tables for combined insert/update/delete operations.
- Distributed MERGE: MERGE operations are automatically distributed across executors in cluster mode.
- Data Safety: Duplicate source keys are detected and prevented to avoid data loss during MERGE operations.
- Chunked Delete Filters: Large MERGE delete filter lists are chunked to prevent stack overflow with Vortex IN-list expressions.
PARTITION BY Support for Cayenne
SQL Partition Management: Spice now supports PARTITION BY for Cayenne (opens in a new tab)-backed CREATE TABLE statements, enabling partition definitions to be expressed directly in SQL and persisted in the Cayenne catalog.
Key improvements:
- SQL Partition Definition: Define Cayenne table partitioning directly in SQL using
CREATE TABLE ... PARTITION BY (...).
- Partition Validation: Partition expressions are parsed and validated during DDL analysis before table creation.
- Persisted Partition Metadata: Partition metadata is stored in the Cayenne catalog and can be reloaded by the runtime after restart.
- Distributed DDL Support: Partition metadata is forwarded when
CREATE TABLE is distributed to executors in cluster mode.
- Improved Type Support: Partition utilities now support newer string scalar variants such as
Utf8View.
Example:
CREATE TABLE events (id INT, region TEXT, ts TIMESTAMP) PARTITION BY (region)
CREATE TABLE events (id INT, region TEXT, ts TIMESTAMP) PARTITION BY (region)
Catalog Connector Enhancements
Spice now includes additional catalog connectors for major database systems, improving schema discovery and federation workflows across external data systems.
Key improvements:
Example PostgreSQL catalog configuration:
catalogs:
- from: pg
name: pg
include:
- 'public.*'
params:
pg_host: localhost
pg_port: 5432
pg_user: postgres
pg_pass: ${secrets:POSTGRES_PASSWORD}
pg_db: my_database
pg_sslmode: disable
catalogs:
- from: pg
name: pg
include:
- 'public.*'
params:
pg_host: localhost
pg_port: 5432
pg_user: postgres
pg_pass: ${secrets:POSTGRES_PASSWORD}
pg_db: my_database
pg_sslmode: disable
JSON Ingestion Improvements
JSON ingestion is now more flexible and robust.
Key improvements:
- More JSON Formats: Added support for single-object JSON documents, auto-detected JSON formats, and Socrata SODA responses.
json_pointer Extraction: Extract nested payloads before schema inference and reading using RFC 6901 JSON Pointer syntax.- Better Auto-Detection: JSON format detection now handles arrays, objects, JSONL, and BOM-prefixed input more reliably, including single multi-line objects.
- SODA Support: Added schema extraction and data conversion for Socrata Open Data API responses.
- Broader Compatibility: Improved handling for BOM-prefixed files, CRLF-delimited JSONL, nested payloads, mixed structures, and wrapped documents.
Example using json_pointer to extract nested data from an API response:
datasets:
- from: https://api.example.com/v1/data
name: users
params:
json_pointer: /data/users
datasets:
- from: https://api.example.com/v1/data
name: users
params:
json_pointer: /data/users
DataFusion v52.4.0 Upgrade
Apache DataFusion (opens in a new tab) has been upgraded from v52.2.0 to v52.4.0, with aligned updates across arrow-rs, datafusion-federation, and datafusion-table-providers.
Key improvements:
- DataFusion v52.4.0: Brings the latest fixes and compatibility improvements across query planning and execution.
- Strict Overflow Handling:
try_cast_to now uses strict cast to return errors on overflow instead of silently producing NULL values.
- Federation Fix: Fixed SQL unparsing for Inexact filter pushdown with aliases.
- Partial Aggregation Optimization: Improved partial aggregation performance for
FlightSQLExec.
Dependency Upgrades
| Dependency | Version / Update |
|---|
| Turso (libsql) | v0.5.3 (from v0.4.4) |
| iceberg-rust | v0.9 |
| Vortex | Map type support, stack-safe IN-lists |
| arrow-rs | Arrow v57.2.0 |
| datafusion-federation | Updated for DataFusion v52.4.0 alignment |
| datafusion-table-providers | Updated for DataFusion v52.4.0 alignment |
| datafusion-ballista | Bumped to fix BatchCoalescer schema mismatch panic |
Other Improvements
-
Cayenne released as RC: Cayenne (opens in a new tab) data accelerator is now promoted to release candidate status.
-
File Update Acceleration Mode: Added mode: file_update acceleration mode for file-based data refresh.
-
spice completions Command: New CLI command for generating shell completion scripts, with auto-detection of shell directory.
-
--endpoint Flag: Added --endpoint flag to spice run with scheme-based routing for custom endpoints.
-
mTLS Client Auth: Added mTLS client authentication support to the spice sql REPL.
-
DynamoDB DML: Implemented DML (INSERT, UPDATE, DELETE) support for the DynamoDB (opens in a new tab) table provider.
-
Caching Retention: Added retention policies for cached query results.
-
GraphQL Custom Auth Headers: Added custom authorization header support for the GraphQL (opens in a new tab) connector.
-
ClickHouse Date32 Support: Added Date32 type support for the ClickHouse (opens in a new tab) connector.
-
AWS IAM Role Source: Added iam_role_source parameter for fine-grained AWS credential configuration.
-
S3 Metadata Columns: Metadata columns renamed to _location, _last_modified, _size for consistency, with more robust handling in projected queries.
-
S3 URL Style: Added s3_url_style parameter for S3 connector (opens in a new tab) URL addressing (path-style vs virtual-hosted). Useful for S3-compatible stores like MinIO:
params:
s3_endpoint: https://minio.local:9000
s3_url_style: path
params:
s3_endpoint: https://minio.local:9000
s3_url_style: path
-
S3 Parquet Performance: Improved S3 parquet read performance.
-
HTTP Caching: Transient HTTP (opens in a new tab) error responses such as 429 and 5xx are no longer cached, preventing stale error payloads from being served from cache.
-
HTTP Connector Metadata: Added response_headers as structured map data for HTTP (opens in a new tab) datasets.
-
Views on_zero_results: Accelerated views now support on_zero_results: use_source to fall back to the source when no results are found:
views:
- name: sales_summary
sql: |
SELECT region, SUM(amount) as total
FROM sales
GROUP BY region
acceleration:
enabled: true
on_zero_results: use_source
views:
- name: sales_summary
sql: |
SELECT region, SUM(amount) as total
FROM sales
GROUP BY region
acceleration:
enabled: true
on_zero_results: use_source
-
Flight DoPut Ingestion Metrics: Added rows_written and bytes_written metrics for Flight DoPut (opens in a new tab) / ADBC ETL ingestion.
-
EXPLAIN ANALYZE Metrics: Added metrics for EXPLAIN ANALYZE in FlightSQLExec.
-
Scheduler Executor Metrics: Added scheduler_active_executors_count metric for monitoring active executors.
-
Query Memory Limit: Updated default query memory limit from 70% to 90%, with GreedyMemoryPool for improved memory management.
-
MetastoreTransaction Support: Added transaction support to prevent concurrent metastore transaction conflicts.
-
Iceberg REST Catalog: Coerce unsupported Arrow types to Iceberg v2 equivalents in the REST catalog API.
-
CDC Cache Invalidation: Improved cache invalidation for CDC-backed datasets.
-
Spice.ai Connector Alignment: Parameter names aligned across catalog and data connectors for Spice.ai Cloud.
-
Cayenne File Size: Cayenne now correctly respects the configured target file size (defaults to 128MB).
-
Cayenne Primary Keys: Properly set primary_keys/on_conflict for Cayenne tables.
-
Turso Metastore Performance: Cached metastore connections and prepared statements for improved Turso (opens in a new tab) and SQLite (opens in a new tab) metastore performance.
-
Turso SQL Robustness: More robust SQL unparsing and date comparison handling for Turso.
-
Dictionary Type Normalization: Normalize Arrow Dictionary types for DuckDB and SQLite acceleration.
-
GitHub Connector Resilience: Improved GraphQL client resilience, performance, and ref filter handling.
-
ODBC Fix: Fixed ODBC queries silently returning 0 rows on query failure.
-
Anthropic Fixes: Fixed compatibility issues with Anthropic model provider.
-
v1/responses API Fix: The /v1/responses API now correctly preserves client instructions when system_prompt is set.
-
Shared Acceleration Snapshots: Show an error when snapshots are enabled on a shared acceleration file.
-
Distributed Mode Error Handling: Improved error handling for distributed mode and state_location configuration.
-
Helm Chart: Added support for ServiceAccount annotations and AWS IRSA example.
-
Perplexity Removed: Removed Perplexity model provider support.
-
Rust v1.93.1: Upgraded Rust toolchain to v1.93.1.
Contributors
Breaking Changes
- S3 metadata columns renamed: S3 metadata columns renamed from
location, last_modified, size to _location, _last_modified, _size.
v1/evals API removed: The /v1/evals endpoint has been removed.- Perplexity removed: Perplexity model provider support has been removed.
- Default query memory limit changed: Default query memory limit increased from 70% to 90%.
Upgrading
To upgrade to v2.0.0-rc.2, use one of the following methods:
CLI:
spice upgrade v2.0.0-rc.2
spice upgrade v2.0.0-rc.2
Homebrew:
brew upgrade spiceai/spiceai/spice
brew upgrade spiceai/spiceai/spice
Docker:
Pull the spiceai/spiceai:2.0.0-rc.2 image:
docker pull spiceai/spiceai:2.0.0-rc.2
docker pull spiceai/spiceai:2.0.0-rc.2
For available tags, see DockerHub (opens in a new tab).
Helm:
helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.0.0-rc.2
helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.0.0-rc.2
AWS Marketplace:
Spice is available in the AWS Marketplace (opens in a new tab).
What's Changed
Changelog
- ci: fix E2E CLI upgrade test to use latest release for spiced download by @phillipleblanc (opens in a new tab) in #9613 (opens in a new tab)
- fix(DF): Lazily initialize BatchCoalescer in RepartitionExec to avoid schema type mismatch by @sgrebnov (opens in a new tab) in #9623 (opens in a new tab)
- feat: Implement catalog connectors for various databases by @lukekim (opens in a new tab) in #9509 (opens in a new tab)
- Refactor and clean up code across multiple crates by @lukekim (opens in a new tab) in #9620 (opens in a new tab)
- fix: Improve error handling for distributed mode and state_location configuration by @lukekim (opens in a new tab) in #9611 (opens in a new tab)
- Properly install postgres in
install-postgres action by @krinart (opens in a new tab) in #9629 (opens in a new tab)
- fix: Use Python venv for schema validation in CI by @phillipleblanc (opens in a new tab) in #9637 (opens in a new tab)
- Update spicepod.schema.json by @app/github-actions (opens in a new tab) in #9640 (opens in a new tab)
- Update testoperator dispatch to use release/2.0 branch by @phillipleblanc (opens in a new tab) in #9641 (opens in a new tab)
- fix: Align CUDA asset names in Dockerfile and install tests with build output by @phillipleblanc (opens in a new tab) in #9639 (opens in a new tab)
- Fix expect test scripts in E2E Installation AI test by @sgrebnov (opens in a new tab) in #9643 (opens in a new tab)
- testoperator for partitioned arrow accelerator by @Jeadie (opens in a new tab) in #9635 (opens in a new tab)
- Remove default 1s refresh_check_interval from spidapter for hive datasets by @phillipleblanc (opens in a new tab) in #9645 (opens in a new tab)
- Fix scheduler panic and cancel race condition by @phillipleblanc (opens in a new tab) in #9644 (opens in a new tab)
- Align Spice.ai connector parameter names across catalog/data connectors by @lukekim (opens in a new tab) in #9632 (opens in a new tab)
- docs: update distribution details and add NAS support in release notes by @lukekim (opens in a new tab) in #9650 (opens in a new tab)
- Enable
postgres-accel in CI builds for benchmarks by @sgrebnov (opens in a new tab) in #9649 (opens in a new tab)
- perf: Cache Turso metastore connection across operations by @penberg (opens in a new tab) in #9646 (opens in a new tab)
- Add 'scheduler_state_location' to spidapter by @Jeadie (opens in a new tab) in #9655 (opens in a new tab)
- Implement Cayenne S3 Express multi-zone live test with data validation by @lukekim (opens in a new tab) in #9631 (opens in a new tab)
- chore(spidapter): bump default memory limit from 8Gi to 32Gi by @phillipleblanc (opens in a new tab) in #9661 (opens in a new tab)
- perf: Use prepare_cached() in Turso and SQLite metastore backends by @penberg (opens in a new tab) in #9662 (opens in a new tab)
- Improve CDC cache invalidation by @krinart (opens in a new tab) in #9651 (opens in a new tab)
- Refactor Cayenne IDs to use UUIDv7 strings by @lukekim (opens in a new tab) in #9667 (opens in a new tab)
- fix: add liveness check for dead executors in partition routing by @Jeadie (opens in a new tab) in #9657 (opens in a new tab)
- fix(s3): Fix metadata column schema mismatches in projected queries by @sgrebnov (opens in a new tab) in #9664 (opens in a new tab)
- s3_metadata_columns tests: include test for location outside table prefix by @sgrebnov (opens in a new tab) in #9676 (opens in a new tab)
- docs: Update DuckDB, GCS, Git connector and Cayenne documentation by @lukekim (opens in a new tab) in #9671 (opens in a new tab)
- Add s3_url_style support for S3 connector URL addressing by @phillipleblanc (opens in a new tab) in #9642 (opens in a new tab)
- Consolidate E2E workflows and require WSL for Windows runtime by @lukekim (opens in a new tab) in #9660 (opens in a new tab)
- Upgrade to Rust v1.93.1 by @lukekim (opens in a new tab) in #9669 (opens in a new tab)
- Security fixes and improvements by @lukekim (opens in a new tab) in #9666 (opens in a new tab)
- feat(flight): add DoPut rows/bytes written metrics for DoPut ETL ingestion tracking by @phillipleblanc (opens in a new tab) in #9663 (opens in a new tab)
- Skip caching http error response + add
response_headers by @krinart (opens in a new tab) in #9670 (opens in a new tab)
- refactor: Remove
v1/evals functionality by @Jeadie (opens in a new tab) in #9420 (opens in a new tab)
- Make a test harness for Distributed Spice integration tests by @Jeadie (opens in a new tab) in #9615 (opens in a new tab)
- Enable
on_zero_results: use_source for views by @krinart (opens in a new tab) in #9699 (opens in a new tab)
- fix(spidapter): Lower memory limit, passthrough AWS secrets, override flight URL by @peasee (opens in a new tab) in #9704 (opens in a new tab)
- Show an error on a shared acceleration file with snapshots enabled by @krinart (opens in a new tab) in #9698 (opens in a new tab)
- Fixes for anthropic by @Jeadie (opens in a new tab) in #9707 (opens in a new tab)
- Use
max_partitions_per_executor in allocate_initial_partitions by @Jeadie (opens in a new tab) in #9659 (opens in a new tab)
- [SpiceDQ] Accelerations must have partition key by @Jeadie (opens in a new tab) in #9711 (opens in a new tab)
- Upgrade to Turso v0.5 by @lukekim (opens in a new tab) in #9628 (opens in a new tab)
- feat: Rename metadata columns to _location, _last_modified, _size by @phillipleblanc (opens in a new tab) in #9712 (opens in a new tab)
- fix: bump datafusion-ballista to fix BatchCoalescer schema mismatch panic by @phillipleblanc (opens in a new tab) in #9716 (opens in a new tab)
- fix: Ensure Cayenne respects target file size by @peasee (opens in a new tab) in #9730 (opens in a new tab)
- refactor: Make DDL preprocessing generic from Iceberg DDL processing by @peasee (opens in a new tab) in #9731 (opens in a new tab)
- [SpiceDQ] Distribute query of Cayenne Catalog to executors with data by @Jeadie (opens in a new tab) in #9727 (opens in a new tab)
- Properly set
primary_keys/on_conflict for Cayenne tables by @krinart (opens in a new tab) in #9739 (opens in a new tab)
- Add executor resource and replica support to cloud app config by @ewgenius (opens in a new tab) in #9734 (opens in a new tab)
- feat: Support PARTITION BY in Cayenne Catalog table creation by @peasee (opens in a new tab) in #9741 (opens in a new tab)
- Update datafusion and related packages to version 52.3.0 by @lukekim (opens in a new tab) in #9708 (opens in a new tab)
- Route FlightSQL statement updates through QueryBuilder by @phillipleblanc (opens in a new tab) in #9754 (opens in a new tab)
- JSON file format improvements by @lukekim (opens in a new tab) in #9743 (opens in a new tab)
- [SpiceDQ] Partition Cayenne catalogs writes through to executors by @Jeadie (opens in a new tab) in #9737 (opens in a new tab)
- Update to DF v52.3.0 versions of datafusion & datafusion-tableproviders by @lukekim (opens in a new tab) in #9756 (opens in a new tab)
- Make S3 metadata column handling more robust by @sgrebnov (opens in a new tab) in #9762 (opens in a new tab)
- Fetch API keys from dedicated endpoint instead of apps response by @phillipleblanc (opens in a new tab) in #9767 (opens in a new tab)
- Update arrow-rs, datafusion-federation, and datafusion-table-providers dependencies by @phillipleblanc (opens in a new tab) in #9769 (opens in a new tab)
- Chunk metastore batch inserts to respect SQLite parameter limits by @phillipleblanc (opens in a new tab) in #9770 (opens in a new tab)
- Improve JSON SODA support by @lukekim (opens in a new tab) in #9795 (opens in a new tab)
- Add ADBC Data Connector by @lukekim (opens in a new tab) in #9723 (opens in a new tab)
- docs: Release Cayenne as RC by @peasee (opens in a new tab) in #9766 (opens in a new tab)
- cli[feat]: cloud mode to use region-specific endpoints by @lukekim (opens in a new tab) in #9803 (opens in a new tab)
- Include updated JSON formats in HTTPS connector by @lukekim (opens in a new tab) in #9800 (opens in a new tab)
- Flight DoPut: Partition-aware write-through forwarding by @Jeadie (opens in a new tab) in #9759 (opens in a new tab)
- Pass through authentication to ADBC connector by @lukekim (opens in a new tab) in #9801 (opens in a new tab)
- Move scheduler_state_location from adapter metadata to env var by @phillipleblanc (opens in a new tab) in #9802 (opens in a new tab)
- Fix Cayenne DoPut upsert returning stale data after 3+ writes by @phillipleblanc (opens in a new tab) in #9806 (opens in a new tab)
- Fix JSON column projection producing schema mismatch by @sgrebnov (opens in a new tab) in #9811 (opens in a new tab)
- Fix http connector by @krinart (opens in a new tab) in #9818 (opens in a new tab)
- Fix ADBC Connector build and test by @lukekim (opens in a new tab) in #9813 (opens in a new tab)
- Support update & delete DML for distributed cayenne catalog by @Jeadie (opens in a new tab) in #9805 (opens in a new tab)
- Set allow_http param when S3 endpoint uses http scheme by @phillipleblanc (opens in a new tab) in #9834 (opens in a new tab)
- fix: Cayenne Catalog DDL requires a connected executor in distributed mode by @Jeadie (opens in a new tab) in #9838 (opens in a new tab)
- fix: Add conditional put support for file:// scheduler state location by @Jeadie (opens in a new tab) in #9842 (opens in a new tab)
- fix: Require the DDL primary key contain the partition key by @Jeadie (opens in a new tab) in #9844 (opens in a new tab)
- fix: Databricks SQL Warehouse schema retrieval with INLINE disposition and async retry by @lukekim (opens in a new tab) in #9846 (opens in a new tab)
- Filter pushdown improvements for SqlTable by @lukekim (opens in a new tab) in #9852 (opens in a new tab)
- feat: add iam_role_source parameter for AWS credential configuration by @lukekim (opens in a new tab) in #9854 (opens in a new tab)
- Fix ODBC queries silently returning 0 rows on query failure by @lukekim (opens in a new tab) in #9864 (opens in a new tab)
- feat(adbc): Add ADBC catalog connector with schema/table discovery by @lukekim (opens in a new tab) in #9865 (opens in a new tab)
- Make Turso SQL unparsing more robust and fix date comparisons by @lukekim (opens in a new tab) in #9871 (opens in a new tab)
- Fix Flight/FlightSQL filter precedence and mutable query consistency by @lukekim (opens in a new tab) in #9876 (opens in a new tab)
- Partial Aggregation optimisation for
FlightSQLExec by @lukekim (opens in a new tab) in #9882 (opens in a new tab)
- fix: v1/responses API preserves client instructions when system_prompt is set by @Jeadie (opens in a new tab) in #9884 (opens in a new tab)
- feat: emit
scheduler_active_executors_count and use it in spidapter by @Jeadie (opens in a new tab) in #9885 (opens in a new tab)
- feat: Add custom auth header support for GraphQL connector by @krinart (opens in a new tab) in #9899 (opens in a new tab)
- Add --endpoint flag to spice run with scheme-based routing by @lukekim (opens in a new tab) in #9903 (opens in a new tab)
- When executor connects, send DDL for existing tables by @Jeadie (opens in a new tab) in #9904 (opens in a new tab)
- fix: Improve ADBC driver shutdown handling and error classification by @lukekim (opens in a new tab) in #9905 (opens in a new tab)
- fix: require all executors to succeed for distributed DML (DELETE/UPDATE) forwarding by @Jeadie (opens in a new tab) in #9908 (opens in a new tab)
- fix(cayenne catalog): fix catalog refresh race condition causing duplicate primary keys by @Jeadie (opens in a new tab) in #9909 (opens in a new tab)
- Remove Perplexity support by @Jeadie (opens in a new tab) in #9910 (opens in a new tab)
- Fix refresh_sql support for debezium constraints by @krinart (opens in a new tab) in #9912 (opens in a new tab)
- Implement DML for DynamoDBTableProvider by @lukekim (opens in a new tab) in #9915 (opens in a new tab)
- chore: Update iceberg-rust fork to v0.9 by @lukekim (opens in a new tab) in #9917 (opens in a new tab)
- Run physical optimizer on
FallbackOnZeroResultsScanExec fallback plan by @sgrebnov (opens in a new tab) in #9927 (opens in a new tab)
- Improve Databricks error message when dataset has no columns by @sgrebnov (opens in a new tab) in #9928 (opens in a new tab)
- Delta Lake: fix data skipping for >= timestamp predicates by @sgrebnov (opens in a new tab) in #9932 (opens in a new tab)
- fix: Ensure distributed Cayenne DML inserts are forwarded to executors by @Jeadie (opens in a new tab) in #9948 (opens in a new tab)
- Add full query federation support for ADBC data connector by @lukekim (opens in a new tab) in #9953 (opens in a new tab)
- Make time_format deserialization case-insensitive by @vyershov (opens in a new tab) in #9955 (opens in a new tab)
- Hash ADBC join-pushdown context to prevent credential leaks in EXPLAIN plans by @lukekim (opens in a new tab) in #9956 (opens in a new tab)
- fix: Normalize Arrow Dictionary types for DuckDB and SQLite acceleration by @sgrebnov (opens in a new tab) in #9959 (opens in a new tab)
- ADBC BigQuery: Improve BigQuery dialect date/time and interval SQL generation by @lukekim (opens in a new tab) in #9967 (opens in a new tab)
- Make
BigQueryDialect more robust and add BigQuery TPC-H benchmark support by @lukekim (opens in a new tab) in #9969 (opens in a new tab)
- fix: Show proper unauthorized error instead of misleading runtime unavailable by @lukekim (opens in a new tab) in #9972 (opens in a new tab)
- fix: Enforce target_chunk_size as hard maximum in chunking by @lukekim (opens in a new tab) in #9973 (opens in a new tab)
- Add caching retention by @krinart (opens in a new tab) in #9984 (opens in a new tab)
- fix: improve Databricks schema error detection and messages by @lukekim (opens in a new tab) in #9987 (opens in a new tab)
- fix: Set default S3 region for opendal operator and fix cayenne nextest by @phillipleblanc (opens in a new tab) in #9995 (opens in a new tab)
- fix(PostgreSQL): fix schema discovery for PostgreSQL partitioned tables by @sgrebnov (opens in a new tab) in #9997 (opens in a new tab)
- fix: Defer cache size check until after encoding for compressed results by @krinart (opens in a new tab) in #10001 (opens in a new tab)
- fix: Rewrite numeric BETWEEN to CAST(AS REAL) for Turso by @lukekim (opens in a new tab) in #10003 (opens in a new tab)
- fix: Handle integer time columns in append refresh for all accelerators by @sgrebnov (opens in a new tab) in #10004 (opens in a new tab)
- fix: preserve s3a:// scheme when building OpenDalStorageFactory with custom endpoint by @phillipleblanc (opens in a new tab) in #10006 (opens in a new tab)
- Fix ISO8601 time_format with Vortex/Cayenne append refresh by @sgrebnov (opens in a new tab) in #10009 (opens in a new tab)
- fix: Address data correctness bugs found in audit by @sgrebnov (opens in a new tab) in #10015 (opens in a new tab)
- fix(federation): fix SQL unparsing for Inexact filter pushdown with alias by @lukekim (opens in a new tab) in #10017 (opens in a new tab)
- Improve GitHub connector ref handling and resilience by @lukekim (opens in a new tab) in #10023 (opens in a new tab)
- feat: Add spice completions command for shell completion generation by @lukekim (opens in a new tab) in #10024 (opens in a new tab)
- fix: Fix data correctness bugs in DynamoDB decimal conversion and GraphQL pagination by @sgrebnov (opens in a new tab) in #10054 (opens in a new tab)
- Implement RefreshDataset for distributed control stream by @Jeadie (opens in a new tab) in #10055 (opens in a new tab)
- perf: Improve S3 parquet read performance by @sgrebnov (opens in a new tab) in #10064 (opens in a new tab)
- fix: Prevent write-through stalls and preserve PartitionTableProvider during catalog refresh by @Jeadie (opens in a new tab) in #10066 (opens in a new tab)
- feat:
spice completions auto-detects shell directory and writes file by @lukekim (opens in a new tab) in #10068 (opens in a new tab)
- fix: Bug in DynamoDB, GraphQL, and ISO8601 refresh data handling by @sgrebnov (opens in a new tab) in #10063 (opens in a new tab)
- fix partial aggregation deduplication on string checking by @lukekim (opens in a new tab) in #10078 (opens in a new tab)
- fix: add MetastoreTransaction support to prevent concurrent transaction conflicts by @phillipleblanc (opens in a new tab) in #10080 (opens in a new tab)
- fix: Use GreedyMemoryPool, add spidapter query memory limit arg by @phillipleblanc (opens in a new tab) in #10082 (opens in a new tab)
- feat: Add metrics for EXPLAIN ANALYZE in FlightSQLExec by @lukekim (opens in a new tab) in #10084 (opens in a new tab)
- Use strict cast in
try_cast_to to error on overflow instead of silent NULL by @sgrebnov (opens in a new tab) in #10104 (opens in a new tab)
- feat: Implement MERGE INTO for Cayenne catalog tables by @peasee (opens in a new tab) in #10105 (opens in a new tab)
- feat: Add distributed MERGE INTO support for Cayenne catalog tables by @peasee (opens in a new tab) in #10106 (opens in a new tab)
- Improve JSON format auto-detection for single multi-line objects by @lukekim (opens in a new tab) in #10107 (opens in a new tab)
- Add mode: file_update acceleration mode by @krinart (opens in a new tab) in #10108 (opens in a new tab)
- Coerce unsupported Arrow types to Iceberg v2 equivalents in REST catalog API by @peasee (opens in a new tab) in #10109 (opens in a new tab)
- fix: Update default query memory limit to 90% from 70% by @phillipleblanc (opens in a new tab) in #10112 (opens in a new tab)
- feat: Add mTLS client auth support to spice sql REPL by @lukekim (opens in a new tab) in #10113 (opens in a new tab)
- fix(datafusion-federation): report error on overflow instead of silent NULL by @sgrebnov (opens in a new tab) in #10124 (opens in a new tab)
- fix: Prevent data loss in MERGE when source has duplicate keys by @peasee (opens in a new tab) in #10126 (opens in a new tab)
- feat: Add ClickHouse Date32 type support by @sgrebnov (opens in a new tab) in #10132 (opens in a new tab)
- Add Delta Lake column mapping support (Name/Id modes) by @sgrebnov (opens in a new tab) in #10134 (opens in a new tab)
- fix: Restore Turso numeric BETWEEN rewrite lost in DML revert by @lukekim (opens in a new tab) in #10139 (opens in a new tab)
- fix: Enable arm64 Linux builds with fp16 and lld workarounds by @lukekim (opens in a new tab) in #10142 (opens in a new tab)
- fix: remove double trailing slash in Unity Catalog storage locations by @sgrebnov (opens in a new tab) in #10147 (opens in a new tab)
- fix: Improve GitHub GraphQL client resilience and performance by @lukekim (opens in a new tab) in #10151 (opens in a new tab)
- Enable reqwest compression and optimize HTTP client settings by @lukekim (opens in a new tab) in #10154 (opens in a new tab)
- fix: executor startup failures by @Jeadie (opens in a new tab) in #10155 (opens in a new tab)
- feat: Distributed runtime.task_history support by @Jeadie (opens in a new tab) in #10156 (opens in a new tab)
- fix: Preserve timestamp timezone in DDL forwarding to executors by @peasee (opens in a new tab) in #10159 (opens in a new tab)
- feat: Per-model rate-limited concurrent AI UDF execution by @Jeadie (opens in a new tab) in #10160 (opens in a new tab)
- fix(Turso): Reject subquery/outer-ref filter pushdown in Turso provider by @lukekim (opens in a new tab) in #10174 (opens in a new tab)
- Fix linux/macos
spice upgrade by @phillipleblanc (opens in a new tab) in #10194 (opens in a new tab)
- Improve CREATE TABLE LIKE error messages, success output, EXPLAIN, and validation by @peasee (opens in a new tab) in #10203 (opens in a new tab)
- fix: chunk MERGE delete filters and update Vortex for stack-safe IN-lists by @peasee (opens in a new tab) in #10207 (opens in a new tab)
- Propagate
runtime.params.parquet_page_index to Delta Lake connector by @sgrebnov (opens in a new tab) in #10209 (opens in a new tab)
- Properly mark dataset as Ready on Scheduler by @Jeadie (opens in a new tab) in #10215 (opens in a new tab)
- fix: handle Utf8View/LargeUtf8 in GitHub connector ref filters by @lukekim (opens in a new tab) in #10217 (opens in a new tab)
- fix(databricks): Fix schema introspection and timestamp overflow by @lukekim (opens in a new tab) in #10226 (opens in a new tab)
- fix(databricks): Fix schema introspection failures for non-Unity-Catalog environments by @lukekim (opens in a new tab) in #10227 (opens in a new tab)
- feat: Add pagination support to HTTP data connector by @lukekim (opens in a new tab) in #10228 (opens in a new tab)
- feat(databricks): DESCRIBE TABLE fallback and source-native type parsing for Lakehouse Federation by @lukekim (opens in a new tab) in #10229 (opens in a new tab)
- fix(databricks): harden HTTP retries, compression, and token refresh by @lukekim (opens in a new tab) in #10232 (opens in a new tab)
- feat[helm chart]: Add support for ServiceAccount annotations and AWS IRSA example by @peasee (opens in a new tab) in #9833 (opens in a new tab)
- fix: Log warning and fall back gracefully on Cayenne config change by @krinart (opens in a new tab) in #9092 (opens in a new tab)
- fix: Handle engine mismatch gracefully in snapshot fallback loop by @krinart (opens in a new tab) in #9187 (opens in a new tab)
Full Changelog: https://github.com/spiceai/spiceai/compare/v2.0.0-rc.1...v2.0.0-rc.2 (opens in a new tab)
