date: 2024-07-15 title: 'Spice v0.15.2-alpha (July 15, 2024)' type: blog authors: [lukekim] categories: [release] tags: [catalog providers, data catalog tables, unity catalog, databricks, mysql, postgresql]

The v0.15.2-alpha minor release focuses on enhancing stability, performance, and introduces Catalog Providers (opens in a new tab) for streamlined access to Data Catalog tables. Unity Catalog, Databricks Unity Catalog, and the Spice.ai Cloud Platform Catalog are supported in v0.15.2-alpha. The reliability of federated query push-down has also been improved for the MySQL, PostgreSQL, ODBC, S3, Databricks, and Spice.ai Cloud Platform data connectors.

Highlights in v0.15.2-alpha

Catalog Providers: Catalog Providers (opens in a new tab) streamline access to Data Catalog tables. Initial catalog providers supported are Databricks Unity Catalog (opens in a new tab), Unity Catalog (opens in a new tab) and Spice.ai Cloud Platform Catalog.

For example, to configure Spice to connect to tpch tables in the Spice.ai Cloud Platform Catalog use the new catalogs: section in the spicepod.yml:

catalogs:
  - name: spiceai
    from: spiceai
    include:
      - tpch.*

catalogs:
  - name: spiceai
    from: spiceai
    include:
      - tpch.*

sql> show tables
+---------------+--------------+---------------+------------+
| table_catalog | table_schema | table_name    | table_type |
+---------------+--------------+---------------+------------+
| spiceai       | tpch         | region        | BASE TABLE |
| spiceai       | tpch         | part          | BASE TABLE |
| spiceai       | tpch         | customer      | BASE TABLE |
| spiceai       | tpch         | lineitem      | BASE TABLE |
| spiceai       | tpch         | partsupp      | BASE TABLE |
| spiceai       | tpch         | supplier      | BASE TABLE |
| spiceai       | tpch         | nation        | BASE TABLE |
| spiceai       | tpch         | orders        | BASE TABLE |
| spice         | runtime      | query_history | BASE TABLE |
+---------------+--------------+---------------+------------+

Time: 0.001866958 seconds. 9 rows.

sql> show tables
+---------------+--------------+---------------+------------+
| table_catalog | table_schema | table_name    | table_type |
+---------------+--------------+---------------+------------+
| spiceai       | tpch         | region        | BASE TABLE |
| spiceai       | tpch         | part          | BASE TABLE |
| spiceai       | tpch         | customer      | BASE TABLE |
| spiceai       | tpch         | lineitem      | BASE TABLE |
| spiceai       | tpch         | partsupp      | BASE TABLE |
| spiceai       | tpch         | supplier      | BASE TABLE |
| spiceai       | tpch         | nation        | BASE TABLE |
| spiceai       | tpch         | orders        | BASE TABLE |
| spice         | runtime      | query_history | BASE TABLE |
+---------------+--------------+---------------+------------+

Time: 0.001866958 seconds. 9 rows.

ODBC Data Connector Push-Down: The ODBC Data Connector (opens in a new tab) now supports query push-down for joins, improving performance for joined datasets configured with the same odbc_connection_string.

Improved Spicepod Validation Improved spicepod.yml validation has been added, including warnings when loading resources with duplicate names (datasets, views, models, embeddings).

Breaking Changes

None.

Contributors

• @phillipleblanc
• @peasee
• @y-f-u
• @ewgenius
• @Sevenannn
• @sgrebnov
• @lukekim

What's Changed

Dependencies

• Upgraded Apache DataFusion (opens in a new tab) to v40.0.0 (opens in a new tab).

Commits

• Update to next release version v0.15.2-alpha by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1901 (opens in a new tab)
• release: Update helm 0.15.1-alpha by @peasee in https://github.com/spiceai/spiceai/pull/1902 (opens in a new tab)
• fix: Detect and error on duplicate component names on spiced (re)load by @peasee in https://github.com/spiceai/spiceai/pull/1905 (opens in a new tab)
• fix: flaky test - test_refresh_status_change_to_ready by @y-f-u in https://github.com/spiceai/spiceai/pull/1908 (opens in a new tab)
• Add support for parsing catalog from Spicepod. by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1903 (opens in a new tab)
• Add catalog component to Runtime by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1906 (opens in a new tab)
• Adds a RuntimeBuilder and make most items on Runtime private by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1913 (opens in a new tab)
• Bump zerovec-derive from 0.10.2 to 0.10.3 by @dependabot in https://github.com/spiceai/spiceai/pull/1914 (opens in a new tab)
• Add separate tagged image with enabled models feature by @ewgenius in https://github.com/spiceai/spiceai/pull/1909 (opens in a new tab)
• Update datafusion-table-providers to use newest head by @Sevenannn in https://github.com/spiceai/spiceai/pull/1927 (opens in a new tab)
• Add MySQL support for TPC-H test data generation script by @sgrebnov in https://github.com/spiceai/spiceai/pull/1932 (opens in a new tab)
• fix: Expose ODBC task errors if error is before data stream begins by @peasee in https://github.com/spiceai/spiceai/pull/1924 (opens in a new tab)
• Use public.ecr.aws/docker/library/{postgres/mysql}:latest for integration test images by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1934 (opens in a new tab)
• Implement spice.ai CatalogProvider by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1925 (opens in a new tab)
• fix: validate time column and time format when constructing accelerated table refresh by @y-f-u in https://github.com/spiceai/spiceai/pull/1926 (opens in a new tab)
• Add support for filtering tables included by a catalog by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1933 (opens in a new tab)
• Add UnityCatalog catalog provider by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1940 (opens in a new tab)
• Implement Databricks catalog provider by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1941 (opens in a new tab)
• Copy params into dataset_params by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1947 (opens in a new tab)
• Make integration tests more stable by using logged-in registry during CI by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1955 (opens in a new tab)
• Add integration test for Spice.ai catalog provider by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1956 (opens in a new tab)
• Add GET /v1/catalogs API and catalogs CMD by @lukekim in https://github.com/spiceai/spiceai/pull/1957 (opens in a new tab)
• feat: Enable ODBC JoinPushDown with hashed connection string by @peasee in https://github.com/spiceai/spiceai/pull/1954 (opens in a new tab)
• Fix bug: arrow acceleration reports zero results during refresh by @sgrebnov in https://github.com/spiceai/spiceai/pull/1962 (opens in a new tab)
• Revert "fix: validate time column and time format when constructing accelerated table refresh" by @y-f-u in https://github.com/spiceai/spiceai/pull/1964 (opens in a new tab)
• fix: Update arrow-odbc to use our fork for pending fixes by @peasee in https://github.com/spiceai/spiceai/pull/1965 (opens in a new tab)
• Upgrade to DataFusion 40 by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1963 (opens in a new tab)
• Do exchange shouldn't require table to be writable by @Sevenannn in https://github.com/spiceai/spiceai/pull/1958 (opens in a new tab)
• Use custom dialect rule for flight federated request by @y-f-u in https://github.com/spiceai/spiceai/pull/1946 (opens in a new tab)
• upgrade datafusion federation to have the table rewrite fix for tpch-q9 by @y-f-u in https://github.com/spiceai/spiceai/pull/1970 (opens in a new tab)
• Create v0.15.2-alpha.md Release notes by @digadeesh in https://github.com/spiceai/spiceai/pull/1969 (opens in a new tab)
• Fix Unity Catalog API response for Azure Databricks by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1973 (opens in a new tab)
• Update acknowledgements by @github-actions in https://github.com/spiceai/spiceai/pull/1976 (opens in a new tab)

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.15.1-alpha...v0.15.2-alpha (opens in a new tab)

Resources

• Getting started with Spice.ai (opens in a new tab)
• Documentation (opens in a new tab)

Community

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

• Twitter: @spice_ai (opens in a new tab)
• Slack: spiceai.org/slack (opens in a new tab)
• Telegram: Spice AI Discussion (opens in a new tab)
• Reddit: https://www.reddit.com/r/spiceai (opens in a new tab)
• Email: hey@spice.ai (opens in a new tab)