date: 2024-06-17 title: 'Spice v0.14-alpha (June 17, 2024)' type: blog authors: [sgrebnov] categories: [release] tags: [accelerated datasets, data integrity, primary keys, constraints, indexes, graphql, logging]

The v0.14-alpha release focuses on enhancing accelerated dataset performance and data integrity, with support for configuring primary keys and indexes. Additionally, the GraphQL data connector been introduced, along with improved dataset registration and loading error information.

Highlights

• Accelerated Datasets: Ensure data integrity using primary key and unique index constraints. Configure conflict handling to either upsert new data or drop it. Create indexes on frequently filtered columns for faster queries on larger datasets.

• GraphQL Data Connector: Initial support for using GraphQL as a data source.

Example Spicepod showing how to use primary keys and indexes with accelerated datasets:

datasets:
  - from: eth.blocks
    name: blocks
    acceleration:
      engine: duckdb # Use DuckDB acceleration engine
      primary_key: '(hash, timestamp)'
      indexes:
        number: enabled # same as `CREATE INDEX ON blocks (number);`
        '(number, hash)': unique # same as `CREATE UNIQUE INDEX ON blocks (number, hash);`
      on_conflict:
        '(hash, number)': drop # possible values: drop (default), upsert
        '(hash, timestamp)': upsert

datasets:
  - from: eth.blocks
    name: blocks
    acceleration:
      engine: duckdb # Use DuckDB acceleration engine
      primary_key: '(hash, timestamp)'
      indexes:
        number: enabled # same as `CREATE INDEX ON blocks (number);`
        '(number, hash)': unique # same as `CREATE UNIQUE INDEX ON blocks (number, hash);`
      on_conflict:
        '(hash, number)': drop # possible values: drop (default), upsert
        '(hash, timestamp)': upsert

Primary Keys, constraints, and indexes are currently supported when using SQLite, DuckDB, and PostgreSQL acceleration engines.

Learn more with the indexing (opens in a new tab) quickstart and the primary key (opens in a new tab) sample.

Read the Local Acceleration (opens in a new tab) documentation.

Breaking Changes

None.

Contributors

• @phillipleblanc
• @ewgenius
• @sgrebnov
• @Jeadie
• @digadeesh
• @gloomweaver
• @y-f-u
• @lukekim
• @edmondop

What's Changed

Dependencies

• Apache DataFusion: Upgraded from 38.0.0 to 39.0.0
• Apache Arrow/Parquet: Upgraded from 51.0.0 to 52.0.0
• Rust: Upgraded from 1.78.0 to 1.79.0

Commits

• Update Helm chart for v0.13.3-alpha by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1671 (opens in a new tab)
• Bump version to v0.14.0-alpha by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1673 (opens in a new tab)
• Dependency upgrades: DataFusion 39, Arrow/Parquet 52, object_store 0.10.1, arrow-odbc 11.1.0 by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1674 (opens in a new tab)
• Generate unique runtime instance name and store in runtime.metrics table by @ewgenius in https://github.com/spiceai/spiceai/pull/1678 (opens in a new tab)
• Proper support for Snowflake TIMESTAMP_NTZ by @sgrebnov in https://github.com/spiceai/spiceai/pull/1677 (opens in a new tab)
• Enable tpch_q2 and tpch_q21 in the benchmark queries by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1679 (opens in a new tab)
• Start runtime metrics recorder after loading secrets and extensions by @ewgenius in https://github.com/spiceai/spiceai/pull/1680 (opens in a new tab)
• Validate table constraints (Primary Keys/Unique Index) on accelerated tables by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1658 (opens in a new tab)
• Store labels as JSON string in runtime.metrics by @ewgenius in https://github.com/spiceai/spiceai/pull/1681 (opens in a new tab)
• Atomic updates for DuckDB tables with constraints by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1682 (opens in a new tab)
• Rename metrics column labels to properties and make it nullable by @ewgenius in https://github.com/spiceai/spiceai/pull/1686 (opens in a new tab)
• Fix federation_optimizer_rule schema error for tpch_q7, tpch_q8, tpch_q9, tpch_q14 by @sgrebnov in https://github.com/spiceai/spiceai/pull/1683 (opens in a new tab)
• Better prompt for /v1/assist by @Jeadie in https://github.com/spiceai/spiceai/pull/1685 (opens in a new tab)
• Support stream in v1/assist by @Jeadie in https://github.com/spiceai/spiceai/pull/1653 (opens in a new tab)
• Fix cache hit rate chart loading for Grafana v9.5 by @sgrebnov in https://github.com/spiceai/spiceai/pull/1691 (opens in a new tab)
• Update ROADMAP.md to include data connector statuses by @digadeesh in https://github.com/spiceai/spiceai/pull/1684 (opens in a new tab)
• Support primary_key in Spicepod and create in accelerated table by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1687 (opens in a new tab)
• Datasets with schema support for availability monitoring by @sgrebnov in https://github.com/spiceai/spiceai/pull/1690 (opens in a new tab)
• Improve dataset registration output by @sgrebnov in https://github.com/spiceai/spiceai/pull/1692 (opens in a new tab)
• Readme: update dataset registration traces by @sgrebnov in https://github.com/spiceai/spiceai/pull/1694 (opens in a new tab)
• Improved error logging for datasets load error by @edmondop in https://github.com/spiceai/spiceai/pull/1695 (opens in a new tab)
• Improve ArrayDistance scalar UDF by @Jeadie in https://github.com/spiceai/spiceai/pull/1697 (opens in a new tab)
• Implement on_conflict behavior for accelerated tables with constraints by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1688 (opens in a new tab)
• Fix datasets live update (Spice file watcher) by @sgrebnov in https://github.com/spiceai/spiceai/pull/1702 (opens in a new tab)
• Grafana Dashboard: replace Quantile with Percentile filter by @sgrebnov in https://github.com/spiceai/spiceai/pull/1703 (opens in a new tab)
• refresh with append overlap by @y-f-u in https://github.com/spiceai/spiceai/pull/1706 (opens in a new tab)
• Fix error message on DuckDB constraint violation by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1709 (opens in a new tab)
• Add warning when configuring indexes/primary_key/on_conflict for Arrow engine. by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1710 (opens in a new tab)
• ensure schema to be existing when query timestamp during refresh by @y-f-u in https://github.com/spiceai/spiceai/pull/1711 (opens in a new tab)
• Improve README clarity and add comparison table by @lukekim in https://github.com/spiceai/spiceai/pull/1713 (opens in a new tab)
• Update acknowledgements by @github-actions in https://github.com/spiceai/spiceai/pull/1716 (opens in a new tab)
• Update README.md to include GraphQL data connector in supported table by @digadeesh in https://github.com/spiceai/spiceai/pull/1717 (opens in a new tab)
• Fix quoting issue for databricks identifier by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1718 (opens in a new tab)

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.13.3-alpha...v0.14.0-alpha (opens in a new tab)

Resources

• Getting started with Spice.ai (opens in a new tab)
• Documentation (opens in a new tab)

Community

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

• Twitter: @spice_ai (opens in a new tab)
• Slack: spiceai.org/slack (opens in a new tab)
• Telegram: Spice AI Discussion (opens in a new tab)
• Reddit: https://www.reddit.com/r/spiceai (opens in a new tab)
• Email: hey@spice.ai (opens in a new tab)