spiceai/docs

date: 2024-07-01 title: 'Spice v0.15-alpha (July 1, 2024)' type: blog authors: [lukekim] categories: [release] tags: [debezium connector, cdc, kafka, data-connector, client sdk, data refresh retries]

The v0.15-alpha release introduces support for streaming databases changes with Change Data Capture (CDC) into accelerated tables via a new Debezium connector, configurable retry logic for data refresh, and the release of a new C# SDK to build with Spice in Dotnet.

Highlights in v0.15-alpha

Debezium data connector with Change Data Capture (CDC): Sync accelerated datasets with Debezium data sources over Kafka in real-time.
Data Refresh Retries: By default, accelerated datasets attempt to retry data refreshes on transient errors. This behavior can be configured using refresh_retry_enabled and refresh_retry_max_attempts.
C# Client SDK: A new C# Client SDK (opens in a new tab) has been released for developing applications in Dotnet.

Debezium data connector with Change Data Capture (CDC)

Integrating Debezium CDC is straightforward. Get started with the Debezium CDC Sample (opens in a new tab), read more about CDC (opens in a new tab) in Spice, and read the Debezium data connector documentation (opens in a new tab).

Example Spicepod using Debezium CDC:

datasets:
  - from: debezium:cdc.public.customer_addresses
    name: customer_addresses_cdc
    params:
      debezium_transport: kafka
      debezium_message_format: json
      kafka_bootstrap_servers: localhost:19092
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      refresh_mode: changes

datasets:
  - from: debezium:cdc.public.customer_addresses
    name: customer_addresses_cdc
    params:
      debezium_transport: kafka
      debezium_message_format: json
      kafka_bootstrap_servers: localhost:19092
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      refresh_mode: changes

Data Refresh Retries

Example Spicepod configuration limiting refresh retries to a maximum of 10 attempts:

datasets:
  - from: eth.blocks
    name: blocks
    acceleration:
      refresh_retry_enabled: true
      refresh_retry_max_attempts: 10
      refresh_check_interval: 30s

datasets:
  - from: eth.blocks
    name: blocks
    acceleration:
      refresh_retry_enabled: true
      refresh_retry_max_attempts: 10
      refresh_check_interval: 30s

Breaking Changes

None.

New Contributors

@rupurt made their first contribution in https://github.com/spiceai/spiceai/pull/1791 (opens in a new tab)

Contributors

What's Changed

Dependencies

No major dependency updates.

Commits

Update version to 0.15.0-alpha by @ewgenius in https://github.com/spiceai/spiceai/pull/1784 (opens in a new tab)
Update helm for v0.14.1-alpha by @ewgenius in https://github.com/spiceai/spiceai/pull/1786 (opens in a new tab)
Run PR checks on PRs merging into feature-- branches by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1788 (opens in a new tab)
Enable retries for accelerated table refresh by @sgrebnov in https://github.com/spiceai/spiceai/pull/1762 (opens in a new tab)
enable more tpch benchmark queries as a result of decimal unparsing by @y-f-u in https://github.com/spiceai/spiceai/pull/1790 (opens in a new tab)
add nix flake by @rupurt in https://github.com/spiceai/spiceai/pull/1791 (opens in a new tab)
Support local and HF embedding models by @Jeadie in https://github.com/spiceai/spiceai/pull/1789 (opens in a new tab)
fix(bin/spice): Implement custom Unmarshaller for DatasetOrReference by @peasee in https://github.com/spiceai/spiceai/pull/1787 (opens in a new tab)
For windows, move symlink -> symlink_file. by @Jeadie in https://github.com/spiceai/spiceai/pull/1793 (opens in a new tab)
docs: Add PULL_REQUEST_TEMPLATE.md by @peasee in https://github.com/spiceai/spiceai/pull/1794 (opens in a new tab)
Fix Unsupported DataType: conversion for time predicates by @sgrebnov in https://github.com/spiceai/spiceai/pull/1795 (opens in a new tab)
Use incremental backoff for initial dataset registration retries by @sgrebnov in https://github.com/spiceai/spiceai/pull/1805 (opens in a new tab)
Basic HTTP/S connector by @Jeadie in https://github.com/spiceai/spiceai/pull/1792 (opens in a new tab)
Scale support for Snowflake fixed-point numbers by @sgrebnov in https://github.com/spiceai/spiceai/pull/1804 (opens in a new tab)
bump datafusion federation to resolve the join query failures by @y-f-u in https://github.com/spiceai/spiceai/pull/1806 (opens in a new tab)
fix: Stream PostgreSQL data in by @peasee in https://github.com/spiceai/spiceai/pull/1798 (opens in a new tab)
Remove clippy::module_name_repetitions lint by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1812 (opens in a new tab)
Improve Snowflake fixed-point numbers casting by @sgrebnov in https://github.com/spiceai/spiceai/pull/1809 (opens in a new tab)
Case insensitive secret getter by @ewgenius in https://github.com/spiceai/spiceai/pull/1813 (opens in a new tab)
refactor: Format TOML with Taplo by @peasee in https://github.com/spiceai/spiceai/pull/1808 (opens in a new tab)
feat: Update PR template, add label enforcement in PR by @peasee in https://github.com/spiceai/spiceai/pull/1815 (opens in a new tab)
fix bug that append may miss updates when the incremental changes are not able to be contained in one record batch by @y-f-u in https://github.com/spiceai/spiceai/pull/1817 (opens in a new tab)
add integration test for inner join across federated table and accelerated table by @y-f-u in https://github.com/spiceai/spiceai/pull/1811 (opens in a new tab)
Unify spicepod.llms into spicepod.models and refactor UX of spicepod.models by @Jeadie in https://github.com/spiceai/spiceai/pull/1818 (opens in a new tab)
Fix issue with querying accelerated tables where the dataset name has a schema by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1823 (opens in a new tab)
Fix schema support for refresh_sql and improve e2e tests by @sgrebnov in https://github.com/spiceai/spiceai/pull/1826 (opens in a new tab)
feat: Add GraphQL unnesting by @peasee in https://github.com/spiceai/spiceai/pull/1822 (opens in a new tab)
fix: Allow kind/optimization labels, increase Postgres test timeout by @peasee in https://github.com/spiceai/spiceai/pull/1830 (opens in a new tab)
Implement Real-time acceleration updates via Debezium CDC by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1832 (opens in a new tab)
Remove println statement from PG Connector by @sgrebnov in https://github.com/spiceai/spiceai/pull/1835 (opens in a new tab)
Don't try to "hot reload" Debezium accelerated datasets by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1837 (opens in a new tab)
Create v1/search that performs vector search. by @Jeadie in https://github.com/spiceai/spiceai/pull/1836 (opens in a new tab)
Align spicepod UX of embeddings with models by @Jeadie in https://github.com/spiceai/spiceai/pull/1829 (opens in a new tab)
Add "cmake-build" feature to rdkafka for windows by @Jeadie in https://github.com/spiceai/spiceai/pull/1840 (opens in a new tab)
Add a better error message when trying to configure refresh_mode=changes on a data connector that doesn't support it. by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1839 (opens in a new tab)

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.14.1-alpha...v0.15.0-alpha (opens in a new tab)

Resources

Community

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

Twitter: @spice_ai (opens in a new tab)
Slack: spiceai.org/slack (opens in a new tab)
Telegram: Spice AI Discussion (opens in a new tab)
Reddit: https://www.reddit.com/r/spiceai (opens in a new tab)
Email: hey@spice.ai (opens in a new tab)

spiceai/docs/README.md