spiceai/docs

Help Login

trunk

Edit on GitHub

Fork

/docs/website/releases/v0.16-alpha.md

spiceai/docs | Spice Cloud Platform

trunk

Edit on GitHub

Fork

/docs/website/releases/v0.16-alpha.md

spiceai/docs/README.md

date: 2024-07-22 title: 'Spice v0.16-alpha (July 22, 2024)' type: blog authors: [lukekim] categories: [release] tags: [secret store, odbc connector, beta milestone, java sdk, federated sql]

The v0.16-alpha release is the first candidate release for the beta milestone on a path to finalizing the v1.0 developer and user experience. Upgraders should be aware of several breaking changes designed to improve the Secrets configuration experience and to make authoring spicepod.yml files more consistent. See the [Breaking Changes](#Breaking Changes) section below for details. Additionally, the Spice Java SDK was released, providing Java developers a simple but powerful native experience to query Spice.

Highlights in v0.16-alpha

Secret Stores: More than one Secret Store can now be specified. For example, to configure Spice with both Environment Variable and AWS Secrets Manager Secret Stores, use the following secrets configuration in spicepod.yaml:

Secrets managed by configured Secret Stores can be referenced in component params using the syntax ${<store_name>:<key>}. E.g.

Java Client SDK: The Spice Java SDK has been released for JDK 17 or greater.
Federated SQL Query: Significant stability and reliability improvements have been made to federated SQL query support in most data connectors.
ODBC Data Connector: Providing a specific SQL dialect to query ODBC data sources is now supported using the sql_dialect param. For example, when querying Databricks using ODBC, the databricks dialect can be specified to ensure compatibility. Read the ODBC Data Connector documentation for more details.

Breaking Changes

Secret Stores: Secret Stores support has been overhauled including required changes to spicepod.yml schema. File based secrets stored in the ~/.spice/auth file are no longer supported. See Secret Stores Documentation for full reference.

To upgrade Secret Stores, rename any parameters ending in _key to remove the _key suffix and specify a secret inline via the secret replacement syntax (${<secret_store>:<key>}):

to:

And ensure the MY_PG_PASS environment variable is set.

Datasets: The default value of time_format has changed from unix_seconds to timestamp.

To upgrade:

HTTP Port: The default HTTP port has changed from port 3000 to port 8090 to avoid conflicting with frontend apps which typically use the 3000 range. If an SDK is used, upgrade it at the same time as the runtime.

To upgrade and continue using port 3000, run spiced with the --http command line argument:

HTTP Metrics Port: The default HTTP Metrics port has changed from port 9000 to 9090 to avoid conflicting with other metrics protocols which typically use port 9000.

To upgrade and continue using port 9000, run spiced with the metrics command line argument:

GraphQL Data Connector: json_path has been replaced with json_pointer to access nested data from the result of the GraphQL query. See the GraphQL Data Connector documentation for full details and RFC-6901 - JSON Pointer.

To upgrade, change:

To:

Data Connector Configuration: Consistent connector name prefixing has been applied to connector specific params parameters. Prefixed parameter names helps ensure parameters do not collide.

For example, the Databricks data connector specific params are now prefixed with databricks:

To upgrade:

Refer to the Data Connector documentation for parameter naming changes in this release.

Clickhouse Data Connector: The clickhouse_connection_timeout parameter has been renamed to connection_timeout as it applies to the client and is not Clickhouse configuration itself.

To upgrade, change:

To:

Contributors

@y-f-u
@phillipleblanc
@ewgenius
@github-actions
@sgrebnov
@lukekim
@digadeesh
@peasee
@Sevenannn

What's Changed

Dependencies

No major dependency updates.

Commits

bump helm chart versions to 0.15.2-alpha by @y-f-u in https://github.com/spiceai/spiceai/pull/1975
Remove unused Cargo.toml fields by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1981
Update version to 0.16.0-beta by @ewgenius in https://github.com/spiceai/spiceai/pull/1983
Update spicepod.schema.json by @github-actions in https://github.com/spiceai/spiceai/pull/1984
Enable sqlite acceleration testing in E2E by @sgrebnov in https://github.com/spiceai/spiceai/pull/1980
Revert "Revert "fix: validate time column and time format when constructing accelerated table refresh"" by @y-f-u in https://github.com/spiceai/spiceai/pull/1982
Add Datadog dashboard skeleton by @sgrebnov in https://github.com/spiceai/spiceai/pull/1971
Format Cargo.toml with taplo by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1988
Spice cli spice chat command, to interact with deployed spiced instance in spice.ai cloud by @ewgenius in https://github.com/spiceai/spiceai/pull/1990
Use platform api /v1/chat/completions with streaming in spice chat cli command by @ewgenius in https://github.com/spiceai/spiceai/pull/1998

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.15.2-alpha...v0.16.0-alpha

Resources

Community

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

Twitter: @spice_ai
Slack: spiceai.org/slack
Telegram: Spice AI Discussion
Reddit: https://www.reddit.com/r/spiceai
Email: hey@spice.ai

spiceai/docs/README.md

date: 2024-07-22 title: 'Spice v0.16-alpha (July 22, 2024)' type: blog authors: [lukekim] categories: [release] tags: [secret store, odbc connector, beta milestone, java sdk, federated sql]

Highlights in v0.16-alpha

Secret Stores: More than one Secret Store can now be specified. For example, to configure Spice with both Environment Variable and AWS Secrets Manager Secret Stores, use the following secrets configuration in spicepod.yaml:

Secrets managed by configured Secret Stores can be referenced in component params using the syntax ${<store_name>:<key>}. E.g.

Java Client SDK: The Spice Java SDK has been released for JDK 17 or greater.
Federated SQL Query: Significant stability and reliability improvements have been made to federated SQL query support in most data connectors.
ODBC Data Connector: Providing a specific SQL dialect to query ODBC data sources is now supported using the sql_dialect param. For example, when querying Databricks using ODBC, the databricks dialect can be specified to ensure compatibility. Read the ODBC Data Connector documentation for more details.

Breaking Changes

Secret Stores: Secret Stores support has been overhauled including required changes to spicepod.yml schema. File based secrets stored in the ~/.spice/auth file are no longer supported. See Secret Stores Documentation for full reference.

To upgrade Secret Stores, rename any parameters ending in _key to remove the _key suffix and specify a secret inline via the secret replacement syntax (${<secret_store>:<key>}):

to:

And ensure the MY_PG_PASS environment variable is set.

Datasets: The default value of time_format has changed from unix_seconds to timestamp.

To upgrade:

HTTP Port: The default HTTP port has changed from port 3000 to port 8090 to avoid conflicting with frontend apps which typically use the 3000 range. If an SDK is used, upgrade it at the same time as the runtime.

To upgrade and continue using port 3000, run spiced with the --http command line argument:

HTTP Metrics Port: The default HTTP Metrics port has changed from port 9000 to 9090 to avoid conflicting with other metrics protocols which typically use port 9000.

To upgrade and continue using port 9000, run spiced with the metrics command line argument:

GraphQL Data Connector: json_path has been replaced with json_pointer to access nested data from the result of the GraphQL query. See the GraphQL Data Connector documentation for full details and RFC-6901 - JSON Pointer.

To upgrade, change:

To:

Data Connector Configuration: Consistent connector name prefixing has been applied to connector specific params parameters. Prefixed parameter names helps ensure parameters do not collide.

For example, the Databricks data connector specific params are now prefixed with databricks:

To upgrade:

Refer to the Data Connector documentation for parameter naming changes in this release.

Clickhouse Data Connector: The clickhouse_connection_timeout parameter has been renamed to connection_timeout as it applies to the client and is not Clickhouse configuration itself.

To upgrade, change:

To:

Contributors

@y-f-u
@phillipleblanc
@ewgenius
@github-actions
@sgrebnov
@lukekim
@digadeesh
@peasee
@Sevenannn

What's Changed

Dependencies

No major dependency updates.

Commits

bump helm chart versions to 0.15.2-alpha by @y-f-u in https://github.com/spiceai/spiceai/pull/1975
Remove unused Cargo.toml fields by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1981
Update version to 0.16.0-beta by @ewgenius in https://github.com/spiceai/spiceai/pull/1983
Update spicepod.schema.json by @github-actions in https://github.com/spiceai/spiceai/pull/1984
Enable sqlite acceleration testing in E2E by @sgrebnov in https://github.com/spiceai/spiceai/pull/1980
Revert "Revert "fix: validate time column and time format when constructing accelerated table refresh"" by @y-f-u in https://github.com/spiceai/spiceai/pull/1982
Add Datadog dashboard skeleton by @sgrebnov in https://github.com/spiceai/spiceai/pull/1971
Format Cargo.toml with taplo by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1988
Spice cli spice chat command, to interact with deployed spiced instance in spice.ai cloud by @ewgenius in https://github.com/spiceai/spiceai/pull/1990
Use platform api /v1/chat/completions with streaming in spice chat cli command by @ewgenius in https://github.com/spiceai/spiceai/pull/1998

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.15.2-alpha...v0.16.0-alpha

Resources

Community

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

Twitter: @spice_ai
Slack: spiceai.org/slack
Telegram: Spice AI Discussion
Reddit: https://www.reddit.com/r/spiceai
Email: hey@spice.ai

secrets:
  - from: env
    name: env
  - from: aws_secrets_manager:my_secret_name
    name: aws_secret

secrets:
  - from: env
    name: env
  - from: aws_secrets_manager:my_secret_name
    name: aws_secret

datasets:
  - from: postgres:my_table
    name: my_table
    params:
      pg_host: localhost
      pg_port: 5432
      pg_pass: ${ env:MY_PG_PASS }

datasets:
  - from: postgres:my_table
    name: my_table
    params:
      pg_host: localhost
      pg_port: 5432
      pg_pass: ${ env:MY_PG_PASS }

datasets:
  - from: postgres:my_table
    name: my_table
    params:
      pg_host: localhost
      pg_port: 5432
      pg_pass_key: my_pg_pass

datasets:
  - from: postgres:my_table
    name: my_table
    params:
      pg_host: localhost
      pg_port: 5432
      pg_pass_key: my_pg_pass

datasets:
  - from: postgres:my_table
    name: my_table
    params:
      pg_host: localhost
      pg_port: 5432
      pg_pass: ${secrets:my_pg_pass}

datasets:
  - from: postgres:my_table
    name: my_table
    params:
      pg_host: localhost
      pg_port: 5432
      pg_pass: ${secrets:my_pg_pass}

datasets:
  - from:
    name: my_dataset
    # Explicitly define format when not specified.
    time_format: unix_seconds

datasets:
  - from:
    name: my_dataset
    # Explicitly define format when not specified.
    time_format: unix_seconds

# Using Dockerfile or spiced directly
spiced --http 127.0.0.1:3000

# Using Dockerfile or spiced directly
spiced --http 127.0.0.1:3000

# Using Dockerfile or spiced directly
spiced --metrics 127.0.0.1:9000

# Using Dockerfile or spiced directly
spiced --metrics 127.0.0.1:9000

json_path: my.json.path

json_path: my.json.path

json_pointer: /my/json/pointer

json_pointer: /my/json/pointer

datasets:
  - from: databricks:spiceai.datasets.my_awesome_table # A reference to a table in the Databricks unity catalog
    name: my_delta_lake_table
    params:
      mode: spark_connect
      endpoint: dbc-a1b2345c-d6e7.cloud.databricks.com
      token: MY_TOKEN

datasets:
  - from: databricks:spiceai.datasets.my_awesome_table # A reference to a table in the Databricks unity catalog
    name: my_delta_lake_table
    params:
      mode: spark_connect
      endpoint: dbc-a1b2345c-d6e7.cloud.databricks.com
      token: MY_TOKEN

datasets:
  # Example for Spark Connect
  - from: databricks:spiceai.datasets.my_awesome_table # A reference to a table in the Databricks unity catalog
    name: my_delta_lake_table
    params:
      mode: spark_connect
      databricks_endpoint: dbc-a1b2345c-d6e7.cloud.databricks.com # Now prefixed with databricks
      databricks_token: ${secrets:my_token} # Now prefixed with databricks

datasets:
  # Example for Spark Connect
  - from: databricks:spiceai.datasets.my_awesome_table # A reference to a table in the Databricks unity catalog
    name: my_delta_lake_table
    params:
      mode: spark_connect
      databricks_endpoint: dbc-a1b2345c-d6e7.cloud.databricks.com # Now prefixed with databricks
      databricks_token: ${secrets:my_token} # Now prefixed with databricks

clickhouse_connection_timeout: time

clickhouse_connection_timeout: time

connection_timeout: time

connection_timeout: time