Announcing the release of Spice v1.6.1! ⚡
Spice 1.6.1 is a patch release that provides improved Kafka type inference and JSON flattening support, alongside several bug fixes.
Improved Kafka Type Inference: Improve Kafka type inference by configuring the number of Kafka messages sampled during schema inference. Increasing the sample size can improve the robustness and reliability of inferred schemas, especially in cases where data contains optional fields or varying structures.
Example spicepod.yml:
For details, see the Kafka Data Connector Documentation.
Improved Kafka JSON Support: Enable nested JSON Kafka messages to be represented in flattened JSON format for the dataset schema.
Example spicepod.yml:
For example, the object:
With flatten_json: true the result is:
With flatten_json: false or ommitted the result is:
For details, see the Kafka Data Connector Documentation.
No breaking changes.
No new cookbook recipes added in this release.
The Spice Cookbook includes 77 recipes to help you get started with Spice quickly and easily.
To upgrade to v1.6.1, use one of the following methods:
CLI:
Homebrew:
Docker:
Pull the spiceai/spiceai:1.6.1 image:
For available tags, see DockerHub.
Helm:
AWS Marketplace:
🎉 Spice is now available in the AWS Marketplace!
dataset:
- from: kafka:orders_events
name: orders
params:
schema_infer_max_records: 100 # Default 1.
dataset:
- from: kafka:orders_events
name: orders
params:
schema_infer_max_records: 100 # Default 1.
dataset:
- from: kafka:orders_events
name: orders
params:
flatten_json: true # default false
dataset:
- from: kafka:orders_events
name: orders
params:
flatten_json: true # default false
+------------------------+-----------+-------------+
| column_name | data_type | is_nullable |
+------------------------+-----------+-------------+
| order_id | Utf8 | YES |
| customer.id | Int64 | YES |
| customer.name | Utf8 | YES |
| customer.premium | Boolean | YES |
| customer.contact.email | Utf8 | YES |
| customer.contact.phone | Utf8 | YES |
| discount | Float64 | YES |
| shipped | Boolean | YES |
+------------------------+-----------+-------------+
+------------------------+-----------+-------------+
| column_name | data_type | is_nullable |
+------------------------+-----------+-------------+
| order_id | Utf8 | YES |
| customer.id | Int64 | YES |
| customer.name | Utf8 | YES |
| customer.premium | Boolean | YES |
| customer.contact.email | Utf8 | YES |
| customer.contact.phone | Utf8 | YES |
| discount | Float64 | YES |
| shipped | Boolean | YES |
+------------------------+-----------+-------------+
+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+
| column_name | data_type | is_nullable |
+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+
| order_id | Utf8 | YES |
| customer | Struct([Field { name: "id", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "name", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "premium", data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "contact", data_type: Struct([Field { name: "email", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "phone", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]) | YES |
| discount | Float64 | YES |
| shipped | Boolean | YES |
+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+
+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+
| column_name | data_type | is_nullable |
+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+
| order_id | Utf8 | YES |
| customer | Struct([Field { name: "id", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "name", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "premium", data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "contact", data_type: Struct([Field { name: "email", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "phone", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]) | YES |
| discount | Float64 | YES |
| shipped | Boolean | YES |
+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+
spice upgrade
spice upgrade
brew upgrade spiceai/spiceai/spice
brew upgrade spiceai/spiceai/spice
docker pull spiceai/spiceai:1.6.1
docker pull spiceai/spiceai:1.6.1
helm repo update
helm upgrade spiceai spiceai/spiceai
helm repo update
helm upgrade spiceai spiceai/spiceai
{
"order_id": "a1f2c3d4-1111-2222-3333-444455556666",
"customer": {
"id": 101,
"name": "Alice",
"premium": true,
"contact": {
"email": "alice@example.com",
"phone": "555-1234"
}
},
"discount": 5.0,
"shipped": false
}
{
"order_id": "a1f2c3d4-1111-2222-3333-444455556666",
"customer": {
"id": 101,
"name": "Alice",
"premium": true,
"contact": {
"email": "alice@example.com",
"phone": "555-1234"
}
},
"discount": 5.0,
"shipped": false
}