1---
2date: 2024-03-28
3title: 'Adding Spice - The Next Generation of Spice.ai OSS'
4type: blog
5linkTitle: 'Adding Spice'
6authors: [phillipleblanc]
7categories: [spiceai]
8---
9
10import ReactPlayer from 'react-player';
11
12**TL;DR:** We've rebuilt [Spice.ai OSS](https://github.com/spiceai/spiceai) from the ground up in Rust, as a unified SQL query interface and portable runtime to locally materialize, accelerate, and query datasets sourced from any database, data warehouse or data lake. Learn more at [github.com/spiceai/spiceai](https://github.com/spiceai/spiceai).
13
14In September, 2021, we [introduced](https://blog.spiceai.org/posts/2021/09/07/introducing-spice.ai-open-source-time-series-ai-for-developers/) Spice.ai OSS as a runtime for building AI-driven applications using time-series data.
15
16We quickly ran into a big problems in making these applications work... data, the fuel for intelligent software, was painfully difficult to access, operationalize, and use, not only in machine learning, but also in web frontends, backend applications, dashboards, data pipelines, and notebooks. And we had to make hard tradeoffs between cost and query performance.
17
18We felt this pain every day building 100TB+ scale data and AI systems for the [Spice.ai Cloud Platform](https://spice.ai/). So we took our learnings and infused them back into Spice.ai OSS with the capabilities we wished we had.
19
20We rebuilt Spice.ai OSS from the ground up in Rust, as a unified SQL query interface and portable runtime to locally materialize, accelerate, and query data tables sourced from any database, data warehouse or data lake.
21
22
23
24Spice is a fast, lightweight (< 150Mb), single-binary, designed to be deployed alongside your application, dashboard, and within your data or machine learning pipelines. Spice federates SQL query across databases (MySQL, PostgreSQL, etc.), data warehouses (Snowflake, BigQuery, etc.) and data lakes (S3, MinIO, Databricks, etc.) so you can easily use and combine data wherever it lives. Datasets, declaratively defined, can be materialized and accelerated using your engine of choice, including DuckDB, SQLite, PostgreSQL, and in-memory Apache Arrow records, for ultra-fast, low-latency query. Accelerated engines run in your infrastructure giving you flexibility and control over price and performance.
25
26### Before Spice
27
28
29Figure 2. Before Spice, applications submit many queries to external data sources.
30
31### With Spice
32
33
34
35### Use-Cases
36
37The next-generation of Spice.ai OSS enables:
38
39**Better applications.** Accelerate and co-locate data with frontend and backend applications, for high concurrent queries, serving more users with faster page loads and data updates. [Try the CQRS sample app](https://github.com/spiceai/samples/tree/trunk/acceleration#local-materialization-and-acceleration-cqrs-sample).
40
41**Snappy dashboards, analytics, and BI.** Faster, more responsive dashboards without massive compute costs. Spice supports Arrow Flight SQL (JDBC/ODBC/ADBC) for connectivity with Tableau, Looker, PowerBI, and more. [Watch the Apache Superset with Spice demo.](https://github.com/spiceai/samples/blob/trunk/sales-bi/README.md)
42
43**Faster data pipelines, machine learning training and inference.** Co-locate datasets with pipelines where the data is needed to minimize data-movement and improve query performance. [Predict hard drive failure with the SMART data demo.](https://github.com/spiceai/demos/tree/trunk/smart-demo#spiceai-smart-demo)
44
45**Easily query many data sources.** Federated SQL query across databases, data warehouses, and data lakes using [Data Connectors](https://docs.spiceai.org/data-connectors).
46
47### Community Built
48
49Spice is open-source, Apache 2.0 licensed, and is built using industry-leading technologies including Apache DataFusion, Arrow, and Arrow Flight SQL. We're launching with several built-in [Data Connectors](https://docs.spiceai.org/data-connectors) and [Accelerators](https://docs.spiceai.org/data-accelerators) and Spice is extensible so more will be added in each release. If you're interested in contributing, we'd love to welcome you to the community!
50
51### Getting Started
52
53You can download and run Spice in less than 30 seconds by following the quickstart at [github.com/spiceai/spiceai](https://github.com/spiceai/spiceai#quickstart).
54
55<div style={{display: 'flex', justifyContent: 'center', marginBottom: '15px'}}>
56 <ReactPlayer
57 controls
58 url='https://www.youtube.com/watch?v=t3B2AyshVY0&list=PLesJrUXEx3U-dQul0PqLV3TGTdUmr3B6e&index=1&pp=gAQBiAQB'
59 />
60</div>
61
62### Conclusion
63
64Spice, rebuilt in Rust, introduces a unified SQL query interface, making it simpler and faster to build data-driven applications. The lightweight Spice runtime is easy to deploy and makes it possible to materialize and query data from any source quickly and cost-effectively. Applications can serve more users, dashboards and analytics can be snappier, and data and ML pipelines finish faster, without the heavy lifting of managing data.
65
66For developers this translates to less time wrangling data and more time creating innovative applications and business value.
67
68Check out and [star the project on GitHub](https://github.com/spiceai/spiceai)!
69
70Thank you,
71
72Phillip
73
1---
2date: 2024-03-28
3title: 'Adding Spice - The Next Generation of Spice.ai OSS'
4type: blog
5linkTitle: 'Adding Spice'
6authors: [phillipleblanc]
7categories: [spiceai]
8---
9
10import ReactPlayer from 'react-player';
11
12**TL;DR:** We've rebuilt [Spice.ai OSS](https://github.com/spiceai/spiceai) from the ground up in Rust, as a unified SQL query interface and portable runtime to locally materialize, accelerate, and query datasets sourced from any database, data warehouse or data lake. Learn more at [github.com/spiceai/spiceai](https://github.com/spiceai/spiceai).
13
14In September, 2021, we [introduced](https://blog.spiceai.org/posts/2021/09/07/introducing-spice.ai-open-source-time-series-ai-for-developers/) Spice.ai OSS as a runtime for building AI-driven applications using time-series data.
15
16We quickly ran into a big problems in making these applications work... data, the fuel for intelligent software, was painfully difficult to access, operationalize, and use, not only in machine learning, but also in web frontends, backend applications, dashboards, data pipelines, and notebooks. And we had to make hard tradeoffs between cost and query performance.
17
18We felt this pain every day building 100TB+ scale data and AI systems for the [Spice.ai Cloud Platform](https://spice.ai/). So we took our learnings and infused them back into Spice.ai OSS with the capabilities we wished we had.
19
20We rebuilt Spice.ai OSS from the ground up in Rust, as a unified SQL query interface and portable runtime to locally materialize, accelerate, and query data tables sourced from any database, data warehouse or data lake.
21
22
23
24Spice is a fast, lightweight (< 150Mb), single-binary, designed to be deployed alongside your application, dashboard, and within your data or machine learning pipelines. Spice federates SQL query across databases (MySQL, PostgreSQL, etc.), data warehouses (Snowflake, BigQuery, etc.) and data lakes (S3, MinIO, Databricks, etc.) so you can easily use and combine data wherever it lives. Datasets, declaratively defined, can be materialized and accelerated using your engine of choice, including DuckDB, SQLite, PostgreSQL, and in-memory Apache Arrow records, for ultra-fast, low-latency query. Accelerated engines run in your infrastructure giving you flexibility and control over price and performance.
25
26### Before Spice
27
28
29Figure 2. Before Spice, applications submit many queries to external data sources.
30
31### With Spice
32
33
34
35### Use-Cases
36
37The next-generation of Spice.ai OSS enables:
38
39**Better applications.** Accelerate and co-locate data with frontend and backend applications, for high concurrent queries, serving more users with faster page loads and data updates. [Try the CQRS sample app](https://github.com/spiceai/samples/tree/trunk/acceleration#local-materialization-and-acceleration-cqrs-sample).
40
41**Snappy dashboards, analytics, and BI.** Faster, more responsive dashboards without massive compute costs. Spice supports Arrow Flight SQL (JDBC/ODBC/ADBC) for connectivity with Tableau, Looker, PowerBI, and more. [Watch the Apache Superset with Spice demo.](https://github.com/spiceai/samples/blob/trunk/sales-bi/README.md)
42
43**Faster data pipelines, machine learning training and inference.** Co-locate datasets with pipelines where the data is needed to minimize data-movement and improve query performance. [Predict hard drive failure with the SMART data demo.](https://github.com/spiceai/demos/tree/trunk/smart-demo#spiceai-smart-demo)
44
45**Easily query many data sources.** Federated SQL query across databases, data warehouses, and data lakes using [Data Connectors](https://docs.spiceai.org/data-connectors).
46
47### Community Built
48
49Spice is open-source, Apache 2.0 licensed, and is built using industry-leading technologies including Apache DataFusion, Arrow, and Arrow Flight SQL. We're launching with several built-in [Data Connectors](https://docs.spiceai.org/data-connectors) and [Accelerators](https://docs.spiceai.org/data-accelerators) and Spice is extensible so more will be added in each release. If you're interested in contributing, we'd love to welcome you to the community!
50
51### Getting Started
52
53You can download and run Spice in less than 30 seconds by following the quickstart at [github.com/spiceai/spiceai](https://github.com/spiceai/spiceai#quickstart).
54
55<div style={{display: 'flex', justifyContent: 'center', marginBottom: '15px'}}>
56 <ReactPlayer
57 controls
58 url='https://www.youtube.com/watch?v=t3B2AyshVY0&list=PLesJrUXEx3U-dQul0PqLV3TGTdUmr3B6e&index=1&pp=gAQBiAQB'
59 />
60</div>
61
62### Conclusion
63
64Spice, rebuilt in Rust, introduces a unified SQL query interface, making it simpler and faster to build data-driven applications. The lightweight Spice runtime is easy to deploy and makes it possible to materialize and query data from any source quickly and cost-effectively. Applications can serve more users, dashboards and analytics can be snappier, and data and ML pipelines finish faster, without the heavy lifting of managing data.
65
66For developers this translates to less time wrangling data and more time creating innovative applications and business value.
67
68Check out and [star the project on GitHub](https://github.com/spiceai/spiceai)!
69
70Thank you,
71
72Phillip
73