Spice AI Announces Contribution of TableProviders for PostgreSQL, MySQL, DuckDB, and SQLite to the Apache DataFusion Project
Spice AI has contributed new TableProviders for PostgreSQL, MySQL, DuckDB, and SQLite to the Apache DataFusion project. This addition reflects our commitment to building together in the data and AI ecosystem and supporting the open-source community.
What is Apache DataFusion?
Apache DataFusion is a high-performance query engine built on Apache Arrow. It allows you to execute SQL queries quickly and efficiently on data stored in various formats. By using the in-memory columnar format of Apache Arrow, DataFusion speeds up data processing and works natively with other Arrow-based tools.
About the Spice OSS Project
Spice OSS is an open-source project from Spice AI that provides developers with a unified SQL query interface to locally materialize, accelerate, and query datasets from any database, data warehouse, or data lake. Spice OSS incorporates Apache DataFusion as its SQL query engine.
Our goal with Spice OSS is to make data and AI-driven development more accessible. By contributing to projects like DataFusion and Arrow, we can help making accessing and using data better for everyone building in the space.
New TableProviders for DataFusion
We've initially added TableProviders for PostgreSQL, MySQL, DuckDB, and SQLite to DataFusion. This expands the range of data sources you can query using DataFusion. And we plan to add more in the future.
- PostgreSQL: A robust and extensible open-source relational database.
- MySQL: A reliable and user-friendly open-source relational database.
- DuckDB: An in-process SQL OLAP database for analytical queries.
- SQLite: A lightweight, disk-based database commonly used in embedded systems.
These new TableProviders make DataFusion even more versatile, allowing you to work with your existing databases and data lakes more easily.
Our Commitment to Data and AI
At Spice AI, we believe in the power of open-source and the potential of data and AI. By contributing to Apache DataFusion, we're helping to advance data processing technology and make powerful tools available to developers everywhere.
To learn more about Spice AI and our open-source projects, check out our GitHub repository. You can also explore Apache DataFusion at datafusion.apache.org.
Stay tuned for more updates from Spice AI as we continue to contribute to the data and AI ecosystem.
Get the latest insights
New releases, tutorials, platform updates, and more.
Explore more Spice resources
Tutorials, docs, and blog posts to help you go deeper with Spice.
Real-Time Control Plane Acceleration with DynamoDB Streams
How to sync DynamoDB data to thousands of nodes with sub-second latency using a two-tier architecture with DynamoDB Streams and Spice acceleration.
How we use Apache DataFusion at Spice AI
Why we chose to build on DataFusion and how we extended it with custom TableProviders, optimizer rules, and UDFs for federated SQL
Real-Time Hybrid Search Using RRF: A Hands-On Guide with Spice
Surfacing relevant answers to searches across datasets has historically meant navigating significant tradeoffs. Keyword (or lexical) search is fast, cheap, and commoditized, but limited by the constraints of exact matching. Vector (or semantic) search captures nuance and intent, but can be slower, harder to debug, and expensive to run at scale. Combining both usually entails standing up multiple engines […]
See Spice in action
Get a guided walkthrough of how development teams use Spice to query, accelerate, and integrate AI for mission-critical workloads.
Get a demo
