Migrating Off Dremio: A Phased Guide for Data Teams

TL;DR

Spice.ai offers an open-source, deploy-anywhere alternative to Dremio that is purpose-built for data-intensive applications and AI agents.

Development teams can take a phased migration path off Dremio without a forced cutover:

• Run side-by-side. Deploy Spice alongside Dremio; Spice can federate queries through Dremio's Arrow Flight endpoint while you incrementally move workloads.
• Migrate sources one at a time. Move each dataset and its associated Reflections to Spice, validating results while Dremio stays available as a reference. Layer on additional Spice capabilities as your use cases require.
• Decommission Dremio. When the last dataset is off Dremio, shut it down.

Each phase changes one thing at a time, making it straightforward to validate correctness, compare performance, and isolate any issues before moving on.

The remainder of this guide introduces Spice, explains how Dremio concepts map to Spice and the areas that require planning, and then walks through each phase of the migration.

What is Spice.ai

Spice.ai is an open-source (Apache 2.0) SQL query, hybrid search, and AI-inference engine written in Rust. Point Spice at your data sources, define what you want to accelerate, and Spice handles federation, materialization, caching, CDC, search, and LLM inference through a unified SQL interface.

At a high level, Spice offers:

• Deploy-anywhere. A single binary or container that runs on a laptop, bare metal, any cloud, or Kubernetes. There's no mandatory coordinator-executor topology.
• 40+ data connectors. Postgres, MySQL, MongoDB, Snowflake, Databricks, BigQuery, S3/Iceberg, Delta Lake, DuckDB, Dremio, and more.
• High-throughput replication. Built-in CDC from PostgreSQL, MySQL, MongoDB, and DynamoDB. Deploy a Spice analytics replica alongside your operational database in minutes that absorbs large analytical queries while your database keeps serving transactions.
• Sub-second acceleration. Cayenne, Spice's columnar accelerator, materializes working sets in-memory or on disk for millisecond query performance on data that's seconds fresh.
• AI and search. Embedded LLM inference, hybrid search, text-to-SQL, and OpenAI-compatible APIs in the same runtime.
• Built on open standards. Apache DataFusion, Apache Arrow, Apache Arrow Flight, Vortex, Apache Iceberg, ADBC, and more.

For more details, see docs.spiceai.org or the GitHub repo.

Migration guide

Phase 0: Plan the migration

Before moving workloads, review which Dremio concepts map directly to Spice and which require planning.

What maps cleanly

• Standard SQL. ANSI SQL, common aggregation functions, joins, window functions, CTEs, subqueries.
• Arrow Flight / Flight SQL clients. If your tools connect to Dremio via Arrow Flight, they will also connect to Spice.
• Iceberg table reads. Spice reads Iceberg natively through the same catalogs Dremio uses: REST catalog or AWS Glue.
• S3 / ADLS / GCS Parquet files. Direct object storage reads work.
• JDBC / ODBC / ADBC connections. Standard BI tool connectivity.

See Appendix: Dremio -> Spice concept map for details on how specific Dremio concepts map to Spice.

What needs planning

• Dremio-specific SQL functions. Functions like CONVERT_FROM, FLATTEN, and Dremio-specific date/time handling don't exist in DataFusion (the query engine Spice is built on). Audit your query inventory and map them to DataFusion equivalents or define SQL UDFs. See Appendix: Dremio -> Spice Function Mapping.
• VDS hierarchies. Dremio's nested Virtual Dataset layers don't have a one-to-one equivalent. Flatten them into Spice views or materialized datasets. This is usually an improvement: deep VDS chains are a common source of planning overhead in Dremio.
• Reflection management. Dremio's automated Reflection recommendations don't exist in Spice. You explicitly declare what to accelerate in a Spicepod (a configuration package that defines application-specific datasets, catalogs, models, and secrets). More transparent and predictable, but the acceleration decisions are yours. Step 2b discusses how to replace raw and aggregation Reflections with Spice equivalents.
• Dremio wiki and dataset documentation. If you use Dremio's built-in dataset descriptions and wiki, plan to move that metadata into your catalog or docs system.
• Fine-grained security policies. Re-implement Dremio row filters and column masks explicitly. Spice.ai Enterprise provides OIDC-authenticated Cedar policies for row filtering and column masking. For OSS deployments, preserve those controls at the source or isolate datasets in separate runtimes. mTLS secures transport; it does not replace authorization.
• Arctic / Nessie catalog management. If you use Dremio-managed Nessie catalogs, point Spice at your Iceberg REST catalog (or Polaris) directly. Catalog branching and tagging are catalog-side operations, not query-engine-side.
• Autoscaling. Dremio Cloud's elastic engine scaling has a different operational model. In Spice, use your infrastructure's scaling mechanisms (Kubernetes HPA, cloud autoscaling groups) around the spiced process (the Spice.ai runtime binary).

Once you have a full audit of the areas unique to your environment, you're ready to begin the migration.

Phase 1: Run side-by-side

Deploy Spice alongside your existing Dremio cluster. Use Spice's Dremio connector to federate queries through Dremio's Arrow Flight endpoint. Your applications keep working exactly as they do today. Spice is just another query endpoint you can validate against.

# Phase 1: federate through Dremio
datasets:
  - from: dremio:my_schema.orders
    name: orders
    params:
      dremio_endpoint: grpc://dremio-coordinator:32010
      dremio_username: ${secrets:DREMIO_USER}
      dremio_password: ${secrets:DREMIO_PASS}

At this point you can move queries to using the data available within Spice. Most queries using standard SQL should work as they are. For non-standard SQL functions you may be using, review the Dremio -> Spice Function Mapping.

What to validate at this stage:

• Result correctness. Run your query suite against both endpoints. Compare result sets.
• Client compatibility. Connect your BI tools (Tableau, Grafana, Superset, etc.) to Spice's Flight SQL endpoint and confirm they work.
• Latency baseline. Measure query latency through Spice -> Dremio. Latency should be roughly equivalent to querying Dremio directly, since Spice is passing through.

Phase 2: Migrate sources one at a time

For each dataset sourced from Dremio, repeat the following steps:

• Point Spice dataset at the underlying source (Postgres, Iceberg, S3, Snowflake, etc.).
• Replace Dremio Reflections with Spice Accelerations and Accelerated Views.
• Validate and layer on new capabilities.

As an example, we'll apply these steps to the my_schema.orders table from phase 1.

Step 2a: Point Spice at the underlying source

Swap from: dremio:schema.table to from: postgres:public.table (e.g. Iceberg or Snowflake). This removes the Dremio hop for that dataset. Queries still work, they just now federate to the underlying source instead of through Dremio.

# Phase 2a: direct source
datasets:
  - from: postgres:public.orders
    name: orders
    params:
      pg_connection_string: ${secrets:PG_CONN}

Validation: When you swap a dataset from from: dremio:schema.table to a direct source connection, the only thing that changes is where the data comes from. Validate that the data is the same: compare results, the schema, and measure query latency against the direct source. Without acceleration, federated queries go straight to the source - expect latency comparable to querying the source directly. This is your baseline for the next step.

Step 2b: Replace Dremio Reflections with Spice accelerations and accelerated views

For each Dremio Reflection on that source's datasets, add the equivalent Spice acceleration.

Raw Reflections become Spice accelerations. Enable acceleration on the dataset and choose an engine and refresh strategy:

# Phase 2b: replace a raw Reflection
datasets:
  - from: postgres:public.orders
    name: orders
    params:
      pg_connection_string: ${secrets:PG_CONN}

    acceleration:
      enabled: true
      engine: cayenne # or: arrow, duckdb, sqlite, postgres, turso
      mode: file
      refresh_mode: full
      refresh_check_interval: 10m

Aggregation Reflections become accelerated views. Spice does not transparently rewrite existing queries to use an accelerated view, so update affected workloads to query the named view explicitly. Then define and accelerate the view:

# Phase 2b: replace an aggregation Reflection
views:
  - name: orders_daily_summary
    sql: |
      SELECT
        DATE_TRUNC('day', order_date) AS day,
        region,
        COUNT(*) AS order_count,
        SUM(total) AS revenue
      FROM orders
      GROUP BY 1, 2
    acceleration:
      enabled: true
      engine: cayenne

Validation: When you add Spice accelerations to replace Dremio Reflections, you're swapping one materialization layer for another. Dremio is still running, so you can A/B them directly. Compare the latency, verify that accelerations pick up source changes within the configured interval, and verify that the rollup logic for accelerated views produces the same totals. Pay special attention to NULL handling, floating-point precision, and timezone-sensitive DATE_TRUNC boundaries.

Step 2c: Layer on new capabilities

Once the source is migrated and accelerations are validated, layer on capabilities Dremio didn't provide as your use cases demand.

CDC for sub-second freshness. Spice captures changes directly from the source database's replication stream - PostgreSQL WAL, MongoDB change streams, MySQL binlog, or DynamoDB streams - and applies them to the local acceleration without an external pipeline (Kafka, Flink, Debezium) required. Enable CDC by setting refresh_mode: changes on an accelerated dataset. Measure freshness by inserting a row into the source and observing how long until it's queryable in Spice. Under normal load, Cayenne CDC from PostgreSQL typically delivers sub-two-second freshness. This replaces the pattern where teams ran a separate ingestion pipeline alongside Dremio to keep a materialized view current.

Hybrid search. Spice supports vector search, full-text search, keyword search, and reciprocal rank fusion (RRF) ranking - all queryable via SQL UDTFs (vector_search, text_search, rrf, rerank). Enable hybrid search by adding an embedding model to the Spicepod and configuring a vector index on the target column (search docs). This means datasets that previously required a separate search index (Elasticsearch, OpenSearch, Pinecone) can be searched and queried in the same engine, against the same data, with results that are always consistent with the latest acceleration refresh.

Embedded inference. Spice includes an OpenAI-compatible AI gateway that routes to hosted models (OpenAI, Anthropic, xAI, Amazon Bedrock) or runs local models on CUDA/Metal. Configure a model in the Spicepod and call it via the /v1/chat/completions HTTP API or from SQL UDFs (AI gateway docs). This replaces the pattern where applications queried Dremio for data and then made a separate call to an LLM API - Spice serves both from the same runtime, with the data already colocated.

Phase 3: Decommission Dremio

Decommission Dremio only after every dataset, VDS/view, client, scheduled job, security policy, and catalog dependency has migrated and the Dremio endpoint shows no production traffic for an agreed observation period. Retain backups and a rollback plan until validation is complete.

Getting started

A Dremio migration does not need to be a single cutover. Start by deploying Spice alongside Dremio, move one dataset at a time, and validate each change before continuing.

This migration plan stays incremental and limits the scope at every step. If a query result, schema, or performance characteristic diverges, you can isolate the cause to the dataset or migration step that just changed.

To begin, install Spice in minutes and configure one representative dataset:

# Install
curl https://install.spiceai.org | /bin/bash

# Initialize a project
spice init my-migration
cd my-migration

# Configure your first dataset
spice dataset configure

# Start the runtime
spice run

# Query
spice sql
sql> SELECT count(*) FROM orders WHERE order_date > '2026-01-01';

From there, use the migration phases in this guide to move the next dataset and validate the results.

We'd love to hear from you in the Spice Community Slack channel if you have any questions or need help getting started.

Resources

• Spice.ai OSS docs
• GitHub repo - Apache 2.0
• Data Connectors reference
• Spicepod reference
• Spice.ai Cloud Platform - managed option
• Talk to an engineer - walk through your specific deployment

Appendix: Dremio -> Spice concept map

If you're coming from Dremio, here's how the core concepts translate.

Sources and Federation

Reflections / Materialization

Query Engine and Connectivity

Security and Operations

Appendix: Dremio -> Spice Function Mapping

This table covers the most common Dremio-specific functions. Audit your full query inventory for anything not listed. See Spice AI documentation for available Scalar Functions, Aggregate Functions, etc.