Data lake accelerator: low-latency SQL on S3, Iceberg, and Delta Lake

How does a data lake accelerator work?

A data lake accelerator copies the tables an application queries most from object storage into a local query engine, then keeps that copy current. Spice does this per dataset, in three steps defined in a Spicepod:

1. Connect the dataset to its source with a data connector: S3, Azure Blob Storage, Google Cloud Storage, Apache Iceberg, AWS Glue, Delta Lake, Databricks, or DuckLake.
2. Materialize the dataset in an acceleration engine. Most engines are embedded in the Spice runtime, next to the application.
3. Refresh the local copy on an interval or from a change stream, with the refresh mode that matches how the source data changes. In caching mode, Spice instead caches query results, reads the source on a miss, and revalidates cached results in the background.

Applications query accelerated and non-accelerated datasets through the same SQL endpoint. Datasets without acceleration stay federated, so one query can join a local copy of an Iceberg table with a live PostgreSQL table. The SQL federation and acceleration engine plans both paths, and the data acceleration guide covers the general pattern.

datasets:
  - from: s3://my-bucket/events/
    name: events
    time_column: event_time
    params:
      file_format: parquet
    acceleration:
      enabled: true
      engine: cayenne
      mode: file
      refresh_mode: append
      refresh_check_interval: 1m

Which acceleration engine fits a data lake workload?

Pick the engine by dataset size and query shape. Applications query every engine through the same Spice SQL endpoint. Switching engines changes the engine setting and engine-specific settings such as mode, because Arrow runs in memory only and Cayenne runs on files only. Queries that call engine-specific SQL functions can also need changes.

Spice Cayenne stores data in the Vortex columnar format and keeps metadata in a separate embedded engine, which avoids the single-file limits of DuckDB and SQLite. On the TPC-H SF-100 query set, Spice 2.0 with Cayenne finishes in 88.3 seconds on one node, compared with 138.3 seconds for the same runtime with DuckDB. The Spice benchmarks page lists the hardware and runtime version for each run.

Which refresh mode should a data lake dataset use?

Match the refresh mode to how the source table changes:

Two settings bound the size of the local copy. refresh_sql filters what each refresh loads, for example the last 30 days of a multi-year table. Retention policies evict rows from the local copy: retention_period removes rows older than a set age, based on the dataset's time_column, and retention_sql removes rows that match a DELETE predicate. Eviction runs only when retention_check_enabled is true and retention_check_interval is set.

When should you query the data lake directly instead?

Leave a dataset federated when queries scan its full history, run once, or touch data that no application reads repeatedly. Acceleration pays off on the working set that applications query again and again. For large scans of federated object storage, distributed query spreads the work across a cluster: on TPC-H SF-100 over federated S3 Parquet, three Ballista executors finish 2.9x faster than one node. To serve both patterns from one platform, see the operational data lakehouse use case. To run data lake acceleration as a managed service, compare Spice Cloud plans and pricing.