Apache Iceberg at Spice AI: How we Query, Accelerate, and Write to Open Table Formats

Spice AI

Spice Open-Source

Data Federation

Data Acceleration

Open-Table-Formats

Phillip LeBlanc

Phillip LeBlanc

Co-Founder and CTO of Spice AIFebruary 25, 2026
Apache Iceberg at Spice AI

Apache Iceberg has become the default open table format for production data lakes, bringing ACID transactions, schema evolution, and time travel to data stored in object storage. But Iceberg alone doesn't solve the performance problem: querying data in object storage like S3 carries inherent latency that's too high for many application workloads.

At Spice AI, we integrate Iceberg as a first-class data source - customers can connect to Iceberg catalogs and query tables with full SQL semantics, then selectively accelerate hot datasets for sub-millisecond reads. This post explains how that integration works, from connecting your first catalog to the query engine internals.

What this post covers:

  • What Apache Iceberg is and why it matters for data lakes
  • The latency gap that Iceberg leaves open for application workloads
  • How Spice connects to Iceberg catalogs, accelerates tables, and writes data back
  • How the query engine works under the hood (Apache DataFusion, catalog providers, query optimization)
  • Production lessons from running Iceberg at scale

This article is the second part of our Engineering at Spice AI series, where we share technical deep-dives into the open-source technologies and practices that power the Spice.ai compute engine.

  1. Apache DataFusion at Spice AI: The query engine
  2. Apache Iceberg at Spice AI: Open table format and SQL-based ingestion
  3. Rust at Spice AI: The systems programming foundation
  4. Apache Arrow at Spice AI: The core data format
  5. DuckDB at Spice AI: Embedded analytics acceleration
  6. Vortex at Spice AI: Columnar compression for Cayenne, Spice's premier data accelerator

Want to skip ahead? The Iceberg catalog connector docs walk through connecting a local REST catalog in under 5 minutes. The rest of this post explains how the integration works under the hood.

What is Apache Iceberg?

Apache Iceberg is an open table format - a specification for how to organize metadata and data files so that data stored in object storage (S3, GCS, HDFS) can support database-level properties like ACID transactions, schema evolution, and time travel. It's not a storage system or a query engine, but a layer that sits between the two.

Unlike the older Hive approach, which treats directories as tables and relies on naming conventions for partitioning, Iceberg maintains explicit metadata files that track:

  • Current and historical table schemas - so readers can handle schema changes without rewriting data
  • Which data files belong to the table - including which rows have been deleted
  • Partition specifications and their evolution - old and new partitioning schemes coexist
  • Immutable snapshots - for time travel queries and concurrent-write safety
  • File-level and column-level statistics - for pruning data files before reading them

These capabilities are organized into three layers:

  • The metadata layer: JSON and Avro files tracking table schema, partitions, and data files
  • The data layer: Parquet files containing the actual data in object storage
  • A catalog API: The standard interface for table discovery and atomic updates
Iceberg Catalog(REST, Glue, Hive, Hadoop) Table Metadata(Schema, Partitions, Snapshots) Data Files(Parquet in object storage, e.g. S3)

This metadata-driven design is what gives Iceberg its core strengths. We'll return to specific features - ACID transactions, hidden partitioning, schema evolution, time travel, and file-level pruning - later in this post, in the context of how Spice uses them.

The latency gap: Why Iceberg alone isn't enough for applications

Iceberg solves the thorny data lake consistency and reliability challenges, but there's still an inherent performance gap for latency-sensitive workloads. Object storage like S3 carries 50-200ms per-request latency by design, and resolving an Iceberg query requires multiple metadata round-trips (manifest list, manifests, then data files) before touching a single row. Parquet itself is optimized for bulk analytical scans, not the high-concurrency, low-cardinality reads that application and increasingly AI agent workloads demand.

Consider a concrete example: a 10TB Iceberg table of user events, partitioned by date, stored in S3. Your data team runs ad-hoc queries over years of data - Iceberg's partition pruning makes this efficient. But your application serves a dashboard that queries the last 7 days of data, thousands of times per second.

The common workaround is to copy hot data into a faster system (ClickHouse, Redis, etc.) and introduce an ETL pipeline to keep it in sync with the lake. This works, but it introduces drift between systems, duplicates storage costs, and adds another operational surface to maintain.

This is the gap Spice is designed to fill.

How Spice.ai bridges the gap

Spice.ai is a data compute engine that lets you query data across sources using standard SQL. It connects to databases, data lakes, and warehouses - including Iceberg catalogs - and presents them through a single query interface. The key capability for Iceberg workloads is acceleration: Spice can materialize a hot working set into a faster local engine (DuckDB, SQLite, or Spice Cayenne), then keep it automatically synchronized with the source. Iceberg stays the authoritative source of truth; Spice handles the caching and refresh logic so you don't have to build a separate pipeline.

Spice is built on Apache DataFusion for query planning and execution, and on Apache Arrow as the in-memory columnar format. We'll cover the DataFusion internals later in this post; for now, the important thing is that Spice treats Iceberg as a first-class catalog - connect once, query across data sources, then choose which datasets to accelerate.

event_date max < 2024-01-01 user_id out of range Predicate matches Read Manifests Check Statistics File matches predicate? Skip file Scan file

Spice supports several deployment topologies. The sidecar pattern works well for workloads that need low-latency and high-concurrency. In Kubernetes, Spice runs as a second container in the same pod as your application. Communication goes over localhost instead of the network, which removes round-trip overhead. Each pod has its own copy of the accelerated data, so reads never leave the machine and each instance fails independently. The trade-off is resource duplication: every pod stores its own copy, which costs extra memory and compute. This works best when scale is moderate and sub-millisecond reads justify the extra resources - for example, a dashboard serving thousands of concurrent users.

Cold queries(full history, ad-hoc) Hot queries(recent data, dashboards) Application LayerSQL queries via Spice Spice Query Engine(Apache DataFusion + acceleration layer) Iceberg on S3(full dataset)10TB historicalPartition pruningQuery in 500ms-5s DuckDB Acceleration(last 7 days, cached)50GB on local NVMeSub-10ms queriesAuto-refresh

Connecting Iceberg to Spice

Configuration in Spice uses the spicepod - a YAML file that declares which data sources to connect and how to access them. To connect an Iceberg catalog, add a catalogs entry pointing to the catalog's REST endpoint (or Glue, or Hadoop):

catalogs:
  - from: iceberg:http://localhost:8181/v1/namespaces
    name: ice
    params:
      iceberg_s3_endpoint: http://localhost:9000
      iceberg_s3_access_key_id: admin
      iceberg_s3_secret_access_key: password

On startup, Spice connects to the catalog, discovers all namespaces (databases) and tables, and registers them for SQL access:

spice run
# 2025-01-27T19:08:37Z  INFO Registered catalog 'ice' with 1 schema and 8 tables

Every table is now queryable - no additional configuration needed:

sql> SHOW TABLES;

+--------+-------+------+
| table_catalog | table_schema | table_name |
+--------+-------+------+
| ice           | tpch_sf1     | lineitem   |
| ice           | tpch_sf1     | nation     |
| ice           | tpch_sf1     | orders     |
| ice           | tpch_sf1     | customer   |
+--------+-------+------+

sql> SELECT COUNT(*) FROM ice.tpch_sf1.lineitem;

+------+
| count(*)  |
+------+
| 6001215   |
+------+
Time: 0.186233833 seconds

Tables registered this way go directly to the Iceberg source. This is the simplest setup, but it means query latency depends on the source (S3, network, file format, etc.). For low-latency use cases, the next step is acceleration.

Accelerating Iceberg tables

Tables registered through a catalog are queryable but not accelerated. To accelerate a table, register it as a dataset in the spicepod. This gives you control over the acceleration engine, refresh schedule, and which subset of data to cache:

datasets:
  - from: ice.analytics.events
    name: events
    acceleration:
      enabled: true
      engine: cayenne
      refresh_sql: |
        SELECT * FROM events
        WHERE event_date > NOW() - INTERVAL '7 days'
      refresh_check_interval: 10m

The from: field references the catalog table (ice.analytics.events), so the Iceberg catalog still serves as the source of truth. The dataset definition adds the acceleration layer on top.

What happens behind the scenes:

  1. On startup, Spice executes the refresh_sql against the Iceberg table
  2. Results are loaded into a local acceleration engine (Spice supports DuckDB, SQLite, Arrow, and Spice Cayenne for queries with low memory and DuckDB-like scale)
  3. Queries against events hit the acceleration engine (local NVMe storage)
  4. Every 10 minutes, Spice re-executes the refresh SQL to pull new data

You don't have to choose between Iceberg and fast queries. Iceberg stays your source of truth. Acceleration adds a transparent cache for predictable, low-latency reads.

Writing back to Iceberg

Spice supports INSERT operations with full ACID guarantees via Iceberg's transaction protocol (check out the Iceberg cookbook for a full example). To enable writes, set access: read_write on the catalog:

catalogs:
  - from: iceberg:http://localhost:8181/v1/namespaces
    access: read_write  #required to enable INSERT operations
    name: ice
    params:
      iceberg_s3_endpoint: http://localhost:9000
      iceberg_s3_access_key_id: admin
      iceberg_s3_secret_access_key: password
      iceberg_s3_region: us-east-1
- Insert new rows
INSERT INTO ice.tpch_sf1.region (r_regionkey, r_name, r_comment)
VALUES (5, 'ANTARCTICA', 'A cold and remote region');

- Verify
SELECT * FROM ice.tpch_sf1.region WHERE r_regionkey = 5;
SELECT * FROM ice.tpch_sf1.nation WHERE n_nationkey = 25;

How this works under the hood:

  1. Spice writes new Parquet files to S3
  2. Creates a new manifest file listing the new files
  3. Creates a new snapshot referencing the updated manifest
  4. Atomically updates the metadata pointer in the catalog

If another writer commits between steps 1-3, Spice's commit fails and retries. This is Iceberg's optimistic concurrency in action - every write creates a new snapshot, and the catalog uses atomic compare-and-swap operations to ensure only one writer wins. The loser retries from the latest snapshot.

Under the hood: How Spice queries Iceberg

Now that we've covered the user-facing configuration, let's look at how the query engine processes an Iceberg query. This section introduces the internal components - if you're primarily interested in using Spice with Iceberg, the sections above have you covered.

Apache DataFusion: Spice's query engine

Spice is built on Apache DataFusion, an extensible SQL query engine written in Rust. DataFusion handles SQL parsing, query planning, and execution. (We covered DataFusion in depth in the first post in this series.)

DataFusion organizes data sources into a three-level hierarchy: Catalog -> Schema -> Table. This maps directly to Iceberg's own structure:

  • Catalog = Iceberg catalog (REST endpoint, Glue database, or Hadoop warehouse)
  • Schema = Iceberg namespace
  • Table = Iceberg table

Iceberg namespaces can technically be nested (catalog.a.b.c.table), but most catalog implementations use a single level.

When you add an Iceberg catalog to the spicepod, Spice creates three types of DataFusion components to handle it:

  • Catalog connectors to connect to REST, Glue, or Hadoop catalogs
  • CatalogProvider / SchemaProvider to discover namespaces and list tables
  • TableProvider to read and write individual Iceberg tables

How a query flows through the system

When you run SELECT * FROM ice.db.events, here's how Spice resolves it:

User QuerySELECT * FROM iceberg.db.table Name Resolution / Table Lookupiceberg.db.table → IcebergTableProvider IcebergTableProvider(iceberg-datafusion crate)Reads metadata, plans which Parquetfiles to read for the scan Parquet Files(S3, GCS, HDFS, Local FS)

DataFusion parses the three-part table name (ice.db.table), resolves ice to the Iceberg CatalogProvider, db to the SchemaProvider for that namespace, and table to an IcebergTableProvider. The TableProvider then uses Iceberg's metadata to plan the most efficient read from S3.

Schema discovery: Loading tables from a namespace

Each Iceberg namespace becomes a DataFusion SchemaProvider. On initialization, the provider:

  1. Lists all tables in the namespace from the catalog
  2. Filters them against glob patterns if an inclusion list is configured
  3. Loads each table concurrently

Performance note: Loading table metadata is the slow path - each table requires fetching metadata from S3 (metadata.json, manifest lists). For catalogs with hundreds of tables, this can take 10-30 seconds on startup. Use glob filtering to scope the inclusion list to only the tables a given deployment needs. This keeps startup time predictable regardless of catalog size.

Query optimization through Iceberg metadata

Spice uses the community iceberg-datafusion crate to connect Iceberg tables to DataFusion. When a query hits an Iceberg table, the TableProvider uses Iceberg's metadata to minimize the data read from S3:

  • Manifest pruning - filter expressions skip manifest files that can't contain matching rows
  • Predicate pushdown - query predicates push down to the Parquet reader, so only matching row groups are scanned
  • Column projection - only requested columns are read from Parquet files

This means a query like SELECT customer_id FROM events WHERE event_date = '2025-01-15' might read 50MB instead of 10TB - partition pruning eliminates irrelevant files, and column projection skips every column except customer_id.

Key Iceberg features Spice leverages

The query optimization described above depends on several foundational Iceberg features. Here's a closer look at each.

ACID transactions via optimistic concurrency

Every write creates a new metadata file pointing to a new snapshot. The catalog uses atomic compare-and-swap operations to commit changes. If two writers conflict, one fails and retries:

Read snapshot S1 Read snapshot S1 Add files, commit snapshot S2 Commit successful Add files, try to commit snapshot S3 Rejected (S2 already committed) Retry from S2 Writer A Catalog Writer B

This comes with a tradeoff - too many concurrent changes by multiple writers will result in a lot of wasted work as rejected writes retry. But the benefit is that writers can work independently, with the catalog serving as the coordination point.

Hidden partitioning

Iceberg maintains a separate partition spec that maps logical columns to physical partition values via transforms. Unlike Hive-style partitioning, users never query partition columns directly - partitioning is always defined as a transform on existing columns.

For example, consider a table:

events (
    event_id   BIGINT,
    user_id    BIGINT,
    event_time DATE
)

With the partition spec:

PARTITION SPEC
    day(event_time)

Iceberg physically stores a derived partition value such as event_time_day = 2024-01-15, but this field is not part of the table schema. When a user writes WHERE event_time = '2024-01-15', Iceberg rewrites the predicate automatically using the partition transform and prunes data files accordingly.

Changing the partitioning strategy only requires updating the spec. Old data remains partitioned under the old spec; new data uses the new one. Iceberg's metadata tracks which data files belong to which spec, and during scan planning it evaluates predicates across all active specs.

Schema evolution without rewrites

Iceberg tracks schema evolution in metadata; each data file records which schema version it was written with. Readers handle differences automatically:

  • New columns in old files -> return NULL
  • Renamed columns -> metadata tracks the mapping
  • Type promotions (int -> long) -> handled transparently

Time travel

Every commit creates an immutable snapshot. You can query as of any snapshot ID or timestamp. The tradeoff is that many small changes create many snapshots, increasing query planning time.

File-level pruning via metadata statistics

Each manifest file stores min/max values for every column in every data file. Before scanning any Parquet, Iceberg can eliminate files that cannot contain matching data:

WHERE event_date > '2024-01-01' AND user_id = 12345
event_date max < 2024-01-01 user_id out of range Predicate matches Read Manifests Check Statistics File matches predicate? Skip file Scan file

Parquet files from modern writers also include these statistics in the Parquet footer - but Iceberg's metadata-level statistics enable pruning without opening the files at all.

Write predicates that Iceberg can optimize:

- Good: Iceberg can prune partitions
SELECT * FROM events WHERE event_date > '2024-01-01'

- Less optimal: Function prevents pushdown
SELECT * FROM events WHERE YEAR(event_date) = 2024

Error handling: Making failures actionable

Iceberg operations can fail in many ways - network issues, permission errors, corrupted metadata, or missing files. Generic error messages make these hard to debug. Spice maps Iceberg errors to specific messages that tell you what went wrong and what to do about it.

For example, if you have a typo in a namespace name:

Generic Iceberg error:

Unexpected => Failed to execute http request, NoSuchNamespaceException

Spice error:

The namespace 'analytics_prod' does not exist in the Iceberg catalog, verify the namespace name and try again.

Or if the catalog URL is wrong:

Generic Iceberg error:

Unexpected => Failed to execute http request, source: error sending request for url (http://localhoster:8181/v1/config)

Spice error:

Failed to connect to the Iceberg catalog or object store at (http://localhoster:8181/v1/config). Verify the Iceberg catalog is accessible and try again.

The same approach applies across TLS certificate errors, unsupported features, and invalid table metadata - each error includes context about the cause and a suggested next step.

Production lessons

After running Iceberg in production, here are some lessons learned:

1. Catalog discovery is a startup bottleneck

Listing all namespaces and tables requires many metadata requests. Each table load fetches metadata from S3, and for catalogs with hundreds of tables, startup can take 10-30 seconds. We address this with:

  • Concurrency control - a semaphore caps concurrent table loads at 10 to avoid overwhelming the catalog or object store
  • Include patterns - glob filters scope discovery to only the tables a deployment needs
  • Lazy loading (planned) - fetch table metadata on first query instead of at startup, trading startup time for first-query latency

2. S3 request signing can expire on large tables

AWS SigV4 signatures have a fixed validity window. When loading a large Iceberg table, Spice may queue hundreds of S3 requests for metadata files, manifests, and Parquet files. If requests wait too long in the queue, AWS rejects them with RequestTimeTooSkewed. This looks like a permissions error, but it's actually a timing issue - the signature expired before the request was sent. Concurrency limiting (point 1) helps here by keeping the request queue from growing too large.

3. Environment credentials can leak into explicit configs

The iceberg-rust library uses OpenDAL for S3 access. By default, OpenDAL loads credentials from multiple sources - explicit properties, environment variables, and ~/.aws/config. Even when you provide credentials explicitly in the spicepod, OpenDAL can pick up AWS_SESSION_TOKEN from the environment and mix it with your explicit access key. This causes authentication failures that are hard to trace because the credentials you configured are correct - it's the extra session token that's wrong. Spice now sets s3.disable-config-load=true to prevent this. If you hit unexpected auth errors with Iceberg, check whether AWS credentials are set in your environment.

4. Not every table in a catalog is an Iceberg table

When connecting to AWS Glue, the catalog may contain Hive, Parquet, or CSV tables alongside Iceberg tables. Loading a non-Iceberg table as Iceberg fails. Spice handles this by skipping non-Iceberg tables with a warning instead of failing the entire catalog. If you see "Failed to load table" warnings on startup, this is likely why - it's not an error, it's Spice filtering out tables it can't read as Iceberg.

Our contributions to iceberg-rust

Spice depends on iceberg-rust for catalog access, metadata handling, and Parquet scanning. We maintain a fork for changes that haven't been upstreamed yet, and contribute fixes back to the upstream project as they stabilize. Merged upstream to apache/iceberg-rust:

Current limitations and future work

What works today:

  • Read all Iceberg tables (v1 and v2 format)
  • INSERT new data with snapshot isolation
  • Partition evolution - old and new partitioning schemes coexist transparently

Current limitations:

  • No UPDATE or DELETE. Row-level mutations are not yet supported.
  • No incremental refresh. Acceleration re-fetches the entire dataset on each refresh. Incremental refresh (only fetch new data) is in development.
  • No automatic catalog refresh. Schema changes in the Iceberg catalog require restarting Spice. Periodic auto-refresh is planned.

What's next:

  • DELETE FROM via equality delete files (merge-on-read)
  • Incremental acceleration refresh - only pull new partitions since the last refresh
  • Metadata-only queries - answer COUNT, MIN, MAX from Iceberg statistics without reading data files
  • Lazy table loading - defer metadata fetches until first query to reduce startup time
  • Better predicate pushdown to S3 Select for highly selective queries

Apache Iceberg + Spice AI: Summary

Apache Iceberg provides ACID transactions, schema evolution, and open file formats for data lakes. Spice adds federated SQL access with local acceleration on top. Together, they give you:

  • Iceberg's reliability - ACID guarantees and schema evolution over open Parquet files
  • Spice's flexibility - query Iceberg alongside Postgres, Snowflake, and other sources in one SQL interface
  • Sub-millisecond reads - accelerate frequently queried datasets locally while Iceberg remains the source of truth

Run Iceberg locally in under 5 minutes using the Spice .ai Iceberg Catalog Connector.

Get Started with Apache Iceberg + Spice AI

Experience federated SQL, acceleration, and ACID-compliant writes with Apache Iceberg and Spice AI. Follow the Iceberg Catalog Connector guide to run locally in minutes.

View Iceberg Connector Guide

content stat graphiccontent stat graphiccontent stat orb
Share
twitter logolinkedin logomailto logo
copy link logo
Get the latest insights

New releases, tutorials, platform updates, and more.