Data lakes on object storage (S3, GCS, Azure Blob) have a fundamental limitation: raw files on disk have no inherent concept of a "table." There is no built-in support for atomic writes, schema consistency, concurrent readers and writers, or querying data as it existed at a point in the past.

Both Apache Iceberg and Delta Lake solve this by adding a metadata layer on top of Parquet files stored in object storage. They track which files belong to the current version of a table, manage schema changes, record an auditable history of every modification, and provide the isolation guarantees that analytics and data pipelines depend on.

Both formats are production-ready and widely adopted. Choosing between them requires understanding their architectural differences -- particularly around metadata design, partition evolution, and engine compatibility -- rather than asking which one is "better."

How Each Format Works

Apache Iceberg Metadata Architecture

Apache Iceberg uses a three-level metadata hierarchy:

1. Metadata files -- one per snapshot, recording the table schema, partition spec, and a reference to the snapshot's manifest list
2. Manifest lists -- one per snapshot, listing all manifest files with partition-level statistics
3. Manifest files -- listing individual data files with column-level min/max statistics and null counts

This hierarchy is self-contained in the storage system. The Iceberg catalog (Hive Metastore, AWS Glue, REST catalog, or others) stores only a pointer to the current metadata file. Reading the table means following the pointer chain: catalog → metadata file → manifest list → manifest files → data files.

The design separates the catalog from the metadata. Changing the catalog implementation (e.g., from Hive Metastore to a REST catalog) does not require migrating table metadata.

Delta Lake Metadata Architecture

Delta Lake uses a transaction log -- the _delta_log directory -- consisting of ordered JSON commit files:

_delta_log/
  000000000000000000000.json  # Version 0
  000000000000000000001.json  # Version 1
  000000000000000000010.json  # Version 10
  000000000000000000010.checkpoint.parquet  # Checkpoint at version 10

Each JSON commit file records the actions for that version: which files were added, which were removed, and any metadata changes. To read the current state, a reader replays the log from the last checkpoint forward. Periodic Parquet checkpoints (every 10 commits by default) prevent full log replay from the beginning.

Delta Lake's catalog integration is through the metastore (typically the Hive Metastore or Unity Catalog) which registers the table's storage location. The transaction log is stored alongside the data files.

Feature Comparison

Key Differences

Partition Evolution

This is one of the most meaningful practical differences between the two formats.

Iceberg supports partition evolution natively. A table can start with monthly partitioning, be re-partitioned to daily without rewriting existing data, and queries spanning both old and new partitions work correctly. Iceberg tracks which partition spec applies to each data file in the manifest. Changing the partition scheme is a metadata operation.

Delta Lake added partition transforms (liquid clustering) as a later feature. Traditional Hive-style partitioning in Delta Lake requires specifying partition columns at table creation and changing them requires data rewriting. Liquid clustering (Delta Lake's replacement for static partitioning) addresses this by organizing data with space-filling curves that can be re-keyed without data rewrites, but it is a different mechanism than Iceberg's partition evolution.

Hidden Partitioning

Iceberg decouples the logical query from the physical partition layout. A filter like WHERE event_date > '2026-03-01' automatically benefits from partition pruning even if the table is partitioned by month(event_date). Users do not need to know or reference the partition column names. This prevents a common class of performance bugs where users write queries without matching partition expressions and accidentally trigger full table scans.

Delta Lake (with Hive-style partitioning) requires that filter expressions match the physical partition column. A filter on event_date benefits from pruning only if the table is partitioned on event_date directly (or if liquid clustering is used). Liquid clustering provides some automatic pruning through its zone map statistics, but the user experience is closer to DuckDB's min/max zone maps than Iceberg's hidden partitioning.

Engine Compatibility

Iceberg was designed from the start to be engine-agnostic. It defines an open specification that any engine can implement. Native read and write support exists in Spark, Trino, Flink, Dremio, Snowflake External Tables, Google BigQuery (via manifest export), Amazon Athena, and federated query engines like Spice.

Delta Lake originated in the Databricks ecosystem and has its deepest integration with Apache Spark. The Delta Kernel project provides a standalone library for reading Delta tables outside Spark, which has expanded engine support (Trino, Flink, Athena, BigQuery). For teams already using Databricks, Delta Lake's Unity Catalog and Databricks platform integrations offer first-class governance, lineage, and access control features that Iceberg does not match in the Databricks context.

Metadata Scalability

At tables with millions of data files -- common in large-scale production data lakes -- metadata performance becomes a critical concern.

Iceberg avoids directory listing entirely. The manifest list for a snapshot immediately enumerates all manifest files, and each manifest file enumerates its data files. Planning time is proportional to the number of manifest files read, and manifest-level partition statistics allow the planner to skip entire manifests without reading them. This scales well to tables with tens of millions of files.

Delta Lake must replay the transaction log from the last checkpoint. At high write frequencies, the number of JSON files accumulating between checkpoints can grow large, and the checkpoint frequency may need tuning. The recent move toward Parquet and Avro checkpoints with more granular metadata improves scalability, but log replay is still the core access pattern.

Decision Framework

Choose Apache Iceberg when:

• Multi-engine access is required. If you query the same tables from Spark, Trino, Flink, and a federated SQL engine, Iceberg's broad native support is a significant advantage.
• Partition evolution is important. If partition schemes will change over time without full data rewrites, Iceberg's native partition evolution is the right design choice.
• You value catalog independence. Iceberg's self-contained metadata design makes it straightforward to switch catalog implementations without migrating table data.
• Tables have many files (10M+). Iceberg's manifest-based planning avoids the log replay scalability limitations that can affect Delta Lake at extreme scale.
• You are not primarily on Databricks. If your stack includes Trino, Dremio, Starburst, or open-source Spark on non-Databricks infrastructure, Iceberg's engine-agnostic design fits more naturally.

Choose Delta Lake when:

• Databricks is your primary platform. Delta Lake has the deepest Databricks integration, including Unity Catalog governance, Delta Live Tables pipelines, and Databricks-managed compaction and optimization.
• Your team is primarily on Apache Spark. Delta Lake's Spark integration is the most mature and best-maintained. Spark's DeltaTable API provides a rich set of operational commands beyond what Iceberg's Spark integration offers.
• You need the Change Data Feed. Delta Lake's CDF exposes row-level changes between versions as a structured stream, which is useful for incremental processing pipelines. Iceberg has similar capabilities through its streaming API, but Delta Lake's CDF has broader tooling support.
• Liquid clustering meets your partitioning needs. For tables where dynamic clustering is more important than partition evolution, liquid clustering simplifies operations.

Iceberg and Delta Lake in Spice

Spice connects to both Apache Iceberg and Delta Lake tables as federated data sources through its SQL federation layer.

For Iceberg tables in Amazon S3, GCS, or Azure Blob Storage, Spice reads the Iceberg metadata to identify relevant data files, applies partition pruning based on the partition spec, and uses file-level statistics from manifest files for predicate pushdown. This allows SQL federation over Iceberg tables with efficient scans.

For Delta Lake tables, Spice connects through the delta-rs Rust library, which implements the Delta Lake protocol without requiring Spark or the JVM. It reads the transaction log to identify the current set of active files, applies predicate pushdown through Apache DataFusion's optimizer, and supports optional local acceleration.

Both formats can be joined in a single federated query:

SELECT i.customer_id, d.event_type, COUNT(*) AS count
FROM iceberg.orders i
JOIN delta.events d ON i.customer_id = d.customer_id
WHERE i.order_date > '2026-01-01'
GROUP BY i.customer_id, d.event_type

For workloads requiring sub-second query performance, Spice supports accelerating both Iceberg and Delta Lake tables locally, with incremental refresh using each format's native change detection mechanism.

Advanced Topics

Iceberg REST Catalog and Delta Lake Unity Catalog

The catalog layer determines how table metadata is discovered and how concurrent writes are coordinated.

Iceberg REST catalog is a standardized HTTP API that any catalog provider can implement. It decouples table discovery from the backend store, enabling catalog providers to add authorization, caching, multi-tenancy, and access control logic behind the standard API. The REST catalog uses optimistic concurrency: clients send the current metadata location alongside the new metadata, and the server rejects commits where another writer has intervened.

Delta Lake Unity Catalog (Databricks) provides a centralized metadata, governance, and access control layer for Delta tables. It integrates with Databricks' RBAC, column-level masking, row-level security, and lineage tracking. Outside Databricks, the open-source Unity Catalog project provides a standalone implementation, though with fewer integrations.

Compaction and Maintenance

Both formats accumulate small files over time (from streaming ingestion or frequent small writes) and require periodic compaction.

Iceberg compaction rewrites small files into larger ones using the REWRITE DATA FILES procedure. It can be run in-engine (via Spark) or via the Iceberg API. Snapshot expiration removes old snapshots and their unreferenced data files.

Delta Lake compaction uses the OPTIMIZE command, which rewrites small files into larger target sizes (default 1 GB). VACUUM removes files that are no longer referenced by any table version within the retention window.

Both approaches are equivalent operationally; the syntax and scheduling differ by engine.

Row-Level Operations Performance

Row-level deletes and updates are an area where both formats have evolved significantly.

Iceberg v2 introduced position deletes and equality deletes, which avoid full file rewrites for small delete operations. Position deletes specify exact file path and row offset. Equality deletes specify column values that identify rows to delete. Over time, accumulated delete files degrade read performance, and compaction merges them with data files.

Delta Lake deletion vectors serve the same purpose -- recording logically deleted rows as a compact bitmap without rewriting data files. Deletion vectors are applied during scan, and compaction eventually produces clean files without deleted rows.