Apache Iceberg: Complete Guide to Open Table Format

Apache Iceberg is an open table format for large analytic datasets, designed to address the limitations of Hive-style table management on cloud object storage. Originally developed at Netflix and contributed to the Apache Software Foundation, Iceberg provides ACID transactions, schema evolution, hidden partitioning, and time travel capabilities on top of standard file formats such as Parquet, ORC, and Avro. It has become one of the dominant open table formats in the data lakehouse ecosystem, with broad adoption across Snowflake, Databricks, AWS, Google Cloud, and Apache Flink.

What Is Apache Iceberg?

The fundamental problem Iceberg solves is that cloud object storage — Amazon S3, Azure Data Lake Storage, Google Cloud Storage — is an excellent place to store large amounts of data cheaply, but it was never designed to function as a database. Traditional approaches to querying data on object storage, epitomised by Apache Hive, relied on directory structures and partition naming conventions that created significant reliability and performance problems at scale. Iceberg replaces this directory-based approach with a proper table metadata layer that tracks the exact set of files belonging to a table snapshot, enabling reliable, performant, and concurrent access without the limitations of Hive metastore.

The "open" in open table format is as important as the technical capabilities. Unlike proprietary warehouse formats, Iceberg tables can be read and written by any compatible engine. This engine-agnostic design means organisations are not locked into a single vendor: Spark, Trino, Flink, Dremio, StarRocks, Snowflake, and BigQuery can all access the same Iceberg tables directly. Multi-engine compatibility is a defining advantage of the open table format approach over closed alternatives and directly supports data lineage tracking across heterogeneous processing environments.

Why Open Table Formats Matter

Before open table formats, data lakes had a fundamental weakness: they were write-once, read-many systems. You could add files but not update or delete individual records efficiently. ACID transactions — the guarantee that writes either fully succeed or fully fail, and that readers never see partial writes — were the exclusive domain of relational databases and data warehouses. Data lakes were fast and cheap but unreliable for workloads that required consistent views.

Open table formats like Apache Iceberg bring database-grade reliability to cloud object storage without sacrificing the flexibility and cost advantages of the lake model. They do this by maintaining a separate metadata layer — a series of manifest files and a table metadata JSON file — that tracks which data files belong to the current snapshot of the table. This metadata layer is what enables ACID transactions: a write creates a new snapshot atomically, and readers always see a consistent snapshot regardless of what writes are happening concurrently.

For medallion architecture implementations, open table formats are essential. The Bronze, Silver, and Gold layers each require different write patterns — high-throughput append at Bronze, CDC upserts at Silver, aggregation and overwrite at Gold — and open table formats handle all of these patterns reliably within a single storage layer.

Key Features of Apache Iceberg

ACID Transactions. Iceberg provides full ACID transaction support through snapshot isolation. Every write operation — INSERT, UPDATE, DELETE, or MERGE — produces a new table snapshot by creating a new metadata file pointing to a new set of data files. The previous snapshot remains intact and visible to readers until they commit to reading a newer snapshot. This optimistic concurrency model allows many concurrent readers and writers without locking, which is essential for large-scale analytics workloads and enables GDPR right-to-erasure compliance through row-level deletes.

Schema Evolution. One of Iceberg's most practically valuable features is its support for schema evolution without rewriting existing data. Tables can have columns added, renamed, reordered, or dropped without touching the underlying Parquet files. Iceberg tracks column identities using stable integer column IDs rather than column names, so renaming a column does not break existing files written before the rename. This is a significant improvement over Hive-style tables, where schema changes often required expensive full table rewrites.

Hidden Partitioning. Traditional Hive partitioning requires users to know the partitioning scheme of a table and include partition columns in their queries to avoid full table scans. Iceberg introduces hidden partitioning, which decouples the logical table schema from the physical partitioning layout. Partitions are defined as transform functions applied to table columns — for example, partitioning by the month derived from a timestamp column — and the query engine automatically uses the partition layout to prune files without requiring users to filter on the partition column explicitly. When the partitioning strategy needs to change, Iceberg supports partition evolution without rewriting old files.

Time Travel and Snapshot History. Iceberg retains the full history of a table's snapshots, enabling time travel queries that read the state of a table at any point in its history. This is invaluable for debugging data quality issues, auditing historical values, reproducing past results, and implementing slowly changing dimensions. Time travel is addressed either by snapshot ID or by timestamp using SQL syntax such as SELECT * FROM table FOR SYSTEM_TIME AS OF '2025-01-01'.

Row-Level Operations. Iceberg supports row-level DELETE, UPDATE, and MERGE through two storage strategies. Copy-on-Write (CoW) rewrites affected data files entirely, producing clean files optimised for read performance. Merge-on-Read (MoR) writes small delete files alongside the original data files and merges them at read time, offering faster writes at slightly more complex read cost. The choice between these strategies depends on the workload's read/write ratio.

Apache Iceberg vs Delta Lake vs Apache Hudi

The three dominant open table formats — Apache Iceberg, Delta Lake, and Apache Hudi — share a common goal but have different design philosophies and strengths.

Delta Lake, developed by Databricks, was the first widely adopted open table format and pioneered the lakehouse concept. It is tightly integrated with Apache Spark and the Databricks platform. Delta Lake's primary strengths are its mature Spark integration, its Z-ordering clustering capability, and its broad adoption within the Databricks ecosystem. Its main limitation compared to Iceberg is that it has historically been less engine-agnostic: while Delta Lake has opened its format specification and added support for non-Spark engines, Iceberg has a longer track record of genuine multi-engine support.

Apache Hudi, originally developed at Uber, excels at high-frequency upsert workloads. Its Merge-on-Read storage model and record-level indexing make it particularly well suited to CDC pipelines that continuously upsert individual records from operational databases. Hudi's indexing mechanisms provide faster upsert performance than Iceberg or Delta Lake for these workloads, at the cost of greater operational complexity.

Iceberg's primary advantages are its genuine openness, its superior multi-engine compatibility, and its clean separation between the table format specification and specific engine implementations. The Iceberg specification is independently governed by the Apache Software Foundation, which makes it the most vendor-neutral choice. For organisations operating heterogeneous environments — combining Spark, Trino, Flink, and a cloud data warehouse — Iceberg is typically the most practical choice.

Iceberg Catalogs and Metadata Integration

An Iceberg table's metadata must be tracked in a catalog that records where the current metadata file for each table is stored. Several catalog implementations are available, each with different trade-offs.

The Hive Metastore is the most widely used catalog for Iceberg tables in existing Hadoop and Spark environments. The AWS Glue Data Catalog is a fully managed alternative popular in AWS-native stacks, offering serverless operation and tight integration with Athena, EMR, and Lake Formation.

Project Nessie is an open-source catalog developed by Dremio that provides Git-like branching and tagging semantics for Iceberg tables. With Nessie, data teams can create isolated branches of their data lake for testing schema changes or running experimental transformations, then merge those changes back to the main branch when ready — a capability that has no equivalent in traditional data warehouse environments.

The REST catalog specification is an emerging standard that defines a vendor-neutral HTTP API for Iceberg catalog operations. REST-compatible catalogs include Polaris Catalog (open-sourced by Snowflake) and Unity Catalog. The REST spec's adoption is accelerating Iceberg's multi-cloud and multi-vendor interoperability.

Industry Adoption

Apache Iceberg has achieved remarkable breadth of adoption across both cloud providers and data platform vendors. Snowflake has made Iceberg a first-class citizen through Iceberg Tables, which allow Snowflake to read and write Iceberg tables stored in customer-owned S3 or ADLS, enabling true data portability without vendor lock-in. Databricks has added Iceberg read and write support through its UniForm feature, which writes Delta Lake tables that are simultaneously valid Iceberg tables, bridging the two formats. AWS supports Iceberg natively across S3 Tables, Athena, EMR, and Glue. Google BigQuery supports BigLake tables on Iceberg, and Apache Flink has first-class Iceberg sink support for streaming pipelines.

This level of cross-vendor adoption is unprecedented for a data format and reflects the industry's recognition that data portability — the ability to move workloads between engines and platforms without reformatting data — is a strategic imperative for large organisations.

For teams migrating existing Hive or Parquet-based lakes to Iceberg, the most common starting point is using Apache Spark's Iceberg connector to convert existing tables in place using the CALL catalog.system.migrate() procedure, which rewrites table metadata without copying data files. This allows teams to adopt Iceberg incrementally without a big-bang migration.

Data Governance and Apache Iceberg

Apache Iceberg's rich metadata layer creates excellent foundations for data governance. Every table snapshot is recorded with a timestamp, the operation that created it, and the identity of the process that wrote it. This audit history is directly accessible through Iceberg's metadata tables — snapshots, history, manifests, and files — which can be queried using standard SQL to answer questions about what changed, when, and how.

For organisations using a metadata management platform like Dawiso, Iceberg's metadata layer is a rich source of technical lineage and data quality signals. Dawiso can harvest table metadata from Iceberg catalogs — column names, types, partition specifications, snapshot history — and surface it in the data catalog alongside business glossary definitions, ownership records, and quality scores. Column-level lineage across Iceberg tables populated by Spark or dbt pipelines can be tracked in Dawiso, enabling analysts to trace a metric in a Gold reporting table all the way back to the raw source files in the Bronze layer of a medallion architecture.

Access control for Iceberg tables is typically implemented at the catalog layer. Apache Ranger, AWS Lake Formation, and Unity Catalog all provide row-level and column-level security for Iceberg tables, allowing data governance policies defined in a central governance platform to be enforced at the storage layer without requiring application-level filtering.

File management is an important operational consideration. Iceberg's snapshot model retains old data files until they are explicitly expired, which means storage costs grow without active maintenance. The expire_snapshots and rewrite_data_files table maintenance procedures should be run on a regular schedule to remove orphaned files and compact small files created by frequent streaming writes.