What is Apache Iceberg?

Apache Iceberg is an open table format for huge analytic datasets, built to give data lakes the reliability that data warehouses had always taken for granted. It originated at Netflix in 2017, where Ryan Blue and Daniel Weeks built it to solve real production problems with Hive tables at scale. Netflix donated it to the Apache Software Foundation in 2018, and it graduated to a top-level project in May 2020.

The core idea: Iceberg sits as a layer between your storage system (S3, HDFS, ADLS) and whatever query engine you're using. It's not a database, not a server, and there's nothing to run or deploy. It's a specification — a set of rules for how to organize Parquet files, track metadata, and manage table state — plus open-source libraries that implement those rules. Spark, Trino, Flink, Presto, Snowflake, Dremio, Athena, BigQuery, and Databricks all speak Iceberg natively, which means any of them can read from and write to the same tables without stepping on each other.

Why Iceberg Exists

To understand what Iceberg solves, it helps to trace how we got here. Data warehouses dominated for decades — batch overnight ETL, specialized hardware, predictable schema. Then around 2010, Hadoop arrived and changed the model: dump everything into a distributed file system cheaply, worry about structure later. The schema-on-read idea made sense at the time because managing schema was genuinely painful, and the volumes of data organizations were dealing with were breaking traditional warehouse economics.

But schema does matter. And consistency matters. And the ability to safely modify data without corrupting concurrent reads matters. Early data lakes — and even Hive tables on HDFS — didn't have any of that. If you wanted to update a row, you rewrote the whole partition. If two writers touched the same table at the same time, you got lucky or you got corrupted data. Changing a column name required rewriting every file or breaking downstream readers. Querying a large table meant listing every file in a directory hierarchy, which became comically slow at scale.

Iceberg and formats like it emerged from those pain points. Netflix had tens of petabytes in Hive tables and was hitting all of them.

How Apache Iceberg Works

Iceberg organizes table data through four layers of metadata stacked on top of ordinary data files.

At the bottom are the data files — typically Parquet, though ORC and Avro are also supported. These are the same columnar files you'd find in any data lake, and they live in whatever object storage you're using.

Above that, manifest files track which data files belong to a table. But they're not just path lists — each manifest entry also records column-level statistics: value counts, null counts, and the min/max values for each column. Those statistics are what lets query engines skip files entirely during planning without opening them.

Multiple ingest operations produce multiple manifest files, so manifest lists collect them. A manifest list represents one complete version of a table's file set, and it's what a query engine reads to figure out which manifests — and therefore which data files — it needs to touch.

The metadata file holds everything together. It contains the table's schema, partition spec, and a history of snapshots, where each snapshot points to a manifest list. This is what makes time travel possible and what enables atomic schema changes — you're never modifying files in place, you're always creating a new snapshot that points to a new view of the data while the old snapshots remain intact and queryable.

On top of all that sits a catalog — a lightweight lookup service that maps a table name to its current metadata file location. The catalog can be the Hive Metastore, a JDBC-backed database, AWS Glue, Project Nessie, or any other compatible implementation. The query engine asks the catalog where to find a table, the catalog points to the metadata file, and the rest unwinds from there.

This structure — all JSON and Parquet in a bucket — is what delivers ACID transactions, time travel, schema evolution, and efficient file-level skipping, without requiring any specialized storage or a dedicated server.

Key Features

ACID transactions with snapshot isolation. Concurrent writers work against the same table safely. Iceberg resolves conflicts at commit time, not at write time, using optimistic concurrency. Readers always see a consistent snapshot of the table and are never affected by in-progress writes.

Hidden partitioning. Most table formats require you to partition explicitly and write queries that match the partition scheme — WHERE dt = '2024-01-01' has to match the dt partition exactly or you scan everything. Iceberg tracks the relationship between partition values and the transform that produced them, so the engine can prune correctly even if your query uses a different expression. You can also change the partition layout without touching existing data — old files stay on the old layout, new files use the new one, and queries span both transparently.

Schema evolution. Iceberg identifies columns by an internal integer ID, not by name or position. That means you can rename, reorder, add, or drop columns and old files continue to be read correctly — the engine maps the stored column IDs to the current schema. This is not the case with most other formats, where column order in the file is assumed to match the schema's column order.

Time travel. Every write produces a new snapshot; no snapshot is deleted until you explicitly run expiry. You can query any historical snapshot by ID or by timestamp, which is useful for debugging, auditing, or reproducing the state of a dataset at a specific point in time.

Scalable metadata. The file-skipping statistics in manifest files mean query planning doesn't depend on opening data files or listing directories. This scales to tables with millions of files where directory-listing approaches become unusable.

Where Iceberg Fits in a Data Lakehouse Stack

A data lakehouse built on Iceberg typically has five layers:

1. Object storage — S3, GCS, ADLS, or HDFS holds the actual files.
2. File format — Parquet is the standard choice for analytics; Iceberg also supports ORC and Avro.
3. Table format — Apache Iceberg provides the metadata layer that makes the files behave like managed tables.
4. Catalog — AWS Glue, Nessie, the Hive Metastore, or an Iceberg REST Catalog tracks what tables exist and where their metadata lives.
5. Query engine — Spark for batch processing, Trino or Dremio for interactive queries, Flink for streaming — all reading the same tables.

The practical benefit over a traditional data warehouse architecture is that you're not making a second copy of your data into a proprietary system. Everything stays in open formats on cheap storage, and any engine that understands Iceberg can use it. When a better query engine comes out, you can adopt it without migrating your data.

Apache Iceberg vs Delta Lake vs Apache Hudi

All three are open table formats solving the same core problem. The differences are mostly about ecosystem fit and specific feature tradeoffs.

Iceberg is the right choice when you care about engine portability and want to avoid coupling your table format to a specific vendor. Delta Lake is a perfectly good format if your organization is already deep in the Databricks ecosystem and staying there. Hudi tends to show up in pipelines with heavy upsert and change-data-capture requirements, where its record-level indexing is a genuine advantage.

Cloud and Platform Support

Every major cloud platform now has native Iceberg support:

AWS offers Amazon S3 Tables for managed Iceberg storage, with Athena, EMR, Glue, and Redshift all able to read and write Iceberg tables on S3. AWS also supports the Iceberg REST Catalog standard for catalog interoperability.

Google Cloud has BigQuery BigLake Iceberg Tables, with streaming ingestion, auto-reclustering, and Vertex AI integration.

Snowflake provides managed Iceberg Tables with full DML support, plus the Snowflake Open Catalog (a managed version of Apache Polaris, the open-source REST Catalog reference implementation).

Databricks added full Iceberg support through Unity Catalog's REST Catalog API — a significant shift following their 2024 acquisition of Tabular, the company founded by Iceberg's original creators.

Microsoft Fabric supports Iceberg via metadata virtualization and Data Factory integration.

The Iceberg REST Catalog

The REST Catalog is a vendor-neutral HTTP API specification for catalog operations — creating tables, listing namespaces, committing metadata updates. Any query engine that implements the REST Catalog client can work with any catalog backend that implements the server side, regardless of how either is built.

Apache Polaris (contributed to the Apache Software Foundation by Snowflake) is the reference open-source implementation. AWS, Google Cloud, Snowflake, Databricks, and Dremio all offer REST Catalog-compatible services. The practical effect is that you can swap catalog backends without changing your query engines, and new engines don't need to implement custom integrations for each catalog.

Iceberg V3 Specification

The V3 spec, ratified in spring 2025, adds capabilities that had been limitations of the format:

Deletion vectors replace positional delete files for row-level deletes. Instead of maintaining a separate file listing deleted row positions, V3 uses a compact binary bitmap. This makes UPDATE and DELETE operations significantly cheaper, both at write time and read time.

Row-level lineage adds metadata fields to individual records for precise change tracking, which simplifies CDC pipelines and audit requirements.

New data types include a Variant type for semi-structured data, geospatial types (geometry and geography), and nanosecond-precision timestamps — gaps that had required workarounds in V2.

Is Iceberg Parquet? Is It a Database?

Two questions that come up often:

Iceberg is not Parquet. Parquet is a file format for storing columnar data efficiently. Iceberg is a table format that manages a collection of Parquet files — it adds the metadata layer that gives those files transactional semantics, schema management, and partition tracking. Iceberg tables almost always store data as Parquet, but they could use ORC or Avro instead. Parquet alone has no concept of tables, schemas, or transactions.

Iceberg is not a database. There's no Iceberg server process to run. It's a specification plus libraries. The "database-like" features come from the metadata layer living alongside the data files in object storage, interpreted by whichever engine you're using.

Netflix, Apple, Airbnb, LinkedIn, Salesforce, Tencent, and Bloomberg are among the organizations running Iceberg at production scale.