How to architect a modern data lake in 2026 - Apache Iceberg tables on S3 object storage, seconds-fresh streaming ingestion with Flink and Kafka, and federated REST catalogs (Polaris, Lakekeeper, Glue, Unity) that let multiple engines share one set of tables.
The phrase "data lake" used to mean a directory of Parquet files in a bucket and a Hive metastore that knew where they lived. That design still works, and it still fails in the usual ways: no atomic writes, no safe schema changes, queries that scan partitions they should skip, and a metastore that becomes a single point of contention. A lake built in 2026 looks different. The design space has narrowed to Apache Iceberg tables on object storage, streaming ingestion, and a REST catalog any engine can talk to.
Of those three, the catalog is the one decision that can still lock you in, so it is the one to make first. Storage and table format are settled. The catalog is where teams still pick something closed, discover a year later that every table registration and every grant lives inside it, and then pay to move. This post walks through the layers, which catalog options interoperate, how to land data with seconds of latency, and the operational work the design assumes you will do.
This is the build piece. If you want the category-level distinctions first, read EDW vs data lake vs lakehouse. For the warehouse-side view of the same convergence, see data warehouse architecture in 2026. For how to organize the data once it lands, see the medallion architecture.
Object Storage Is the Foundation, Not a Detail
All table data and metadata live as immutable objects in cloud object storage (Amazon S3, Google Cloud Storage, or Azure Data Lake Storage), and compute is a separate, stateless tier that can be scaled, replaced, or run in parallel without moving the data. Storage is cheap, durable, and effectively infinite; compute is expensive and bursty. Decoupling them lets you keep one physical copy of a dataset and point Trino, Spark, Flink, and a warehouse at it independently.
Object storage gives you eleven nines of durability and near-linear read throughput, but it is not a filesystem, and treating it like one is where lakes go bad. Two physical details dominate performance. First, file size. Iceberg's default write.target-file-size-bytes is 512 MB, and that is a fine target for batch-written tables. For streaming tables, where writers flush on a commit interval rather than on size, expect files in the low hundreds of megabytes after compaction and plan for that. Hundreds of tiny files turn a query into thousands of HTTP GETs and balloon metadata; multi-gigabyte files kill read parallelism and make rewrites expensive. Second, layout. Iceberg's hidden partitioning means partition values live in table metadata, not in the object path. Iceberg still writes dt=2026-01-01/ prefixes by default, but no engine relies on them, so you can change the partition spec without rewriting data, and you avoid both the small-files explosion of over-partitioning and the full-table scans of under-partitioning.
AWS made this easier in late 2024 with Amazon S3 Tables, a bucket type that stores fully managed Iceberg tables and runs compaction and maintenance for you. If you are AWS-native, use it: the file-optimization work it takes off your plate is exactly the work most teams under-staff. The cost is control over how and when maintenance runs, and a lake that only exists on AWS. On a self-managed lake you own that work, which we come back to below.
Iceberg Is the Table Layer
The table format is what turns a pile of objects into something you can safely write to from more than one job at a time. Apache Iceberg has become the default. It tracks table state through a tree of metadata files, manifest lists, and manifests that point at the actual data files, which is what gives it atomic commits, snapshot isolation, time travel, schema evolution, and partition evolution without rewriting data. Delta Lake and Apache Hudi solve the same problems; the gravity in 2026 is around Iceberg because the major catalogs and cloud vendors standardized on its REST API. For a format-level comparison see Iceberg vs Delta Lake, and for how the internals fit together, the Iceberg architecture deep dive.
Format version matters when you design in 2026. Iceberg 1.10 (September 2025) finalized format v3: deletion vectors instead of positional delete files, row lineage, and a variant type for semi-structured columns. Engine support is uneven. Databricks took v3 GA in May 2026, while Athena still only creates and operates on v2 tables as of this writing. If any engine in your stack cannot read v3, create tables as v2 and upgrade later, because a v3 table is invisible to a v2-only reader.
The practical payoff is concurrency. Multiple writers can commit to the same table using optimistic concurrency: each commit swaps a metadata pointer atomically, and a losing writer retries against the new snapshot. Readers always see a consistent snapshot and never observe a half-written commit. A raw Parquet-on-S3 lake cannot give you that, and it is why "schema migration broke the nightly job" stops being a recurring incident.
Iceberg does not manage itself. Every commit creates a new snapshot and may add small files; left alone, a busy table accumulates thousands of snapshots and millions of objects. Compaction, snapshot expiration, and orphan-file cleanup are part of the design. Skip them and read latency climbs week over week until someone notices the lake is slow and nobody knows why. We cover the routine in Iceberg table maintenance best practices.
The Catalog Layer: Federated, Not Locked In
The catalog is the service that maps table names to their current metadata location and brokers commits. The old default, the Hive Metastore, was a JDBC-backed service that every engine integrated with separately and that became a bottleneck and a liability. The Iceberg REST Catalog specification replaced that model with a single OpenAPI-defined HTTP contract: implement the spec once on the server, and Spark, Flink, Trino, and PyIceberg all talk to it through one client. Swapping catalogs then becomes a config change on the engine side rather than a rewrite.
"Federated" is the second shift. A federated catalog can present tables that physically live in other catalogs under one namespace and access model. Apache Polaris, donated to the Apache Software Foundation in August 2024 and promoted to a Top-Level Project in February 2026, can federate to external Iceberg REST catalogs, Hive Metastores, and BigQuery Metastore, exposing their tables read-only under one namespace. On the cloud side, AWS shipped catalog federation in the Glue Data Catalog and a Glue Iceberg REST endpoint that lets engines reach S3 Tables and federate to Databricks Unity Catalog through the same API. Both federations are read-only. Writes still go to the catalog that owns the table, which is why federation is a migration bridge and not a way to avoid choosing.
What lock-in costs, concretely: every table registration, every namespace, and every grant lives in the catalog. Moving off a closed one means re-registering each table's current metadata location in the new catalog, re-creating every role and permission there, and repointing every engine, ideally without a window where two catalogs both think they own commits. For a few hundred tables that is a weekend; for ten thousand with per-column grants it is a project.
| Catalog | Type | Governance / lock-in | Best fit |
|---|---|---|---|
| Hive Metastore | Legacy, self-hosted | No federation; engine-specific quirks | Existing Hadoop estates being migrated off |
| AWS Glue Data Catalog | Managed (AWS) | REST endpoint + read-only federation to Unity/external | AWS-native lakes, S3 Tables |
| Apache Polaris | Open source (ASF) | Vendor-neutral, read-only federation, RBAC | Multi-engine or multi-cloud, avoiding lock-in |
| Lakekeeper | Open source | REST spec + OpenFGA authorization, OPA bridge, audit events | Self-hosted lakes wanting fine-grained governance |
| Databricks Unity Catalog | Managed (Databricks) | Iceberg REST read/write interop since May 2026; richest inside Databricks | Databricks-centric platforms |
Polaris is the vendor-neutral standard with broad backing. Lakekeeper is a Rust REST catalog with OpenFGA-based authorization, an OPA bridge so Trino enforces the same policies, and audit events for authorization decisions, which makes it attractive when governance is the deciding factor. Both implement the same REST spec, so an engine pointed at one can be repointed at the other with a config change.
Streaming Ingestion: Seconds, Not Hours
Batch ingestion into a lake is a solved problem. The question in 2026 is how fresh you can make the data, and the answer is seconds. Two patterns dominate, and they differ mainly in how much infrastructure you want to run.
The lighter option is the Apache Iceberg project's own Kafka Connect sink: a connector that reads Kafka topics, buffers records, and commits to Iceberg on a configurable interval, with automatic table creation, schema evolution from the Schema Registry, multi-table fan-out, and Debezium and DMS transforms for CDC streams. It needs only a Kafka Connect cluster. The heavier and more capable option is Apache Flink, which commits Iceberg snapshots on checkpoint completion to get exactly-once delivery, and handles CDC changelogs by writing them as equality deletes on v2 tables with a primary key, so updates and deletes from an upstream database land correctly rather than as appends. The Flink Dynamic Iceberg Sink (November 2025; needs Iceberg 1.10+ and Flink 1.20, 2.0, or 2.1) extends this to route a single stream into many tables with schema evolution and no downtime, which removes the old one-job-per-table tax. We go deeper in Flink and Iceberg.
| Pattern | Latency | Exactly-once | CDC / upserts | Ops cost |
|---|---|---|---|---|
| Batch (Spark/Trino INSERT) | Minutes to hours | Via job idempotency | Manual MERGE | Low |
| Kafka Connect Iceberg sink | Seconds to minutes | Yes (commit coordinator over a control topic; relies on KIP-447, Kafka 2.5+) | CDC via Debezium/DMS transforms and id columns | Low (Connect cluster) |
| Flink streaming sink | Seconds | Yes (checkpoint = snapshot) | Native (equality deletes, v2 table with primary key) | Higher (Flink cluster) |
The pitfall with streaming into Iceberg is the small-files problem made continuous. A sink committing every few seconds writes a steady drip of small files and a snapshot per commit, so size the commit interval and the compaction schedule together. Worked example: a topic at 5,000 events/s of 1 KB each is about 5 MB/s. A Flink job checkpointing every 10 seconds on a table partitioned by hour writes one roughly 50 MB file per checkpoint per partition, which is 360 files and 360 snapshots an hour. Compact hourly and you fold each hour into three or four 512 MB files; expire snapshots older than a day and you keep the metadata tree at a few thousand entries instead of tens of thousands. Drop the checkpoint interval to 2 seconds for fresher data and the file count goes up fivefold, so compaction has to run more often to keep pace. ClickHouse can also query Iceberg tables directly for low-latency analytics over this data; see our ClickHouse and Iceberg integration guide.
A Reference Architecture and What Breaks First
An AWS-native lake looks like this: sources stream into Kafka; Flink writes Iceberg tables to S3 Tables; the Glue Data Catalog REST endpoint fronts those tables; and Trino, Spark, Athena, Redshift, and ClickHouse all query the same physical tables through that one catalog. A vendor-neutral variant swaps Glue for Polaris or Lakekeeper and plain S3 for S3 Tables, and keeps everything else.
What breaks first, in the order we usually see it:
- Small files. The streaming table nobody scheduled compaction for is slow within a month. On S3 Tables the bucket handles it; on plain S3 the job is yours, and streaming tables need it hourly, not nightly.
- Format version mismatch. A writer defaults to v3, and Athena or an older Trino cannot read the table. Pin the format version per table until every reader is upgraded.
- Governance forks. Permissions defined in Trino do not apply in Spark. Put access control, lineage, and audit in the catalog so policy is enforced once, which is the case for Lakekeeper or Polaris over Glue if you need per-column grants across engines.
- The catalog you cannot leave. Federation is read-only, so a closed catalog stays the write owner until you do the re-registration project described above. Pick a REST-spec catalog on day one.
Designing or operating a lake on Iceberg and S3? Our data engineering consulting team builds and tunes open lakehouse architectures end to end, from streaming ingestion and catalog selection to the Iceberg maintenance that keeps them fast. Get in touch.
