What is a Kafka Topic?

A Kafka topic is the fundamental unit of organization in Apache Kafka -- a named, append-only log of events that producers write to and consumers read from. Think of it as a durable, distributed, ordered sequence of records, identified by a name like orders, user-clicks, or payments.completed. Producers append to the tail; consumers read at their own pace from any position. Unlike a traditional message queue, the log isn't consumed away -- records stay on disk for as long as the retention configuration allows.

A topic in Kafka is closer to a database table than to a queue, but a table with two unusual properties: it's append-only, and it's partitioned across many machines for parallelism and scale. Everything interesting about how Kafka behaves -- ordering, throughput, durability, scalability -- flows from how topics are structured underneath.

How a Kafka Topic Is Structured

A topic is not stored as a single log. It's split into one or more partitions, and each partition is the actual append-only log that lives on disk. A topic with 12 partitions is twelve independent logs that happen to share a name.

Partitions are the unit of parallelism. Each partition is owned by one broker (the leader) and replicated to others (followers) for durability. Producers write to a specific partition, and consumers in a consumer group each get assigned one or more partitions. More partitions means more parallelism -- but also more file handles, more metadata, more rebalancing overhead. Modern Kafka clusters routinely run topics with hundreds or thousands of partitions; very large topics push into the tens of thousands.

Offsets identify position within a partition. Each record appended to a partition gets a monotonically increasing 64-bit integer called an offset. Consumers track their own position by committing offsets; they can replay from any earlier offset, jump to the latest, or start from a specific timestamp. Offsets are per-partition -- there is no global ordering across a topic.

Records are key/value pairs with headers and a timestamp. Keys are optional but consequential: by default, records with the same key go to the same partition (via hash partitioning), which is what guarantees per-key ordering. Values are arbitrary bytes -- usually JSON, Avro, Protobuf, or JSON Schema with a schema registry gating the structure.

Replication factor controls durability. Each partition is replicated to N brokers (typically 3 in production). One is the leader and handles all reads and writes; the rest stay in sync as followers. If the leader fails, one of the in-sync replicas (ISRs) takes over. min.insync.replicas controls how many replicas must acknowledge a write before it's considered durable -- typically set to 2 with replication factor 3 for safe, available writes.

What Topics Actually Guarantee

The guarantees are precise and worth knowing exactly.

Per-partition ordering. Records appended to a partition are read in the same order. Across partitions, no order is guaranteed. If global order matters for a topic, the topic must have a single partition -- which caps its throughput to a single broker.

Per-key ordering (when keys are used). If producers use the default partitioner and a consistent key, all records for a given key land on the same partition and are therefore read in order. This is the usual way to preserve per-entity ordering (per user, per order, per device) while still scaling out across many partitions.

At-least-once delivery by default. With idempotent producers (enable.idempotence=true, the default in modern Kafka), exactly-once semantics within a single producer session are achievable. Transactional producers extend this to multi-partition atomic writes. Consumers can also opt into "read committed" isolation to ignore aborted transactions.

Durability up to min.insync.replicas. Producers can choose acks=0 (fire and forget), acks=1 (leader only), or acks=all (all in-sync replicas). For data you actually care about, acks=all with min.insync.replicas=2 is the safe configuration.

Retention: Time-Based and Compacted Topics

Kafka topics don't delete records after consumption. Retention is governed by the topic's configuration, and there are two distinct retention modes.

Time- or size-based retention. The default. Configured via retention.ms (how long to keep records) and retention.bytes (size cap per partition). Records older than the limit are deleted in whole-segment chunks. Common values range from hours (high-volume operational topics) to days or weeks (analytics-feeding topics) to infinite (event sourcing, immutable history). Storage costs scale linearly with retention, which is why modern Kafka offerings increasingly support tiered storage that moves older segments to object storage like S3.

Log compaction (cleanup.policy=compact). Instead of deleting old records by time, compacted topics retain only the most recent value for each key. The topic becomes a durable, replayable changelog: at any point, replaying it reconstructs the current state of every key. This is the foundation for Kafka Streams' state stores, for replication patterns, and for "topic-as-table" architectures.

Hybrid: cleanup.policy=compact,delete. Retains the latest record per key and enforces a time/size cap. Useful when state needs to expire after some period (e.g., a session table).

Producers, Consumers, and Consumer Groups

A topic has many producers writing in parallel and many consumers reading in parallel. Kafka's consumer model is what makes the design work at scale.

Producers decide which partition a record goes to. The default behavior is to hash the key; records without a key get distributed round-robin or via a sticky partitioner that batches efficiently. Custom partitioners are supported for advanced cases.

Consumer groups. Consumers join a group identified by group.id. Within a group, partitions are distributed across members -- each partition is read by exactly one consumer. Adding consumers up to the partition count increases parallelism; adding beyond that gives idle consumers. Different consumer groups read the same topic independently, each tracking its own offsets. This is how Kafka supports both queue-style consumption (one group) and pub/sub fan-out (many groups) from the same topic.

Rebalancing. When consumers join or leave a group, partitions are reassigned. Naive rebalancing causes "stop-the-world" pauses; modern Kafka uses cooperative or incremental rebalancing to keep most consumers running through the change.

Topic Configuration in Practice

A small number of topic-level settings drive most of the operational behavior.

Changing partition count after creation is supported but has consequences: existing records don't re-shuffle, so per-key ordering across the partition count change can break. Picking the right partition count up front matters.

Topics as Streams: Beyond Messaging

The fact that Kafka topics are durable, ordered, replayable logs -- not transient queues -- is what enabled the architectural shift toward Kafka as a streaming platform rather than a message bus.

Event sourcing. Topics become the system of record. Application state is derived from replaying events. State stores (RocksDB-backed in Kafka Streams, Flink's managed state) are rebuilt by replaying compacted topics.

Stream processing. Apache Flink, Kafka Streams, ksqlDB, and Spark Structured Streaming all treat topics as unbounded streams of events to compute over. Joins, aggregations, windows, and pattern detection happen continuously over the topic.

Change data capture. CDC tools like Debezium read database transaction logs and emit row-level changes onto Kafka topics -- often with one topic per source table. Compacted topics make the resulting changelogs replayable.

Data integration. Kafka Connect moves data between topics and external systems (databases, S3, Elasticsearch, OpenSearch, warehouses, search engines). The topic is the integration boundary.

Real-time analytics. Topics feed real-time analytical stores like ClickHouse for sub-second dashboards. The same topic often feeds a warehouse through batch ingestion in parallel.

Common Patterns and Anti-Patterns

One topic per logical event type. A topic should represent a single, well-defined event stream -- orders.created, payments.completed, users.profile-updated. Combining unrelated event types into one topic to "save topics" makes consumers, schema evolution, and retention policies harder.

Use keys deliberately. Keys drive partitioning and therefore ordering. Picking a key that has too few distinct values (e.g., a region with 4 possible values across 50 partitions) creates skew. Picking too many (e.g., a UUID per record when ordering doesn't matter) is fine but wastes the key's purpose.

Don't use topics as a long-term database substitute. Kafka can hold data forever, especially with tiered storage. That doesn't mean it should. For random-access reads of historical state, a warehouse, lakehouse, or operational database is the right tool. Kafka is for streams, not for arbitrary lookups.

Watch out for hot partitions. A handful of high-traffic keys can saturate one partition while others sit idle. Monitor per-partition throughput. For pathological cases, custom partitioning or salted keys are the fix.

Schema management isn't optional. Producers and consumers evolve independently. A schema registry (Confluent Schema Registry, Apicurio, AWS Glue Schema Registry) enforcing backward and forward compatibility prevents silent breakage.

Plan for replay. The whole point of a durable log is replayability. Build consumers that can be reset to an earlier offset cleanly -- idempotent writes, no side effects that can't be re-applied.

Topics Without ZooKeeper: KRaft Mode

Through Kafka 2.x, ZooKeeper held cluster metadata, including topic configurations and partition assignments. KRaft mode -- KIP-500 -- replaced ZooKeeper with a Raft-based metadata quorum inside Kafka itself. KRaft has been the default for new clusters since Kafka 3.3 (production-ready) and ZooKeeper support was removed in Kafka 4.0 (2025).

For topic management this changes nothing at the API level -- the same kafka-topics CLI, the same admin APIs, the same configuration model. It changes a lot operationally: simpler cluster topology, faster controller failover, faster partition creation, and significantly higher partition limits per cluster.

Where Kafka Topics Fit in the Broader Stack

Topics are the durable backbone of modern event-driven and streaming architectures. They decouple producers from consumers, buffer load mismatches, preserve replayable history, and provide the integration substrate that lets dozens of systems share data without point-to-point coupling. Around them sit:

• Stream processors: Apache Flink, Kafka Streams, ksqlDB
• Connectors: Kafka Connect to/from databases, Iceberg lakehouses, warehouses, search engines
• Storage destinations: data lakes, warehouses, ClickHouse, OpenSearch
• Schema management: Confluent Schema Registry, Apicurio
• Observability: lag monitoring, partition skew detection, broker metrics

For a deeper architectural pattern that combines Kafka topics with Flink and ClickHouse for real-time analytics, see The KFC Architecture Blueprint.

BigDataBoutique and Apache Kafka

We design, operate, and tune large Kafka deployments -- topic design, partition strategy, replication and durability, KRaft migrations, performance tuning, and integration with Flink, ClickHouse, and the broader data stack. See our Apache Kafka consulting and Kafka performance tuning services, or get in touch to discuss your architecture.