A practical guide to Kafka usage patterns including Pub/Sub vs Observer, solving dual writes, CQRS, Event Sourcing, Event Collaboration, and the Outbox pattern - with guidance on when to use each and anti-patterns to avoid.

Kafka has become a cornerstone of modern distributed architectures, but its flexibility can be a double-edged sword. Engineers often struggle to determine when Kafka is the right tool and which patterns to apply. This post covers practical usage patterns - from basic Pub/Sub to advanced Event Collaboration - with clear guidance on trade-offs and when each pattern makes sense.

Understanding these patterns is not about adopting the most sophisticated approach, but about choosing the right level of complexity for your specific requirements. Let's start with the fundamentals.

Observer vs Pub/Sub: Understanding the Core Difference

The Observer pattern establishes direct coupling between a subject and its observers. When the subject changes state, it directly notifies all registered observers. This tight coupling has its place - reactive programming frameworks leverage it to enable back-pressure negotiation, where observers can signal the subject to slow down when overwhelmed.

Pub/Sub introduces an intermediary: the event channel. Publishers write messages without knowing who consumes them; subscribers receive messages without knowing their origin. This decoupling is Kafka's strength. Publishers and subscribers can evolve independently - you can add new consumers, scale existing ones, or replace implementations without affecting other parts of the system.

Kafka excels specifically in scenarios with multiple subscribers consuming the same data stream. While traditional message queues can implement Pub/Sub, Kafka's architecture - with its persistent, partitioned log - is purpose-built for high-throughput, multi-consumer scenarios. This architectural decision has implications for the patterns we'll explore next.

Solving Dual Writes with Kafka

Consider a common scenario: a web application that must write to a relational database, update a Redis cache, and index data in Elasticsearch. With direct writes (the Observer pattern approach), you face several problems:

  • Partial failures: What happens when the database write succeeds but Elasticsearch is unavailable?
  • Inconsistent state: Different systems may have different versions of the same data.
  • Back-pressure cascading: The slowest system dictates overall throughput, and your application must buffer data locally.

Kafka provides a clean solution. Your application writes to Kafka once - a fast, highly-available, replicated operation. Each downstream system consumes from Kafka at its own pace:

Web App → Kafka Topic → Consumer A → PostgreSQL
                       → Consumer B → Redis
                       → Consumer C → Elasticsearch

This pattern is particularly valuable with Elasticsearch, where indexing throughput varies with cluster load. By placing Kafka in front, you absorb traffic spikes without losing data. The consumers can catch up during quieter periods.

The key insight is that Kafka acts as a buffer that absorbs the mismatch between your application's write rate and your downstream systems' processing capacity. Your application is no longer coupled to the performance characteristics of any individual database.

CQRS and Event Sourcing: When Complexity Pays Off

CQRS (Command Query Responsibility Segregation) separates your system into a command model (writes) and a query model (reads). This separation allows you to optimize each path independently. With Kafka and a stream processing framework like Kafka Streams or Flink, the command path writes events to Kafka, and processors transform these events into query-optimized views.

When CQRS makes sense: Complex queries requiring aggregations, joins across entities, or pre-computed views. If your query model benefits from denormalized data structures that differ significantly from your write model, CQRS can eliminate expensive real-time computations.

When to avoid CQRS: Simple CRUD applications. If your reads and writes operate on the same data shape, CQRS adds architectural complexity without proportional benefit.

Event Sourcing takes this further by treating the event log as the source of truth. Instead of storing current state, you store the sequence of events that led to that state. Kafka Streams' state stores provide in-memory, distributed key-value storage that can be rebuilt by replaying events from the beginning of a topic.

Kafka's compaction feature supports this pattern by periodically removing older versions of records with the same key, keeping only the latest. However, a critical warning: do not use compaction for business logic like deduplication. Compaction is a space optimization that runs on its own schedule - it's not a real-time guarantee. For deduplication, use proper stream processing logic in Kafka Streams or your application layer.

Event Collaboration and the Outbox Pattern

Event Collaboration extends Event Sourcing across service boundaries. Each microservice publishes state changes to Kafka, and other services maintain local replicas of the data they need. When Service A publishes an order update, Service B - which needs order data - applies that event to its local state store.

The power of this pattern lies in fault isolation. If Service A becomes unavailable, Service B continues operating with its local copy. Services are decoupled not just in code, but in runtime availability.

Adopting Event Collaboration requires organizational change. Teams must shift from "I provide APIs that others call" to "I publish my state changes and let others consume what they need." This raises legitimate concerns:

  • Contract management: You need API-first practices for asynchronous events, not just REST endpoints.
  • Consumer visibility: Implement a self-service registry where teams request topic access, giving producers visibility into who consumes their data and enabling coordinated schema evolution.

For legacy systems that cannot publish directly to Kafka, the Outbox pattern provides a bridge. Instead of modifying the application to write to Kafka, you create an outbox table in its database. The application writes events to this table as part of its normal transactions. A Change Data Capture (CDC) tool like Debezium monitors the table and publishes new rows to Kafka.

This approach works because writing to a database table is something legacy systems already do. The outbox table acts as a contract - a structured format designed for downstream consumption, rather than raw database change events that expose internal schema details.

Anti-Patterns to Avoid

Kafka is not a silver bullet. The most common anti-pattern is treating Kafka as a drop-in replacement for traditional message queues. Key differences to remember:

  • Ordering and exactly-once semantics: While Kafka supports these, MQ systems were designed for transactional, ordered delivery from the start. If you need strict ordering or exactly-once processing between two services, evaluate whether a traditional MQ might be simpler.
  • Single consumer scenarios: Kafka's architecture shines with multiple consumers. For point-to-point messaging, the overhead may not be justified.

Another anti-pattern is building heavy abstraction layers on top of Kafka - custom authentication frameworks, organization-wide message schemas, proprietary client libraries. The Kafka ecosystem evolves rapidly, and these custom layers become expensive to maintain and update. Instead:

  • Provide Kafka as a service with light governance.
  • Focus on self-service access control and schema registries.
  • Let teams use standard Kafka clients and connectors.

Finally, avoid using a single Kafka cluster for everything. Logging, analytics/CDC, and microservice integration have different requirements. Separate clusters allow you to tune configurations, manage capacity, and evolve independently.

Key Takeaways

  • Pub/Sub decoupling is Kafka's core strength - use it to isolate systems and manage back-pressure.
  • Dual writes are an anti-pattern; let Kafka buffer between your application and downstream systems.
  • CQRS and Event Sourcing add complexity - adopt them only when query optimization or event replay genuinely benefits your use case.
  • Event Collaboration requires organizational buy-in and tooling (registries, schema management) to succeed.
  • The Outbox pattern bridges legacy systems to event-driven architectures via CDC.
  • Avoid the ESB trap: don't over-engineer Kafka with custom frameworks. Keep it simple and let the ecosystem do the heavy lifting.

Choose the pattern that matches your actual requirements. Start simple, and evolve toward more sophisticated patterns only when the trade-offs justify the added complexity.