Kafka Deep-Dive Workshop

• Basic overview of Kafka
• Broker, Consumer, Producer and Consumer groups
• Topics, partitions and replicas
• Logs, segments and lag
• Working with Kafka

• Measuring a naive pipeline with default values
• Compression impact
• Message formats (Avro/Json/Protobuf)
• Bulking
• Acks
• Delivery semantics - At most once / At least once
• Tweaking broker thread pools

• Product wants to see their metrics within 5 seconds - now what?
• Measuring latency
• Understanding Kafka artificial delays
• What about delays coming from our application?
• Consumer lag analysis and responses for Downstream issues
• We just made throughput optimizations, what is their impact on latency?
• Can't I just add more partitions?
• How does the broker state impact our latency?

• Are we highly available?
• In-Sync replicas & acks
• Replication lag
• Quotas
• Data retention - compacted topics vs retention by time/size
• Retry mechanisms in Kafka
• Consumer lag analysis and responses for Broker outages
• Schema evolution with schema registry

• Ordering guarantees in Kafka - Writing a custom partition selection
• Idempotency
• Transactions - Read Committed vs Read Uncommitted
• Manual offset commits
• Consumer group rebalances and their impact (Consumer assignment strategies, Group member IDs)
• Analysis of temporary failures (Overview of timeouts configurations and other potential failures)
• Avoiding Consumer Group rebalances shaking things up due to temporary failures

Just kidding, the right title is "Deployment, Sizing and Costs".

• Right number of partitions
• Analyzing Replication costs
• Calculating compute costs
• Data retention costs
• Separation of concerns (Multi cluster / single cluster etc)

• Fault tolerance with Consumer and Producers
• Load testing and making sure our Kafka cluster and applications are up to the task
• Transactions - Read Committed vs Read Uncommitted
• Monitoring and whats behind the metrics
• Load distribution
• Take aways to your daily work