Kafka Deep-Dive Workshop
Everything you need to know to work with Kafka effectively and optimize your Kafka clusters
1 day
20
Instructor-led, hands-on exercises
Hebrew / English
Bring your own (installation instructions will be sent prior to course start)
Included
Kafka is the world's most widely used queue system and message bus, and knowing how to use it correctly is paramount to keeping it stable and highly-performant.
This one day hands-on workshop is designed to get you into the details, and sharing a lot of good advice and important information. Learn directly from our experts who consult on Kafka projects daily how Kafka should be used, and what mistakes to avoid.
Objectives
You will learn:
- Using Kafka correctly.
- Message formats and schemas.
- Message delivery guarantees.
- Maximizing throughput and minimizing latency.
- Durability and high-availability.
- Monitoring and performance tuning.
Prerequisites
Developers, DevOps and SREs with at least some Kafka knowledge and experience.
Syllabus
- Basic overview of Kafka
- Broker, Consumer, Producer and Consumer groups
- Topics, partitions and replicas
- Logs, segments and lag
- Working with Kafka
- Measuring a naive pipeline with default values
- Compression impact
- Message formats (Avro/Json/Protobuf)
- Bulking
- Acks
- Delivery semantics - At most once / At least once
- Tweaking broker thread pools
- Product wants to see their metrics within 5 seconds - now what?
- Measuring latency
- Understanding Kafka artificial delays
- What about delays coming from our application?
- Consumer lag analysis and responses for Downstream issues
- We just made throughput optimizations, what is their impact on latency?
- Can't I just add more partitions?
- How does the broker state impact our latency?
- Are we highly available?
- In-Sync replicas & acks
- Replication lag
- Quotas
- Data retention - compacted topics vs retention by time/size
- Retry mechanisms in Kafka
- Consumer lag analysis and responses for Broker outages
- Schema evolution with schema registry
- Ordering guarantees in Kafka - Writing a custom partition selection
- Idempotency
- Transactions - Read Committed vs Read Uncommitted
- Manual offset commits
- Consumer group rebalances and their impact (Consumer assignment strategies, Group member IDs)
- Analysis of temporary failures (Overview of timeouts configurations and other potential failures)
- Avoiding Consumer Group rebalances shaking things up due to temporary failures
Just kidding, the right title is "Deployment, Sizing and Costs".
- Right number of partitions
- Analyzing Replication costs
- Calculating compute costs
- Data retention costs
- Separation of concerns (Multi cluster / single cluster etc)
- Fault tolerance with Consumer and Producers
- Load testing and making sure our Kafka cluster and applications are up to the task
- Transactions - Read Committed vs Read Uncommitted
- Monitoring and whats behind the metrics
- Load distribution
- Take aways to your daily work
