Kafka Deep-Dive Workshop Training Course - BigData Boutique

Kafka Deep-Dive Workshop

Everything you need to know to work with Kafka effectively and optimize your Kafka clusters

1 day
20
Instructor-led, hands-on exercises
Hebrew / English
Bring your own (installation instructions will be sent prior to course start)
Included
Enroll Now
Itamar Syn-Hershko Itamar Syn-Hershko
Itamar is a BigData and search technologies expert, and BigData Boutique's CTO. Equipped with many years of experience building and optimizing data platforms for organizations in various scales, Itamar's current job is to keep BigData Boutique's team engaged with the latest technologies and working with customers to optimize their journey in the world of BigData.

Kafka is the world's most widely used queue system and message bus, and knowing how to use it correctly is paramount to keeping it stable and highly-performant.

This one day hands-on workshop is designed to get you into the details, and sharing a lot of good advice and important information. Learn directly from our experts who consult on Kafka projects daily how Kafka should be used, and what mistakes to avoid.

Objectives

You will learn:

  • Using Kafka correctly.
  • Message formats and schemas.
  • Message delivery guarantees.
  • Maximizing throughput and minimizing latency.
  • Durability and high-availability.
  • Monitoring and performance tuning.

Prerequisites

Developers, DevOps and SREs with at least some Kafka knowledge and experience. Platform agnostic - hands-on exercises are using Kibana end-to-end.

Syllabus

  • Basic overview of Kafka
  • Broker, Consumer, Producer and Consumer groups
  • Topics, partitions and replicas
  • Logs, segments and lag
  • Working with Kafka
  • Measuring a naive pipeline with default values
  • Compression impact
  • Message formats (Avro/Json/Protobuf)
  • Bulking
  • Acks
  • Delivery semantics - At most once / At least once
  • Tweaking broker thread pools
  • Product wants to see their metrics within 5 seconds - now what?
  • Measuring latency
  • Understanding Kafka artificial delays
  • What about delays coming from our application?
  • Consumer lag analysis and responses for Downstream issues
  • We just made throughput optimizations, what is their impact on latency?
  • Can't I just add more partitions?
  • How does the broker state impact our latency?
  • Are we highly available?
  • In-Sync replicas & acks
  • Replication lag
  • Quotas
  • Data retention - compacted topics vs retention by time/size
  • Retry mechanisms in Kafka
  • Consumer lag analysis and responses for Broker outages
  • Schema evolution with schema registry
  • Ordering guarantees in Kafka - Writing a custom partition selection
  • Idempotency
  • Transactions - Read Committed vs Read Uncommitted
  • Manual offset commits
  • Consumer group rebalances and their impact (Consumer assignment strategies, Group member IDs)
  • Analysis of temporary failures (Overview of timeouts configurations and other potential failures)
  • Avoiding Consumer Group rebalances shaking things up due to temporary failures
  • Right number of partitions
  • Analyzing Replication costs
  • Calculating compute costs
  • Data retention costs
  • Separation of concerns (Multi cluster / single cluster etc)
  • Fault tolerance with Consumer and Producers
  • Load testing and making sure our Kafka cluster and applications are up to the task
  • Transactions - Read Committed vs Read Uncommitted
  • Monitoring and whats behind the metrics
  • Load distribution
  • Take aways to your daily work

Ready to get started?

Enroll Now
We use cookies to provide an optimized user experience and understand our traffic. To learn more, read our use of cookies; otherwise, please choose 'Accept Cookies' to continue using our website.