Kafka: a map of traps for the enlightened dev and op by Emmanuel Bernard And Clement Escoffier

While looking simple, Kafka hides a surprising complexity. In this Deep Dive, we will explore a map of Kafka kingdom's traps, monsters, and treasures. First from the point of view of the **client side app and its developers**: serialization, partitioning in a polyglot world, impact of rebalancing, retry and orders, assignment policy, commits and schema loss, DLQ problems, etc. Then from the point of view of the **operations**: how to configure the Kafka cluster, how to validate that it works as expected, which metrics are useful, how are guarantees managed, what’s the impact of sharing resources between a Kafka and… another Kafka. This deep dive will make you explore advanced concepts that influence the development of a well-behaving Kafka application. And because we’ve got to go live one day, we will discover how a cloud provider handles its fleet of Kafka at scale. This retrospective comes from the work Clément and Emmanuel did around Kafka libraries development and the management of Kafka in the cloud. Dev traps, ops traps, they will be discovered through a fun and practical focused recipe format. EMMANUEL BERNARD Emmanuel is Java Champion, Distinguished Engineer and Chief Architect Application Cloud Services at Red Hat. His work is Open Source. He is most well known for his contributions and lead of the [Quarkus](https://quarkus.io) and [Hibernate](https://hibernate.org) projects as well as his contribution to Java standards. His most recent endeavour is building a [Kafka as a Service managed by Red Hat](https://red.ht/trykafka). Prior to that was [Quarkus](https://quarkus.io) (A Kubernetes Native Java stack tailored for GraalVM & OpenJDK HotSpot, crafted from the best of breed Java libraries and standards). He speaks regularly at various conferences and JUGs, including JavaOne, Red Hat Summit and Devoxx. He hosts several podcasts including Les Cast Codeurs. You can follow him on twitter at @emmanuelbernard #kafka #quarkus #debezium #infinispan #hibernate-orm #hibernate-search #hibernate-validator #hibernate-ogm #ceylon #JPA #bean-validation #lescastcodeurs CLEMENT ESCOFFIER Clement Escoffier (@clementplop) is a senior principal software engineer at Red Hat. He is a Java Champion and also acts as a Reactive Architect at Red Hat. Before joining Red Hat, Clement had several professional lives, from academic positions to management. He contributed to projects and products, touching many domains and technologies such as OSGi, mobile, continuous delivery, and DevOps. Clement has always been interested in software engineering, distributed systems, and event-driven architectures. More recently, he focused on Reactive Systems, Cloud-Native applications, and Kubernetes. Clement is an active contributor to many open-source projects such as Apache Felix, Eclipse Vert.x, SmallRye, Mutiny, and Quarkus. I recently authored the "Reactive Systems in Java" book. ------------------------------------------------------------ INTRO * visuals & editing by @Mercator * music : Avocado by Ephixa