Running Kafka in Kubernetes: a practical guide by Katherine Stanley

The rise of Apache Kafka as the de-facto standard for event streaming has coincided with the rise of Kubernetes for cloud-native applications. While Kubernetes is a great choice for any distributed system, that doesn't mean it is easy to deploy and maintain a Kafka cluster running on it. At IBM we have hands-on experience with running Kafka in Kubernetes and in this session I will share our top tips for a smooth ride. I will show an example deployment of Kafka on Kubernetes and step through the system to explain the common pitfalls and how to avoid them. This will include the Kubernetes objects to use, resource considerations and connecting applications to the cluster. I will also discuss useful Kafka metrics to include in Kubernetes liveness and readiness probes. Finally I will introduce some of the tools available to help automate the management of your Kafka deployment.