Kafka, DevOps...And Resilience for All • James Bowkett • GOTO 2021

This presentation was recorded at GOTOpia Chicago 2021. #GOTOcon #GOTOpia http://gotochgo.com James Bowkett - Lead Consultant at OpenCredo @OpencredoItd ABSTRACT So you’ve made the enlightened choice to use Kafka in your architecture, and you know you need to provision your cluster using infrastructure as code (IAC), but why stop at just provisioning brokers? In this talk we will explore the available options to make deploying your Kafka-based applications more repeatable, resilient and observable. We’ll look at specific examples and techniques for applying IAC to the following areas : • Topic and partition creation • Secrets management • Configuration options I’ll also share some lessons I’ve learned related to tooling, monitoring and backup management. And finally we’ll look at some resilience do’s and don’ts with some specific examples of where these tools have helped, and where their absence has hindered [...] TIMECODES 00:00 Intro 02:16 Kafka - what's the difference? 03:49 Key concepts 07:30 What can I DevOpsify? 10:40 What are the options? 11:21 Bash 12:17 Terraform 13:33 Kafka config manager 14:47 Secrets management 16:48 In an ideal world 18:33 In a k8s-native world 20:27 Backups 21:54 Backup options 25:37 What about application parameters? 31:20 Monitoring 35:18 Monitoring tooling 36:33 Tooling - commercial 37:14 Tooling - open source 38:01 Key takeaways 40:12 Outro Download slides and read the full abstract here: https://gotochgo.com/2021/sessions/1758 RECOMMENDED BOOKS Liz Rice • Container Security • https://amzn.to/3oU4iJe Liz Rice • Kubernetes Security • https://www.oreilly.com/library/view/kubernetes-security/9781492039075 Brendan Burns, Joe Beda & Kelsey Hightower • Kubernetes: Up and Running • https://amzn.to/3wrtwlp Chris Richardson • Microservices Patterns • https://amzn.to/2SOnQ7h John Arundel & Justin Domingus • Cloud Native DevOps with Kubernetes • https://amzn.to/3hKZvI5 Michael Hausenblas & Stefan Schimanski • Programming Kubernetes • https://amzn.to/3qTvKch Sam Newman • Monolith to Microservices • https://amzn.to/2Nml96E Sam Newman • Building Microservices • https://amzn.to/3dMPbOs Ronnie Mitra & Irakli Nadareishvili • Microservices: Up and Running• https://amzn.to/3c4HmmL Mitra, Nadareishvili, McLarty & Amundsen • Microservice Architecture • https://amzn.to/3fVNAb0 https://twitter.com/GOTOcon https://www.linkedin.com/company/goto- https://www.facebook.com/GOTOConferences #ApacheKafka #Kafka #DevOps #Resilience #IAC #Bash #Terraform #Kubernetes #k8s #MirrorMaker Looking for a unique learning experience? Attend the next GOTO conference near you! Get your ticket at https://gotopia.tech SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily. https://www.youtube.com/user/GotoConferences/?sub_confirmation=1