Optimizing Amazon EventBridge Pipes with Apache Kafka • Daniele Frasca & Roman Boiko • GOTO 2024

This presentation was recorded at GOTO EDA Day 2024. #GOTOcon #GOTOeda #GOTOldn http://gotoldn.com Daniele Frasca - Team Lead of Content Discovery and AWS Serverless Community Builder Roman Boiko - Specialist Solution Architect for Serverless ORIGINAL TALK TITLE Optimizing Amazon EventBridge Pipes with Apache Kafka at ProSiebenSat.1 RESOURCES Daniele https://twitter.com/dfrasca80 https://www.linkedin.com/in/daniele-frasca https://dfrasca.hashnode.dev https://github.com/ymwjbxxq Roman https://twitter.com/romannboiko https://www.linkedin.com/in/romboiko https://github.com/roman-boiko ABSTRACT ProSiebenSat.1 Media SE is one of the largest German television companies operating in the entertainment sector, offering various products. Joyn, their streaming application used in the DACH regions across multiple platforms. Serving millions of customers during major events, such as popular shows or live sports, is crucial, where customers expect a fast, seamless experience. Slow responses or unavailability leads to frustration, dissatisfaction, and loss of trust. In this presentation, we will guide you through the journey of optimizing Kafka within our system, starting from its initial status upon takeover. We will tell why we went with Confluent Kafka and not with Amazon MSK. We will look into integrating Kafka with Amazon EventBridge Pipes and share encountered limitations and issues. Next, we will introduce the specific architectural patterns such as the Claim-check and Point-to-point fan-out to enhance resilience and efficiency. The talk also outlines a wishlist for AWS, highlighting desired improvements in payload size, monitoring capabilities, and processing speed. We will end with lessons learned and strategies for leveraging Kafka more effectively in complex systems. [...] TIMECODES 00:00 Intro 01:44 The challenge 03:57 Why Kafka? 09:12 EventBridge with Kafka 11:18 EventBridge with Kafka alternative 17:05 How we did this 23:26 Implemented solution 29:51 AWS wishes 34:49 Summary 36:25 Outro Download slides and read the full abstract here: https://gotoldn.com/2024-eda-aws-day/sessions/3171 RECOMMENDED BOOKS James Urquhart • Flow Architectures • https://amzn.to/3Tyz8cY Adam Bellemare • Building Event-Driven Microservices • https://amzn.to/3WfNKfM Peter Sbarski • Serverless Architectures on AWS • https://amzn.to/3hJzEUM Michael Stack • Event-Driven Architecture in Golang • https://amzn.to/3G5e8ST Ford, Richards, Sadalage & Dehghani • Software Architecture: The Hard Parts • https://amzn.to/3v4pKQS Gerardus Blokdyk • Event-Driven Architecture EDA • https://amzn.to/3FOfUHE https://twitter.com/GOTOcon https://www.linkedin.com/company/goto- https://www.instagram.com/goto_con https://www.facebook.com/GOTOConferences #EventBridge #ApacheKafka #ProSiebenSat1 #AWS #EDA #EventDriven #EventDrivenArchitecture #Serverless #DanieleFrasca #RomanBoko Looking for a unique learning experience? Attend the next GOTO conference near you! Get your ticket at https://gotopia.tech Sign up for updates and specials at https://gotopia.tech/newsletter SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily. https://www.youtube.com/user/GotoConferences/?sub_confirmation=1