Kafka Connect: Build & Run Data Pipelines • Kate Stanley, Mickael Maison & Danica Fine • GOTO 2024

This interview was recorded for the GOTO Book Club. #GOTOcon #GOTObookclub http://gotopia.tech/bookclub Read the full transcription of the interview here: https://gotopia.tech/episodes/341 Kate Stanley - Principal Software Engineer at Red Hat & Co-Author of "Kafka Connect" Mickael Maison - Senior Principal Software Engineer at Red Hat & Co-Author of "Kafka Connect" @MickaelMaison Danica Fine - Lead Developer Advocate, Open Source at Snowflake @DanicaFine RESOURCES Kate https://x.com/KateStanley91 https://fosstodon.org/@katheris https://github.com/katheris https://www.linkedin.com/in/katherine-kate-stanley-3796b579 Mickael https://bsky.app/profile/mickaelmaison.bsky.social https://mas.to/@MickaelMaison https://x.com/MickaelMaison https://github.com/mimaison https://stackoverflow.com/users/1765189/mickael-maison https://www.linkedin.com/in/mickaelmaison https://mickaelmaison.com Danica https://bsky.app/profile/thedanicafine.bsky.social https://x.com/TheDanicaFine https://data-folks.masto.host/@thedanicafine https://www.linkedin.com/in/danica-fine https://github.com/danicafine https://linktr.ee/thedanicafine Links https://kafka.apache.org https://flink.apache.org https://debezium.io https://strimzi.io DESCRIPTION Used by more than 80% of Fortune 100 companies, Apache Kafka has become the de facto event streaming platform. Kafka Connect is a key component of Kafka that lets you flow data between your existing systems and Kafka to process data in real time. With this practical guide, authors Mickael Maison and Kate Stanley show data engineers, site reliability engineers, and application developers how to build data pipelines between Kafka clusters and a variety of data sources and sinks. Kafka Connect allows you to quickly adopt Kafka by tapping into existing data and enabling many advanced use cases. No matter where you are in your event streaming journey, Kafka Connect is the ideal tool for building a modern data pipeline. • Learn Kafka Connect's capabilities, main concepts, and terminology • Design data and event streaming pipelines that use Kafka Connect • Configure and operate Kafka Connect environments at scale • Deploy secured and highly available Kafka Connect clusters • Build sink and source connectors and single message transforms and converters * Book description: © O'Reilly: https://www.oreilly.com/library/view/kafka-connect/9781098126520 TIMECODES 00:00 Intro 03:48 Understanding Kafka Connect & its versatility 07:31 Key use cases for Kafka Connect 13:38 Challenging Kafka Connect terminology 16:25 Second section of the book focuses on data engineers 25:11 Optimizing Kafka Connect 43:08 Expanding Kafka Connect: Resources, insights & community 48:00 Outro RECOMMENDED BOOKS Kate Stanley & Mickael Maison • Kafka Connect • https://amzn.to/40Jq5Jz Shapira, Palino, Sivaram & Petty • Kafka: The Definitive Guide • https://amzn.to/3ULdz8A Viktor Gamov, Dylan Scott & Dave Klein • Kafka in Action • https://amzn.to/3CrB8KU Adrienne Braganza Tacke • "Looks Good to Me": Constructive Code Reviews • https://www.manning.com/books/looks-good-to-me Martin Fowler • Refactoring • https://amzn.to/3EVcHXQ Matthew Skelton & Manuel Pais • Team Topologies • http://amzn.to/3sVLyLQ https://twitter.com/GOTOcon https://www.linkedin.com/company/goto- https://www.instagram.com/goto_con https://www.facebook.com/GOTOConferences #KafkaConnect #ApacheKafka #Kafka #DataPipeline #EventStreaming #CloudNative #SoftwareEngineering #Programming #KateStanley #MickaelMaison #DanicaFine #BookClub CHANNEL MEMBERSHIP BONUS Join this channel to get early access to videos & other perks: https://www.youtube.com/channel/UCs_tLP3AiwYKwdUHpltJPuA/join Looking for a unique learning experience? Attend the next GOTO conference near you! Get your ticket at https://gotopia.tech SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily. https://www.youtube.com/user/GotoConferences/?sub_confirmation=1