Orchestrating Your Data Pipelines with Apache Airflow • Ricardo Sueiras • GOTO 2022

This presentation was recorded at GOTO Amsterdam 2022. #GOTOcon #GOTOams http://gotoams.nl Ricardo Sueiras - Principal Advocate in Open Source at AWS & Experienced Coach & Mentor @ricardoasueiras RESOURCES GitHub Repo: https://github.com/094459/building-data-pipelines-apache-airflow https://airflow.apache.org ABSTRACT Apache Airflow continues to gain adoption for its ability to configure data pipelines as scripts, monitor them with a great user interface, and extend their functionality through a set of community-developed and custom plugins. In this session, Ricardo will introduce Apache Airflow and its key components and then dive into building some simple pipelines exploring these and so you can begin to get an idea of how you can use Apache Airflow to orchestrate your own data pipelines. [...] TIMECODES 00:00 Intro 01:10 What is in your data pipeline? 02:33 Introducing Apache Airflow 12:11 Demo 18:13 Building our data pipeline demo 20:23 Demo 30:52 Apache Airflow evolution 31:48 How to contribute 32:10 Outro Read the full abstract here: https://gotoams.nl/2022/sessions/2014/orchestrating-your-data-pipelines-with-apache-airflow RECOMMENDED BOOKS Bas P. Harenslak & Julian Rutger de Ruiter • Data Pipelines with Apache Airflow • https://amzn.to/3HlM0LC Danil Zburivsky & Lynda Partner • Designing Cloud Data Platforms • https://amzn.to/3Rgr6lD Gareth Eagar • Data Engineering with AWS • https://amzn.to/3DqiGCN Martin Kleppmann • Designing Data-Intensive Applications • https://amzn.to/3wEWLnn https://twitter.com/GOTOcon https://www.linkedin.com/company/goto- https://www.instagram.com/goto_con https://www.facebook.com/GOTOConferences #ApacheAirflow #Airflow #AWS #Orchestrating #DataPipelines #OpenSource #AirflowDAG CHANNEL MEMBERSHIP BONUS Join this channel to get early access to videos & other perks: https://www.youtube.com/channel/UCs_tLP3AiwYKwdUHpltJPuA/join Looking for a unique learning experience? Attend the next GOTO conference near you! Get your ticket at https://gotopia.tech Sign up for updates and specials at https://gotopia.tech/newsletter SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily. https://www.youtube.com/user/GotoConferences/?sub_confirmation=1