Cloud Native Data Pipelines • Sid Anand • GOTO 2017

This presentation was recorded at GOTO Chicago 2017. #GOTOcon #GOTOchgo http://gotochgo.com Sid Anand - Data Architect at Agari ABSTRACT Big Data companies (e.g. LinkedIn, Facebook, Google, and Twitter) have historically built custom data pipelines over bare metal in custom-designed data centers. In order to meet strict requirements on data security, fault-tolerance, cost control, job scalability, network topology, and compute and storage placement, they need to closely manage their core [...] TIMECODES 0:00 Introduction 0:16 About Me 1:06 Agari : What We Do 4:55 Cloud Native Data Pipelines 6:26 Desirable Qualities of a Resilient Data Pipeline 10:23 Use-Case : Message Scoring 12:58 Architectural Components 16:55 Tackling Cost 23:09 ASG : Queue-based 24:40 ASG - Build & Deploy 27:33 Tackling Operability : Requirements 28:27 Apache Airflow - Authoring DAGS 28:56 Apache Airflow - Perf. Insights 31:15 Apache Airflow - Alerting 33:29 Stream Processing Architecture 39:10 What is Avro? 40:38 Why is Avro Useful? 43:23 Avro Schema Example 43:53 Avro Schema Data File Example 44:26 Avro Schema Streaming Example 45:01 Avro Schema Registry Download slides and read the full abstract here: https://gotochgo.com/2017/sessions/80 RECOMMENDED BOOKS Liz Rice • Container Security • https://amzn.to/3oU4iJe Liz Rice • Kubernetes Security • https://www.oreilly.com/library/view/kubernetes-security/9781492039075 Brendan Burns, Joe Beda & Kelsey Hightower • Kubernetes: Up and Running • https://amzn.to/3wrtwlp John Arundel & Justin Domingus • Cloud Native DevOps with Kubernetes • https://amzn.to/3hKZvI5 Kasun Indrasiri & Sriskandarajah Suhothayan • Design Patterns for Cloud Native Applications • https://amzn.to/3yCFxWE Michael Hausenblas & Stefan Schimanski • Programming Kubernetes • https://amzn.to/3qTvKch Alexander Raul • Cloud Native with Kubernetes • https://amzn.to/3yw9ckc Nigel Poulton • The Kubernetes Book • https://amzn.to/3dW8ViU Marko Luksa • Kubernetes in Action • https://amzn.to/3dXk2Im https://twitter.com/GOTOcon https://www.linkedin.com/company/goto- https://www.facebook.com/GOTOConferences #CloudNative #DataPipelines #AWScloud Looking for a unique learning experience? Attend the next GOTO conference near you! Get your ticket at https://gotopia.tech Sign up for updates and specials at https://gotopia.tech/newsletter SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily. https://www.youtube.com/user/GotoConferences/?sub_confirmation=1