Resilient Predictive Data Pipelines • Siddharth "Sid" Anand • GOTO 2016

This presentation was recorded at GOTO Chicago 2016 http://gotochgo.com Siddharth "Sid" Anand - Data Architect at Agari Inc ABSTRACT Big Data companies (e.g. LinkedIn, Facebook, Google, and Twitter) have historically built custom data pipelines over bare metal in custom-designed data centers. In order to meet strict requirements on data security, fault-tolerance, cost control, job scalability, and uptime, they need [...] TIMECODES 0:00 Introduction 0:33 About Me 3:21 Motivation 3:50 Data Products 6:10 Serving + Data Pipelines 8:05 The Blast Radius Problem 12:11 Timeliness 20:52 SOS - Dead Letter Queue 22:52 SNS + SOS Design Pattern 25:02 What Does Agari Do? 33:22 Apache Airflow - Authoring DAGS 34:25 Apache Airflow - Perf. Insights 34:50 Apache Airflow - Alerting 35:53 NRT Architecture 39:10 Schema Registry 40:53 What is AWS Lambda? 41:54 Elastic Stream Processing 42:30 Open Source Plans 42:38 Questions? (@r39132) 42:44 SOFTWARE DEVELOPMENT CONFERENCE Download slides and read the full abstract here: https://gotocon.com/chicago-2016/presentation/Resilient%20Predictive%20Data%20Pipelines https://twitter.com/gotochgo https://www.facebook.com/GOTOConference http://gotocon.com Looking for a unique learning experience? Attend the next GOTO conference near you! Get your ticket at https://gotopia.tech Sign up for updates and specials at https://gotopia.tech/newsletter SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily. https://www.youtube.com/user/GotoConferences/?sub_confirmation=1