How Data & Software Eng. Teams Collaborate to Ensure Smooth Data Integrations • Sam Bail • GOTO 2023

This presentation was recorded at GOTO Chicago 2023. #GOTOcon #GOTOchgo https://gotochgo.com Sam Bail - Principal Data Engineer at Collectors ORIGINAL TALK TITLE Bridging the Gap: How Data and Software Engineering Teams Can Work Together to Ensure Smooth Data Integrations RESOURCES https://sambail.com https://github.com/spbail https://linkedin.com/in/spbail ABSTRACT Imagine this scenario: The engineering team at your company is about to launch a new product or feature. The pressure is on the data team to deliver metrics and stats from the get-go, as everyone is keen to see how the new feature is performing. Behind the scenes, the data team is scrambling to understand how to connect to the new systems, transform the data, interpret it, and turn it into actionable insights. In this talk, I will walk the audience through the different steps necessary to ensure both data teams and software engineers are ready to integrate with a new production system and deliver correct and reliable analytics from day 1. I will cover topics such as infrastructure and data access considerations, data models, data flow and state changes, and data quality checks. I will also touch upon “data contracts”, a concept that’s been talked about a lot recently in the data space, and how these can be established to preserve consistent high-quality data integrations. Both data practitioners and software engineers will benefit from this talk and learn how to “bridge the gap” between the two worlds. [...] TIMECODES 00:00 Intro 00:46 TL;DR 01:55 About me 03:31 Agenda 04:16 The problem 06:42 1. Logistics 13:44 2. Infrastructure 19:29 3. Data model 26:40 4. Application & data flow 33:27 Data contracts 38:13 Wrapping up 41:48 Outro Download slides and read the full abstract here: https://gotochgo.com/2023/sessions/2572 RECOMMENDED BOOKS Phil Winder • Reinforcement Learning • https://amzn.to/3t1S1VZ Kelleher & Tierney • Data Science (The MIT Press Essential Knowledge series) • https://amzn.to/3AQmIRg Lakshmanan, Robinson & Munn • Machine Learning Design Patterns • https://amzn.to/2ZD7t0x Lakshmanan, Görner & Gillard • Practical Machine Learning for Computer Vision • https://amzn.to/3m9HNjP Aurélien Géron • Hands-On Machine Learning with Scikit-Learn, Keras & TensorFlow • https://amzn.to/2XZaQy8 https://twitter.com/GOTOcon https://www.linkedin.com/company/goto- https://www.instagram.com/goto_con https://www.facebook.com/GOTOConferences #DataScience #DataEngineering #DataTeam #SoftwareEngineering #Programming #DataIntegration #DataFlow #ApplicationFlow #SLA Looking for a unique learning experience? Attend the next GOTO conference near you! Get your ticket at https://gotopia.tech Sign up for updates and specials at https://gotopia.tech/newsletter SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily. https://www.youtube.com/user/GotoConferences/?sub_confirmation=1