10 Learnings From Running Production Infrastructure at Google • Christof Leng • GOTO 2023

This presentation was recorded at GOTO Amsterdam 2023. #GOTOcon #GOTOams https://gotoams.nl Christof Leng - Lead for Google's SRE Engagement Model and SRE Review Programs @ChristofLeng ORIGINAL TALK TITLE Ten Things We've Learned From Running Production Infrastructure at Google RESOURCES https://www.oreilly.com/library/view/enterprise-roadmap-to/9781098117740 https://sre.google/resources Christof https://twitter.com/Moroquen https://github.com/christof-leng https://linkedin.com/in/christofleng ABSTRACT Google’s production infrastructure might be one of the most complex machines that humanity has built so far. It is constantly changing and evolving. Site Reliability Engineers (SREs) are the specialists to manage and improve the architectures, tooling, and operational procedures that enable Google to keep its products reliable, scalable, efficient, and agile. This talk will discuss a number of fundamental organizational principles that Google SRE has learned over the years. [...] TIMECODES 00:00 Intro 05:26 Culture 07:00 1. Reliability can't be taken for granted 10:42 2. Cattle vs. Pets 14:11 3. Blamelessness 16:15 4. Measure what matters 19:22 A word on Ops 20:16 5. Failure modes 21:59 6. No heroes 25:58 7. Automation 27:55 Change is constant 28:05 8. Change is No. 1 reason for outages 30:42 9. Outages are inevitable 34:30 10. No haunted graveyards 36:53 What did we learn? 38:10 Outro Download slides and read the full abstract here: https://gotoams.nl/2023/sessions/2477 RECOMMENDED BOOKS Murphy, Beyer, Jones & Petoff • Site Reliability Engineering • https://amzn.to/2Vg6Mbr Beyer, Murphy, Rensin, Kawahara & Thorne • The Site Reliability Workbook • https://amzn.to/3N5sjvk Adkins, Beyer, Blankinship, Lewandowski, Oprea & Stubblefield• Building Secure and Reliable Systems • https://amzn.to/3GoZI08 Nora Jones & Casey Rosenthal • Chaos Engineering • https://amzn.to/3hUmuAH Russ Miles • Learning Chaos Engineering • https://amzn.to/3hCiUe8 https://twitter.com/GOTOcon https://www.linkedin.com/company/goto- https://www.instagram.com/goto_con https://www.facebook.com/GOTOConferences #SRE #SiteReliabilityEngineering #ChaosEngineering #AtGoogle #ProductionInfrastructure #ChristofLeng #OrganizationalPrinciples #OrganizationalCulture #Change #Simplicity Looking for a unique learning experience? Attend the next GOTO conference near you! Get your ticket at https://gotopia.tech Sign up for updates and specials at https://gotopia.tech/newsletter SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily. https://www.youtube.com/user/GotoConferences/?sub_confirmation=1