The Evolution of Hadoop at Spotify Through Failures & Pain • Josh Baer & Rafal Wojdyla • GOTO 2015

This presentation was recorded at GOTO Copenhagen 2015 http://gotocph.com Josh Baer - Hadoop Product Owner at Spotify Rafal Wojdyla - Engineer at Spotify ABSTRACT The quickest way to learn and evolve infrastructure is by encountering obstacles and being forced to overcome limitations that keep you inches away from project goals. At Spotify, we've encountered many of these obstacles and frustrations as we grew our Hadoop cluster from a few machines in an office closet aggregating [...] Download slides and read the full abstract here: http://gotocon.com/cph-2015/presentation/The%20Evolution%20of%20Hadoop%20at%20Spotify%20-%20Through%20Failures%20and%20Pain TIMECODES 0:00 Introduction 1:22 Overview 2:40 What is Spotify? 4:09 Powered by Data 7:23 Moving Data to Hadoop 9:27 LogArchiver 13:07 Workflow Management Fail! 17:37 Hadoop Availability 18:33 How did we do? 20:32 What happened in the last quarter? 24:47 Lessons Learned 28:53 Going from Python to JVM 30:01 Moving from Python to Crunch 31:07 Crunch vs Hadoo Streaming Benchmark 31:56 Let's Review 33:00 Growth of Hadoop vs. Spotify Users 33:28 Explosive Growth 36:39 Inviso 37:18 Hadoop Report Card 38:45 Apache Spark with Zeppelin 40:10 Takeaways https://twitter.com/gotocph https://www.facebook.com/GOTOConference http://gotocon.com #Spotify #Hadoop #EvolutionOfHadoop #ProgrammingAtSpotify #Programming Looking for a unique learning experience? Attend the next GOTO conference near you! Get your ticket at https://gotopia.tech Sign up for updates and specials at https://gotopia.tech/newsletter