Using serverless functions for real-time observability - Liz Fong-Jones - CPH DevFest 2024

This talk was recorded at Copenhagen Developers Festival in Copenhagen, Denmark. #cphdevfest #ndcconferences #developer #softwaredeveloper Attend the next NDC conference near you: https://ndcconferences.com https://cphdevfest.com/ Subscribe to our YouTube channel and learn every day: / @NDC Follow our Social Media! https://www.facebook.com/ndcconferences https://twitter.com/NDC_Conferences https://www.instagram.com/ndc_conferences/ #bigdata #microservices #devops #serverless In this talk, you will learn how a sub-second latency query engine inspired by Facebook's Scuba was extended from running in RAM only, to querying the most recent data that could still fit on local SSD, to querying months of data at a time using cloud storage and serverless functions. We'll describe the pitfalls of managing lambdas at scale, including impatience, maximum concurrency, runtime and architecture configuration/experimentation, and the price/performance of renting tens of thousands of parallel workers across the x86 and arm ecosystems.