Building a Real-Time Analytics Database • Tim Berglund • GOTO 2023

This presentation was recorded at GOTO Chicago 2023. #GOTOcon #GOTOchgo https://gotochgo.com Tim Berglund - VP DevRel at StarTree & Author of "Gradle Beyond the Basics" @tlberglund@StarTree ORIGINAL TALK TITLE Building a Real-Time Analytics Database: A 'Choose Your Own Adventure' Journey RESOURCES https://pinot.apache.org https://twitter.com/startreedata https://www.linkedin.com/company/startreedata https://dev.startree.ai https://stree.ai/slack Tim http://timberglund.com https://twitter.com/tlberglund https://www.linkedin.com/in/tlberglund ABSTRACT Have you ever stopped to think about how to build a database? The thing is, there isn't just one way, as we can see by the massive number of data infrastructure options we have to choose from. It's a nonstop series of tradeoffs, each motivated by the constraints the database wants to satisfy. An in-memory transactional database would be one thing. A general-purpose, single-server relational database would be another. A low-latency, horizontally scalable analytics database would be...the journey we're going to take. In this talk, we'll start by picking a data model, make decisions about serialization and storage, choose indexing strategies, pick a query language, and figure out how to scale, eventually ending up with something that looks remarkably like Apache Pinot, a real-time analytics database. Pinot was built on a journey like this, always optimized for ultra low-latency, user-facing analytics at scale. In the real world, Pinot is used by applications like LinkedIn and UberEats to expose the state of the system not just to internal decision-makers, but to the users of the system itself, including all of us people who consumers of analytical queries. By focusing on the internals of Pinot and the tradeoffs made along the way to build a database of its kind, we'll see how it enables a new class of applications that every user of a system into a decision maker. [...] Download slides and read the full abstract here: https://gotochgo.com/2023/sessions/2522 RECOMMENDED BOOKS Tim Berglund • Gradle Beyond the Basics • https://amzn.to/3fSjfMD Tim Berglund & Matthew McCullough • Building and Testing with Gradle • https://amzn.to/3VaBY6g Mark Needham • Building Real-Time Analytics Systems • https://amzn.to/41AOZJd Gwen Shapira, Todd Palino, Rajini Sivaram & Krit Petty • Kafka: The Definitive Guide • https://amzn.to/41AVlrO Adi Polak • Scaling Machine Learning with Spark • https://amzn.to/3N9vx1H https://twitter.com/GOTOcon https://www.linkedin.com/company/goto- https://www.facebook.com/GOTOConferences #ApachePinot #Analytics #RealTime #RealTimeAnalytics #TimBerglund #StarTree #StarTreeCloud #Cloud #ApachePinotTutorial #ApachePinotTraining #OLAP #OLTP #LowLatency #ApacheZooKeeper #ApacheHelix #Hadoop #ApacheSpark Looking for a unique learning experience? Attend the next GOTO conference near you! Get your ticket at https://gotopia.tech Sign up for updates and specials at https://gotopia.tech/newsletter SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily. https://www.youtube.com/user/GotoConferences/?sub_confirmation=1