Java, How Fast Can You Parse 1 Billion Rows of Weather Data? • Roy van Rijn • GOTO 2024

This presentation was recorded at GOTO Amsterdam 2024. #GOTOcon #GOTOams https://gotoams.nl Roy van Rijn - Experienced Developer & Architect, Robotics Enthusiast & Hobby Mathematician @royvanrijn ORIGINAL TALK TITLE How Fast Can You Parse a File with 1 Billion Rows of Weather Data Using Java? RESOURCES https://x.com/royvanrijn https://www.linkedin.com/in/royvanrijn https://github.com/royvanrijn https://royvanrijn.com Links https://adventofcode.com https://x.com/gunnarmorling https://www.morling.dev ABSTRACT Last January a challenge was posted online by Gunnar Morling: How fast can you parse a file with 1 billion rows of weather data using Java? Little did I know this deceivingly simple question would lead me down a path that taught me all about: parallelism, memory mapped files, SWAR techniques (SIMD as a register), bit twiddling, branchless code, mechanical sympathy, Graal native compilation and finally... I even turned to the dark side: using sun.misc.Unsafe. Join me in this deep dive where I'll explain all the code changes and tricks that took me from the reference implementation which processes the billion records in 4+ minutes, to processing everything in under 2 seconds. Who knew Java could be this fast? [...] TIMECODES 00:00 Intro 01:49 The challenge 06:07 Watch, learn, adopt, experiment 08:00 Mechanical sympathy 09:32 Temperature as integer 10:37 Memory mapped files 11:54 Getting unsafe 13:31 SWAR 17:22 Stringless 18:18 Branchless programming 20:35 Parse the temperature 30:14 Keeping track 36:22 Which JVM? 37:21 Graal (native-image) 39:38 Summary 40:50 Results 42:00 Outro Download slides and read the full abstract here: https://gotoams.nl/2024/sessions/3164 RECOMMENDED BOOKS Monica Beckwith • JVM Performance Engineering • https://amzn.to/3zuJ7Ig Scott Oaks • Java Performance • https://amzn.to/4eNhlH4 Trisha Gee, Kathy Sierra & Bert Bates • Head First Java • https://amzn.to/3k59BJ6 Trisha Gee & Kevlin Henney • 97 Things Every Java Programmer Should Know • https://amzn.to/3kiTwJJ https://twitter.com/GOTOcon https://www.linkedin.com/company/goto- https://www.instagram.com/goto_con https://www.facebook.com/GOTOConferences #Java #JVM #GraalVM #Parsing #Parallelism #MemoryMappedFiles #SWAR #BitTwiddling #BranchlessCode #MechanicalSympathy #GraalNative #JavaProgramming #AdventOfCode #1BillionRowChallenge #GunnarMorling #RoyvanRijn CHANNEL MEMBERSHIP BONUS Join this channel to get early access to videos & other perks: https://www.youtube.com/channel/UCs_tLP3AiwYKwdUHpltJPuA/join Looking for a unique learning experience? Attend the next GOTO conference near you! Get your ticket at https://gotopia.tech Sign up for updates and specials at https://gotopia.tech/newsletter SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily. https://www.youtube.com/user/GotoConferences/?sub_confirmation=1