Uncluster Your Data Science Using Vaex • Maarten Breddels & Jovan Veljanoski • GOTO 2021

This presentation was recorded at GOTO Copenhagen 2021. #GOTOcon #GOTOcph http://gotocph.com Maarten Breddels - Independent developer and consultant, co-founder of vaex.io Jovan Veljanoski - Machine learning specialist at Cloud Technology Solutions and co-founder of vaex.io ABSTRACT Would you like to build an snappy dashboard visualising hundreds of millions of data points, or interactively explore hundreds of Gigabytes of data, all of that using a single machine? Meet Vaex - an out of core DataFrame library in Python that can do all the typical data manipulations, filtering, and aggregations on a billion rows in real time & on a single computer. This approach empowers your team and allows them to focus much more on the business problem, as it removes the large DevOps overhead of configuring and maintaining a cluster. Vaex fully supports Apache Arrow, which both facilitates the interoperability with other systems and enables storage and manipulation of more complex data structures like lists [...] TIMECODES 00:00 Intro 00:50 Motivation 05:20 Vaex 06:14 Concepts: Memory mapping 07:56 Concepts: Column based storage 09:37 Concepts: No memory copies 10:50 Concepts: Compute & expression system 12:23 Vaex.io: From academic project to consultancy 13:30 Demo 32:32 In production 34:23 In the wild 35:00 In production: Dash example 36:42 Vaex.io: Consultancy 37:13 Summary 37:58 Outro Download slides and read the full abstract here: https://gotocph.com/2021/sessions/1922/uncluster-your-data-science-using-vaex RECOMMENDED BOOKS Phil Winder • Reinforcement Learning • https://amzn.to/3t1S1VZ Kelleher & Tierney • Data Science (The MIT Press Essential Knowledge series) • https://amzn.to/3AQmIRg Lakshmanan, Robinson & Munn • Machine Learning Design Patterns • https://amzn.to/2ZD7t0x Lakshmanan, Görner & Gillard • Practical Machine Learning for Computer Vision • https://amzn.to/3m9HNjP Aurélien Géron • Hands-On Machine Learning with Scikit-Learn, Keras & TensorFlow • https://amzn.to/2XZaQy8 https://twitter.com/GOTOcon https://www.linkedin.com/company/goto- https://www.facebook.com/GOTOConferences #Vaex #ApacheArrow #DataScience #AI #ML #ArtificialIntelligence #MachineLearning #DataFrame #Programming #VaexIO #Astronomy Looking for a unique learning experience? Attend the next GOTO conference near you! Get your ticket at https://gotopia.tech SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily. https://www.youtube.com/user/GotoConferences/?sub_confirmation=1