Experimenting with Distributed Data Processing in Haskell • Utku Demir • YOW! 2019

This presentation was recorded at YOW! 2019. #GOTOcon #YOW https://yowcon.com Utku Demir - Software Engineer at Movio RESOURCES https://mastodon.nz/@utdemir https://twitter.com/utdemir https://www.linkedin.com/in/utdemir https://github.com/utdemir https://utdemir.com ABSTRACT #ApacheSpark is one of the most popular data processing frameworks in the world, and is widely used in the enterprise. Its popularity is due in no small part to its adoption of the functional paradigm: it demonstrates the advantages of purity, higher-order functions and laziness to simplify the processing of large datasets. #Haskell excels at all of those things; so it is only natural to think that Haskell would be a good fit for distributed data processing. #Tweag.io's Sparkle and #Soostone's Hadron are a few examples in the Haskell ecosystem. 'distributed-dataset'[1] is a framework written in Haskell designed to efficiently process large amount of data. With the StaticPointers extension of #GHC Haskell, we are able to distribute a computation across different machines; and using the technique described by Matei et al[2] that led to Apache Spark, we can express and execute large scale data transforms using a pretty DSL. In this talk, I am going to give a brief introduction to the library, and then move on to explaining the key implementation ideas and the advantages that Haskell offers to distributed data processing. [...] RECOMMENDED BOOKS Rebecca Skinner • Effective Haskell • https://amzn.to/3SxTpwY Vitaly Bragilevsky • Haskell in Depth • https://amzn.to/3EXpmbeSaša Jurić • Elixir in Action • https://amzn.to/2RZh5eN Dave Thomas • Programming Elixir ≥ 1.6: Functional • https://amzn.to/34Dw3O5 McCord, Tate & Valim • Programming Phoenix 1.4 • https://amzn.to/3zcUqj4 https://twitter.com/GOTOcon https://www.linkedin.com/company/goto- https://www.facebook.com/GOTOConferences #DistributedDataProcessing #DataProcessing #Programming #ProgrammingLanguage #SoftwareEngineering #FP #FunctionalProgramming #UtkuDemir #YOWcon Looking for a unique learning experience? Attend the next GOTO conference near you! Get your ticket at https://gotopia.tech Sign up for updates and specials at https://gotopia.tech/newsletter SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily. https://www.youtube.com/user/GotoConferences/?sub_confirmation=1