Logging Apache Spark How we made it easy - Simona Meriam - NDC Oslo 2021

Are you familiar with the following Scenario? You're running your Apache Spark app on EMR, and the log file gets pretty heavy. You try and open it through the AWS UI, or download it straight to your computer. You end up connecting to the server running your driver or any of your executors, relentlessly searching your logs while simultaneously looking at Ganglia and the Spark UI for additional logs and metrics. If you are, this talk is exactly for you. Let me tell you how made it all easy with just some bootstrap actions, some bash scripts, Beats and Elastic. Customizable per app logging, with less searching of big log files and more looking into useful Kibana dashboards. This architecture is not nice to have, it's essential. Check out more of our featured speakers and talks at https://ndcconferences.com/ https://ndcoslo.com/