Talks - Amitosh Swain: Testing Data Pipelines

Hello 👋! I'll review few great ways to test data pipelines in this talk. This approach's primary goal is to ensure data flows smoothly through the pipelines by quickly identifying and fixing any problems. While the talk uses Airflow as the base, the techniques presented are toolkit-agnostic. When it comes to testing pipelines, the process is similar to testing software applications. It includes running unit tests for each pipeline component, integration tests for the entire pipeline, and end-to-end tests to ensure accurate data output. However, I'll also discuss unique methods like data snapshot testing and online and offline data quality checks. Slides: https://pycon-assets.s3.amazonaws.com/2024/media/presentation_slides/46/2024-05-17T03%3A49%3A26.293574/_Testing_Data_Pipelines1.pdf