This presentation highlights practical validation techniques to prevent misconfigurations and enhance reliability in Apache Airflow environments. We cover two key safeguards: validating that sensors are correctly tied to their upstream tasks, and checking that critical DAGs have PagerDuty alerting enabled. Both validations are automated and integrated into our CI/CD pipeline using GitHub Actions, ensuring continuous enforcement and early detection of potential issues before deployment. In addition, we’ve implemented a solution to track Service Level Objectives (SLOs) for our DAGs, enabling better insight into reliability and performance over time. These checks form a practical defense against operational blind spots, promoting workflow reliability and robust incident response. Join us as we uncover practical strategies to streamline workflow monitoring and enhance system resilience using Apache Airflow's robust capabilities.
talk-data.com
R
Speaker
Robert Bemmann
1
talks
Senior Data Engineer
GetYourGuide
Senior Data Engineer at GetYourGuide.
Bio from: Berlin Apache Airflow Meetup at GetYourGuide!
Filtering by:
Berlin Apache Airflow Meetup at GetYourGuide!
×
Filter by Event / Source
Talks & appearances
Showing 1 of 1 activities