Running Airflow at scale for thousands of workflows across multiple teams introduces challenges around standardization, governance, and isolation. At Booking, we've built a multi-tenant Airflow platform that serves over 4,000 workflows using a custom DSL defined in workflow.yaml files. In this talk, I'll show how we use automated DAG generation to bring structure to complexity, how we achieved horizontal scalability by decoupling orchestration from execution, and how reusable step templates help us enforce governance--without sacrificing workflow isolation. You'll leave with a blueprint for taming Airflow at scale.
talk-data.com
O
Speaker
Omid Karami
1
talks
Software Engineer
Booking.com
Software Engineer at Booking.com.
Bio from: Amsterdam Apache Airflow® Meetup at Booking.com!
Filter by Event / Source
Talks & appearances
1 activities · Newest first