Have you ever added a DAG file and had no clue what happened to it? You’re not alone! With default settings, Airflow can wait up to 5 minutes before processing new DAG files. In this talk, I’ll discuss the implementation of an event-based DAG parser that immediately processes changes in the DAGs folder. As a result, changes are reflected immediately in the Airflow UI. In this talk I will cover: A demonstration of the event-based DAG parser and the fast Airflow UI experience How the current DAG parser implementation and configuration works How an event-based DAG parser is implemented
talk-data.com
B
Speaker
Bas Harenslak
1
talks
Staff Architect
Astronomer
Staff Architect at Astronomer.
Bio from: Amsterdam Apache Airflow® Meetup at Booking.com!
Filtering by:
Airflow Summit 2023
×
Filter by Event / Source
Talks & appearances
Showing 1 of 4 activities