This talk is all about how we at Jagex manage DAGs at scale. Focusing of following challenges faced and how we resolved them; Keeping track of airflow state Keeping track of each DAG state DAGs as git submodules Updating airflow with new dags How to seamlessly automate airflow deployment How to avoid package dependency conflicts
talk-data.com
A
Speaker
Anum Sheraz
1
talks
Software (Data) Engineer
Filtering by:
Airflow Summit 2022
×
Filter by Event / Source
Talks & appearances
Showing 1 of 1 activities