As Data Engineers, our jobs regularly include scheduling or scaling workflows. But have you ever asked yourself, can I scale my scheduling ? It turns out that you can! But doing so raises a number of issues that need to be addressed. In this talk we’ll be: Recapping Asset-aware scheduling in Apache Airflow Discussing diverse methods to upscale our scheduling Solving the issue of maintaining our Airflow Asset synchronized between instances Comparing our professional push based solution and the built-in solution from AIP-82 and the pros and cons of each method. I hope you will enjoy it!
talk-data.com
S
Speaker
Sébastien Crocquevieille
1
talks
Senior Data Engineer
Numberly
Filter by Event / Source
Talks & appearances
1 activities · Newest first