talk-data.com
Airflow Summit
session
2021-07-01
Autoscaling in Airflow - Lessons learned
Event:
Airflow Summit 2021
Description
Autoscaling in Airflow - what we learnt based on Cloud Composer case. We would like to present how we approach the autoscaling problem for Airflow running in Kubernetes in Cloud Composer: how we calculate our autoscaling metric, what problem we had for scaling down and how did we solve it. Also we share an ideas on what and how we could improve the current solution