Topic

Cloud Composer

Google Cloud Composer

workflow_orchestration airflow google_cloud

Activities

1

tagged

Activity Trend

7 peak/qtr

2020-Q1 2026-Q1

Top Events

Airflow Summit 2025 7 Airflow Summit 2020 3 Airflow Summit 2021 3 O'Reilly Data Engineering Books 2 Airflow Summit 2023 2 Airflow Summit 2022 1

Top Speakers

Rafal Biegacz 5 Michał Modras (Google) 3 Piotr Wieczorek 3 Leah Cole 3 Bartosz Jankiewicz 2 Adi Wijaya 2 Filip Knapik 2 Przemek Więch 2 Priyanka Samanta 1 Marek Gawiński (Allegro) 1 Anita Fronczak 1 Gangfeng Huang 1

Activities

Showing filtered results

All Video Podcast Book

Filtering by: Anita Fronczak ×

Autoscaling in Airflow - Lessons learned

2021-07-01 · Airflow Summit 2021

session

by Anita Fronczak

Airflow Cloud Computing Kubernetes

Autoscaling in Airflow - what we learnt based on Cloud Composer case. We would like to present how we approach the autoscaling problem for Airflow running in Kubernetes in Cloud Composer: how we calculate our autoscaling metric, what problem we had for scaling down and how did we solve it. Also we share an ideas on what and how we could improve the current solution