talk-data.com talk-data.com

Airflow Summit session 2025-07-01

Scaling Airflow at OpenAI

Event: Airflow Summit 2025

Speakers

Howie Wang

Member of Technical Staff at OpenAI

Ping Zhang

Member of Technical Staff at OpenAI

Topics

Airflow Kubernetes LLM

Description

This talk shares how we scaled and hardened OpenAI’s Airflow deployment to orchestrate thousands of workflows on Kubernetes. We’ll cover key architecture choices, scaling strategies, and reliability improvements - along with practical lessons learned.