This talk shares how we scaled and hardened OpenAI’s Airflow deployment to orchestrate thousands of workflows on Kubernetes. We’ll cover key architecture choices, scaling strategies, and reliability improvements - along with practical lessons learned.
talk-data.com
Speaker
Ping Zhang
3
talks
Filter by Event / Source
Talks & appearances
3 activities · Newest first
Go over the yesterday, today and tomorrow for Airflow in Airbnb. Share our learnings and vision in Airflow core and around Airflow in its eco system. Starting with the history of Airflow in Airbnb, briefly describe how Airflow is used and the high level overview of Airflow in Airbnb. Then going into out current setup of Airflow, short term plans, learnings and best practises of Airflow. And finally talk about our the roadmap and vision of Airflow in Airbnb. Aside with Airflow core, we would love to also talk about what we’ve done inside the Airflow ecosystem, including frameworks built on top of Airflow, workflow development tools, etc. The recording for this session is not available.
With 34 new contributors, this best-selling handbook provides comprehensive coverage of the main topics in pure and applied graph theory. This second edition incorporates 14 new sections. Each chapter includes lists of essential definitions and facts, accompanied by examples, tables, remarks, and, in some cases, conjectures and open problems.