talk-data.com talk-data.com

A

Speaker

Alvin Zhang

1

talks

Software Engineer at Doordash

Filter by Event / Source

Talks & appearances

1 activities · Newest first

Search activities →

In large organizations, multiple Apache Airflow instances often arise organically—driven by team-specific needs, distinct use cases, or tiered workloads. This fragmentation introduces complexity, operational overhead, and higher infrastructure costs. To address these challenges, we developed the “Orchestration Frederator,” a solution designed to unify and horizontally scale multiple Airflow deployments seamlessly. This session will detail our journey in implementing Orchestration Frederator, highlighting how we achieved: Horizontal Scalability: Seamlessly scaling Airflow across multiple instances without operational overhead. End-to-End Data Lineage: Constructing comprehensive data lineage across disparate Airflow deployments to simplify monitoring and debugging. Multi-Region Support: Introducing multi-region capabilities, enhancing reliability and disaster recovery. Unified Ecosystem: Consolidating previously fragmented Airflow environments into a cohesive orchestration platform. Join us to explore practical strategies, technical challenges, lessons learned, and best practices for enhancing scalability, reliability, and maintainability in large-scale Airflow deployments.