talk-data.com talk-data.com

Airflow Summit session 2025-07-01

Scaling and Unifying Multiple Airflow Instances with Orchestration Frederator

Topics

Description

In large organizations, multiple Apache Airflow instances often arise organically—driven by team-specific needs, distinct use cases, or tiered workloads. This fragmentation introduces complexity, operational overhead, and higher infrastructure costs. To address these challenges, we developed the “Orchestration Frederator,” a solution designed to unify and horizontally scale multiple Airflow deployments seamlessly. This session will detail our journey in implementing Orchestration Frederator, highlighting how we achieved: Horizontal Scalability: Seamlessly scaling Airflow across multiple instances without operational overhead. End-to-End Data Lineage: Constructing comprehensive data lineage across disparate Airflow deployments to simplify monitoring and debugging. Multi-Region Support: Introducing multi-region capabilities, enhancing reliability and disaster recovery. Unified Ecosystem: Consolidating previously fragmented Airflow environments into a cohesive orchestration platform. Join us to explore practical strategies, technical challenges, lessons learned, and best practices for enhancing scalability, reliability, and maintainability in large-scale Airflow deployments.