Apache Airflow is the go-to platform for data orchestration, while dbt is widely recognized for analytical data transformations. Using astronomer-cosmos library, integrating dbt projects into Airflow becomes straightforward, allowing each dbt model to be transformed into individual tasks or task groups equipped with Airflow features like retries and callbacks. However, organizing dbt models into separate Airflow DAGs based on domain or user-defined filters presents challenges in maintaining dependencies across these distinct DAGs. Ensuring that downstream dbt tasks only execute after the corresponding upstream tasks in different DAGs have successfully completed is crucial for data consistency—yet this functionality is not supported by default. Join GetYourGuide as we explore our method for dynamically creating inter-DAG sensors in Airflow using Astronomer Cosmos for dbt. We will show how we maintained dbt model dependencies across multiple DAGs, making our pipeline modular, scalable, and robust.

dbt has become the de facto standard for transforming data in modern analytics stacks. But as projects grow, so does the question: where should dbt run in production, and how can we make it faster? In this talk, we’ll compare the performance trade-offs between running dbt natively and orchestrating it through Airflow using Cosmos, with a focus on workflow efficiency at scale. Using a 200-model dbt project as a case study, we’ll show how workflow execution time in Cosmos was reduced from 15 minutes to just 5 minutes. We’ll also discuss opportunities to push performance further—ranging from better DAG optimization to warehouse-aware scheduling strategies. Whether you’re a data engineer, analytics engineer, or platform owner, you’ll leave with practical strategies to optimize dbt execution and inspiration for what’s next in large-scale orchestration