talk-data.com
Airflow Summit
session
2021-07-01
Operating contexts: patterns around defining how a DAG should behave in dev, staging, prod & beyond
Event:
Airflow Summit 2021
Description
As people define and publish a DAG, it can be really useful to make it clear how this DAG should behave under different “operating contexts”. Common operating contexts may match your different environments (dev / staging / prod) and/or match your operating needs (quick run, full backfill, test run, …). Over the years, patterns have emerged around workflow authors, teams and organizations, and little has been shared as to how to approach this. In this talk, we’ll talk about what an “operating context” is, why it’s useful, and describe common patterns and best practices around this topic.