talk-data.com talk-data.com

M

Speaker

Madison Swain-Bowden

1

talks

Staff Data Engineer Automattic
Filtering by: Airflow Summit 2021 ×

Filter by Event / Source

Talks & appearances

Showing 1 of 3 activities

Search activities →

We will describe how we were able to build a system in Airflow for MySQL to Redshift ETL pipelines defined in pure Python using dataclasses. These dataclasses are then used to dynamically generate DAGs depending on pipeline type. This setup allows us to implement robust testing, validation, alerts, and documentation for our pipelines. We will also describe the performance improvements we achieved by upgrading to Airflow 2.0.