This session explores how to bring unit testing to SQL pipelines using Airflow. I’ll walk through the development of a SQL testing library that allows isolated testing of SQL logic by injecting mock data into base tables. To support this, we built a type system for AWS Glue tables using Pydantic, enabling schema validation and mock data generation. Over time, this type system also powered production data quality checks via a custom Airflow operator. Learn how this approach improves reliability, accelerates development, and scales testing across data workflows.
talk-data.com
Topic
AWS Glue
etl
data_catalog
aws
1
tagged
Activity Trend
10
peak/qtr
2020-Q1
2026-Q1
Top Events
AWS re:Invent 2024
15
O'Reilly Data Engineering Books
6
Data + AI Summit 2025
3
Data Engineering Podcast
2
O'Reilly Data Science Books
2
Databricks DATA + AI Summit 2023
2
Leaders of Analytics
1
Data Expo NL 2025
1
dbt Coalesce 2025
1
PyData Amsterdam 2025
1
Airflow Summit 2023
1
Experiencing Data w/ Brian T. O’Neill (AI & data product management leadership—powered by UX design)
1
Filtering by:
Gurmeet Saran
×