This session explores how to bring unit testing to SQL pipelines using Airflow. I’ll walk through the development of a SQL testing library that allows isolated testing of SQL logic by injecting mock data into base tables. To support this, we built a type system for AWS Glue tables using Pydantic, enabling schema validation and mock data generation. Over time, this type system also powered production data quality checks via a custom Airflow operator. Learn how this approach improves reliability, accelerates development, and scales testing across data workflows.
talk-data.com
Topic
Pydantic
schemas
python
data_modeling
data_validation
1
tagged
Activity Trend
5
peak/qtr
2020-Q1
2026-Q1
Top Events
PyConDE & PyData Berlin 2023
3
Airflow Summit 2025
2
DataTopics: All Things Data, AI & Tech
2
PyData Rhein-Main I Security Risks in AI & Structured Automation with Agentic AI
1
Python Data Science Day
1
May Members Talk Evening
1
PyData Hamburg July 24th Meetup
1
PyDataMCR August
1
Node.js Meetup #43 (Pizza 🍕 and Drinks 🍺)
1
O'Reilly Data Engineering Books
1
Filtering by:
Gurmeet Saran
×