talk-data.com
Airflow Summit
session
2023-07-01
Data at Rest: Bringing granular quality into flowing pipelines
Event:
Airflow Summit 2023
Speakers
Topics
Description
You’ve got your pipelines flowing … how much do you know about the data inside? Most teams have some coverage with unit/contract/expectations tests, and you might have other quality checks. But it can be very ad-hoc and disorganized. You want to do more to beef up data quality and observability … does that mean you just need to write more tests and assertions? Come learn about the best way to see your data’s quality alongside DAGs in a familiar context. We’ll review 3 common tools to get a handle on quality in a cohesive way across all your DAGs: Great Expectations Monte Carlo Data Databand