Data contracts have been much discussed in the community of late, with a lot of curiosity around how to approach this concept in practice. We believe data contracts need a harmonizing layer to manage data quality in a uniform manner across a fragmented stack. We are calling this harmonizing layer the Control Plane for Data - powered by the common thread across these systems: metadata. For teams already orchestrating pipelines with Airflow, data contacts can be an effective way to process data that meets preset quality standards. With a control plane as a connecting layer, producers can build data contracts that consumers can rely on, ensuring DAGs only run when a contract is valid. Producers can govern how workflows should behave, and consumers receive the tooling they need to only opt into high quality data. Learn how to use data contracts and DataHub to make your Airflow pipelines more reliable - as well as other use cases that can help build a simpler, more flexible data stack.
talk-data.com
Speaker
Shirshanka Das
1
talks
CTO
Acryl Data
My career in software engineering has led me to some incredible companies, including PayPal, Yahoo, and LinkedIn. I spent 10 years navigating the complex data challenges that came up as LinkedIn grew to become one of the most influential platforms in the world. I developed DataHub, an open-source metadata platform that helps address issues that arise when managing and leveraging data at scale. In 2021, I co-founded Acryl Data to help enterprises adopt DataHub and scale the open-source community.
Bio from: dbt Coalesce 2023
Filtering by:
Airflow Summit 2023
×
Filter by Event / Source
Talks & appearances
Showing 1 of 5 activities