talk-data.com

Summary Data contracts are both an enforcement mechanism for data quality, and a promise to downstream consumers. In this episode Tom Baeyens returns to discuss the purpose and scope of data contracts, emphasizing their importance in achieving reliable analytical data and preventing issues before they arise. He explains how data contracts can be used to enforce guarantees and requirements, and how they fit into the broader context of data observability and quality monitoring. The discussion also covers the challenges and benefits of implementing data contracts, the organizational impact, and the potential for standardization in the field.

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data managementData lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end-to-end data lakehouse platform built on Trino, the query engine Apache Iceberg was designed for, with complete support for all table formats including Apache Iceberg, Hive, and Delta Lake. Trusted by teams of all sizes, including Comcast and Doordash. Want to see Starburst in action? Go to dataengineeringpodcast.com/starburst and get $500 in credits to try Starburst Galaxy today, the easiest and fastest way to get started using Trino.At Outshift, the incubation engine from Cisco, they are driving innovation in AI, cloud, and quantum technologies with the powerful combination of enterprise strength and startup agility. Their latest innovation for the AI ecosystem is Motific, addressing a critical gap in going from prototype to production with generative AI. Motific is your vendor and model-agnostic platform for building safe, trustworthy, and cost-effective generative AI solutions in days instead of months. Motific provides easy integration with your organizational data, combined with advanced, customizable policy controls and observability to help ensure compliance throughout the entire process. Move beyond the constraints of traditional AI implementation and ensure your projects are launched quickly and with a firm foundation of trust and efficiency. Go to motific.ai today to learn more!Your host is Tobias Macey and today I'm interviewing Tom Baeyens about using data contracts to build a clearer API for your dataInterview IntroductionHow did you get involved in the area of data management?Can you describe the scope and purpose of data contracts in the context of this conversation?In what way(s) do they differ from data quality/data observability?Data contracts are also known as the API for data, can you elaborate on this?What are the types of guarantees and requirements that you can enforce with these data contracts?What are some examples of constraints or guarantees that cannot be represented in these contracts?Are data contracts related to the shift-left?Data contracts are also known as the API for data, can you elaborate on this?The obvious application of data contracts are in the context of pipeline execution flows to prevent failing checks from propagating further in the data flow. What are some of the other ways that these contracts can be integrated into an organization's data ecosystem?How did you approach the design of the syntax and implementation for Soda's data contracts?Guarantees and constraints around data in different contexts have been implemented in numerous tools and systems. What are the areas of overlap in e.g. dbt, great expectations?Are there any emerging standards or design patterns around data contracts/guarantees that will help encourage portability and integration across tooling/platform contexts?What are the most interesting, innovative, or unexpected ways that you have seen data contracts used?What are the most interesting, unexpected, or challenging lessons that you have learned while working on data contracts at Soda?When are data contracts the wrong choice?What do you have planned for the future of data contracts?Contact Info LinkedInParting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?Closing Announcements Thank you for listening! Don't forget to check out our other shows. Podcast.init covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.Links SodaPodcast EpisodeJBossData ContractAirflowUnit TestingIntegration TestingOpenAPIGraphQLCircuit Breaker PatternSodaCLSoda Data ContractsData MeshGreat Expectationsdbt Unit TestsOpen Data ContractsODCS == Open Data Contract StandardODPS == Open Data Product SpecificationThe intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Session:

For decades I've been designing, architecting, and implementing analytical solutions using MSFT data technologies, and I'm still not bored.

I consider the current times being the most exciting ones. New products/companies promising to solve analytical challenges easily are entering the stage almost every day. But what I consider even more important, not to say challenging is the advent of new concepts. Whereas I can ignore a new product easily to avoid re-training hundreds of users and other aspects like becoming a Junior again, new concepts are a different beast.

I consider concepts the building blocks that fuel any business or technology-focused initiative. The data warehouse concept is about creating a single point of truth (next to some other things). The concept of a lake house is about - what?

Ignoring a concept can become the root cause of demise, or as I put it sometimes: death by arrogance.

In this session, I will provide an overview of current concepts like data product, data contract, data lake house, and many other data ... concepts, why I consider some of them more important than others, compare concepts with similar approaches from the past, and of course: I will provide ideas how these concepts can be implemented using Microsoft Fabric and why they should - from a business perspective.

Speaker:

Tom Martens

Thomas "Tom" Martens has been awarded as an MSFT Data Platform MVP and works as Solution Architect at Munich Re (www.munichre.com). For 20+ years, Tom delivers Business Intelligence, Data Warehousing, and Analytics solutions. His current interest is in data visualization and applying analytical methods to small and large amounts of data, next to providing the Power BI Platform to users for tackling analytical challenges. Tom is a regular speaker at international conferences and user meetings. Tom is the co-author of the book "Pro DAX with Power BI."

Session: For decades I've been designing, architecting, and implementing analytical solutions using MSFT data technologies, and I'm still not bored.

I consider the current times being the most exciting ones. New products/companies promising to solve analytical challenges easily are entering the stage almost every day. But what I consider even more important, not to say challenging is the advent of new concepts. Whereas I can ignore a new product easily to avoid re-training hundreds of users and other aspects like becoming a Junior again, new concepts are a different beast. I consider concepts the building blocks that fuel any business or technology-focused initiative. The data warehouse concept is about creating a single point of truth (next to some other things). The concept of a lake house is about - what? Ignoring a concept can become the root cause of demise, or as I put it sometimes: death by arrogance.

In this session, I will provide an overview of current concepts like data product, data contract, data lake house, and many other data ... concepts, why I consider some of them more important than others, compare concepts with similar approaches from the past, and of course: I will provide ideas how these concepts can be implemented using Microsoft Fabric and why they should - from a business perspective.

Speaker: Tom Martens Thomas "Tom" Martens has been awarded as an MSFT Data Platform MVP and works as Solution Architect at Munich Re (www.munichre.com). For 20+ years, Tom delivers Business Intelligence, Data Warehousing, and Analytics solutions. His current interest is in data visualization and applying analytical methods to small and large amounts of data, next to providing the Power BI Platform to users for tackling analytical challenges. Tom is a regular speaker at international conferences and user meetings. Tom is the co-author of the book "Pro DAX with Power BI."

Delta Lake has quickly grown in usage across data lakes everywhere due to the growing use cases that require DML capabilities that Delta Lake brings. Outside of support for ACID transactions, users want the ability to interactively query the data in their data lake. This is where a query engine like Trino (formerly PrestoSQL) comes in. Starburst provides an enterprise version of the popular Trino MPP SQL query engine and has recently open sourced their Delta Lake connector.

In this talk, Tom and Claudius will talk about the connector, its features, and how their users are taking advantage of expanding the functionality of their data lakes with improved performance and the ability to handle colliding modifications. Get started with this feature-rich and open stack without the need of a multi-million dollar budget.

Connect with us: Website: https://databricks.com Facebook: https://www.facebook.com/databricksinc Twitter: https://twitter.com/databricks LinkedIn: https://www.linkedin.com/company/data... Instagram: https://www.instagram.com/databricksinc/