talk-data.com talk-data.com

Meetup talk 2025-11-27 at 18:05

Data pipeline troubleshooting: Root cause analysis with lineage metadata

Description

Remember when debugging streaming data pipelines felt like playing detective at a crime scene, where the evidence kept shifting? Well, grab your magnifying glass because we’re about to turn you into Sherlock Holmes of the streaming world. We’ll simulate a disruptive change in an order processing pipeline that captures database changes with Debezium, processes them through Apache Flink, and tracks lineage metadata with OpenLineage and Marquez.