talk-data.com
Big Data LDN/Paris
Face To Face
2025-09-24 at 15:20
Offload Ks of SQL tenants with Debezium and DLTs
Event:
Big Data LDN 2025
Speakers
Description
How to move data from thousands of SQL databases to data lake with no impact on OLTP? We'll explore the challenges we faced while migrated legacy batch data flows to event-based architecture. A key challenge for our data engineers was the multi-tenant architecture of our backend, meaning that we had to handle the same SQL schema on over 15k databases. We'll present the journey employing Debezium, Azure Event Hub, Delta Live tables and the extra tooling we had to put in place.