talk-data.com talk-data.com

Company

Teamsystem

Speakers

2

Activities

1

Speakers from Teamsystem

Talks & appearances

1 activities from Teamsystem speakers

Face To Face
Marco Santoni (Data Engineering Manager) , Andrea Romeo (Platform Engineer)

How to move data from thousands of SQL databases to data lake with no impact on OLTP? We'll explore the challenges we faced while migrated legacy batch data flows to event-based architecture. A key challenge for our data engineers was the multi-tenant architecture of our backend, meaning that we had to handle the same SQL schema on over 15k databases. We'll present the journey employing Debezium, Azure Event Hub, Delta Live tables and the extra tooling we had to put in place.