talk-data.com talk-data.com

Meetup talk 2025-07-02 at 18:35

Migrating Millions of Users with Debezium, Apache Kafka, and an Acyclic Synchronization Algorithm

Description

In this talk we will look into the details of how Kleinanzeigen, a leader in classifieds business in Germany, built a data migration system using Apache Kafka and Debezium that migrated millions of users' data from a legacy to a new platform and allowed bi-directional data sync between them in real time. We will also discover how the system allowed user's data to be updated on both platforms (partially, under certain conditions) while keeping the entire system in sync. Finally, we will learn how the system leveraged a logical clock to implement a custom synchronization algorithm that helped avoid infinite update loops between the platforms.