In this talk we will look into the details of how Kleinanzeigen, a leader in classifieds business in Germany, built a data migration system using Apache Kafka and Debezium that migrated millions of users' data from a legacy to a new platform and allowed bi-directional data sync between them in real time. We will also discover how the system allowed user's data to be updated on both platforms (partially, under certain conditions) while keeping the entire system in sync. Finally, we will learn how the system leveraged a logical clock to implement a custom synchronization algorithm that helped avoid infinite update loops between the platforms.
talk-data.com
M
Speaker
MD Sayem Ahmed
2
talks
MTS 2, Software Engineer
Kleinanzeigen
MD Sayem Ahmed works at Kleinanzeigen as a software engineer specializing in distributed systems. He works on initiatives that often involve both organisational and technical challenges that span across multiple teams. A major part of his work involves creating systems that can scale to meet both operational and business needs while ensuring resilience and fault tolerance.
Bio from: Debezium, Apache Kafka®, and an Acyclic Synchronization Algorithm
Filter by Event / Source
Talks & appearances
2 activities · Newest first
In this talk we will look into the details of how Kleinanzeigen, a leader in classifieds business in Germany, built a data migration system using Apache Kafka and Debezium that migrated millions of users' data from a legacy to a new platform and allowed bi-directional data sync between them in real time. We will also discover how the system allowed user's data to be updated on both platforms (partially, under certain conditions) while keeping the entire system in sync. Finally, we will learn how the system leveraged a logical clock to implement a custom synchronization algorithm that helped avoid infinite update loops between the platforms.