Abstract: You've been tasked with implementing a data streaming pipeline for propagating data changes from your operational Postgres database to a search index in OpenSearch. Data views in OS should be denormalized for fast querying, and of course there should be no noticeable impact on the production database. In this session we'll discuss how to build this data pipeline using two popular open-source projects: Debezium for log-based change data capture (CDC) and Apache Flink for stream processing. Join us for this talk and learn about: * Setting up change data streams with Debezium * Efficiently building nested data structures from 1:n joins * Deployment options: Kafka Connect vs. Flink CDC
talk-data.com
G
Speaker
Gunnar Morling
1
talks
Software Engineer and open-source enthusiast
Decodable
Gunnar Morling is a Software Engineer at Decodable, focusing on stream processing with Apache Flink. Formerly at Red Hat, he led the Debezium project for change data capture. He is a Java Champion and has founded open source projects such as JfrUnit, kcctl, and MapStruct. He blogs at morling.dev and speaks at conferences including QCon, JavaOne, and Devoxx.
Bio from: Data Council 2023
Filtering by:
Berlin Open Source Data Infrastructure Meetup - November 2023
×
Filter by Event / Source
Talks & appearances
Showing 1 of 6 activities