talk-data.com talk-data.com

B

Speaker

Bruno Cadonna

1

talks

Senior Software Developer II Confluent

Bruno is a committer and PMC member for Apache Kafka and a Software Developer at Confluent. At Confluent he has worked on Kafka Streams, Java Kafka clients, and ksqlDB. Prior to Confluent he was a Software Developer at SAP where he worked on a distributed in-memory computing engine for Big Data. Bruno holds a Ph.D. in Computer Science from the Free University of Bozen-Bolzano, Italy, and he held a postdoc position at Humboldt-Universität zu Berlin. His academic research focused on data stream and event processing. Bruno loves well-written code and hates inline comments.

Bio from: IN PERSON! Apache Kafka® Meetup Berlin - January 2024

Filtering by: IN PERSON! Apache Kafka® Meetup Berlin - January 2024 ×

Filter by Event / Source

Talks & appearances

Showing 1 of 1 activities

Search activities →

Restoring local state in Kafka Streams applications is indispensable for recovering after a failure or for moving stream processors between Kafka Streams clients. However, restoration has a reputation for being operationally problematic, because a Streams client occupied with restoration of some stream processors blocks other stream processors that are ready from processing new records. When the state is large this can have a considerable impact on the overall throughput of the Streams application. Additionally, when failures interrupt restoration, restoration restarts from the beginning, thus negatively impacting throughput further.\n\nIn this talk, we will explain how Kafka Streams currently restores local state and processes records. We will show how we decouple processing from restoring by moving restoration to a dedicated thread and how throughput profits from this decoupling. We will present how we avoid restarting restoration from the beginning after a failure. Finally, we will talk about the concurrency and performance problems that we had to overcome and we will present benchmarks that show the effects of our improvements.