talk-data.com

Join us for a focused, 45-minute online meetup which we’ll walk through our architecture, the key design decisions, and the lessons learned in building for scale. The result: a fully managed, easy-to-operate system benchmarked at 100 GB/s throughput, delivering cost-effective, near-real-time ingestion with exceptional query performance.

From Kafka to Iceberg: High-Throughput Ingestion at Starburst Real-time analytics is critical for modern businesses — but bridging the gap between fast-moving Kafka streams and query-ready lakehouse tables remains a complex challenge. At Starburst, we encountered this firsthand while ingesting our internal telemetry data into Iceberg tables. Existing solutions fell short, plagued by issues such as lack of exactly-once guarantees, limited scalability, head-of-line blocking, small file proliferation, and high operational overhead and cost.

This prompted us to rethink streaming ingestion from the ground up. Our goal was a fully managed, highly available system that makes data actionable within minutes, optimized for both performance and usability. We designed and built a custom Kafka-to-Iceberg ingestion service that directly writes data in Iceberg format with strong guarantees, minimal latency, and continuous data maintenance for optimal query performance. Along the way, we developed novel techniques — such as Iceberg-aware commit coordination and adaptive Kafka consumer assignment — to overcome typical ingestion bottlenecks and deliver best-in-class price/performance.

Speaker: Lakshmikant (Pachu) Shrinivas, Staff Software Engineer at Starburst, hosted by Lester Martin, Developer Advocate at Starburst.

Alright Bristol, we’re finally here.

We’re buzzing to launch the Databricks Bristol Meetup and get the Bristol Databricks community together. Huge thanks to iO Associates for being our host and cheerleader in Bristol, and to Advancing Analytics and Databricks for backing the group and helping us build something that’s genuinely useful for practitioners.

Expect a relaxed evening of good people, practical Databricks chat, and proper networking (the kind where you actually meet people, not just collect LinkedIn connections you’ll never speak to again). Whether you’re deep in Spark, living your best Lakehouse life, or just Databricks-curious, you’re very welcome.

What to expect

• A friendly, community-first meetup (no sales, no nonsense)
• A quick Databricks “what’s new” segment to kick things off
• A keynote from Databricks
• Drinks, pizza, and plenty of time to chat with other data folks

Agenda 5.30pm – Arrival + networking drinks 6.00pm – Introduction from Simon Whiteley, followed by What’s new in Databricks 6.30pm – Keynote from Holly Smith (Databricks Staff Developer Advocate) 7.00pm – Drinks + pizza + networking

Venue Hosted at iO Associates, Bristol. (St Bartholomew's House, Bristol BS1 2NH) Spaces will be limited, so please RSVP. If you can’t make it, do update your RSVP so someone else can grab your spot.

Who should come?

• Data engineers, analysts, scientists, platform folks, architects
• People building on Databricks (or evaluating it)
• Anyone into modern data + AI and learning from others doing it for real

In this 2-hour hands-on workshop, you'll build an end-to-end streaming analytics pipeline that captures live cryptocurrency prices, processes them in real-time, and uses AI to forecast the future. Ingest live crypto data into Apache Kafka using Kafka Connect; tame that chaos with Apache Flink's stream processing; freeze streams into queryable Apache Iceberg tables using Tableflow; and forecast price trends with Flink AI.