talk-data.com talk-data.com

Filter by Source

Select conferences and events

Activities & events

Title & Speakers Event

Learn to build, containerize, deploy, and extend interactive 3D digital twin applications using NVIDIA Omniverse Kit App Streaming. This hands-on course walks you through creating a streaming-ready Kit app, deploying it for web clients, and enabling real-time, bi-directional data interaction—empowering scalable, remote visualization and collaboration across digital twin workflows.

Please RSVP and arrive at least 5 minutes before the start time, at which point remaining spaces are open to standby attendees.

Azure Microsoft Data Streaming
Microsoft Ignite 2025

Unlock the Power of Data with CrateDB & AWS**: A Hands-On Workshop.**

Join us for a dynamic, hands-on morning that will transform the way you think about data analytics and AI. Designed for data-driven professionals across sectors, this event delivers hands-on sessions, real-world AI applications, and the latest in scalable analytics. Secure your spot and discover how CrateDB and AWS are shaping the future of data.

We’ll dive into the world of multi-model databases and explore how time-series data can be enriched with geospatial and vector data. From there, we will build an AI agent that enables interpreting data using natural language with the help of an LLM.

Hands-On Workshops with CrateDB’s Solutions Engineers

Get ready to work directly with CrateDB’s solutions engineers as they guide you through two engaging hands-on sessions, using Jupyter notebooks. Everything runs smoothly in the cloud, so you can focus on learning and experimenting.

What to Bring

Bring your laptop and your curiosity! No software installation is required as all labs run in the cloud. Basic SQL and Python knowledge is recommended but not essential.

Register Today – Don’t Miss Out!

Fill in the registration form to save your seat for this educational morning. After submission, you will receive an email within 48/72 hours to confirm your registration. This session is also a great opportunity to participate in deep technical discussions and exchange ideas with your peers. Don’t miss this chance to elevate your technical skills and discover how modern data technologies can fuel innovation!

About CrateDB

CrateDB is a distributed database that empowers organizations with a Unified Data Layer to process, analyze, and act on high-velocity data at scale. Designed for speed, scalability, and flexibility, CrateDB seamlessly integrates real-time analytics, full-text and vector search, and AI-driven capabilities into a single, cohesive platform.

CrateDB Cloud on AWS offers a seamless and scalable SQL database solution for managing all types of data for advanced analytics and machine learning applications. Available on AWS, CrateDB Cloud enables the deployment of modern, scalable applications, ensuring compliance with strict security standards.

About AWS

Amazon Web Services (AWS) is a widely adopted cloud platform, offering +200 fully featured services from data centers globally. AWS empowers organizations of all sizes - from startups to enterprises, government agencies, and nonprofits - to innovate, scale, and transform their operations with secure, reliable, and cost-effective cloud infrastructure. AWS enables customers to build, deploy, and manage applications with unmatched flexibility and scalability, helping them accelerate growth and drive meaningful impact across industries.

Venue

Amazon Web Services Munich Office Oskar-von-Miller-Ring 20 80333 München Germany Google maps

CrateDB European City Tour Munich

As AI agents become more autonomous and capable, integration becomes a key challenge. In this session, we’ll introduce Agentic AI and how MCP (Model Context Protocol) — an open standard developed by Anthropic — is transforming the way agents connect with external tools. Think of MCP as the “USB-C for AI” that enables seamless, no-code integration between agents and thousands of applications.

We’ll walk through how Zapier’s support for MCP integration allows AI agents to trigger over 35,000 actions across 7,000+ apps — without needing any manual API integration. From sending emails to updating spreadsheets and triggering webhooks, agents can now perform complete workflows end-to-end through MCP-integrated automation.

You’ll see a live demo of how an AI agent connects to a Zapier-generated MCP server URL and executes automated tasks — no code required. We’ll also explore practical agent design strategies for real-world implementation, from orchestrating actions and assigning roles to handling retries and errors at scale.

What We Will Cover:

  • Understand what Agentic AI is and how MCP integration enables scalable, real-time integrations with external tools
  • Learn how Zapier’s MCP feature works and how it empowers agents to perform thousands of actions across major platforms
  • See a live demo of an AI agent triggering a multi-step workflow via a Zapier MCP integration endpoint — with no manual API calls
  • Explore design strategies for agent orchestration: retries, error handling, and modular role assignments
  • Discover how to build and deploy intelligent, automated workflows using MCP integration — and apply these patterns to your own projects
  • Interactive Element: Through live demos and audience Q&A, participants will gain hands-on insights into agent-based automation powered by MCP integration, and leave with a scalable framework they can begin applying immediately.
Agentic AI with MCP Integration

As AI agents become more autonomous and capable, integration becomes a key challenge. In this session, we’ll introduce Agentic AI and how MCP (Model Context Protocol) — an open standard developed by Anthropic — is transforming the way agents connect with external tools. Think of MCP as the “USB-C for AI” that enables seamless, no-code integration between agents and thousands of applications.

We’ll walk through how Zapier’s support for MCP integration allows AI agents to trigger over 35,000 actions across 7,000+ apps — without needing any manual API integration. From sending emails to updating spreadsheets and triggering webhooks, agents can now perform complete workflows end-to-end through MCP-integrated automation.

You’ll see a live demo of how an AI agent connects to a Zapier-generated MCP server URL and executes automated tasks — no code required. We’ll also explore practical agent design strategies for real-world implementation, from orchestrating actions and assigning roles to handling retries and errors at scale.

What We Will Cover:

  • Understand what Agentic AI is and how MCP integration enables scalable, real-time integrations with external tools
  • Learn how Zapier’s MCP feature works and how it empowers agents to perform thousands of actions across major platforms
  • See a live demo of an AI agent triggering a multi-step workflow via a Zapier MCP integration endpoint — with no manual API calls
  • Explore design strategies for agent orchestration: retries, error handling, and modular role assignments
  • Discover how to build and deploy intelligent, automated workflows using MCP integration — and apply these patterns to your own projects
  • Interactive Element: Through live demos and audience Q&A, participants will gain hands-on insights into agent-based automation powered by MCP integration, and leave with a scalable framework they can begin applying immediately.
Agentic AI with MCP Integration

As AI agents become more autonomous and capable, integration becomes a key challenge. In this session, we’ll introduce Agentic AI and how MCP (Model Context Protocol) — an open standard developed by Anthropic — is transforming the way agents connect with external tools. Think of MCP as the “USB-C for AI” that enables seamless, no-code integration between agents and thousands of applications.

We’ll walk through how Zapier’s support for MCP integration allows AI agents to trigger over 35,000 actions across 7,000+ apps — without needing any manual API integration. From sending emails to updating spreadsheets and triggering webhooks, agents can now perform complete workflows end-to-end through MCP-integrated automation.

You’ll see a live demo of how an AI agent connects to a Zapier-generated MCP server URL and executes automated tasks — no code required. We’ll also explore practical agent design strategies for real-world implementation, from orchestrating actions and assigning roles to handling retries and errors at scale.

What We Will Cover:

  • Understand what Agentic AI is and how MCP integration enables scalable, real-time integrations with external tools
  • Learn how Zapier’s MCP feature works and how it empowers agents to perform thousands of actions across major platforms
  • See a live demo of an AI agent triggering a multi-step workflow via a Zapier MCP integration endpoint — with no manual API calls
  • Explore design strategies for agent orchestration: retries, error handling, and modular role assignments
  • Discover how to build and deploy intelligent, automated workflows using MCP integration — and apply these patterns to your own projects
  • Interactive Element: Through live demos and audience Q&A, participants will gain hands-on insights into agent-based automation powered by MCP integration, and leave with a scalable framework they can begin applying immediately.
Agentic AI with MCP Integration
DevFest Berlin 2024 2024-11-23 · 08:00

DevFest Berlin is back! This year back to Humboldt University of Berlin, with more than 25 talks & workshops, you can expect a whole day of learning, socialising, and engaging with a vibrant Berlin Tech community!

🎫 Get you ticket here: pretix.eu/devfestberlin/2024/ 🖍 Call for Papers still open: pretalx.com/devfest-berlin-2024/cfp

Agenda

Day 1

9:00 AM: Registration & Coffee 🥐 ☕️

9:45 AM: 🎤 Welcoming

10:00 AM: 🎤 Katya Vinnichenko - Introduction to Google Principles of Responsible AI

This year's DevFest explores how AI can improve lives globally, from business to healthcare to education. At Google we acknowledge AI's potential, while also recognising the challenges it presents. Thus, we are committed to helping you build and use AI responsibly, ensuring fairness and ethical practices.

In my talk you will learn: the main principles of responsible AI at Google; the ethical implications of AI; best practices for developing AI systems and integrating AI into Google products and services; last but not least – how AI will change the role of the developer as we know it.

10:50 AM: 🎤 Oleksii Antypov - DMARC Demystified

Discover the essential framework behind DMARC and how it secures email communication across the internet. This session covers the historical evolution of email security, dives into the common challenges of implementing DMARC, and provides actionable best practices for protecting your domain. Ideal for developers, security professionals, and anyone interested in safe email practices.

In a world where phishing and email spoofing are constant threats, DMARC stands as a vital defense mechanism. “DMARC Demystified” takes you through a journey from the origins of email security to the modern challenges and solutions that DMARC offers. We'll explore how DMARC works with SPF and DKIM, why it’s essential for organizations of all sizes, and the practical steps to ensure smooth implementation.

Expect an interactive timeline tracing the milestones of email security, detailed breakdowns of real-world cases, and insights into optimizing DMARC. Walk away with a deeper understanding of email protection, armed with knowledge to strengthen your email systems and protect against threats.

11:40 AM: 🎤 Marcin Chudy - Demystifying App Architecture: The LeanCode Guide

At LeanCode we developed over 40 Flutter apps, spanning from huge enterprise apps to nimble startup ventures. Some were developed by a single Flutter dev, some came into light through collaborative efforts across multiple teams. Each of them was different. Each of them presented unique challenges and taught us invaluable lessons.

In this talk, we invite you to explore different approaches to architecting Flutter apps. Central to our narrative will be the concept of architectural drivers - key factors or priorities that steer our decisions about how the app is structured and designed. We'll show how we leverage our experience when approaching new projects. Drawing from our successes and failures, we'll present our current Flutter stack which enables us to craft robust, scalable, and maintainable applications. While there is no silver bullet for Flutter architecture, we can still have some sensible defaults.

Why do we use BLoC for state management? Why not Riverpod? Why do we love hook

12:30 PM: 🎤 Danny Preussler - Ten things you heard about testing that might be wrong

Testing became an essential part of Android development. Many conference talks have been given and even more best practices have been written.

But what if, as time evolved, some of the things we thought were true, changed?

Let’s start questioning some of these in this talk: Are flaky tests fixable? Are mocks even harmful? Is DI about testing? Did we understand testing in isolation properly? Is the test pyramid still valid? And in times of AI, should we generate tests?

Come and join my session to learn more!

1:10 PM: Lunch 🍔🥤

2:40 PM: 🎤 Andrey Sitnik - Privacy-first architecture: alternatives to GDPR popup and local-first

Why and how modern developers could increase the privacy of modern Web.

The popularity of clouds, the rise of huge monopolies across the internet, and the growth of shady data brokers recently have made the world a much more dangerous place for ordinary people—here is how we fix it.

In this talk, Andrey Sitnik, the creator of PostCSS and the privacy-first open-source RSS reader, will explain how we can stop this dangerous trend and make the web a private place again. — Beginners will find simple steps, which can be applied to any website — Advanced developers will get practical insights into new local-first architecture — Privacy experts could find useful unique privacy tricks from a global world perspective and beyond just U.S. privacy risks

3:30 PM: 🎤 Raphaël VO - Largest Contentful Paint - The unheard story

Largest Contentful Paint (LCP) is more than a speed metric — it's the unseen factor shaping user experiences and impacting SEO. While often overlooked, LCP reveals when a page’s core content is truly ready, affecting how users perceive load time and usability. This talk uncovers LCP’s role, why it matters more than we think, and simple strategies to boost LCP for better engagement and rankings. Discover the hidden story behind one of web performance’s most crucial, yet understated metrics.

Did you know the speed of a single webpage element could decide if users stay or leave? Largest Contentful Paint (LCP) is that hidden hero, quietly working to load the most important content quickly. This talk unveils LCP’s role in creating faster, more engaging web experiences and why it’s key to winning user loyalty. Dive into the “unheard story” of LCP and discover practical tips to make your site not only faster but unforgettable.

4:20 PM: 🎤 Ash Davies - Navigation in a Multiplatform World: Choosing the Right Framework for your App

Navigation in mobile, desktop, and web applications is such a fundamental part of how we structure our architecture. In order to both obtain functional clarity, and abstraction from platform level implementation.

For a long time, there have been options available specific to each platform, and even options part of the platform framework itself. Though it can be difficult to find the right option for platform-agnostic code, ensuring consistency. Some go one step further, providing an opinionated guide on how to architecture your application.

In this talk, I'll evaluate the options available, how they differ, and to what type of applications they are best suited. Including how to get started with them, and the best practice guidelines on how to get the most out of them, for your application.

5:10 PM: 🎤 Vadim Makeev - You don’t know MathML. Almost nobody does

Do you speak math? Me neither. Still, math formulas have always been around: from Wikipedia articles to JavaScript APIs and even CSS docs. It looks so alien that I never had a clue how to express it on the web. Apparently, there’s a markup language for that. HTML for content, SVG for vector graphics, and MathML for math! And it’s pretty cross-browser, too. Let’s dive into the basics and quirks of the language of the universe. Even if math is not your love language, you might learn something interesting about the web platform.

Day 2

9:00 AM: Registration & Coffee 🥐 ☕️

10:00 AM: 🎤 Alex Mir – Accessibility matters

The regulators are here and now businesses will care about the a11y. Let's make the a11y compliance not just a formal check. I believe that it is our job as industry experts to understand why it is important and get our products ready for all groups of people.

10:50 AM: 🎤 Marco Gomiero - From Android to Multiplatform and beyond

With Kotlin Multiplatform getting increasingly established, many Android libraries became multiplatform.

But how to make an existing Android library multiplatform?

In this talk, we will cover the common challenges faced while migrating Android libraries to Kotlin Multiplatform, like handling platform-specific dependencies, re-organizing the project structure without losing the contributor's history, testing on multiple platforms, and publishing the library.

11:20 AM: 🎤 Muhammad Salman Bediya - Crucial Performance Issue in Flutter Apps: Memory Leaks

Memory leaks can be hard to spot but have a big impact on the performance of Flutter apps, especially those running for long periods. In this talk, we’ll explore the most common reasons memory leaks happen in Flutter and Dart, focusing on how asynchronous programming and Streams can make them more challenging. You’ll learn practical tips to identify and fix these issues, helping your apps run smoother and more efficiently.

11:40 AM: 🎤 Andrii Raikov - Maximizing Scalability with Go and Redis: A Telemetry Processing Journey

At Delivery Hero, we process 10,000 requests per second using Go and Redis. Join us to learn how this powerful duo handles high-load telemetry data efficiently and cost-effectively, with scalability, resource optimization, and continuous innovation through customized data flows.

12:30 PM: 🎤 Tomek Porozynski - Can You Outsmart an AI? Adventures in Prompt Hacking

In this talk combined with hands-on elements, participants will engage in a series of live prompt hacking challenges, accessible directly through their mobile devices. The workshop begins with simple prompt injection techniques and progressively moves to more sophisticated manipulation strategies. After each successful hack, I'll analyze what made it work and transform these insights into practical defense mechanisms.

Attendees will learn: Common vulnerabilities in AI prompt design, Practical techniques for prompt injection attacks, Essential strategies for securing chatbot applications, Best practices for implementing defensive layers, Real-world examples of prompt security failures and successes

Perfect for developers working with AI models, security enthusiasts, or anyone interested in building safer AI applications. No specialized tools needed - just bring your phone and creativity! You'll leave with concrete techniques for both testing and securing your AI systems against prompt manipulation attacks.

1:10 PM: Lunch 🍔🥤

2:40 PM: 🎤 Cesar Martinez - Domain Driven Design Fundamentals for Frontend Developers

What can we learn from Domain Driven Design and how to start applying its teachings in your frontend codebase.

3:30 PM: 🎤 Vadym Pinchuk - Effortless optimization of Flutter apps: performance tips for developers

In this session, we’ll dive into effortless yet impactful ways to optimize your Flutter applications. Performance improvements don’t always require a full rewrite—sometimes, small adjustments can lead to big gains. We'll explore practical tips and tricks for enhancing app speed, responsiveness, and efficiency with minimal effort. From reducing widget rebuilds to handling large data efficiently and managing state effectively, this talk will provide developers with actionable insights to deliver a smoother user experience. Whether you’re a beginner or an experienced Flutter dev, you’ll walk away with easy-to-apply techniques to optimize your apps without breaking a sweat.

4:20 PM: 🎤 Ian Ballantyne - Generative AI on Mobile and Web with Google AI Edge

Generative AI is no longer limited to execution in the cloud. Small language models, such as Gemma 2B, are quickly becoming small and powerful enough for on-device AI, offering benefits like low latency, offline functionality, privacy, and cost-effectiveness. Google AI Edge, with MediaPipe and LiteRT (formerly Tensorflow Lite), enables the development and deployment of efficient on-device AI models. These frameworks handle the complexities of model execution and hardware acceleration, allowing developers to focus on creating innovative AI experiences.

Think generative AI is just about chatbots? Think again. This talk will go beyond basic conversations with language models and explore how on-device generative AI can be integrated into everyday apps ready to help with tasks, answer questions, and provide creative inspiration, all powered by the information located on-device. Imagine truly useful apps that are quick to respond and still work without an internet connection.

5:10 PM: 🎤 Bogdan Plieshka - Automated Testing Layers in a multidimensional Monorepo: Fast-tracking Quality for hundreds apps

In this talk, I’ll dive into the testing layers that make up our quality pipeline at Zattoo, including static analysis, unit, system, and end-to-end testing.

We’ll discuss the concept of quality gates, shift-left approach, and affected domain recognition, which helps us maintain reliability across a large, dynamic codebase, bringing total quality feedback for contributors to 3 minutes.

I’ll share practices for achieving scalable, fast testing in a high-complexity environment, offering insights for anyone working with large-scale applications or monorepos and looking to streamline QA processes.

Day 3

9:00 AM: Registration & Coffee 🥐 ☕️

10:00 AM: 🎤 Inès Mir & Doruk Deniz Kutukculer - Fellowship of Product. How your team setup affects your experience

Did you know there are 2 types of team formation in tech? These formations can change your experience in the team drastically and you better recognise them early to adjust your expectations from the job. And even more importantly, you need to show different qualities on job interviews to get this job in a particular team formation!

Deniz Doruk Kuetuekcueler, a head of engineering, and Inès Mir, a principal product designer, are trying to figure out how design and engineering can effectively work together in these setups.

10:50 AM: 🎤 Alireza Rahmaty - How we automate the App Release Monitoring at GetYourGuide

App release monitoring (ARM) represents a suite of innovative tools designed to monitor the health and stability of iOS and Android app releases. These tools provide real-time updates by sending notifications to Slack channels and logging the app's status throughout the release process. At GetYourGuide, we have developed an ARM to monitor the rollout of our Android and iOS apps from the moment they are submitted to the App Store & Google Play until they are fully released. We ship releases faster and with more confidence using ARM!

11:40 AM: 🎤 Aleksandr Gorbunov - Flutter for frontenders or There and Back Again

Every developer, regardless of specialization, may encounter the need to create a UI for a client application. The choice of technology may depend on the developer, or it may be pre-determined by the client, as happened in my case.

The peculiarity is that, coming from frontend development in JavaScript, I started building user interfaces in Flutter.

Today, there is a vast number of technologies that enable the development of cross-platform applications. These technologies are evolving rapidly, attracting large communities, and more frequently, companies are adopting them. For example, Flutter is a powerful framework that allows developers to create cross-platform applications.

With a high probability, every developer may encounter the need to use such development tools, and it’s great that frameworks like Flutter come with detailed documentation and extensive community support, making it relatively easy to start developing with them. Although, at first glance, everything might not seem smooth, and the desire to revert to familiar methods may arise.

12:05 PM: 🎤 Muhammad Salman Bediya - Crucial Performance Issue in Flutter Apps: Memory Leaks

Memory leaks can be hard to spot but have a big impact on the performance of Flutter apps, especially those running for long periods. In this talk, we’ll explore the most common reasons memory leaks happen in Flutter and Dart, focusing on how asynchronous programming and Streams can make them more challenging. You’ll learn practical tips to identify and fix these issues, helping your apps run smoother and more efficiently.

12:30 PM: 🎤 Ole Bulbuk - Native GUIs For All

Traditionally native GUIs are highly platform dependent and often specific for one programming language. In this talk we will explore a way to create GUI applications that supports virtually all platforms and any programming language. It is very effective and easy to use, too.

1:10 PM: Lunch 🍔🥤

2:40 PM: 🎤 Nicole Terc - Tap it! Shake it! Fling it! Sheep it! - The Gesture Animations Dance!

Let's have fun with animations, gestures and sensors!

Using Compose Multiplatform, we'll go over how to create animations using gestures and sensor events for Android & iOS. We'll cover some basics like how to get the device motion and position information, how to track gestures in the screen, and how you can combine them with animations to have fun!

After this talk, you'll have a better understanding on how to use the sensor frameworks, how to make your own gesture effects, and how to create interesting animations in an easy way.

Keep it fun, keep it animated!

3:30 PM: 🎤 Andrii Khrystian - From waves to widgets: Sound processing in Flutter

In this talk, we'll explore how to work with sound in Flutter apps. We'll go over the basics of adding sound effects and processing audio to make your apps more interesting. You'll learn how to handle audio files and integrate them smoothly with your Flutter projects. This session is great for anyone looking to add audio features to their apps simply and effectively.

4:20 PM: 🎤 Randy Nel Gupta - From Practice: Migration of an Order Processing System to the Cloud

A case study on how an order processing system, processing 50,000 orders daily for an international retailer spread across multiple continents and jurisdictions, is migrated to the cloud. The legacy system is implemented in PL/SQL and must be migrated during ongoing operations.

The presentation will cover all aspects from testing, monitoring, to development and the application of Site Reliability Engineering.

Furthermore, less technical topics will be introduced, such as the systematic composition of teams to ensure the necessary technical as well as domain-specific expertise.

4:50 PM: 🎤 Wietse Venema - Running open large language models in production with serverless GPUs

Many developers are interested in running open large language models, such as Google's Gemma and Llama. Open models give you full control over the deployment options, the timing of model upgrades, the private data that goes into the model, and the ability to fine-tune on specific tasks such as data extraction. Hugging Face TGI is a popular open-source LLM inference server, and Hugging Face TRL is excellent for fine-tuning. You’ll learn how to build and deploy an application that uses an open model on Google Cloud Run with cost-effective GPUs that scale down to zero instances.

Day 4

9:00 AM: Registration & Coffee 🥐 ☕️

10:00 AM: 🎤 Daniel Stamer & Diana Nanova - Workshop: From Prototype to Production

In this hands-on technical workshop participants will work on a hilarious web service prototype and deploy it to the cloud, set up build and deployment pipelines, extend the code base to leverage GenAI functionality, use SRE practices to effectively operate the application and finally strengthen the security posture of the overall software delivery process to guard against supply chain attacks.

1:10 PM: Lunch 🍔🥤

2:40 PM: 🎤 John Nguyen - Building a Chrome Extension using Gemini and Langchain

In this workshop, you will learn the basics of creating a Google Chrome Extension (which will also work on any Chromium-based Browser). We will build a simple Page summarizer using Bun, Typescript, Gemini, and LangChain. We will learn the anatomy of the manifest.json for building a Chrome Extension, Bun's bundler, how to interact with Gemini, and why LangChain is a good idea here.

3:45 PM: 🎤 Guillaume Vernade - How to make the most of Gemini multimodal capabilities?

We all know that in Tech there are always dozens of way of doing anything. But what if we could only use LLM for a first investigation? Let me show you how I'm trying to solve the mystery of who killed my pond's fishes using the power of Gemini.

Day 5

9:00 AM: Registration & Coffee 🥐 ☕️

10:00 AM: 🎤 Mario Bodemann & Joost van Dijk - Workshop: Passkeys on Android: How to get rid of passwords

Passwords. Or two factors? What about multiple factors? Which email did you register with? Why is 'password123' not working on this side, that is password is shared everywhere else?

If you recognize some of those questions, I am happy to add another couple: What are passkeys? Or how about: How to use passkeys to replace passwords in an Android app?

In this workshop I will walk through the later two questions: How to build an Android App that registers and signs users in, using passkeys. Expect a quick explanation of this fancy new technology, why it will replace passwords and how you can store them either on your mobile devices or on dedicated hardware. Following that, a fictive application and service will be built to show you how to use those passkeys and which moving pieces you will need.

Expect to use you Android Studio with Kotlin and common best practices to build an Android app, talking to the public available backend.

11:05 AM: 🎤 Anton Borries - Workshop: Adding Homescreen Widgets to Flutter Apps

HomeScreen Widgets are a great way to provide more Information to your Users right on their HomeScreens providing more ways for your App to appear in User's lives and help them achieve their goals.

In this Workshop we'll look at the necessary steps needed in order to add HomeScreen Widgets to Flutter Apps using the home_widget package

12:10 PM: 🎤 Elena Grahovac - Workshop: Mastering Multiple Engineering Leadership Roles for Maximum Impact

As an engineering manager or technical leader, navigating multiple roles that demand a diverse set of skills is a common yet challenging part of the job.

In this workshop, we will explore how to effectively balance these multiple roles and responsibilities in a complex engineering environment. Participants will be guided through the creation of their own leadership framework, tailored to adapt to the unique situations and styles of each individual. Beginning with identifying core values and responsibilities, the framework is elaborated into an actionable plan to succeed.

This workshop not only offers an opportunity for reflection on personal and professional development but also provides tools and insights to enhance management capabilities and team dynamics. Join us to cultivate a comprehensive approach to leadership that aligns with your unique role, responsibilities, and personal style.

1:10 PM: Lunch 🍔🥤

2:40 PM: 🎤 Gus Martins - Workshop: Gemma for Everyone: Your First Steps with Open Models and AI

Dive into the world of open models and AI with Gemma! This workshop will guide you through the basics of using Gemma, Google's powerful family of language models. Learn how to harness Gemma's capabilities for tasks like text generation, question answering, and more. We'll also explore how to fine-tune Gemma on your own data, allowing you to create custom AI solutions tailored to your needs. No prior experience with large language models is required!

3:45 PM: 🎤 Shahriyar Rzayev - Learn Flask the hard way: Introduce Architecture Patterns

Flask is a popular and flexible web framework for Python, but building scalable and maintainable Flask applications can be challenging without a solid understanding of architecture patterns. This workshop aims to provide participants with a detailed explanation of applying architecture patterns to Flask projects. By exploring various design principles and best practices, attendees will learn how to structure their Flask applications for improved scalability, modularity, and maintainability.

Focusing on the Repository, Unit of Work, and Use Cases patterns, attendees will gain experience in applying these patterns to enhance code organization, maintainability, and testability. All these layers are wired together using Dependency Injection, which is yet another powerful tool to use in your applications.

The application we are going to build is stored in: https://github.com/ShahriyarR/hexagonal-flask-blog-tutorial

We are going to completely rewrite the official Blog application described in Flask documentation by applying architecture patterns.

All abstraction layers are covered by unit and integration tests, which will give the attendees a detailed view of why it is important to structure the application using architecture patterns.


Speakers

Aleksandr Gorbunov - Smart Steel Technologies (Full Stack Developer)

A skilled developer specializing in JavaScript (JS) and TypeScript (TS), with strong expertise in frontend development. Proficient in the Vue ecosystem (Vue2, Vue3, Composition API, Nuxt 3), using Webpack and Vite for project bundling. Experienced in testing with Vitest, Cypress, and Jest. Adept in CSS preprocessors like SASS and Stylus. Additionally, has solid knowledge of Flutter and experie…

Andrey Sitnik - Evil Martians (Lead Engineer)

With more than 20 years in open source, Andrey Sitnik created a few popular CSS tools (PostCSS, Autoprefixer), local-first framework (Logux), and many small libraries with millions of downloads (like Nano ID).

Andrii Khrystian - Dynatrace (Senior Flutter Developer)

GDG Linz organiser. Senior Flutter Developer at Dynatrace. Public speaker and tech writer

Andrii Raikov - Delivery Hero SE (Principal Software Engineer)

Andrii is a Principal Software Engineer at Delivery Hero. He has a total of 15 years of experience with Ruby and has been very passionate about Go for the last 5 years.

Anton Borries - 1KOMMA5° (Software Engineer)

Anton is a Software Engineer working at 1KOMMA5° He loves building great UI and UX using Flutter. Coming from an Android Background the gap between Flutter and native Features has always tickled his interest. This has lead him into improving the experience of developing HomeScreen Widgets for Flutter Apps

Ash Davies

Google Developer Expert for Android, enthusiastic speaker, lead engineer at ImmobilieenScout24, Kotlin aficionado, spends more time travelling than working.

Daniel Stamer - Google (Cloud Customer Engineer)

Daniel is passionate about building modern cloud-native applications on Google's serverless technologies. He works with digital natives out of Germany’s startup capital Berlin and helps to modernize applications or build brand new ones in the cloud.

Danny Preussler - SoundCloud (Android Platform Lead)

Danny is a developer by heart, living in Berlin and leading the Android team at SoundCloud. He worked for companies like Groupon, Viacom, eBay and Alcatel and started his mobile career long before any Android with Java ME and Blackberry applications. Danny writes and talks about mobile development and testing regularly and is a Google Developer Expert for Android and Kotlin.

Elena Grahovac - FerretDB (Director of Engineering)

Elena has been in software engineering since 2007, focusing on backend systems and infrastructure. Having played the roles of both individual contributor and engineering manager, Elena is passionate about combining technical expertise with strong team collaboration. A dedicated advocate of DevOps practices, she aims to enhance workflows and bring teams together. Elena believes in helping peopl…

Gus Martins - Google (Developer Advocate)

Katya Vinnichenko - Google (Program Manager)

Katya is a Program Manager at Google Developer Relations team. Currently she is leading the Google Developer Groups across Europe, the Middle East and Africa.

Marcin Chudy - LeanCode (Senior Flutter Developer)

Marcin is a Senior Flutter Developer at LeanCode, currently playing tech lead role in a big project for the banking sector. Previously worked with backend, web frontend with React, finally settling on mobile and falling in love with Flutter at first sight. After work, he enjoys dancing salsa and bachata and attends metal concerts. Marcin is a Senior Flutter Developer at LeanCode and has …

Marco Gomiero - Airalo (Senior Android Developer | Kotlin GDE)

Marco is an Android engineer, currently working at Airalo. He is a Google Developer Expert for Kotlin, he loves Kotlin and he has experience with native Android and native iOS development, as well as cross-platform development with Flutter and Kotlin Multiplatform. In his spare time, he writes and maintains open-source code, he shares his dev experience by writing on his blog, speaking a…

Mario Bodeman - Yubico (Android Developer Advocate)

Speaker of talks, coder of code, doer of dones.

Muhammad Bediya

Muhammad Salman is a Senior Software Engineer specializing in mobile app development with a focus on building scalable, high-quality applications using Flutter, React Native, Xamarin, and Swift. With experience leading frontend teams on enterprise-level projects that have reached over 1.5 million users, he brings a strong commitment to creating impactful, user-centered solutions. A dedic…

Nicole Terc

Android GDE, Boardgame lover, videogame addict and origami enthusiast, Nicole self taught herself to code and has been fooling around with the Android ecosystem for more than 10 years. She has participated in a diverse variety of projects for several clients around the world, including video streaming, news, social media and public transport applications. Regardless of what the current adventu…

Ole Bulbuk - Ardan Labs

Ole is a backend engineer since the nineties. He has been working for many companies big and small and seen many projects fail or succeed. He loves to be part of the global Go community and working on projects that make the world a better place. In his spare time he is co-organising the Berlin chapter of GDG Golang, develops open source software and enjoys time with his family.

Oleksii Antypov - DmarcDkim.com (Founder & CEO)

Experienced CTO specializing in early-stage startups. Formerly with Rocket Internet and PocketBook, now focused on accelerating global DMARC adoption. Originally from Ukraine, I relocated to Berlin in 2015 to deepen my expertise in building successful startups from the ground up.

Raphaël VO - Ekino (Senior Software Engineer)

I’m Raphael Vo, a passionate Senior Software Engineer with over 10 years of experience, specializing in Angular and frontend development. I love turning complex ideas into delightful user experiences and tackling challenges creatively and enthusiastically. When I'm not coding, you’ll find me diving into the latest tech trends or enjoying epic board game nights with friends. As an aspiring spea…

Vadim Makeev

Frontend developer in love with the Web, browsers, bicycles, and podcasting. He/him, MDN technical writer, Google Developer Expert.

Alex Mir - mobile.de (Frontend Engineer)

Frontend Engineer at car retail platform mobile.de (part of Adevinta / ex-Ebay)

Alireza Rahmaty - GetYourGuide (Android Developer)

I am Alireza, an Android developer with 6+ years of experience building apps. I have experience building server-driven UI apps, complex UI, localisation and testing, and CI/CDI. I sometimes go hiking and play video games.

Cesar Martinez - Meyer Sound (Web Developer)

Web developer with around 10 years of experience and a passion for software architecture. Currently working at Meyer Sound.

Bogdan Plieshka - Zattoo (Principal Engineer)

Engineer with over a decade of Frontend development experience, passionate about automation, accessibility, and scaling complex systems. Working at Zattoo as a Principal Engineer, focusing on delivering frontend solutions across Web, React, and React Native for streaming media content.Organizer of the React Berlin Meetup, actively contributing to the development community.

Diana Nanova - Google (Customer Engineering Manager)

Diana is a Customer Engineering Manager at Google Cloud. Based in the German tech startup capital Berlin, Diana helps digital native customers and startups across various industries to leverage the capabilities of Google Cloud and loves championing for Google culture.

Doruk Deniz Kutukculer - Zalando (Head of Engineering)

IT professional and a leader with over 15 years of experience in the industry. Currently a Head of Engineering at Zalando.

Guillaume Vernade - Google (AI Dev Rel)

I've been a jack-of-all-trades in the Tech industry, starting as a prototyper building apps on Google Glasses and the first Android watches, then became a Product Owner and an Agile coach. I realized my childhood dream of becoming a video game producer then came back to my other passion: AI.

Ian Ballantyne - Google (AI DevRel)

Ian is a Developer Relations Engineer for AI at Google. Currently he works on generative AI, such as Gemini and Gemma. He is passionate about on-device AI, using technologies such as Google AI Edge to deploy artificial intelligence to web and mobile devices. He has been in Developer Relations at Google for 9 years specializing in helping partners and developers unlock the capability of Google …

Inès Mir - Zalando (Principal Product Designer)

A principal product designer at Zalando and a content creator.

John Nguyen - Eon (Backend Developer)

Fullstack developer with a knack for whipping up code recipes using my secret ingredients: a dash of JavaScript, a pinch of Python, and a whole lot of serverless magic John's journey in software development began as a PHP developer, but he later transitioned to front-end development and became passionate about all things related to Javascript. While working as a data DevOps engineer in a…

Joost van Dijk - Yubico (Developer Advocate)

Joost van Dijk is a developer advocate at Yubico. As the inventor of the YubiKey, Yubico makes secure login easy and available for everyone. Joost focuses on securing digital identities and accelerating the adoption of open authentication standards as part of Yubico’s developer program.

Randy Gupta

Randy is a Google Developer Expert for Cloud and also Organizer of the GDG Düsseldorf. With a professional experience of more 25 years in software development he is focused today on building microservices applications on top of Kubernetes.

Shahriyar Rzayev - Nord Security (Senior Software Engineer)

Senior Software Engineer @ Nord Security. Moving forward on Clean Code and Clean Architecture. Previous accomplishments include contributing to open source, providing technical direction, and sharing knowledge about Clean Code and Architectural patterns. An empathetic team player and mentor. Azerbaijan Python Group Leader. Former QA Engineer and Bug Hunter.

Tomek Porożyński - Atos

Vadym Pinchuk - Sky (Mobile Software Engineer)

Vadym, a seasoned software engineer, possesses a wealth of experience in Android application development. He has skillfully transitioned his expertise to cross-platform development, utilizing Flutter. Throughout his career, Vadym has collaborated with a diverse range of companies, from industry giants like Samsung, Volvo, Bosch, and Instagram to smaller start-ups. Leveraging his extensiv…

Wietse Venema - Google (Google Cloud Engineer)

Wietse Venema is an engineer at Google Cloud. He wrote the O’Reilly book on Cloud Run.

Hosts

Seemran Xec - Sawayo (Software Engineer)

A focused developer possessing professional experience of 6+ years in software development for product-based and service-based industries, with businesses acquiring valuable insight and implementing best practices. Collaborated with startups and other businesses as a freelancer/consultant to build, design, and manage the product. I'm passionate about what I do and a lifelong learner.

Louis Tsai - Zalando SE (GDG Organizer)

Alex Mir - mobile.de (Frontend Engineer)

Frontend Engineer at car retail platform mobile.de (part of Adevinta / ex-Ebay)

Jhoon Saravia - Greenmates (Mobile Engineer)

Software consultant and developer, experienced in Android, Flutter and Full-stack. Interested in working on DEI initiatives as a complement to my core work. Particularly interested in technology, gadgetry, the future, the combination of those three and the impact that driving Diversity, Equity and Inclusion has on all of them both in and out of the workplace.Amateur photographer a…

Matthias Geisler - Thermondo (Senior Software Engineer)

True believer in (Kotlin) Multiplatform and working with it for over 4 years now. Builds solutions for Android. Maintainer and developer of KMock. Co-Organizer of KUG Berlin, GDG Android Berlin, Rust Berlin and XTC Berlin.

Emy Jamalian - Atlas Metrics (Software QA Engineer)

Complete your event RSVP here: https://gdg.community.dev/events/details/google-gdg-berlin-presents-devfest-berlin-2024/.

DevFest Berlin 2024
Eric Sammer – Founder @ Decodable , Tobias Macey – host

Summary

Building streaming applications has gotten substantially easier over the past several years. Despite this, it is still operationally challenging to deploy and maintain your own stream processing infrastructure. Decodable was built with a mission of eliminating all of the painful aspects of developing and deploying stream processing systems for engineering teams. In this episode Eric Sammer discusses why more companies are including real-time capabilities in their products and the ways that Decodable makes it faster and easier.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare production and development environments and column-level lineage to show you the exact impact of every code change on data, metrics, and BI tools, keeping your team productive and stakeholders happy. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting dataengineeringpodcast.com/datafold You shouldn't have to throw away the database to build with fast-changing data. You should be able to keep the familiarity of SQL and the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. With Materialize, you can! It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. Whether it’s real-time dashboarding and analytics, personalization and segmentation or automation and alerting, Materialize gives you the ability to work with fresh, correct, and scalable results — all in a familiar SQL interface. Go to dataengineeringpodcast.com/materialize today to get 2 weeks free! As more people start using AI for projects, two things are clear: It’s a rapidly advancing field, but it’s tough to navigate. How can you get the best results for your use case? Instead of being subjected to a bunch of buzzword bingo, hear directly from pioneers in the developer and data science space on how they use graph tech to build AI-powered apps. . Attend the dev and ML talks at NODES 2023, a free online conference on October 26 featuring some of the brightest minds in tech. Check out the agenda and register today at Neo4j.com/NODES. Your host is Tobias Macey and today I'm interviewing Eric Sammer about starting your stream processing journey with Decodable

Interview

Introduction How did you get involved in the area of data management? Can you describe what Decodable is and the story behind it?

What are the notable changes to the Decodable platform since we last spoke? (October 2021) What are the industry shifts that have influenced the product direction?

What are the problems that customers are trying to solve when they come to Decodable? When you launched your focus was on SQL transformations of streaming data. What was the process for adding full Java support in addition to SQL? What are the developer experience challenges that are particular to working with streaming data?

How have you worked to address that in the Decodable platform and interfaces?

As you evolve the technical and product direction, what is your heuristic for balancing the unification of interfaces and system integration against the ability to swap different components or interfaces as new technologies are introduced? What are the most interesting, innovative, or unexpected ways that you have seen Decodable used? What are the most interesting, unexpected, or challenging lessons that you have learned while working on Decodable? When is Decodable the wrong choice? What do you have planned for the future of Decodable?

Contact Info

esammer on GitHub LinkedIn

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.init covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story. To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers

Links

Decodable

Podcast Episode

Understanding the Apache Flink Journey Flink

Podcast Episode

Debezium

Podcast Episode

Kafka Redpanda

Podcast Episode

Kinesis PostgreSQL

Podcast Episode

Snowflake

Podcast Episode

Databricks Startree Pinot

Podcast Episode

Rockset

Podcast Episode

Druid InfluxDB Samza Storm Pulsar

Podcast Episode

ksqlDB

Podcast Episode

dbt GitHub Actions Airbyte Singer Splunk Outbox Pattern

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Sponsored By: Neo4J: NODES Conference Logo

NODES 2023 is a free online conference focused on graph-driven innovations with content for all skill levels. Its 24 hours are packed with 90 interactive technical sessions from top developers and data scientists across the world covering a broad range of topics and use cases. The event tracks: - Intelligent Applications: APIs, Libraries, and Frameworks – Tools and best practices for creating graph-powered applications and APIs with any software stack and programming language, including Java, Python, and JavaScript - Machine Learning and AI – How graph technology provides context for your data and enhances the accuracy of your AI and ML projects (e.g.: graph neural networks, responsible AI) - Visualization: Tools, Techniques, and Best Practices – Techniques and tools for exploring hidden and unknown patterns in your data and presenting complex relationships (knowledge graphs, ethical data practices, and data representation)

Don’t miss your chance to hear about the latest graph-powered implementations and best practices for free on October 26 at NODES 2023. Go to Neo4j.com/NODES today to see the full agenda and register!Rudderstack: Rudderstack

Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstackMaterialize: Materialize

You shouldn't have to throw away the database to build with fast-changing data. Keep the familiar SQL, keep the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date.

That is Materialize, the only true SQL streaming database built from the ground up to meet the needs of modern data products: Fresh, Correct, Scalable — all in a familiar SQL UI. Built on Timely Dataflow and Differential Dataflow, open source frameworks created by cofounder Frank McSherry at Microsoft Research, Materialize is trusted by data and engineering teams at Ramp, Pluralsight, Onward and more to build real-time data products without the cost, complexity, and development time of stream processing.

Go to materialize.com today and get 2 weeks free!Datafold: Datafold

This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare…

AI/ML Airbyte Analytics Flink API Kinesis BI CI/CD Cloud Computing Data Engineering Data Management Data Quality Data Science Databricks Dataflow Datafold dbt Druid GitHub Java JavaScript Kafka Modern Data Stack Microsoft Neo4j postgresql Python Redpanda SaaS Singer Snowflake Splunk SQL Data Streaming
Data Engineering Podcast

Please register through the following link:

  • https://redis.com/events-and-webinars/build-and-deploy-scalable-real-time-applications/

When Redis meets JSON, microservices, Clouds and Containers.

Join us for our next meetup in the Redis London office where we will showcase how easy it is to build real-time applications using Redis Stack, RedisInsight, JSON, and search.

We’ll show how to scale both Redis deployments and microservices – whether you’re using managed services on public cloud with Redis Enterprise Cloud, or retaining more architecture options and control with Kubernetes-based deployment and automation.

The meetup will consist of 2 sessions:

Session 1: Real-time scalable applications powered by Redis, JSON, Vector Similarity and Search

Get customers the data they need – faster.

Developers and architects who build applications turn to real-time search for the performance required to keep users happy – because the competition is always just a click away.

Speed up your applications with Redis’ real-time indexing, querying, and full-text search engine for JSON. Search and Query is a flexible indexing, query and search engine that delivers sub-millisecond results.

We’ll also touch on Redis’ vector similarity search capabilities. Search and Query integrates with Java with Lettuce, Jedis or even Spring Boot / Spring Data with easy-to-use object mapping and repository interfaces and annotations.

You will learn how Search and Query overcomes common application challenges, including:

  • Ingest performance: To search fast, you need to ingest and index data fast for both key value and JSON, all powered by Redis’ in-memory architecture.
  • Rich querying: Applications need current and immediately usable data. Discover how to combine queries, aggregation, full text search and vector similarity search in a cohesive, simple architecture.
  • Speed and scalability: Redis Enterprise in-memory database scales across database shards and cluster nodes, providing high concurrency. It also includes data protection with persistence and backup built in.
  • Be productive and run anywhere: Experiment on desktop with Redis Stack and RedisInsight, deploy to cloud in seconds, or to on-premises with Redis Enterprise.

Session 2: Microservices, Redis, Cloud and Kubernetes - patterns for deploying quickly, reliably and at-scale.

Deploying apps using the KubernetesAPI has made life easier for developers, especially when deploying across multiple regions or environments. But why stop there? Use the KubernetesAPI to provision and scale Redis databases as well, with the size, reliability and capabilities your app needs. – all fully automated so you can focus on your app rather than the infrastructure.

But that’s not all. In a distributed environment you need a lightweight, asynchronous way to communicate between applications – one that’s also scalable, resilient, and reliable. We’ll show how Redis Streams couples applications/microservices loosely, allowing graceful degradation and restore at the same time. That’s all while keeping data-duplication, bandwidth, and egress costs at a minimum.

Finally, as part of your journey to cloud, consider deploying Redis as a fully managed service. Discover the available options to deploy Redis natively to Google Cloud, Amazon, or Azure and see how to start right away with small caches, large databases with persistence, and Redis database with JSON/RediSearch support.

In this session you will learn:

  • How to build resilient, scalable, and reactive applications using Redis Streams
  • How to use Search and Query and JSON to go beyond basic caching scenarios
  • How to provision and scale Redis databases using the Kubernetes API or your preferred cloud
Build and Deploy Scalable Real-Time Applications

Event Full!

Join PingCAP and AWS for an informative, hands-on, in-person workshop!

Space is limited, please register here to reserve your seat: https://www.pingcap.com/event/building-scalable-microservices-with-tidb-and-aws-lambda/

Time: July 24th, 10 AM – 2 PM ET

Location: AWS New York Office

Address: 7 West 34th Street, New York, NY 10001​

Modern data-intensive applications require an equally modern data stack that can process ever-growing transactional and analytical workloads at scale without provisioning or managing servers.

Join PingCAP and AWS for an in-person workshop where you’ll learn about TiDB Serverless, AWS Lambda, and Amazon QuickSight. You’ll explore how to combine them to build scalable, highly-available microservices while generating real-time insights directly from raw application data.

Attendees will discover the capabilities and benefits of developing a high volume of transactional data and – as a bonus – analyze that same data set in real time.

During this event, we’ll deploy an application that showcases Amazon QuickSight picking up real-time data changes in TiDB Serverless. These changes will be user-issued via calls to API endpoints run by AWS Lambda functions.

Attendees will be given step-by-step instructions on how to reproduce the demo application in a provided AWS environment.

Breakfast and lunch will be available. We’ll also have some free swag on hand for giveaway.

Who should attend:

Application developers, architects, and DBAs

What you will learn:

  • How to connect TiDB Serverless, AWS Lambda, and Amazon QuickSight to create a scalable, ACID-compliant application stack that provides real-time analytics on transactional data.
  • How AWS Lambda enables developers to run code without provisioning or managing servers.
  • How Amazon QuickSight can connect to TiDB Serverless and provide insights into application data.
  • What problems TiDB Serverless solves as a backend for microservice applications.
  • How TiDB Serverless’ architecture specifically helps with new paradigms of online applications.

Speakers

Ayan Ray Senior Partner Solutions Architect – Data & Analytics, AWS Ayan Ray specializes in architecting well-architected solutions with AWS Services and AWS partner products. He has deep expertise in architecting solutions in the field of Data Analytics, Serverless, and Microservices.

Sam Dillard Principal Product Manager, PingCAP Sam Dillard is an experienced Product Manager in both the OLTP and OLAP database spaces, with a specialty in distributed systems and data engineering.

AWS Immersion Day - New York City

Event Full!

Join PingCAP and AWS for an informative, hands-on, in-person workshop!

Space is limited, please register here to reserve your seat: https://www.pingcap.com/event/building-scalable-microservices-with-tidb-and-aws-lambda/

Time: July 24th, 10 AM – 2 PM ET

Location: AWS New York Office

Address: 7 West 34th Street, New York, NY 10001​

Modern data-intensive applications require an equally modern data stack that can process ever-growing transactional and analytical workloads at scale without provisioning or managing servers.

Join PingCAP and AWS for an in-person workshop where you’ll learn about TiDB Serverless, AWS Lambda, and Amazon QuickSight. You’ll explore how to combine them to build scalable, highly-available microservices while generating real-time insights directly from raw application data.

Attendees will discover the capabilities and benefits of developing a high volume of transactional data and – as a bonus – analyze that same data set in real time.

During this event, we’ll deploy an application that showcases Amazon QuickSight picking up real-time data changes in TiDB Serverless. These changes will be user-issued via calls to API endpoints run by AWS Lambda functions.

Attendees will be given step-by-step instructions on how to reproduce the demo application in a provided AWS environment.

Breakfast and lunch will be available. We’ll also have some free swag on hand for giveaway.

Who should attend:

Application developers, architects, and DBAs

What you will learn:

  • How to connect TiDB Serverless, AWS Lambda, and Amazon QuickSight to create a scalable, ACID-compliant application stack that provides real-time analytics on transactional data.
  • How AWS Lambda enables developers to run code without provisioning or managing servers.
  • How Amazon QuickSight can connect to TiDB Serverless and provide insights into application data.
  • What problems TiDB Serverless solves as a backend for microservice applications.
  • How TiDB Serverless’ architecture specifically helps with new paradigms of online applications.

Speakers

Ayan Ray Senior Partner Solutions Architect – Data & Analytics, AWS Ayan Ray specializes in architecting well-architected solutions with AWS Services and AWS partner products. He has deep expertise in architecting solutions in the field of Data Analytics, Serverless, and Microservices.

Sam Dillard Principal Product Manager, PingCAP Sam Dillard is an experienced Product Manager in both the OLTP and OLAP database spaces, with a specialty in distributed systems and data engineering.

AWS Immersion Day - New York City

Event Full!

Join PingCAP and AWS for an informative, hands-on, in-person workshop!

Space is limited, please register here to reserve your seat: https://www.pingcap.com/event/building-scalable-microservices-with-tidb-and-aws-lambda/

Time: July 24th, 10 AM – 2 PM ET

Location: AWS New York Office

Address: 7 West 34th Street, New York, NY 10001​

Modern data-intensive applications require an equally modern data stack that can process ever-growing transactional and analytical workloads at scale without provisioning or managing servers.

Join PingCAP and AWS for an in-person workshop where you’ll learn about TiDB Serverless, AWS Lambda, and Amazon QuickSight. You’ll explore how to combine them to build scalable, highly-available microservices while generating real-time insights directly from raw application data.

Attendees will discover the capabilities and benefits of developing a high volume of transactional data and – as a bonus – analyze that same data set in real time.

During this event, we’ll deploy an application that showcases Amazon QuickSight picking up real-time data changes in TiDB Serverless. These changes will be user-issued via calls to API endpoints run by AWS Lambda functions.

Attendees will be given step-by-step instructions on how to reproduce the demo application in a provided AWS environment.

Breakfast and lunch will be available. We’ll also have some free swag on hand for giveaway.

Who should attend:

Application developers, architects, and DBAs

What you will learn:

  • How to connect TiDB Serverless, AWS Lambda, and Amazon QuickSight to create a scalable, ACID-compliant application stack that provides real-time analytics on transactional data.
  • How AWS Lambda enables developers to run code without provisioning or managing servers.
  • How Amazon QuickSight can connect to TiDB Serverless and provide insights into application data.
  • What problems TiDB Serverless solves as a backend for microservice applications.
  • How TiDB Serverless’ architecture specifically helps with new paradigms of online applications.

Speakers

Ayan Ray Senior Partner Solutions Architect – Data & Analytics, AWS Ayan Ray specializes in architecting well-architected solutions with AWS Services and AWS partner products. He has deep expertise in architecting solutions in the field of Data Analytics, Serverless, and Microservices.

Sam Dillard Principal Product Manager, PingCAP Sam Dillard is an experienced Product Manager in both the OLTP and OLAP database spaces, with a specialty in distributed systems and data engineering.

AWS Immersion Day - New York City

Event Full!

Join PingCAP and AWS for an informative, hands-on, in-person workshop!

Space is limited, please register here to reserve your seat: https://www.pingcap.com/event/building-scalable-microservices-with-tidb-and-aws-lambda/

Time: July 24th, 10 AM – 2 PM ET

Location: AWS New York Office

Address: 7 West 34th Street, New York, NY 10001​

Modern data-intensive applications require an equally modern data stack that can process ever-growing transactional and analytical workloads at scale without provisioning or managing servers.

Join PingCAP and AWS for an in-person workshop where you’ll learn about TiDB Serverless, AWS Lambda, and Amazon QuickSight. You’ll explore how to combine them to build scalable, highly-available microservices while generating real-time insights directly from raw application data.

Attendees will discover the capabilities and benefits of developing a high volume of transactional data and – as a bonus – analyze that same data set in real time.

During this event, we’ll deploy an application that showcases Amazon QuickSight picking up real-time data changes in TiDB Serverless. These changes will be user-issued via calls to API endpoints run by AWS Lambda functions.

Attendees will be given step-by-step instructions on how to reproduce the demo application in a provided AWS environment.

Breakfast and lunch will be available. We’ll also have some free swag on hand for giveaway.

Who should attend:

Application developers, architects, and DBAs

What you will learn:

  • How to connect TiDB Serverless, AWS Lambda, and Amazon QuickSight to create a scalable, ACID-compliant application stack that provides real-time analytics on transactional data.
  • How AWS Lambda enables developers to run code without provisioning or managing servers.
  • How Amazon QuickSight can connect to TiDB Serverless and provide insights into application data.
  • What problems TiDB Serverless solves as a backend for microservice applications.
  • How TiDB Serverless’ architecture specifically helps with new paradigms of online applications.

Speakers

Ayan Ray Senior Partner Solutions Architect – Data & Analytics, AWS Ayan Ray specializes in architecting well-architected solutions with AWS Services and AWS partner products. He has deep expertise in architecting solutions in the field of Data Analytics, Serverless, and Microservices.

Sam Dillard Principal Product Manager, PingCAP Sam Dillard is an experienced Product Manager in both the OLTP and OLAP database spaces, with a specialty in distributed systems and data engineering.

AWS Immersion Day - New York City

Event Full!

Join PingCAP and AWS for an informative, hands-on, in-person workshop!

Space is limited, please register here to reserve your seat: https://www.pingcap.com/event/building-scalable-microservices-with-tidb-and-aws-lambda/

Time: July 24th, 10 AM – 2 PM ET

Location: AWS New York Office

Address: 7 West 34th Street, New York, NY 10001​

Modern data-intensive applications require an equally modern data stack that can process ever-growing transactional and analytical workloads at scale without provisioning or managing servers.

Join PingCAP and AWS for an in-person workshop where you’ll learn about TiDB Serverless, AWS Lambda, and Amazon QuickSight. You’ll explore how to combine them to build scalable, highly-available microservices while generating real-time insights directly from raw application data.

Attendees will discover the capabilities and benefits of developing a high volume of transactional data and – as a bonus – analyze that same data set in real time.

During this event, we’ll deploy an application that showcases Amazon QuickSight picking up real-time data changes in TiDB Serverless. These changes will be user-issued via calls to API endpoints run by AWS Lambda functions.

Attendees will be given step-by-step instructions on how to reproduce the demo application in a provided AWS environment.

Breakfast and lunch will be available. We’ll also have some free swag on hand for giveaway.

Who should attend:

Application developers, architects, and DBAs

What you will learn:

  • How to connect TiDB Serverless, AWS Lambda, and Amazon QuickSight to create a scalable, ACID-compliant application stack that provides real-time analytics on transactional data.
  • How AWS Lambda enables developers to run code without provisioning or managing servers.
  • How Amazon QuickSight can connect to TiDB Serverless and provide insights into application data.
  • What problems TiDB Serverless solves as a backend for microservice applications.
  • How TiDB Serverless’ architecture specifically helps with new paradigms of online applications.

Speakers

Ayan Ray Senior Partner Solutions Architect – Data & Analytics, AWS Ayan Ray specializes in architecting well-architected solutions with AWS Services and AWS partner products. He has deep expertise in architecting solutions in the field of Data Analytics, Serverless, and Microservices.

Sam Dillard Principal Product Manager, PingCAP Sam Dillard is an experienced Product Manager in both the OLTP and OLAP database spaces, with a specialty in distributed systems and data engineering.

AWS Immersion Day - New York City

Event Full!

Join PingCAP and AWS for an informative, hands-on, in-person workshop!

Space is limited, please register here to reserve your seat: https://www.pingcap.com/event/building-scalable-microservices-with-tidb-and-aws-lambda/

Time: July 24th, 10 AM – 2 PM ET

Location: AWS New York Office

Address: 7 West 34th Street, New York, NY 10001​

Modern data-intensive applications require an equally modern data stack that can process ever-growing transactional and analytical workloads at scale without provisioning or managing servers.

Join PingCAP and AWS for an in-person workshop where you’ll learn about TiDB Serverless, AWS Lambda, and Amazon QuickSight. You’ll explore how to combine them to build scalable, highly-available microservices while generating real-time insights directly from raw application data.

Attendees will discover the capabilities and benefits of developing a high volume of transactional data and – as a bonus – analyze that same data set in real time.

During this event, we’ll deploy an application that showcases Amazon QuickSight picking up real-time data changes in TiDB Serverless. These changes will be user-issued via calls to API endpoints run by AWS Lambda functions.

Attendees will be given step-by-step instructions on how to reproduce the demo application in a provided AWS environment.

Breakfast and lunch will be available. We’ll also have some free swag on hand for giveaway.

Who should attend:

Application developers, architects, and DBAs

What you will learn:

  • How to connect TiDB Serverless, AWS Lambda, and Amazon QuickSight to create a scalable, ACID-compliant application stack that provides real-time analytics on transactional data.
  • How AWS Lambda enables developers to run code without provisioning or managing servers.
  • How Amazon QuickSight can connect to TiDB Serverless and provide insights into application data.
  • What problems TiDB Serverless solves as a backend for microservice applications.
  • How TiDB Serverless’ architecture specifically helps with new paradigms of online applications.

Speakers

Ayan Ray Senior Partner Solutions Architect – Data & Analytics, AWS Ayan Ray specializes in architecting well-architected solutions with AWS Services and AWS partner products. He has deep expertise in architecting solutions in the field of Data Analytics, Serverless, and Microservices.

Sam Dillard Principal Product Manager, PingCAP Sam Dillard is an experienced Product Manager in both the OLTP and OLAP database spaces, with a specialty in distributed systems and data engineering.

AWS Immersion Day - New York City

Event Full!

Join PingCAP and AWS for an informative, hands-on, in-person workshop!

Space is limited, please register here to reserve your seat: https://www.pingcap.com/event/building-scalable-microservices-with-tidb-and-aws-lambda/

Time: July 24th, 10 AM – 2 PM ET

Location: AWS New York Office

Address: 7 West 34th Street, New York, NY 10001​

Modern data-intensive applications require an equally modern data stack that can process ever-growing transactional and analytical workloads at scale without provisioning or managing servers.

Join PingCAP and AWS for an in-person workshop where you’ll learn about TiDB Serverless, AWS Lambda, and Amazon QuickSight. You’ll explore how to combine them to build scalable, highly-available microservices while generating real-time insights directly from raw application data.

Attendees will discover the capabilities and benefits of developing a high volume of transactional data and – as a bonus – analyze that same data set in real time.

During this event, we’ll deploy an application that showcases Amazon QuickSight picking up real-time data changes in TiDB Serverless. These changes will be user-issued via calls to API endpoints run by AWS Lambda functions.

Attendees will be given step-by-step instructions on how to reproduce the demo application in a provided AWS environment.

Breakfast and lunch will be available. We’ll also have some free swag on hand for giveaway.

Who should attend:

Application developers, architects, and DBAs

What you will learn:

  • How to connect TiDB Serverless, AWS Lambda, and Amazon QuickSight to create a scalable, ACID-compliant application stack that provides real-time analytics on transactional data.
  • How AWS Lambda enables developers to run code without provisioning or managing servers.
  • How Amazon QuickSight can connect to TiDB Serverless and provide insights into application data.
  • What problems TiDB Serverless solves as a backend for microservice applications.
  • How TiDB Serverless’ architecture specifically helps with new paradigms of online applications.

Speakers

Ayan Ray Senior Partner Solutions Architect – Data & Analytics, AWS Ayan Ray specializes in architecting well-architected solutions with AWS Services and AWS partner products. He has deep expertise in architecting solutions in the field of Data Analytics, Serverless, and Microservices.

Sam Dillard Principal Product Manager, PingCAP Sam Dillard is an experienced Product Manager in both the OLTP and OLAP database spaces, with a specialty in distributed systems and data engineering.

AWS Immersion Day - New York City
Jowanza Joseph – author

Every enterprise application creates data, including log messages, metrics, user activity, and outgoing messages. Learning how to move these items is almost as important as the data itself. If you're an application architect, developer, or production engineer new to Apache Pulsar, this practical guide shows you how to use this open source event streaming platform to handle real-time data feeds. Jowanza Joseph, staff software engineer at Finicity, explains how to deploy production Pulsar clusters, write reliable event streaming applications, and build scalable real-time data pipelines with this platform. Through detailed examples, you'll learn Pulsar's design principles, reliability guarantees, key APIs, and architecture details, including the replication protocol, the load manager, and the storage layer. This book helps you: Understand how event streaming fits in the big data ecosystem Explore Pulsar producers, consumers, and readers for writing and reading events Build scalable data pipelines by connecting Pulsar with external systems Simplify event-streaming application building with Pulsar Functions Manage Pulsar to perform monitoring, tuning, and maintenance tasks Use Pulsar's operational measurements to secure a production cluster Process event streams using Flink and query event streams using Presto

data data-engineering apache-pulsar Flink API Big Data Presto Data Streaming
O'Reilly Data Engineering Books
Matteo Merli – guest , Tobias Macey – host

Summary The way to build maintainable software and systems is through composition of individual pieces. By making those pieces high quality and flexible they can be used in surprising ways that the original creators couldn’t have imagined. One such component that has gone above and beyond its originally envisioned use case is BookKeeper, a distributed storage system that is optimized for durability and speed. In this episode Matteo Merli shares the story behind the creation of BookKeeper, the various ways that it is being used today, and the architectural aspects that make it such a strong building block for projects such as Pulsar. He also shares some of the other interesting systems that have been built on top of it and an amusing war story of running it at scale in its early years.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show! RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming. With RudderStack you can use all of your customer data to answer more difficult questions and then send those insights to your whole customer data stack. Sign up free at dataengineeringpodcast.com/rudder today. We’ve all been asked to help with an ad-hoc request for data by the sales and marketing team. Then it becomes a critical report that they need updated every week or every day. Then what do you do? Send a CSV via email? Write some Python scripts to automate it? But what about incremental sync, API quotas, error handling, and all of the other details that eat up your time? Today, there is a better way. With Census, just write SQL or plug in your dbt models and start syncing your cloud warehouse to SaaS applications like Salesforce, Marketo, Hubspot, and many more. Go to dataengineeringpodcast.com/census today to get a free 14-day trial. Your host is Tobias Macey and today I’m interviewing Matteo Merli about Apache BookKeeper, a scalable, fault-tolerant, and low-latency storage service optimized for real-time workloads

Interview

Introduction How did you get involved in the area of data management? Can you describe what BookKeeper is and the story behind it? What are the most notable features/capabilities of BookKeeper? What are some of the ways that BookKeeper is being used? How has your work on Pulsar influenced the features and product direction of BookKeeper? Can you describe the architecture of a BookKeeper cluster?

How have the design and goals of BookKeeper changed or evolved over time?

What is the impact of record-oriented storage on data distribution/allocation within the cluster when working with variable record sizes? What are some of the operational considerations that users should be aware of? What are some of the most interesting/compelling features from your perspective? What are some of the most often overlooked or misunderstood capabilities of BookKeeper? What are the most interesting, innovative, or unexpected ways that you have seen BookKeeper used? What

API BigQuery Cloud Computing CSV Data Engineering Data Management dbt DWH Hubspot Kubernetes Marketing Python Redshift SaaS Snowflake SQL Data Streaming
Jorge Sancha – CEO @ Tinybird , Tobias Macey – host

Summary Building an API for real-time data is a challenging project. Making it robust, scalable, and fast is a full time job. The team at Tinybird wants to make it easy to turn a continuous stream of data into a production ready API or data product. In this episode CEO Jorge Sancha explains how they have architected their system to handle high data throughput and fast response times, and why they have invested heavily in Clickhouse as the core of their platform. This is a great conversation about the challenges of building a maintainable business from a technical and product perspective.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show! RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming. With RudderStack you can use all of your customer data to answer more difficult questions and then send those insights to your whole customer data stack. Sign up free at dataengineeringpodcast.com/rudder today. Ascend.io — recognized as a 2021 Gartner Cool Vendor in Enterprise AI Operationalization and Engineering—empowers data teams to to build, scale, and operate declarative data pipelines with 95% less code and zero maintenance. Connect to any data source using Ascend’s new flex code data connectors, rapidly iterate on transformations and send data to any destination in a fraction of the time it traditionally takes—just ask companies like Harry’s, HNI, and Mayvenn. Sound exciting? Come join the team! We’re hiring data engineers, so head on over to dataengineeringpodcast.com/ascend and check out our careers page to learn more. Your host is Tobias Macey and today I’m interviewing Jorge Sancha about Tinybird, a platform to easily build analytical APIs for real-time data

Interview

Introduction How did you get involved in the area of data management? Can you start by describing what you are building at Tinybird and the story behind it? What are some of the types of use cases that your customers are focused on? What are the areas of complexity that come up when building analytical APIs that are often overlooked when first designing a system to operate on and expose real-time data?

What are the supporting systems that are necessary and useful for operating this kind of system which contribute to the overall time and engineering cost beyond the baseline functionality?

How is the Tinybird platform architected?

How have the goals and implementation of Tinybird changed or evolved since you first began building it?

What was your criteria for selecting the core building block of your platform, and how did that lead to your choice to build on top of Clickhouse? What are some of the sharp edges that you have run into while operating Clickhouse?

What are some of the custom tools or systems that you have built to help deal with them?

What are some of the performance challenges that an API built with Tinybird might run into?

What are the considerations that users should be

AI/ML API BigQuery ClickHouse Cloud Computing Data Engineering Data Management DWH Kubernetes Redshift Snowflake Data Streaming
Kenny Gorman – Founder and CEO @ Eventador , Tobias Macey – host

Summary Modern applications frequently require access to real-time data, but building and maintaining the systems that make that possible is a complex and time consuming endeavor. Eventador is a managed platform designed to let you focus on using the data that you collect, without worrying about how to make it reliable. In this episode Eventador Founder and CEO Kenny Gorman describes how the platform is architected, the challenges inherent to managing reliable streams of data, the simplicity offered by a SQL interface, and the interesting projects that his customers have built on top of it. This was an interesting inside look at building a business on top of open source stream processing frameworks and how to reduce the burden on end users.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, a 40Gbit public network, fast object storage, and a brand new managed Kubernetes platform, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. And for your machine learning workloads, they’ve got dedicated CPU and GPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show! Your host is Tobias Macey and today I’m interviewing Kenny Gorman about the Eventador streaming SQL platform

Interview

Introduction How did you get involved in the area of data management? Can you start by describing what the Eventador platform is and the story behind it?

How has your experience at ObjectRocket influenced your approach to streaming SQL? How do the capabilities and developer experience of Eventador compare to other streaming SQL engines such as ksqlDB, Pulsar SQL, or Materialize?

What are the main use cases that you are seeing people use for streaming SQL?

How does it fit into an application architecture? What are some of the design changes in the different layers that are necessary to take advantage of the real time capabilities?

Can you describe how the Eventador platform is architected?

How has the system design evolved since you first began working on it? How has the overall landscape of streaming systems changed since you first began working on Eventador? If you were to start over today what would you do differently?

What are some of the most interesting and challenging operational aspects of running your platform? What are some of the ways that you have modified or augmented the SQL dialect that you support?

What is the tipping point for when SQL is insufficient for a given task and a user might want to leverage Flink?

What is the workflow for developing and deploying different SQL jobs?

How do you handle versioning of the queries and integration with the software development lifecycle?

What are some data modeling considerations that users should be aware of?

What are some of the sharp edges or design pitfalls that users should be aware of?

What are some of the most interesting, innovative, or unexpected ways that you have seen your customers use your platform? What are some of the most interesting, unexpected, or challenging lessons that you have learned in the process of building and scaling Eventador? What do you have planned for the future of the platform?

Contact Info

LinkedIn Blog @kennygorman on Twitter kgorman on Twitter

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don’t forget to check out our other show, Podcast.init to learn about the Python language, its community, and the innovative ways it is being used. Visit t

AI/ML Flink Data Engineering Data Management Data Modelling Kubernetes Python SQL Data Streaming
Adam Kocoloski – guest , Tobias Macey – host

Summary CouchDB is a distributed document database built for scale and ease of operation. With a built-in synchronization protocol and a HTTP interface it has become popular as a backend for web and mobile applications. Created 15 years ago, it has accrued some technical debt which is being addressed with a refactored architecture based on FoundationDB. In this episode Adam Kocoloski shares the history of the project, how it works under the hood, and how the new design will improve the project for our new era of computation. This was an interesting conversation about the challenges of maintaining a large and mission critical project and the work being done to evolve it.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, a 40Gbit public network, fast object storage, and a brand new managed Kubernetes platform, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. And for your machine learning workloads, they’ve got dedicated CPU and GPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show! Are you spending too much time maintaining your data pipeline? Snowplow empowers your business with a real-time event data pipeline running in your own cloud account without the hassle of maintenance. Snowplow takes care of everything from installing your pipeline in a couple of hours to upgrading and autoscaling so you can focus on your exciting data projects. Your team will get the most complete, accurate and ready-to-use behavioral web and mobile data, delivered into your data warehouse, data lake and real-time streams. Go to dataengineeringpodcast.com/snowplow today to find out why more than 600,000 websites run Snowplow. Set up a demo and mention you’re a listener for a special offer! Setting up and managing a data warehouse for your business analytics is a huge task. Integrating real-time data makes it even more challenging, but the insights you obtain can make or break your business growth. You deserve a data warehouse engine that outperforms the demands of your customers and simplifies your operations at a fraction of the time and cost that you might expect. You deserve ClickHouse, the open-source analytical database that deploys and scales wherever and whenever you want it to and turns data into actionable insights. And Altinity, the leading software and service provider for ClickHouse, is on a mission to help data engineers and DevOps managers tame their operational analytics. Go to dataengineeringpodcast.com/altinity for a free consultation to find out how they can help you today. You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management. For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Corinium Global Intelligence, ODSC, and Data Council. Upcoming events include the Software Architecture Conference in NYC, Strata Data in San Jose, and PyCon US in Pittsburgh. Go to dataengineeringpodcast.com/conferences to learn more about these and other events, and take advantage of our partner discounts to save money when you register today. Your host is Tobias Macey and today I’m interviewing Adam Kocoloski about CouchDB and the work being done to migrate the storage layer to FoundationDB

Interview

Introduction How did you get involved in the area of data management? Can you starty by describing what CouchDB is?

How did you get involved in the CouchDB project and what is your current role in the community?

What are the use cases that it is well suited for? Can you share some of the history of CouchDB and its role in the NoSQL movement? How is CouchDB currently architected and how has it evolved since it was first introduced? What have been the benefits and challenges of Erlang as the runtime for CouchDB? How is the current storage engine implemented and what are its shortcomings? What problems are you trying to solve by replatforming on a new storage layer?

What were the selection criteria for the new storage engine and how did you structure the decision making process? What was the motivation for choosing FoundationDB as opposed to other options such as rocksDB, levelDB, etc.?

How is the adoption of FoundationDB going to impact the overall architecture and implementation of CouchDB? How will the use of FoundationDB impact the way that the current capabilities are implemented, such as data replication? What will the migration path be for people running an existing installation? What are some of the biggest challenges that you are facing in rearchitecting the codebase? What new capabilities will the FoundationDB storage layer enable? What are some of the most interesting/unexpected/innovative ways that you have seen CouchDB used?

What new capabilities or use cases do you anticipate once this migration is complete?

What are some of the most interesting/unexpected/challenging lessons that you have learned while working with the CouchDB project and community? What is in store for the future of CouchDB?

Contact Info

LinkedIn @kocolosk on Twitter kocolosk on GitHub

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

Apache CouchDB FoundationDB

Podcast Episode

IBM Cloudant Experimental Particle Physics FPGA == Field Programmable Gate Array Apache Software Foundation CRDT == Conflict-free Replicated Data Type

Podcast Episode

Erlang Riak RabbitMQ Heisenbug Kubernetes Property Based Testing

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Support Data Engineering Podcast

AI/ML Analytics Big Data ClickHouse Cloud Computing Data Engineering Data Lake Data Management DevOps DWH GitHub IBM Kubernetes NoSQL Snowplow Data Streaming