Lakeflow Connect streamlines the ingestion of incremental data from popular databases like SQL Server and PostgreSQL. In this session, we’ll review best practices for networking, security, minimizing database load, monitoring and more — tailored to common industry scenarios. Join us to gain practical insights into Lakeflow Connect's functionality so that you’re ready to build your own pipelines. Whether you're looking to optimize data ingestion or enhance your database integrations, this session will provide you with a deep understanding of how Lakeflow Connect works with databases.

Hundreds of customers are already ingesting data with Lakeflow Connect from SQL Server, Salesforce, ServiceNow, Google Analytics, SharePoint, PostgreSQL and more to unlock the full power of their data. Lakeflow Connect introduces built-in, no-code ingestion connectors from SaaS applications, databases and file sources to help unlock data intelligence. In this demo-packed session, you’ll learn how to ingest ready-to-use data for analytics and AI with a few clicks in the UI or a few lines of code. We’ll also demonstrate how Lakeflow Connect is fully integrated with the Databricks Data Intelligence Platform for built-in governance, observability, CI/CD, automated pipeline maintenance and more. Finally, we’ll explain how to use Lakeflow Connect in combination with downstream analytics and AI tools to tackle common business challenges and drive business impact.

Future-proof your data architecture: Learn how DoorDash built a data lakehouse powered by Starburst to achieve a 20-30% faster time to insights. Akshat Nair shares lessons learned about what drove DoorDash to move beyond Snowflake to embrace the lakehouse. He will share his rationale for selecting Trino as their lakehouse query engine and why his team chose Starburst over open source. Discover how DoorDash seamlessly queries diverse sources, including Snowflake, Postgres, and data lake table formats, achieving faster data-driven decision-making at scale with cost benefits.

Streaming data with Apache Kafka® has become the backbone of modern day applications. While streams are ideal for continuous data flow, they lack built-in querying capability. Unlike databases with indexed lookups, Kafka's append-only logs are designed for high throughput processing, not for on-demand querying. This necessitates teams to build additional infrastructure to enable query capabilities for streaming data. Traditional methods replicate this data into external stores such as relational databases like PostgreSQL for operational workloads and object storage like S3 with Flink, Spark, or Trino for analytical use cases. While useful sometimes, these methods deepen the divide between operational and analytical estates, creating silos, complex ETL pipelines, and issues with schema mismatches, freshness, and failures.\n\nIn this session, we’ll explore and see live demos of some solutions to unify the operational and analytical estates, eliminating data silos. We’ll start with stream processing using Kafka Streams, Apache Flink®, and SQL implementations, then cover integration of relational databases with real-time analytics databases such as Apache Pinot® and ClickHouse. Finally, we’ll dive into modern approaches like Apache Iceberg® with Tableflow, which simplifies data preparation by seamlessly representing Kafka topics and associated schemas as Iceberg or Delta tables in a few clicks. While there's no single right answer to this problem, as responsible system builders, we must understand our options and trade-offs to build robust architectures.

Summary In this episode of the Data Engineering Podcast Jeremy Edberg, CEO of DBOS, about durable execution and its impact on designing and implementing business logic for data systems. Jeremy explains how DBOS's serverless platform and orchestrator provide local resilience and reduce operational overhead, ensuring exactly-once execution in distributed systems through the use of the Transact library. He discusses the importance of version management in long-running workflows and how DBOS simplifies system design by reducing infrastructure needs like queues and CI pipelines, making it beneficial for data pipelines, AI workloads, and agentic AI.

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data managementData migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.Your host is Tobias Macey and today I'm interviewing Jeremy Edberg about durable execution and how it influences the design and implementation of business logicInterview IntroductionHow did you get involved in the area of data management?Can you describe what DBOS is and the story behind it?What is durable execution?What are some of the notable ways that inclusion of durable execution in an application architecture changes the ways that the rest of the application is implemented? (e.g. error handling, logic flow, etc.)Many data pipelines involve complex, multi-step workflows. How does DBOS simplify the creation and management of resilient data pipelines? How does durable execution impact the operational complexity of data management systems?One of the complexities in durable execution is managing code/data changes to workflows while existing executions are still processing. What are some of the useful patterns for addressing that challenge and how does DBOS help?Can you describe how DBOS is architected?How have the design and goals of the system changed since you first started working on it?What are the characteristics of Postgres that make it suitable for the persistence mechanism of DBOS?What are the guiding principles that you rely on to determine the boundaries between the open source and commercial elements of DBOS?What are the most interesting, innovative, or unexpected ways that you have seen DBOS used?What are the most interesting, unexpected, or challenging lessons that you have learned while working on DBOS?When is DBOS the wrong choice?What do you have planned for the future of DBOS?Contact Info LinkedInParting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?Closing Announcements Thank you for listening! Don't forget to check out our other shows. Podcast.init covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.Links DBOSExactly Once SemanticsTemporalSempahorePostgresDBOS TransactPython Typescript Idempotency KeysAgentic AIState MachineYugabyteDBPodcast EpisodeCockroachDBSupabaseNeonPodcast EpisodeAirflowThe intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Experience the power of AlloyDB Omni, a cutting-edge PostgreSQL-compatible database designed for multicloud and hybrid cloud environments. This session explores how AlloyDB Omni accelerates the development of modern applications, enabling generative AI experiences, efficient vector search, real-time operational analytics, and scalable transactional performance. We’ll also showcase how to run your applications on multiple clouds using Aiven’s seamless managed service, and how to supercharge hybrid cloud deployments with cloud-ready partners.

Time to make generative AI a reality for your application. This session is all about how to build high-performance gen AI applications fast with Cloud SQL for MySQL and PostgreSQL. Learn about Google Cloud’s innovative full-stack solutions that make gen AI app development, deployment, and operations simple and easy – even when deploying high-performance, production-grade applications. We’ll highlight best practices for getting started with Vertex AI, Cloud Run, Google Kubernetes Engine, and Cloud SQL, so that you can focus on gen AI application development from the get-go.

Build resilient, scalable applications that thrive in the face of increasing demands. Cloud SQL offers new features designed to optimize performance, availability, and cost efficiency for MySQL and PostgreSQL databases, managed replica pools, and connection pooling. Learn how to make downtime a thing of the past, implement advanced disaster recovery strategies, and maximize your application’s performance. Join our demo-packed session for a deep dive into these new Cloud SQL capabilities and best practices.

Learn how Database Migration Service can help you modernize your SQL Server databases to unleash the power of cloud databases and open source PostgreSQL! Convert your SQL Server schema and T-SQL code to PostgreSQL dialect with a click of a button in the DMS Conversion Workspace. Some objects could not be fully converted? Gemini can suggest a fix. Not yet familiar with PostgreSQL features? Ask Gemini to teach you how to convert SQL Server features to PostgreSQL equivalent ones. While Gemini is there - ask it to optimize the converted code or add some comments to explain the business logic. Once your database is fully converted and optimized you can migrate the data with minimal downtime using the change data capture powered migration job and complete your migration journey.

AI is revolutionizing observability. Learn about Cloud SQL AI-powered Database Insights and how it can help you optimize your queries and boost database performance. We’ll dive deep into the new Insights capabilities for MySQL, PostgreSQL, and SQL Server, including the Gemini-powered chat agent. Learn how to troubleshoot those tricky database performance issues and get practical tips to improve the performance of your applications.

This talk demonstrates a fashion app that leverages the power of AlloyDB, Google Cloud’s fully managed PostgreSQL-compatible database, to provide users with intelligent recommendations for matching outfits. User-uploaded data of their clothes triggers a styling insight on how to pair the outfit with matching real-time fashion advice. This is enabled through an intuitive contextual search (vector search) powered by AlloyDB and Google’s ScaNN index to deliver faster vector search results, low-latency querying, and response times. While we’re at it, we’ll showcase the power of the AlloyDB columnar engine on joins required by the application to generate style recommendations. To complete the experience, we’ll engage the Vertex AI Gemini API package from Spring and LangChain4j integrations for generative recommendations and a visual representation of the personalized style. This entire application is built on a Java Spring Boot framework and deployed serverlessly on Cloud Run, ensuring scalability and cost efficiency. This talk explores how these technologies work together to create a dynamic and engaging fashion experience.

Newt Global's DMAP revolutionizes Oracle/MS-SQL to PostgreSQL migrations. This automated solution streamlines the entire process, from initial planning to final production deployment. Key advantages are 1. Container-driven parallelization: Dramatically reduces migration timelines by harnessing powerful computing resources. 2. Unmatched speed: For medium complexity databases, DMAP achieves in 12 weeks what other tools take 12 months due to its advanced automation capabilities, including streamlined application and complex code translation.

This Session is hosted by a Google Cloud Next Sponsor.
Visit your registration profile at g.co/cloudnext to opt out of sharing your contact information with the sponsor hosting this session.

PostgreSQL is one of the most popular open source databases for application development.  Furthermore, native support for vector search makes PostgreSQL an excellent choice for generative AI app development too. Learn why Google Cloud is the best place for your PostgreSQL workloads across Cloud SQL, AlloyDB and Spanner database offerings. Also hear from Salesloft about their journey to Google Cloud and how they chose which database service to select for their workloads.

AlloyDB for PostgreSQL is Google Cloud’s high-performance, PostgreSQL-compatible database service for your demanding online transaction processing (OLTP), hybrid transactional and analytical, and generative AI applications. In this session, we’ll cover what’s new with AlloyDB, including critical enterprise features across performance, availability, ease of use, and more. You’ll also learn how customers are quickly migrating to AlloyDB with Database Migration Service and how they’ve successfully scaled their workloads with AlloyDB, saving money while doing so.

Unlock the power of generative AI and data. Join experts from LlamaIndex and Google Cloud databases and learn how to seamlessly integrate LlamaIndex with AlloyDB and Cloud SQL for PostgreSQL, enabling your apps to reason, act on your data, and leverage the performance of Google Cloud. We’ll share real-world examples and code. Discover new possibilities for building advanced gen AI applications.

Migrate PostgreSQL to AlloyDB with Database Migration Service (DMS). This hands-on lab covers setting up Virtual Private Cloud (VPC) peering, configuring a continuous migration job, migrating data, and verifying the results. Learn to propagate live updates to your AlloyDB instance.

If you register for a Learning Center lab, please ensure that you sign up for a Google Cloud Skills Boost account for both your work domain and personal email address. You will need to authenticate your account as well (be sure to check your spam folder!). This will ensure you can arrive and access your labs quickly onsite. You can follow this link to sign up!

Is siloed data hindering your operations? Learn how Nuro consolidated their transactional, relational, and vector data sets on AlloyDB for PostgreSQL, and how they’re now able to do operational analysis, real-time analytics, and business intelligence (BI) reports on the same platform. Join this panel session to discover best practices for unifying vector and relational data, and learn how Nuro is now able to satisfy their self-driven car analytics use cases in a cost-effective way.

This session presents Schnucks’, a midwest grocer’s migration of their E-commerce application from Oracle Database to Cloud SQL PostgreSQL. It will cover challenges such as addressing the complexities of even "simple" schemas, testing data movement possibilities to minimize downtime, and transforming the database tier. Hear about the business impact, including cost savings, increased database-application proximity and the potential for similar future migrations, allowing for direct integration options from Google CloudSQL to Google Gemini AI.

This Session is hosted by a Google Cloud Next Sponsor.
Visit your registration profile at g.co/cloudnext to opt out of sharing your contact information with the sponsor hosting this session.