talk-data.com talk-data.com

Databricks Summit talk 2025-06-10 at 20:50

Lakeflow Declarative Pipelines Integrations and Interoperability: Get Data From — and to — Anywhere

Event: Data + AI Summit 2025

Speakers

Ryan Nienhuis

Sr. Staff Product Manager · Databricks

Topics

API Azure Cosmos Data Lakehouse Delta ETL/ELT Kafka MongoDB Python Spark

Description

This session is repeated.In this session, you will learn how to integrate Lakeflow Declarative Pipelines with external systems in order to ingest and send data virtually anywhere. Lakeflow Declarative Pipelines is most often used in ingestion and ETL into the Lakehouse. New Lakeflow Declarative Pipelines capabilities like the Lakeflow Declarative Pipelines Sinks API and added support for Python Data Source and ForEachBatch have opened up Lakeflow Declarative Pipelines to support almost any integration. This includes popular Apache Spark™ integrations like JDBC, Kafka, External and managed Delta tables, Azure CosmosDB, MongoDB and more.