talk-data.com talk-data.com

Databricks Summit talk 2025-06-10 at 17:20

Top Performance and Cost Optimizations for Lakeflow Declarative Pipelines

Description

Lakeflow Declarative Pipelines simplifies pipeline development and management — but how do you optimize for performance and cost? In this session, we’ll explore practical strategies for tuning Lakeflow Declarative Pipelines, including when and how to use autoscaling, Photon and different node types. We'll also cover how to monitor resource usage and decide when serverless is the right choice. You'll learn best practices drawn from real-world customer implementations, along with an overview of the latest performance enhancements available in serverless Lakeflow Declarative Pipelines.