talk-data.com

Azure Data Factory (ADF) is Microsoft’s central orchestration and integration service for reliable and scalable data movement. Combined with Azure Databricks as a powerful platform for data processing and transformation, it enables a flexible and future-proof data engineering architecture. In this session, participants will gain a practical, real-world overview of how to implement complex end-to-end data loading processes with ADF and Databricks - whether on-premises, in the cloud, or in hybrid scenarios. We will cover the full lifecycle of a Data Factory and its integration with Databricks:

· Architecture & best practices for high-performance, scalable, and maintainable pipelines · Source and target connectivity (SQL Server, Azure Storage, APIs, Data Lake, and more) · Databricks integration within ADF pipelines as the central transformation tool · Transformations with Databricks notebooks – from simple cleansing to complex business logic · Parameterization & reusability of pipelines, datasets, and notebook executions · DevOps integration: Building a complete CI/CD pipeline in Azure DevOps or GitHub Actions · Infrastructure as Code with Bicep and Terraform for repeatable, parameterized deployments · Monitoring & error handling – from alerting to automated remediation

The session guides participants through a real-world end-to-end solution based on an actual customer project. The focus is on practical architecture decisions, proven best practices, and common pitfalls - rather than abstract concepts. Participants will leave with a clear, actionable roadmap that enables them to successfully implement similar data integration scenarios with Azure Data Factory, Azure Databricks, and DevOps-based deployment.