Faced with the limitations of a legacy, on-prem data stack and scalability bottlenecks in MicroStrategy, Saddle Creek Logistics Services needed a modern solution to handle massive data volumes and accelerate insight delivery. By migrating to a cloud-native architecture powered by Sigma and Databricks, the team achieved significant performance gains and operational efficiency. In this session, Saddle Creek will walk through how they leveraged Databricks’ cloud-native processing engine alongside a unified governance layer through Unity Catalog to streamline and secure downstream analytics in Sigma. Learn how embedded dashboards and near real-time reporting—cutting latency from 9 minutes to just 3 seconds—have empowered data-driven collaboration with external partners and driven a major effort to consolidate over 30,000 reports and objects to under 1,000.

At Kaizen Gaming, data drives our decision-making, but rapid growth exposed inefficiencies in our legacy cloud setup — escalating costs, delayed insights and scalability limits. Operating in 18 countries with 350M daily transactions (1PB+), shared quotas and limited cost transparency hindered efficiency. To address this, we redesigned our cloud architecture with Data Landing Zones, a modular framework that decouples resources, enabling independent scaling and cost accountability. Automation streamlined infrastructure, reduced overhead and enhanced FinOps visibility, while Unity Catalog ensured governance and security. Migration challenges included maintaining stability, managing costs and minimizing latency. A phased approach, Delta Sharing, and DBx Asset Bundles simplified transitions. The result: faster insights, improved cost control and reduced onboarding time, fostering innovation and efficiency. We share our transformation, offering insights for modern cloud optimization.

With hundreds of millions viewing broadcasts from news to sports, Fox relies on a sophisticated and trusted architecture ingesting 100+ data sources, carefully governed to improve UX across products, drive sales and marketing, and ensure KPI tracking. Join Oliver Gomes, VP of Enterprise and Data Platform at Fox, and Prukalpa Sankar of Atlan to learn how true partnership helps their team navigate opportunities from Governance to AI. To govern and democratize their multi-cloud data platform, Fox chose Atlan to make data accessible and understandable for more users than ever before. Their team then used a data product approach to create a shared language using context from sources like Unity Catalog at a single point of access, no matter the underlying technology. Now, Fox is defining an ambitious future for Metadata. With Atlan and Iceberg driving interoperability, their team prepares to build a “control plane”, creating a common system of trust and governance.

In this course, you'll learn concepts and perform labs that showcase workflows using Unity Catalog - Databricks' unified and open governance solution for data and AI. We'll start off with a brief introduction to Unity Catalog, discuss fundamental data governance concepts, and then dive into a variety of topics including using Unity Catalog for data access control, managing external storage and tables, data segregation, and more. Pre-requisites: Beginner familiarity with the Databricks Data Intelligence Platform (selecting clusters, navigating the Workspace, executing notebooks), cloud computing concepts (virtual machines, object storage, etc.), production experience working with data warehouses and data lakes, intermediate experience with basic SQL concepts (select, filter, groupby, join, etc), beginner programming experience with Python (syntax, conditions, loops, functions), beginner programming experience with the Spark DataFrame API (Configure DataFrameReader and DataFrameWriter to read and write data, Express query transformations using DataFrame methods and Column expressions, etc.) Labs: Yes Certification Path: Databricks Certified Data Engineer Associate

In this course, you’ll learn how to orchestrate data pipelines with Lakeflow Jobs (previously Databricks Workflows) and schedule dashboard updates to keep analytics up-to-date. We’ll cover topics like getting started with Lakeflow Jobs, how to use Databricks SQL for on-demand queries, and how to configure and schedule dashboards and alerts to reflect updates to production data pipelines. Pre-requisites: Beginner familiarity with the Databricks Data Intelligence Platform (selecting clusters, navigating the Workspace, executing notebooks), cloud computing concepts (virtual machines, object storage, etc.), production experience working with data warehouses and data lakes, intermediate experience with basic SQL concepts (select, filter, groupby, join, etc), beginner programming experience with Python (syntax, conditions, loops, functions), beginner programming experience with the Spark DataFrame API (Configure DataFrameReader and DataFrameWriter to read and write data, Express query transformations using DataFrame methods and Column expressions, etc.) Labs: No Certification Path: Databricks Certified Data Engineer Associate

Auto Loader is the definitive tool for ingesting data from cloud storage into your lakehouse. In this session, we’ll unveil new features and best practices that simplify every aspect of cloud storage ingestion. We’ll demo out-of-the-box observability for pipeline health and data quality, walk through improvements for schema management, introduce a series of new data formats and unveil recent strides in Auto Loader performance. Along the way, we’ll provide examples and best practices for optimizing cost and performance. Finally, we’ll introduce a preview of what’s coming next — including a REST API for pushing files directly to Delta, a UI for creating cloud storage pipelines and more. Join us to help shape the future of file ingestion on Databricks.

In this course, you’ll learn how to define and schedule data pipelines that incrementally ingest and process data through multiple tables on the Data Intelligence Platform, using Lakeflow Declarative Pipelines in Spark SQL and Python. We’ll cover topics like how to get started with Lakeflow Declarative Pipelines, how Lakeflow Declarative Pipelines tracks data dependencies in data pipelines, how to configure and run data pipelines using the Lakeflow Declarative Pipelines. UI, how to use Python or Spark SQL to define data pipelines that ingest and process data through multiple tables on the Data Intelligence Platform, using Auto Loader and Lakeflow Declarative Pipelines, how to use APPLY CHANGES INTO syntax to process Change Data Capture feeds, and how to review event logs and data artifacts created by pipelines and troubleshoot syntax.By streamlining and automating reliable data ingestion and transformation workflows, this course equips you with the foundational data engineering skills needed to help kickstart AI use cases. Whether you're preparing high-quality training data or enabling real-time AI-driven insights, this course is a key step in advancing your AI journey.Pre-requisites: Beginner familiarity with the Databricks Data Intelligence Platform (selecting clusters, navigating the Workspace, executing notebooks), cloud computing concepts (virtual machines, object storage, etc.), production experience working with data warehouses and data lakes, intermediate experience with basic SQL concepts (select, filter, groupby, join, etc), beginner programming experience with Python (syntax, conditions, loops, functions), beginner programming experience with the Spark DataFrame API (Configure DataFrameReader and DataFrameWriter to read and write data, Express query transformations using DataFrame methods and Column expressions, etc.)Labs: NoCertification Path: Databricks Certified Data Engineer Associate

This course is designed for data professionals who want to explore the data warehousing capabilities of Databricks. Assuming no prior knowledge of Databricks, it provides an introduction to leveraging Databricks as a modern cloud-based data warehousing solution. Learners will explore how use the Databricks Data Intelligence Platform to ingest, transform, govern, and analyze data efficiently. Learners will also explore Genie, an innovative Databricks feature that simplifies data exploration through natural language queries. By the end of this course, participants will be equipped with the foundational skills to implement and optimize a data warehouse using Databricks. Pre-requisites: Basic understanding of SQL and data querying concepts General knowledge of data warehousing concepts, including tables, schemas, and ETL/ELT processes is recommended Some experience with BI and/or data visualization tools is helpful but not required Labs: Yes

In this course, you’ll learn how to have efficient data ingestion with Lakeflow Connect and manage that data. Topics include ingestion with built-in connectors for SaaS applications, databases and file sources, as well as ingestion from cloud object storage, and batch and streaming ingestion. We'll cover the new connector components, setting up the pipeline, validating the source and mapping to the destination for each type of connector. We'll also cover how to ingest data with Batch to Streaming ingestion into Delta tables, using the UI with Auto Loader, automating ETL with Lakeflow Declarative Pipelines or using the API.This will prepare you to deliver the high-quality, timely data required for AI-driven applications by enabling scalable, reliable, and real-time data ingestion pipelines. Whether you're supporting ML model training or powering real-time AI insights, these ingestion workflows form a critical foundation for successful AI implementation.Pre-requisites: Beginner familiarity with the Databricks Data Intelligence Platform (selecting clusters, navigating the Workspace, executing notebooks), cloud computing concepts (virtual machines, object storage, etc.), production experience working with data warehouses and data lakes, intermediate experience with basic SQL concepts (select, filter, groupby, join, etc), beginner programming experience with Python (syntax, conditions, loops, functions), beginner programming experience with the Spark DataFrame API (Configure DataFrameReader and DataFrameWriter to read and write data, Express query transformations using DataFrame methods and Column expressions, etc.Labs: NoCertification Path: Databricks Certified Data Engineer Associate

The only certainty in life is that the pendulum will always swing. Recently, the pendulum has been swinging towards repatriation. However, the infrastructure needed to build and operate AI systems using Python in a sovereign (even air-gapped) environment has changed since the shift towards the cloud. This talk will introduce the infrastructure you need to build and deploy Python applications for AI - from data processing, to model training and LLM fine-tuning at scale to inference at scale. We will focus on open-source infrastructure including: a Python library server (Pypi, Conda, etc) and avoiding supply chain attacks a container registry that works at scale a S3 storage layer a database server with a vector index

In this session, Learn how to seamlessly integrate powerful GenAI platform features into your apps—without the complexity of managing infrastructure. This session explores how to combine cloud-hosted AI models with MongoDB to enable smarter, scalable AI experiences. You’ll discover streamlined integration patterns, cost-efficient deployment tips, and practical prompt-engineering techniques to take your GenAI project from prototype to production with ease on DigitalOcean GenAI platform.

Supported by Our Partners •⁠ Statsig ⁠ — ⁠ The unified platform for flags, analytics, experiments, and more. •⁠ Sinch⁠ — Connect with customers at every step of their journey. •⁠ Modal⁠ — The cloud platform for building AI applications. — How has Microsoft changed since its founding in 1975, especially in how it builds tools for developers? In this episode of The Pragmatic Engineer, I sit down with Scott Guthrie, Executive Vice President of Cloud and AI at Microsoft. Scott has been with the company for 28 years. He built the first prototype of ASP.NET, led the Windows Phone team, led up Azure, and helped shape many of Microsoft’s most important developer platforms. We talk about Microsoft’s journey from building early dev tools to becoming a top cloud provider—and how it actively worked to win back and grow its developer base. In this episode, we cover: • Microsoft’s early years building developer tools  • Why Visual Basic faced resistance from devs back in the day: even though it simplified development at the time • How .NET helped bring a new generation of server-side developers into Microsoft’s ecosystem • Why Windows Phone didn’t succeed  • The 90s Microsoft dev stack: docs, debuggers, and more • How Microsoft Azure went from being the #7 cloud provider to the #2 spot today • Why Microsoft created VS Code • How VS Code and open source led to the acquisition of GitHub • What Scott’s excited about in the future of developer tools and AI • And much more! — Timestamps (00:00) Intro (02:25) Microsoft’s early years building developer tools (06:15) How Microsoft’s developer tools helped Windows succeed (08:00) Microsoft’s first tools were built to allow less technically savvy people to build things (11:00) A case for embracing the technology that’s coming (14:11) Why Microsoft built Visual Studio and .NET (19:54) Steve Ballmer’s speech about .NET (22:04) The origins of C# and Anders Hejlsberg’s impact on Microsoft  (25:29) The 90’s Microsoft stack, including documentation, debuggers, and more (30:17) How productivity has changed over the past 10 years  (32:50) Why Gergely was a fan of Windows Phone—and Scott’s thoughts on why it didn’t last (36:43) Lessons from working on (and fixing)  Azure under Satya Nadella  (42:50) Codeplex and the acquisition of GitHub (48:52) 2014: Three bold projects to win the hearts of developers (55:40) What Scott’s excited about in new developer tools and cloud computing  (59:50) Why Scott thinks AI will enhance productivity but create more engineering jobs — The Pragmatic Engineer deepdives relevant for this episode: • Microsoft is dogfooding AI dev tools’ future • Microsoft’s developer tools roots • Why are Cloud Development Environments spiking in popularity, now? • Engineering career paths at Big Tech and scaleups • How Linux is built with Greg Kroah-Hartman — See the transcript and other references from the episode at ⁠⁠https://newsletter.pragmaticengineer.com/podcast⁠⁠ — Production and marketing by ⁠⁠⁠⁠⁠⁠⁠⁠https://penname.co/⁠⁠⁠⁠⁠⁠⁠⁠. For inquiries about sponsoring the podcast, email [email protected].

Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe

This session will share proven enterprise architecture best practices for augmenting Snowflake with data virtualization to deliver real-time insights. We'll explore how to address latency-sensitive use cases—such as month-end financial reconciliations—while ensuring data security and supporting cloud migration using Denodo. Attendees will learn how the combination of Snowflake and Denodo enables scalable, low-latency analytics across highly customized and distributed data environments.

Enterprises that possess high-quality data and governance and attest to the trust worthiness of their data among stakeholders have doubled the return on investment (ROI) from their AI. Learn how leading organizations today are implementing an open and trusted data foundation to secure and maximize the value of both their structured and unstructured data, accessing siloed data across hybrid cloud, cost-optimizing growing data workloads, and preparing and delivering high-quality, governed data for AI with a new approach to a data fabric architecture.

D&A leaders scaling generative AI training and inference must navigate new technologies including GPUs and AI processors, both on-premises and in the cloud. In this session, we provide a pragmatic framework to map GenAI model sizes to infrastructures across on-premises and cloud environments.

CDAOs and AI leaders are grappling with two crucial questions: 1. What public cloud provider should we choose for AI and GenAI initiatives, and 2. how do we assemble the right cloud architecture to scale and deploy AI more effectively?
This session compares public cloud AI and Generative AI architectures from AWS, Azure and GCP and provides insights on their points of differentiation.

Managing cloud costs requires accurate resource tagging, but maintaining completeness and accuracy is a challenge. In this session, Mitchell Ertle (Senior Partner Solutions Architect) and Josue Bogran (Data & AI Architect) demonstrate how Sigma and Databricks combine to streamline FinOps and resource management with AI-driven cost attribution and workflow automation.

Through a practical demonstration, you'll see:

Identify and classify untagged Databricks pipelines with a cost attribution app Use GenAI from Databricks to suggest tags with human-in-the-loop approval Enable bidirectional data flow between Sigma and Databricks for real-time updates Automate workflows with Sigma’s actions framework Ensure security and governance by inheriting Unity Catalog permissions Discover why this combination is powerful—Sigma provides intuitive application building while Databricks delivers computation, AI/ML capabilities, and data storage. These platforms create solutions business users can interact with directly, without technical expertise.

Whether in data engineering, finance, or operations, learn how Sigma + Databricks can automate workflows, optimize costs, and drive business impact.

➡️ Learn more about Data Apps: https://www.sigmacomputing.com/product/data-applications?utm_source=youtube&utm_medium=organic&utm_campaign=data_apps_conference&utm_content=pp_data_apps

➡️ Sign up for your free trial: https://www.sigmacomputing.com/go/free-trial?utm_source=youtube&utm_medium=video&utm_campaign=free_trial&utm_content=free_trial

sigma #sigmacomputing #dataanalytics #dataanalysis #businessintelligence #cloudcomputing #clouddata #datacloud #datastructures #datadriven #datadrivendecisionmaking #datadriveninsights #businessdecisions #datadrivendecisions #embeddedanalytics #cloudcomputing #SigmaAI #AI #AIdataanalytics #AIdataanalysis #GPT #dataprivacy #python #dataintelligence #moderndataarchitecture

The enterprise software landscape is at a pivotal turning point. For decades, organizations have been trapped in a cycle of siloed applications—first in on-premise data centers, and then repackaged as cloud SaaS solutions. While infrastructure has become more flexible, scalable, and cost-effective, the applications running on top remain frustratingly rigid, expensive, and disconnected.

In this session, Mike Palmer (CEO of Sigma) and Carter Busse (CIO of Workato) discuss the shift from "best-of-breed" point solutions to an "end-to-end" approach powered by data apps. They'll explore:

Why traditional SaaS applications force organizations to adapt their workflows to software limitations rather than the other way around How the centralization of data in cloud warehouses creates the foundation for building custom, integrated workflows Real-world examples of organizations replacing expensive, disconnected tools with purpose-built data apps The future of enterprise software, including predictions on how AI will reshape application development and data accessibility Practical strategies for starting your data apps journey without creating new technology sprawl Learn how forward-thinking organizations are using data apps to create workflows that better match their business needs, increase decision-making velocity, boost accuracy, and dramatically reduce software costs—all while maintaining enterprise-grade governance and security.

➡️ Learn more about Data Apps: https://www.sigmacomputing.com/product/data-applications?utm_source=youtube&utm_medium=organic&utm_campaign=data_apps_conference&utm_content=pp_data_apps

➡️ Sign up for your free trial: https://www.sigmacomputing.com/go/free-trial?utm_source=youtube&utm_medium=video&utm_campaign=free_trial&utm_content=free_trial

sigma #sigmacomputing #dataanalytics #dataanalysis #businessintelligence #cloudcomputing #clouddata #datacloud #datastructures #datadriven #datadrivendecisionmaking #datadriveninsights #businessdecisions #datadrivendecisions #embeddedanalytics #cloudcomputing #SigmaAI #AI #AIdataanalytics #AIdataanalysis #GPT #dataprivacy #python #dataintelligence #moderndataarchitecture

Sales forecasting and demand planning are critical business processes, but most organizations still rely on spreadsheets—leading to version control issues, fragmented approvals, and lack of historical tracking. Traditional BI tools struggle to solve these problems because they don’t allow for cell-level edits, inline comments, and structured approval workflows in a governed way.

In this session, Ian Reed will demonstrate how to:

Enable real-time forecasting by replacing manual spreadsheets with a structured, cloud-based data app Allow for cell-level edits and inline commentary so teams can capture assumptions behind forecast changes Implement an automated approval workflow with proper governance Integrate seamlessly with live data sources for continuous updates and visibility into actual vs. forecasted performance Track historical changes and maintain audit trails of all modifications

With Sigma, demand planning is no longer a fragmented, error-prone process—it’s a seamless, governed workflow that scales with business growth. Join this session for a demo and a step-by-step walkthrough of how this app was built, proving that anyone can create a highly customizable, enterprise-grade demand planning system in Sigma without deep technical expertise.

➡️ Learn more about Data Apps: https://www.sigmacomputing.com/product/data-applications?utm_source=youtube&utm_medium=organic&utm_campaign=data_apps_conference&utm_content=pp_data_apps

➡️ Sign up for your free trial: https://www.sigmacomputing.com/go/free-trial?utm_source=youtube&utm_medium=video&utm_campaign=free_trial&utm_content=free_trial

sigma #sigmacomputing #dataanalytics #dataanalysis #businessintelligence #cloudcomputing #clouddata #datacloud #datastructures #datadriven #datadrivendecisionmaking #datadriveninsights #businessdecisions #datadrivendecisions #embeddedanalytics #cloudcomputing #SigmaAI #AI #AIdataanalytics #AIdataanalysis #GPT #dataprivacy #python #dataintelligence #moderndataarchitecture

Traditional analytics platforms help you understand what happened, but what about acting on those insights? Sigma Data Apps bridges this gap, allowing you to create purpose-built applications that combine live data, human inputs, and automated workflows—all connected directly to your cloud data warehouse.

In this kickoff, the Sigma team demonstrates how the platform has evolved beyond dashboards to enable true end-to-end workflows:

Learn how Writeback allows you to collect and update data directly in your cloud warehouse See how the Actions Framework automates processes with a single click Explore application layouts with modals, popovers, and tab containers to create intuitive user experiences Discover how to integrate AI models from your data warehouse for advanced analytics

Whether you're looking to replace expensive departmental software, enhance spreadsheet-based processes, or create entirely new workflows, this session will show you how Sigma's Data Apps can help you build solutions that match your exact business needs—in days, not months, without writing a single line of code.

➡️ Learn more about Data Apps: https://www.sigmacomputing.com/product/data-applications?utm_source=youtube&utm_medium=organic&utm_campaign=data_apps_conference&utm_content=pp_data_apps

➡️ Sign up for your free trial: https://www.sigmacomputing.com/go/free-trial?utm_source=youtube&utm_medium=video&utm_campaign=free_trial&utm_content=free_trial

sigma #sigmacomputing #dataanalytics #dataanalysis #businessintelligence #cloudcomputing #clouddata #datacloud #datastructures #datadriven #datadrivendecisionmaking #datadriveninsights #businessdecisions #datadrivendecisions #embeddedanalytics #cloudcomputing #SigmaAI #AI #AIdataanalytics #AIdataanalysis #GPT #dataprivacy #python #dataintelligence #moderndataarchitecture