Today’s organizations need faster, more reliable insights — but metric sprawl and inconsistent KPIs make that difficult. In this session, you’ll learn how Unity Catalog Metrics helps unify business semantics across your organization. Define your KPIs once, apply enterprise-grade governance with fine-grained access controls, auditing and lineage, and use them across any Databricks tool — from AI/BI Dashboards and Genie to notebooks and Lakeflow. You’ll learn how to eliminate metric chaos by centrally defining and governing metrics with Unity Catalog. You’ll walk away with strategies to boost trust through built-in governance and empower every team — regardless of technical skill — to work from the same certified metrics.

With hundreds of millions viewing broadcasts from news to sports, Fox relies on a sophisticated and trusted architecture ingesting 100+ data sources, carefully governed to improve UX across products, drive sales and marketing, and ensure KPI tracking. Join Oliver Gomes, VP of Enterprise and Data Platform at Fox, and Prukalpa Sankar of Atlan to learn how true partnership helps their team navigate opportunities from Governance to AI. To govern and democratize their multi-cloud data platform, Fox chose Atlan to make data accessible and understandable for more users than ever before. Their team then used a data product approach to create a shared language using context from sources like Unity Catalog at a single point of access, no matter the underlying technology. Now, Fox is defining an ambitious future for Metadata. With Atlan and Iceberg driving interoperability, their team prepares to build a “control plane”, creating a common system of trust and governance.

The Houston Food Bank (HFB) is the largest food bank in the country, serving 18 southeast Texas counties and distributing over 120 million meals in the last fiscal year through our network of 1,600+ community partners to the 1 million-plus food-insecure persons in the region.

Over the last 2+ years, HFB has leveraged dbt to modernize our data stack. Initially working with dbt Core, our data team's engineers centralized, streamlined, and automated data pipelines to provide critical KPIs to HFB Leadership. Fast-forward to today, our data team of 10, which includes engineers, analysts, and other specialists, uses dbt Cloud to manage all data transformations in our data warehouse, which now supports 30+ integrations and 70+ reports that deliver 180+ metrics to stakeholders across the organization. This organizational transformation has saved countless hours for our staff, improved organizational trust in data significantly by identifying and managing sources of truth, and delivered key insights to stakeholders across our entire organization.

A handful of examples include: - Identifying corporate donor opportunities by mining donor and volunteer data - Increasing the number of opportunities for federal and grant-based funding by being able to generate metrics across an ever-increasing number of data sources - Assessing the efficiency of school-based programs by analyzing the proportion and volume of students served to the food-insecure population of that school

HFB is committed to being a data leader in the food banking space, and we’re hoping our journey using dbt can inspire other non-profits to leverage the platform as well.

Speakers: Erwin Kristel Data Analyst Houston Food Bank

Benjamin Herndon-Miller Data Engineer Houston Food Bank

Susan Quiros Data Analyst II Houston Food Bank

Read the blog to learn about the latest dbt Cloud features announced at Coalesce, designed to help organizations embrace analytics best practices at scale https://www.getdbt.com/blog/coalesce-2024-product-announcements

dbt packages are libraries for dbt. Packages can produce information about best practice for your dbt project (ex: dbt project evaluator) and cloud warehouse cost overviews. Unfortunately, all theses KPIs are stored in your data warehouse and it can be painful and expensive to create data visualization dashboards. This application build automatically dashboards from dbt packages that you are using. You just need to parameter your dbt Cloud API key - that's it! In this session, you'll learn how.

Speaker: Adrien Boutreau, Head of Analytics Engineers , Infinite Lambda

Register for Coalesce at https://coalesce.getdbt.com

As the backbone of Australia’s supply chain, the Australia Rail Track Corporation (ARTC) plays a vital role in the management and monitoring of goods transportation across 8,500km of its rail network throughout Australia. ARTC provides weighbridges along their track which read train weights as they pass at speeds of up to 60 kilometers an hour. This information is highly valuable and is required both by ARTC and their customers to provide accurate haulage weight details, analyze technical equipment, and help ensure wagons have been loaded correctly.

A total of 750 trains run across a network of 8500 km in a day and generate real-time data at approximately 50 sensor platforms. With the help of structured streaming and Delta Lake, ARTC was able to analyze and store:

• Precise train location
• Weight of the train in real-time
• Train crossing time to the second level
• Train speed, temperature, sound frequency, and friction
• Train schedule lookups

Once all the IoT data has been pulled together from an IoT event hub, it is processed in real-time using structured streaming and stored in Delta Lake. To understand the train GPS location, API calls are then made per minute per train from the Lakehouse. API calls are made in real-time to another scheduling system to lookup customer info. Once the processed/enriched data is stored in Delta Lake, an API layer was also created on top of it to expose this data to all consumers.

The outcome: increased transparency on weight data as it is now made available to customers; we built a digital data ecosystem that now ARTC’s customers use to meet their KPIs/ planning; the ability to determine temporary speed restrictions across the network to improve train scheduling accuracy and also schedule network maintenance based on train schedules and speed.

Talk by: Deepak Sekar and Harsh Mishra

Connect with us: Website: https://databricks.com Twitter: https://twitter.com/databricks LinkedIn: https://www.linkedin.com/company/databricks Instagram: https://www.instagram.com/databricksinc Facebook: https://www.facebook.com/databricksinc

ABOUT THE TALK: In this talk, Felipe Hoffa and Daniel Myers present an honest take of their wildly different approaches to Developer Relations and how both have been critical in building Snowflake's world-class developer community and ecosystem from the ground up. Learn how they define DevRel KPIs & metrics and daily challenges they face and lessons learned along the way. You might even get inspired to become a Developer Advocate after understanding the different ways to engage with the Snowflake community and what's next for Snowflake Developer Relations.

ABOUT THE SPEAKERS: Felipe Hoffa is the Data Cloud Advocate at Snowflake. Previously he worked at Google, as a Developer Advocate on Data Analytics for BigQuery, after joining as a Software Engineer. He moved from Chile to San Francisco in 2011. His goal is to inspire developers and data scientists around the world to analyze and understand their data in ways they never could before.

Daniel Myers is in Developer Relations and previously held roles at different companies, including Google, Cisco, and Fujitsu. In addition, he led and founded multiple startups.

ABOUT DATA COUNCIL: Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers.

Make sure to subscribe to our channel for the most up-to-date talks from technical professionals on data related topics including data infrastructure, data engineering, ML systems, analytics and AI from top startups and tech companies.

FOLLOW DATA COUNCIL: Twitter: https://twitter.com/DataCouncilAI LinkedIn: https://www.linkedin.com/company/datacouncil-ai/

For shop owners to maximize revenue, they need to ensure that the right products are available on the right shelf at the right time. So, how does one assort the right mix of products to make max profit & reduce inventory pressure? Today, these decisions are led by human knowledge of trends & inputs from salespeople. This is error prone and cannot scale with a growing product assortment & varying demand patterns. Mindtree has analyzed this problem and built a cloud-based AI/ML solution that provides contextual, real-time insights and optimizes inventory management. In this presentation, you will hear our solution approach to help global CPG organization, promote new products, increase demand across their product offerings and drive impactful insights. You will also learn about the technical solution architecture, orchestration of product and KPI generation using Databricks, AI/ML models, heterogenous cloud platform options for deployment and rollout, scale-up and scale-out options.

Connect with us: Website: https://databricks.com Facebook: https://www.facebook.com/databricksinc Twitter: https://twitter.com/databricks LinkedIn: https://www.linkedin.com/company/data... Instagram: https://www.instagram.com/databricksinc/

In this session, we'll go over several use-cases and describe the process of improving our spark structured streaming application micro-batch time from ~55 to ~30 seconds in several steps.

Our app is processing ~ 700 MB/s of compressed data, it has very strict KPIs, and it is using several technologies and frameworks such as: Spark 3.1, Kafka, Azure Blob Storage, AKS and Java 11.

We'll share our work and experience in those fields, and go over a few tips to create better Spark structured streaming applications.

The main areas that will be discussed are: Spark Configuration changes, code optimizations and the implementation of the Spark custom data source.

Connect with us: Website: https://databricks.com Facebook: https://www.facebook.com/databricksinc Twitter: https://twitter.com/databricks LinkedIn: https://www.linkedin.com/company/data... Instagram: https://www.instagram.com/databricksinc/

Accelerate the path from data to decisions with the the Tellius AI-driven Decision Intelligence platform powered by Databricks Delta Lake. Empower business users and data teams to analyze data residing in the Delta Lake to understand what is happening in their business, uncover the reasons why metrics change, and get recommendations on how to impact outcomes. Learn how organizations derive value from Delta Lakehouse with a modern analytics experience that unifies guided insights, natural language search, and automated machine learning to speed up data-driven decision making at cloud scale.

In this session, we will showcase how customers: - Discover changes in KPIs and investigate the reasons why metrics change with AI-powered automated analysis - Empower business users and data analysts to iteratively explore data to identify trend drivers, uncover new customer segments, and surface hidden patterns in data - Simplify and speed-up analysis from massive datasets on Databrick Delta lake

Connect with us: Website: https://databricks.com Facebook: https://www.facebook.com/databricksinc Twitter: https://twitter.com/databricks LinkedIn: https://www.linkedin.com/company/data... Instagram: https://www.instagram.com/databricksinc/

In this presentation we will talk about how we will use available NER based sensitive data detection methods, automated record of activity processing on top of spark and feast for collaborative intelligent analytics & governed data sharing. Information sharing is the key to successful business outcomes but it's complicated by sensitive information both user centric and business centric.

Our presentation is motivated by the need to share key KPIs, outcomes for health screening data collected from various surveys to improve care and assistance. In particular, collaborative information sharing was needed to help with health data management, early detection and prevention of disease KPIs. We will present a framework or an approach we have used for these purposes.

Connect with us: Website: https://databricks.com Facebook: https://www.facebook.com/databricksinc Twitter: https://twitter.com/databricks LinkedIn: https://www.linkedin.com/company/data... Instagram: https://www.instagram.com/databricksinc/