This session will explore a few key metrics that can be used to steer your business in the right direction. Retention rate, lifetime value and payback time are crucial for estimating growth, understanding user behavior and determining marketing spend. Those are needed in turn for companies to make informed decisions and drive the business forward so it is important to get them right. While some rules may be specific for each company, this talk will present the work undertaken at Rebtel to calculate these metrics using SQL, and how you could implement similar models quickly.

Speaker: Quentin Coviaux Data Engineer Rebtel

Read the blog to learn about the latest dbt Cloud features announced at Coalesce, designed to help organizations embrace analytics best practices at scale https://www.getdbt.com/blog/coalesce-2024-product-announcements

Explore the latest features added to dbt Core this year, shaped by community feedback and collaboration. Learn how community-driven efforts influenced development and get a sneak peek at what’s coming next for dbt Core. Whether you’re a seasoned user or new to dbt, this session will keep you in the loop on the latest innovations and what’s ahead.

Speaker: Grace Goheen Product Manager dbt Labs

Read the blog to learn about the latest dbt Cloud features announced at Coalesce, designed to help organizations embrace analytics best practices at scale https://www.getdbt.com/blog/coalesce-2024-product-announcements

Over the past 8 years, dbt, Cloud data warehouses and the dbt viewpoint have dramatically changed the workflow for data practitioners, raising the bar on what great data work looks like and altering the nature of the types of problems we focus on day to day. Jason Ganz lived this transition firsthand and now, he believes, we’re on the cusp of another transformation in how data work gets done. Come hear about how new technologies like Generative AI and new workflows like the Analytics Development Lifecycle will transform data work and how to think about that in your own role and career trajectory.

Speaker: Jason Ganz Senior Manager, Developer Experience dbt Labs

Read the blog to learn about the latest dbt Cloud features announced at Coalesce, designed to help organizations embrace analytics best practices at scale https://www.getdbt.com/blog/coalesce-2024-product-announcements

"Look at these beautiful dbt models! Why are we still experiencing the same friction with stakeholders?" This talk from experts at dbt Labs argues that we solved the first stage of building "Transformations" (via the 2024 State of Analytics Engineering report) and now we're now in the second stage: "The Philosophy of Transformations". And all roads lead to "metrics".

Speakers: Erica Louie Sr. Analytics Manager dbt Labs

Andrew Escay Lead Data Analyst dbt Labs

Read the blog to learn about the latest dbt Cloud features announced at Coalesce, designed to help organizations embrace analytics best practices at scale https://www.getdbt.com/blog/coalesce-2024-product-announcements

The dbt Labs 2024 State of Analytics Engineering report highlights that stakeholder data literacy remains a problem in the modern data workplace. Data stakeholders and data professionals can both benefit from learning foundational data literacy concepts that foster their ability to reason about working with data in a business environment. In this talk, an expert from Great Expectations covers two key concepts that they've applied in their own career when framing data fundamentals: “the data supply chain” and “ML in a nutshell.”

Speaker: Rachel House Senior Developer Advocate Great Expectations

Read the blog to learn about the latest dbt Cloud features announced at Coalesce, designed to help organizations embrace analytics best practices at scale https://www.getdbt.com/blog/coalesce-2024-product-announcements

This talk isn't about data quality tests. Why? Well, because there’s no shortage of tools and processes for testing your data, monitoring your data, and alerting your team when the pipeline breaks. Instead, this is a talk about what happens after the alarm goes off.

Data incidents can quickly snowball into much more than simply fixing the issue. Juggling comms, diagnosing the issue, ownership of the failure, your DMs lighting up…we’ve all been there and understand just how stressful it can be.

In this session, Matilda covers a number of tool and product agnostic approaches teams can adopt to improve how they manage data incidents. These are practical steps any data team can follow to improve how they’re resolving, communicating, and learning from their incidents, with the goal of providing more resilient data systems.

Speaker: Matilda Hultgren Data Analyst - Product incident.io

Read the blog to learn about the latest dbt Cloud features announced at Coalesce, designed to help organizations embrace analytics best practices at scale https://www.getdbt.com/blog/coalesce-2024-product-announcements

Community Spotlight Honorees Bruno and Dakota from phData cover various advanced pipelines and when to implement them.

Speakers: Dakota Kelley Sr Solution Architect phData

Bruno Souza de Lima Senior Data Engineer phData

Read the blog to learn about the latest dbt Cloud features announced at Coalesce, designed to help organizations embrace analytics best practices at scale https://www.getdbt.com/blog/coalesce-2024-product-announcements

The Houston Food Bank (HFB) is the largest food bank in the country, serving 18 southeast Texas counties and distributing over 120 million meals in the last fiscal year through our network of 1,600+ community partners to the 1 million-plus food-insecure persons in the region.

Over the last 2+ years, HFB has leveraged dbt to modernize our data stack. Initially working with dbt Core, our data team's engineers centralized, streamlined, and automated data pipelines to provide critical KPIs to HFB Leadership. Fast-forward to today, our data team of 10, which includes engineers, analysts, and other specialists, uses dbt Cloud to manage all data transformations in our data warehouse, which now supports 30+ integrations and 70+ reports that deliver 180+ metrics to stakeholders across the organization. This organizational transformation has saved countless hours for our staff, improved organizational trust in data significantly by identifying and managing sources of truth, and delivered key insights to stakeholders across our entire organization.

A handful of examples include: - Identifying corporate donor opportunities by mining donor and volunteer data - Increasing the number of opportunities for federal and grant-based funding by being able to generate metrics across an ever-increasing number of data sources - Assessing the efficiency of school-based programs by analyzing the proportion and volume of students served to the food-insecure population of that school

HFB is committed to being a data leader in the food banking space, and we’re hoping our journey using dbt can inspire other non-profits to leverage the platform as well.

Speakers: Erwin Kristel Data Analyst Houston Food Bank

Benjamin Herndon-Miller Data Engineer Houston Food Bank

Susan Quiros Data Analyst II Houston Food Bank

Read the blog to learn about the latest dbt Cloud features announced at Coalesce, designed to help organizations embrace analytics best practices at scale https://www.getdbt.com/blog/coalesce-2024-product-announcements

This session is a practical guide to changing how you operate in response to the Cambrian explosion of AI and LLM technologies. In your future, everyone at the company will have access to an LLM with unfettered access to your data warehouse. Do you feel afraid? The Data team at Hex did too. They'll share how they had to change how they worked to adapt, and what data leaders and practitioners need to be thinking about for their own teams.

Speakers: Amanda Fioritto Senior Analytics Engineer Hex Technologies

Erika Pullum Analytics Engineer Hex Technologies

Read the blog to learn about the latest dbt Cloud features announced at Coalesce, designed to help organizations embrace analytics best practices at scale https://www.getdbt.com/blog/coalesce-2024-product-announcements

Since the beginning of 2024, the Warner Brothers Discovery team supporting the CNN data platform has been undergoing an extensive migration project from dbt Core to dbt Cloud. Concurrently, the team is also segmenting their project into multi-project frameworks utilizing dbt Mesh. In this talk, Zachary will review how this transition has simplified data pipelines, improved pipeline performance and data quality, and made data collaboration at scale more seamless.

He'll discuss how dbt Cloud features like the Cloud IDE, automated testing, documentation, and code deployment have enabled the team to standardize on a single developer platform while also managing dependencies effectively. He'll share details on how the automation framework they built using Terraform streamlines dbt project deployments with dbt Cloud to a ""push-button"" process. By leveraging an infrastructure as code experience, they can orchestrate the creation of environment variables, dbt Cloud jobs, Airflow connections, and AWS secrets with a unified approach that ensures consistency and reliability across projects.

Speakers: Mamta Gupta Staff Analytics Engineer Warner Brothers Discovery

Zachary Lancaster Manager, Data Engineering Warner Brothers Discovery

Read the blog to learn about the latest dbt Cloud features announced at Coalesce, designed to help organizations embrace analytics best practices at scale https://www.getdbt.com/blog/coalesce-2024-product-announcements

There are undoubtedly similarities between the disciplines of analytics engineering and DevOps: in fact, dbt was founded with the goal of helping data professionals embrace DevOps principles as part of the data workflow. As the embedded DevOps engineer for a mature analytics engineering function, Katie Claiborne, Founding Analytics Engineer at Duet, observed parallels between analytics-as-code and infrastructure-as-code, particularly tools like Terraform. In this talk, she'll examine how analytics engineering is a means of empowerment for data practitioners and discuss infrastructure engineering as a means of scaling dbt Cloud deployments. Learn about similarities between analytics and infrastructure configuration tools, how to apply the concepts you've learned about analytics engineering towards new disciplines like DevOps, and how to extend engineering principles beyond data transformation and into the world of infrastructure.

Speaker: Katie Claiborne Founding Analytics Engineer Duet

Read the blog to learn about the latest dbt Cloud features announced at Coalesce, designed to help organizations embrace analytics best practices at scale https://www.getdbt.com/blog/coalesce-2024-product-announcements

Carta prioritizes ensuring that their data is accessible only to the right individuals. Previously, they used functional groups to manage data access, but this approach often fell short in perfectly governing granular data sets. Recently, Carta developed a new access management system utilizing dbt lineage and dbt grants. These tools enable them to automatically propagate data access tags defined in dbt sources. This innovative system allows them to confidently ensure that individuals have appropriate access to data.

Speaker: Marco Albuquerque Senior Engineering Manager - Data Engineering Carta

Read the blog to learn about the latest dbt Cloud features announced at Coalesce, designed to help organizations embrace analytics best practices at scale https://www.getdbt.com/blog/coalesce-2024-product-announcements

The data team at SurveyMonkey, the global leader in survey software, oversees heavy data transformation in dbt Cloud — both to power current business-critical projects, and also to migrate legacy workloads. Much of that transformation work is taking raw data — either from legacy databases or their cloud data warehouse (Snowflake) — and making it accessible and useful for downstream users. And to Samiksha Gour, Senior Data Engineering Manager at SurveyMonkey, each of these projects is not considered complete unless the proper checks, monitors, and alerts are in place.

Join Samiksha in this informative session as she walks through how her team uses dbt and their data observability platform Monte Carlo to ensure proper governance, gain efficiencies by eliminating duplicate testing and monitoring, and use data lineage to ensure upstream and downstream continuity for users and stakeholders.

Speaker: Samiksha Gour Senior Data Engineering Manager SurveyMonkey

Read the blog to learn about the latest dbt Cloud features announced at Coalesce, designed to help organizations embrace analytics best practices at scale https://www.getdbt.com/blog/coalesce-2024-product-announcements

Learn how Virgin Media O2 uses dbt Cloud to enhance call center efficiency, personalize customer communications, and accelerate data science workflows. In this session, we will share details about our innovative continuous flow system, developed using best practices from Toyota Kanban, and how it helps reduce operational waste and costs. We will also highlight a number of capabilities within dbt Cloud that support continuous data flows by automating manual tasks.

Read the blog to learn about the latest dbt Cloud features announced at Coalesce, designed to help organizations embrace analytics best practices at scale https://www.getdbt.com/blog/coalesce-2024-product-announcements

Speakers: Arun Kumaravel Senior Analytics Engineer Virgin Media O2

Oliver Burt Lead Analytics Engineer Virgin Media o2

Gordon Curzon Head of Analytics Engineering Virgin Media O2

In today's data-driven world, harnessing the power of data is no longer an option but a necessity for businesses to thrive. For product engineering teams in particular, timely access to accurate and contextual data is crucial for making informed decisions and monitoring success. In this conversation, Aakriti Kaul and Scott Henry, Data Scientists at Cisco, dive into Duo Security’s data modernization journey, bolstered by dbt Cloud and embedded context in data, aimed at empowering product teams with data access and insights to drive innovation.

At the end of this session we hope to leave attendees with the following takeaways: • Understand how an Embedded Data science model creates value across Product, Engineering and Data teams • Learn practical strategies for implementing dbt within product development workflows to accelerate decision making and drive innovation, in partnership with Analytics Engineering teams • Gain insights from real-world case studies of Duo’s Product Data teams that have successfully leveraged dbt to provide access to data and insights for product teams • Gain insights from our organizational experience using dbt to provide product teams with self-service access to contextual datasets

The presentation is designed for data scientists, analytics engineers and other professionals involved in product development who are interested in leveraging data to drive decision making and embedding context within their data workflows. Whether you're new to dbt or looking to optimize your existing data analytics workflows, this session will provide valuable insights and practical strategies for harnessing the power of dbt in partnership with product engineering teams.

Speakers: Aakriti Kaul Data Scientist Duo Security @ Cisco

Scott Henry Data Scientist Duo Security @ Cisco

Read the blog to learn about the latest dbt Cloud features announced at Coalesce, designed to help organizations embrace analytics best practices at scale https://www.getdbt.com/blog/coalesce-2024-product-announcements

At dbt Labs, open source innovation is at the core of all that we do. The movement that is dbt would only happen thanks to you, our strong and vibrant dbt community. Join dbt Labs’ community leaders Grace Goheen, Jeremy Cohen, and Amada Echeverria as we celebrate the community with our annual recognition awards, share the latest innovations in open source, and share tales for the community with an interactive panel.

What does it mean to be successful with AI? Is it validating that it could prove long term value? Finding a way to innovate faster than before? And what role does dbt play in unlocking AI’s potential? dbt Labs COO Brandon Sweeney is joined by Fifth Third Bank and Optum UHG to talk about AI and the role dbt plays in accelerating business growth.

Read the blog to learn about the latest dbt Cloud features announced at Coalesce, designed to help organizations embrace analytics best practices at scale https://www.getdbt.com/blog/coalesce-2024-product-announcements

dbt Labs co-founder and CEO, Tristan Handy, unveils his vision for the analytics development lifecycle, highlighting how our mission to make data and AI more accessible and trustworthy is fueling innovation. Hear from data leaders who have unlocked incredible business value with dbt Cloud at scale, and get an exclusive look at the groundbreaking product features that are launching soon. And remember, what happens in Vegas could change the future of analytics and AI.

Read the blog to learn more about the product announcements: https://www.getdbt.com/blog/coalesce-2024-product-announcements

Speakers: Tristan Handy Founder & CEO dbt Labs

Amy Chen Product Manager dbt Labs

Greg McKeon Staff Product Manager dbt Labs

Roxi Dahlke Product Manager dbt Labs

James Dorado VP, Data Analytics Bilt Rewards

Tobias Humpert Siemens Data Cloud Product Owner Siemens AG

Yannick Misteli Head of Engineering Roche

A recap of Coalesce 2024—from dbt Labs—in Las Vegas. Coalesce is the premier analytics engineering conference for data practitioners looking to hone their craft, connect with peers, and get energized about the future of data.

Summary In this episode of the Data Engineering Podcast Lukas Schulte, co-founder and CEO of SDF, explores the development and capabilities of this fast and expressive SQL transformation tool. From its origins as a solution for addressing data privacy, governance, and quality concerns in modern data management, to its unique features like static analysis and type correctness, Lucas dives into what sets SDF apart from other tools like DBT and SQL Mesh. Tune in for insights on building a business around a developer tool, the importance of community and user experience in the data engineering ecosystem, and plans for future development, including supporting Python models and enhancing execution capabilities. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data managementImagine catching data issues before they snowball into bigger problems. That’s what Datafold’s new Monitors do. With automatic monitoring for cross-database data diffs, schema changes, key metrics, and custom data tests, you can catch discrepancies and anomalies in real time, right at the source. Whether it’s maintaining data integrity or preventing costly mistakes, Datafold Monitors give you the visibility and control you need to keep your entire data stack running smoothly. Want to stop issues before they hit production? Learn more at dataengineeringpodcast.com/datafold today!Your host is Tobias Macey and today I'm interviewing Lukas Schulte about SDF, a fast and expressive SQL transformation tool that understands your schemaInterview IntroductionHow did you get involved in the area of data management?Can you describe what SDF is and the story behind it?What's the story behind the name?What problem are you solving with SDF?dbt has been the dominant player for SQL-based transformations for several years, with other notable competition in the form of SQLMesh. Can you give an overview of the venn diagram for features and functionality across SDF, dbt and SQLMesh?Can you describe the design and implementation of SDF?How have the scope and goals of the project changed since you first started working on it?What does the development experience look like for a team working with SDF?How does that differ between the open and paid versions of the product?What are the features and functionality that SDF offers to address intra- and inter-team collaboration?One of the challenges for any second-mover technology with an established competitor is the adoption/migration path for teams who have already invested in the incumbent (dbt in this case). How are you addressing that barrier for SDF?Beyond the core migration path of the direct functionality of the incumbent product is the amount of tooling and communal knowledge that grows up around that product. How are you thinking about that aspect of the current landscape?What is your governing principle for what capabilities are in the open core and which go in the paid product?What are the most interesting, innovative, or unexpected ways that you have seen SDF used?What are the most interesting, unexpected, or challenging lessons that you have learned while working on SDF?When is SDF the wrong choice?What do you have planned for the future of SDF?Contact Info LinkedInParting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?Links SDFSemantic Data Warehouseasdf-vmdbtSoftware Linting)SQLMeshPodcast EpisodeCoalescePodcast EpisodeApache IcebergPodcast EpisodeDuckDB Podcast Episode SDF Classifiersdbt Semantic Layerdbt expectationsApache DatafusionIbisThe intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA