Behind any good DataOps within a Modern Data Stack (MDS) architecture is a solid DevOps design! This is particularly pressing when building an MDS solution at scale, as reliability, quality and availability of data requires a very high degree of process automation while remaining fast, agile and resilient to change when addressing business needs.

While DevOps in Data Engineering is nothing new - for a broad-spectrum solution that includes data warehouse, BI, etc seemed either a bit out of reach due to overall complexity and cost - or simply overlooked due to perceived issues around scaling often attributed to the challenges of automation in CI/CD processes. However, this has been fast changing with tools such as dbt having super cool features which allow a very high degree of autonomy in the CI/CD processes with relative ease, with flexible and cutting edge features around pre-commits, Slim CI, etc.

In this session, Datatonic covers the challenges around building and deploying enterprise-grade MDS solutions for analytics at scale and how they have used dbt to address those - especially around near-complete autonomy to the CI/CD processes!

Speaker: Ash Sultan, Lead Data Architect, Datatonic

Register for Coalesce at https://coalesce.getdbt.com

In this session, Alex Higgs unveils the potential of Data Vault 2.0, an often overlooked but powerful data warehousing method. Discover how it offers scalability, agility, and flexibility to your data solutions.

Key Highlights: - Explore the origins and essence of Data Vault 2.0 - Learn how Data Vault 2.0 streamlines big data solutions for scalability. - See how it integrates with dbt via AutomateDV for faster time to value. - Understand how AutomateDV simplifies Data Vault 2.0 data warehouses, freeing data teams from intricate SQL.

Speaker: Alex Higgs, Senior Consultant Data Engineer, Datavault

Register for Coalesce at https://coalesce.getdbt.com

Every day, Trade Me gets 1.5 million new listings and 20 million listing views. With all that data comes the difficulty of managing a complex data ecosystem. This got the Trade Me team thinking: "Which problems are we trying to solve? How can we increase speed to customer value?" Using this framework, the team developed a new mission statement: "To build a data warehouse that analysts love to use." In this session, Trade Me shares exactly how they achieved that vision, with a focus on planning, data operating models, and database architecture.

Speaker: Lance Witheridge, Data Modernisation Lead, Trade Me

Register for Coalesce at https://coalesce.getdbt.com

dbt packages are libraries for dbt. Packages can produce information about best practice for your dbt project (ex: dbt project evaluator) and cloud warehouse cost overviews. Unfortunately, all theses KPIs are stored in your data warehouse and it can be painful and expensive to create data visualization dashboards. This application build automatically dashboards from dbt packages that you are using. You just need to parameter your dbt Cloud API key - that's it! In this session, you'll learn how.

Speaker: Adrien Boutreau, Head of Analytics Engineers , Infinite Lambda

Register for Coalesce at https://coalesce.getdbt.com

Over the past four years, the data team at EQT has leveraged dbt and Snowflake to create a myriad of data products across the company. With a rapidly growing organization and increased demands for timely and accurate data, their immense monolithic dbt repository has become challenging to maintain. Learn about the best practices they are adopting to keep the platform in shape and scale with the business.

Speaker: Erik Lehto, Senior analytics engineer, EQT

Register for Coalesce at https://coalesce.getdbt.com

Traditional data warehouses excel at churning through terabytes of data for historical analysis. But for real-time, business-critical use cases, traditional data warehouses can’t produce results fast enough—and they still rack up a huge bill in the process.

So when Ramp’s data engineering team needed to serve complex analytics queries on the critical path of their production application, they knew they needed a new tool for the job. Enter Materialize, the first operational data warehouse. Like a traditional data warehouse, Materialize centralizes the data from all of a business’s production systems, from application databases to SaaS tools. But unlike a traditional data warehouse, Materialize enables taking immediate and automatic action when that data changes. Queries that once took hours or minutes to run are up-to-date in Materialize within seconds.

This talk presents how Ramp is unlocking new real-time use cases using Materialize as their operational data warehouse. The best part? The team still uses dbt for data modeling and deployment management, just like they are able to with their traditional batch workloads.

Speakers: Nikhil Benesch, CTO, Materialize; Ryan Delgado, Staff Software Engineer, Data Platform, Ramp

Register for Coalesce at https://coalesce.getdbt.com

It’s 2023, why are software engineers still breaking analytics reporting? We’ve all been there, being alerted by an analyst or C-level stakeholders, saying “this report is broken”, only to spend hours determining that an engineer deleted a column on the source database that is now breaking your pipeline and reporting.

At Xometry, the data engineering team wanted to fix this problem at its root and give the engineering teams a clear and repeatable process that allowed them to be the owners of their own database data. Xometry named the process DPICT (data pipeline contract) and built several internal tools that integrated seamlessly with their developer’s microservice toolsets.

Their software engineers mostly build their database microservices using Postgres, and bring in the data using Fivetran. Using that as the baseline, the team created a set of tools that would allow the engineers to quickly build the staging layer of their database in the data warehouse (Snowflake), but also alert them of the consequences of removing a table or column in downstream reporting.

In this talk, Jisan shares the nuts and bolts of the designed solution and process that allowed the team to onboard 13 different microservices seamlessly, working with multiple domains and dozens of developers. The process also helped software engineers to own their own data and realize their impact. The team has saved hundreds hours of data engineering time and resources not having to chase down what changed upstream to break data. Overall, this process has helped to bring transparency to the whole data ecosystem.

Speaker: Jisan Zaman, Data Engineering Manager, Xometry

Register for Coalesce at https://coalesce.getdbt.com

The Guild team migrated to Snowflake and dbt for their data warehousing needs and immediately saw the benefits of standardizing model structure, DRYer logic, data lineage ,and automated testing on Pull Requests.

But leveraging dbt didn’t solve everything. Pain points around maintaining model logic, handling historical data, and integrating data from over 60 source systems meant that analysts still struggled to provide a unified view of the business. The team knew that they needed to level up their processes and modeling again, and chose to adopt Data Vault (DV).

Brandon and Rebecca take you behind the scenes of this decision to explain the benefits of Data Vault. They highlight DV’s ability to handle complex data integration requirements while remaining agile and demonstrate that it complements other modern data concepts like domain-driven design and data mesh.

Attendees learn what Data Vault is, when it can be a key component of a successful data strategy, and instances where it’s not the right fit. Walk away with practical tips to successfully transition based on a real-world implementation.

Guild transformed their data warehouse; you can too!

Speakers: Brandon Taylor, Senior Data Architect, Guild; Rebecca Di Bari Staff Data Engineer , Guild

Register for Coalesce at https://coalesce.getdbt.com

Learn how ClickUp uses dbt, dbt packages, and Snowflake to save on storage and compute costs using Slim CI and how they empower a data warehouse centric culture across Sales, Marketing, Product Growth, Finance, and RevOps all while maintaining one monolithic dbt build job.

Speaker: Michael Revelo, Data Platform Lead , ClickUp

Register for Coalesce at https://coalesce.getdbt.com

Performance is a crucial factor in delivering timely and accurate data to organizations. However, debugging the performance of dbt models can be a challenge, as most resources available focus on legacy databases or tips for specific data engines that do not translate to modern data platforms.

In this talk, Juan Manuel Perafan focuses on optimizing performance for dbt users, without focusing on any specific data warehouse. He explores the commonalities across most data warehouses and provides practical tips and strategies for improving the performance of dbt models. From query optimization to materialization strategies.

Whether you're new to dbt or a seasoned user, this talk provides valuable insights and best practices for improving the performance of your dbt models.

Speaker: Juan Manuel Perafan, Analytics Engineer, Xebia

Register for Coalesce at https://coalesce.getdbt.com

Business processes are the foundation of any organization, directing entities towards achieving specific outcomes. These processes can be simple or complex and may take days or even months to complete. Insights into business processes can be determined through three categories: occurrence, volume, and velocity.

In this presentation, Routable’s Director of Data & Analytics discusses the technical and process complexities involved in creating data models in a data warehouse using dbt Cloud. The session also provides tips to make the process easier and explains how to expose this data to users using Looker.

Speaker: Jason Hodson, Director, Data & Analytics, Routable

Register for Coalesce at https://coalesce.getdbt.com

Bluecore works with the largest retail brands around the world to engage shoppers and keep them coming back. In this talk, you’ll learn how the team at Bluecore went about creating, scaling, and maturing an analytics data warehouse in BigQuery to orchestrate 10,000+ models every 30 minutes without bankrupting the company.

Speakers: Adam Whitaker, Analytics Lead, bluecore; Nicole Dallar-Malburg, Analytics Engineer, Bluecore

Register for Coalecse at https://coalesce.getdbt.com/