The session details how our lean central data team has achieved significant output by: Deploying the optimal tools (dbt and Lightdash) that supercharge DevEx Democratizing dbt development effectively Leveraging AI driven development Effectively using Kanban prioritization

Discover how Workday built an AI-powered self-service analytics platform that doesn't compromise on governance using Lightdash. We'll discuss concrete strategies for transforming your data organization from a bottleneck into a competitive advantage, backed by real implementation insights.

Analytics engineers are at a crossroads. Back in 2018, dbt paved the way for for this new kind of data professional, people who had technical ability and could understand business context. But here's the thing: AI is automating traditional tasks like pipeline building and dashboard creation. So then what happens to analytics engineers? They don't disappear - they evolve.

The same skills that made analytics engineers valuable also make them perfect for a new role I'm calling 'Analytics Intelligence Engineers.' Instead of writing SQL, they're writing the context that makes AI actually useful for business users.

In this talk, I'll show you what this evolution looks like day-to-day. We'll explore building semantic layers, crafting AI context, and measuring AI performance - all through real examples using Lightdash. You'll see how the work shifts from data plumbing to data intelligence, and walk away with practical tips for making AI tools more effective in your organization. Whether you're an analytics engineer wondering about your future or a leader planning your data strategy, this session will help you understand where the field is heading and how to get there.

In 2021, semantic layers took the data world by storm—it felt like they came out of nowhere, then everyone was talking about them. Since then, companies have been built (and some have failed) on the promise of the semantic layer, blog posts have debated their rise and fall, and data teams are still left wondering: was it all just hype, or are semantic layers truly the next big thing in data? In this talk, I'll explore the evolution of the semantic layer and answer the burning question: Is it here to stay, or is it just a passing trend?

Speaker: Katie Hindson Head of Product and Data Lightdash

Read the blog to learn about the latest dbt Cloud features announced at Coalesce, designed to help organizations embrace analytics best practices at scale https://www.getdbt.com/blog/coalesce-2024-product-announcements

Eric will share the team's experience with dbt and tell the development story of bringing Workday FinOps cost analysis to cloud engineers and stakeholders. He will describe how the team is using dbt, Trino and Lightdash to build a new data platform that is now a key part of their data-driven business decision process in multiple organizations within Workday. Plus, he'll show how they created a secure, efficient, and scalable platform — through dbt governance features — to drive those successful data projects.

Speakers: Eric Pu Senior Software Engineer Workday

Pattabhi Nanduri FinOps Data Engineer Workday

Read the blog to learn about the latest dbt Cloud features announced at Coalesce, designed to help organizations embrace analytics best practices at scale https://www.getdbt.com/blog/coalesce-2024-product-announcements

Lightdash evaluation, integration, and user experience at JustWatch.

ABOUT THE TALK: Building data tools requires us to not only think about the data team, but also about the people that the data team is serving: business users, or "non-data team people".

This talk will go over how it's super important to consider these two personas when building data tools, but it can also be a bit complicated. We will talk through a few principles we can use to build data products that are great for everyone (not just the data team!)

ABOUT THE SPEAKER: As a product manager with a background in data science, Katie Hindson loves building data products. Currently, she's working at Lightdash, an open-source BI tool that instantly turns your dbt project into a full-stack BI platform. Katie is really interested in the interaction between data teams, their tools, and the rest of the company - because the best data teams are the ones that can help everyone at the company make better decisions, faster.

ABOUT DATA COUNCIL: Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers.

Make sure to subscribe to our channel for the most up-to-date talks from technical professionals on data related topics including data infrastructure, data engineering, ML systems, analytics and AI from top startups and tech companies.

FOLLOW DATA COUNCIL: Twitter: https://twitter.com/DataCouncilAI LinkedIn: https://www.linkedin.com/company/datacouncil-ai/

Summary The market for business intelligence has been going through an evolutionary shift in recent years. One of the driving forces for that change has been the rise of analytics engineering powered by dbt. Lightdash has fully embraced that shift by building an entire open source business intelligence framework that is powered by dbt models. In this episode Oliver Laslett describes why dashboards aren’t sufficient for business analytics, how Lightdash promotes the work that you are already doing in your data warehouse modeling with dbt, and how they are focusing on bridging the divide between data teams and business teams and the requirements that they have for data workflows.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show! Are you bored with writing scripts to move data into SaaS tools like Salesforce, Marketo, or Facebook Ads? Hightouch is the easiest way to sync data into the platforms that your business teams rely on. The data you’re looking for is already in your data warehouse and BI tools. Connect your warehouse to Hightouch, paste a SQL query, and use their visual mapper to specify how data should appear in your SaaS systems. No more scripts, just SQL. Supercharge your business teams with customer data using Hightouch for Reverse ETL today. Get started for free at dataengineeringpodcast.com/hightouch. Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days. Datafold helps Data teams gain visibility and confidence in the quality of their analytical data through data profiling, column-level lineage and intelligent anomaly detection. Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. Go to dataengineeringpodcast.com/datafold today to start a 30-day trial of Datafold. Your host is Tobias Macey and today I’m interviewing Oliver Laslett about Lightdash, an open source business intelligence system powered by your dbt models

Interview

Introduction How did you get involved in the area of data management? Can you describe what Lightdash is and the story behind it?

What are the main goals of the project? Who are the target users, and how has that profile informed your feature priorities?

Business intelligence is a market that has gone through several generational shifts, with products targeting numerous personas and purposes. What are the capabilities that make Lightdash stand out from the other options? Can you describe how Lightdash is architected?

How have the design and goals of the system changed or evolved since you first began working on it? What have been the most challenging engineering problems that you have dealt with?

How does the approach that you are taking with Lightdash compare to systems such as Transform and Metriql that aim to provide a dedicated metrics layer? Can you describe the workflow for som

Discover how semantic layers (dbt/Lightdash) empower all kinds of users to navigate their data without technical knowledge.