We strive for our dbt project to be ready by 9am for our stakeholders. Should be easy, right? Except that our dbt project consists of around 450 dbt models and over 30 sources. Some of those sources are ready as early as midnight but some as late as 4am, and in total our project takes around 4 hours to run. Join as us we walk through the evolution of our dbt run setup, from one selector, to a set of parallel commands, to today's setup -- a dynamic lineage in Airflow which runs models when and only when the upstream source is ready. It's finished when the Tableau datasource is refreshed and our stakeholders can start their day with the latest data.
talk-data.com
K
Speaker
Katie Scheitzer
1
talks
Senior Analytics Engineer
Studyportals
Katie is a Senior Data Analytics Engineer, who has more than 10 years of experience working in data, inlcuding web analytics, data science, and analytics engineering. She is originally from the US, but came to the Netherlands to get her Masters in Data Science and has been working here since.
Bio from: 22th Eindhoven Data Community Meetup | Studyportals
Filter by Event / Source
Talks & appearances
1 activities · Newest first