We strive for our dbt project to be ready by 9am for our stakeholders. Should be easy, right? Except that our dbt project consists of around 450 dbt models and over 30 sources. Some of those sources are ready as early as midnight but some as late as 4am, and in total our project takes around 4 hours to run. Join as us we walk through the evolution of our dbt run setup, from one selector, to a set of parallel commands, to today's setup -- a dynamic lineage in Airflow which runs models when and only when the upstream source is ready. It's finished when the Tableau datasource is refreshed and our stakeholders can start their day with the latest data.
talk-data.com
A
Speaker
Armand Duijen
1
talks
Data Engineer
Studyportals
Armand is a Data Engineer from Eindhoven, working at Studyportals. He has nearly five years of experience building and maintaining data pipelines in the AWS cloud. With a background in Computer Science from TU/e, he also explores machine learning to help Studyportals predict student behavior. Outside of work, he enjoys running and exploring new places.
Bio from: 22th Eindhoven Data Community Meetup | Studyportals
Filtering by:
22th Eindhoven Data Community Meetup | Studyportals
×
Filter by Event / Source
Talks & appearances
Showing 1 of 1 activities