talk-data.com talk-data.com

S

Speaker

Soren Archibald

1

talks

Consulting Lead at QBiz, Inc.
Filtering by: Airflow Summit 2023 ×

Filter by Event / Source

Talks & appearances

Showing 1 of 2 activities

Search activities →

The ability to create DAGs programmatically opens up new possibilities for collaboration between Data Science and Data Engineering. Engineering and DevOPs are typically incentivized by stability whereas Data Science is typically incentivized by fast iteration and experimentation. With Airflow, it becomes possible for engineers to create tools that allow Data Scientists and Analysts to create robust no-code/low-code data pipelines for feature stores. We will discuss Airlow as a means of bridging the gap between data infrastructure and modeling iteration as well as examine how a Qbiz customer did just this by creating a tool which allows Data Scientists to build features, train models and measure performance, using cloud services, in parallel.