talk-data.com talk-data.com

L

Speaker

Luigi Di Tacchio

1

talks

Sr. Software Engineer FreeWheel, a Comcast Company

I am a Sr. Software Engineer at FreeWheel, a Comcast Company, where I build scalable data pipelines and products using Spark and Databricks to leverage Comcast's viewership data. I began my career as a researcher at the University at Buffalo (SUNY) before moving into industry. For over five years at FreeWheel, I have focused on data infrastructure within Comcast advertising. At the Data + AI Summit, I will share how my team rebuilt our CI/CD infrastructure from an imperative to a declarative paradigm using Hatch and Databricks Asset Bundles.

Bio from: Data + AI Summit 2025

Filter by Event / Source

Talks & appearances

1 activities · Newest first

Search activities →
From Imperative to Declarative Paradigm: Rebuilding a CI/CD Infrastructure Using Hatch and DABs

Building and deploying Pyspark pipelines to Databricks should be effortless. However, our team at FreeWheel has, for the longest time, struggled with a convoluted and hard-to-maintain CI/CD infrastructure. It followed an imperative paradigm, demanding that every project implement custom scripts to build artifacts and deploy resources, and resulting in redundant boilerplate code and awkward interactions with the Databricks REST API. We set our mind on rebuilding it from scratch, following a declarative paradigm instead. We will share how we were able to eliminate thousands of lines of code from our repository, create a fully configuration-driven infrastructure where projects can be easily onboarded, and improve the quality of our codebase using Hatch and Databricks Asset Bundles as our tools of choice. In particular, DAB has made deploying across our 3 environments a breeze, and has allowed us to quickly adopt new features as soon as they are released by Databricks.