talk-data.com talk-data.com

Superweek talk 2024-02-01

A SWEET FINISH: DATA INGESTION PIPELINES WITHOUT THE HEADACHES

Description

We all know that data, like wine and cheese, becomes more valuable when combined. And, just like wine and cheese, they can lead to serious headaches. Whether you are emailing Excel files around, capturing data from thousands of IoT-devices, or just joining your Google Analytics and sales data, you can benefit from following a structured process to minimize your headaches. After debugging yet another failed pipeline I have distilled my experience of building data ingestion pipelines in 8 simple (though not necessarily easy) steps from setting up triggers to archiving and retention.