talk-data.com
Superweek
talk
2024-02-01
A SWEET FINISH: DATA INGESTION PIPELINES WITHOUT THE HEADACHES
Event:
Superweek 2024
Speakers
Topics
Description
We all know that data, like wine and cheese, becomes more valuable when combined. And, just like wine and cheese, they can lead to serious headaches. Whether you are emailing Excel files around, capturing data from thousands of IoT-devices, or just joining your Google Analytics and sales data, you can benefit from following a structured process to minimize your headaches. After debugging yet another failed pipeline I have distilled my experience of building data ingestion pipelines in 8 simple (though not necessarily easy) steps from setting up triggers to archiving and retention.