talk-data.com talk-data.com

S

Speaker

Sung Yun

1

talks

Software Engineer at Bloomberg

Filter by Event / Source

Talks & appearances

1 activities · Newest first

Search activities →

As a team that has built a Time-Series Data Lakehouse at Bloomberg, we looked for a workflow orchestration tool that could address our growing scheduling requirements. We needed a tool that was reliable and scalable, but also could alert on failures and delays to enable users to recover quickly from them. From using triggers over simple sensors to implementing custom SLA monitoring operators, we explore our choices in designing Airflow DAGs to create a reliable data delivery pipeline that is optimized for failure detection and remediation.