Workload Orchestration is at the heart of a successful Data lakehouse implementation. Especially for the “house” part which represents the Datawarehouse workloads which often are complex because of the very nature of warehouse data, which have dependency orchestration problems. We at Asurion have spent years in perfecting the Airflow solution to make it a super power for our Data Engineers. We have innovated in key areas like single operator for all use cases, auto DAG code generation, custom UI components for Data Engineers, monitoring tools etc. With over a few million job runs per year running on a platform with over 3 nines of availability, we have condensed years of our learnings into valuable ideas that can inspire and help all other Data enthusiasts. This session is going to walk the audience through some blind spots and pain points of Airflow architecture, scaling, engineering culture.
talk-data.com
Speaker
Rajesh Gundugollu
1
talks
Solving and simplifying petabyte scale problems!
Rajesh is an experienced Data and Thought Leader with accomplished history of making an impact. Driven by Customer Obsession, Simplified Design, Imagination, Creativity and Passion to innovate. Rajesh as a Director of Engineering and Architecture for Data, ML and AI, helped rapid innovation of Data and ML platforms at Asurion to stay simple, available, and cost effective while processing petabytes of data and helping business unlock insights from data.
Bio from: Data Universe 2024
Filtering by:
Airflow Summit 2023
×
Filter by Event / Source
Talks & appearances
Showing 1 of 2 activities