Last year, we shared how LinkedIn’s continuous deployment platform (LCD) leveraged Apache Airflow to streamline and automate deployment workflows. LCD is the deployment platform inside Linkedin which is actively used by all engineers (10000+) at Likedin. This year, we take a deeper dive into the challenges, solutions, and engineering innovations that helped us scale Airflow to support thousands of concurrent tasks while maintaining usability and reliability. Key Takeaways: Abstracting Airflow for a Better User Experience – How we designed a system where users could define and update their workflows without directly interacting with Airflow. Scaling to 10,000+ Concurrent Tasks – The architectural and configuration changes that enabled us to scale execution efficiently. Enhanced Observability & Monitoring – The tools and techniques we implemented to track Airflow’s health, detect failures, and improve reliability. Lessons from the Field – Key learnings, trade-offs, and best practices for managing large-scale Airflow deployments.
talk-data.com
Speaker
Arun Kumar
1
talks
Arun Kumar serves as a Principal Product Manager, specializing in cybersecurity and enterprise data protection. With extensive experience in leading strategic security initiatives, he focuses on implementing multi-layered defense mechanisms to mitigate data exfiltration risks across diverse organizational landscapes. He is recognized for his ability to translate complex security challenges into actionable solutions, fostering resilient data protection frameworks.
At Microsoft Ignite, he will share insights into building comprehensive data security strategies that align with evolving threat landscapes and regulatory requirements.
Bio from: Microsoft Ignite 2025
Filter by Event / Source
Talks & appearances
Showing 1 of 2 activities