The talk will cover how we use Airflow at the heart of our Workflow Management Platform(WFM) at Booking.com, enabling our internal users to orchestrate big data workflows on Booking Data Exchange(BDX). High level overview of the talk: Adapting open source Airflow helm chart to spin up Airflow installation in Booking Kubernetes Service (BKS) Coming up with Workflow definition format (yaml) Conversion of workflow.yaml to workflow.py DAGs Usage of Deferrable operators to provide standard step templates to users Workspaces (collection of workflows), using it to ensure role based access to DAG permissions for users Using okta for authentication Alerting, monitoring, logging Plans to shift to Astronomer
talk-data.com
Topic
YAML
Yet Another Markup Language (YAML)
data_serialization
configuration_file_format
human_readable
file_format
1
tagged
Activity Trend
9
peak/qtr
2020-Q1
2026-Q1
Top Events
Airflow Summit 2025
3
dbt Coalesce 2023
3
O'Reilly Data Engineering Books
2
Airflow Summit 2024
2
Data Engineering Podcast
1
dbt Coalesce 2022
1
PyData Rhein-Main I Security Risks in AI & Structured Automation with Agentic AI
1
DataTopics: All Things Data, AI & Tech
1
AI Camp NYC: GenAI, LLMs and Agent
1
Big Data LDN 2025
1
Data Expo NL 2025
1
AI and Deep Learning for Enterprise #11
1
Filtering by:
Madhav Khakhar
×