At Trendyol, Turkey’s leading e-commerce company, Apache Airflow powers our task orchestration, handling DAGs with 500+ tasks, complex interdependencies, and diverse environments. Managing on-prem Airflow instances posed challenges in scalability, maintenance, and deployment. To address these, we built TaskHarbor, a fully managed orchestration platform with a hybrid architecture—combining Airflow on GKE with on-prem resources for optimal performance and efficiency. This talk covers how we: Enabled seamless DAG synchronization across environments using GCS Fuse. Optimized workload distribution via GCP’s HTTPS & TCP Load Balancers. Automated infrastructure provisioning (GKE, CloudSQL, Kubernetes) using Terraform. Simplified Airflow deployments by replacing Helm YAML files with a custom templating tool, reducing configurations to 10-15 lines. Built a fully automated deployment pipeline, ensuring zero developer intervention. We enhanced efficiency, reliability, and automation in hybrid orchestration by embracing a scalable, maintainable, and cloud-native strategy. Attendees will obtain practical insights into architecting Airflow at scale and optimizing deployments.
talk-data.com
Topic
YAML
Yet Another Markup Language (YAML)
data_serialization
configuration_file_format
human_readable
file_format
1
tagged
Activity Trend
9
peak/qtr
2020-Q1
2026-Q1
Top Events
Airflow Summit 2025
3
dbt Coalesce 2023
3
O'Reilly Data Engineering Books
2
Airflow Summit 2024
2
Data Engineering Podcast
1
dbt Coalesce 2022
1
PyData Rhein-Main I Security Risks in AI & Structured Automation with Agentic AI
1
DataTopics: All Things Data, AI & Tech
1
AI Camp NYC: GenAI, LLMs and Agent
1
Big Data LDN 2025
1
Data Expo NL 2025
1
AI and Deep Learning for Enterprise #11
1
Filtering by:
Salih Goktug Kose
×