At Wix more often than not business analysts build workflows themselves to avoid data engineers being a bottleneck. But how do you enable them to create SQL ETLs starting when dependencies are ready and sending emails or refreshing Tableau reports when the work is done? One simple answer may be to use Airflow. The problem is every BA cannot be expected to know Python and Git so well that they will create thousands of DAGs easily. To bridge this gap we have built a web-based IDE, called Quix, that allows simple notebook-like development of Trino SQL workflows and converts them to Airflow DAGs when a user hits the “schedule” button. During the talk we will go through the problems of building a reliable and extendable DAG generating tool, why we preferred Airflow over Apache Oozie and also tricks (sharding, HA-mode, etc) allowing Airflow to run 8000 active DAGs on a single cluster in k8s.
talk-data.com
Topic
Trino
Apache Trino
query_engine
big_data
analytics
1
tagged
Activity Trend
14
peak/qtr
2020-Q1
2026-Q1
Top Events
Data Engineering Podcast
57
Databricks DATA + AI Summit 2023
5
O'Reilly Data Engineering Books
3
Data + AI Summit 2025
2
Apache Iceberg Paris Community Meetup #2
1
dbt Coalesce 2022
1
Airflow Summit 2025
1
O'Reilly Data Science Books
1
Data Council Austin 2024 - Day 1
1
Airflow Summit 2024
1
gartner-data-analytics-uk-2025
1
Big Data & AI Paris 2025
1
Filtering by:
Airflow Summit 2024
×