talk-data.com talk-data.com

M

Speaker

Mocheng Guo

1

talks

Software Engineer at Airbnb
Filtering by: Airflow Summit 2022 ×

Filter by Event / Source

Talks & appearances

Showing 1 of 1 activities

Search activities →

In this talk we want to present how Airbnb extends the REST api to support on-demand workload. A DAG object is created from a local environment like Jupyter notebook, serialized into binary and transported to the API. The API persists the DAG object into the meta DB and Airflow scheduler and worker are extended to process this new kind of DAG.