talk-data.com talk-data.com

Airflow Summit session 2024-07-01

Orchestration of ML workloads via Airflow & GKE Batch

Event: Airflow Summit 2024

Speakers

Rafal Biegacz

Senior Engineering Manager (Cloud Composer, Google)

Piotr Leśniak

Sr Staff Software Engineer at Google

Topics

AI/ML Airflow API Kubernetes

Description

During this talk we are going to given an overview of different orchestration approaches (Kubeflow, Ray, Airflow, etc.) when running ML workloads on Kubernetes and specifically we will focus on how to use Kubernetes Batch API and Kubernetes Operators to run complex ML workloads.