Topic

Amazon EMR

big_data hadoop aws

Activities

1

tagged

Activity Trend

6 peak/qtr

2020-Q1 2026-Q1

Top Events

AWS re:Invent 2024 12 Data Engineering Podcast 6 Databricks DATA + AI Summit 2023 6 O'Reilly Data Engineering Books 5 Data + AI Summit 2025 3 Airflow Summit 2020 1 Airflow Summit 2023 1

Top Speakers

Tobias Macey 6 Rick Sears (AWS) 3 Martin Holste (Trellix) 2 Trâm Ngọc Phạm 1 Tim Kessler (Redox, Inc.) 1 Mark Greville (Workhuman) 1 Josh Beemster (Snowplow) 1 Rob Goretsky 1 Dipankar Ghosal 1 Kevin Schmidt (Google) 1 Kinshuk Pahare (AWS) 1 Christopher Locklin 1

Activities

Showing filtered results

All Video Podcast Book

Filtering by: Airflow Summit 2020 ×

Migrating Airflow-based Spark jobs to Kubernetes - the native way

2020-07-01 · Airflow Summit 2020

session

by Roi Teveth (Nielsen Identity Engine) , Itai Yaffe (Nielsen Identity Engine)

Airflow AWS GCP Kubernetes Spark

At Nielsen Identity Engine, we use Spark to process 10’s of TBs of data. Our ETLs, orchestrated by Airflow, spin-up AWS EMR clusters with thousands of nodes per day. In this talk, we’ll guide you through migrating Spark workloads to Kubernetes with minimal changes to Airflow DAGs, using the open-sourced GCP Spark-on-K8s operator and the native integration we recently contributed to the Airflow project.