talk-data.com talk-data.com

PyData talk 2025-09-26 at 09:05

Declarative Feature Engineering: Bridging Spark and Flink with a Unified DSL

Description

Building ML features at scale shouldn’t require every ML Scientist to become an expert in Spark or Flink. At Adyen, the Feature Platform team built a Python-based DSL that lets data scientists define features declaratively — while automatically generating the necessary batch or real-time pipelines behind the scenes.