talk-data.com talk-data.com

PyData talk 2025-12-09 at 11:20

Efficient Time-Series Forecasting with Thousands of Local Models on Databricks

Description

In industries like energy and retail, forecasting often requires local models when each time series has unique behavior — though training thousands of them can be overwhelming. However, training and managing thousands of such models presents scalability and operational challenges. This talk shows how we scaled local models on Databricks by leveraging the Pandas API on Spark, and shares practical lessons on storage, reuse, and scaling challenges to make this approach efficient when it’s truly needed