talk-data.com talk-data.com

PyData talk 2025-12-09 at 12:50

Scaling Python to thousands of nodes with Ray

Description

Python is the language of choice for anything to do with AI and ML. While that has made it easy to write code for one machine, it's much more difficult to run workloads across clusters of thousands of nodes. Ray allows you to do just that. I'll demonstrate how to implement this open source tool with a few lines of code. As a demo project, I'll show how I built a RAG for the Wheel of Time series.