talk-data.com
PyData
talk
2025-09-26 at 12:55
Resource Monitoring and Optimization with Metaflow
Event:
PyData Amsterdam 2025
Speakers
Topics
Description
Metaflow is a powerful workflow management framework for data science, but optimizing its cloud resource usage still involves guesswork. We have extended Metaflow with a lightweight resource tracking tool that automatically monitors CPU, memory, GPU, and more, then recommends the most cost-effective cloud instance type for future runs. A single line of code can save you from overprovisioned costs or painful job failures!