talk-data.com talk-data.com

Google Cloud Next session 2025-04-10 at 23:00

Monitor performance for LLM training and inference workloads on GKE

Event: Google Cloud Next '25

Speakers

Jie Wu

Software Engineer · Google Cloud

James Maffey

Product Manager · Google Cloud

Topics

AI/ML Kubernetes LLM Cloud Computing GCP

Description

Struggling to monitor the performance and health of your large language model (LLM) deployments on Google Kubernetes Engine (GKE)? This session unveils how the Google Cloud Observability suite provides a comprehensive solution for monitoring leading AI model servers like Ray, NVIDIA Triton, vLLM, TGI, and others. Learn how our one-click setup automatically configures dashboards, alerts, and critical metrics – including GPU and TPU utilization, latency, throughput, and error analysis – to enable faster troubleshooting and optimized performance. Discover how to gain complete visibility into your LLM infrastructure.