talk-data.com talk-data.com

Meetup talk 2024-05-15 at 19:00

ML infrastructure and serving LLMs at scale

Event: Mastering LLM Serving and Management at Grammarly

Speakers

Christoph Stuber

Software Engineer · Grammarly

Topics

llms ml infrastructure third-party llm apis in-house llms load testing

Description

This talk covers Grammarly's approach to using a combination of third-party LLM APIs and in-house LLMs, the role of LLMs in Grammarly's product offerings, an overview of the tools and processes used in our ML infrastructure, and how we address challenges such as access, cost control, and load testing of LLMs, sharing our experience in optimizing and serving LLMs.