talk-data.com talk-data.com

Meetup talk 2025-10-29 at 19:00

WTF is Temperature: LLM inference in Practice

Event: PyData Trójmiasto #37

Speakers

Dariusz Piotrowski

Senior Machine Learning Engineer (contractor) · Amazon Robotics

Topics

LLM inference tokenization sampling strategies quantization kv caching prompt caching beam search

Description

Ever wondered what actually happens when you call an LLM API? This talk breaks down the inference pipeline from tokenization to text generation, explaining what's really going on under the hood. He will walk through the key sampling strategies and their parameters - temperature, top-p, top-k, beam search. We'll also cover performance tricks like quantization, KV caching, and prompt caching that can speed things up significantly. If time allows, we will also touch on some use-case-specific techniques like pass@k and majority voting.