In this talk, we will examine how LLM outputs are evaluated by potential end users versus professional linguist-annotators, as two ways of ensuring alignment with real-world user needs and expectations. We will compare the two approaches, highlight the advantages and recurring pitfalls of user-driven annotation, and share the mitigation techniques we have developed from our own experience.

LLMs have unlocked new opportunities in NLP with their possible applications. Features that used to take months to be planned and developed now require a day to be prototyped. But how can we make sure that a successful prototype will turn into a high-quality feature useful for millions of customers? In this talk, we will explore real examples of the challenges that arise when ensuring the quality of LLM outputs and how we address them at Grammarly.