In this talk, we will examine how LLM outputs are evaluated by potential end users versus professional linguist-annotators, as two ways of ensuring alignment with real-world user needs and expectations. We will compare the two approaches, highlight the advantages and recurring pitfalls of user-driven annotation, and share the mitigation techniques we have developed from our own experience.

How can we influence quality during the prompt creation stage, as well as how to work with already-generated text—improving it, identifying errors, and filtering out undesirable results. We'll explore linguistic approaches that help achieve better, more controlled outcomes from LLMs.