In this talk, we will examine how LLM outputs are evaluated by potential end users versus professional linguist-annotators, as two ways of ensuring alignment with real-world user needs and expectations. We will compare the two approaches, highlight the advantages and recurring pitfalls of user-driven annotation, and share the mitigation techniques we have developed from our own experience.