How Sampling Limits LLM Vocabulary

Large Language Models (LLMs) are often praised for their impressive command of language, but why do they repeatedly rely on the same words and expressions? While previous research has linked this phenomenon to model training and alignment, a new study by researchers from the Information and Processing Telecommunications Center (IPTC) at the Universidad Politécnica de Madrid (UPM) shows that part of the answer lies in the way LLMs generate text.

The accompanying article, Are There Words LLMs Won’t Say? explains how LLMs predict the next token by assigning probabilities to every possible candidate before applying sampling methods such as Top-k, Top-p, and Min-p. Although these decoding strategies improve fluency and coherence, they also remove many low-probability candidates, making some perfectly valid words impossible to generate despite being present in the model’s vocabulary.

Building on this concept, the IPTC researchers introduced the Word Coverage Score (WCS) in their preprint, Lost in Sampling: Assessing Lexical Reachability in LLMs via the Word Coverage Score. Rather than measuring what a model knows, WCS measures what it can actually produce under real decoding conditions, quantifying how many contextually appropriate words remain reachable after sampling filters are applied.

Experiments across several state-of-the-art open-weight LLMs reveal that commonly recommended decoding settings systematically reduce lexical diversity by preventing a significant number of suitable words from ever being generated. The results suggest that vocabulary limitations are not solely a consequence of model training or alignment, but also of the inference algorithms used during text generation.

As AI-generated content becomes increasingly widespread, the researchers argue that preserving lexical diversity should become a key objective in the design of future language models. The Word Coverage Score provides a practical framework for evaluating this trade-off, helping developers balance text coherence with the richness and variety that characterize human language.

You can read the preprint here: https://arxiv.org/pdf/2605.27268

And here to the blog article: https://samermakes.com/blog/WCS/

Javier Conde Díaz: GS / ORCID / LinkedIn

Carlos Arriaga Prieto: GS / ORCID / LinkedIn

Pedro Reviriego: GS / ORCID / LinkedIn

LinkedIn: https://www.linkedin.com/company/iptc-upm/

For more information: www.iptc.upm.es

Share this: