Large language models (LLMs) often feel like a knowledgeable conversation partner. Behind the scenes, their work is much narrower: at each step, they predict the next piece of text that is most likely to appear. Understanding the three parts of this process—tokens, probabilities, and selection methods—helps you assess when AI answers are trustworthy and when they should be double-checked.
The unit of work is tokens, not words
LLMs do not read text word by word. Google's Machine Learning Crash Course refers to tokens as the smallest units in language modeling, and a token can be a word, part of a word, or a single character. For example, "unwatched" is broken down into un, watch, and ed.
For English text, the rough benchmark is about 4 characters or three-quarters of a word per token, so 400 tokens are roughly equivalent to 300 words. OpenAI's help documentation confirms this figure and adds that other languages have different character-word-token relationships. This means that the number of Indonesian words in a single request cannot be directly converted to tokens using English figures. The input and output length limits are also counted in tokens, and the limits vary by model.
What is generated is a list of probabilities
After reading the entire context, the model assigns probabilities to many candidate tokens that could follow. Google provides the example sentence "When I hear rain on my roof, I ___ in my kitchen" with the following candidates:
| Candidate | Probability |
|---|---|
| cook soup | 9.4% |
| warm up a kettle | 5.2% |
| cower | 3.6% |
| nap | 2.5% |
| relax | 2.2% |
Note that even the best choice is only about one-tenth. Many answers are plausible, and the model does not "know" which one is correct, only which is more likely.
Two ways to choose from that list
The Hugging Face Transformers library documentation distinguishes between two basic strategies:
- Greedy search: always takes the most likely token. Suitable for short outputs without a need for creativity, but according to the documentation, it tends to repeat itself in long texts.
- Sampling: randomly selects tokens according to their probability distribution, so every token with a probability above zero has a chance of being chosen. This strategy reduces repetition and produces more diverse outputs.
That is why the same question can yield slightly different answers each time. The sampling settings in AI services determine how far the selection can deviate from the top choice.
What it means for everyday use
- Fluency does not equal accuracy. The model is trained to produce coherent phrases, not to verify every fact. Anthropic's documentation acknowledges that even the most advanced models sometimes generate factually incorrect text.
- Allow for uncertainty. Anthropic recommends explicitly stating that the model can respond with insufficient information, as this is said to drastically reduce false information.
- Request citation basis for each claim, and discard claims that lack support in the documents you provide.
- Run it twice and compare. If answers vary on the same factual question, that is a signal to check the original sources.
- Count tokens, not words, when your document is lengthy, as context limits are calculated in that unit.
These techniques reduce errors but do not eliminate them. For high-stakes decisions, AI results still need human validation.
If you find it easier to grasp this concept through verbal explanation, there is a short clip on Taalenta.
Sources
- Google for Developers, Machine Learning Crash Course: Large language models, Tokens and next-token probabilities
- OpenAI Help Center: What are tokens and how to count them?
- Hugging Face Docs: Text generation strategies
- Anthropic Docs: Reduce hallucinations