How does a language model actually "think"?
One example per building block – to remember it
Try it yourself: temperature
The trick behind seemingly clever answers
A language model like ChatGPT or Claude only predicts, at each step, which piece of text most likely comes next – trained on huge amounts of text. Out of this one simple ability, sheer scale produces behavior that looks remarkably intelligent.
The four building blocks that explain the behavior
Tokens – the building blocks of language: A language model doesn't read or write in whole words, but in tokens – small text chunks that can also be parts of words. Probability – how the next word gets chosen: For every next token, the model computes probabilities for all the candidates in question and picks one – token by token, until a whole answer emerges. Context window – the memory for a conversation: The model only "knows" what's in the current context window. Start a new chat, and that memory is gone – unless an app builds a separate memory layer on top. No truth check – why hallucinations happen: The model has no built-in mechanism that checks whether a statement is true – it optimizes for plausible-sounding text. That's why false statements can sound just as confident as correct ones.
Why this matters for you as a decision-maker
These four points explain why a language model isn't a database: important facts should be checked, important context has to be given explicitly, and answers are probabilities, not guarantees. This is exactly why Beyond Prompt deliberately uses deterministic logic instead of generative models for pricing and calculations (see the blog article on quote automation) – the language model gets to understand and phrase things, but a rule decides.
Key takeaways
- ✓A language model only predicts the most likely next piece of text at each step – it doesn't "understand" the way a human does.
- ✓Text gets split into tokens, which can also be parts of words – not always whole words.
- ✓The model only knows what's in the current context window – no automatic memory across separate conversations.
- ✓There's no built-in truth check – so confidently phrased but false answers can occur (hallucination).
- ✓Important facts and calculations belong checked, or handled by deterministic logic – not trusted blindly to the language model.
→ Compare models technically: open-source LLMs compared
Want to understand more deeply how language models actually work?