How an LLM generates text: tokens, attention, temperature

Middle PRO

Essence

An LLM generates text one token at a time. At each step the model looks at all the preceding text, computes a probability for every token in its vocabulary and picks one. The chosen token is appended, and the loop repeats. There is no explicit "answer plan": a long coherent answer is the result of thousands of such steps.

tokens → embeddings → [attention → MLP] × N layers → logits → probabilities → pick a token
                                                                                  ↓
                                                                append to the input and repeat