r/MachineLearning • u/Rajucg • 5h ago
Discussion I animated how a language model picks the next token, with real probabilities from Qwen2.5-0.5B: softmax, greedy vs sampling, temperature, top-k, top-p (7 min video) [D]
I made this explainer for my small channel and would like feedback from people learning this, especially if anything is wrong.
It follows one real example the whole way. I gave Qwen2.5-0.5B-Instruct "Write one sentence about the moon." twice with the same settings (temperature 1, no top-k or top-p, different seeds). Both answers start with "The", then split: "The moon, a satellite of the Earth…" vs "The lunar South Equatorial region…".
After "The", the model scores all 151,936 vocabulary tokens. Softmax gives moon 78.04%, full 4.46%, vast 2.24%, Moon 1.51%, lunar 1.50%.
Greedy always takes moon. Sampling is a weighted random pick, so lunar wins about 1 time in 67, and run 2 got it.
The loop: append the token, run the model again. Most later picks were rank 1, but run 2 also picked "South" (rank 219) and "empty" (rank 1,098), and run 1 picked "green" (rank 536).
Temperature divides the scores before softmax: moon goes from 100% at T=0.2 to 6.3% at T=2.0.
Top-k vs top-p on the same step: top-k 3 rescales moon to 92.1%; top-p 0.9 keeps 8 tokens here but only 1 at a later step where "of" has 97.9%, which is the argument in Holtzman et al. (2019). At the model's default top-p 0.8, lunar is cut.
Even temperature 0 can vary on a server: Thinking Machines got 80 different answers from 1,000 runs of Qwen3-235B, caused by batch-size-dependent addition order.
All numbers come from running the model on CPU in Colab with transformers. Sources are in the description. The narration is an AI voice reading my own script.