How LLMs Generate Text: Logits, Temperature, Top-k, and Top-p

What happens between a prompt and the next piece of an answer? Part of AI Engineering for Backend Developers.

You ask a language model to finish this sentence: After lunch, I usually drink. It might continue with “tea,” “coffee,” or “water.” Run the same prompt again and the continuation may change. Where does that choice come from?

The model produces scores for possible next tokens. A separate set of rules turns those scores into a choice. Logits are the scores. Temperature changes their relative influence. Top-k and top-p restrict which candidates can be sampled. The chosen token joins the context, and the process repeats.

This article starts with ordinary words and small numbers. No machine learning background is required. The formulas, short Python examples, and later engineering details are optional. I will use one invented example throughout, so every probability can be checked.

One token at a time

A token is a unit of text represented by an ID in the model’s vocabulary. It can be a word, part of a word, punctuation, or another supported symbol. The vocabulary is the set of token IDs the model can produce. For readability, our examples pretend each displayed word is one token. Real tokenization may split it differently or include a leading space.

An autoregressive language model generates a continuation using the prompt and the tokens already generated. That sounds more complicated than the actual loop: score the next token, choose one, append it, and score the next token again.

Suppose the first choice is “tea.” The next prediction uses After lunch, I usually drink tea. A period may now be likely. If the first choice was “coffee,” the next scores can be different. The model does not keep drawing from the same fixed list for the whole answer. Every choice changes the context for what follows.

Generation ends when a stopping condition is reached. An end-of-sequence token, often abbreviated EOS, can signal completion. Applications can also stop at a configured token limit or stop sequence. Hitting the length limit does not mean the answer is complete; it can end halfway through a sentence. See the Hugging Face text-generation guide.

The earlier tokenization article explains the text units. The transformer article explains how the model processes context. Here, the focus is the decision made from its output scores.

Logits: scores before probabilities

At a generation step, the model’s output layer, often called the language-model head, maps its internal representation to one score per vocabulary token. These raw scores are called logits. Larger means more preferred at this step, before additional decoding rules are applied.

A logit is not a percentage. It can be negative, positive, or zero, and the logits do not have to add up to anything. A function called softmax converts them into positive probabilities that sum to 1. A list of probabilities over all candidates is a probability distribution.

Here is our complete, invented six-token vocabulary. The logits are chosen so that softmax at temperature 1 gives round probabilities. They are not captured from a real model. Real vocabularies contain many more candidates.

“Tea” has the highest score even though its logit is negative. That is fine: it is less negative than the other scores. Adding the same constant to every logit would leave the softmax probabilities unchanged. What matters is their relative separation.

Optional formula: for token i with logit zᵢ, softmax assigns P(i) = exp(zᵢ) / Σⱼ exp(zⱼ). The exponential turns scores into positive weights; the division makes those weights add up to 1. The symbol Σ means “add over all candidates.”

In this example, the full-precision logits are log(0.40), log(0.25), and so on. Exponentiating them gives the original probabilities, whose sum is already 1. Using the rounded logits printed in the table produces slightly different numbers.

Greedy decoding versus sampling

Decoding means the process used to turn the model’s scores into output tokens. Greedy decoding always chooses the highest-scoring allowed next token. In our example it selects “tea,” then repeats the process using the updated context.

Sampling makes a weighted random choice. With the unmodified distribution, “tea” gets a 40% chance on this step, “coffee” 25%, and “water” 15%. Less likely does not mean impossible. “Soup” still has a 4% chance.

Imagine 100 independent draws from this one distribution. You would expect about 40 “tea” results, but you would not require exactly 40. Random variation is part of sampling. These repeated draws are a teaching experiment, not how a full sentence is generated.

Greedy selection is useful when variation is unwanted, but it does not verify facts. Sampling gives alternative continuations, but randomness does not create missing knowledge. The generation-strategies guide distinguishes these decoding approaches.

Temperature: change how concentrated the choices are

Temperature, usually written T, rescales the logits before softmax. For a positive temperature, divide every logit by T, then calculate the probabilities. It changes the probability spread; it does not retrain the model or change its stored knowledge.

• T below 1: the distribution becomes sharper. Higher-scoring candidates gain probability relative to lower-scoring ones.
• T = 1: no temperature rescaling. This does not guarantee that other filters or penalties are disabled.
• T above 1: the distribution becomes flatter. Lower-scoring candidates receive more probability.

At temperature 0.5, “tea” rises from 40% to about 61.5%. At temperature 2, it falls to about 27.7%. The ranking is unchanged: “tea” still comes before “coffee.” Positive temperature changes the gaps, not the order, when applied to the same finite logits without other changes.

This is why “creativity slider” is only a loose shorthand. A higher temperature makes less-preferred continuations more likely. Some are interesting; others are incoherent or incorrect. The setting has no direct way to judge creativity or truth.

Optional deeper intuition: the odds between two tokens satisfy P(i) / P(j) = exp((zᵢ − zⱼ) / T). At temperature 1, tea-to-coffee odds are 0.40 / 0.25 = 1.6. At temperature 0.5 they become 1.6² = 2.56. At temperature 2 they become √1.6 ≈ 1.265. Lower temperature amplifies an existing preference.

What about zero? Dividing by zero is undefined. For a unique highest logit, the positive-temperature distribution approaches greedy selection as T approaches zero. Some products use a setting of zero as shorthand for greedy behavior; others reject it. For example, Hugging Face exposes greedy generation through do_sample=False, with num_beams=1. Check the actual API instead of inserting zero into the formula.

Top-k: keep a fixed number of candidates

Top-k sampling keeps the k highest-scoring candidates, gives the others zero probability, and samples from the retained set. The remaining probabilities must be rescaled to sum to 1 again. This rescaling is called renormalization.

With k = 2, our original distribution keeps only tea and coffee. Their retained probability mass is 0.40 + 0.25 = 0.65. “Probability mass” just means the sum of probabilities in the set.

Water, juice, milk, and soup now have zero chance on this step. Top-k does not pick the winning token by itself. It picks the candidate set; sampling chooses a token from that set. With k = 1, only the highest candidate survives, so there is no sampling variety at that step.

A fixed count can be awkward. When the model strongly prefers one token, a large k can still admit many weak alternatives. When several continuations look equally reasonable, a small k can remove useful options. Top-k does not adjust its count according to how concentrated the distribution is.

Top-p: keep enough probability mass

Top-p sampling, also called nucleus sampling, sorts candidates by probability and keeps the smallest leading set whose cumulative probability reaches or exceeds p. It then renormalizes and samples from that set. The nucleus is simply this retained set.

With p = 0.75, start at the most likely token and add probabilities: tea gives 40%; adding coffee gives 65%; adding water gives 80%. We stop there. The token that crosses the threshold stays. Keeping exactly 75% would require cutting part of a token’s probability, which is not what this rule does.

The “p” is a cumulative probability threshold, not a percentage of vocabulary entries. Top-p 0.75 does not mean “keep 75% of all tokens.” It can keep one token on an easy step and many tokens on an uncertain one.

Consider another invented distribution: 0.92, 0.03, 0.02, 0.01, 0.01, 0.01. With top-p 0.75, the first token alone crosses the threshold, so it is the only candidate. With our original distribution, the same threshold keeps three. That adaptive size is the main difference from top-k.

The nucleus-sampling paper proposed truncating the unreliable tail of a model’s distribution for open-ended generation. Its results motivate the method; they do not establish one best threshold for every model or application. Values such as 0.9 or 0.95 are settings to evaluate, not universal quality guarantees.

How the controls interact

For this article’s experiment, I use this explicit order: temperature → softmax → top-k, if supplied → top-p on the retained distribution → renormalize → sample. This is a teaching convention. A serving system may also apply penalties, grammar constraints, token bans, or other transformations, and its exact order matters.

Temperature can change which tokens survive top-p. In our example, top-p 0.75 keeps two tokens at T = 0.5, three at T = 1, and four at T = 2. The candidate ranking stays the same, but the cumulative probabilities move.

Top-k and top-p can also restrict each other. At T = 1, top-k 3 keeps tea, coffee, and water with renormalized probabilities 0.50, 0.3125, 0.1875. Applying top-p 0.75 next keeps only tea and coffee because 0.50 + 0.3125 = 0.8125. Applying top-p 0.75 alone would keep all three.

This is an easy source of confusing experiments: a setting looks ineffective because another setting already removed its alternatives. Read your model or provider’s generation configuration documentation. Defaults, supported controls, filtering order, and tie handling are implementation details, not universal properties of LLMs.

For learning, change one control at a time. Start from an explicitly understood baseline, print the candidates, and only then combine settings. In a production API, record the effective configuration, including inherited defaults.

Five short Python examples

Each block runs independently using only Python’s standard library. Paste one into a Python session, or save it in a file and run it with python3. These are small examples with fixed inputs, so you can focus on one idea at a time. They do not load a model or call a paid service.

1. Turn logits into probabilities

The scores below are invented using log so that softmax recovers the same six probabilities as the article.

from math import exp, log

tokens = ["tea", "coffee", "water", "juice", "milk", "soup"]
logits = [log(p) for p in [0.40, 0.25, 0.15, 0.10, 0.06, 0.04]]
weights = [exp(z - max(logits)) for z in logits]
total = sum(weights)
probs = [round(w / total, 2) for w in weights]
print(dict(zip(tokens, probs)))

Output:
{'tea': 0.4, 'coffee': 0.25, 'water': 0.15, 'juice': 0.1, 'milk': 0.06, 'soup': 0.04}

What it shows: Tea has a 40% chance. Subtracting the largest logit keeps exponentiation stable without changing the probabilities.

2. Compare greedy selection with sampling

Both choices use the same probabilities. Greedy takes the maximum; sampling makes a weighted random draw.

from random import Random

tokens = ["tea", "coffee", "water", "juice", "milk", "soup"]
probs = [0.40, 0.25, 0.15, 0.10, 0.06, 0.04]
print("Greedy:", tokens[probs.index(max(probs))])
print("Sample:", Random(0).choices(tokens, weights=probs)[0])

Output:
Greedy: tea
Sample: juice

What it shows: Greedy selects tea because its probability is highest. This sampled draw selects juice, which has a 10% chance. The seed 0 makes this illustrative draw repeatable in the same Python environment.

3. Change temperature

Use the same logits with three positive temperatures. Each printed list follows the order tea, coffee, water, juice, milk, soup.

from math import exp, log

logits = [log(p) for p in [0.40, 0.25, 0.15, 0.10, 0.06, 0.04]]
for temperature in (0.5, 1.0, 2.0):
    weights = [exp((z - max(logits)) / temperature) for z in logits]
    total = sum(weights)
    print(temperature, [round(w / total, 3) for w in weights])

Output:
0.5 [0.615, 0.24, 0.086, 0.038, 0.014, 0.006]
1.0 [0.4, 0.25, 0.15, 0.1, 0.06, 0.04]
2.0 [0.277, 0.219, 0.17, 0.139, 0.107, 0.088]

What it shows: Tea rises to about 61.5% at temperature 0.5 and falls to about 27.7% at temperature 2. The token ranking stays the same.

4. Keep the top two candidates

Sort by probability, retain two tokens, and divide by their combined probability to renormalize.

probs = {"tea": 0.40, "coffee": 0.25, "water": 0.15,
         "juice": 0.10, "milk": 0.06, "soup": 0.04}
kept = sorted(probs, key=probs.get, reverse=True)[:2]
mass = sum(probs[token] for token in kept)
print({token: round(probs[token] / mass, 4) for token in kept})

Output:
{'tea': 0.6154, 'coffee': 0.3846}

What it shows: Only tea and coffee remain, at about 61.54% and 38.46%. This block selects the candidates; a weighted draw would choose the next token.

5. Keep enough probability for top-p

Add candidates from most likely downward until their total reaches or exceeds 0.75. Keep the token that crosses the threshold.

probs = {"tea": 0.40, "coffee": 0.25, "water": 0.15,
         "juice": 0.10, "milk": 0.06, "soup": 0.04}
kept = []
mass = 0.0
for token in sorted(probs, key=probs.get, reverse=True):
    kept.append(token)
    mass += probs[token]
    if mass >= 0.75:
        break
print({token: round(probs[token] / mass, 4) for token in kept})

Output:
{'tea': 0.5, 'coffee': 0.3125, 'water': 0.1875}

What it shows: The retained mass is 0.80, so tea, coffee, and water survive. After renormalization their probabilities are 50%, 31.25%, and 18.75%.

Keep the boundary clear: top-k and top-p above filter the original distribution independently. To combine controls, follow the explicit order described earlier. These examples demonstrate one prediction step. A full answer needs fresh model scores after each selected token is added to the context.

Real libraries may handle tied scores or minimum candidate counts differently. Check their documented token filters. Rounding here is only for displaying results; retain full precision when chaining operations.

Deeper point: a greedy step is not a global optimum

Choosing the best next token sounds like it should produce the most likely whole answer. It does not follow. A complete sequence’s probability multiplies the conditional probabilities of its tokens. A strong first choice can lead to weaker continuations.

In the diagram, greedy decoding chooses A because 0.60 exceeds 0.40, then X because 0.51 exceeds 0.49. The path probability is 0.60 × 0.51 = 0.306. But B followed by X scores 0.40 × 0.90 = 0.360. The locally preferred first choice loses the comparison between complete paths.

All paths here have two tokens, so length does not complicate the comparison. More elaborate methods, such as beam search, keep several candidate prefixes to explore alternatives. They add computation and still optimize a scoring objective. The highest-probability text is not automatically the best explanation, the most factual answer, or the most appropriate response.

What sampling settings cannot guarantee

Low temperature is not a factuality check

A model can strongly prefer an incorrect continuation. Lowering temperature can make that same mistake more repeatable. A token probability measures the model’s next-token preference under a context and configuration. It is not the probability that a factual claim is true.

For a factual application, improve the evidence available to the model and verify important outputs. Retrieval can supply source material; tests, validators, or human review can check relevant properties. These solve different problems from deciding how randomly to choose a token. The embeddings and vector-search article explains the retrieval side.

A seed is only part of reproducibility

A random seed fixes the starting state of a random-number generator. Repeating a model request can still produce different results if the model version, prompt formatting, execution path, hardware, or numerical behavior changes. Hosted services may also provide weaker guarantees than a local toy script. The PyTorch reproducibility note explains why a seed alone does not guarantee identical results across platforms and releases.

When comparing settings, record the model revision, complete input, tokenizer or chat template, sampling configuration, output limit, and software versions where you control them. Reproducibility is an end-to-end property. Greedy selection removes one source of randomness; it does not make every serving system bit-for-bit deterministic.

Valid structure and correct content are separate

If an application needs JSON, lowering temperature does not enforce a JSON grammar. Use the system’s supported structured-output or constrained-decoding feature, then validate the result. A grammar can restrict which tokens are allowed, but syntactically valid JSON can still contain a wrong amount or an invented identifier.

Likewise, top-k is not a search over documents or a count of complete answers. It limits next-token candidates at each step. Changing k does not by itself remove the model’s need to calculate vocabulary scores, so it is not a reliable shortcut to lower inference cost.

A practical way to choose settings

I would start with the model’s documented generation configuration and a small set of representative prompts. The right experiment depends on the job. For extraction, check required fields and exact values. For explanations, check correctness and coverage. For creative writing, inspect diversity as well as coherence.

• Establish a baseline: save the full configuration and outputs. Include awkward prompts, short inputs, and cases where the model should admit missing information.
• Change one setting: vary temperature or one candidate filter while holding the rest fixed. Sample several outputs per prompt when evaluating a stochastic policy. One attractive response proves little.
• Inspect failures: separate missing knowledge, invalid structure, repetition, and unwanted randomness. Fix the actual cause instead of expecting one decoding knob to solve every problem.
• Check stopping and latency: record whether answers finish naturally or hit the output limit, along with response time and token use. More output is not automatically a better answer.

There is no single “best temperature.” A setting is useful when it improves the behavior you measured for your model, task, and inputs. Keep an evaluation set and revisit it when the model or prompt changes.

The mental model to keep

The model scores. The decoding policy reshapes and filters. A token is chosen. The context grows. Repeat. Logits are raw scores, temperature changes their probability spread, top-k limits the candidate count, and top-p adapts that count to cumulative probability. None of these settings verifies whether the answer is true.

For a first experiment, run one short block, change its setting, and compare the output before combining controls. For a real service, add task-specific evaluation, validation, and clear stopping rules. Understanding the small example makes those larger choices easier to reason about.

Previous: Embeddings & Vector Spaces: What They Actually Mean. Explore the AI Engineering for Backend Developers series.