How LLMs Learn: Pretraining, SFT, RLHF, and Alignment

Predicting plausible text and behaving like a useful assistant are related, but different, abilities. Part 6 of AI Engineering for Backend Developers.

Imagine asking a language model to explain a failed deployment. It produces a fluent continuation that resembles a forum discussion. You wanted a direct answer with a few diagnostic steps. It understood enough language to continue the text, but its response did not match the task.

Architecture alone does not explain that difference. Architecture describes the mechanism. Context supplies the current input. Training shapes which continuations the model favors.

I find three learning signals useful to keep separate: predict the next token, imitate a demonstrated response, and favor a response that evaluators prefer. These lead into pretraining, supervised fine-tuning, and preference optimization. The boundaries are more useful than memorizing one supposedly universal training pipeline.

What changes when a model learns?

A model’s parameters, often called weights, are numbers used in its calculations. Training adjusts those numbers so that its predictions better match an objective. The objective is expressed through a loss: a numerical measure of how poorly a prediction matches the training target.

A simplified training step looks like this:

• Read a batch of examples and make predictions using the current parameters.
• Calculate the loss for the positions being trained.
• Compute gradients: how the loss would change with small parameter changes.
• Use an optimizer to turn those gradients into parameter updates.

Making predictions is called the forward pass. Backpropagation calculates gradients by working backward through the model’s computations. The optimizer then applies an update. These are separate operations: calculating a loss alone does not change any weights.

An optimizer is the update procedure. Its settings, including the learning rate, influence the size and direction of changes. Repeating this process over many batches changes the model. A checkpoint saves a version of its parameters, often alongside state needed to resume training.

During ordinary inference, the model uses its parameters to produce an answer without updating them. Examples in a prompt can strongly change the answer, but they do not ordinarily trigger gradient updates. The GPT-3 paper studied this kind of learning from examples in context while keeping model parameters fixed.

Retrieval works on that input side too: the application finds documents and includes them in the request. Saving a chat or supplying a corrected policy does not, by itself, retrain the model. An application can arrange a separate training process, including updates to small adapters, but that is an explicit operation with its own data and evaluation requirements.

Pretraining: learn to predict what comes next

For the autoregressive text models discussed in this series, a central pretraining task is next-token prediction. Given preceding tokens, predict the next token found in the training text. A token can be a word fragment, punctuation, or another vocabulary unit, as the tokenization article explains.

The text supplies the targets. Humans do not have to label the next token separately for every position. This is often called self-supervised learning: the training signal is constructed from the data itself.

The GPT-2 paper describes this autoregressive objective. There are other model objectives, but next-token prediction connects directly to the generation mechanism already covered in this series.

Suppose a training sequence contains “The cache entry has expired.” At one position, the model receives the preceding tokens and assigns probabilities to possible continuations. Training rewards assigning more probability to the observed next token. The same idea applies repeatedly across sequences.

Predicting text across varied material encourages learning patterns useful for grammar, code, facts, and relationships between ideas. It can also reproduce errors, biases, and memorized training examples. A target token is correct for the training sequence; that does not establish that the sentence containing it is factually correct.

Data selection therefore matters. Repeated low-quality material does not become authoritative because there is a lot of it. Filtering, duplication, coverage, and data rights shape the material available for learning. The GPT-3 paper describes one particular data mixture and training approach; it is not a specification for every current or proprietary model.

One probability, one loss

For a small example, pretend expired is one token in our vocabulary. The training prefix is The cache entry has, and the target next token is expired. Three hypothetical predictions assign that target probabilities of 0.1, 0.5, and 0.9.

These are three alternative predictions, not probabilities for three tokens in one distribution. For this target, the negative log-likelihood loss is loss = -ln p(target). The symbol ln means the natural logarithm. The resulting losses are approximately:

• Probability 0.1: loss 2.3026.
• Probability 0.5: loss 0.6931.
• Probability 0.9: loss 0.1054.

Assigning greater probability to the observed target produces a smaller loss. Training normally combines losses across many target positions and examples. Reducing one example’s loss does not guarantee that every other task improves.

This short Python example checks the arithmetic. Save it as learning_loss.py and run python3 learning_loss.py. It uses only the standard library.

from math import isfinite, log


def target_loss(probability):
    if (
        type(probability) not in (int, float)
        or not 0 < probability <= 1
        or not isfinite(probability)
    ):
        raise ValueError("Probability must be finite and in (0, 1]")
    return -log(probability)


if __name__ == "__main__":
    # Three invented predictions, not a model training run.
    for probability in (0.1, 0.5, 0.9):
        print(f"p(target)={probability:.1f}, loss={target_loss(probability):.4f}")

The output shows the three losses listed above. The function rejects zero, probabilities above one, and non-finite values such as infinity or NaN. This calculates a loss from invented probabilities. It does not train a neural network or report a model benchmark.

A real training implementation must also calculate how the loss depends on its parameters, then update them. The text-generation article explains how model scores become next-token probabilities. Here, a known training target gives those probabilities a learning signal.

SFT: learn from demonstrated responses

Supervised fine-tuning, or SFT, continues training with selected examples of desired behavior. A typical assistant example contains a user request and a demonstration of a useful response.

For the deployment question, a demonstration might acknowledge the missing error message, request the relevant log excerpt, and suggest a few safe diagnostic checks. The training target is that demonstrated response. Repeated examples teach patterns such as answering directly, following a format, asking for clarification, and handling a particular task.

In standard next-token SFT, teacher forcing means predictions are conditioned on the example’s actual preceding tokens. While learning the second sentence, the model sees the demonstrated first sentence, rather than an earlier sentence it sampled itself. During ordinary generation it must continue from its own generated tokens.

SFT changes the parameters and therefore the predictions. The resulting assistant still generates a response one token at a time. It has learned from a different selection of targets; it has not acquired a separate answering mechanism.

The loss can apply to different positions depending on the dataset and trainer configuration. A prompt-completion setup may train only on completion tokens; a conversational setup may select assistant turns. Other configurations include more positions. A loss mask selects which targets contribute to the loss. It does not mean the prompt is hidden from attention. The TRL SFT documentation describes these options.

SFT can change factual associations as well as response style. However, repeatedly updating weights is usually an awkward way to deliver a policy that changes every week. A source-backed retrieval path can provide the current policy and its version. SFT is more directly useful when you need consistent behavior across examples of a recurring task.

It can also teach mistakes. Confident but incorrect demonstrations reward confident errors. A dataset containing only easy cases may teach little about ambiguity. Include examples reflecting the situations the application must handle, including missing information and appropriate refusal.

Fine-tuning may update all parameters or use a parameter-efficient method that trains a smaller set, such as adapter parameters. That changes the implementation cost and what gets updated. It does not remove the need for suitable examples or held-out evaluation.

Classic RLHF: learn from comparisons

A good response can be easier to recognize than to write from scratch. Reinforcement learning from human feedback, or RLHF, uses human judgments to help shape model behavior. The InstructGPT paper provides an influential historical recipe:

• Train an initial assistant with supervised demonstrations.
• Have people rank candidate responses, then train a reward model from those preferences.
• Optimize the assistant to receive higher reward while constraining how far it moves from a reference model.

The reward model learns to score a prompt and candidate response together, reflecting the comparison data. The generating model is called the policy: given the current context, it determines probabilities for possible next tokens.

For a fictional prompt, “The deployment failed. What should I do?”, consider two candidates:

• Preferred: “Share the failed step and error message. First check the deployment logs and recent configuration changes.”
• Rejected: “The database is definitely down. Restart it immediately.”

The preference criteria here are to acknowledge missing evidence, avoid inventing a cause, and suggest proportionate diagnostic steps. Another wording could satisfy the same criteria. A preference pair describes which response was favored under a rubric, rather than a unique correct string.

In that recipe, a reinforcement-learning algorithm called Proximal Policy Optimization, or PPO, updates the policy using scores from generated responses. A KL penalty discourages excessive movement away from the supervised reference. KL is a measure of difference between probability distributions; here it helps restrain the optimization.

Humans provide the comparison data. They are not necessarily watching the training loop and grading every generated token live. The trained reward model supplies the scalable proxy signal during this optimization stage.

That distinction is essential. A higher reward means better performance according to the learned scoring function. It does not prove truth, complete source coverage, or safe execution. Suppose reviewers often prefer detailed explanations. A reward model could learn surface cues associated with detail, and optimization could amplify them without improving the underlying answer.

This gap between the proxy and the desired outcome is one route to reward hacking: improving the measured score through behavior that does not fulfill the intended goal. Reference constraints help control changes, but they do not make an imperfect reward model a truth verifier.

The reward-model overoptimization study explores this problem using a synthetic setup in which a larger reward model stands in for the target measure. It supports taking proxy errors seriously, while remaining distinct from a complete evaluation against human goals.

DPO and AI feedback change the recipe

RLHF is not a mandatory three-stage assembly line shared by every model. Training pipelines can combine different data sources, objectives, and optimization methods. Two alternatives help make that distinction concrete.

Direct Preference Optimization

Direct Preference Optimization, or DPO, trains from preferred and rejected response pairs. Its standard formulation uses a preference objective based on their probabilities relative to a reference policy, without separately training an explicit reward model and running a PPO rollout loop. See the DPO paper.

The useful engineering distinction is the training arrangement. Classic RLHF learns a separate scorer, then optimizes against it. Standard DPO updates the policy directly from preference pairs under its objective. That can simplify a workflow, but preference quality and evaluation still matter. “Preferred” remains a judgment under some criteria, not an automatic factual label.

DPO still uses gradient training to update parameters. Both responses contribute to its preference objective. It is different from selecting the better answer only after generation, or performing SFT solely on the preferred response.

Constitutional AI and AI-generated feedback

Feedback can also come from another model. In Constitutional AI, a set of written principles guides critique and revision of responses. Revised responses support supervised training. AI-generated preference judgments then support a reinforcement-learning stage, often described as RLAIF, or reinforcement learning from AI feedback.

The paper’s setup replaced human harmlessness comparisons with AI feedback while retaining human helpfulness comparisons. It should not be summarized as removing every human label or judgment. People still choose principles, data, evaluation criteria, and the surrounding process.

AI feedback can expand the amount of feedback available, but it also carries the evaluating model’s limitations. If the evaluator misses a subtle factual error, generating more judgments does not automatically correct it. Test the feedback process itself, as well as the final model.

Alignment is a goal wider than an algorithm

Alignment concerns how model behavior relates to intended goals, instructions, and constraints. Helpful answers, honest uncertainty, appropriate refusal, and reliable instruction following can all be part of that goal. Different applications and stakeholders can disagree about the desired balance.

SFT, RLHF, DPO, and AI feedback are methods that can influence behavior. Naming a method does not establish that a model is aligned for your application. A model can follow the requested format while giving the wrong answer, or refuse harmless requests so often that it becomes unusable.

For a backend service, keep critical controls in the application. Check authorization before exposing a document. Validate amounts and identifiers before executing an operation. Require confirmation or human review where the product calls for it. Source grounding, output validation, and permission checks cover responsibilities that training alone cannot guarantee.

Choose the intervention that matches the failure

Before collecting training data, inspect representative failures. The model may lack current evidence, misunderstand the request, produce the wrong format, or favor an unhelpful response. These call for different interventions.

Start with a held-out task set before changing prompts or weights. Keep evaluation examples separate from training data, including close duplicates and paraphrases where possible. For document-heavy tasks, splitting by source or date can expose failures hidden by a random split.

Evaluate the behavior you need: factual support, task completion, format validity, appropriate abstention, and permission boundaries. Include difficult and ambiguous cases. Compare against the earlier model, and check whether an improvement in the target task causes regressions elsewhere.

Training data also needs ownership and provenance. Confirm that you can use it, remove unnecessary private information, and record where examples and preference labels came from. A training example can become part of learned behavior, so routine debug logs are a poor default dataset.

Version the dataset, training configuration, and resulting checkpoint together. Keep a rollback path and evaluate the actual deployed combination of model, prompt, retrieval, and tools. A good checkpoint cannot compensate for the application supplying the wrong policy or granting excessive access.

Pretraining learns predictive structure. SFT supplies demonstrations. Preference optimization changes which responses are favored. Each relies on a learning signal with limits. Understanding those signals makes it easier to choose between better input, better examples, and a different optimization process.

Previous: Context Windows Are Not Memory: Limits, Prompt Placement, and Lost-in-the-Middle. Revisit How LLMs Generate Text for the prediction step, or explore the AI Engineering for Backend Developers series.