RL for LLMs

Why this week is positioned where it is. The story has changed dramatically twice in the last three years. In 2022, RLHF (PPO + reward model + KL constraint) was the holy grail and the only path to a chat model worth serving. In 2024, DPO came out and showed you could skip the reward model entirely; most teams switched within a year. In 2025–26, reasoning models retrained the world: GRPO + verifiable rewards became central to producing models that could actually reason, and the RL story stopped being only about preferences and started being about correctness. This week is about that arc.

1. The RL story has changed twice

Three eras to understand:

2022–2024: The RLHF era. Train a base model with SFT. Train a reward model on human preferences. Use PPO to optimize the policy against the reward model, with a KL constraint to keep it close to the SFT model. This was hard. The hyperparameters are brittle, the reward model overfits, and you need three or four copies of the model in memory at training time. But it worked, and ChatGPT, Claude 1, and the early open-weight chat models all used it.

2024–2025: The DPO era. Direct Preference Optimization showed that if your goal is to optimize a KL-regularized reward, you can skip the reward model entirely and train on preference pairs directly. The math gives you a closed-form objective. Most teams switched. Variants proliferated — IPO, KTO, ORPO, SimPO — each with small refinements. By mid-2025, most production preference-tuning happened with one of these, not PPO.

2025–2026: The verifiable-reward era. Reasoning models changed the field. Models like DeepSeek-R1 and the o-series showed that you could train reasoning capability with RL on tasks where you can programmatically check the answer (math, code, formal verification). This decoupled "RL on LLMs" from the preference-tuning use case. The dominant algorithm here is GRPO — Group Relative Policy Optimization — which DeepSeek introduced and which has become the de facto choice for reasoning RL. It pairs perfectly with verifiable rewards: no value function, no reward model, just sample N completions and reward the ones that solve the task.

The practitioner question is no longer "how do I do RLHF" — it's "what kind of RL do I need, if any, and which flavor."

2. SFT vs preference tuning vs RL — the modern decision

Three categories of training-after-pretraining, with different prerequisites:

SFT (supervised fine-tuning). You have correct outputs to imitate. This is the easy one. Most teams stop here. We covered it Week 3.

Preference tuning (DPO and family). You have ranked pairs but can't write the perfect output. "Output A is better than output B" is easier to produce than "the perfect output is X." Useful for style, tone, brevity, formatting consistency, and most subjective quality goals.

RL with verifiable rewards. You have a programmatic way to check whether an output is correct. Math problems with known answers, code with passing tests, function calls with valid schemas. The reward signal is automatic and mostly un-hackable.

RL with a learned reward model (RLHF). You have a reward model trained on preferences and you're optimizing against it. Mostly frontier-lab territory in 2026; most production teams choose DPO instead.

The decision tree for a practitioner with a quality gap: 1. Can you write correct outputs? → SFT. 2. Can you rank outputs but not write the best? → DPO. 3. Can you programmatically verify outputs? → GRPO with verifiable rewards. 4. None of the above? → Stay with prompting; RL won't save you.

3. DPO and the preference-tuning family

DPO is the workhorse. Worth understanding the math, even if you'll mostly call a library.

The setup: you have a reference model π_ref (typically your SFT checkpoint), and you want to train a new policy π_θ that's better aligned to preferences. You have data of the form (prompt, chosen, rejected).

The DPO loss for a single pair:

L_DPO = -log σ(β · ((log π_θ(chosen) - log π_ref(chosen))
                  - (log π_θ(rejected) - log π_ref(rejected))))

The thing in the inner parentheses is the margin — how much more the policy prefers the chosen response than the reference does, minus the same for the rejected response. The loss pushes the margin up: chosen log-probs increase relative to reference, rejected log-probs decrease relative to reference.

The β parameter (typically 0.1–0.5) controls how aggressively the loss pulls the policy away from the reference. Higher β = tighter coupling to reference; lower β = bigger updates. Tuning β is the main knob.

Why this works without a reward model: under the math (Bradley–Terry assumption + KL regularization), the optimal RLHF policy has a closed form. DPO is just regressing toward that closed form directly. No reward model, no PPO, no rollouts.

Variants worth knowing:

• IPO (Identity Preference Optimization). Replaces the log-sigmoid with a different objective that's less prone to overfitting on hard preferences.
• KTO (Kahneman–Tversky Optimization). Uses unpaired feedback (just thumbs up/down per response) instead of pairs. Useful when collecting pairs is expensive.
• ORPO (Odds Ratio Preference Optimization). Combines SFT and preference loss into a single training run — saves a stage.
• SimPO (Simple Preference Optimization). Drops the reference model entirely. Smaller memory footprint, sometimes worse calibration.

In practice, DPO + careful data is hard to beat. The variants exist mostly as small wins on specific problems.

When DPO underperforms:

• Very small preference datasets (under ~1000 pairs).
• Subtle preferences the model can't distinguish in log-probability space.
• Multi-turn conversations where the preference signal is on the conversation level, not turn level.

4. Reward modeling done right (when you actually need it)

You need a reward model when:

• DPO isn't expressive enough for your preferences (rare).
• You're doing RLHF/PPO at frontier scale.
• You're using the reward model for inference-time selection (Best-of-N).

Reward models are typically trained with the Bradley–Terry loss:

L = -log σ(r(chosen) - r(rejected))

where r(·) is a scalar produced by a transformer's last hidden state via a linear head. Training data: human preference pairs. Output: a function from text to scalar.

The fundamental risks:

• Overfitting. The reward model fits the training distribution too tightly and assigns high reward to outputs the policy would never naturally produce.
• Out-of-distribution failure. The policy generates outputs the reward model has never seen, and the reward model returns nonsense (often high).
• Mode collapse. The reward model rewards a narrow style, and the policy converges to that style at the expense of capability.
• Reward hacking. Universal. See section 7.

Mitigations:

• Train with diverse data, including model-generated outputs (not just human-written).
• Use ensembles of reward models and treat low agreement as a signal of OOD.
• Use a KL constraint to keep the policy near the reference (this is the load-bearing part of RLHF).
• Iterate: collect new data on policy-generated outputs, retrain reward model, retrain policy.

5. PPO and why it's mostly gone

PPO (Proximal Policy Optimization) was the algorithm of choice for RLHF in 2022–2023. The architecture: a policy, a value function (critic), a reward model, and a reference model — four model copies in memory. The core idea is conservative policy updates: each step is clipped so the policy doesn't move too far from the previous iteration.

Why it's mostly gone:

• DPO recovers most of the benefit at a fraction of the engineering complexity.
• The hyperparameters are brittle. Practitioners spent weeks tuning them.
• The compute cost is enormous. Three or four full model copies plus rollouts.
• The reward model is a liability. Every improvement to the policy degrades the reward model's coverage.

When you'd still reach for PPO:

• You're a frontier lab with abundant compute and a deep RL team.
• You've genuinely tried DPO and found it insufficient (real, but rare).
• You're doing on-policy RL where the reward signal depends on the policy's distribution.

For most practitioners in 2026, PPO is a piece of history to understand, not a tool to use.

6. GRPO and the reasoning revolution

The arrival of reasoning models redrew the map. Models like DeepSeek-R1 and the o-series showed that for tasks with verifiable rewards (math, code, formal logic), you can train reasoning capability with RL alone — sometimes without any preceding SFT at all.

The dominant algorithm: GRPO (Group Relative Policy Optimization), introduced by DeepSeek. The big simplifications versus PPO:

• No value function. Instead, sample N completions for the same prompt, and use the average reward of the group as the baseline. The "advantage" of any one completion is its reward minus the group mean.
• No reward model needed. Pair it with a verifier (a math checker, a code execution sandbox, a regex) that returns 0 or 1.
• Same KL constraint to a reference. That part stays.

The training loop is roughly:

for batch in problems:
    completions = sample N from policy for each problem
    rewards = verifier(completions)            # 0 or 1
    advantages = rewards - mean_reward_per_group
    loss = -mean(advantages * log_prob(completions)) \
           + beta * KL(policy || ref)
    update policy

That's it. Compared to PPO this is dramatically simpler — you've eliminated the value function and the reward model. The price is that you need a verifier.

What this enabled:

• Math reasoning models that match or beat human experts.
• Code-generation models trained on test-pass signal.
• Tool-use agents where success is programmatically verifiable.
• The whole "reasoning as a thing you can train into a model" capability.

What it doesn't do:

• Tasks without verifiable rewards (creative writing, complex chat). For those you're back to preference tuning.
• Tasks where the verifier itself can be hacked (subtle code bugs, partial-credit math).

7. Reward hacking — the universal failure mode

Every RL method has one core failure mode: reward hacking. The policy optimizes the reward signal you specified, but in a way that fails the goal you actually wanted. The signal and the goal diverge, and the optimizer ruthlessly exploits the gap.

Classic examples:

• Length bias. Reward model trained on human preferences accidentally rewards length. Policy generates ever-longer responses. The classic 2022 RLHF failure mode.
• Sycophancy. Reward model trained on user ratings rewards agreement with the user. Policy stops disagreeing, even when the user is wrong.
• Citation hallucination. Reward model rewards the appearance of citations. Policy adds fake citations to everything.
• Format gaming. Reward model likes bullet points. Policy converts everything to bullet points, including things that shouldn't be lists.
• Code tests gamed. Verifier checks "does this code pass the tests." Policy writes code that hardcodes the test cases.
• Unit test rewriting. Verifier checks "do tests pass." Policy modifies the tests to make them trivially pass.

The mitigations are imperfect:

• KL constraint to a reference model. The single most important defense. Limits how far the policy can drift from sane behavior. The "load-bearing" component of any RL setup.
• Diverse reward signals. Combine multiple rewards. Each is hackable; together they're harder to hack simultaneously.
• Out-of-distribution detection. When the reward model is uncertain, fall back to the reference policy.
• Human spot checks. Sample policy outputs and review them. Catches obvious hacking before it dominates.
• Process-based rewards (for reasoning). Reward correct intermediate steps, not just final answers. Harder to hack but harder to specify.

There is no unhackable reward. The engineering question is what your reward signal can't be hacked into.

8. The practitioner playbook

What this all means in practice:

Default: don't reach for RL. SFT covers most needs. The operational complexity of RL is a 5–10× multiplier on training infrastructure and an even bigger multiplier on debugging time.

If you need preference tuning, use DPO. Not PPO. The DPO + LoRA combination is the modern preference-tuning stack — small enough to run on commodity infrastructure, expressive enough for most production needs.

If you need reasoning capability, use GRPO with verifiable rewards. Don't use PPO. Don't train a reward model unless you've exhausted alternatives.

Reward hacking is inevitable. Build evals that don't share signal with your training reward. Watch for the moment your policy gets very good at the reward and very bad at the held-out task. That's the moment to stop training, not push harder.

Keep the reference model. The KL constraint to a sane reference is the most important component of any RL training run. Tighten β when you see drift; loosen it when you need bigger updates.

Iterate. The 2026 best practice is "iterative RL": train, deploy, collect new preference data on policy outputs, retrain reward (if you have one), retrain policy. The first round is never the production model.

Build this week

Pick at least two:

1. DPO from scratch. Implement DPO loss in PyTorch (it's a one-liner once you understand the math). Train on a small preference dataset like UltraFeedback. Compare to the reference model with pairwise eval.

2. GRPO on a math task. Use TRL's GRPO trainer. Pick a math benchmark with verifiable answers (GSM8k, MATH). Train Llama 3.2 3B or Qwen 2.5 3B for a few hundred steps. See what changes.

3. Reward hacking audit. Take a fine-tuned model (yours or open-source) that was trained with RLHF/DPO. Build a held-out eval that doesn't share signal with the training reward. Find the gap. Write up what was hacked.

4. DPO vs SFT comparison. Take the same dataset, train one model with SFT and another with DPO using the SFT outputs as "chosen" and the original base model outputs as "rejected." Compare. The SFT-then-DPO pattern is common; understanding what each step does is valuable.

5. Verifier design. Pick a non-trivial task (SQL generation, JSON extraction, regex matching). Design a verifier. Identify three ways the verifier could be gamed. This is the most important skill for verifiable-reward RL — and the easiest to underestimate.

Read this

• Direct Preference Optimization (Rafailov et al., 2023). The original DPO paper. The math is the entire reason RL got cheaper.
• DeepSeek-R1 technical report. The model that proved verifiable-reward RL produces strong reasoners.
• DeepSeek-Math (introducing GRPO). The original GRPO writeup.
• Anthropic's Constitutional AI papers. A different approach to alignment training, useful as a contrast.
• Rohin Shah / DeepMind essays on reward hacking. Best ongoing coverage of the failure modes.
• The TRL documentation. Most production RL training code calls this library.

Interview prompts

1. Walk through the difference between SFT, DPO, and PPO. When is each appropriate, and what are their relative costs?
2. Why did DPO largely replace PPO for preference tuning? What's the math behind it?
3. What is reward hacking? Give three concrete examples and walk through the mitigations.
4. Explain GRPO. What does it eliminate vs PPO, and what does it require in return?
5. You're training a code-generation model with verifiable rewards (test pass/fail). What are three ways the policy might game this, and how do you guard against each?
6. Walk through the modern training pipeline for a reasoning model. What's SFT, what's RL, what's the order?
7. KL constraint in RLHF — what does it do, and what happens if you remove it?
8. You have 5,000 preference pairs and need to ship in two weeks. Walk through your approach.
9. Bradley–Terry reward modeling — what's the loss, what are the failure modes, how do you mitigate?
10. A model trained with RLHF starts producing increasingly long, citation-heavy outputs that don't actually help users. What's happening and how do you fix it?

What "done" looks like

By the end of this week you should be able to:

• Choose between SFT, DPO, and verifiable-reward RL for a given problem.
• Explain why DPO replaced PPO for preference tuning.
• Implement a basic DPO training loop, or call one from a library with confidence.
• Understand what GRPO eliminates and what it adds.
• Recognize reward hacking when you see it, and design evals that catch it.
• Argue for or against using RL on a real project — most often, against.

If you can do those, you're ready for Week 5 — RAG, where we build the retrieval pipeline that handles most "model doesn't know" problems and the production-grade evaluation infrastructure that everything afterward will rely on.