Fine-tuning

Why this week is short relative to its importance. Fine-tuning dominated practical LLM work in 2022–2023. By 2026, base models have gotten so capable and prompting/RAG have improved so much that fine-tuning has been demoted to a specialized tool — the option of last resort, but irreplaceable for the 10% of problems where it's the right answer. Most teams that fine-tune today either didn't need to (and made things worse) or genuinely needed to but can't articulate why. Both errors are common. The framing is harder than the implementation.

1. The fine-tuning decision

Default: don't fine-tune. When you encounter a quality gap on an LLM task, work the ladder in order:

1. Better prompting. System prompt revisions, few-shot examples, structured outputs. Free, fast, often sufficient.
2. RAG. If the gap is "the model doesn't know X," retrieve X. Week 5.
3. Stronger model. Sometimes the cheapest fix is just upgrading from Haiku to Sonnet, or from open-weight to frontier. Run the eval before assuming it's a prompting problem.
4. Fine-tuning. The model has the capability but the wrong default behavior, or you need a smaller model to match a bigger one's quality, or the task is so specialized that prompting can't bridge the gap.

Each rung adds operational complexity by roughly 5–10×. Prompting is a string. RAG is an indexing pipeline plus a retrieval system. Fine-tuning adds dataset curation, training infrastructure, evaluation, deployment, and ongoing maintenance against base-model drift. Don't take that step until you've exhausted the cheaper options.

Indicators that fine-tuning IS the right answer: - The model has the capability but a wrong default style or tone you can't prompt your way out of. - The task uses a specialized vocabulary or output format the base model handles inconsistently. - You need a smaller, faster model to match a larger one's quality (cost or latency reasons). - You're shipping a function-calling agent at scale where structured-output reliability matters. - You've exhausted prompting and your evals show a stable, reproducible gap.

Indicators that fine-tuning is NOT the answer: - You don't have evals. Build them first. You cannot fine-tune productively without them. - You haven't tried structured outputs or schema-constrained decoding. - You haven't enabled prompt caching. - Your problem is "the model doesn't know our internal facts" — that's RAG. - You have fewer than ~500 high-quality, hand-checked examples. - You're hoping fine-tuning will fix accuracy on factual recall. - You're trying to "fix hallucinations." Fine-tuning rarely does this; structured outputs and RAG do.

The hardest version of this is "I tried prompting for a few hours, it didn't work, let me fine-tune." That path leads to a fine-tuned model that performs worse than the original prompting attempt would have if you'd given it another day. Most prompting "failures" are fixable in another two hours of prompt iteration plus a real eval suite.

2. The flavors of fine-tuning

Pick the right one and the rest is mechanical.

Continued pre-training. Adds new general knowledge to a base model by training on raw unlabeled text. Rare in practitioner work. Almost always what you want is one of the supervised flavors below.

Supervised fine-tuning (SFT). Train on input/output pairs in your task's format. The default form of fine-tuning, and what most people mean when they say "fine-tuning." We'll spend most of this week on SFT.

Preference tuning (DPO and family). Train on pairs of preferred vs rejected responses. Useful when you can rank outputs but can't write the perfect output. We cover the RL-for-LLMs family in Week 4.

RLHF with a reward model. Full reinforcement learning. Almost always overkill outside frontier labs; the engineering cost is enormous and DPO recovers most of the benefit at a fraction of the complexity.

For most production needs in 2026: SFT first, then DPO if you have ranked data, almost never RLHF.

3. Full fine-tuning vs LoRA vs adapters

For SFT, you have three options for what you update.

Full fine-tuning. Update every weight in the model. Most expressive, most expensive. For a 70B model in fp16, weights and gradients alone need ~280 GB of GPU memory, before optimizer states. In practice that's an 8-GPU H100 node minimum, training over days. Used when LoRA quality is insufficient and the budget exists.

LoRA — Low-Rank Adaptation. Train small low-rank "adapter" matrices that get added to existing weight matrices at inference time. Trains 0.1–1% of the parameters, recovers 90–99% of full fine-tuning quality on most tasks. Default choice in 2026.

QLoRA. LoRA on top of a 4-bit-quantized base model. Lets you fine-tune a 70B model on a single H100, or an 8B model on a consumer GPU. The quality loss from 4-bit base is usually negligible because the LoRA adapters compensate. This is what most practitioners actually use today.

Other adapter methods (Prefix tuning, IA3, etc.). Similar spirit to LoRA, less commonly used in 2026. LoRA won the standard wars.

The math: LoRA on a single linear layer of dimensions [d_in × d_out] with rank r adds r × (d_in + d_out) parameters instead of d_in × d_out. For a typical attention or MLP layer with d=4096, rank-16 LoRA replaces 16M parameters with 130k — a 100× reduction. You can apply LoRA to any subset of the model's linear layers. The conventional setup applies it to all attention projections (q, k, v, o) and all MLP layers (gate, up, down).

4. The dataset is where it lives or dies

Fine-tuning succeeds or fails on data quality. Three principles, each violated by most failed fine-tuning projects:

Quality beats quantity. 500 hand-curated, format-perfect, edge-case-covering examples beat 50,000 noisy ones. The dominant failure mode is "we scraped 100k examples from production logs" — production logs contain wrong outputs, misformatted outputs, and outputs that the user disliked. Training on those teaches the model to be wrong consistently.

Coverage beats volume. When you find that your fine-tuned model fails on a specific input pattern, the answer is usually not "more data" — it's "more data on this specific pattern." Five new examples that target the failure mode beat 5000 random examples that don't.

Format consistency. The model's chat template, special tokens, system prompt presentation — all of these need to match exactly between training and inference. The single most common deployment bug for fine-tuned models is that the inference-time chat template doesn't match the one used at training time. Always tokenize your training data through the same code path that will tokenize at inference.

Synthetic data is now standard. Distill from a stronger model: have GPT-5 or Opus generate input/output pairs in your task format, then fine-tune a smaller model on those. With careful filtering, this works astonishingly well. Most of the open-source fine-tuned models you respect were trained largely on outputs from frontier closed models. Whether this is legal under each provider's TOS is a separate (and active) question.

Hold out a test set. Set aside 10–20% of your data and never look at it during training. This is the only honest way to know whether you've improved. Most teams skip this and regret it within a month.

5. Evaluation before, during, and after

If you don't have an eval suite when you start fine-tuning, stop and build one first. This is non-negotiable. Without evals you cannot tell whether your fine-tune helped or hurt, you cannot detect catastrophic forgetting, and you cannot make a confident production deployment decision.

What an eval suite needs:

• A held-out test set. Examples your model has never seen. Same distribution as production traffic, ideally hand-curated.
• Pairwise comparison with the base model. For each test example, run base and fine-tuned, then have an LLM-as-judge (or human) pick which is better. Track win/lose/tie rates.
• Behavioral regression tests. Specific inputs the model used to handle correctly that it must continue to handle correctly. The fine-tune is a regression risk — guard against it.
• Format compliance. If your task requires structured output, score format validity separately from content quality.
• Catastrophic forgetting checks. Sample 50 prompts from the base model's general capability surface (math, coding, reasoning) and verify the fine-tune still answers them well. This is the most-skipped test, and the most painful when you skip it.

A common pattern worth internalizing: build evals first, then prompt-engineer to get as far as you can on those evals, then decide whether to fine-tune based on the remaining gap. You'll often find prompting got you 95% of the way and fine-tuning isn't needed.

6. The 2026 toolchain

The fine-tuning ecosystem stabilized around a small set of tools. Pick from the right tier for your situation.

Self-hosted training: - Hugging Face TRL + PEFT. The canonical libraries. Highly configurable, somewhat verbose. What every other tool wraps. - Axolotl. Opinionated YAML-driven trainer built on top of TRL. The default for most open-source fine-tuning projects. - Unsloth. Optimized kernels, dramatically faster training (often 2–5×). Constrained to specific architectures but the constraints loosen each release. Use this when you can.

Managed training: - Together AI, Modal, Anyscale, RunPod. GPU rentals or fully managed training jobs. Best for one-off projects or teams without ML infrastructure. - OpenAI fine-tuning API, Anthropic fine-tuning (Claude on Bedrock). Hosted fine-tuning on closed models. Convenient, expensive, opaque. Useful when you want a fine-tuned closed-model and don't want to manage anything.

Decision rule: If you'll fine-tune more than once a quarter, invest in the self-hosted toolchain. Below that, managed is faster to results.

7. Serving fine-tuned models

This is where many fine-tuning projects quietly die: the model trains fine, but serving it economically is harder than expected.

Single LoRA. Trivial. Load the base model, apply the adapter, serve. vLLM, TGI, SGLang all handle this directly. Cost is identical to serving the base model.

Multi-LoRA serving. Serve hundreds of LoRAs from a single base model on the same GPU, switching adapters per request. This is the architecture that makes per-tenant fine-tuning economically viable. vLLM's LoRA support, S-LoRA, and Punica all enable this. The ops trick: keep the base model in GPU memory, swap adapters in fast memory or load on demand.

Full fine-tuned models. No sharing. You're serving N copies of the model for N customers. Expensive at scale — this is one of the strongest arguments for LoRA over full fine-tuning when you have multi-tenant requirements.

Cost model that works in 2026: shared base model + per-tenant LoRA + prompt caching on the system prompt = sustainable economics for personalized agents at scale. Without LoRA, the math doesn't close.

8. Common pitfalls (and the failure modes you'll actually hit)

Catastrophic forgetting. The model gets great at your task and worse at everything else. Especially common with small datasets and high learning rates. Mitigations: lower learning rate, smaller LoRA rank, mix in a small fraction of general-purpose examples, evaluate on capability-surface tasks after training.

Distribution shift. Your training data was clean and well-formed; production traffic isn't. The model performs great in eval and poorly in the wild. Fix: sample real production traffic into your training set; evaluate against production-shaped distributions.

Format leakage. If every training example starts with User: and ends with Assistant:, the model learns those tokens are mandatory. Production inputs that don't have them confuse the model. Fix: vary the format, or normalize aggressively.

Quality drift over time. Fine-tuned models age fast. Base models keep improving; your fine-tune doesn't. The breakeven moment — when the latest base model with good prompting beats your year-old fine-tune — usually arrives within 6–12 months. Plan for re-training, or plan for replacement.

The "good model" trap. Six months ago you fine-tuned and shipped. Today, the new Sonnet beats your fine-tuned model on the same task with prompting alone. This is not a hypothetical — it has happened to most teams that fine-tuned in 2023. Default to fine-tuning the latest base, and assume your work has a half-life.

Build this week

Pick at least two:

1. Build evals before fine-tuning. Pick a task you suspect needs fine-tuning. Build the eval suite first: held-out test set, pairwise comparison, behavioral regressions. Run the base model against it and write up how far prompting alone gets you. Often this is the entire project.

2. LoRA on a small model. Pick Llama 3.2 1B or 3B. Use Unsloth or Axolotl. Fine-tune on a small dataset (1–2k examples) for a clearly-bounded task: classification, format conversion, or specific style. Serve it locally and run your eval suite.

3. QLoRA on a bigger model. Same exercise on Llama 3.1 8B with QLoRA on a single 24GB consumer GPU. The point of this exercise is to feel how much QLoRA opens up — fine-tuning a usable 8B model on hardware you might own personally.

4. Reproduce a published distillation. Pick a paper or open dataset where someone fine-tuned a small model on outputs from a larger one. Reproduce the experiment. Compare quality across base, distilled, and the original teacher.

5. Multi-LoRA serving. Train 2–3 LoRAs on different tasks (different chat styles, different formats). Serve them with vLLM's LoRA support and benchmark switching cost.

Read this

• LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021) — the original paper. Read for the math.
• QLoRA (Dettmers et al., 2023) — quantization-aware fine-tuning. Read for the engineering.
• The Hugging Face PEFT documentation — practical reference for everything LoRA-adjacent.
• Anthropic's fine-tuning best practices (Claude on Bedrock docs) — opinionated, well-engineered guide. Even if you're not using Claude, the principles transfer.
• Eugene Yan, Don't fine-tune that model — the canonical framing of when fine-tuning is wrong.
• Karpathy's tokenization tutorial — not about fine-tuning per se, but you'll inevitably hit tokenizer issues during fine-tuning, and his treatment is the clearest in the field.

Interview prompts

1. Walk through the decision tree from "we need better quality" to "fine-tune this model." When does fine-tuning win, and what's the cheapest cost-effective alternative you'd try first?
2. LoRA vs full fine-tuning — when is each appropriate? What's the math behind LoRA's parameter efficiency?
3. Walk through how you'd build the eval suite for a fine-tuning project. What's in it, what's not, and how do you guard against catastrophic forgetting?
4. What is catastrophic forgetting and how do you detect/mitigate it?
5. You're given 100k examples of customer support transcripts to fine-tune on. What do you do first?
6. QLoRA — what does it cost in quality, and what does it gain you in operational flexibility?
7. Multi-LoRA serving — describe the architecture and what makes it economically viable.
8. A fine-tuned model that worked great six months ago is now underperforming the base model with good prompting. What do you do?
9. You have a fine-tuning provider that charges X/M tokens for training and Y/M tokens for serving. Walk through the cost model versus self-hosted training.
10. Synthetic data from a frontier model — when does this work, when does it fail, and what are the legal/ethical considerations?

What "done" looks like

By the end of this week you should be able to:

• Look at a quality gap and choose correctly between prompting, RAG, and fine-tuning.
• Build an eval suite for a fine-tuning project before you start training.
• Articulate the LoRA math and pick a reasonable rank for a target model.
• Know when QLoRA is appropriate and what it costs you.
• Set up serving for fine-tuned models, including multi-LoRA where it applies.
• Recognize the "good model trap" — that fine-tunes age, and you should plan for replacement.

If you can do those, you're ready for Week 4 — RL for LLMs, where we cover DPO, PPO/GRPO, and the modern reasoning-model training stack.