Interactive · Week 12 · Section 2

3am. Something's wrong. What do you do?

Three incident scenarios — silent quality regression after deploy, agent loop cost explosion, prompt injection wave — each with four interventions ranging from "best response" to "this is why we keep losing trust." Pick the scenario, watch the metrics drift, choose your intervention, see the blast radius. The visceral lesson: 80% of LLM incident response is having the right instinct in the first 10 minutes. Strong on-call engineers know which lever to pull before the page even fires; weak ones discover their playbook in the middle of the incident.

What separates strong on-call from weak

Strong on-call has internalized the playbook. They know which intervention fits which incident class before the page fires. The first 10 minutes are about pattern-matching, not investigation.
Containment beats diagnosis early. The fastest action that reduces blast radius is usually the right first step, even if it's not the final fix. Rollback/disable/throttle. Diagnose afterward.
Symptom vs cause matters. Tightening verifiers when the model regressed treats the symptom — costs go up, false-rejects rise, and the underlying problem persists. Rolling back addresses the cause.
Defense in depth is built before incidents. The team that has feature flags, automatic rollback, per-task budgets, input filters — they don't run faster during incidents, they run *less*. The architecture does the work.
Postmortems are the multiplier. The blast radius of incident #1 is the cost. The blast radius of incident #2 is what you didn't learn from #1. Strong teams write the lesson into the system before #2 arrives.

3am. Something's wrong. What do you do?

Production dashboard · 60-minute window

Intervention menu · pick one

Incident outcome

What separates strong on-call from weak