Interactive · Week 8 · Section 6

How noisy is your LLM judge?

Three raters grade the same 50 responses: two humans and an LLM-as-judge. Move the subjectivity slider — from "did the response include the correct refund amount?" (factual, binary) to "was this response delightful?" (subjective). Watch how all three raters fall apart together as subjectivity rises, and how the LLM judge falls apart faster than humans. The visceral lesson: LLM-as-judge isn't free, and the noise depends entirely on what you're evaluating.

Task subjectivity 0.50

0 · binary fact-check 1 · subjective quality

Example task at this subjectivity

—

Human A vs truth

—

accuracy

Human B vs truth

—

accuracy

LLM judge vs truth

—

accuracy

Judge below human

—

accuracy gap

Per-sample verdicts · 50 samples per rater

green = matches truth · red = disagrees

Human A moderate-noise rater —

Human B correlated with Human A —

LLM judge independent · noisier —

Pairwise agreement

how often any two raters give the same verdict

Human A vs Human B

—

inter-human agreement (the ceiling)

Human A vs LLM judge

—

judge approximating human A

Human B vs LLM judge

—

judge approximating human B

What's happening here

Each sample has a deterministic ±1 ground-truth signal. Each rater's "perception" = truth + (rater-specific noise) × subjectivity_factor, thresholded at 0 to produce a binary verdict. Human A and Human B share 0.5 noise correlation (raters share biases). LLM noise is independent and ~30% larger in magnitude than human noise (representing imperfect calibration). At subjectivity = 0, all raters converge on truth; at subjectivity = 1, raters diverge — humans more correlated with each other than with the LLM. This is a model, not measurement; real numbers depend on task and judge model. Source-of-truth lives as <JudgeAgreementVisualizer/> in the curriculum codebase.