Post 03
Calibrating a judge
A judge that agrees with you 90% of the time still reports a biased pass rate. The calibrate loop measures that bias against human labels and corrects for it — with strict train/dev/test discipline so the final number is honest.
Why "it agrees most of the time" isn't enough
An auto-judge is itself a noisy classifier. If you just count how often it says "Pass" and report that as your pass rate, you inherit every false pass and false fail it makes. Two judges with the same accuracy can report different rates depending on which way they're wrong. So calibration has two jobs: push the judge's agreement with humans as high as it'll go, and then mathematically correct whatever error remains.
Three splits, one rule
The labeled dataset is split three ways, stratified by label balance. The one rule that makes the whole thing valid: examples never leak across splits, and the test set is measured only once.
Train
10–20%Only if the judge is LLM-based and uses few-shot examples. The examples it learns from are drawn from here — never from dev or test.
Dev
40–45%The iteration set. Measured every loop, refined against, measured again. You are allowed to overfit to this — that's what it's for.
Test
40–45%Touched exactly once, at the very end. The only unbiased read on how the judge actually performs. Peeking here during iteration invalidates it.
The two numbers that matter
Every measurement is a confusion matrix of judge verdict vs human label. From it come the two rates the loop targets — treating "Fail" as the positive class the judge is trying to catch.
Of the real failures, how many the judge caught. Low TPR = too strict's opposite — it's missing failures. Target ≥ 0.90.
Of the real passes, how many the judge let through. Low TNR = it's flagging good outputs as failures. Target ≥ 0.90.
The loop reads the gap and refines
Each dev-set measurement points at a specific fix. A refinement subagent gets the misclassified slices and the directive below, proposes an edit to the prompt or code, and the loop re-measures. It stops on the first of: both targets hit, a two-iteration plateau, max iterations, or a refiner that has nothing left to change.
Too strict
The judge fails things it should pass. Inspect the false fails; clarify the Pass definition, add Pass examples, loosen overly strict rules.
Too lenient
The judge passes things it should fail. Inspect the false passes; strengthen the Fail definition, add Fail examples, tighten the rules.
Under-powered
Recommend a more capable model or more code structure — but surface it to the user. The loop never silently swaps the model.
Too coarse
Two flat iterations means the failure mode is doing too much. Recommend decomposing it into atomic sub-checks, and halt.
The math: Rogan-Gladen correction
Once the test TPR and TNR are known, you can recover the true rate from the
judge's biased observed rate. If p_obs
is the fraction the judge flags, the corrected true rate
θ̂ is:
Worked through with a judge that tested at TPR 0.92, TNR 0.88, observing a failure rate of 0.80:
The raw 0.80 understated the truth; corrected, the real failure rate is ~0.85. A bootstrap over the test labels gives a 95% confidence interval (here roughly [0.78, 0.91]) so you report a range, not a false-precision point. Skipping this correction and quoting the raw judge rate is the most common way an "evaluated" number lies.
Reading the confidence interval
The corrected rate is still just one number computed from a finite test set — measure a different handful of traces and you'd get a slightly different answer. The confidence interval is how we put honest error bars on it. It's computed by bootstrapping: take the test set, resample it with replacement to make a new "what-if" test set of the same size, recompute TPR, TNR and the corrected rate on that resample, and repeat ~2,000 times. You end up with 2,000 plausible answers; the middle 95% of them — the 2.5th to the 97.5th percentile — is the [low, high] range you report.
The width of that range is the real signal of judge quality. Two judges can land on the same point estimate yet deserve completely different levels of trust:
Tight interval
High TPR/TNR on a healthy test set (~100 labels). The resamples all agree, so the band is narrow — a few points wide. You can act on this number: "≈85%, give or take 3."
Wide interval
Same headline 0.85, but low TPR and/or too few test labels make the resamples scatter. The truth could be anywhere from "barely passing" to "nearly perfect" — the number can't be trusted to decide anything.
Two things tighten the band: more labeled test
data (below ~60 labels intervals get wide fast) and a
higher TPR — because the correction divides by
TPR + TNR − 1, a low TPR amplifies
every bit of sampling noise into a wider interval. A judge that only reports a tight interval is
one whose number you can build on.
Then it's certified — or it isn't
The judge's metadata records exactly how it ended, and only a certified judge is trusted by the downstream ablate loop. This is the judge-certification checkpoint — you see these numbers before the judge is believed.
certified
Both targets hit on the test set. Trusted.
certified_minimum
Cleared the floor (≥ 0.80) but not the target. Usable with caution.
stalled
Couldn't clear the floor. Returns a stall reason instead of a false pass.