A clean pass rate is not calibration

Sixteen out of sixteen. Every summary my eval graded came back faithful, first try. That number should make you suspicious of the eval, not proud of the system, and it’s the reason this post exists: a same-day Bedrock experiment on my own blog, built to find out what it actually takes to trust an LLM-as-judge score before you wire one into anything that matters.

The setup

Sixteen posts, two Bedrock models in my own AWS account, one small repo: do-you-trust-it-evals.

• Claude Haiku 4.5 (us.anthropic.claude-haiku-4-5-20251001-v1:0) writes a 2-3 sentence summary of each of my 16 published posts, instructed to use only claims present in the source.
• Claude Sonnet 4.5 (us.anthropic.claude-sonnet-4-5-20250929-v1:0) grades each summary against its source: faithfulness 1-5, completeness 1-5, a list of unsupported claims, and a PASS/FAIL verdict. The rubric treats a conditional claim stated as a universal (“sometimes” becoming “always”) as a hallucination, not just an invented fact.

Three separate checks feed the same judge before I’d trust it:

flowchart TB
    posts[16 published posts] --> sum[Claude Haiku 4.5 summarises each]
    sum --> judge[Claude Sonnet 4.5 judges faithfulness]

    subgraph r1["Round 1 — baseline"]
        direction LR
        b1[16 summaries] --> b2[16/16 PASS]
    end

    subgraph r2["Round 2 — calibration"]
        direction LR
        c1[5 single-variable corruptions] --> c2[5/5 caught, correctly named]
    end

    subgraph r3["Round 3 — self-consistency"]
        direction LR
        s1[Same input, 4 repeat calls] --> s2[Stable verdict + score]
    end

    judge --> r1
    judge --> r2
    judge --> r3
    r1 --> gate{Do you trust it?}
    r2 --> gate
    r3 --> gate
    gate --> ship[Only then: gate a real deploy]

Round 1: the baseline that should worry you

Cold run, no tuning: 16 out of 16 summaries passed, faithfulness 5 across the board. I didn’t take the judge’s word for it. The summary for the CDK post claims “nine specific pitfalls”, and the source has exactly nine, numbered ## Gotcha #1 through ## Gotcha #9. The summary for the prompt-caching post claims a 99.9% hit ratio on Nova Pro and a 78% billing reduction on Sonnet, and both numbers sit in that post’s own measurement table. The summaries were genuinely faithful. This wasn’t a rubber stamp catching nothing because there was nothing to catch.

A 100% pass rate proves the eval didn’t break on the easy case. It proves nothing about whether the judge would catch a hard one. In my experience most eval writeups stop here and call it done.

Round 2: calibrate with a single variable

So I stopped grading the summariser and started grading the judge. Five corrupted summaries, each with exactly one injected error and nothing else touched, so a FAIL verdict can only be explained by that one change:

Injection	Post	What changed
Fabricated exact number	three-things-bedrock-workload	One sentence inserted
Modality broadening	agents-need-a-harness	One clause replaced
Misattributed real number (from a different post)	part-6-cost-performance-prompt-caching	One clause replaced
Fabricated named entity	llm-is-not-a-security-boundary	One clause appended
Count inflation (nine → twelve)	part-2-cdk-infrastructure-bedrock-agentcore	One word changed

5 out of 5 caught, and in every case the judge’s own hallucination list named the exact injected error, not something else in the summary. On the CDK case it went further than I’d engineered: it also flagged that my corrupted summary had quietly compressed “Match.arrayWith being order-sensitive” into a vaguer “array matching order sensitivity”, a real precision loss I hadn’t deliberately planted. Two of the five categories are the ones I’d expect to be hardest for a judge: modality broadening and the misattributed-but-real number, because nothing in either summary is a “fake fact” sitting in isolation, they’re both true statements wearing the wrong coat.

The single-variable constraint matters more than it sounds. A corrupted test case that changes three things at once and gets flagged tells you something failed, not which change caused it. If you can’t point to the one thing you changed, you don’t have a calibration result, you have a guess with a percentage attached.

Round 3: does the judge agree with itself

Calibration tells you the judge can tell known-bad from known-good. It says nothing about whether the same input gets the same grade twice. Published research says you can’t assume it does: Fiona Lau’s “Same Input, Different Scores: A Multi Model Study on the Inconsistency of LLM Judge” (2026) found substantial score variability across judge models, “despite expectations of stability at temperature=0,” and flagged completeness scoring as showing the largest fluctuations of the metrics tested. That’s the specific axis worth checking, not just the pass/fail verdict.

A clean input repeating cleanly is the least informative thing to test, so I ran two: the unmodified, round-1-passing summary for three-things-bedrock-workload, and the round-2 corrupted case for agents-need-a-harness (the modality-broadening one, a genuinely borderline call). Each went back through the same judge four more times, same rubric, temperature 0, both faithfulness and completeness tracked.

Easy case: verdict PASS, faithfulness 5, completeness 5, all four runs, matching Round 1. Hard case: verdict FAIL, faithfulness 2, completeness 3, all four runs. Both scores held steady on both inputs, on this judge, on this day, on the exact metric the cited paper flags as the shakiest.

Why three rounds, not one

A per-item LLM-as-judge score means three different things, and most teams only check the first:

1. Does it pass a clean input? (Round 1.) Necessary, proves almost nothing on its own.
2. Does it catch a broken one, and can you prove which break it caught? (Round 2.) This is where a multi-variable corrupted test quietly lies to you, by giving you a FAIL you can’t attribute.
3. Does it hold the same grade on a re-run? (Round 3.) In my experience the one most pipelines skip entirely, because it costs nothing to skip and nothing visibly breaks until it does.

If you’re using an LLM judge to gate a deploy or score a golden dataset, a single pass/fail number per item, checked once, is a thinner signal than it looks like.

What I now do

• Treat a 100% pass rate as an instruction to calibrate, not a result to report. If your eval never fails, that’s a claim about your eval, not your system.
• Corrupt one variable per test case, and check it. Diff the corrupted input against the original before you run it. If more than one thing changed, redo it; a FAIL you can’t attribute to a specific cause isn’t evidence.
• Cover more than one failure category. A fabricated fact, a scope-broadened claim, and a misattributed-but-real number fail differently. A judge that’s fine on one can still miss another.
• Re-grade the same unmodified input more than once before trusting a single run. It’s a few extra API calls. Published research says self-consistency isn’t guaranteed even at temperature 0.

The honest caveat

This is a small experiment: 16 posts, one judge model, one rubric, five corrupted cases, eight repeat runs across two inputs, one day. I’m not claiming Sonnet 4.5 is reliable as a judge in general, or that these results hold at production scale, on other content, or with other models. What I checked, and what I’d stand behind, is narrower: on this specific pairing of models, this rubric, this content, the judge discriminated correctly across five distinct failure categories and held its grade, on both an easy and a borderline input, across four repeats each.

One more limit worth naming directly: every corruption in Round 2 was a benign, accidental-looking error, the kind a well-meaning summariser might actually produce. None of it tests whether the judge, or the summariser feeding it, can be manipulated by adversarial content sitting in the source document itself, which is the sharper question if you’re using this pattern to gate anything security-relevant rather than just catching sloppy summaries. That’s a different, harder experiment, and this post isn’t it.

Clearing three rounds on benign failure modes is the floor I’d want before trusting an LLM judge with anything real, not the ceiling.

Full methodology, all three scripts, and the raw JSON for every round are in the repo: github.com/rajmurugan01/do-you-trust-it-evals.

---

Series

This is Part 1 of Do You Trust It?, a series on how you actually know an AI system is good once it’s live, not according to the dashboard. The last series, Production AI, Honestly, measured and caged the probabilistic layer. This one asks what comes after: how you know the thing you built is actually trustworthy, and how much testing that confidence really costs.

If you’re wiring an LLM judge into a deploy gate or a golden-dataset pipeline and want a second pair of eyes on it, happy to compare notes. Find me on LinkedIn or via rajmurugan.com.

Next in Do You Trust It?: offline evals versus live production signals, and the regression gate that’s supposed to catch a silent quality drop before your users do.