A score is only as trustworthy as the judge that produced it

An automated judge is a measurement instrument, not ground truth. If the instrument prefers the first answer, rewards repetition, mistakes polished formatting for quality, or cannot verify a difficult factual claim, a rising dashboard score may describe the judge rather than the product.

The warning is not hypothetical. the MT-Bench study documented position and verbosity bias. Style Outweighs Substance reported that judges in its alignment-benchmark setting implicitly reweighted criteria and could be more forgiving of factual errors than conspicuous stylistic violations. Research on format bias has also found preferences associated with lists, links, bold text, emojis, and length.

Difficulty matters too. JudgeBench constructed 350 pairs in which one response was objectively correct and the other contained a subtle error. In its reported historical setup, a vanilla GPT-4o judge achieved 50.86% overall accuracy, while an Arena-Hard prompt using GPT-4o achieved 56.57%. Those figures are not universal model rankings; they are evidence that a seemingly capable judge can struggle on a difficult, specifically constructed test. See the JudgeBench paper.

In this lab, calibration means empirical validation against blinded human labels and invariance tests. It is not formal probability calibration unless the judge also emits a meaningful confidence estimate.

What would make a judge trustworthy?

A useful judge should satisfy five operational properties:

• Validity: it rewards the qualities named in the rubric, especially correctness and instruction following.
• Invariance: irrelevant changes in order, formatting, or harmless wording do not flip the verdict.
• Discrimination: a small but material factual error is detected even when the incorrect answer looks polished.
• Stability: repeated runs and model updates do not create unexplained discontinuities.
• Auditability: prompts, model identifiers, source packets, parses, failures, and human labels are retained.

Agreement alone is insufficient. A judge can agree with humans on easy examples while failing precisely where a regression suite is supposed to protect the product. The report therefore needs overall results, difficulty slices, manipulation slices, and uncertainty intervals.

The threat model

A large systematic study found that position effects vary by judge and task and are strongly affected by the quality gap between candidates. That supports testing order on every pair, not on a small convenience sample. See Judging the Judges.

The calibration lab: a preregistered design

Controlled pair bank
        ├── blinded human labels ──┐
        ├── judge 1: holistic ─────┤
        ├── judge 2: decomposed ───┼── metrics by factor and difficulty
        └── judge 3: reference-first┘        └── deployment gates + audit queue

1. Build 120 base tasks and 560 controlled pairs

Use 40 evidence-seeking tasks, 40 quantitative or logical reasoning tasks, and 40 procedural or synthesis tasks. Balance the collection across 40 easy, 40 medium, and 40 hard items. For evidence-seeking and synthesis tasks, create length, style, citation, correctness, and identity-control pairs. For reasoning tasks, omit citation pairs unless external evidence is genuinely required. This produces 560 planned pairs.

Start from one reviewed canonical response. Create each variant by changing only one declared dimension. Store a character-level diff and a short mutation note. A length variant may repeat or expand existing content but may not add a new claim. A style variant may change headings, bullets, tone, or sentence structure but not evidence. A correctness variant should change one consequential fact or step. This is a causal test set, so uncontrolled rewrites are defects.

Citation pairs should use a bundled source packet rather than fabricated links. The valid response must cite evidence that supports the nearby claim. The degraded response may cite an irrelevant passage, a source that does not entail the claim, or omit support for a material statement. Citation presence is not citation quality: ALCE evaluates answer correctness and citation quality as distinct dimensions.

Each record should contain base_id, pair_id, domain, difficulty, manipulation, candidate identities, display order, expected relationship, source-packet IDs, mutation diff, and provenance. Keep candidate identity separate from the A/B display label.

2. Collect blinded human labels

Hide the manipulation type, expected label, generator, and candidate provenance. Randomize display order. Two qualified annotators independently choose A, B, tie, or abstain and record concise reason codes: correctness, instruction following, citation support, completeness, clarity, or style. Use a third annotator to adjudicate disagreements.

The rubric should be hierarchical: first instruction compliance, then factual and logical correctness, then citation support where required, then completeness and clarity. Style breaks a tie only when the higher-priority dimensions are equivalent. Difficult factual, mathematical, legal, medical, or scientific items require an annotator with appropriate competence or an objectively checkable reference.

Publish raw agreement and chance-adjusted agreement, not just adjudicated labels. Cohen’s original paper provides the foundation for kappa on nominal labels. Low human agreement is a property of the item or rubric that must be reported, not silently erased.

3. Compare three judge configurations

1. J1 — Holistic pairwise: show the task and two candidates, provide a short rubric, and request one verdict.
2. J2 — Decomposed rubric: require separate assessments of instruction following, correctness, evidence support, completeness, and style before applying the declared hierarchy.
3. J3 — Reference-first: create or supply a reference checklist before exposing the candidates, then compare each response against that checklist and the source packet.

Use the same underlying model snapshot for all three configurations so the experiment isolates judging procedure rather than model capability. Freeze the exact model identifier, system prompt, rubric, decoding parameters, parser, and date. JudgeLM motivates swap and reference controls, while CALM illustrates the value of controlled, principle-guided modifications.

Every configuration evaluates every pair in both orders. Normalize the result back to candidate identity. Require machine-readable output such as:

{"winner": "A|B|TIE|ABSTAIN", "confidence": null, "reason_codes": []}

Track malformed output, refusal, timeout, and missing verdict as explicit outcomes. Do not quietly retry until a desired parse appears.

Metrics that belong in the visual report

Agreement with blinded humans

Report exact agreement for A, B, tie, and abstain. Also publish a strict result that counts a judge tie against a decisive human label, plus a secondary policy result if the production system intentionally maps uncertainty to human review.

Confusion matrices

A single accuracy number cannot show whether the judge systematically converts human ties into wins, favors candidate A, or misses incorrect-but-polished answers. Publish counts and row-normalized percentages for every configuration.

Order-swap consistency

Map both verdicts to candidate identity and compute the proportion that agree after swapping. Also report first-position preference, second-position preference, and contradiction rate. JudgeBench similarly evaluates both orders and treats contradictory swapped verdicts as a reliability failure.

Manipulation and difficulty slices

For content-preserving length and style pairs, measure how often the modified answer wins instead of tying. For citation and correctness pairs, measure how often the better-supported or correct answer wins. Cross-tabulate every metric by difficulty. Research has shown that judge behavior can vary with difficulty and judging strategy; see the LREC-COLING study.

Bootstrap confidence intervals

Resample by base_id, not by individual row, because variants from one task are dependent. For each of 10,000 resamples, include all pairs belonging to every selected base task and recompute the statistic. Publish percentile intervals for each metric and for paired differences between configurations. Bootstrapping originates with Efron’s foundational work.

def cluster_bootstrap(df, statistic, B=10_000, seed=7):
    rng = np.random.default_rng(seed)
    ids = df['base_id'].unique()
    estimates = []
    for _ in range(B):
        chosen = rng.choice(ids, len(ids), replace=True)
        sample = pd.concat(
            [df[df['base_id'].eq(i)] for i in chosen],
            ignore_index=True
        )
        estimates.append(statistic(sample))
    return np.quantile(estimates, [0.025, 0.975])

If a judge emits confidence, evaluate that confidence separately. A confident explanation is not evidence that the verdict is calibrated.

How to read the result without fooling yourself

Do not select the configuration with the highest overall agreement and stop. A useful configuration might have slightly lower aggregate agreement but much better factual-error detection, citation discrimination, and order consistency. Conversely, a judge that excels on easy style pairs but fails hard correctness pairs should not control a correctness-sensitive release gate.

Predeclare the decision rule before examining results. There is no universal safe threshold. A reasonable structure is a two-key gate: the automated score must improve with an interval that excludes the team’s material-regression boundary, and a blinded human sample must confirm the direction on critical slices. If the judge and humans disagree near a release threshold, the system should pause rather than average away the conflict.

A practical workflow for regression tests and dashboards

1. Calibrate before deployment. Run the full lab on tasks representative of production, not only public chat benchmarks.
2. Freeze and identify the instrument. Version the judge model, prompt, rubric, parser, source-access policy, and aggregation code together.
3. Keep a sentinel set. Re-run a protected subset on every judge change to detect discontinuities before comparing product versions.
4. Show uncertainty and slices. A dashboard should display the point estimate, confidence interval, sample count, judge version, order consistency, and critical error categories.
5. Audit score movements. For every claimed improvement, inspect a stratified sample of changed verdicts, including gains and losses.
6. Recalibrate after change. A model update, prompt edit, new source-retrieval path, parser change, or task-distribution shift creates a new instrument.

The strongest operational question is not “What score did the judge assign?” It is “On which kinds of examples has this judge earned the right to influence a decision?”

Human-in-the-loop safety

Candidate outputs are untrusted input. Delimit them as data, instruct the judge not to follow embedded directives, and test prompt-injection examples in which a candidate asks the evaluator to award it a win. Preserve the raw request and response so a reviewer can diagnose failures.

Automated judgments should not be the sole authority for high-impact medical, legal, financial, employment, safety, or access decisions. Route abstentions, order contradictions, low-confidence cases, unsupported citations, and material human disagreements to review. Humans also need safeguards: balanced assignment, source access, domain matching, documented adjudication, and permission to abstain.

Public artifact manifest

Assign a license only after verifying that every prompt, source packet, and output can be redistributed. Remove personal data and secrets. Publish checksums so later revisions are distinguishable from the original release.

Limitations

• Human consensus is a reference, not infallible truth. Expert tasks may require objective verification beyond preference labels.
• Controlled mutations improve causal interpretability but cannot reproduce every interaction among style, length, correctness, and user context.
• The task mixture determines the meaning of the aggregate score. Results do not automatically transfer to a different product distribution.
• Hosted models may change, and nominally deterministic settings may still produce variation. Raw runs and dates therefore matter.
• Bootstrap intervals quantify sampling uncertainty but do not correct a biased dataset, rubric, annotator pool, or judge.
• A public test can eventually become familiar to model developers. Maintain a private sentinel set alongside the public educational artifact.

The decision rule

Trust an LLM judge only within the slices where it has demonstrated agreement, invariance, and error sensitivity—and only for as long as the judge configuration remains unchanged. A rising score is evidence of product improvement only after the measurement system survives its own regression test.