Choose what the judge is allowed to decide
Model graders can be useful for qualitative criteria such as whether an answer addresses a question or whether an explanation follows a stated rubric. Deterministic checks are usually more appropriate for a schema violation, a missing required field, a known unauthorized write or an exact numeric constraint.
The original MT-Bench and Chatbot Arena study examined limitations including position and verbosity bias in LLM judging. Its historical benchmark results do not establish the reliability of a current model on a different product task. Use the paper to motivate tests for bias, not as a borrowed accuracy claim.[1]
| Requirement | Appropriate starting point |
|---|---|
| Valid JSON schema | Deterministic parser and schema validator. |
| No unauthorized mutation | Execution and policy evidence. |
| Required numeric total | Independent calculation with defined tolerance. |
| Explanation follows the evidence | Calibrated judge plus reviewed disagreement samples. |
| High-consequence domain decision | A task-specific review process with appropriate human responsibility. |
Write a rubric that supports disagreement
A rubric should say what counts as acceptable, what makes an answer unacceptable and what evidence is needed. Replace “is this good?” with observable criteria. For a support response, the answer may need to identify the relevant incident, use the current account policy and avoid promising an action that was not authorized.
Include examples near the decision boundary, not only obvious good and bad answers. Ask reviewers to label them independently where practical, then resolve disagreements in the rubric. If humans cannot explain why an item passes, an automated grader is unlikely to make that ambiguity disappear.
Keep a calibration set separate from the material used to tune the grader prompt. Version both. Official evaluation guidance also recommends task-specific evaluation and comparison against human judgments rather than relying only on intuition.[2]
Worked example: 90% agreement is not the whole result
Suppose a reviewed synthetic set contains 80 acceptable answers and 20 unacceptable answers. The judge accepts 76 of the acceptable answers and rejects four. It correctly rejects 14 unacceptable answers but accepts six. Overall agreement is (76 + 14) / 100 = 90%.
| Human reference | Judge accepts | Judge rejects |
|---|---|---|
| Acceptable: 80 | 76 | 4 |
| Unacceptable: 20 | 6 | 14 |
The judge detects 14/20 unacceptable answers: 70% failure-detection recall. It misses 6/20, or 30%, of the failures. Of the 82 answers it accepts, six are unacceptable: about 7.3% false acceptance within the accepted group. These different denominators answer different questions.
A 90% agreement headline could therefore hide a release process that admits too many bad answers. Inspect the consequence and distribution of those six misses. Missing a minor style issue and missing an unauthorized disclosure should not be treated as equivalent grading errors.
Test the judge as a system
- Position: reverse the order of candidate answers and measure whether preferences change without a substantive reason.
- Verbosity: compare concise and expanded answers that contain the same required information.
- Style: vary formatting and confidence of tone while holding correctness constant.
- Evidence: remove or contradict a source and check whether the judge notices.
- Injection: include answer text that tries to instruct the grader to pass it.
- Repeatability: rerun a fixed sample to understand the grader’s own variability.
These tests probe whether the grader is measuring the intended criterion or a proxy. If it rewards length, a model can improve the score by writing more without improving the answer. If it follows instructions inside the candidate response, the evaluation boundary itself is compromised.
Do not give the grader unnecessary tool access or sensitive context. Candidate outputs are untrusted inputs to the grading system. Keep the rubric and expected output contract separate from the material being judged.
Keep release policy outside the grader
The grader should return a structured result with a criterion-level decision, relevant evidence and an explicit ungradable state when appropriate. Application code applies the release policy. A failed schema check or known prohibited action should not be overridden because the judge awarded a high overall score.
| Observation | Treatment |
|---|---|
| Critical deterministic failure | Block the relevant release regardless of average judge score. |
| Judge cannot assess the evidence | Mark ungraded and route for review; do not silently count as a pass. |
| High disagreement in a task class | Investigate or restrict that class before trusting automation. |
| Calibrated ordinary cases | Use the grader within the validated scope and monitor drift. |
See how critical failures remain visible when the aggregate pass rate is high.
Recalibrate when the measurement changes
A new judge model, prompt, rubric, product task or data distribution can change the meaning of the score. Preserve a comparison set and review the disagreement before treating old and new scores as one continuous series. Version identifiers belong in the evaluation record.
Use the judge where it reduces review effort without hiding important errors. Where it does not, narrow the criterion, improve the reference or retain human review. The objective is a reliable release decision, not a fully automated score for every task.
Sources and scope
Technical references inform the cited statements. The decision frameworks and synthetic examples are TeqEngine’s editorial guidance.
- Zheng et al.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenahttps://arxiv.org/abs/2306.05685
- OpenAI: Evaluation best practiceshttps://developers.openai.com/api/docs/guides/evaluation-best-practices