# LLM-as-judge evaluation

> Grade open-ended outputs with a model and a rubric — after checking the grader agrees with you.

Concept · LLMs in production · https://levelcode.org/concepts/llm-as-judge

Some outputs can be checked with code: the JSON parses, the label matches, the SQL returns the right rows. Prefer that whenever you can. But "is this summary faithful?" or "did the support reply solve the problem politely?" has no exact answer. **LLM-as-judge** uses a model to grade those outputs against a rubric, so you can score hundreds of cases on every prompt change.

## A good judge prompt

```text
You are grading a customer-support reply.

<question>{{question}}</question>
<reference_facts>{{facts}}</reference_facts>
<reply>{{reply}}</reply>

Check each criterion:
1. Every factual claim in the reply is supported by reference_facts.
2. The reply answers the customer's actual question.
3. No promises about refunds or timelines that the facts don't state.

Think it through in <reasoning>, then output <verdict>PASS</verdict> or <verdict>FAIL</verdict>
and the number of the first criterion that failed.
```

- **Make specific, binary checks rather than a 1–10 score.** "Mentions the refund window: yes/no" is far more consistent than "helpfulness: 7."
- **Give the judge reference material** (source documents, expected facts) instead of relying on what it happens to know.
- **Let it reason before the verdict,** and keep the reasoning so you can debug disagreements.
- **Use a strong model at low [temperature](https://levelcode.org/concepts/sampling-temperature)** for judging.

## Check the judge itself

A judge is only useful if it agrees with people. Before you trust it:

1. Have humans label 50–100 outputs.
2. Run the judge on the same outputs.
3. Measure agreement, read the disagreements, and fix the rubric until it's high enough to rely on.

Recheck whenever you change the judge prompt or model.

## Known biases

- **Position bias:** in A-vs-B comparisons, the judge favours whichever answer comes first. Run both orders.
- **Length bias:** longer answers look more thorough. Add "length is not a criterion" and check for it.
- **Self-preference:** a model can rate its own family's style higher. Consider judging with a different model.

## In practice

Run the eval suite in CI on every prompt, model or retrieval change, and track pass rate per criterion, not one blended number. Keep real failures from production in the test set; that's what turns [few-shot](https://levelcode.org/concepts/few-shot-prompting) tweaks and model swaps from guesswork into measurements. The [evals lesson](https://levelcode.org/roadmaps/ai-engineer/evals) covers building the dataset.