# Eval Tier 2 — Probabilistic Evals (LLM-as-judge)

Runs on high-stakes outputs and a statistical sample of routine ones.

## What it checks
Semantic quality that deterministic rules cannot capture:
- Faithfulness to source material
- Consistency with stated intent
- Absence of hallucination against grounded facts
- Style and voice adherence
- Reasoning quality on multi-step outputs

## Judge prompt template
```
You are a judge evaluating whether the OUTPUT below meets the CRITERIA.

CRITERIA:
{list of criteria specific to this output type}

INPUT (what the agent was given):
{input}

OUTPUT (what the agent produced):
{output}

For each criterion, score 0-5 and explain briefly. At the end, give an overall
PASS / FAIL / NEEDS_HUMAN_REVIEW.

Be strict. If any criterion is 2 or below, the output should not PASS.
If you would need more context to judge fairly, return NEEDS_HUMAN_REVIEW.
```

## Adjudication rules
- Sample rate: 100% on high-stakes outputs (customer-facing, irreversible)
- Sample rate: 5-10% on routine outputs (for drift monitoring)
- Judge disagreement with primary: escalate to arbiter
- Judge NEEDS_HUMAN_REVIEW: always escalate

## Failure handling
- Track PASS rate over rolling windows (24h, 7d, 30d)
- Alert on threshold breach
- Never quietly lower thresholds