# Eval Tier 3 — Operational Evals

Runs on batches of outputs in production. Catches what per-output review cannot.

## What it catches
- **Drift** — eval pass rate degrading quietly over time
- **Cross-output inconsistency** — same input producing different outputs
- **Distributional shift** — the input distribution changing without the eval set updating
- **Tool misuse trends** — tool invocation patterns that are individually reasonable but collectively wrong
- **Cascading hallucinations** — fabricated facts propagating through multi-agent chains
- **Cost anomalies** — token spend patterns that suggest runaway loops or inefficiency

## Implementation pattern
- Run auditor agent on N-hour rolling windows of outputs
- Compare aggregate metrics against baselines
- Alert on statistical anomaly (2-sigma, 3-sigma)
- Never rely on humans to notice drift — the whole point is that they can't

## Rules
- Tier 3 is what saves your program six months in. Tier 1 catches individual failures. Tier 3 catches structural failures.
- The auditor agent must have a different model from the primary and validator. Common-mode failures are the enemy.
- Auditor findings are always human-reviewed. This tier is not autonomous.