Evals (inferential feedback)

Evals are the inferential, LLM-based feedback control. Where a sensor checks structure deterministically, an eval judges meaning: is this PRD clear, is this design's trade-off real, does this plan actually follow from the PRP. An eval is an LLM-judge scored 0-10 against a rubric, with a pass threshold of 8.0 and up to 3 retries.

Why you need inferential controls

No regex can tell you a problem statement is vague, a hypothesis is unfalsifiable, or a design hid its hardest trade-off. That is semantic judgment, and only a language model can do it at this granularity. The cost: it is probabilistic, so you calibrate it with a rubric and weights rather than trusting a bare "rate this 1-10".

Anatomy of an eval

A rubric is a markdown file: weighted dimensions, each with a 0/5/10 anchor, a threshold, and a strict output format.

# Eval: Design Quality
Type: LLM-judge
Mode: quality gate
Threshold: weighted total >= 8.0

## Rubric
### Requirements rigor (weight 15%)
Numeric targets + back-of-envelope math?
- 10: numeric targets + sizing math
- 5: some numbers, no math
- 0: prose only
...

## On failure (total below 8.0)
Retry. Regenerate lowest-scoring sections only. Max 3 attempts.

## Output format
{ "scores": {...}, "weighted_total": 0.0, "feedback": ["dimension: issue with line ref"] }

weighted_total = sum(score x weight%) / 100. The judge must cite a line or section when it scores a dimension below 7, so the feedback is actionable, not a vibe.

The retry loop

generate -> sensors (hard gate) -> eval (score)
   ^                                   |
   |          below 8.0                v
   +------ regenerate weak sections ---+   (max 3 attempts, then blocker)

Critically, the retry regenerates only the low-scoring dimensions, not the whole artifact, and the judge's per-dimension feedback says which. After 3 failed attempts the stage returns a blocker rather than shipping something sub-threshold.

Weights encode what matters

The weights are where you express priorities. In the PRD-quality rubric, metric completeness and clarity carry the most weight because a PRD lives or dies on a crisp problem and measurable success. In the design-quality rubric, architecture + deep dives and trade-off discipline dominate because that is what separates a staff design from a box diagram. Tuning weights is a "human on the loop" action: you change what the gate cares about once, and it applies to every future artifact.

Calibration and anti-rubber-stamp

An eval that always passes is worthless. Two practices keep it honest:

• Anchored scales. Each dimension defines what 0, 5, and 10 look like, so the judge is not guessing what "7" means.
• Adversarial framing where it matters. The system-architect's design-review-depth eval scores a review and explicitly fails one that rubber-stamps: a review with no named gaps, flat severity, or generic "consider improving" advice scores low by design.

Evals in harness-kit

The PASS/FAIL variant: spec-satisfied

The spec-driven loop (/sse:sdd) uses a different eval shape. Instead of a 0-10 score, spec-satisfied returns PASS/FAIL against the PRP's Success criteria (verifiable) and Validation gates. A FAIL re-enters the dev↔test loop with a next_iter_focus hint; the loop runs at most 3 iterations. It runs in a fresh session with no worker context, so the judge is not biased by the implementer's narrative. This is an eval used as a goal predicate rather than a quality score.

Sensor first, eval second

Order matters: sensors run before evals. There is no point spending an LLM call scoring the prose of an artifact that is missing half its sections. The deterministic gate clears the cheap, objective failures first; the eval judges only well-formed artifacts.

See also

• Sensors: the deterministic gate that runs first
• Guides: feedforward that reduces how often evals fail
• Pipeline and Stages: where evals sit in a stage

What the score is not

The judge returns per-dimension scores and a weighted total, and until recently nothing checked that the total followed from the scores. It does now: pipe the judge's JSON through the score verifier, which reads the weights out of the rubric and recomputes.

.claude/scripts/eval-score.py --rubric evals/prd-quality.md --scores judge.json

Exit 0 prints the number the approval marker should carry. Exit 2 means the judge inflated the total, scored a dimension the rubric does not weight, or skipped one it does, in which case the score is meaningless and you do not approve on it.

That fixes the computable half only. Two limits stand, and it is worth being blunt about them:

• The scores are not validated against human labels. 8.0 is a convention, not a calibrated boundary. Husain argues uncalibrated 1-10 scales mean different things to different graders and that binary judgments plus measured human agreement are what make an eval trustworthy. We do not do that yet.
• A fresh evaluator is not an unbiased one. Dispatching a judge with no prior context removes the author's stake in the text, but LLM judges still inflate scores for output from their own family (Panickssery et al.). A fresh Claude judging Claude is still self-preference. Fixing it properly needs a different model family or a human.

Read the feedback, not just the number. The score is a signal that catches weak artifacts, not a measurement.

This page mirrors Evals (inferential feedback) in the wiki. Edit it there.