Evals (inferential feedback)

A sensor checks structure with regex. An eval judges meaning, such as whether the plan follows from the PRP. Each eval is a weighted rubric scored 0 to 10; pass is 8.0, with up to 3 retries.

Anatomy of a rubric

Rubrics live in .claude/agents/{agent}/evals/*.md. Each ### Name (weight N%) dimension carries atomic yes/no checks: - check: lines are questions for the judge, - absent: lines are regexes run in code (banned words, em dash, ASCII box diagrams).

### Metric completeness (weight 20%)
- check: Every metric in `sections.success_metrics` has a baseline value.
- check: Every metric in `sections.success_metrics` has a target value.
- check: `sections.success_metrics` states a kill criterion with a numeric threshold.

### Voice (weight 5%)
- absent: (?i)\b(delve|leverage|utilize|unlock|streamline|robust|cutting-edge|seamless|best-in-class)\b
- absent: (?m)^\s*\+[-=]{3,}\+

A dimension scores 10 times the share of its checks met; the 0/5/10 anchors only break ties.

Verified arithmetic

eval-score.py recomputes the weighted total from the rubric weights and refuses a judge whose arithmetic or keys do not match.

python3 .claude/scripts/eval-score.py --rubric evals/prd-quality.md --scores judge.json

Exit	Meaning
`0`	consistent and at or above 8.0; prints the score for the approval marker
`1`	below threshold, retry
`2`	malformed output, wrong total, or missing or extra dimensions

The retry loop

Sensors run first, so no judge call goes to a malformed artifact. A failed eval regenerates only the failed checks, which are the feedback verbatim. After 3 failed attempts the stage returns a blocker.

flowchart LR
  G[generate] --> S[sensors]
  S -->|fail| G
  S -->|pass| E[eval]
  E -->|below 8.0, max 3| R[regenerate failed checks]
  R --> S
  E -->|8.0 or more| A[approve]

Two judges

/hk:eval jev | local picks the judge per project. The choice lives in .claude/hk-config.json, asked once at install, default local.

local is a fresh Claude evaluator that sees only the artifact and the rubric. jev is Jev by TypeSafe AI, a System One model that returns calibrated probabilities instead of text, called through .claude/scripts/jev-judge.py. Each check is one Noul (yes/no) question. The artifact goes split by ## section (sections.success_metrics), so a check reads only the part it names; a check on a missing section fails without a call. Every run appends to .claude/runtime/outputs/evals/jev-judge.jsonl.

Escalation to Claude

The run escalates to the Claude evaluator when Jev is unsure on checks that decide the result: the total recomputed with unsure answers forced to no and to yes lands on both sides of 8.0. It also escalates on no key, an artifact over 32k tokens, or an API error.

The key lives in an env var (default TYPESAFE_API_KEY) or in the env block of .claude/settings.local.json, never in a committed file. Jev is a paid API, about $0.042 per million input tokens, so it runs locally and never in CI. spec-satisfied and the readiness gates always use Claude.

Evals per stage

Stage	Evals
`prd`	`prd-quality`, `prd-readiness`
`prp`	`prp-quality`, `prp-context-readiness`
`plan`	`plan-quality`
`dev`	`dev-quality`
`test`	`test-quality`
`pr`	`pr-quality`
`sdd` loop	`spec-satisfied`
system design	`design-quality`, `design-review-depth`

Weights say what each gate cares about; PRD quality puts 20% each on clarity and metric completeness. You tune a weight once and it applies to every future artifact. design-review-depth fails a review that rubber-stamps: no named gaps, flat severity, generic advice.

spec-satisfied

/sse:sdd uses spec-satisfied, which returns PASS or FAIL against the PRP's Success criteria (verifiable) and Validation gates. A FAIL re-enters the dev and test loop with a next_iter_focus hint, up to 3 iterations. It runs in a fresh session with no worker context.

Why this shape

A Claude judge grading Claude output inflates scores (Wataoka et al.; Panickssery et al.); a different family reduces this, it does not remove it. Atomic checklists raise judge agreement (CheckEval, TICK). Escalation follows Trust or Escalate. Jev checks follow its guidance: one judgment per question, no counting, filter state.

What the score is not

Neither judge is validated against human labels yet, and 8.0 is a convention, not a calibrated boundary. Hamel Husain recommends about 100 labeled examples per failure mode; the jsonl log is the start of that set. Read the failed checks, not only the number.