arcodangeandClaude Fable 5
ceb4321224
- runs/2026-07-18/: the 8 sha256-pinned verifier transcripts (4 runtimes × 2 tests), blind-judging verdicts (2 independent judges per cell, unanimous), the erp#56 builder-bench journal + prompt + caps, and the evidence README with the parity table. - run-verifier.sh: mistral runtime drops the tool-filter flag (--enabled-tools with a no-match pattern hangs vibe 2.21.0); plain -p with --max-turns 1. Verdicts: Mistral (vibe -p, mistral-medium-3.5) and Ornith 35B (hermes MLX) reach verdict parity with the Claude baseline on both tests → admitted to verifier duty. Qwen2.5-7B-4bit fails both → the honest small-model floor. Builder bench: erp#56 completed by the Mistral runtime, 0 code corrections, 261 s, acceptance run clean (0 bank-UNKNOWN) → merged as PR #68. Closes #63 (with the paired factory qa-strategy PR). Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VRShc4QhLLU73FLHx9vskh
3.0 KiB
Harness portability run — 2026-07-18 (erp#63)
Verifier-parity + builder-bench evidence for erp#63.
Protocol per fleet/harness/README.md: same assembled prompts
(sha256-pinned in each transcript), temperature 0, responses judged blind
(runtime identity stripped — blinded-mapping.json re-attaches it) by 2
independent context-free judges per cell (strict + skeptic lens), unanimous on
all 8 cells (judging-verdicts-20260718.json).
Verifier parity table
| Runtime | locate-test | backlog audit | Latency (s) | Admitted to verifier duty |
|---|---|---|---|---|
Claude (claude-fable-5, context-free subagent) |
PASS | PASS | 11 / 29 | baseline |
Mistral (vibe -p, mistral-medium-3.5) |
PASS | PASS | 14 / 102 | yes |
Ornith (leonsarmiento/Ornith-1.0-35B-5bit-mlx, hermes 127.0.0.1:18080) |
PASS | PASS | 18 / 51 | yes |
MLX small (mlx-community/Qwen2.5-7B-Instruct-4bit) |
FAIL (environment rules incomplete) | FAIL (Q1: anchored on the PRD-authoring PR-log row, missed the #38 ship) | 7 / 41 | no |
Two non-Claude verifiers reach verdict parity with the Claude baseline on both tests → the cross-family verification rule is codified in the PRD qa-strategy (paired factory PR).
Deviation, stated honestly: the issue names Qwen3 14B/30B-A3B or Mistral Small 3.x as the MLX candidate class; only Qwen2.5-7B-4bit and Llama-3.2-3B are resident on the M4 today. The 7B-4bit result is the honest small-model floor — the named larger candidates remain to be benched once pulled. Ornith (35B) covers the "hermes + local model" verifier claim.
Builder bench — erp#56 under the Mistral runtime
One tightly-footered issue run end-to-end by vibe -p (mistral-medium-3.5)
via vibe-builder.sh, unchanged Execution footer,
caps --max-turns 60 --max-price 3:
| Metric | Value |
|---|---|
| Completion | both file-side deliverables correct (annual fix + personal-card lane note), committed with a conventional message |
| Code corrections by the operator | 0 |
| Interventions | 1, environmental: relinking the shared skill .env into the worktree so the acceptance command could run; the builder correctly reported the missing credentials instead of fabricating output |
| Wall-clock | 261 s |
Acceptance (bin/arcodange bank match --since 2026-01-01) |
run post-relink from the builder's branch: 0 bank-UNKNOWN, no stale gap annotation for the now-recorded AI subscriptions → merged as erp PR #68 |
Full journal: builder-known-patterns-fix-journal.json (raw vibe --output json);
prompt: builder-bench-56-prompt.md; caps + wall-clock: the .meta.json.
Files
{locate,backlog}-<runtime>-<ts>.json— the 8 verifier transcripts (prompt sha256, input sha256s, response, latency).blinded-mapping.json— cell-id → runtime identity (judging was blind).judging-verdicts-20260718.json— 16 judge verdicts (2 per cell), item-level notes.builder-*— the erp#56 bench artifacts.