arcodangeandClaude Fable 5
ceb4321224
- runs/2026-07-18/: the 8 sha256-pinned verifier transcripts (4 runtimes × 2 tests), blind-judging verdicts (2 independent judges per cell, unanimous), the erp#56 builder-bench journal + prompt + caps, and the evidence README with the parity table. - run-verifier.sh: mistral runtime drops the tool-filter flag (--enabled-tools with a no-match pattern hangs vibe 2.21.0); plain -p with --max-turns 1. Verdicts: Mistral (vibe -p, mistral-medium-3.5) and Ornith 35B (hermes MLX) reach verdict parity with the Claude baseline on both tests → admitted to verifier duty. Qwen2.5-7B-4bit fails both → the honest small-model floor. Builder bench: erp#56 completed by the Mistral runtime, 0 code corrections, 261 s, acceptance run clean (0 bank-UNKNOWN) → merged as PR #68. Closes #63 (with the paired factory qa-strategy PR). Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VRShc4QhLLU73FLHx9vskh
52 lines
3.0 KiB
Markdown
52 lines
3.0 KiB
Markdown
# Harness portability run — 2026-07-18 (erp#63)
|
|
|
|
Verifier-parity + builder-bench evidence for [erp#63](https://gitea.arcodange.lab/arcodange-org/erp/issues/63).
|
|
Protocol per [`fleet/harness/README.md`](../../README.md): same assembled prompts
|
|
(sha256-pinned in each transcript), temperature 0, responses judged **blind**
|
|
(runtime identity stripped — `blinded-mapping.json` re-attaches it) by 2
|
|
independent context-free judges per cell (strict + skeptic lens), unanimous on
|
|
all 8 cells (`judging-verdicts-20260718.json`).
|
|
|
|
## Verifier parity table
|
|
|
|
| Runtime | locate-test | backlog audit | Latency (s) | Admitted to verifier duty |
|
|
| --- | --- | --- | --- | --- |
|
|
| Claude (`claude-fable-5`, context-free subagent) | PASS | PASS | 11 / 29 | baseline |
|
|
| **Mistral** (`vibe -p`, `mistral-medium-3.5`) | PASS | PASS | 14 / 102 | **yes** |
|
|
| **Ornith** (`leonsarmiento/Ornith-1.0-35B-5bit-mlx`, hermes `127.0.0.1:18080`) | PASS | PASS | 18 / 51 | **yes** |
|
|
| MLX small (`mlx-community/Qwen2.5-7B-Instruct-4bit`) | FAIL (environment rules incomplete) | FAIL (Q1: anchored on the PRD-authoring PR-log row, missed the #38 ship) | 7 / 41 | no |
|
|
|
|
Two non-Claude verifiers reach verdict parity with the Claude baseline on both
|
|
tests → the cross-family verification rule is codified in the PRD qa-strategy
|
|
(paired factory PR).
|
|
|
|
**Deviation, stated honestly:** the issue names Qwen3 14B/30B-A3B or
|
|
Mistral Small 3.x as the MLX candidate class; only Qwen2.5-7B-4bit and
|
|
Llama-3.2-3B are resident on the M4 today. The 7B-4bit result is the honest
|
|
small-model floor — the named larger candidates remain to be benched once
|
|
pulled. Ornith (35B) covers the "hermes + local model" verifier claim.
|
|
|
|
## Builder bench — erp#56 under the Mistral runtime
|
|
|
|
One tightly-footered issue run end-to-end by `vibe -p` (`mistral-medium-3.5`)
|
|
via [`vibe-builder.sh`](../../bin/vibe-builder.sh), unchanged Execution footer,
|
|
caps `--max-turns 60 --max-price 3`:
|
|
|
|
| Metric | Value |
|
|
| --- | --- |
|
|
| Completion | both file-side deliverables correct (annual fix + personal-card lane note), committed with a conventional message |
|
|
| Code corrections by the operator | **0** |
|
|
| Interventions | 1, environmental: relinking the shared skill `.env` into the worktree so the acceptance command could run; the builder correctly *reported* the missing credentials instead of fabricating output |
|
|
| Wall-clock | 261 s |
|
|
| Acceptance (`bin/arcodange bank match --since 2026-01-01`) | run post-relink from the builder's branch: **0 bank-UNKNOWN**, no stale gap annotation for the now-recorded AI subscriptions → merged as erp PR #68 |
|
|
|
|
Full journal: `builder-known-patterns-fix-journal.json` (raw `vibe --output json`);
|
|
prompt: `builder-bench-56-prompt.md`; caps + wall-clock: the `.meta.json`.
|
|
|
|
## Files
|
|
|
|
- `{locate,backlog}-<runtime>-<ts>.json` — the 8 verifier transcripts (prompt sha256, input sha256s, response, latency).
|
|
- `blinded-mapping.json` — cell-id → runtime identity (judging was blind).
|
|
- `judging-verdicts-20260718.json` — 16 judge verdicts (2 per cell), item-level notes.
|
|
- `builder-*` — the erp#56 bench artifacts.
|