arcodangeandClaude Fable 5
ceb4321224
- runs/2026-07-18/: the 8 sha256-pinned verifier transcripts (4 runtimes × 2 tests), blind-judging verdicts (2 independent judges per cell, unanimous), the erp#56 builder-bench journal + prompt + caps, and the evidence README with the parity table. - run-verifier.sh: mistral runtime drops the tool-filter flag (--enabled-tools with a no-match pattern hangs vibe 2.21.0); plain -p with --max-turns 1. Verdicts: Mistral (vibe -p, mistral-medium-3.5) and Ornith 35B (hermes MLX) reach verdict parity with the Claude baseline on both tests → admitted to verifier duty. Qwen2.5-7B-4bit fails both → the honest small-model floor. Builder bench: erp#56 completed by the Mistral runtime, 0 code corrections, 261 s, acceptance run clean (0 bank-UNKNOWN) → merged as PR #68. Closes #63 (with the paired factory qa-strategy PR). Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VRShc4QhLLU73FLHx9vskh
50 lines
1.3 KiB
JSON
50 lines
1.3 KiB
JSON
{
|
|
"cell-01": {
|
|
"file": "backlog-claude-baseline-20260718.json",
|
|
"runtime": "claude",
|
|
"model": "claude-fable-5 (context-free subagent)",
|
|
"latency_s": null
|
|
},
|
|
"cell-02": {
|
|
"file": "backlog-mistral-20260718T194815.json",
|
|
"runtime": "mistral",
|
|
"model": "vibe-active-model",
|
|
"latency_s": 102
|
|
},
|
|
"cell-03": {
|
|
"file": "backlog-mlx-Qwen2.5-7B-Instruct-4bit-20260718T185141.json",
|
|
"runtime": "mlx",
|
|
"model": "mlx-community/Qwen2.5-7B-Instruct-4bit",
|
|
"latency_s": 18
|
|
},
|
|
"cell-04": {
|
|
"file": "backlog-ornith-20260718T185050.json",
|
|
"runtime": "ornith",
|
|
"model": "leonsarmiento/Ornith-1.0-35B-5bit-mlx",
|
|
"latency_s": 51
|
|
},
|
|
"cell-05": {
|
|
"file": "locate-claude-baseline-20260718.json",
|
|
"runtime": "claude",
|
|
"model": "claude-fable-5 (context-free subagent)",
|
|
"latency_s": null
|
|
},
|
|
"cell-06": {
|
|
"file": "locate-mistral-20260718T194754.json",
|
|
"runtime": "mistral",
|
|
"model": "vibe-active-model",
|
|
"latency_s": 14
|
|
},
|
|
"cell-07": {
|
|
"file": "locate-mlx-Qwen2.5-7B-Instruct-4bit-20260718T184831.json",
|
|
"runtime": "mlx",
|
|
"model": "mlx-community/Qwen2.5-7B-Instruct-4bit",
|
|
"latency_s": 7
|
|
},
|
|
"cell-08": {
|
|
"file": "locate-ornith-20260718T185032.json",
|
|
"runtime": "ornith",
|
|
"model": "leonsarmiento/Ornith-1.0-35B-5bit-mlx",
|
|
"latency_s": 18
|
|
}
|
|
} |