Part of the AI back-office PRD — model-fleet › harness portability + decision D2. Operator direction 2026-07-15: the harness loop (builder sessions, cold verifiers — first proven on Claude with #38/PR #62) must also run on Mistral or on hermes + Ornith / MLX-served local models; Claude is an escalation tier, not a hard dependency.
Deliverables
Verifier parity (the cheap, high-value half): re-run the erp#38 locate-test (prompt preserved in the #38 evidence comment) and ONE cold-reader backlog audit with:
Ornith via hermes (127.0.0.1:18080),
an MLX-served candidate on the M4 (Qwen3 14B/30B-A3B or Mistral Small 3.x class),
Mistral API (Large or Medium).
Same prompts, same inputs → verdict parity table vs the Claude run (which PASSED). A verifier tier is admitted when it reaches parity on both tests.
Cross-family verification rule: once ≥1 non-Claude verifier is admitted, codify in the PRD qa-strategy: the independent verifier SHOULD be a different model family than the builder (cross-family refutation > same-family agreement). PRD PR paired with this issue.
Scoped builder bench: ONE small, tightly-footered issue (a #56-class fix) attempted end-to-end by a Mistral-driven runtime (hermes skill or SDK-equivalent) against the unchanged Execution footer + acceptance gates; measure completion, intervention count, wall-clock. No gate is relaxed for the bench — that's the point.
Findings feed the D2 settlement (orchestration runtime) — ADR via factory#22 when D2 closes with phase 3.
Acceptance: parity table posted here (three runtimes × two verifier tests, real transcripts linked); qa-strategy cross-family rule merged if admitted; builder-bench report with numbers (even a failed bench is a valid result — it sets the complexity ceiling honestly).
Execution — Blocked by: none (locate-test fixtures exist on main since #62; hermes+Ornith live) · Blocks: D2 ADR; cross-family verifier policy · Human gates: none (Mistral API key assumed in skills .env; if absent, one-time operator step). Start:claude --worktree harness-portability from the erp repo root; first command: replay the locate-test prompt against Ornith (curl 127.0.0.1:18080 per hermes conventions) with AGENTS.md as the only input. Done means: evidence comment here (parity table + transcripts) → paired PRD PR merged → Closes #63.
Part of the **AI back-office PRD** — [model-fleet › harness portability](https://gitea.arcodange.lab/arcodange-org/factory/src/branch/main/vibe/PRD/ai-back-office/model-fleet.md#harness-portability) + decision **D2**. Operator direction 2026-07-15: the harness loop (builder sessions, cold verifiers — first proven on Claude with #38/PR #62) must also run on **Mistral** or on **hermes + Ornith / MLX-served local models**; Claude is an escalation tier, not a hard dependency.
**Deliverables**
1. **Verifier parity (the cheap, high-value half):** re-run the erp#38 **locate-test** (prompt preserved in the [#38 evidence comment](https://gitea.arcodange.lab/arcodange-org/erp/issues/38)) and ONE cold-reader backlog audit with:
- **Ornith** via hermes (`127.0.0.1:18080`),
- an **MLX-served candidate** on the M4 (Qwen3 14B/30B-A3B or Mistral Small 3.x class),
- **Mistral API** (Large or Medium).
Same prompts, same inputs → verdict parity table vs the Claude run (which PASSED). A verifier tier is admitted when it reaches parity on both tests.
2. **Cross-family verification rule:** once ≥1 non-Claude verifier is admitted, codify in the PRD qa-strategy: *the independent verifier SHOULD be a different model family than the builder* (cross-family refutation > same-family agreement). PRD PR paired with this issue.
3. **Scoped builder bench:** ONE small, tightly-footered issue (a #56-class fix) attempted end-to-end by a **Mistral-driven runtime** (hermes skill or SDK-equivalent) against the unchanged Execution footer + acceptance gates; measure completion, intervention count, wall-clock. No gate is relaxed for the bench — that's the point.
4. Findings feed the **D2 settlement** (orchestration runtime) — ADR via [factory#22](https://gitea.arcodange.lab/arcodange-org/factory/issues/22) when D2 closes with phase 3.
**Acceptance:** parity table posted here (three runtimes × two verifier tests, real transcripts linked); qa-strategy cross-family rule merged if admitted; builder-bench report with numbers (even a failed bench is a valid result — it sets the complexity ceiling honestly).
---
**Execution** — Blocked by: none (locate-test fixtures exist on main since #62; hermes+Ornith live) · Blocks: D2 ADR; cross-family verifier policy · Human gates: none (Mistral API key assumed in skills `.env`; if absent, one-time operator step).
**Start:** `claude --worktree harness-portability` from the erp repo root; first command: replay the locate-test prompt against Ornith (`curl 127.0.0.1:18080` per hermes conventions) with `AGENTS.md` as the only input.
**Done means:** evidence comment here (parity table + transcripts) → paired PRD PR merged → `Closes #63`.
arcodange
added this to the P3 — Standing fleet milestone 2026-07-15 18:49:39 +02:00
FAIL (Q1 anchored on the PRD-authoring PR-log row, missed the #38 ship)
❌
Deviation, stated honestly: the issue's MLX candidate class (Qwen3 14B/30B-A3B / Mistral Small 3.x) is not resident on the M4; the 7B-4bit run is the honest small-model floor, the named candidates remain to be benched when pulled. Ornith (35B) covers the "hermes + local model" verifier claim.
2. Cross-family rule: with two non-Claude verifiers admitted, the rule is codified in the PRD qa-strategy › Independent verification — paired factory PR.
3. Builder bench — erp#56 end-to-end under the Mistral runtime (vibe-builder.sh, unchanged Execution footer, caps --max-turns 60 --max-price 3):
Interventions: 1, environmental — relinking the shared skill .env into the worktree for the acceptance command; the builder correctly reported the missing credentials instead of fabricating output.
Wall-clock: 261 s.
Acceptance: bin/arcodange bank match --since 2026-01-01 from the builder's branch → 12 matched, 1 internal, 2 avoir-netted, 5 bank-known (all legitimate: fee, capital, top-up pair, URSSAF), 0 bank-UNKNOWN, 0 dol-only-API, 7 dol-only-personal (CCA1, expected) → merged as PR #68, closed#56.
D2 signal: the harness loop (builder + cold verifier + evidence) ran without Claude in the builder seat on a #56-class issue; complexity ceiling above that (multi-file, ambiguity, fiscal reasoning) stays Claude-default until benched otherwise — per the PRD migration ladder.
**Evidence — parity table + builder bench (2026-07-18).** Full transcripts committed under [`fleet/harness/runs/2026-07-18/`](https://gitea.arcodange.lab/arcodange-org/erp/src/branch/arcodange/harness-portability/fleet/harness/runs/2026-07-18) (branch `arcodange/harness-portability`, harness home `fleet/harness/`).
**1. Verifier parity — same assembled prompts (sha256-pinned), temperature 0, blind-judged** (runtime identity stripped; 2 independent context-free judges per cell, strict + skeptic lens; **unanimous 16/16**):
| Runtime | locate-test (erp#38 prompt, verbatim) | cold-reader backlog audit | Admitted |
| --- | --- | --- | --- |
| Claude (`claude-fable-5`, context-free subagent) | PASS | PASS | baseline |
| **Mistral** — `vibe -p`, `mistral-medium-3.5` | PASS (14 s) | PASS (102 s) | **✅ verifier duty** |
| **Ornith** — `leonsarmiento/Ornith-1.0-35B-5bit-mlx`, hermes `127.0.0.1:18080` | PASS (18 s) | PASS (51 s) | **✅ verifier duty** |
| MLX small — `Qwen2.5-7B-Instruct-4bit` | FAIL (environment-rules half missing) | FAIL (Q1 anchored on the PRD-authoring PR-log row, missed the #38 ship) | ❌ |
Deviation, stated honestly: the issue's MLX candidate class (Qwen3 14B/30B-A3B / Mistral Small 3.x) is not resident on the M4; the 7B-4bit run is the honest small-model floor, the named candidates remain to be benched when pulled. Ornith (35B) covers the "hermes + local model" verifier claim.
**2. Cross-family rule**: with two non-Claude verifiers admitted, the rule is codified in the PRD [qa-strategy › Independent verification](https://gitea.arcodange.lab/arcodange-org/factory/src/branch/main/vibe/PRD/ai-back-office/qa-strategy.md#independent-verification--no-self-grading) — paired factory PR.
**3. Builder bench — erp#56 end-to-end under the Mistral runtime** (`vibe-builder.sh`, unchanged Execution footer, caps `--max-turns 60 --max-price 3`):
- **Completion**: both file-side deliverables correct (Mistral annual + Anthropic personal-card lane), conventional commit. **0 code corrections.**
- **Interventions**: 1, environmental — relinking the shared skill `.env` into the worktree for the acceptance command; the builder correctly **reported** the missing credentials instead of fabricating output.
- **Wall-clock**: 261 s.
- **Acceptance**: `bin/arcodange bank match --since 2026-01-01` from the builder's branch → `12 matched, 1 internal, 2 avoir-netted, 5 bank-known (all legitimate: fee, capital, top-up pair, URSSAF), 0 bank-UNKNOWN, 0 dol-only-API, 7 dol-only-personal (CCA1, expected)` → merged as **PR #68**, closed #56.
**D2 signal**: the harness loop (builder + cold verifier + evidence) ran without Claude in the builder seat on a #56-class issue; complexity ceiling above that (multi-file, ambiguity, fiscal reasoning) stays Claude-default until benched otherwise — per the PRD migration ladder.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Part of the AI back-office PRD — model-fleet › harness portability + decision D2. Operator direction 2026-07-15: the harness loop (builder sessions, cold verifiers — first proven on Claude with #38/PR #62) must also run on Mistral or on hermes + Ornith / MLX-served local models; Claude is an escalation tier, not a hard dependency.
Deliverables
127.0.0.1:18080),Same prompts, same inputs → verdict parity table vs the Claude run (which PASSED). A verifier tier is admitted when it reaches parity on both tests.
Acceptance: parity table posted here (three runtimes × two verifier tests, real transcripts linked); qa-strategy cross-family rule merged if admitted; builder-bench report with numbers (even a failed bench is a valid result — it sets the complexity ceiling honestly).
Execution — Blocked by: none (locate-test fixtures exist on main since #62; hermes+Ornith live) · Blocks: D2 ADR; cross-family verifier policy · Human gates: none (Mistral API key assumed in skills
.env; if absent, one-time operator step).Start:
claude --worktree harness-portabilityfrom the erp repo root; first command: replay the locate-test prompt against Ornith (curl 127.0.0.1:18080per hermes conventions) withAGENTS.mdas the only input.Done means: evidence comment here (parity table + transcripts) → paired PRD PR merged →
Closes #63.Evidence — parity table + builder bench (2026-07-18). Full transcripts committed under
fleet/harness/runs/2026-07-18/(brancharcodange/harness-portability, harness homefleet/harness/).1. Verifier parity — same assembled prompts (sha256-pinned), temperature 0, blind-judged (runtime identity stripped; 2 independent context-free judges per cell, strict + skeptic lens; unanimous 16/16):
claude-fable-5, context-free subagent)vibe -p,mistral-medium-3.5leonsarmiento/Ornith-1.0-35B-5bit-mlx, hermes127.0.0.1:18080Qwen2.5-7B-Instruct-4bitDeviation, stated honestly: the issue's MLX candidate class (Qwen3 14B/30B-A3B / Mistral Small 3.x) is not resident on the M4; the 7B-4bit run is the honest small-model floor, the named candidates remain to be benched when pulled. Ornith (35B) covers the "hermes + local model" verifier claim.
2. Cross-family rule: with two non-Claude verifiers admitted, the rule is codified in the PRD qa-strategy › Independent verification — paired factory PR.
3. Builder bench — erp#56 end-to-end under the Mistral runtime (
vibe-builder.sh, unchanged Execution footer, caps--max-turns 60 --max-price 3):.envinto the worktree for the acceptance command; the builder correctly reported the missing credentials instead of fabricating output.bin/arcodange bank match --since 2026-01-01from the builder's branch →12 matched, 1 internal, 2 avoir-netted, 5 bank-known (all legitimate: fee, capital, top-up pair, URSSAF), 0 bank-UNKNOWN, 0 dol-only-API, 7 dol-only-personal (CCA1, expected)→ merged as PR #68, closed #56.D2 signal: the harness loop (builder + cold verifier + evidence) ran without Claude in the builder seat on a #56-class issue; complexity ceiling above that (multi-file, ambiguity, fiscal reasoning) stays Claude-default until benched otherwise — per the PRD migration ladder.