feat(fleet): multi-runtime harness — verifier tests + capped builder shell
The harness layer (builder sessions, cold verifiers, evidence flow) gets a committable home, per the PRD model-fleet § harness portability and erp#63: - fleet/harness/verifier/: the two canonical verifier tests (locate-test, cold-reader backlog audit) with pinned inputs, verbatim prompts, ground truth and pass rules — judged context-free, never self-graded. - fleet/harness/bin/run-verifier.sh: runs a test against any OpenAI-style local endpoint (Ornith/MLX) or vibe -p (Mistral); emits sha256-pinned JSON transcripts. - fleet/harness/bin/vibe-builder.sh: the bounded shell for scoped builders and recurring tasks — refuses the trunk (linked-worktree guard), hard --max-turns/--max-price caps, full JSON journal per run. - fleet/README.md layout + AGENTS.md Fleet section updated in the same change (same-change freshness rule). Part of erp#63 (harness portability spike, D2). Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VRShc4QhLLU73FLHx9vskh
This commit is contained in:
@@ -0,0 +1,44 @@
|
||||
# Verifier test — cold-reader backlog audit
|
||||
|
||||
The fresh-reader smoke test from the PRD
|
||||
[closure protocol](https://gitea.arcodange.lab/arcodange-org/factory/src/branch/main/vibe/PRD/ai-back-office/STATUS.md),
|
||||
run standalone: a cold reader with only the two orientation documents must
|
||||
reconstruct where the work stands. Used for verifier admission (erp#63).
|
||||
|
||||
## Inputs
|
||||
|
||||
| Document | Pinned source |
|
||||
| --- | --- |
|
||||
| `AGENTS.md` | the erp repo root, at the commit under test |
|
||||
| `STATUS.md` | `vibe/PRD/ai-back-office/STATUS.md` from the **factory** repo, origin/main at run time |
|
||||
|
||||
The runtime gets both documents embedded in the prompt and **nothing else**.
|
||||
|
||||
## Prompt (verbatim)
|
||||
|
||||
> You are a cold reader auditing the Arcodange AI back-office backlog. Using ONLY
|
||||
> the two documents below — no other knowledge, no tools — answer:
|
||||
> 1. What shipped most recently?
|
||||
> 2. What should be worked on next, and why that item?
|
||||
> 3. What would you verify before trusting these documents, and in what order?
|
||||
|
||||
`bin/run-verifier.sh backlog <runtime> --status-file <path>` assembles this
|
||||
prompt with both documents appended under `--- AGENTS.md ---` and
|
||||
`--- STATUS.md ---` delimiters.
|
||||
|
||||
## Rubric
|
||||
|
||||
| Question | Substantively correct means |
|
||||
| --- | --- |
|
||||
| 1 — shipped | names the most recent ✅ item(s) the STATUS actually records (e.g. the fleet scaffold / latest merged PR in its ledger) — not an older phase, not an open issue |
|
||||
| 2 — next | applies the resume protocol: top **unblocked** issue of the **earliest open milestone by due date**, and skips issues whose only open step is a `[HUMAN]` gate (surfaced, not stalled on) |
|
||||
| 3 — trust | states the trust order live system > code/git log > STATUS > PRD leaves > memories (wording may vary; the direction must be right), and proposes checking Last Updated / verifying claims against the live repo |
|
||||
|
||||
## Pass rule
|
||||
|
||||
PASS = substantively correct on all three questions. Minor omissions that do not
|
||||
invert the protocol (e.g. not listing every milestone) do not fail; inverting
|
||||
the trust order, proposing a blocked/`[HUMAN]`-gated issue as "next" without
|
||||
flagging the gate, or citing shipped work the documents do not support, fails.
|
||||
Judged by an independent context-free judge — never by the session that produced
|
||||
the run.
|
||||
Reference in New Issue
Block a user