arcodangeandClaude Fable 5
31158b05fa
The PARA Obsidian vault (arcodange/SecondBrain — git-synced, sb.py digest/inbox/gitea-ingest jobs on the hermes cron ticker, local Ornith model, mcp-obsidian access) enters the PRD as a first-class component: new T17 knowledge capture & retrieval fiche, knowledge-layer section in the architecture (ERP = book of record, vault = context + institutional memory, append-only idempotent deposits, trusted-but-stale retrieval), hermes/Ornith recognized as the resident M4 runtime (D2 leaning, new D7 cluster<->vault access decision), foundation ledger row, diagram + goals updated (mermaid revalidated, 231 links/anchors re-checked green). Co-Authored-By: Claude Fable 5 <[email protected]>
5.7 KiB
5.7 KiB
vibe > PRD > AI back-office > QA strategy
QA strategy — how "done and safe" is proven
Status: In design Last Updated: 2026-07-11 Up: AI back-office hub Related: POC plan · Challenges · Agent architecture
The fleet's product is trustworthy books, so QA is not a phase — it is the operating system of the fleet: evals gate autonomy, writes assert their own deltas, fiscal outputs prove €-parity, and operations prove their own liveness.
Golden datasets
- Sources: real history — the 2026 mailbox (labeled by Claude, corrected by the human), every supplier invoice already recorded, filed declarations, bank feeds. Volumes are small, so every real item is a test case; synthetic edge cases (weird layouts, multi-rate invoices, credit notes) and adversarial injection fixtures pad the set.
- Storage: in the private Gitea (business data stays in the lab); one folder per atom:
inputs/,expected/,scoringscript. The datasets grow as a by-product of operation — every human correction, rejection reason, and reclassification is captured into the set (the approval card's "reject with reason" is a labeling interface). - Scoring: field-level, not document-level — a 9/10-fields extraction is a failed document but 90 % field accuracy; both numbers are tracked. Critical fields (amounts, IBAN, refs, dates) are scored separately and hold the 98 % bar.
Eval harness
- Per-atom regression: any change to an atom (prompt, model, version bump in the registry) re-runs its golden set; scores are committed alongside the change (a PR that degrades an atom's score is visible as such).
- Injection suite: every atom that reads untrusted content runs the adversarial fixtures; a single leak (instruction obeyed, field fabricated under influence) is a blocking failure regardless of the accuracy score.
- Disagreement telemetry: dual-extraction disagreement rates and escalation rates are recorded per run — a drift upward is an early-warning signal before accuracy visibly drops.
Autonomy promotion gates
Per atom, mechanical, recorded in the registry (ladder):
| Transition | Gate |
|---|---|
| A0 → A1 | golden set exists; atom passes it at its accuracy bar (≥ 98 % critical fields for extraction atoms). |
| A1 → A2 | ≥ 20 consecutive real items where the human's action was approve as-is (any field correction resets the counter); injection suite green. |
| A2 → A3 | read-only/reversible atoms only; 3 clean months at A2 + human sampling audit (random 10 % re-verified) with zero material findings. |
| Demotion | any incident (wrong write approved, missed deadline, injection leak) drops the atom one level; the path back up is the same gates, not seniority. |
Write-path QA
- Predicted-delta assertion: every rehearsed manifest re-reads what the sandbox created and diffs it against the draft before the approval card goes out; a mismatch is a bug, never a "close enough".
- Post-write verification: after promote, the prod object is re-read and compared again; the pre/post snapshot pair (T13) must show exactly the journaled writes and nothing else.
- Idempotency tests: every write atom's test suite replays its own manifest twice and asserts a no-op second pass.
Fiscal parity checks
- Dry-run €-parity: fiscal sheets (T10) are compared to actually-filed values to the euro (POC-4); divergences block autonomy and open an investigation (books error vs. atom error — both are findings).
- Expert checkpoint: before the first agent-prepared filing of a new declaration type (first CA3 in 2027, first liasse), an expert-comptable (or SIE confirmation) validates the method once; after that, parity checks carry the load.
- Reconciliation invariant: the weekly zero-unexplained-deltas bar (T07) is itself a standing QA on the books.
Ops QA
- Heartbeats + silence alarms: every standing loop reports; the daily digest states "all green" explicitly — a missing digest is the alarm (C7).
- Monthly restore drill: latest prod backup restored into the sandbox + smoke-check, automated with a human-read report (T14).
- Quarterly game-day: deliberately take one tier down (revoke the cloud key, cordon the inference node, sleep the laptop) and verify the degraded-mode table holds in practice — same philosophy as the safe-prod-like-environment drills.
- Weekly ops review (human, ~10 min): escalation/quarantine/disagreement rates, DLQ age, digest accuracy spot-check, and the standing question: which atom cost more than it saved this week?
Evidence trail
Every month yields an audit pack: the coherence audit (T15), the month's run journals, snapshot content-hashes, approval-card decisions, and fiscal sheets — archived in git + GED. The pack is written for a third party (expert-comptable, auditor, or a future operator): it must let them reconstruct what the fleet did and why without access to this PRD or any chat history. A distilled summary of each pack also lands in the second brain (T17), so institutional memory outlives both chat logs and this repo.