factory/vibe/PRD/ai-back-office/qa-strategy.md

vibe > PRD > AI back-office > QA strategy

QA strategy — how "done and safe" is proven

Status: In design Last Updated: 2026-07-11 Up: AI back-office hub Related: POC plan · Challenges · Agent architecture

The fleet's product is trustworthy books, so QA is not a phase — it is the operating system of the fleet: evals gate autonomy, writes assert their own deltas, fiscal outputs prove €-parity, and operations prove their own liveness.

Golden datasets

• Sources: real history — the 2026 mailbox (labeled by Claude, corrected by the human), every supplier invoice already recorded, filed declarations, bank feeds. Volumes are small, so every real item is a test case; synthetic edge cases (weird layouts, multi-rate invoices, credit notes) and adversarial injection fixtures pad the set.
• Storage: in the private Gitea (business data stays in the lab); one folder per atom: inputs/, expected/, scoring script. The datasets grow as a by-product of operation — every human correction, rejection reason, and reclassification is captured into the set (the approval card's "reject with reason" is a labeling interface).
• Scoring: field-level, not document-level — a 9/10-fields extraction is a failed document but 90 % field accuracy; both numbers are tracked. Critical fields (amounts, IBAN, refs, dates) are scored separately and hold the 98 % bar.

Eval harness

• Per-atom regression: any change to an atom (prompt, model, version bump in the registry) re-runs its golden set; scores are committed alongside the change (a PR that degrades an atom's score is visible as such). ERP-dependent eval runs target a fresh sandbox checkpoint — the iso-prod refresh is a reproducible fixture (environments).
• Injection suite: every atom that reads untrusted content runs the adversarial fixtures; a single leak (instruction obeyed, field fabricated under influence) is a blocking failure regardless of the accuracy score.
• Disagreement telemetry: dual-extraction disagreement rates and escalation rates are recorded per run — a drift upward is an early-warning signal before accuracy visibly drops.

Autonomy promotion gates

Per atom, mechanical, recorded in the registry (ladder):

Transition	Gate
A0 → A1	golden set exists; atom passes it at its accuracy bar (≥ 98 % critical fields for extraction atoms).
A1 → A2	≥ 20 consecutive real items where the human's action was approve as-is (any field correction resets the counter); injection suite green.
A2 → A3	read-only/reversible atoms only; 3 clean months at A2 + human sampling audit (random 10 % re-verified) with zero material findings.
Demotion	any incident (wrong write approved, missed deadline, injection leak) drops the atom one level; the path back up is the same gates, not seniority.

Write-path QA

• Compliance linter: promote-plan rejects any manifest operation outside the production ledger grammar (mutating a validated document, supplying a manual ref where Dolibarr owns the sequence, detaching a GED original); the linter carries its own test suite of forbidden manifests.
• Provenance check (anti-hallucination): before any approval card, the deterministic checker re-verifies every critical field against its source anchor — document text re-extracted, bank feed re-pulled fresh (write contract); a field without a passing anchor blocks the card. Hallucination is mechanically detectable, not merely reviewable; the checker's own test suite includes seeded-wrong-value fixtures that must FAIL.
• Predicted-delta assertion: every rehearsed manifest re-reads what the sandbox created and diffs it against the draft before the approval card goes out; a mismatch is a bug, never a "close enough".
• Post-write verification: after promote, the prod object is re-read and compared again; the pre/post snapshot pair (T13) must show exactly the journaled writes and nothing else — and the diff must be pure-append (a mutation of a validated document is an incident, not a diff).
• Ledger & FEC checks: quarterly FEC export validated with the DGFiP Test Compta Demat tool (once the accounting-module binding is verified — compliance); numbering gaplessness across validate + avoir cycles; BlockedLog chain verification if adopted. All rehearsed on a sandbox checkpoint before running against prod.
• Idempotency tests: every write atom's test suite replays its own manifest twice and asserts a no-op second pass.

Fiscal parity checks

• Dry-run €-parity: fiscal sheets (T10) are compared to actually-filed values to the euro (POC-4); divergences block autonomy and open an investigation (books error vs. atom error — both are findings).
• Expert checkpoint: before the first agent-prepared filing of a new declaration type (first CA3 in 2027, first liasse), an expert-comptable (or SIE confirmation) validates the method once; after that, parity checks carry the load.
• Reconciliation invariant: the weekly zero-unexplained-deltas bar (T07) is itself a standing QA on the books.

Ops QA

• Heartbeats + silence alarms: every standing loop reports; the daily digest states "all green" explicitly — a missing digest is the alarm (C7).
• Monthly restore drill: latest prod backup restored into the sandbox + smoke-check, automated with a human-read report (T14).
• Quarterly game-day: deliberately take one tier down (revoke the cloud key, cordon the inference node, sleep the laptop) and verify the degraded-mode table holds in practice — same philosophy as the safe-prod-like-environment drills.
• Weekly ops review (human, ~10 min): escalation/quarantine/disagreement rates, DLQ age, digest accuracy spot-check, and the standing question: which atom cost more than it saved this week?

Independent verification — no self-grading

Work is never attested by the session that produced it. Milestone closures (closure protocol), POC exit gates, and autonomy promotions are verified by a context-free subagent prompted to refute ("find why this is NOT done / NOT at the bar"), whose only inputs are the repo, the issues, and the run journals — never the author's conversation. A refutation the author cannot resolve with evidence blocks the gate; the verifier's verdict is attached to the artifact it gates (milestone, registry autonomy field, POC record). This extends to the process level the principle the pipelines already run at the data level (dual independent extraction, seeded-wrong fixtures that must FAIL) and that the PRD itself was built with (fresh-reader review before first publication).

Cross-family verification. The harness runs on more than one model family (model-fleet › harness portability); a runtime is admitted to verifier duty by verdict parity with the Claude baseline on the two canonical verifier tests (locate-test + cold-reader backlog audit — erp#63, harness home erp:fleet/harness/). Once at least one non-Claude verifier is admitted, the independent verifier SHOULD be a different model family than the builder — a foreign family refuting the builder is stronger evidence than the builder's family agreeing with itself. Same-family verification stays acceptable when no admitted cross-family verifier fits the role, and the run journal records which family verified what.

Evidence trail

Every month yields an audit pack: the coherence audit (T15), the month's run journals, snapshot content-hashes, approval-card decisions, and fiscal sheets — archived in git + GED. This pack is deliberately shaped as the documented-control set of the piste d'audit fiable (CGI art. 289 VII — compliance): the invoice ↔ service ↔ payment linkage is evidenced continuously, not reconstructed under audit. The pack is written for a third party (expert-comptable, auditor, or a future operator): it must let them reconstruct what the fleet did and why without access to this PRD or any chat history. A distilled summary of each pack also lands in the second brain (T17), so institutional memory outlives both chat logs and this repo.