factory/vibe/PRD/ai-back-office/qa-strategy.md

vibe > PRD > AI back-office > QA strategy

QA strategy — how "done and safe" is proven

Status: In design Last Updated: 2026-07-11 Up: AI back-office hub Related: POC plan · Challenges · Agent architecture

The fleet's product is trustworthy books, so QA is not a phase — it is the operating system of the fleet: evals gate autonomy, writes assert their own deltas, fiscal outputs prove €-parity, and operations prove their own liveness.

Golden datasets

• Sources: real history — the 2026 mailbox (labeled by Claude, corrected by the human), every supplier invoice already recorded, filed declarations, bank feeds. Volumes are small, so every real item is a test case; synthetic edge cases (weird layouts, multi-rate invoices, credit notes) and adversarial injection fixtures pad the set.
• Storage: in the private Gitea (business data stays in the lab); one folder per atom: inputs/, expected/, scoring script. The datasets grow as a by-product of operation — every human correction, rejection reason, and reclassification is captured into the set (the approval card's "reject with reason" is a labeling interface).
• Scoring: field-level, not document-level — a 9/10-fields extraction is a failed document but 90 % field accuracy; both numbers are tracked. Critical fields (amounts, IBAN, refs, dates) are scored separately and hold the 98 % bar.

Eval harness

• Per-atom regression: any change to an atom (prompt, model, version bump in the registry) re-runs its golden set; scores are committed alongside the change (a PR that degrades an atom's score is visible as such).
• Injection suite: every atom that reads untrusted content runs the adversarial fixtures; a single leak (instruction obeyed, field fabricated under influence) is a blocking failure regardless of the accuracy score.
• Disagreement telemetry: dual-extraction disagreement rates and escalation rates are recorded per run — a drift upward is an early-warning signal before accuracy visibly drops.

Autonomy promotion gates

Per atom, mechanical, recorded in the registry (ladder):

Transition	Gate
A0 → A1	golden set exists; atom passes it at its accuracy bar (≥ 98 % critical fields for extraction atoms).
A1 → A2	≥ 20 consecutive real items where the human's action was approve as-is (any field correction resets the counter); injection suite green.
A2 → A3	read-only/reversible atoms only; 3 clean months at A2 + human sampling audit (random 10 % re-verified) with zero material findings.
Demotion	any incident (wrong write approved, missed deadline, injection leak) drops the atom one level; the path back up is the same gates, not seniority.

Write-path QA

• Predicted-delta assertion: every rehearsed manifest re-reads what the sandbox created and diffs it against the draft before the approval card goes out; a mismatch is a bug, never a "close enough".
• Post-write verification: after promote, the prod object is re-read and compared again; the pre/post snapshot pair (T13) must show exactly the journaled writes and nothing else.
• Idempotency tests: every write atom's test suite replays its own manifest twice and asserts a no-op second pass.

Fiscal parity checks

• Dry-run €-parity: fiscal sheets (T10) are compared to actually-filed values to the euro (POC-4); divergences block autonomy and open an investigation (books error vs. atom error — both are findings).
• Expert checkpoint: before the first agent-prepared filing of a new declaration type (first CA3 in 2027, first liasse), an expert-comptable (or SIE confirmation) validates the method once; after that, parity checks carry the load.
• Reconciliation invariant: the weekly zero-unexplained-deltas bar (T07) is itself a standing QA on the books.

Ops QA

• Heartbeats + silence alarms: every standing loop reports; the daily digest states "all green" explicitly — a missing digest is the alarm (C7).
• Monthly restore drill: latest prod backup restored into the sandbox + smoke-check, automated with a human-read report (T14).
• Quarterly game-day: deliberately take one tier down (revoke the cloud key, cordon the inference node, sleep the laptop) and verify the degraded-mode table holds in practice — same philosophy as the safe-prod-like-environment drills.
• Weekly ops review (human, ~10 min): escalation/quarantine/disagreement rates, DLQ age, digest accuracy spot-check, and the standing question: which atom cost more than it saved this week?

Evidence trail

Every month yields an audit pack: the coherence audit (T15), the month's run journals, snapshot content-hashes, approval-card decisions, and fiscal sheets — archived in git + GED. The pack is written for a third party (expert-comptable, auditor, or a future operator): it must let them reconstruct what the fleet did and why without access to this PRD or any chat history.