docs(prd): AI back-office — agent fleet for daily admin & accounting
New PRD tree vibe/PRD/ai-back-office/ (hub + 6 leaves + STATUS): task inventory T01-T16 with mode operatoire, atom/contract architecture on the ADR-0003 write gate, four-tier model fleet (Claude/Mistral/M4/Pi), 12 challenges with mitigations, POC plan with exit criteria, QA strategy with autonomy promotion gates. Index row + bidirectional backlinks (erp guidebook, safe-prod PRD). Co-Authored-By: Claude Fable 5 <[email protected]>
This commit is contained in:
@@ -0,0 +1,56 @@
|
||||
[vibe](../../README.md) > [PRD](../README.md) > [AI back-office](README.md) > **QA strategy**
|
||||
|
||||
# QA strategy — how "done and safe" is proven
|
||||
|
||||
> **Status:** In design
|
||||
> **Last Updated:** 2026-07-11
|
||||
> **Up:** [AI back-office hub](README.md)
|
||||
> **Related:** [POC plan](poc-plan.md) · [Challenges](challenges.md) · [Agent architecture](agent-architecture.md)
|
||||
|
||||
The fleet's product is *trustworthy books*, so QA is not a phase — it is the operating system of the fleet: evals gate autonomy, writes assert their own deltas, fiscal outputs prove €-parity, and operations prove their own liveness.
|
||||
|
||||
## Golden datasets
|
||||
|
||||
- **Sources:** real history — the 2026 mailbox (labeled by Claude, corrected by the human), every supplier invoice already recorded, filed declarations, bank feeds. Volumes are small, so *every* real item is a test case; synthetic edge cases (weird layouts, multi-rate invoices, credit notes) and **adversarial injection fixtures** pad the set.
|
||||
- **Storage:** in the private Gitea (business data stays in the lab); one folder per atom: `inputs/`, `expected/`, `scoring` script. The datasets grow as a by-product of operation — every human correction, rejection reason, and reclassification is captured into the set (the approval card's "reject with reason" is a labeling interface).
|
||||
- **Scoring:** field-level, not document-level — a 9/10-fields extraction is a *failed* document but 90 % field accuracy; both numbers are tracked. Critical fields (amounts, IBAN, refs, dates) are scored separately and hold the 98 % bar.
|
||||
|
||||
## Eval harness
|
||||
|
||||
- **Per-atom regression:** any change to an atom (prompt, model, version bump in the registry) re-runs its golden set; scores are committed alongside the change (a PR that degrades an atom's score is visible as such).
|
||||
- **Injection suite:** every atom that reads untrusted content runs the adversarial fixtures; a single leak (instruction obeyed, field fabricated under influence) is a blocking failure regardless of the accuracy score.
|
||||
- **Disagreement telemetry:** dual-extraction disagreement rates and escalation rates are recorded per run — a drift upward is an early-warning signal *before* accuracy visibly drops.
|
||||
|
||||
## Autonomy promotion gates
|
||||
|
||||
Per atom, mechanical, recorded in the registry ([ladder](README.md#the-autonomy-ladder)):
|
||||
|
||||
| Transition | Gate |
|
||||
| --- | --- |
|
||||
| A0 → A1 | golden set exists; atom passes it at its accuracy bar (≥ 98 % critical fields for extraction atoms). |
|
||||
| A1 → A2 | ≥ 20 consecutive real items where the human's action was *approve as-is* (any field correction resets the counter); injection suite green. |
|
||||
| A2 → A3 | read-only/reversible atoms only; 3 clean months at A2 + human sampling audit (random 10 % re-verified) with zero material findings. |
|
||||
| Demotion | any incident (wrong write approved, missed deadline, injection leak) drops the atom one level; the path back up is the same gates, not seniority. |
|
||||
|
||||
## Write-path QA
|
||||
|
||||
- **Predicted-delta assertion:** every rehearsed manifest re-reads what the sandbox created and diffs it against the draft *before* the approval card goes out; a mismatch is a bug, never a "close enough".
|
||||
- **Post-write verification:** after promote, the prod object is re-read and compared again; the pre/post snapshot pair ([T13](task-inventory.md#t13--erp-snapshot--drift-detection)) must show *exactly* the journaled writes and nothing else.
|
||||
- **Idempotency tests:** every write atom's test suite replays its own manifest twice and asserts a no-op second pass.
|
||||
|
||||
## Fiscal parity checks
|
||||
|
||||
- **Dry-run €-parity:** fiscal sheets ([T10](task-inventory.md#t10--tva-preparation)) are compared to actually-filed values to the euro ([POC-4](poc-plan.md#poc-4--tva-dry-run)); divergences block autonomy and open an investigation (books error vs. atom error — both are findings).
|
||||
- **Expert checkpoint:** before the first agent-prepared filing of a new declaration type (first CA3 in 2027, first liasse), an expert-comptable (or SIE confirmation) validates the method once; after that, parity checks carry the load.
|
||||
- **Reconciliation invariant:** the weekly zero-unexplained-deltas bar ([T07](task-inventory.md#t07--bank-reconciliation)) is itself a standing QA on the books.
|
||||
|
||||
## Ops QA
|
||||
|
||||
- **Heartbeats + silence alarms:** every standing loop reports; the daily digest states "all green" explicitly — a *missing* digest is the alarm ([C7](challenges.md#c7--silent-failures-in-unattended-operation)).
|
||||
- **Monthly restore drill:** latest prod backup restored into the sandbox + smoke-check, automated with a human-read report ([T14](task-inventory.md#t14--backup--restore-verification)).
|
||||
- **Quarterly game-day:** deliberately take one tier down (revoke the cloud key, cordon the inference node, sleep the laptop) and verify the [degraded-mode table](model-fleet.md#degraded-modes) holds in practice — same philosophy as the [safe-prod-like-environment](../safe-prod-like-environment/README.md) drills.
|
||||
- **Weekly ops review (human, ~10 min):** escalation/quarantine/disagreement rates, DLQ age, digest accuracy spot-check, and the standing question: *which atom cost more than it saved this week?*
|
||||
|
||||
## Evidence trail
|
||||
|
||||
Every month yields an audit pack: the coherence audit ([T15](task-inventory.md#t15--monthly-coherence-audit)), the month's run journals, snapshot content-hashes, approval-card decisions, and fiscal sheets — archived in git + GED. The pack is written for a third party (expert-comptable, auditor, or a future operator): it must let them reconstruct *what the fleet did and why* without access to this PRD or any chat history.
|
||||
Reference in New Issue
Block a user