factory/vibe/PRD/ai-back-office/qa-strategy.md

[vibe](../../README.md) > [PRD](../README.md) > [AI back-office](README.md) > **QA strategy**

# QA strategy — how "done and safe" is proven

> **Status:** In design
> **Last Updated:** 2026-07-11
> **Up:** [AI back-office hub](README.md)
> **Related:** [POC plan](poc-plan.md) · [Challenges](challenges.md) · [Agent architecture](agent-architecture.md)

The fleet's product is *trustworthy books*, so QA is not a phase — it is the operating system of the fleet: evals gate autonomy, writes assert their own deltas, fiscal outputs prove €-parity, and operations prove their own liveness.

## Golden datasets

- **Sources:** real history — the 2026 mailbox (labeled by Claude, corrected by the human), every supplier invoice already recorded, filed declarations, bank feeds. Volumes are small, so *every* real item is a test case; synthetic edge cases (weird layouts, multi-rate invoices, credit notes) and **adversarial injection fixtures** pad the set.
- **Storage:** in the private Gitea (business data stays in the lab); one folder per atom: `inputs/`, `expected/`, `scoring` script. The datasets grow as a by-product of operation — every human correction, rejection reason, and reclassification is captured into the set (the approval card's "reject with reason" is a labeling interface).
- **Scoring:** field-level, not document-level — a 9/10-fields extraction is a *failed* document but 90 % field accuracy; both numbers are tracked. Critical fields (amounts, IBAN, refs, dates) are scored separately and hold the 98 % bar.

## Eval harness

- **Per-atom regression:** any change to an atom (prompt, model, version bump in the registry) re-runs its golden set; scores are committed alongside the change (a PR that degrades an atom's score is visible as such). ERP-dependent eval runs target a **fresh sandbox checkpoint** — the iso-prod refresh is a reproducible fixture ([environments](agent-architecture.md#environments--sandbox-vs-production)).
- **Injection suite:** every atom that reads untrusted content runs the adversarial fixtures; a single leak (instruction obeyed, field fabricated under influence) is a blocking failure regardless of the accuracy score.
- **Disagreement telemetry:** dual-extraction disagreement rates and escalation rates are recorded per run — a drift upward is an early-warning signal *before* accuracy visibly drops.

## Autonomy promotion gates

Per atom, mechanical, recorded in the registry ([ladder](README.md#the-autonomy-ladder)):

| Transition | Gate |
| --- | --- |
| A0 → A1 | golden set exists; atom passes it at its accuracy bar (≥ 98 % critical fields for extraction atoms). |
| A1 → A2 | ≥ 20 consecutive real items where the human's action was *approve as-is* (any field correction resets the counter); injection suite green. |
| A2 → A3 | read-only/reversible atoms only; 3 clean months at A2 + human sampling audit (random 10 % re-verified) with zero material findings. |
| Demotion | any incident (wrong write approved, missed deadline, injection leak) drops the atom one level; the path back up is the same gates, not seniority. |

## Write-path QA

- **Compliance linter:** `promote-plan` rejects any manifest operation outside the production [ledger grammar](compliance.md#the-ledger-grammar-production) (mutating a validated document, supplying a manual ref where Dolibarr owns the sequence, detaching a GED original); the linter carries its own test suite of forbidden manifests.
- **Predicted-delta assertion:** every rehearsed manifest re-reads what the sandbox created and diffs it against the draft *before* the approval card goes out; a mismatch is a bug, never a "close enough".
- **Post-write verification:** after promote, the prod object is re-read and compared again; the pre/post snapshot pair ([T13](task-inventory.md#t13--erp-snapshot--drift-detection)) must show *exactly* the journaled writes and nothing else — and the diff must be **pure-append** (a mutation of a validated document is an incident, not a diff).
- **Ledger & FEC checks:** quarterly FEC export validated with the DGFiP *Test Compta Demat* tool (once the accounting-module binding is verified — [compliance](compliance.md#dolibarr-verifications-sandbox-first)); numbering gaplessness across validate + avoir cycles; BlockedLog chain verification if adopted. All rehearsed on a sandbox checkpoint before running against prod.
- **Idempotency tests:** every write atom's test suite replays its own manifest twice and asserts a no-op second pass.

## Fiscal parity checks

- **Dry-run €-parity:** fiscal sheets ([T10](task-inventory.md#t10--tva-preparation)) are compared to actually-filed values to the euro ([POC-4](poc-plan.md#poc-4--tva-dry-run)); divergences block autonomy and open an investigation (books error vs. atom error — both are findings).
- **Expert checkpoint:** before the first agent-prepared filing of a new declaration type (first CA3 in 2027, first liasse), an expert-comptable (or SIE confirmation) validates the method once; after that, parity checks carry the load.
- **Reconciliation invariant:** the weekly zero-unexplained-deltas bar ([T07](task-inventory.md#t07--bank-reconciliation)) is itself a standing QA on the books.

## Ops QA

- **Heartbeats + silence alarms:** every standing loop reports; the daily digest states "all green" explicitly — a *missing* digest is the alarm ([C7](challenges.md#c7--silent-failures-in-unattended-operation)).
- **Monthly restore drill:** latest prod backup restored into the sandbox + smoke-check, automated with a human-read report ([T14](task-inventory.md#t14--backup--restore-verification)).
- **Quarterly game-day:** deliberately take one tier down (revoke the cloud key, cordon the inference node, sleep the laptop) and verify the [degraded-mode table](model-fleet.md#degraded-modes) holds in practice — same philosophy as the [safe-prod-like-environment](../safe-prod-like-environment/README.md) drills.
- **Weekly ops review (human, ~10 min):** escalation/quarantine/disagreement rates, DLQ age, digest accuracy spot-check, and the standing question: *which atom cost more than it saved this week?*

## Evidence trail

Every month yields an audit pack: the coherence audit ([T15](task-inventory.md#t15--monthly-coherence-audit)), the month's run journals, snapshot content-hashes, approval-card decisions, and fiscal sheets — archived in git + GED. This pack is deliberately shaped as the documented-control set of the **piste d'audit fiable** (CGI art. 289 VII — [compliance](compliance.md#obligations--fleet-mechanisms)): the invoice ↔ service ↔ payment linkage is evidenced continuously, not reconstructed under audit. The pack is written for a third party (expert-comptable, auditor, or a future operator): it must let them reconstruct *what the fleet did and why* without access to this PRD or any chat history. A distilled summary of each pack also lands in the second brain ([T17](task-inventory.md#t17--knowledge-capture--retrieval-second-brain)), so institutional memory outlives both chat logs and this repo.