arcodangeandClaude Fable 5
58968135d0
Operator requirement (2026-07-11): mechanical measures against
hallucinated values in agent-performed edits. Four deterministic
layers in agent-architecture: (1) no free-text critical fields —
code copies values from validated extraction/bank-feed payloads,
the model never re-types them; (2) provenance anchors per critical
field {source_sha256, raw_excerpt} re-verified by a deterministic
checker against re-extracted source text (locale-normalized);
(3) cross-system corroboration against FRESH bank pulls + arithmetic
/checksum invariants; (4) read-back closes the loop. A failed check
refuses, never repairs. Woven into hub goal, C1 strategy, and QA
write-path (seeded-wrong-value fixtures must FAIL). Proven live:
the books-regularization pack shipped with a standalone
verify-provenance checker — 36 field-level checks green against
source PDFs + fresh Qonto/Wise feeds before the human gate.
Co-Authored-By: Claude Fable 5 <[email protected]>
7.4 KiB
7.4 KiB
vibe > PRD > AI back-office > QA strategy
QA strategy — how "done and safe" is proven
Status: In design Last Updated: 2026-07-11 Up: AI back-office hub Related: POC plan · Challenges · Agent architecture
The fleet's product is trustworthy books, so QA is not a phase — it is the operating system of the fleet: evals gate autonomy, writes assert their own deltas, fiscal outputs prove €-parity, and operations prove their own liveness.
Golden datasets
- Sources: real history — the 2026 mailbox (labeled by Claude, corrected by the human), every supplier invoice already recorded, filed declarations, bank feeds. Volumes are small, so every real item is a test case; synthetic edge cases (weird layouts, multi-rate invoices, credit notes) and adversarial injection fixtures pad the set.
- Storage: in the private Gitea (business data stays in the lab); one folder per atom:
inputs/,expected/,scoringscript. The datasets grow as a by-product of operation — every human correction, rejection reason, and reclassification is captured into the set (the approval card's "reject with reason" is a labeling interface). - Scoring: field-level, not document-level — a 9/10-fields extraction is a failed document but 90 % field accuracy; both numbers are tracked. Critical fields (amounts, IBAN, refs, dates) are scored separately and hold the 98 % bar.
Eval harness
- Per-atom regression: any change to an atom (prompt, model, version bump in the registry) re-runs its golden set; scores are committed alongside the change (a PR that degrades an atom's score is visible as such). ERP-dependent eval runs target a fresh sandbox checkpoint — the iso-prod refresh is a reproducible fixture (environments).
- Injection suite: every atom that reads untrusted content runs the adversarial fixtures; a single leak (instruction obeyed, field fabricated under influence) is a blocking failure regardless of the accuracy score.
- Disagreement telemetry: dual-extraction disagreement rates and escalation rates are recorded per run — a drift upward is an early-warning signal before accuracy visibly drops.
Autonomy promotion gates
Per atom, mechanical, recorded in the registry (ladder):
| Transition | Gate |
|---|---|
| A0 → A1 | golden set exists; atom passes it at its accuracy bar (≥ 98 % critical fields for extraction atoms). |
| A1 → A2 | ≥ 20 consecutive real items where the human's action was approve as-is (any field correction resets the counter); injection suite green. |
| A2 → A3 | read-only/reversible atoms only; 3 clean months at A2 + human sampling audit (random 10 % re-verified) with zero material findings. |
| Demotion | any incident (wrong write approved, missed deadline, injection leak) drops the atom one level; the path back up is the same gates, not seniority. |
Write-path QA
- Compliance linter:
promote-planrejects any manifest operation outside the production ledger grammar (mutating a validated document, supplying a manual ref where Dolibarr owns the sequence, detaching a GED original); the linter carries its own test suite of forbidden manifests. - Provenance check (anti-hallucination): before any approval card, the deterministic checker re-verifies every critical field against its source anchor — document text re-extracted, bank feed re-pulled fresh (write contract); a field without a passing anchor blocks the card. Hallucination is mechanically detectable, not merely reviewable; the checker's own test suite includes seeded-wrong-value fixtures that must FAIL.
- Predicted-delta assertion: every rehearsed manifest re-reads what the sandbox created and diffs it against the draft before the approval card goes out; a mismatch is a bug, never a "close enough".
- Post-write verification: after promote, the prod object is re-read and compared again; the pre/post snapshot pair (T13) must show exactly the journaled writes and nothing else — and the diff must be pure-append (a mutation of a validated document is an incident, not a diff).
- Ledger & FEC checks: quarterly FEC export validated with the DGFiP Test Compta Demat tool (once the accounting-module binding is verified — compliance); numbering gaplessness across validate + avoir cycles; BlockedLog chain verification if adopted. All rehearsed on a sandbox checkpoint before running against prod.
- Idempotency tests: every write atom's test suite replays its own manifest twice and asserts a no-op second pass.
Fiscal parity checks
- Dry-run €-parity: fiscal sheets (T10) are compared to actually-filed values to the euro (POC-4); divergences block autonomy and open an investigation (books error vs. atom error — both are findings).
- Expert checkpoint: before the first agent-prepared filing of a new declaration type (first CA3 in 2027, first liasse), an expert-comptable (or SIE confirmation) validates the method once; after that, parity checks carry the load.
- Reconciliation invariant: the weekly zero-unexplained-deltas bar (T07) is itself a standing QA on the books.
Ops QA
- Heartbeats + silence alarms: every standing loop reports; the daily digest states "all green" explicitly — a missing digest is the alarm (C7).
- Monthly restore drill: latest prod backup restored into the sandbox + smoke-check, automated with a human-read report (T14).
- Quarterly game-day: deliberately take one tier down (revoke the cloud key, cordon the inference node, sleep the laptop) and verify the degraded-mode table holds in practice — same philosophy as the safe-prod-like-environment drills.
- Weekly ops review (human, ~10 min): escalation/quarantine/disagreement rates, DLQ age, digest accuracy spot-check, and the standing question: which atom cost more than it saved this week?
Evidence trail
Every month yields an audit pack: the coherence audit (T15), the month's run journals, snapshot content-hashes, approval-card decisions, and fiscal sheets — archived in git + GED. This pack is deliberately shaped as the documented-control set of the piste d'audit fiable (CGI art. 289 VII — compliance): the invoice ↔ service ↔ payment linkage is evidenced continuously, not reconstructed under audit. The pack is written for a third party (expert-comptable, auditor, or a future operator): it must let them reconstruct what the fleet did and why without access to this PRD or any chat history. A distilled summary of each pack also lands in the second brain (T17), so institutional memory outlives both chat logs and this repo.