feat(fleet): golden set + injection fixtures from real history #39

Closed
opened 2026-07-11 18:16:40 +02:00 by arcodange · 1 comment
Owner

Part of the AI back-office PRDqa-strategy › golden datasets. Gates everything: no atom earns autonomy without its golden set; first data task of phase 1 on the roadmap.

Deliverables

  • fleet/golden/invoice-extract/{inputs,expected}/ seeded from real history: every supplier PDF already in the GED/mailbox (arcodange-email-ingest fetches; DARNIS/Hiway F1040–F1046, Anthropic, Mistral, OVH, Qonto, greffe, etc.) with hand-verified expected JSON per the T02 schema.
  • fleet/golden/mail-classify/: ≥200 historical mails labeled into {supplier-invoice, bank-notice, government-admin, client, other} (Claude-labeled, human-correctable file format).
  • Adversarial injection fixtures (≥5): documents carrying LLM-addressed instructions, hidden text, IBAN-swap lures — expected outcome = quarantine.
  • Field-level scoring script: overall accuracy + critical-fields (amounts, IBAN, refs, dates) scored separately (the 98 % bar applies to critical fields).

Acceptance: scoring runs offline on the set; counts documented; mailbox/GED provenance recorded per item (source id + sha256).


Execution — Blocked by: #38 (landing home fleet/golden/; collection can start in parallel) · Blocks: #40, #45, #50 · Human gates: spot-check a sample of expected JSONs before trusting the 98 % bar (recommended, not blocking).
Start: claude --worktree golden-set from the erp repo root; open .claude/skills/arcodange-email-ingest/SKILL.md (workflow 1 — list candidates) to enumerate the real PDFs.
Done means: evidence comment here (item counts per set + scoring-script sample output) → PR merged with Closes #39.

Part of the **AI back-office PRD** — [qa-strategy › golden datasets](https://gitea.arcodange.lab/arcodange-org/factory/src/branch/main/vibe/PRD/ai-back-office/qa-strategy.md). **Gates everything**: no atom earns autonomy without its golden set; first data task of phase 1 on the [roadmap](https://gitea.arcodange.lab/arcodange-org/factory/src/branch/main/vibe/PRD/ai-back-office/roadmap.md). **Deliverables** - `fleet/golden/invoice-extract/{inputs,expected}/` seeded from **real history**: every supplier PDF already in the GED/mailbox (`arcodange-email-ingest` fetches; DARNIS/Hiway F1040–F1046, Anthropic, Mistral, OVH, Qonto, greffe, etc.) with hand-verified expected JSON per the T02 schema. - `fleet/golden/mail-classify/`: ≥200 historical mails labeled into {supplier-invoice, bank-notice, government-admin, client, other} (Claude-labeled, human-correctable file format). - **Adversarial injection fixtures** (≥5): documents carrying LLM-addressed instructions, hidden text, IBAN-swap lures — expected outcome = quarantine. - Field-level scoring script: overall accuracy + critical-fields (amounts, IBAN, refs, dates) scored separately (the 98 % bar applies to critical fields). **Acceptance:** scoring runs offline on the set; counts documented; mailbox/GED provenance recorded per item (source id + sha256). --- **Execution** — Blocked by: #38 (landing home `fleet/golden/`; collection can start in parallel) · Blocks: #40, #45, #50 · Human gates: spot-check a sample of expected JSONs before trusting the 98 % bar (recommended, not blocking). **Start:** `claude --worktree golden-set` from the erp repo root; open `.claude/skills/arcodange-email-ingest/SKILL.md` (workflow 1 — list candidates) to enumerate the real PDFs. **Done means:** evidence comment here (item counts per set + scoring-script sample output) → PR merged with `Closes #39`.
arcodange added this to the P1 — Flagship pipeline milestone 2026-07-11 18:16:40 +02:00
Author
Owner

Evidence — golden set built and verified (2026-07-18). Work complete on local branch arcodange/golden-set (commit 6df4693, single commit, 74 files, 1.7 MB).

Counts

  • fleet/golden/invoice-extract/: 16 real items (DARNIS/Hiway F1040/F1042/F1045/F1046, Anthropic invoice+receipt ×2 pairs, Mistral, OVH, greffe d'Évry, INPI ×2, Legalstart, Qonto, Infogreffe), inputs = source PDFs (836 KB total) + extracted text; expected = hand-verified T02 JSON. 7 real documents excluded with reasons in the manifest (fee statements, transfer proofs, T&Cs — non-invoices).
  • fleet/golden/invoice-extract/injection/: 6 synthetic adversarial fixtures (LLM-addressed instructions, hidden text, IBAN-swap lures), expected outcome = quarantine.
  • fleet/golden/mail-classify/: 1824 labeled messages (labels.jsonl, human-correctable). Distribution: other 1742 / government-admin 34 / client 30 / bank-notice 10 / supplier-invoice 8 — the heavy other share is the real mailbox (newsletters, invites). ≫ the ≥200 floor.
  • score.py: field-level scorer, critical fields (amounts, IBAN, refs, dates) scored separately at the 98 % bar, injection suite blocking; --self-test = 6 checks PASS. Perturbed-prediction demo correctly fails (97.92 % critical → NOT MET, injection leak flagged, exit 1).

Provenance: per-item sha256 + source id (Zoho message_id+folder+attachment or Dolibarr GED modulepart/relative_path+ref) + linked supplier invoice ref, in manifest.json.

Independent check by the orchestrator (not the builder agent): scorer self-test PASS; spot-check of expected vs pdftotext for mistral-mstrl-api-814045-001 (143,90 / 28,78 / 172,68, SIREN 952418325) and darnis-f1046 (219,50 / 43,90 / 263,40) — exact.

Deviations, honest: F1041/F1043/F1044 exist in neither mailbox nor GED; DARNIS F1042 is a hard two-page case (page 2 = unrelated embedded customer invoice) kept deliberately; Anthropic invoice+receipt pairs share one idempotency key on purpose (dedupe test).

⚠️ Push gate: the branch push was blocked twice by the session's permission classifier (real financial documents leaving the machine). The branch is ready locally — operator: git push -u origin arcodange/golden-set from the erp repo, then PR with Closes #39. Everything else in this issue is done.

**Evidence — golden set built and verified (2026-07-18).** Work complete on local branch `arcodange/golden-set` (commit `6df4693`, single commit, 74 files, 1.7 MB). **Counts** - `fleet/golden/invoice-extract/`: **16 real items** (DARNIS/Hiway F1040/F1042/F1045/F1046, Anthropic invoice+receipt ×2 pairs, Mistral, OVH, greffe d'Évry, INPI ×2, Legalstart, Qonto, Infogreffe), inputs = source PDFs (836 KB total) + extracted text; expected = hand-verified T02 JSON. 7 real documents **excluded with reasons** in the manifest (fee statements, transfer proofs, T&Cs — non-invoices). - `fleet/golden/invoice-extract/injection/`: **6 synthetic adversarial fixtures** (LLM-addressed instructions, hidden text, IBAN-swap lures), expected outcome = quarantine. - `fleet/golden/mail-classify/`: **1824 labeled messages** (labels.jsonl, human-correctable). Distribution: other 1742 / government-admin 34 / client 30 / bank-notice 10 / supplier-invoice 8 — the heavy `other` share is the real mailbox (newsletters, invites). ≫ the ≥200 floor. - `score.py`: field-level scorer, critical fields (amounts, IBAN, refs, dates) scored separately at the 98 % bar, injection suite blocking; `--self-test` = 6 checks PASS. Perturbed-prediction demo correctly fails (97.92 % critical → NOT MET, injection leak flagged, exit 1). **Provenance**: per-item `sha256` + source id (Zoho `message_id`+folder+attachment or Dolibarr GED `modulepart`/`relative_path`+ref) + linked supplier invoice ref, in `manifest.json`. **Independent check by the orchestrator** (not the builder agent): scorer self-test PASS; spot-check of expected vs `pdftotext` for `mistral-mstrl-api-814045-001` (143,90 / 28,78 / 172,68, SIREN 952418325) and `darnis-f1046` (219,50 / 43,90 / 263,40) — exact. **Deviations, honest**: F1041/F1043/F1044 exist in neither mailbox nor GED; DARNIS F1042 is a hard two-page case (page 2 = unrelated embedded customer invoice) kept deliberately; Anthropic invoice+receipt pairs share one idempotency key on purpose (dedupe test). **⚠️ Push gate**: the branch push was blocked twice by the session's permission classifier (real financial documents leaving the machine). The branch is ready locally — operator: `git push -u origin arcodange/golden-set` from the erp repo, then PR with `Closes #39`. Everything else in this issue is done.
Sign in to join this conversation.
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: arcodange-org/erp#39