Part of the AI back-office PRD — qa-strategy › golden datasets. Gates everything: no atom earns autonomy without its golden set; first data task of phase 1 on the roadmap.
Deliverables
fleet/golden/invoice-extract/{inputs,expected}/ seeded from real history: every supplier PDF already in the GED/mailbox (arcodange-email-ingest fetches; DARNIS/Hiway F1040–F1046, Anthropic, Mistral, OVH, Qonto, greffe, etc.) with hand-verified expected JSON per the T02 schema.
Field-level scoring script: overall accuracy + critical-fields (amounts, IBAN, refs, dates) scored separately (the 98 % bar applies to critical fields).
Acceptance: scoring runs offline on the set; counts documented; mailbox/GED provenance recorded per item (source id + sha256).
Execution — Blocked by: #38 (landing home fleet/golden/; collection can start in parallel) · Blocks: #40, #45, #50 · Human gates: spot-check a sample of expected JSONs before trusting the 98 % bar (recommended, not blocking). Start:claude --worktree golden-set from the erp repo root; open .claude/skills/arcodange-email-ingest/SKILL.md (workflow 1 — list candidates) to enumerate the real PDFs. Done means: evidence comment here (item counts per set + scoring-script sample output) → PR merged with Closes #39.
Part of the **AI back-office PRD** — [qa-strategy › golden datasets](https://gitea.arcodange.lab/arcodange-org/factory/src/branch/main/vibe/PRD/ai-back-office/qa-strategy.md). **Gates everything**: no atom earns autonomy without its golden set; first data task of phase 1 on the [roadmap](https://gitea.arcodange.lab/arcodange-org/factory/src/branch/main/vibe/PRD/ai-back-office/roadmap.md).
**Deliverables**
- `fleet/golden/invoice-extract/{inputs,expected}/` seeded from **real history**: every supplier PDF already in the GED/mailbox (`arcodange-email-ingest` fetches; DARNIS/Hiway F1040–F1046, Anthropic, Mistral, OVH, Qonto, greffe, etc.) with hand-verified expected JSON per the T02 schema.
- `fleet/golden/mail-classify/`: ≥200 historical mails labeled into {supplier-invoice, bank-notice, government-admin, client, other} (Claude-labeled, human-correctable file format).
- **Adversarial injection fixtures** (≥5): documents carrying LLM-addressed instructions, hidden text, IBAN-swap lures — expected outcome = quarantine.
- Field-level scoring script: overall accuracy + critical-fields (amounts, IBAN, refs, dates) scored separately (the 98 % bar applies to critical fields).
**Acceptance:** scoring runs offline on the set; counts documented; mailbox/GED provenance recorded per item (source id + sha256).
---
**Execution** — Blocked by: #38 (landing home `fleet/golden/`; collection can start in parallel) · Blocks: #40, #45, #50 · Human gates: spot-check a sample of expected JSONs before trusting the 98 % bar (recommended, not blocking).
**Start:** `claude --worktree golden-set` from the erp repo root; open `.claude/skills/arcodange-email-ingest/SKILL.md` (workflow 1 — list candidates) to enumerate the real PDFs.
**Done means:** evidence comment here (item counts per set + scoring-script sample output) → PR merged with `Closes #39`.
arcodange
added this to the P1 — Flagship pipeline milestone 2026-07-11 18:16:40 +02:00
Evidence — golden set built and verified (2026-07-18). Work complete on local branch arcodange/golden-set (commit 6df4693, single commit, 74 files, 1.7 MB).
Counts
fleet/golden/invoice-extract/: 16 real items (DARNIS/Hiway F1040/F1042/F1045/F1046, Anthropic invoice+receipt ×2 pairs, Mistral, OVH, greffe d'Évry, INPI ×2, Legalstart, Qonto, Infogreffe), inputs = source PDFs (836 KB total) + extracted text; expected = hand-verified T02 JSON. 7 real documents excluded with reasons in the manifest (fee statements, transfer proofs, T&Cs — non-invoices).
fleet/golden/mail-classify/: 1824 labeled messages (labels.jsonl, human-correctable). Distribution: other 1742 / government-admin 34 / client 30 / bank-notice 10 / supplier-invoice 8 — the heavy other share is the real mailbox (newsletters, invites). ≫ the ≥200 floor.
score.py: field-level scorer, critical fields (amounts, IBAN, refs, dates) scored separately at the 98 % bar, injection suite blocking; --self-test = 6 checks PASS. Perturbed-prediction demo correctly fails (97.92 % critical → NOT MET, injection leak flagged, exit 1).
Provenance: per-item sha256 + source id (Zoho message_id+folder+attachment or Dolibarr GED modulepart/relative_path+ref) + linked supplier invoice ref, in manifest.json.
Independent check by the orchestrator (not the builder agent): scorer self-test PASS; spot-check of expected vs pdftotext for mistral-mstrl-api-814045-001 (143,90 / 28,78 / 172,68, SIREN 952418325) and darnis-f1046 (219,50 / 43,90 / 263,40) — exact.
Deviations, honest: F1041/F1043/F1044 exist in neither mailbox nor GED; DARNIS F1042 is a hard two-page case (page 2 = unrelated embedded customer invoice) kept deliberately; Anthropic invoice+receipt pairs share one idempotency key on purpose (dedupe test).
⚠️ Push gate: the branch push was blocked twice by the session's permission classifier (real financial documents leaving the machine). The branch is ready locally — operator: git push -u origin arcodange/golden-set from the erp repo, then PR with Closes #39. Everything else in this issue is done.
**Evidence — golden set built and verified (2026-07-18).** Work complete on local branch `arcodange/golden-set` (commit `6df4693`, single commit, 74 files, 1.7 MB).
**Counts**
- `fleet/golden/invoice-extract/`: **16 real items** (DARNIS/Hiway F1040/F1042/F1045/F1046, Anthropic invoice+receipt ×2 pairs, Mistral, OVH, greffe d'Évry, INPI ×2, Legalstart, Qonto, Infogreffe), inputs = source PDFs (836 KB total) + extracted text; expected = hand-verified T02 JSON. 7 real documents **excluded with reasons** in the manifest (fee statements, transfer proofs, T&Cs — non-invoices).
- `fleet/golden/invoice-extract/injection/`: **6 synthetic adversarial fixtures** (LLM-addressed instructions, hidden text, IBAN-swap lures), expected outcome = quarantine.
- `fleet/golden/mail-classify/`: **1824 labeled messages** (labels.jsonl, human-correctable). Distribution: other 1742 / government-admin 34 / client 30 / bank-notice 10 / supplier-invoice 8 — the heavy `other` share is the real mailbox (newsletters, invites). ≫ the ≥200 floor.
- `score.py`: field-level scorer, critical fields (amounts, IBAN, refs, dates) scored separately at the 98 % bar, injection suite blocking; `--self-test` = 6 checks PASS. Perturbed-prediction demo correctly fails (97.92 % critical → NOT MET, injection leak flagged, exit 1).
**Provenance**: per-item `sha256` + source id (Zoho `message_id`+folder+attachment or Dolibarr GED `modulepart`/`relative_path`+ref) + linked supplier invoice ref, in `manifest.json`.
**Independent check by the orchestrator** (not the builder agent): scorer self-test PASS; spot-check of expected vs `pdftotext` for `mistral-mstrl-api-814045-001` (143,90 / 28,78 / 172,68, SIREN 952418325) and `darnis-f1046` (219,50 / 43,90 / 263,40) — exact.
**Deviations, honest**: F1041/F1043/F1044 exist in neither mailbox nor GED; DARNIS F1042 is a hard two-page case (page 2 = unrelated embedded customer invoice) kept deliberately; Anthropic invoice+receipt pairs share one idempotency key on purpose (dedupe test).
**⚠️ Push gate**: the branch push was blocked twice by the session's permission classifier (real financial documents leaving the machine). The branch is ready locally — operator: `git push -u origin arcodange/golden-set` from the erp repo, then PR with `Closes #39`. Everything else in this issue is done.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Part of the AI back-office PRD — qa-strategy › golden datasets. Gates everything: no atom earns autonomy without its golden set; first data task of phase 1 on the roadmap.
Deliverables
fleet/golden/invoice-extract/{inputs,expected}/seeded from real history: every supplier PDF already in the GED/mailbox (arcodange-email-ingestfetches; DARNIS/Hiway F1040–F1046, Anthropic, Mistral, OVH, Qonto, greffe, etc.) with hand-verified expected JSON per the T02 schema.fleet/golden/mail-classify/: ≥200 historical mails labeled into {supplier-invoice, bank-notice, government-admin, client, other} (Claude-labeled, human-correctable file format).Acceptance: scoring runs offline on the set; counts documented; mailbox/GED provenance recorded per item (source id + sha256).
Execution — Blocked by: #38 (landing home
fleet/golden/; collection can start in parallel) · Blocks: #40, #45, #50 · Human gates: spot-check a sample of expected JSONs before trusting the 98 % bar (recommended, not blocking).Start:
claude --worktree golden-setfrom the erp repo root; open.claude/skills/arcodange-email-ingest/SKILL.md(workflow 1 — list candidates) to enumerate the real PDFs.Done means: evidence comment here (item counts per set + scoring-script sample output) → PR merged with
Closes #39.Evidence — golden set built and verified (2026-07-18). Work complete on local branch
arcodange/golden-set(commit6df4693, single commit, 74 files, 1.7 MB).Counts
fleet/golden/invoice-extract/: 16 real items (DARNIS/Hiway F1040/F1042/F1045/F1046, Anthropic invoice+receipt ×2 pairs, Mistral, OVH, greffe d'Évry, INPI ×2, Legalstart, Qonto, Infogreffe), inputs = source PDFs (836 KB total) + extracted text; expected = hand-verified T02 JSON. 7 real documents excluded with reasons in the manifest (fee statements, transfer proofs, T&Cs — non-invoices).fleet/golden/invoice-extract/injection/: 6 synthetic adversarial fixtures (LLM-addressed instructions, hidden text, IBAN-swap lures), expected outcome = quarantine.fleet/golden/mail-classify/: 1824 labeled messages (labels.jsonl, human-correctable). Distribution: other 1742 / government-admin 34 / client 30 / bank-notice 10 / supplier-invoice 8 — the heavyothershare is the real mailbox (newsletters, invites). ≫ the ≥200 floor.score.py: field-level scorer, critical fields (amounts, IBAN, refs, dates) scored separately at the 98 % bar, injection suite blocking;--self-test= 6 checks PASS. Perturbed-prediction demo correctly fails (97.92 % critical → NOT MET, injection leak flagged, exit 1).Provenance: per-item
sha256+ source id (Zohomessage_id+folder+attachment or Dolibarr GEDmodulepart/relative_path+ref) + linked supplier invoice ref, inmanifest.json.Independent check by the orchestrator (not the builder agent): scorer self-test PASS; spot-check of expected vs
pdftotextformistral-mstrl-api-814045-001(143,90 / 28,78 / 172,68, SIREN 952418325) anddarnis-f1046(219,50 / 43,90 / 263,40) — exact.Deviations, honest: F1041/F1043/F1044 exist in neither mailbox nor GED; DARNIS F1042 is a hard two-page case (page 2 = unrelated embedded customer invoice) kept deliberately; Anthropic invoice+receipt pairs share one idempotency key on purpose (dedupe test).
⚠️ Push gate: the branch push was blocked twice by the session's permission classifier (real financial documents leaving the machine). The branch is ready locally — operator:
git push -u origin arcodange/golden-setfrom the erp repo, then PR withCloses #39. Everything else in this issue is done.