feat(fleet): golden set + injection fixtures from real history #73

Merged
arcodange merged 1 commits from arcodange/golden-set into main 2026-07-19 09:22:10 +02:00
Owner

Closes #39.

Seeds fleet/golden/ from real Arcodange history (74 files, 1.7 MB — full counts, provenance method and deviations in the #39 evidence comment):

  • invoice-extract/: 16 real supplier documents (inputs = source PDFs + pdftotext layer, expected = hand-verified T02 JSON), 7 real non-invoices excluded with reasons in the manifest.
  • invoice-extract/injection/: 6 synthetic adversarial fixtures (LLM-addressed directives, hidden white text, IBAN-swap, arithmetic-repair lure, fake tool-call, ref hijack) — expected outcome = quarantine, one leak = blocking.
  • mail-classify/: 1824 labeled messages (human-correctable JSONL).
  • score.py: field-level scorer, critical fields at the 98 % bar, injection suite blocking, --self-test built in.

Per-item provenance (source id + sha256) in manifest.json; independently spot-checked by the orchestrator (scorer self-test PASS; expected-vs-pdftotext exact on 2 sampled docs) and corroborated by erp#43's independent re-fetch (4/4 sha256 match).

Push previously gated on the operator (real financial documents) — released by the operator's explicit go this session.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VRShc4QhLLU73FLHx9vskh

Closes #39. Seeds `fleet/golden/` from real Arcodange history (74 files, 1.7 MB — full counts, provenance method and deviations in the [#39 evidence comment](https://gitea.arcodange.lab/arcodange-org/erp/issues/39#issuecomment-2950)): - `invoice-extract/`: **16 real supplier documents** (inputs = source PDFs + pdftotext layer, expected = hand-verified T02 JSON), 7 real non-invoices excluded with reasons in the manifest. - `invoice-extract/injection/`: **6 synthetic adversarial fixtures** (LLM-addressed directives, hidden white text, IBAN-swap, arithmetic-repair lure, fake tool-call, ref hijack) — expected outcome = quarantine, one leak = blocking. - `mail-classify/`: **1824 labeled messages** (human-correctable JSONL). - `score.py`: field-level scorer, critical fields at the 98 % bar, injection suite blocking, `--self-test` built in. Per-item provenance (source id + sha256) in `manifest.json`; independently spot-checked by the orchestrator (scorer self-test PASS; expected-vs-pdftotext exact on 2 sampled docs) and corroborated by erp#43's independent re-fetch (4/4 sha256 match). Push previously gated on the operator (real financial documents) — released by the operator's explicit go this session. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01VRShc4QhLLU73FLHx9vskh
arcodange added 1 commit 2026-07-19 09:22:04 +02:00
Seed the invoice-extract (T02) and mail-classify (T01) golden sets from real
Arcodange history, plus an adversarial injection suite and an offline
field-level scorer.

invoice-extract/
- 16 real supplier PDFs (DARNIS/Hiway F1040/F1042/F1045/F1046, Anthropic
  invoice+receipt x2, Mistral, OVH, greffe d'Evry, INPI x2, Legalstart, Qonto,
  Infogreffe) fetched from the Zoho mailbox + Dolibarr GED, each with a
  hand-verified expected JSON per the T02 schema. Every expected value was
  cross-checked against the pdftotext -layout text and re-validated against the
  deterministic invariants (HT+TVA=TTC, per-rate sums, IBAN mod-97, SIREN Luhn).
- inputs/ carries both the source PDF and its {source_sha256, mime, text} pair.
- 6 SYNTHETIC injection fixtures (LLM-directive, hidden white text, IBAN-swap
  BEC lure, arithmetic-repair lure, fake tool-call, ref-hijack duplicate) whose
  only correct outcome is quarantine; each PDF is marked SYNTHETIC.
- score.py: stdlib-only field-level scorer, critical fields (amounts/IBAN/refs/
  dates) scored separately against the 98% bar, injection leaks blocking; a
  built-in --self-test proves it catches perturbed fields and leaks.
- manifest.json: per-item provenance (mail message id / GED path + sha256),
  linked Dolibarr supplier invoice, a verification note, and the list of real
  documents deliberately excluded (fee statements, payment proofs, La Poste
  receipts with no HT/TVA breakdown) with reasons.

mail-classify/
- 1824 historical mails labeled into {supplier-invoice, bank-notice,
  government-admin, client, other} via sender-domain + subject weak supervision,
  one human-correctable JSONL line per message with confidence + reason +
  message-id provenance. manifest.json records the pull method and distribution.

Docs: golden/README hub, invoice-extract/README (T02 schema + conventions),
injection/README (threat table), mail-classify/README (method + distribution).

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01VRShc4QhLLU73FLHx9vskh
arcodange merged commit 479663e3f8 into main 2026-07-19 09:22:10 +02:00
arcodange deleted branch arcodange/golden-set 2026-07-19 09:22:10 +02:00
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: arcodange-org/erp#73