feat(fleet): golden set + injection fixtures from real history (erp#39)

Seed the invoice-extract (T02) and mail-classify (T01) golden sets from real
Arcodange history, plus an adversarial injection suite and an offline
field-level scorer.

invoice-extract/
- 16 real supplier PDFs (DARNIS/Hiway F1040/F1042/F1045/F1046, Anthropic
  invoice+receipt x2, Mistral, OVH, greffe d'Evry, INPI x2, Legalstart, Qonto,
  Infogreffe) fetched from the Zoho mailbox + Dolibarr GED, each with a
  hand-verified expected JSON per the T02 schema. Every expected value was
  cross-checked against the pdftotext -layout text and re-validated against the
  deterministic invariants (HT+TVA=TTC, per-rate sums, IBAN mod-97, SIREN Luhn).
- inputs/ carries both the source PDF and its {source_sha256, mime, text} pair.
- 6 SYNTHETIC injection fixtures (LLM-directive, hidden white text, IBAN-swap
  BEC lure, arithmetic-repair lure, fake tool-call, ref-hijack duplicate) whose
  only correct outcome is quarantine; each PDF is marked SYNTHETIC.
- score.py: stdlib-only field-level scorer, critical fields (amounts/IBAN/refs/
  dates) scored separately against the 98% bar, injection leaks blocking; a
  built-in --self-test proves it catches perturbed fields and leaks.
- manifest.json: per-item provenance (mail message id / GED path + sha256),
  linked Dolibarr supplier invoice, a verification note, and the list of real
  documents deliberately excluded (fee statements, payment proofs, La Poste
  receipts with no HT/TVA breakdown) with reasons.

mail-classify/
- 1824 historical mails labeled into {supplier-invoice, bank-notice,
  government-admin, client, other} via sender-domain + subject weak supervision,
  one human-correctable JSONL line per message with confidence + reason +
  message-id provenance. manifest.json records the pull method and distribution.

Docs: golden/README hub, invoice-extract/README (T02 schema + conventions),
injection/README (threat table), mail-classify/README (method + distribution).

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01VRShc4QhLLU73FLHx9vskh
This commit is contained in:
2026-07-18 19:07:58 +02:00
co-authored by Claude Fable 5
parent fe6909a2a6
commit 6df4693880
74 changed files with 3631 additions and 1 deletions
+89
View File
@@ -0,0 +1,89 @@
# mail-classify — golden set (T01)
[fleet](../../README.md) > [golden](../README.md) > **mail-classify**
Labeled historical mail for the T01 mailbox-triage classifier
([task-inventory T01](https://gitea.arcodange.lab/arcodange-org/factory/src/branch/main/vibe/PRD/ai-back-office/task-inventory.md#t01--mailbox-triage--routing)):
every message classifies into one of five classes with a confidence and a
one-line reason.
```
{supplier-invoice, bank-notice, government-admin, client, other}
```
## File
`labels.jsonl` — one JSON object per line, one line per mailbox message. This is
the **human-correctable** form the PRD calls for: the machine wrote the seed
labels, a human corrects them in place (the approval card's "reject with reason"
is the labeling interface). Fields:
| Field | Meaning |
| --- | --- |
| `message_id` | Zoho Mail message id (provenance — the source anchor) |
| `folder` | Zoho folder the message lives in (e.g. `/Inbox/books`, `/Inbox/bureaux`) |
| `date` | received date `YYYY-MM-DD` |
| `from` | sender address |
| `subject` | subject line (HTML-unescaped) |
| `has_attachment` | bool |
| `label` | the class (one of the five) |
| `confidence` | seed-labeler confidence 01 |
| `label_reason` | one line explaining the rule that fired |
| `machine_labeled` | `true` — every line is machine-seeded and needs human review |
| `corrected_by` | `null` until a human overrides `label`, then their id/initials |
## How to correct
Edit the `label` field on a line and set `corrected_by` to your id. Keep
`machine_labeled: true` so the split between seed and human-confirmed labels stays
auditable. A corrected line is the ground truth; an uncorrected line is a
candidate. Do **not** delete lines — the label distribution (below) is itself a
property of the set.
## Labeling method (weak supervision)
Labels come from **sender-domain + subject heuristics**, most-specific first — the
T01 target is a small SLM (Pi tier), so these heuristics are the *seed*, not the
model:
1. **calendar/event noise**`other` (invitations always carry an `.ics`; caught first)
2. **supplier-invoice** — known billing domains (mistral, anthropic/stripe, hiway,
ovh, infogreffe, inpi, legalstart, medialex, digitalocean, …) **with an invoice
subject or an attachment in `/Inbox/books` or `/Notification`**. A supplier's
marketing mail with no billing signal falls through to `other`.
3. **bank-notice** — Qonto / Wise / Mercury payment + statement notifications.
4. **government-admin**`*.gouv.fr`, URSSAF, DGFiP, greffe, INPI, PortailPro, Bercy.
5. **client** — the KissMetrics domain + known client contacts (`hendrik@`,
`tdziuba@`); the `/clients/KissMetrics` folder is a known mix of client mail and
third-party ops noise (Sentry, Rollbar, DigitalOcean) routed under it, so only
client-sender mail there is labeled `client`.
6. everything else → `other`.
The seed intentionally favors **precision on the actionable classes** over recall:
better to leave a real invoice in `other` for a human to promote than to
false-positive marketing as an invoice. That bias is why the actionable classes
are small and clean; the human pass raises recall.
## Label distribution (seed, 2026-07-18)
| Label | Count | Share |
| --- | --- | --- |
| `other` | 1742 | 95.5 % |
| `government-admin` | 34 | 1.9 % |
| `client` | 30 | 1.6 % |
| `bank-notice` | 10 | 0.5 % |
| `supplier-invoice` | 8 | 0.4 % |
| **total** | **1824** | |
The heavy `other` share is real: this is a ~6-month-old personal+business mailbox
dominated by newsletters (LinkedIn ~430, Indeed, ODSC, …), event invites and
marketing — exactly what T01 leaves untouched. The actionable classes are small
because the business is young and most supplier bills arrive as card receipts, not
emailed invoices. The count far exceeds the issue's ≥200 floor.
## Provenance
Each line's `message_id` + `folder` is the source anchor back to the Zoho mailbox
(account `[email protected]`, id `7685847000000002002`). The set was
pulled read-only via the `arcodange-email-ingest` Zoho OAuth API across all 14
non-empty folders.