feat(fleet): golden set + injection fixtures from real history (erp#39)
Seed the invoice-extract (T02) and mail-classify (T01) golden sets from real
Arcodange history, plus an adversarial injection suite and an offline
field-level scorer.
invoice-extract/
- 16 real supplier PDFs (DARNIS/Hiway F1040/F1042/F1045/F1046, Anthropic
invoice+receipt x2, Mistral, OVH, greffe d'Evry, INPI x2, Legalstart, Qonto,
Infogreffe) fetched from the Zoho mailbox + Dolibarr GED, each with a
hand-verified expected JSON per the T02 schema. Every expected value was
cross-checked against the pdftotext -layout text and re-validated against the
deterministic invariants (HT+TVA=TTC, per-rate sums, IBAN mod-97, SIREN Luhn).
- inputs/ carries both the source PDF and its {source_sha256, mime, text} pair.
- 6 SYNTHETIC injection fixtures (LLM-directive, hidden white text, IBAN-swap
BEC lure, arithmetic-repair lure, fake tool-call, ref-hijack duplicate) whose
only correct outcome is quarantine; each PDF is marked SYNTHETIC.
- score.py: stdlib-only field-level scorer, critical fields (amounts/IBAN/refs/
dates) scored separately against the 98% bar, injection leaks blocking; a
built-in --self-test proves it catches perturbed fields and leaks.
- manifest.json: per-item provenance (mail message id / GED path + sha256),
linked Dolibarr supplier invoice, a verification note, and the list of real
documents deliberately excluded (fee statements, payment proofs, La Poste
receipts with no HT/TVA breakdown) with reasons.
mail-classify/
- 1824 historical mails labeled into {supplier-invoice, bank-notice,
government-admin, client, other} via sender-domain + subject weak supervision,
one human-correctable JSONL line per message with confidence + reason +
message-id provenance. manifest.json records the pull method and distribution.
Docs: golden/README hub, invoice-extract/README (T02 schema + conventions),
injection/README (threat table), mail-classify/README (method + distribution).
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01VRShc4QhLLU73FLHx9vskh
This commit is contained in:
@@ -0,0 +1,89 @@
|
||||
# mail-classify — golden set (T01)
|
||||
|
||||
[fleet](../../README.md) > [golden](../README.md) > **mail-classify**
|
||||
|
||||
Labeled historical mail for the T01 mailbox-triage classifier
|
||||
([task-inventory T01](https://gitea.arcodange.lab/arcodange-org/factory/src/branch/main/vibe/PRD/ai-back-office/task-inventory.md#t01--mailbox-triage--routing)):
|
||||
every message classifies into one of five classes with a confidence and a
|
||||
one-line reason.
|
||||
|
||||
```
|
||||
{supplier-invoice, bank-notice, government-admin, client, other}
|
||||
```
|
||||
|
||||
## File
|
||||
|
||||
`labels.jsonl` — one JSON object per line, one line per mailbox message. This is
|
||||
the **human-correctable** form the PRD calls for: the machine wrote the seed
|
||||
labels, a human corrects them in place (the approval card's "reject with reason"
|
||||
is the labeling interface). Fields:
|
||||
|
||||
| Field | Meaning |
|
||||
| --- | --- |
|
||||
| `message_id` | Zoho Mail message id (provenance — the source anchor) |
|
||||
| `folder` | Zoho folder the message lives in (e.g. `/Inbox/books`, `/Inbox/bureaux`) |
|
||||
| `date` | received date `YYYY-MM-DD` |
|
||||
| `from` | sender address |
|
||||
| `subject` | subject line (HTML-unescaped) |
|
||||
| `has_attachment` | bool |
|
||||
| `label` | the class (one of the five) |
|
||||
| `confidence` | seed-labeler confidence 0–1 |
|
||||
| `label_reason` | one line explaining the rule that fired |
|
||||
| `machine_labeled` | `true` — every line is machine-seeded and needs human review |
|
||||
| `corrected_by` | `null` until a human overrides `label`, then their id/initials |
|
||||
|
||||
## How to correct
|
||||
|
||||
Edit the `label` field on a line and set `corrected_by` to your id. Keep
|
||||
`machine_labeled: true` so the split between seed and human-confirmed labels stays
|
||||
auditable. A corrected line is the ground truth; an uncorrected line is a
|
||||
candidate. Do **not** delete lines — the label distribution (below) is itself a
|
||||
property of the set.
|
||||
|
||||
## Labeling method (weak supervision)
|
||||
|
||||
Labels come from **sender-domain + subject heuristics**, most-specific first — the
|
||||
T01 target is a small SLM (Pi tier), so these heuristics are the *seed*, not the
|
||||
model:
|
||||
|
||||
1. **calendar/event noise** → `other` (invitations always carry an `.ics`; caught first)
|
||||
2. **supplier-invoice** — known billing domains (mistral, anthropic/stripe, hiway,
|
||||
ovh, infogreffe, inpi, legalstart, medialex, digitalocean, …) **with an invoice
|
||||
subject or an attachment in `/Inbox/books` or `/Notification`**. A supplier's
|
||||
marketing mail with no billing signal falls through to `other`.
|
||||
3. **bank-notice** — Qonto / Wise / Mercury payment + statement notifications.
|
||||
4. **government-admin** — `*.gouv.fr`, URSSAF, DGFiP, greffe, INPI, PortailPro, Bercy.
|
||||
5. **client** — the KissMetrics domain + known client contacts (`hendrik@`,
|
||||
`tdziuba@`); the `/clients/KissMetrics` folder is a known mix of client mail and
|
||||
third-party ops noise (Sentry, Rollbar, DigitalOcean) routed under it, so only
|
||||
client-sender mail there is labeled `client`.
|
||||
6. everything else → `other`.
|
||||
|
||||
The seed intentionally favors **precision on the actionable classes** over recall:
|
||||
better to leave a real invoice in `other` for a human to promote than to
|
||||
false-positive marketing as an invoice. That bias is why the actionable classes
|
||||
are small and clean; the human pass raises recall.
|
||||
|
||||
## Label distribution (seed, 2026-07-18)
|
||||
|
||||
| Label | Count | Share |
|
||||
| --- | --- | --- |
|
||||
| `other` | 1742 | 95.5 % |
|
||||
| `government-admin` | 34 | 1.9 % |
|
||||
| `client` | 30 | 1.6 % |
|
||||
| `bank-notice` | 10 | 0.5 % |
|
||||
| `supplier-invoice` | 8 | 0.4 % |
|
||||
| **total** | **1824** | |
|
||||
|
||||
The heavy `other` share is real: this is a ~6-month-old personal+business mailbox
|
||||
dominated by newsletters (LinkedIn ~430, Indeed, ODSC, …), event invites and
|
||||
marketing — exactly what T01 leaves untouched. The actionable classes are small
|
||||
because the business is young and most supplier bills arrive as card receipts, not
|
||||
emailed invoices. The count far exceeds the issue's ≥200 floor.
|
||||
|
||||
## Provenance
|
||||
|
||||
Each line's `message_id` + `folder` is the source anchor back to the Zoho mailbox
|
||||
(account `[email protected]`, id `7685847000000002002`). The set was
|
||||
pulled read-only via the `arcodange-email-ingest` Zoho OAuth API across all 14
|
||||
non-empty folders.
|
||||
Reference in New Issue
Block a user