Files
erp/fleet/golden/mail-classify
arcodangeandClaude Fable 5 6df4693880 feat(fleet): golden set + injection fixtures from real history (erp#39)
Seed the invoice-extract (T02) and mail-classify (T01) golden sets from real
Arcodange history, plus an adversarial injection suite and an offline
field-level scorer.

invoice-extract/
- 16 real supplier PDFs (DARNIS/Hiway F1040/F1042/F1045/F1046, Anthropic
  invoice+receipt x2, Mistral, OVH, greffe d'Evry, INPI x2, Legalstart, Qonto,
  Infogreffe) fetched from the Zoho mailbox + Dolibarr GED, each with a
  hand-verified expected JSON per the T02 schema. Every expected value was
  cross-checked against the pdftotext -layout text and re-validated against the
  deterministic invariants (HT+TVA=TTC, per-rate sums, IBAN mod-97, SIREN Luhn).
- inputs/ carries both the source PDF and its {source_sha256, mime, text} pair.
- 6 SYNTHETIC injection fixtures (LLM-directive, hidden white text, IBAN-swap
  BEC lure, arithmetic-repair lure, fake tool-call, ref-hijack duplicate) whose
  only correct outcome is quarantine; each PDF is marked SYNTHETIC.
- score.py: stdlib-only field-level scorer, critical fields (amounts/IBAN/refs/
  dates) scored separately against the 98% bar, injection leaks blocking; a
  built-in --self-test proves it catches perturbed fields and leaks.
- manifest.json: per-item provenance (mail message id / GED path + sha256),
  linked Dolibarr supplier invoice, a verification note, and the list of real
  documents deliberately excluded (fee statements, payment proofs, La Poste
  receipts with no HT/TVA breakdown) with reasons.

mail-classify/
- 1824 historical mails labeled into {supplier-invoice, bank-notice,
  government-admin, client, other} via sender-domain + subject weak supervision,
  one human-correctable JSONL line per message with confidence + reason +
  message-id provenance. manifest.json records the pull method and distribution.

Docs: golden/README hub, invoice-extract/README (T02 schema + conventions),
injection/README (threat table), mail-classify/README (method + distribution).

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01VRShc4QhLLU73FLHx9vskh
2026-07-18 19:07:58 +02:00
..

mail-classify — golden set (T01)

fleet > golden > mail-classify

Labeled historical mail for the T01 mailbox-triage classifier (task-inventory T01): every message classifies into one of five classes with a confidence and a one-line reason.

{supplier-invoice, bank-notice, government-admin, client, other}

File

labels.jsonl — one JSON object per line, one line per mailbox message. This is the human-correctable form the PRD calls for: the machine wrote the seed labels, a human corrects them in place (the approval card's "reject with reason" is the labeling interface). Fields:

Field Meaning
message_id Zoho Mail message id (provenance — the source anchor)
folder Zoho folder the message lives in (e.g. /Inbox/books, /Inbox/bureaux)
date received date YYYY-MM-DD
from sender address
subject subject line (HTML-unescaped)
has_attachment bool
label the class (one of the five)
confidence seed-labeler confidence 01
label_reason one line explaining the rule that fired
machine_labeled true — every line is machine-seeded and needs human review
corrected_by null until a human overrides label, then their id/initials

How to correct

Edit the label field on a line and set corrected_by to your id. Keep machine_labeled: true so the split between seed and human-confirmed labels stays auditable. A corrected line is the ground truth; an uncorrected line is a candidate. Do not delete lines — the label distribution (below) is itself a property of the set.

Labeling method (weak supervision)

Labels come from sender-domain + subject heuristics, most-specific first — the T01 target is a small SLM (Pi tier), so these heuristics are the seed, not the model:

  1. calendar/event noiseother (invitations always carry an .ics; caught first)
  2. supplier-invoice — known billing domains (mistral, anthropic/stripe, hiway, ovh, infogreffe, inpi, legalstart, medialex, digitalocean, …) with an invoice subject or an attachment in /Inbox/books or /Notification. A supplier's marketing mail with no billing signal falls through to other.
  3. bank-notice — Qonto / Wise / Mercury payment + statement notifications.
  4. government-admin*.gouv.fr, URSSAF, DGFiP, greffe, INPI, PortailPro, Bercy.
  5. client — the KissMetrics domain + known client contacts (hendrik@, tdziuba@); the /clients/KissMetrics folder is a known mix of client mail and third-party ops noise (Sentry, Rollbar, DigitalOcean) routed under it, so only client-sender mail there is labeled client.
  6. everything else → other.

The seed intentionally favors precision on the actionable classes over recall: better to leave a real invoice in other for a human to promote than to false-positive marketing as an invoice. That bias is why the actionable classes are small and clean; the human pass raises recall.

Label distribution (seed, 2026-07-18)

Label Count Share
other 1742 95.5 %
government-admin 34 1.9 %
client 30 1.6 %
bank-notice 10 0.5 %
supplier-invoice 8 0.4 %
total 1824

The heavy other share is real: this is a ~6-month-old personal+business mailbox dominated by newsletters (LinkedIn ~430, Indeed, ODSC, …), event invites and marketing — exactly what T01 leaves untouched. The actionable classes are small because the business is young and most supplier bills arrive as card receipts, not emailed invoices. The count far exceeds the issue's ≥200 floor.

Provenance

Each line's message_id + folder is the source anchor back to the Zoho mailbox (account [email protected], id 7685847000000002002). The set was pulled read-only via the arcodange-email-ingest Zoho OAuth API across all 14 non-empty folders.