Seed the invoice-extract (T02) and mail-classify (T01) golden sets from real
Arcodange history, plus an adversarial injection suite and an offline
field-level scorer.
invoice-extract/
- 16 real supplier PDFs (DARNIS/Hiway F1040/F1042/F1045/F1046, Anthropic
invoice+receipt x2, Mistral, OVH, greffe d'Evry, INPI x2, Legalstart, Qonto,
Infogreffe) fetched from the Zoho mailbox + Dolibarr GED, each with a
hand-verified expected JSON per the T02 schema. Every expected value was
cross-checked against the pdftotext -layout text and re-validated against the
deterministic invariants (HT+TVA=TTC, per-rate sums, IBAN mod-97, SIREN Luhn).
- inputs/ carries both the source PDF and its {source_sha256, mime, text} pair.
- 6 SYNTHETIC injection fixtures (LLM-directive, hidden white text, IBAN-swap
BEC lure, arithmetic-repair lure, fake tool-call, ref-hijack duplicate) whose
only correct outcome is quarantine; each PDF is marked SYNTHETIC.
- score.py: stdlib-only field-level scorer, critical fields (amounts/IBAN/refs/
dates) scored separately against the 98% bar, injection leaks blocking; a
built-in --self-test proves it catches perturbed fields and leaks.
- manifest.json: per-item provenance (mail message id / GED path + sha256),
linked Dolibarr supplier invoice, a verification note, and the list of real
documents deliberately excluded (fee statements, payment proofs, La Poste
receipts with no HT/TVA breakdown) with reasons.
mail-classify/
- 1824 historical mails labeled into {supplier-invoice, bank-notice,
government-admin, client, other} via sender-domain + subject weak supervision,
one human-correctable JSONL line per message with confidence + reason +
message-id provenance. manifest.json records the pull method and distribution.
Docs: golden/README hub, invoice-extract/README (T02 schema + conventions),
injection/README (threat table), mail-classify/README (method + distribution).
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01VRShc4QhLLU73FLHx9vskh
mail-classify — golden set (T01)
fleet > golden > mail-classify
Labeled historical mail for the T01 mailbox-triage classifier (task-inventory T01): every message classifies into one of five classes with a confidence and a one-line reason.
{supplier-invoice, bank-notice, government-admin, client, other}
File
labels.jsonl — one JSON object per line, one line per mailbox message. This is
the human-correctable form the PRD calls for: the machine wrote the seed
labels, a human corrects them in place (the approval card's "reject with reason"
is the labeling interface). Fields:
| Field | Meaning |
|---|---|
message_id |
Zoho Mail message id (provenance — the source anchor) |
folder |
Zoho folder the message lives in (e.g. /Inbox/books, /Inbox/bureaux) |
date |
received date YYYY-MM-DD |
from |
sender address |
subject |
subject line (HTML-unescaped) |
has_attachment |
bool |
label |
the class (one of the five) |
confidence |
seed-labeler confidence 0–1 |
label_reason |
one line explaining the rule that fired |
machine_labeled |
true — every line is machine-seeded and needs human review |
corrected_by |
null until a human overrides label, then their id/initials |
How to correct
Edit the label field on a line and set corrected_by to your id. Keep
machine_labeled: true so the split between seed and human-confirmed labels stays
auditable. A corrected line is the ground truth; an uncorrected line is a
candidate. Do not delete lines — the label distribution (below) is itself a
property of the set.
Labeling method (weak supervision)
Labels come from sender-domain + subject heuristics, most-specific first — the T01 target is a small SLM (Pi tier), so these heuristics are the seed, not the model:
- calendar/event noise →
other(invitations always carry an.ics; caught first) - supplier-invoice — known billing domains (mistral, anthropic/stripe, hiway,
ovh, infogreffe, inpi, legalstart, medialex, digitalocean, …) with an invoice
subject or an attachment in
/Inbox/booksor/Notification. A supplier's marketing mail with no billing signal falls through toother. - bank-notice — Qonto / Wise / Mercury payment + statement notifications.
- government-admin —
*.gouv.fr, URSSAF, DGFiP, greffe, INPI, PortailPro, Bercy. - client — the KissMetrics domain + known client contacts (
hendrik@,tdziuba@); the/clients/KissMetricsfolder is a known mix of client mail and third-party ops noise (Sentry, Rollbar, DigitalOcean) routed under it, so only client-sender mail there is labeledclient. - everything else →
other.
The seed intentionally favors precision on the actionable classes over recall:
better to leave a real invoice in other for a human to promote than to
false-positive marketing as an invoice. That bias is why the actionable classes
are small and clean; the human pass raises recall.
Label distribution (seed, 2026-07-18)
| Label | Count | Share |
|---|---|---|
other |
1742 | 95.5 % |
government-admin |
34 | 1.9 % |
client |
30 | 1.6 % |
bank-notice |
10 | 0.5 % |
supplier-invoice |
8 | 0.4 % |
| total | 1824 |
The heavy other share is real: this is a ~6-month-old personal+business mailbox
dominated by newsletters (LinkedIn ~430, Indeed, ODSC, …), event invites and
marketing — exactly what T01 leaves untouched. The actionable classes are small
because the business is young and most supplier bills arrive as card receipts, not
emailed invoices. The count far exceeds the issue's ≥200 floor.
Provenance
Each line's message_id + folder is the source anchor back to the Zoho mailbox
(account [email protected], id 7685847000000002002). The set was
pulled read-only via the arcodange-email-ingest Zoho OAuth API across all 14
non-empty folders.