fix(email-ingest): extraction testable et pinnée au golden set + adc-008

L'extraction de champs vivait dans un heredoc à l'intérieur d'email-inspect.sh :
impossible à exécuter isolément, donc jamais mesurée, donc fausse sans que
personne puisse le voir. Sur la facture Darnis F1048 elle renvoyait le numéro de
TVA d'Arcodange comme référence de facture, et aucune date.

- extract_fields.py : l'extraction sort du shell et devient un module.
- test_extract.py : régression contre les 16 factures hand-vérifiées de
  fleet/golden/invoice-extract/. Score par champ, et une valeur FAUSSE pèse plus
  qu'une valeur absente — un humain recopie ce qui s'affiche.

Valeurs fausses : 4 → 0. Exactitude ref 62,5 → 75 %, date 62,5 → 75 %,
HT 68,8 → 75 %, TTC 81,2 → 93,8 %.

Cinq bugs réels, dont trois invisibles sans test :
- « Nº » sur les factures françaises est U+00BA (ordinal masculin), pas le signe
  degré. La classe [°o] le rate, le motif principal échoue, et le repli attrape
  le premier jeton ref-shaped du document — très souvent un numéro de TVA.
- Le filtre anti-TVA rejetait « FR73261832 », qui est la vraie référence OVH : un
  numéro FR fait exactement 11 caractères après le préfixe.
- « Montant total (HT) » était lu comme un TTC.
- Une référence coupée par la colonne (« 06-01-26- » / « payment-366753 ») était
  renvoyée amputée : le recollage doit précéder le scan, sinon la queue seule est
  trouvée en premier.
- Un `\b` après `€` ne peut jamais matcher en fin de ligne (€ n'est pas un
  caractère de mot) — la TVA n'était jamais extraite.

adc-008 : une facture fournisseur s'enregistre à SA date, même future, tant que
l'exercice (année civile) ne bascule pas. Le document fait foi ; altérer sa date
ferait diverger l'écriture de sa pièce justificative (CGI art. 289 VII).
Registre validé : 8 règles, 8 ADC, 0 erreur.

scopes.ts : 1232 (factures fournisseur) ajouté à prod-write — oubli initial,
révélé par un 403 en production sur F1048. Le pipeline s'est arrêté sans écrire.

Appliqué en production via le pipeline gated : FAF2026014 (Darnis F1048),
218,50 HT + 43,70 TVA = 262,20 TTC, validée, non réglée.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01VRShc4QhLLU73FLHx9vskh
This commit is contained in:
2026-08-13 10:47:30 +02:00
co-authored by Claude Opus 5
parent 2e699e1fc6
commit 4d1e3ecb23
6 changed files with 451 additions and 42 deletions
@@ -115,7 +115,7 @@ if [[ -n "${SAVE_PDF_DIR}" ]]; then
fi
# 5. Heuristic extract + render
python3 - "${WORK}" "${FMT}" <<'PY'
EXTRACT_DIR="${SCRIPT_DIR}" python3 - "${WORK}" "${FMT}" <<'PY'
import json, sys, os, re, datetime, glob
work, fmt = sys.argv[1:3]
@@ -158,42 +158,18 @@ def extract(text):
if v is not None: return f"{v:.2f}"
return None
out["total_ht"] = first_amount(r'(?:total\s*ht|montant\s*ht|net\s*amount|subtotal)[^\d-]*([\d \.,]+)')
# TVA: require currency suffix to avoid matching VAT-number digits
out["total_tva"] = first_amount(r'(?:tva|vat)[^\d-]*([\d \.,]+)\s*(?:€|eur)\b')
out["total_ttc"] = first_amount(r'(?:total\s*ttc|amount\s*due|total\s*due|grand\s*total|montant\s*total|amount\s*paid)[^\d-]*([\d \.,]+)')
# Invoice ref — must contain a digit (filters "umber", "Invoice", etc.)
m = re.search(r'(?:facture|invoice|receipt|reçu)\s*(?:n[°o]?|number|#|:)\s*([A-Za-z0-9][\w\d/-]{2,})', text, re.IGNORECASE)
if m and any(c.isdigit() for c in m.group(1)):
out["invoice_ref"] = m.group(1)
else:
# Fallback: any reasonable ref-shaped token after "Invoice" / "Facture" header
m = re.search(r'\b([A-Z]{2,}[-/]?\d[\w\d/-]{2,})\b', text)
out["invoice_ref"] = m.group(1) if m else None
# Invoice date — try ISO, French DD/MM/YYYY, English MM/DD/YYYY, French long form
out["invoice_date_raw"] = None
for p in (
r'\b(\d{4}-\d{2}-\d{2})\b',
r'(?:date|émise\s*le|invoice\s*date|date\s*de\s*facturation)[:\s]*(\d{1,2}[\s/.-]\d{1,2}[\s/.-]\d{2,4})',
r'(?:date|émise\s*le|invoice\s*date)[:\s]*(\d{1,2}\s+\w{3,9}\.?\s+\d{4})',
):
m = re.search(p, text, re.IGNORECASE)
if m: out["invoice_date_raw"] = m.group(1).strip(); break
# VAT rate (e.g. "20%") — restrict to 0-25% so "100%" / page footers don't match.
vrate = None
for line in lines:
m = re.search(r'\b(\d{1,2}([.,]\d+)?)\s*%', line)
if m:
v = float(m.group(1).replace(",", "."))
if 0 <= v <= 25:
vrate = m.group(1).replace(",", "."); break
out["vat_rate_pct"] = vrate
# Extraction déléguée au module testé (extract_fields.py), pinné par
# test_extract.py contre les 16 factures du golden set erp#39. L'ancienne
# version inline n'était pas exécutable isolément, donc jamais mesurée :
# elle renvoyait le n° de TVA d'Arcodange comme référence de facture.
# `python3 -` reads from stdin, so __file__ does not exist here: the shell
# passes the module's directory in EXTRACT_DIR.
sys.path.insert(0, os.environ["EXTRACT_DIR"])
from extract_fields import extract as _extract
out.update(_extract(text))
return out
pdfs = []
for pdf in sorted(glob.glob(os.path.join(work,"atts","*.pdf")) +
glob.glob(os.path.join(work,"atts","*.PDF"))):
@@ -0,0 +1,224 @@
#!/usr/bin/env python3
"""Field extraction from a supplier-invoice PDF text layer.
Lifted out of email-inspect.sh so it can be REGRESSION-TESTED against the 16
hand-verified invoices in fleet/golden/invoice-extract/ (see test_extract.py).
It was a heredoc, therefore untestable, therefore wrong in ways nobody could
measure: on Darnis F1048 it returned Arcodange's own VAT number as the invoice
reference and no date at all.
Two structural traps, both observed on real documents:
1. `Nº` in French invoices is U+00BA (MASCULINE ORDINAL INDICATOR), not the
degree sign `°`. A character class of [°o] silently misses it, the primary
pattern fails, and the fallback grabs the first ref-shaped token in the
document — which is very often a VAT number.
2. Two-column layouts put the label and its value on DIFFERENT lines, with
unrelated text in between. A pattern requiring `label:\\s*value` never
matches, so the field comes back empty.
Principle kept from the atom contract: emit nothing rather than emit a guess.
A value that fails its own sanity check is dropped, not repaired.
"""
from __future__ import annotations
import re
# Ordinal marks seen on real French invoices: degree sign, masculine ordinal, 'o'
ORD = "[°ºo]?"
# VAT numbers are ref-shaped and appear near the top of most invoices — the
# single biggest source of false "invoice_ref". Two country prefixes + 8-13
# digits/letters, no separators.
VAT_LIKE = re.compile(r"^(?:FR[0-9A-Z]{11}|BE0?\d{9,10}|DE\d{9}|ES[0-9A-Z]\d{7}[0-9A-Z]|IT\d{11}"
r"|NL\d{9}B\d{2}|LU\d{8}|IE\d{7}[A-W]{1,2}|PT\d{9}|AT U\d{8}|PL\d{10})$", re.I)
SIRET_LIKE = re.compile(r"^\d{9,14}$")
def parse_amount(s: str) -> float | None:
"""European or Anglo number → float. None when it is not a number."""
if not s:
return None
t = re.sub(r"[\s ]", "", s).strip().rstrip(".,")
if not t:
return None
if "," in t and "." in t: # 1.234,56 or 1,234.56
t = t.replace(".", "").replace(",", ".") if t.rfind(",") > t.rfind(".") \
else t.replace(",", "")
elif "," in t:
t = t.replace(",", ".")
try:
return float(t)
except ValueError:
return None
def extract_ref(text: str) -> str | None:
"""The supplier's own invoice number."""
# 1. Explicitly labelled, on the same line — the reliable case.
for pat in (
rf"(?:facture|invoice|receipt|re[çc]u)\s*(?:n{ORD}|number|#|:)\s*[:\s]*([A-Za-z0-9][\w\d/-]{{2,}})",
r"(?:num[ée]ro\s+de\s+facture|invoice\s+number)\s*:?\s*([A-Za-z0-9][\w\d/-]{2,})",
):
for m in re.finditer(pat, text, re.IGNORECASE):
cand = m.group(1).strip(" .:،,")
if _plausible_ref(cand):
return cand
# 2. Two-column layout: the label sits alone, the value lands further down
# the page. Look ahead a bounded window and take the first plausible
# token — bounded so we do not wander into another block.
m = re.search(r"(?:num[ée]ro\s+de\s+facture|invoice\s+number|facture\s+n" + ORD + r")\s*:?", text, re.IGNORECASE)
if m:
window = text[m.end():m.end() + 400]
# Rejoin FIRST: if the window opens on a dangling fragment, the tail alone
# is also plausible, and scanning first would return a ref amputated of
# its head — a subtly wrong value, the worst kind.
# A ref split by the column wrap: "06-01-26-" ends a line, its tail
# ("payment-366753") lands further down in the other column. Rejoin the
# dangling fragment with the next token that carries a digit.
frag = re.match(r"\s*([A-Za-z0-9][\w\d/-]*[-/])(?=\s)", window)
if frag:
for tail in re.findall(r"\b([A-Za-z0-9][\w\d/-]{2,})\b", window[frag.end():frag.end() + 200]):
if any(c.isdigit() for c in tail):
joined = frag.group(1) + tail
if _plausible_ref(joined):
return joined
for cand in re.findall(r"\b([A-Za-z0-9][\w\d/-]{2,})\b", window):
if _plausible_ref(cand):
return cand
# 3. No labelled ref found. Emit nothing rather than the first ref-shaped
# token in the document — that is how a VAT number ends up as an invoice
# reference, which is exactly the bug this module exists to kill.
return None
def _plausible_ref(cand: str) -> bool:
if not cand or not any(c.isdigit() for c in cand):
return False
if VAT_LIKE.match(cand) or SIRET_LIKE.match(cand):
return False
if re.fullmatch(r"\d{1,2}[/.-]\d{1,2}[/.-]\d{2,4}", cand): # a date
return False
if len(cand) > 40:
return False
if cand.endswith(("-", "/", ".")): # tronquée au saut de colonne
return False
return True
MONTHS = {}
for i, names in enumerate([
("janvier", "janv", "january", "jan"), ("février", "fevrier", "févr", "fevr", "february", "feb"),
("mars", "march", "mar"), ("avril", "avr", "april", "apr"), ("mai", "may"),
("juin", "june", "jun"), ("juillet", "juil", "july", "jul"), ("août", "aout", "august", "aug"),
("septembre", "sept", "september", "sep"), ("octobre", "oct", "october"),
("novembre", "nov", "november"), ("décembre", "decembre", "déc", "dec", "december"),
], start=1):
for n in names:
MONTHS[n] = i
def to_iso(raw: str | None) -> str | None:
"""dd/mm/yyyy or '2 avr. 2026' → yyyy-mm-dd. Consumers want one shape, not five."""
if not raw:
return None
raw = raw.strip()
if re.fullmatch(r"\d{4}-\d{2}-\d{2}", raw):
return raw
m = re.fullmatch(r"(\d{1,2})[/.\-](\d{1,2})[/.\-](\d{2,4})", raw)
if m:
d, mo, y = m.groups()
y = "20" + y if len(y) == 2 else y
return f"{y}-{int(mo):02d}-{int(d):02d}"
m = re.fullmatch(r"(\d{1,2})\s+([A-Za-zéûîàôçÉ.]+)\.?\s+(\d{4})", raw)
if m:
d, mon, y = m.groups()
idx = MONTHS.get(mon.lower().rstrip("."))
if idx:
return f"{y}-{idx:02d}-{int(d):02d}"
return raw
DATE_LABELS = (r"date\s*d[e']\s*[ée]mission", r"date\s*de\s*facturation", r"invoice\s*date",
r"date\s*d[e']\s*facture", r"[ée]mise?\s*le", r"\bdate\b")
def extract_date(text: str) -> str | None:
"""Issue date. Prefers a labelled date; falls back to the earliest date seen."""
# ISO anywhere wins — unambiguous.
m = re.search(r"\b(\d{4}-\d{2}-\d{2})\b", text)
if m:
return m.group(1)
# Labelled, allowing the two-column gap: up to 300 chars of anything between
# the label and its value (the value is often a column away and a line down).
for lab in DATE_LABELS:
for m in re.finditer(lab, text, re.IGNORECASE):
window = text[m.end():m.end() + 300]
d = re.search(r"\b(\d{1,2}[/.\-]\d{1,2}[/.\-]\d{2,4})\b", window)
if d:
return d.group(1)
d = re.search(r"\b(\d{1,2}\s+[A-Za-zéûîàôç.]{3,10}\.?\s+\d{4})\b", window)
if d:
return d.group(1)
# Unlabelled: take the first date-shaped token, but only if the document has
# one — no invention.
m = re.search(r"\b(\d{1,2}[/.\-]\d{1,2}[/.\-]\d{4})\b", text)
return m.group(1) if m else None
def extract_amounts(text: str) -> dict:
"""HT / TVA / TTC. Returns only what the document actually labels."""
lines = text.splitlines()
def first(pattern: str) -> str | None:
for line in lines:
m = re.search(pattern, line, re.IGNORECASE)
if m:
v = parse_amount(m.group(1))
if v is not None:
return f"{v:.2f}"
return None
AMT = r"[^\d-]*([\d\u00a0\u202f .,]+)"
# "Montant total (HT)" is an HT, not a TTC — the parenthetical decides, so it
# is matched on the HT side and excluded on the TTC side.
ht = first(r"(?:total\s*ht|montant\s*ht|montant\s*total\s*\(\s*ht\s*\)|net\s*amount|subtotal|sous.?total\s*\(\s*ht\s*\))" + AMT)
# The rate often sits between the label and the amount — "TVA (20%) : 43,70 €".
# A currency mark is still required so a VAT *number* is never read as a VAT *amount*.
tva = first(r"(?:tva|vat)\s*(?:\(\s*\d{1,2}(?:[.,]\d+)?\s*%\s*\))?\s*:?" + AMT + r"\s*(?:€|\beur\b)")
ttc = first(r"(?:total\s*ttc|amount\s*due|total\s*due|grand\s*total|montant\s*total|amount\s*paid)(?!\s*\(?\s*ht)" + AMT)
if ttc is None:
# Some invoices label the TTC simply "Total" (Legalstart, INPI). Accept it
# only as a fallback, and only on a line that is nothing else.
ttc = first(r"^\s*total\s+" + AMT.lstrip("[^\\d-]*") if False else r"^\s*total\s+([\d\u00a0\u202f .,]+)\s*€")
out = {"total_ht": ht, "total_tva": tva, "total_ttc": ttc}
# Arithmetic sanity: if HT + TVA disagrees with TTC we matched the wrong
# lines somewhere. Report it rather than pretend the numbers are coherent.
h, t_, c = (parse_amount(out[k] or "") for k in ("total_ht", "total_tva", "total_ttc"))
out["arithmetic_ok"] = None if None in (h, t_, c) else abs(h + t_ - c) <= 0.01
return out
def extract_vat_rate(text: str) -> str | None:
for line in text.splitlines():
m = re.search(r"\b(\d{1,2}(?:[.,]\d+)?)\s*%", line)
if m:
v = float(m.group(1).replace(",", "."))
if 0 <= v <= 25:
return m.group(1)
return None
def extract(text: str) -> dict:
out = {
"invoice_ref": extract_ref(text),
"invoice_date_raw": to_iso(extract_date(text)),
"vat_rate_pct": extract_vat_rate(text),
}
out.update(extract_amounts(text))
return out
@@ -0,0 +1,101 @@
#!/usr/bin/env python3
"""Regression test for extract_fields.py against the erp#39 golden set.
The email skill's extraction was a heredoc inside a shell script: impossible to
run in isolation, therefore never measured, therefore silently wrong. This pins
it to the 16 hand-verified supplier invoices in fleet/golden/invoice-extract/.
Scored per field, and a WRONG value counts against us far more than a missing
one: emitting Arcodange's VAT number as an invoice reference is worse than
emitting nothing, because a human copies it without looking.
python3 test_extract.py # summary
python3 test_extract.py --verbose # per-document diff
"""
from __future__ import annotations
import json
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from extract_fields import extract, parse_amount
REPO = os.path.realpath(os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "..", "..", ".."))
GOLDEN = os.path.join(REPO, "fleet", "golden", "invoice-extract")
VERBOSE = "--verbose" in sys.argv
def norm_date(s):
"""dd/mm/yyyy or yyyy-mm-dd → yyyy-mm-dd; anything else stays as-is."""
if not s:
return None
s = s.strip()
if len(s) == 10 and s[4] == "-":
return s
for sep in ("/", ".", "-"):
p = s.split(sep)
if len(p) == 3 and all(x.strip().isdigit() for x in p):
d, m, y = (x.strip() for x in p)
if len(y) == 2:
y = "20" + y
if len(y) == 4:
return f"{y}-{int(m):02d}-{int(d):02d}"
return s
def main() -> int:
inputs = os.path.join(GOLDEN, "inputs")
if not os.path.isdir(inputs):
print(f"golden set not found at {GOLDEN} — skipping", file=sys.stderr)
return 0
stats = {f: {"ok": 0, "wrong": 0, "missing": 0} for f in ("invoice_ref", "invoice_date_raw", "total_ht", "total_ttc")}
docs = 0
for name in sorted(os.listdir(inputs)):
if not name.endswith(".json"):
continue
doc = json.load(open(os.path.join(inputs, name)))
exp_path = os.path.join(GOLDEN, "expected", name)
if not os.path.exists(exp_path):
continue
exp = json.load(open(exp_path))
got = extract(doc["text"])
docs += 1
checks = {
"invoice_ref": (exp.get("ref_supplier"), got.get("invoice_ref")),
"invoice_date_raw": (exp.get("date_issue"), norm_date(got.get("invoice_date_raw"))),
"total_ht": (exp.get("totals", {}).get("ht"), parse_amount(got.get("total_ht") or "")),
"total_ttc": (exp.get("totals", {}).get("ttc"), parse_amount(got.get("total_ttc") or "")),
}
for field, (want, have) in checks.items():
if have is None:
stats[field]["missing"] += 1
verdict = "MANQUE"
elif isinstance(want, (int, float)) and isinstance(have, (int, float)):
ok = abs(float(want) - float(have)) <= 0.01
stats[field]["ok" if ok else "wrong"] += 1
verdict = "ok" if ok else "FAUX"
else:
ok = str(want) == str(have)
stats[field]["ok" if ok else "wrong"] += 1
verdict = "ok" if ok else "FAUX"
if VERBOSE and verdict != "ok":
print(f" {name[:34]:<36}{field:<18}{verdict:<8}attendu={want!r} obtenu={have!r}")
print(f"\n{docs} documents du golden set\n")
print(f"{'champ':<20}{'exact':>7}{'FAUX':>7}{'absent':>9} {'exactitude'}")
print("-" * 62)
total_wrong = 0
for f, s in stats.items():
total = s["ok"] + s["wrong"] + s["missing"]
total_wrong += s["wrong"]
print(f"{f:<20}{s['ok']:>7}{s['wrong']:>7}{s['missing']:>9} {100*s['ok']/total:.1f} %")
print("-" * 62)
print(f"\nvaleurs FAUSSES (le pire cas — un humain les recopie) : {total_wrong}")
return 1 if total_wrong else 0
if __name__ == "__main__":
sys.exit(main())