Learning #4 of the 2026-07-11 rehearsal: manifest B failed mid-run and could
not be re-applied — op 1 (the DARNIS invoice) had already run and a replay
would have duplicated it. Every write op now dedupes BEFORE any POST:
- thirdparty-create.sh: by exact name (promote '#thirdparty:name=' semantics);
ambiguous (2+) aborts; an existing fiche missing the requested role aborts
(refuse-never-repair). Emits {"id", "deduped"} instead of a bare id.
- invoice-create.sh: supplier kind by (socid, ref_supplier) — same key with a
different total aborts as a conflict; customer kind (or supplier without
ref_supplier) by (socid, date, total_ttc ±0.02, line fingerprint) with descs
HTML-unescaped. Credit notes are never candidates. A deduped DRAFT with
validate:true is validated on replay, so an interrupted run converges.
- payment-record.sh: by (invoice, amount, normalized transaction_id), composing
with the erp#37 varchar(50) normalization on BOTH sides so historical
long-form nums still match; same tx + different amount aborts; without a tx
id there is no dedupe key (warned). Dedupe answers id:null (the payments list
exposes no paiement rowid) + the existing bank line.
- All three refuse to POST blind when the dedupe lookup fails with anything but
the documented empty-list 404 (the voir_tous trap would otherwise mint dupes).
- promote-apply.sh: marks each op created / deduped=true inline and totals them
in the summary — an all-deduped second run is visible proof of a no-op.
- promote-plan.sh: advertises each op's dedupe key (and flags tx=MISSING as
'a replay WILL double-pay').
Proof:
- tests/run-tests.sh: 5 new offline cases (11 total) — dedupe hits POST
nothing, conflicts/ambiguity abort pre-POST, long-form history dedupes,
draft convergence validates; stub extended to serve the new lookups with the
live-observed empty behaviors ([] for invoices/payments, 404 for tiers).
- tests/replay-idempotency.sh (new, live): double-applies a self-contained
manifest on the sandbox — run 1 '3 created' (rows 1/1/1), run 2 '3 deduped'
with row counts unchanged and the stored num in erp#37 short form.
- The historic manifest-B now replays on the sandbox as 5/5 deduped, zero new
rows — the exact replay Learning #4 declared impossible.
SKILL.md updated in the same change (per-op dedupe keys, replay-safety section,
gotchas); the 2026-07-11 runbook's Learning #4 carries a dated resolution
addendum.
Closes erp#44.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01VRShc4QhLLU73FLHx9vskh
- runs/2026-07-18/: the 8 sha256-pinned verifier transcripts (4 runtimes ×
2 tests), blind-judging verdicts (2 independent judges per cell, unanimous),
the erp#56 builder-bench journal + prompt + caps, and the evidence README
with the parity table.
- run-verifier.sh: mistral runtime drops the tool-filter flag (--enabled-tools
with a no-match pattern hangs vibe 2.21.0); plain -p with --max-turns 1.
Verdicts: Mistral (vibe -p, mistral-medium-3.5) and Ornith 35B (hermes MLX)
reach verdict parity with the Claude baseline on both tests → admitted to
verifier duty. Qwen2.5-7B-4bit fails both → the honest small-model floor.
Builder bench: erp#56 completed by the Mistral runtime, 0 code corrections,
261 s, acceptance run clean (0 bank-UNKNOWN) → merged as PR #68.
Closes#63 (with the paired factory qa-strategy PR).
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01VRShc4QhLLU73FLHx9vskh
- MISTRAL.AI: update note to reflect annual subscription (Le Chat Pro - Annual)
from invoice MSTRL-API-814045-001, 2026-04-02, 143.90 HT / 172.68 TTC.
Next expected ~2027-04.
- CLAUDE.AI: document payment rail moved to personal card (fk_account=3,
API-invisible) for May/June; reference issue #57.
Both AI subscriptions are now recorded supplier invoices (post-replay).
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <[email protected]>
The harness layer (builder sessions, cold verifiers, evidence flow) gets a
committable home, per the PRD model-fleet § harness portability and erp#63:
- fleet/harness/verifier/: the two canonical verifier tests (locate-test,
cold-reader backlog audit) with pinned inputs, verbatim prompts, ground
truth and pass rules — judged context-free, never self-graded.
- fleet/harness/bin/run-verifier.sh: runs a test against any OpenAI-style
local endpoint (Ornith/MLX) or vibe -p (Mistral); emits sha256-pinned
JSON transcripts.
- fleet/harness/bin/vibe-builder.sh: the bounded shell for scoped builders
and recurring tasks — refuses the trunk (linked-worktree guard), hard
--max-turns/--max-price caps, full JSON journal per run.
- fleet/README.md layout + AGENTS.md Fleet section updated in the same
change (same-change freshness rule).
Part of erp#63 (harness portability spike, D2).
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01VRShc4QhLLU73FLHx9vskh
Part of erp#65 (phase 1). Ledger grammar "thirdparty complete" gets its
op: allowlisted non-ledger fields, per-field diff read-back. Contacts
are born idempotent (dedupe by email then name). Promote ops wired both
targets, offline stub tests, SKILL.md workflows, KM dossier manifest
(unsigned-contract truth fix + EIN-to-collect note).
Co-Authored-By: Claude Fable 5 <[email protected]>
Closes erp#38 deliverables: fleet/ layout, atom.yaml schema documented
in fleet/README.md, 7 class skeletons per the PRD agent-catalog,
invoice-extract as the worked example (contract only — implementation
is erp#40), golden/ + profile/ stubs, AGENTS.md Fleet section with
freshness fixes (fleet/ no longer "not yet landed").
Co-Authored-By: Claude Fable 5 <[email protected]>
The pack (manifests, prelude, runbook, verify-provenance.py PoC) lived
only in an ephemeral session scratchpad while erp#41/#42/#43/#44 now
reference it as fixtures and the prod replay is still pending. 36/36
provenance checks were green at rehearsal time; PDFs are re-fetchable
via arcodange-email-ingest (documented in the pack README).
Co-Authored-By: Claude Fable 5 <[email protected]>
Agents landing in this repo had no entry point: no AGENTS.md, and the
backlog (erp#38-57 on 6 dated milestones, gateway#1-2, factory#22)
was only discoverable from the PRD STATUS in the factory repo. This
seeds the repo-root orientation map: where the work comes from (resume
protocol: top unblocked issue of the earliest open milestone, gitea
MCP pointers, owner gotcha for telegram-gateway), the repo map, the
operating rules (trunk/worktrees, read-only prod, sandbox+promote
gate, append-only ledger, anti-hallucination contract pointers).
Advances erp#38 (AGENTS.md seed; the fleet/ scaffold and the full
fleet section remain in #38's scope).
Co-Authored-By: Claude Fable 5 <[email protected]>
Qonto transaction ids run ~67 chars (<org>-<n>-<n>-transaction-<uuid>) but
Dolibarr stores num_payment in varchar(50) (llx_paiement.num_paiement,
llx_paiementfourn.num_paiement) — POSTing a payment with the raw id fails
HTTP 400 "value too long for type character varying(50)". Parade proven live
on the sandbox (2026-07-11): store the UUID suffix (globally unique, ~37
chars). Wise ids (short numerics) are unaffected.
Writer side — payment-record.sh strips everything through "transaction-"
before POST, announces the normalization on stderr, REFUSES (never truncates)
ids still >50 chars after normalization, and emits the normalized num in the
output JSON.
Reader side — bank-match.sh PASS 0 (exact tx-id, erp#28) now compares BOTH
sides in raw AND canonical short form: Qonto feed ids are carried long+short,
payment nums are normalized on compare — so nums stored short (the varchar(50)
form) and historical long-form nums both keep matching. Wise ids untouched.
Proven offline (no credentials, no network, no sandbox/prod writes):
- arcodange-bank-reco/tests/run-tests.sh — new bank-match --fixtures offline
mode: long feed id ↔ short num, long ↔ long (back-compat), Wise numeric,
each Δ+19d outside the ±7d window so only PASS 0 can pair them (exit 0,
3×[tx-id]); plus the empty-num negative (exit 1, 0 matched).
- dolibarr-sandbox-write/tests/run-tests.sh — payment-record via a stubbed
dol-write.sh (DOL_WRITE hook): long→short in POST body + output JSON, Wise
untouched, >50-after-normalization refused BEFORE any POST, citing
varchar(50).
Both SKILL.md document the canonical short form + the varchar(50) constraint.
Co-Authored-By: Claude Fable 5 <[email protected]>
The checkpoint status/relink-env armed probe calls GET /users/info, which
requires Dolibarr right 251 (user->user->lire). WRITE_IDS didn't include it,
so a freshly provisioned agent answered 403 on the probe — reported NOT armed
— while its key actually authenticates (GET /thirdparties -> 200). Right 251
was granted live in SQL on the sandbox (fk_user=4) today; this persists it in
WRITE_IDS so every future provision grants it.
Also teach both probes to tell the failure modes apart instead of one opaque
message: 401 = key rejected (stale/instance-encrypted -> re-provision),
403 = key OK but right 251 missing (-> grant it / re-provision), 200 = armed.
Docs updated accordingly (checkpoint SKILL.md probe outcomes, sandbox-write
SKILL.md gotcha, test/README.md rights table synced to WRITE_IDS incl. 262/111).
Co-Authored-By: Claude Fable 5 <[email protected]>
After an iso-prod refresh the instance unique-id changes, so an api_key encrypted
with the OLD id can't be decrypted — Dolibarr renders non-UTF-8 bytes in the field.
The POC's generateApiKey reused any non-empty value, so it copied that garbage into
test/.ai_agent_sandbox.key (corrupt key, 401s). Now it reuses ONLY a clean key
(^[A-Za-z0-9_-]{24,}$); otherwise it clears the field and regenerates. So
`checkpoint provision` after a refresh yields a fresh, working key.
Also documents the open PLATFORM follow-ups in ops/backup/README.md (easy to find
when revisiting ERP backups): the orphaned Longhorn `default` recurring-job group
(other cluster volumes have no offsite backup), and verifying the factory
pg_dumpall host cron.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
dolibarr-backup.sh restore --db|--docs <ts> --env <e> --yes — the recovery half of
the dedicated backup. DESTRUCTIVE (gated by --yes + explicit --env): scales the app
to 0, then
- --db: DROP OWNED BY <owner_role> CASCADE + pg_restore --no-owner --role (same
mechanics as sandbox-lifecycle.sh), from s3://.../erp/<env>/db/<ts>.dump;
- --docs: clears /var/www/documents and untars s3://.../erp/<env>/docs/<ts>.tar.gz;
then scales the app back to 1. OWNER_ROLE per env (erp_role / erp_sandbox_role).
The key is the bare <ts> filename from `list`; --db/--docs selects the subpath.
Proven on the sandbox: backup → mutate MAIN_INFO_SOCIETE_NOM to a sentinel →
restore --db → the value reverted to the backup's ('Arcodange'). (First run caught
a path bug — the fetch missed the db/ subdir — now fixed.)
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
tools#5 granted the erp prod Vault policy read on kvv2/data/longhorn/gcs-backup
(applied + verified: 1 changed, 0 destroyed). So the CronJob's VaultStaticSecret
can now resolve the GCS creds.
- backup.enabled: true (prod), vaultS3Path: longhorn/gcs-backup.
- sandbox overlay keeps backup.enabled: false (reproducible; its env policy wasn't
granted the read).
ArgoCD will deploy the CronJob + ConfigMap + VaultStaticSecret in the erp namespace.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Builds on the dedicated backup (erp#31).
Skip-if-unchanged: each half (DB / documents) carries a content fingerprint at
erp/<env>/.fp-{db,docs} and is dumped+uploaded only if it differs from the last
run — a quiet ERP day re-uploads nothing. Fingerprint = durable BUSINESS content
only: DB = count+max(tms) over tms tables EXCEPT volatile churn (llx_const,
llx_user, session/cron); docs EXCLUDE */temp/* (Dolibarr stats cache) — from both
the fingerprint and the tar. Proven live: 1st run uploads both, immediate 2nd run
skips both (uploaded=0).
Automation: the in-container logic moves to chart/files/backup-job.sh (single
source of truth, read by the orchestrator AND the chart). New
chart/templates/backup-cronjob.yaml renders a daily CronJob + ConfigMap +
VaultStaticSecret, gated by backup.enabled (default false). Helm-verified: off by
default (0 CronJobs), on renders correctly, env-aware (PREFIX erp/prod vs
erp/sandbox), script embedded.
Activation (documented): store GCS HMAC creds at kvv2/<backup.vaultS3Path>
(default erp/backup), grant the erp `auth` Vault role read on it (tools change),
set backup.enabled=true. Until then the orchestrator runs on demand.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The accounting data + issued documents are legally retained 10 years and warrant a
backup dedicated to Dolibarr. An audit found the generic Longhorn external backup
NEVER covered the erp volume (its Longhorn volume sits in the orphaned `default`
recurring-job group; the only job has groups=[] → serves nothing; lastBackupAt=never).
So /var/www/documents (invoice PDFs, supplier pieces, contracts, ECM) had zero
offsite copy — only in-cluster replicas.
ops/backup/dolibarr-backup.sh (orchestrator) + ops/backup/backup-job.sh (in-container
logic, env-driven, single source of truth):
- pg_dump -Fc of the DB + tar of the documents PVC (RWX, read-only mount) ->
s3://arcodange-backup/erp/<env>/{db,docs}/<ts>, then tiered prune (daily 30d /
monthly 12m / yearly 10y).
- prod is READ-only (dump+tar read; writes go only to the backup bucket); the DB is
read with the env's own dynamic creds; the GCS HMAC secret is copied transiently
(base64, deleted on exit) and never printed; the whole script ships base64.
- fixes the aws-cli v2.23+ default-checksum incompatibility with GCS/S3-compat
(SignatureDoesNotMatch) via AWS_*_CHECKSUM_*=when_required.
Proven live: sandbox end-to-end (dump+tar+upload+prune, verified in GCS, cleaned up)
and retention logic unit-tested (1100 daily -> 46 kept). The FIRST real prod backup
was taken (erp/prod/db 1.2 MB + erp/prod/docs 12.5 MB) — closing the gap now.
Automation (recurring CronJob in the chart + a dedicated erp Vault policy for its
own S3 creds) is the documented next step; the orchestrator works today on demand.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
A skill + CLI group to drive the ADR-0003 sandbox lifecycle, instead of the manual
kubectl/deno/.env dance:
arcodange sandbox checkpoint status # liveness + is the write agent armed?
arcodange sandbox checkpoint refresh --yes # re-seed iso-prod (DESTRUCTIVE, gated)
arcodange sandbox checkpoint provision # re-create ai_agent_sandbox (Playwright) + relink
arcodange sandbox checkpoint relink-env # rewrite write skill .env from the key + verify
- refresh wraps ops/sandbox/sandbox-lifecycle.sh; requires --yes (it wipes the agent
too, since iso-prod overwrites llx_user). --db-only skips the documents sync.
- provision runs test/provisionSandbox.ts (you do the admin login — PROD creds,
iso-prod) then auto-relinks; relink-env writes .env mode 600 and verifies via
GET /users/info.
- scripts resolve the repo root from ARCO_ROOT (set by bin/arcodange) or their own
path, so they work via the CLI or standalone.
Tested: status reports armed/not-armed correctly; refresh refuses without --yes
(exit 3); relink-env errors with no key (exit 1); help/usage wired.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
refresh-from-prod was structurally broken and silently no-op'd the restore:
1. pg_restore lacked -U, so the postgres image connected as its OS user `root`
and auth-failed. The failure was swallowed by `|| echo "ignorable warnings"`,
so the script reported success while the DROP OWNED had already emptied the DB.
E2's original seed was a manual process, so this path had never really run.
Fix: pass `-h $PGHOST -U $SB_PGUSER`; don't trust pg_restore's exit code (it
returns non-zero on the harmless "schema public already exists" notice) — verify
by counting restored llx_* tables and FAIL the Job if < 250.
2. erp-sandbox is ArgoCD-managed with self-heal ON, which reverts the
`kubectl scale --replicas=0` within seconds — so the seed ran with Dolibarr
still connected. Fix: pause self-heal for the duration, re-arm it after; app
restore + self-heal restoration + secret cleanup are guarded by an EXIT trap so
an interrupt can't strand the sandbox at replicas=0 / self-heal off.
Validated end-to-end on the live sandbox: 295 llx tables, company=Arcodange,
owner=erp_sandbox_role, self-heal re-armed, pod 1/1. README documents the self-heal
pause and the iso-prod consequence (ai_agent_sandbox is wiped → re-provision).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The consumer side of erp#26/#27: now that a règlement stores its originating bank
transaction id (transaction_id -> llx_bank.num_chq), bank-match uses it.
New PASS 0 (exact), highest priority, before wire-ref and amt+date:
- carry each feed movement's own id (Qonto transaction id; Wise activity + transfer
resource id) as feed_ids, and each Dolibarr payment's num.
- match when a payment's num equals a feed id. Tagged [tx-id].
- DATE-WINDOW-INDEPENDENT — the id is proof, so it pairs movements whose bank
settlement and Dolibarr saisie are weeks apart (which amt+date would miss).
Pass 0 runs before the ref index is built, so its matches are excluded from the
later passes (no double-match).
Fixture-proven: a payment dated 15d off the bank movement (outside the ±7d window)
matches via [tx-id] when num carries the Qonto id, and correctly does NOT match
when num is empty.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Make the originating bank transaction id a first-class input on payment-record.sh
so every règlement is tied to the real bank movement at write time.
- `transaction_id` is the canonical field (the Qonto/Wise feed tx id); `num` stays
as a back-compat alias. It's stored on the payment's bank line (llx_bank.num_chq),
the reconciliation key.
- Recording WITHOUT a transaction_id prints a stderr warning (still posts, but won't
auto-reconcile) — nudges the agent to always carry it.
- Output normalises to {id, bank_transaction_id, transaction_id}.
- Promote: manifests' payment ops carry transaction_id; promote-plan shows it
(tx=… or tx=MISSING).
Proven live: customer + supplier record with transaction_id; the `num` alias maps
to the same field; the no-tx warning fires; promote plan/apply carry it through.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
A payment only returned its paiement id, which isn't what bank reconciliation
keys on. payment-record.sh now emits {id, bank_transaction_id, num}:
- bank_transaction_id = the Dolibarr bank line (llx_bank.fk_bank_line) the payment
created, resolved via GET /{invoices|supplierinvoices}/{id}/payments (correlated
by num, else the most recent line). Works for customer and supplier.
- num stores the originating bank tx id (Qonto/Wise) and lands on that bank line's
num_chq — so arcodange-bank-reco can match a règlement to a statement line by id
instead of fuzzy amount/date. Both ends captured at write time.
Proven live: customer {id:13,bank_transaction_id:35,num:QONTO-TX-1234},
supplier {id:16,bank_transaction_id:36,num:WISE-TX-5678}; llx_bank rows 35/36
carry the refs in num_chq. promote-apply still extracts .id unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The two V9 follow-ups, both proven live on the sandbox.
- creditnote-create.sh: `kind:"supplier"` makes an avoir fournisseur on
/supplierinvoices (type=2 + fk_facture_source, carries ref_supplier); default
customer path unchanged. Proven: customer AVC002 (-240) + supplier AVF2026001
(-144, ref_supplier carried, linked to source, validated).
- bank-accounts.sh + `arcodange sandbox accounts`: list bank accounts (id/label/
bank) so a payment can pick its account_id. Needs `banque lire` (rights 111),
now added to the provisioner's WRITE_IDS so fresh runs include it; the existing
ai_agent_sandbox user was granted it live. GET /bankaccounts now returns the 3
accounts (QONTO, WISE EURO, Compte Courant Asso).
- SKILL.md: supplier-avoir example + accounts helper + updated banque-lire note.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Closes the last promote gap: a manifest can now reference records it does NOT
create. A value like "#thirdparty:name=KissMetrics" (or :code=CL0007) is looked
up on the TARGET at apply time and resolved to that target's id — so the same
manifest is portable (sandbox id on --target sandbox, prod id on --target prod).
promote-apply.sh: resolve() gains a "#" branch + a lookup() helper that queries
the target via the GET wrapper with sqlfilters. Supports thirdparty
(name/code/supplier_code) and invoice/supplierinvoice (ref/ref_supplier). A
lookup matching nothing OR more than one record ABORTS the run — it never
guesses, so it cannot write to the wrong entity.
Proven live: "#thirdparty:name=ACME Conseil" resolved to the existing client and
invoiced it; a not-found code and an ambiguous (2-match) name both aborted with
exit 1. Combined with @refs, arbitrary self-contained-or-referential change-sets
now replay cleanly.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The human-gated path that carries a reviewed sandbox change to prod.
- promote-plan.sh: render a manifest (JSON array of write ops with symbolic @refs
instead of ids — portable sandbox->prod) as a human-readable change-set.
- promote-apply.sh <manifest> --target sandbox|prod: replay it, resolving each
@ref to the id actually created during the run (dependent ops wire up). sandbox
rehearses via dol-write.sh; prod via dol-prod-write.sh.
- dol-prod-write.sh: the ONLY prod-write path. Prod key read from the ENVIRONMENT
only (DOLIBARR_PROD_WRITE_KEY, never a stored .env); every write refused unless
ARCO_PROMOTE_CONFIRM=I-UNDERSTAND-THIS-WRITES-PROD.
- create scripts take a DOL_WRITE override so promote-apply reuses them per target.
- bin/arcodange: `promote {plan|apply}` group + example manifest.
- payment-record.sh: fixed supplier payments (payment_mode_id + closepaidinvoices).
Proven live: plan renders; apply --target sandbox replays a 3-op chain with refs
resolved (@tp1->id, invoice socid=@tp1, payment invoice=@inv1); --target prod
without the confirm flag is REFUSED before sending. Supplier payment now works
end-to-end via the script.
Limitation (documented): manifests reference entities they create (@ref);
pre-existing prod entities need business-key resolution (follow-up).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
arcodange
