Learning #4 of the 2026-07-11 rehearsal: manifest B failed mid-run and could not be re-applied — op 1 (the DARNIS invoice) had already run and a replay would have duplicated it. Every write op now dedupes BEFORE any POST: - thirdparty-create.sh: by exact name (promote '#thirdparty:name=' semantics); ambiguous (2+) aborts; an existing fiche missing the requested role aborts (refuse-never-repair). Emits {"id", "deduped"} instead of a bare id. - invoice-create.sh: supplier kind by (socid, ref_supplier) — same key with a different total aborts as a conflict; customer kind (or supplier without ref_supplier) by (socid, date, total_ttc ±0.02, line fingerprint) with descs HTML-unescaped. Credit notes are never candidates. A deduped DRAFT with validate:true is validated on replay, so an interrupted run converges. - payment-record.sh: by (invoice, amount, normalized transaction_id), composing with the erp#37 varchar(50) normalization on BOTH sides so historical long-form nums still match; same tx + different amount aborts; without a tx id there is no dedupe key (warned). Dedupe answers id:null (the payments list exposes no paiement rowid) + the existing bank line. - All three refuse to POST blind when the dedupe lookup fails with anything but the documented empty-list 404 (the voir_tous trap would otherwise mint dupes). - promote-apply.sh: marks each op created / deduped=true inline and totals them in the summary — an all-deduped second run is visible proof of a no-op. - promote-plan.sh: advertises each op's dedupe key (and flags tx=MISSING as 'a replay WILL double-pay'). Proof: - tests/run-tests.sh: 5 new offline cases (11 total) — dedupe hits POST nothing, conflicts/ambiguity abort pre-POST, long-form history dedupes, draft convergence validates; stub extended to serve the new lookups with the live-observed empty behaviors ([] for invoices/payments, 404 for tiers). - tests/replay-idempotency.sh (new, live): double-applies a self-contained manifest on the sandbox — run 1 '3 created' (rows 1/1/1), run 2 '3 deduped' with row counts unchanged and the stored num in erp#37 short form. - The historic manifest-B now replays on the sandbox as 5/5 deduped, zero new rows — the exact replay Learning #4 declared impossible. SKILL.md updated in the same change (per-op dedupe keys, replay-safety section, gotchas); the 2026-07-11 runbook's Learning #4 carries a dated resolution addendum. Closes erp#44. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VRShc4QhLLU73FLHx9vskh
211 lines
10 KiB
Bash
Executable File
211 lines
10 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Record a payment (règlement) on a validated invoice in the SANDBOX —
|
|
# IDEMPOTENT (erp#44): replaying the same règlement is a no-op, not a double
|
|
# payment.
|
|
#
|
|
# Input: a JSON object on stdin (or a file path in $1):
|
|
# invoice_id (required) the invoice to pay
|
|
# kind "customer" | "supplier" (default "customer")
|
|
# mode "VIR" | "CB" | "CHQ" | "LIQ" (default "VIR")
|
|
# account_id (required) the bank account id receiving/paying
|
|
# date "YYYY-MM-DD" (default today)
|
|
# amount (REQUIRED for supplier; customer pays the full remaining)
|
|
# transaction_id (recommended) the originating bank transaction id (the Qonto/Wise
|
|
# tx id from the feed). Stored on the payment's bank line
|
|
# (llx_bank.num_chq) so the règlement reconciles to the feed by id.
|
|
# `num` is a back-compat alias for the same field.
|
|
# NORMALIZED before POST: Dolibarr stores num_payment in varchar(50)
|
|
# (llx_paiement.num_paiement / llx_paiementfourn.num_paiement) while
|
|
# Qonto ids run ~67 chars (<org>-<n>-<n>-transaction-<uuid>), so
|
|
# everything through "transaction-" is stripped and the UUID suffix
|
|
# (globally unique, ~37 chars) is the canonical stored form. Wise ids
|
|
# (short numerics) pass through unchanged. An id still >50 chars after
|
|
# normalization is REFUSED with an error — never truncated silently.
|
|
# comment (optional)
|
|
#
|
|
# Idempotency (erp#44, composing with the erp#37 normalization): BEFORE any POST,
|
|
# the invoice's payment list is fetched and deduped by
|
|
# (invoice, amount, normalized transaction_id):
|
|
# - a row whose num — normalized the same way (historical rows may still carry
|
|
# long-form Qonto ids) — equals the normalized transaction_id is a REPLAY:
|
|
# for supplier payments the amounts must also agree (±0.005; same tx id with
|
|
# a DIFFERENT amount ABORTS as a data conflict); customer payments settle the
|
|
# full remaining so the tx id alone is the key.
|
|
# - a dedupe hit emits {"id": null, "bank_transaction_id": <existing line>,
|
|
# "transaction_id": <normalized>, "deduped": true} and exits 0 WITHOUT
|
|
# posting (the payments list does not expose the paiement rowid — id is null
|
|
# by honesty, the bank line is the stable handle reconciliation keys on).
|
|
# - the SAME normalized tx appearing on 2+ rows of this invoice ABORTS (the
|
|
# target already holds duplicates; never guess).
|
|
# - WITHOUT a transaction_id there is NO dedupe key — the payment posts with a
|
|
# warning (as before) and a replay WILL duplicate it. Always pass the tx id.
|
|
# - a listing failure other than 404 ABORTS: paying blind would double-pay.
|
|
#
|
|
# The invoice must be VALIDATED first (invoice-create.sh ... "validate":true).
|
|
# Emits {id, bank_transaction_id, transaction_id, deduped} on stdout —
|
|
# `transaction_id` is the NORMALIZED num actually stored (what bank-match keys
|
|
# on), `bank_transaction_id` the Dolibarr bank line (llx_bank.fk_bank_line).
|
|
set -euo pipefail
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
W="${DOL_WRITE:-${SCRIPT_DIR}/dol-write.sh}"
|
|
|
|
SRC="${1:-}"
|
|
if [[ -n "${SRC}" && "${SRC}" != "-" ]]; then INPUT="$(cat "${SRC}")"; else INPUT="$(cat)"; fi
|
|
|
|
TMPD="$(mktemp -d -t payrec.XXXXXX)"
|
|
trap 'rm -rf "${TMPD}"' EXIT
|
|
cat > "${TMPD}/map.py" <<'PY'
|
|
import json, sys, datetime, re
|
|
d = json.loads(sys.stdin.read())
|
|
if not d.get("invoice_id"):
|
|
sys.exit("payment-record.sh: 'invoice_id' is required")
|
|
if not d.get("account_id"):
|
|
sys.exit("payment-record.sh: 'account_id' is required")
|
|
# Stable Dolibarr c_paiement ids (sandbox seeded from prod / standard defaults).
|
|
MODE = {"VIR": 2, "CB": 6, "CHQ": 7, "LIQ": 4}
|
|
mode = MODE.get(str(d.get("mode", "VIR")).upper())
|
|
if mode is None:
|
|
sys.exit("payment-record.sh: unknown mode (use VIR|CB|CHQ|LIQ)")
|
|
supplier = d.get("kind", "customer").lower() in ("supplier", "fournisseur")
|
|
ds = d.get("date")
|
|
epoch = int((datetime.datetime.strptime(ds, "%Y-%m-%d") if ds
|
|
else datetime.datetime.now()).timestamp())
|
|
inv = d["invoice_id"]
|
|
# transaction_id is the first-class bank-feed tx id; num is the back-compat alias.
|
|
raw_tx = str(d.get("transaction_id") or d.get("num") or "")
|
|
# Normalize to the canonical short form (erp#37). Dolibarr stores num_payment in
|
|
# varchar(50) (llx_paiement.num_paiement / llx_paiementfourn.num_paiement) and
|
|
# a Qonto id (~67 chars, <org>-<n>-<n>-transaction-<uuid>) blows past it —
|
|
# HTTP 400 "value too long for type character varying(50)". Strip everything
|
|
# through "transaction-" and store the UUID suffix (globally unique). Wise ids
|
|
# (short numerics, no "transaction-") pass through unchanged. bank-match.sh
|
|
# normalizes feed ids the same way, so the short form still reconciles by id.
|
|
tx = re.sub(r'^.*transaction-', '', raw_tx)
|
|
if tx != raw_tx:
|
|
sys.stderr.write("payment-record.sh: transaction_id normalized to %r "
|
|
"(Qonto prefix stripped — num_payment is varchar(50))\n" % tx)
|
|
if len(tx) > 50:
|
|
sys.exit("payment-record.sh: transaction_id %r is %d chars even after "
|
|
"normalization — Dolibarr's num_payment is varchar(50) and silent "
|
|
"truncation would break bank reconciliation; pass a shorter id"
|
|
% (tx, len(tx)))
|
|
if not tx:
|
|
sys.stderr.write("payment-record.sh: WARNING — no transaction_id given; this "
|
|
"règlement won't auto-reconcile to the bank feed AND cannot "
|
|
"be deduped on a replay (erp#44) — a re-run will double-pay\n")
|
|
if supplier:
|
|
if d.get("amount") is None:
|
|
sys.exit("payment-record.sh: supplier payments require an 'amount'")
|
|
endpoint = "/supplierinvoices/%s/payments" % inv
|
|
body = {"datepaye": epoch, "payment_mode_id": mode, "closepaidinvoices": "yes",
|
|
"accountid": d["account_id"],
|
|
"amount": str(d["amount"]), "num_payment": tx,
|
|
"comment": d.get("comment", "")}
|
|
else:
|
|
endpoint = "/invoices/%s/payments" % inv
|
|
body = {"datepaye": epoch, "paymentid": mode, "closepaidinvoices": "yes",
|
|
"accountid": d["account_id"], "num_payment": tx,
|
|
"comment": d.get("comment", "")}
|
|
print(endpoint)
|
|
print(json.dumps(body))
|
|
print(tx)
|
|
print(json.dumps({"supplier": supplier,
|
|
"amount": (float(d["amount"]) if supplier else None)}))
|
|
PY
|
|
|
|
# Dedupe a replay against the invoice's existing payments (erp#44).
|
|
cat > "${TMPD}/dedupe.py" <<'PY'
|
|
import json, sys, os, re
|
|
ded = json.loads(os.environ["DEDUPE"])
|
|
tx = os.environ.get("TX", "")
|
|
try:
|
|
rows = json.load(sys.stdin)
|
|
except Exception:
|
|
rows = []
|
|
rows = rows if isinstance(rows, list) else []
|
|
norm = lambda s: re.sub(r'^.*transaction-', '', str(s or "")) # erp#37, both sides
|
|
hits = [r for r in rows if tx and norm(r.get("num")) == tx]
|
|
if not hits:
|
|
sys.exit(0) # no match -> proceed to POST
|
|
if len(hits) > 1:
|
|
sys.exit("payment-record.sh: ABORT — transaction_id %r already appears on %d "
|
|
"payments of this invoice; the target holds duplicates, refusing to "
|
|
"guess" % (tx, len(hits)))
|
|
r = hits[0]
|
|
if ded["supplier"] and ded["amount"] is not None:
|
|
try:
|
|
got = float(r.get("amount"))
|
|
except (TypeError, ValueError):
|
|
got = None
|
|
if got is None or abs(got - ded["amount"]) > 0.005:
|
|
sys.exit("payment-record.sh: ABORT — transaction_id %r is already recorded "
|
|
"on this invoice with amount %s, but %s was requested. Same tx, "
|
|
"different amount is a conflict to resolve, not a dedupe."
|
|
% (tx, r.get("amount"), ded["amount"]))
|
|
btx = r.get("fk_bank_line")
|
|
print(json.dumps({"id": None,
|
|
"bank_transaction_id": int(btx) if btx and str(btx).isdigit() else btx,
|
|
"transaction_id": tx, "deduped": True}))
|
|
PY
|
|
|
|
# Correlate the created payment back to its bank transaction line. The payments
|
|
# list carries fk_bank_line but not the paiement rowid, so match on the provided
|
|
# transaction_id (the external bank ref), else fall back to the most recent line.
|
|
cat > "${TMPD}/correlate.py" <<'PY'
|
|
import json, sys, os
|
|
rows = json.load(sys.stdin); rows = rows if isinstance(rows, list) else []
|
|
tx = os.environ.get("TX", ""); pid = int(os.environ["PAYID"])
|
|
pick = None
|
|
if tx:
|
|
cand = [r for r in rows if str(r.get("num", "")) == tx]
|
|
if cand:
|
|
pick = cand[-1]
|
|
if pick is None and rows:
|
|
pick = max(rows, key=lambda r: r.get("date", ""))
|
|
btx = (pick or {}).get("fk_bank_line")
|
|
# transaction_id: the num as stored (normalized) — fall back to the sent TX so
|
|
# the JSON always reports the canonical short form even on the recency fallback.
|
|
print(json.dumps({"id": pid,
|
|
"bank_transaction_id": int(btx) if btx and str(btx).isdigit() else btx,
|
|
"transaction_id": (pick or {}).get("num", "") or tx,
|
|
"deduped": False}))
|
|
PY
|
|
|
|
MAPPED="$(printf '%s' "${INPUT}" | python3 "${TMPD}/map.py")"
|
|
ENDPOINT="$(sed -n 1p <<<"${MAPPED}")"
|
|
BODY="$(sed -n 2p <<<"${MAPPED}")"
|
|
TX="$(sed -n 3p <<<"${MAPPED}")"
|
|
DEDUPE="$(sed -n 4p <<<"${MAPPED}")"
|
|
|
|
# --- Pre-POST dedupe (erp#44) — only possible when a transaction_id was given ---
|
|
if [[ -n "${TX}" ]]; then
|
|
set +e
|
|
"${W}" GET "${ENDPOINT}" > "${TMPD}/payments.json" 2> "${TMPD}/list.err"
|
|
rc=$?
|
|
set -e
|
|
if [[ ${rc} -ne 0 ]]; then
|
|
if grep -q "HTTP 404" "${TMPD}/list.err"; then
|
|
printf '[]' > "${TMPD}/payments.json" # empty lists can answer 404 (dolibarr skill gotcha)
|
|
else
|
|
cat "${TMPD}/list.err" >&2
|
|
echo "payment-record.sh: could not list ${ENDPOINT} — refusing to pay blind (dedupe impossible)" >&2
|
|
exit 1
|
|
fi
|
|
fi
|
|
MATCH="$(DEDUPE="${DEDUPE}" TX="${TX}" python3 "${TMPD}/dedupe.py" < "${TMPD}/payments.json")"
|
|
if [[ -n "${MATCH}" ]]; then
|
|
echo "payment-record.sh: transaction_id ${TX} already recorded on this invoice — deduped, no POST" >&2
|
|
printf '%s\n' "${MATCH}"
|
|
exit 0
|
|
fi
|
|
fi
|
|
|
|
PAYID="$("${W}" POST "${ENDPOINT}" "${BODY}")"
|
|
if [[ ! "${PAYID}" =~ ^[0-9]+$ ]]; then
|
|
echo "payment-record.sh: payment POST did not return an id: ${PAYID}" >&2
|
|
exit 1
|
|
fi
|
|
|
|
# Same path serves the GET list; resolve fk_bank_line and emit the enriched record.
|
|
"${W}" GET "${ENDPOINT}" | PAYID="${PAYID}" TX="${TX}" python3 "${TMPD}/correlate.py"
|