fix(txid): normalize bank tx ids to fit Dolibarr's num_payment varchar(50)
Qonto transaction ids run ~67 chars (<org>-<n>-<n>-transaction-<uuid>) but Dolibarr stores num_payment in varchar(50) (llx_paiement.num_paiement, llx_paiementfourn.num_paiement) — POSTing a payment with the raw id fails HTTP 400 "value too long for type character varying(50)". Parade proven live on the sandbox (2026-07-11): store the UUID suffix (globally unique, ~37 chars). Wise ids (short numerics) are unaffected. Writer side — payment-record.sh strips everything through "transaction-" before POST, announces the normalization on stderr, REFUSES (never truncates) ids still >50 chars after normalization, and emits the normalized num in the output JSON. Reader side — bank-match.sh PASS 0 (exact tx-id, erp#28) now compares BOTH sides in raw AND canonical short form: Qonto feed ids are carried long+short, payment nums are normalized on compare — so nums stored short (the varchar(50) form) and historical long-form nums both keep matching. Wise ids untouched. Proven offline (no credentials, no network, no sandbox/prod writes): - arcodange-bank-reco/tests/run-tests.sh — new bank-match --fixtures offline mode: long feed id ↔ short num, long ↔ long (back-compat), Wise numeric, each Δ+19d outside the ±7d window so only PASS 0 can pair them (exit 0, 3×[tx-id]); plus the empty-num negative (exit 1, 0 matched). - dolibarr-sandbox-write/tests/run-tests.sh — payment-record via a stubbed dol-write.sh (DOL_WRITE hook): long→short in POST body + output JSON, Wise untouched, >50-after-normalization refused BEFORE any POST, citing varchar(50). Both SKILL.md document the canonical short form + the varchar(50) constraint. Co-Authored-By: Claude Fable 5 <[email protected]>
This commit is contained in:
Claude Fable 5
@@ -96,9 +96,10 @@ to leave a draft. Emits `{id, ref, ref_supplier, total_ht, total_ttc, statut}`.
|
||||
### 3 · Payment (règlement) — `scripts/payment-record.sh`
|
||||
|
||||
```sh
|
||||
echo '{"invoice_id":19,"mode":"VIR","account_id":1,"transaction_id":"QONTO-TX-1234"}' | scripts/payment-record.sh
|
||||
echo '{"invoice_id":13,"kind":"supplier","mode":"VIR","account_id":1,"amount":96,"transaction_id":"WISE-TX-5678"}' \
|
||||
| scripts/payment-record.sh
|
||||
echo '{"invoice_id":19,"mode":"VIR","account_id":1,"transaction_id":"2159468139"}' | scripts/payment-record.sh
|
||||
echo '{"invoice_id":13,"kind":"supplier","mode":"VIR","account_id":1,"amount":96,
|
||||
"transaction_id":"arcodange-1246-1-transaction-019f14c5-e254-7ac9-9e9f-307ed9-d55f44"}' \
|
||||
| scripts/payment-record.sh # stores num 019f14c5-e254-7ac9-9e9f-307ed9-d55f44
|
||||
```
|
||||
The invoice must be **validated** first. `mode`: `VIR|CB|CHQ|LIQ`. Customer
|
||||
payments settle the full remaining amount and mark the invoice paid; **supplier**
|
||||
@@ -112,9 +113,22 @@ reconciliation matches **by id** rather than by fuzzy amount/date. (`num` is a
|
||||
back-compat alias for the same field.) Recording without it prints a warning — the
|
||||
payment still posts, but it won't auto-reconcile.
|
||||
|
||||
**Canonical short form — the varchar(50) constraint.** Dolibarr stores
|
||||
`num_payment` in **varchar(50)** columns (`llx_paiement.num_paiement`,
|
||||
`llx_paiementfourn.num_paiement`) while Qonto transaction ids run ~67 chars
|
||||
(`<org>-<n>-<n>-transaction-<uuid>`) — POSTing one raw fails with HTTP 400
|
||||
"value too long for type character varying(50)". The script therefore normalizes
|
||||
before POST: everything through `transaction-` is stripped and the **UUID suffix**
|
||||
(globally unique, ~37 chars) is what gets stored. Wise ids (short numerics) pass
|
||||
through unchanged. An id still >50 chars after normalization is **refused** with
|
||||
an explicit error — never truncated silently. `arcodange-bank-reco`'s bank-match
|
||||
normalizes feed ids the same way, so short-form nums keep reconciling by id
|
||||
(historical long-form nums too). Pass the raw feed id; the script does the rest.
|
||||
|
||||
Emits **`{id, bank_transaction_id, transaction_id}`**. `bank_transaction_id` is the
|
||||
Dolibarr bank line (`llx_bank.fk_bank_line`) the payment created — the id the
|
||||
reconciliation (`arcodange-bank-reco`) keys on. Both ends are captured at write time.
|
||||
reconciliation (`arcodange-bank-reco`) keys on. `transaction_id` echoes the
|
||||
**normalized** num actually stored. Both ends are captured at write time.
|
||||
|
||||
### 4 · Credit note (avoir) — `scripts/creditnote-create.sh`
|
||||
|
||||
@@ -180,6 +194,10 @@ sandbox KissMetrics on `--target sandbox` and the prod one on `--target prod`.
|
||||
REST create needs `code_client`/`code_fournisseur = "-1"` to trigger it — the
|
||||
script does this; without it the API errors `ErrorCustomerCodeRequired`.
|
||||
- **Dates** are sent as Unix epochs; pass `date:"YYYY-MM-DD"` or omit for today.
|
||||
- **Qonto tx ids don't fit varchar(50).** `payment-record.sh` strips the
|
||||
`<org>-<n>-<n>-transaction-` prefix and stores the UUID suffix (see workflow 3);
|
||||
ids still >50 chars after that are refused. Offline proof: `tests/run-tests.sh`
|
||||
(stubbed `dol-write.sh` via the `DOL_WRITE` hook — writes nothing anywhere).
|
||||
- **`banque lire`** (rights id 111) is granted → `scripts/bank-accounts.sh` lists
|
||||
accounts (id/label/bank) so a payment can pick its `account_id`. It's in the
|
||||
provisioner's `WRITE_IDS`, so a fresh `provisionSandbox.ts` run includes it.
|
||||
|
||||
@@ -12,10 +12,18 @@
|
||||
# tx id from the feed). Stored on the payment's bank line
|
||||
# (llx_bank.num_chq) so the règlement reconciles to the feed by id.
|
||||
# `num` is a back-compat alias for the same field.
|
||||
# NORMALIZED before POST: Dolibarr stores num_payment in varchar(50)
|
||||
# (llx_paiement.num_paiement / llx_paiementfourn.num_paiement) while
|
||||
# Qonto ids run ~67 chars (<org>-<n>-<n>-transaction-<uuid>), so
|
||||
# everything through "transaction-" is stripped and the UUID suffix
|
||||
# (globally unique, ~37 chars) is the canonical stored form. Wise ids
|
||||
# (short numerics) pass through unchanged. An id still >50 chars after
|
||||
# normalization is REFUSED with an error — never truncated silently.
|
||||
# comment (optional)
|
||||
#
|
||||
# The invoice must be VALIDATED first (invoice-create.sh ... "validate":true).
|
||||
# Emits {id, bank_transaction_id, transaction_id} on stdout. `bank_transaction_id`
|
||||
# Emits {id, bank_transaction_id, transaction_id} on stdout — `transaction_id` is
|
||||
# the NORMALIZED num actually stored (what bank-match keys on). `bank_transaction_id`
|
||||
# is the Dolibarr bank line (llx_bank.fk_bank_line) the payment created — the id
|
||||
# bank reconciliation (arcodange-bank-reco) keys on to link this règlement to a
|
||||
# statement line. Recording without a transaction_id warns (it won't auto-reconcile).
|
||||
@@ -29,7 +37,7 @@ if [[ -n "${SRC}" && "${SRC}" != "-" ]]; then INPUT="$(cat "${SRC}")"; else INPU
|
||||
PYF="$(mktemp -t dolpy.XXXXXX)"; PYF2="$(mktemp -t dolpy2.XXXXXX)"
|
||||
trap 'rm -f "${PYF}" "${PYF2}"' EXIT
|
||||
cat > "${PYF}" <<'PY'
|
||||
import json, sys, datetime
|
||||
import json, sys, datetime, re
|
||||
d = json.loads(sys.stdin.read())
|
||||
if not d.get("invoice_id"):
|
||||
sys.exit("payment-record.sh: 'invoice_id' is required")
|
||||
@@ -46,7 +54,23 @@ epoch = int((datetime.datetime.strptime(ds, "%Y-%m-%d") if ds
|
||||
else datetime.datetime.now()).timestamp())
|
||||
inv = d["invoice_id"]
|
||||
# transaction_id is the first-class bank-feed tx id; num is the back-compat alias.
|
||||
tx = str(d.get("transaction_id") or d.get("num") or "")
|
||||
raw_tx = str(d.get("transaction_id") or d.get("num") or "")
|
||||
# Normalize to the canonical short form. Dolibarr stores num_payment in
|
||||
# varchar(50) (llx_paiement.num_paiement / llx_paiementfourn.num_paiement) and
|
||||
# a Qonto id (~67 chars, <org>-<n>-<n>-transaction-<uuid>) blows past it —
|
||||
# HTTP 400 "value too long for type character varying(50)". Strip everything
|
||||
# through "transaction-" and store the UUID suffix (globally unique). Wise ids
|
||||
# (short numerics, no "transaction-") pass through unchanged. bank-match.sh
|
||||
# normalizes feed ids the same way, so the short form still reconciles by id.
|
||||
tx = re.sub(r'^.*transaction-', '', raw_tx)
|
||||
if tx != raw_tx:
|
||||
sys.stderr.write("payment-record.sh: transaction_id normalized to %r "
|
||||
"(Qonto prefix stripped — num_payment is varchar(50))\n" % tx)
|
||||
if len(tx) > 50:
|
||||
sys.exit("payment-record.sh: transaction_id %r is %d chars even after "
|
||||
"normalization — Dolibarr's num_payment is varchar(50) and silent "
|
||||
"truncation would break bank reconciliation; pass a shorter id"
|
||||
% (tx, len(tx)))
|
||||
if not tx:
|
||||
sys.stderr.write("payment-record.sh: WARNING — no transaction_id given; this "
|
||||
"règlement won't auto-reconcile to the bank feed\n")
|
||||
@@ -83,9 +107,11 @@ if tx:
|
||||
if pick is None and rows:
|
||||
pick = max(rows, key=lambda r: r.get("date", ""))
|
||||
btx = (pick or {}).get("fk_bank_line")
|
||||
# transaction_id: the num as stored (normalized) — fall back to the sent TX so
|
||||
# the JSON always reports the canonical short form even on the recency fallback.
|
||||
print(json.dumps({"id": pid,
|
||||
"bank_transaction_id": int(btx) if btx and str(btx).isdigit() else btx,
|
||||
"transaction_id": (pick or {}).get("num", "")}))
|
||||
"transaction_id": (pick or {}).get("num", "") or tx}))
|
||||
PY
|
||||
|
||||
MAPPED="$(printf '%s' "${INPUT}" | python3 "${PYF}")"
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
#!/usr/bin/env bash
|
||||
# Offline tests for payment-record.sh's transaction_id normalization — the writer
|
||||
# side of the varchar(50) fix (Dolibarr's num_payment columns truncate at 50 chars,
|
||||
# Qonto ids run ~67). Uses tests/stub-dol-write.sh via the DOL_WRITE env hook, so
|
||||
# NOTHING is written to the sandbox or prod.
|
||||
# 1. Long Qonto id → POST carries the UUID suffix; JSON reports it; stderr says so.
|
||||
# 2. Wise numeric id → passes through untouched, no normalization notice.
|
||||
# 3. Id still >50 chars after normalization → refused BEFORE any POST, error
|
||||
# cites varchar(50) (never a silent truncation).
|
||||
set -euo pipefail
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PR="${SCRIPT_DIR}/../scripts/payment-record.sh"
|
||||
STUB="${SCRIPT_DIR}/stub-dol-write.sh"
|
||||
|
||||
fail() { echo "FAIL: $*" >&2; exit 1; }
|
||||
|
||||
bash -n "${PR}" || fail "bash -n payment-record.sh"
|
||||
bash -n "${STUB}" || fail "bash -n stub-dol-write.sh"
|
||||
|
||||
STATE="$(mktemp -d -t prtest.XXXXXX)"
|
||||
trap 'rm -rf "${STATE}"' EXIT
|
||||
|
||||
LONG="arcodange-1246-1-transaction-019f14c5-e254-7ac9-9e9f-307ed9-d55f44"
|
||||
SHORT="019f14c5-e254-7ac9-9e9f-307ed9-d55f44"
|
||||
|
||||
# --- Case 1: long Qonto id is normalized to the UUID suffix ---
|
||||
OUT="$(printf '{"invoice_id":13,"kind":"supplier","account_id":1,"amount":96,"transaction_id":"%s"}' "${LONG}" \
|
||||
| DOL_WRITE="${STUB}" STUB_STATE="${STATE}" bash "${PR}" 2>"${STATE}/stderr1")" \
|
||||
|| fail "long-qonto-id: expected success, got $?"
|
||||
grep -q "\"num_payment\": \"${SHORT}\"" "${STATE}/post_body.json" \
|
||||
|| fail "long-qonto-id: POST body must carry the SHORT num, got: $(cat "${STATE}/post_body.json")"
|
||||
python3 -c "
|
||||
import json, sys
|
||||
o = json.loads('''${OUT}''')
|
||||
assert o['transaction_id'] == '${SHORT}', o
|
||||
assert o['id'] == 77 and o['bank_transaction_id'] == 556, o
|
||||
" || fail "long-qonto-id: output JSON must report the normalized num, got: ${OUT}"
|
||||
grep -q 'normalized' "${STATE}/stderr1" || fail "long-qonto-id: normalization must be announced on stderr"
|
||||
|
||||
# --- Case 2: Wise numeric id passes through unchanged ---
|
||||
OUT="$(printf '{"invoice_id":19,"account_id":2,"transaction_id":"2159468139"}' \
|
||||
| DOL_WRITE="${STUB}" STUB_STATE="${STATE}" bash "${PR}" 2>"${STATE}/stderr2")" \
|
||||
|| fail "wise-id: expected success, got $?"
|
||||
grep -q '"num_payment": "2159468139"' "${STATE}/post_body.json" \
|
||||
|| fail "wise-id: POST body must carry the id untouched"
|
||||
grep -q 'normalized' "${STATE}/stderr2" && fail "wise-id: must NOT announce a normalization"
|
||||
|
||||
# --- Case 3: >50 chars after normalization is refused before any POST ---
|
||||
rm -f "${STATE}/post_body.json" "${STATE}/post_endpoint"
|
||||
BAD="qonto-migration-batch-7-payment-reference-0123456789-0123456789" # 63 chars, no "transaction-"
|
||||
rc=0
|
||||
printf '{"invoice_id":13,"kind":"supplier","account_id":1,"amount":96,"transaction_id":"%s"}' "${BAD}" \
|
||||
| DOL_WRITE="${STUB}" STUB_STATE="${STATE}" bash "${PR}" >/dev/null 2>"${STATE}/stderr3" || rc=$?
|
||||
[[ "${rc}" -ne 0 ]] || fail "overlong-id: must exit non-zero"
|
||||
[[ ! -f "${STATE}/post_body.json" ]] || fail "overlong-id: must refuse BEFORE any POST"
|
||||
grep -q 'varchar(50)' "${STATE}/stderr3" || fail "overlong-id: error must cite the varchar(50) constraint"
|
||||
|
||||
echo "OK: payment-record normalization tests passed (long→short, wise untouched, >50 refused pre-POST)"
|
||||
@@ -0,0 +1,27 @@
|
||||
#!/usr/bin/env bash
|
||||
# Offline stand-in for dol-write.sh, used ONLY by tests/run-tests.sh (injected via
|
||||
# the DOL_WRITE env hook). Records the POST body under $STUB_STATE, then serves it
|
||||
# back as the payments list on GET so payment-record.sh can correlate. Never talks
|
||||
# to any host — sandbox and prod are both out of reach by construction.
|
||||
set -euo pipefail
|
||||
STATE="${STUB_STATE:?stub-dol-write.sh: STUB_STATE not set}"
|
||||
METHOD="$1"; ENDPOINT="$2"; BODY="${3:-}"
|
||||
case "${METHOD}" in
|
||||
POST)
|
||||
printf '%s' "${BODY}" > "${STATE}/post_body.json"
|
||||
printf '%s\n' "${ENDPOINT}" > "${STATE}/post_endpoint"
|
||||
echo "77"
|
||||
;;
|
||||
GET)
|
||||
python3 - "${STATE}/post_body.json" <<'PY'
|
||||
import json, sys
|
||||
body = json.load(open(sys.argv[1]))
|
||||
print(json.dumps([{"num": body.get("num_payment", ""),
|
||||
"date": "2026-06-20 12:00:00",
|
||||
"fk_bank_line": "556"}]))
|
||||
PY
|
||||
;;
|
||||
*)
|
||||
echo "stub-dol-write.sh: unexpected method ${METHOD}" >&2; exit 2
|
||||
;;
|
||||
esac
|
||||
Reference in New Issue
Block a user