Compare commits
24 Commits
4252a88681
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 2154bf319e | |||
| e4a7f99333 | |||
| 04281f0ab7 | |||
| c010099dae | |||
| 4cc5ca39ce | |||
| fbc0cc6962 | |||
| 7264f00ed4 | |||
| 523f0cf001 | |||
| cb17332314 | |||
| c0d5f2e144 | |||
| 354b40549d | |||
| e2fa327361 | |||
| b4bdbe75df | |||
| ec4df4719f | |||
| 444886b91a | |||
| 1d38f25c23 | |||
| 794aa18d2a | |||
| c2d8479f5e | |||
| a1042a483b | |||
| 246c7fc5a9 | |||
| 0f5b6bcbad | |||
| 4b6a5f7529 | |||
| f398003eae | |||
| bd90266372 |
234
.claude/skills/arcodange-bank-reco/SKILL.md
Normal file
234
.claude/skills/arcodange-bank-reco/SKILL.md
Normal file
@@ -0,0 +1,234 @@
|
|||||||
|
---
|
||||||
|
name: arcodange-bank-reco
|
||||||
|
description: Bank-side reconciliation for Arcodange — cross-check Dolibarr customer + supplier payments against the actual movements on Qonto (FR business account, full API access) and Wise (BUSINESS EUR balance, activity-list API). Five workflows — (1) probe / discover auth + IDs; (2) list Qonto transactions for a period; (3) list Wise activities including incoming KissMetrics payments (via /v1/profiles/{pid}/activities — bypasses the EU statement endpoint restriction); (4) match bank movements against Dolibarr payments in three buckets (matched, bank-only, dolibarr-only) with auto-detection of Wise↔Qonto internal consolidations; (5) live balances per account with Dolibarr cross-check per fk_account. Surfaces concrete findings — incoming KM payments not yet entered in Dolibarr, expenses on the bank without supplier invoices recorded, date drift between bank settlement and Dolibarr saisie, and personal-account fk_account=3 movements that are invisible via API. Use when the user asks "réconcilier la banque", "qu'est-ce que la banque a vu que Dolibarr n'a pas", "match bank vs ERP", "audit comptable Arcodange", "did KM actually pay X", "cohort review bank evidence". Depends on `dolibarr` for the ERP side. SKIP for write operations (this is read-only; entries go through Dolibarr UI), for non-Arcodange bank accounts, and for Wise BUSINESS balance-statement endpoints (not available to EU personal tokens — we use the activity list instead).
|
||||||
|
requires:
|
||||||
|
bins: ["curl", "jq", "python3", "openssl"]
|
||||||
|
auth: true
|
||||||
|
---
|
||||||
|
|
||||||
|
# arcodange-bank-reco — close the loop between Dolibarr and the bank
|
||||||
|
|
||||||
|
The V1-V5 skills tell you what Dolibarr *thinks* happened. This one tells you what the **bank actually saw**, and matches the two sides. The three buckets it produces (matched / bank-only / dolibarr-only) are the foundation for any clean accounting audit and any cohort-review evidence pack.
|
||||||
|
|
||||||
|
Depends on the [dolibarr](../dolibarr/SKILL.md) base skill.
|
||||||
|
|
||||||
|
**CLI shortcuts:** `bin/arcodange bank probe | qonto-transactions | wise-transactions | match | balance | curl`
|
||||||
|
|
||||||
|
## Wise SCA & the EU restriction — the path we settled on
|
||||||
|
|
||||||
|
Wise has TWO ways to list movements:
|
||||||
|
|
||||||
|
1. **`/v1/profiles/{pid}/balance-statements/{balanceId}/statement.json`** — the obvious "statement" endpoint. **Returns 403 for EU personal tokens** (FR included). Wise's own docs say it: "Funding transfers and retrieving balance statements via API are not supported except for accounts based in the US, Canada, Australia, New Zealand, Singapore, and Malaysia." Even with full SCA setup (RSA keypair + uploaded public key), the BUSINESS profile statements stay 403. Don't go down that rabbit hole — we did, and Wise just keeps responding `x-2fa-approval-result: REJECTED` regardless.
|
||||||
|
|
||||||
|
2. **`/v1/profiles/{pid}/activities`** — the activity list. **Works for EU personal tokens with no SCA.** Returns incoming + outgoing in a unified HTML-tagged feed. This is what the skill uses. The cost is that amounts come back as `"<positive>+ 5,100.00 EUR</positive>"` strings instead of structured numerics, so we parse HTML out — easy.
|
||||||
|
|
||||||
|
We tried the SCA path first, didn't work, then found this. Documented so the next operator doesn't repeat the dance.
|
||||||
|
|
||||||
|
## Qonto — no surprises
|
||||||
|
|
||||||
|
Qonto's API works as documented:
|
||||||
|
- `Authorization: <login>:<secret_key>` header.
|
||||||
|
- `/v2/organization` lists bank accounts + balances.
|
||||||
|
- `/v2/transactions?bank_account_id=<id>&settled_at_from=<iso>&settled_at_to=<iso>` lists transactions with pagination via `current_page`.
|
||||||
|
|
||||||
|
Wise↔Qonto integration in the Qonto UI does NOT expose Wise data via Qonto's API. We confirmed: no `/v2/external_accounts`, no `/v2/aggregated_accounts` endpoint surfaces Wise transactions. The two banks remain separately queried, then merged in the matching layer.
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
1. Base skill set up ([dolibarr/README.md](../dolibarr/README.md)).
|
||||||
|
2. `.env` extended with:
|
||||||
|
```
|
||||||
|
QONTO_LOGIN=arcodange-XXXXX
|
||||||
|
QONTO_SECRET_KEY=<secret>
|
||||||
|
QONTO_ORG_SLUG=arcodange-XXXXX
|
||||||
|
WISE_API_TOKEN=<token>
|
||||||
|
WISE_PROFILE_ID=<numeric id of the BUSINESS profile>
|
||||||
|
```
|
||||||
|
3. `chmod 600 ~/.config/arcodange-erp/.env`; propagate to the two in-repo hard copies.
|
||||||
|
|
||||||
|
To generate the tokens:
|
||||||
|
- **Qonto**: https://app.qonto.com/ → Settings → Integrations → API → Generate a new key. Copy login + secret (shown once).
|
||||||
|
- **Wise**: https://wise.com/your-account/integrations-and-tools/api-tokens → Add new token → name it `arcodange-bank-reco-readonly` → scope read-only.
|
||||||
|
|
||||||
|
The `WISE_SCA_KEY_PATH` variable exists in the `.env` schema for completeness but is **NOT REQUIRED** today — the activity-list endpoint we use doesn't need SCA. Keep the keypair generated under `~/.config/arcodange-erp/wise-sca-*.pem` so we can revisit if Wise ever opens the statement endpoint to EU tokens.
|
||||||
|
|
||||||
|
## Workflows
|
||||||
|
|
||||||
|
### 1. Probe / discovery
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bin/arcodange bank probe
|
||||||
|
```
|
||||||
|
|
||||||
|
Confirms auth on both banks and prints discovered IDs (Qonto org slug, Wise profile id, balance ids). Run once after any token rotation.
|
||||||
|
|
||||||
|
### 2. Qonto transactions
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bin/arcodange bank qonto-transactions # last 90 days
|
||||||
|
bin/arcodange bank qonto-transactions --month 2026-03
|
||||||
|
bin/arcodange bank qonto-transactions --since 2026-01-01 --until 2026-05-31
|
||||||
|
bin/arcodange bank qonto-transactions --side credit # filter incoming only
|
||||||
|
```
|
||||||
|
|
||||||
|
Captured at [examples/qonto-transactions.txt](examples/qonto-transactions.txt). The full Jan-May 2026 view shows 9 transactions netting to +4191.54 € — exactly the current Qonto balance.
|
||||||
|
|
||||||
|
### 3. Wise activities
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bin/arcodange bank wise-transactions # last 365 days
|
||||||
|
bin/arcodange bank wise-transactions --month 2026-03
|
||||||
|
bin/arcodange bank wise-transactions --type TRANSFER # filter
|
||||||
|
bin/arcodange bank wise-transactions --since 2026-01-01 --enrich # add wire references
|
||||||
|
```
|
||||||
|
|
||||||
|
Captured at [examples/wise-transactions.txt](examples/wise-transactions.txt). With `--enrich`, each TRANSFER is annotated with its wire reference (e.g. `FROM KISSMETRICS HOLDINGS INC FOR INVOICE FAC002CL0001002/ VENDOR:DEV`), which makes manual cross-checking trivial.
|
||||||
|
|
||||||
|
Net = +5308.25 € over the whole period, matches the live balance.
|
||||||
|
|
||||||
|
### 4. Bank ↔ Dolibarr match (the headline)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bin/arcodange bank match --month 2026-03 # one month
|
||||||
|
bin/arcodange bank match --since 2026-01-01 --until 2026-05-31
|
||||||
|
bin/arcodange bank match --month 2026-03 --window-days 14 # looser date tolerance
|
||||||
|
bin/arcodange bank match --include-fees # include cashback / charges in matching
|
||||||
|
```
|
||||||
|
|
||||||
|
Exit 0 if every bank movement and every Dolibarr payment in the window pair up cleanly; exit 1 otherwise (with the unmatched entries surfaced).
|
||||||
|
|
||||||
|
Captured at [examples/bank-match-2026-01-to-05.txt](examples/bank-match-2026-01-to-05.txt) — the V1 baseline.
|
||||||
|
|
||||||
|
**Output buckets:**
|
||||||
|
- `MATCHED` — bank ↔ Dolibarr, with the date delta (`Δ+6d`) so you can see drift between bank settlement and Dolibarr saisie.
|
||||||
|
- `INTERNAL` — Wise↔Qonto consolidations auto-detected by equal-amount opposite-sign on close dates. Excluded from matching against Dolibarr (they're transfers between Arcodange's own accounts, not external operations).
|
||||||
|
- `BANK-ONLY` — bank movements with no Dolibarr counterpart. Each one is either (a) a missing supplier invoice or unrecorded incoming payment, or (b) a Wise platform fee / cashback that doesn't translate to a Dolibarr entry.
|
||||||
|
- `DOLIBARR-ONLY` — Dolibarr payments without a bank movement. Usually fk_account=3 (the CCA1 personal account, not API-visible) or the cancel-and-reissue avoir cycle (bank sees the net, Dolibarr sees the three-way breakdown).
|
||||||
|
|
||||||
|
**Known findings from the V1 baseline** (to raise during cohort review):
|
||||||
|
- **Wise 2026-05-29 +2147 € from Kissmetrics NOT in Dolibarr** — M4 invoice probably emitted but the payment hasn't been entered. Action: enter the payment in Dolibarr.
|
||||||
|
- **+1000 € FOUREZ Quentin on Qonto 2026-01-21** — unknown income, ask Gabriel.
|
||||||
|
- **MISTRAL.AI -172.68 €, CLAUDE.AI -180 €, URSSAF -493 € on Qonto** — bank-only expenses, missing supplier invoices.
|
||||||
|
- **fk_account=3 supplier payments** (~430 € cumul) — paid from G.RADUREAU CCA personal account, not visible via Qonto/Wise APIs. This is normal; documented gap.
|
||||||
|
|
||||||
|
### 5. Live balances
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bin/arcodange bank balance
|
||||||
|
```
|
||||||
|
|
||||||
|
Prints live balances per bank + the Dolibarr-side cumulative-payments-per-fk_account for cross-reference. Captured at [examples/bank-balance.txt](examples/bank-balance.txt).
|
||||||
|
|
||||||
|
Current state (V1 baseline):
|
||||||
|
- Qonto **Compte principal** : 4 191,54 € live
|
||||||
|
- Wise **STANDARD EUR** : 5 308,25 € live
|
||||||
|
- **Total bank-side** : 9 499,79 €
|
||||||
|
|
||||||
|
## Known-patterns catalog ([known-patterns.json](known-patterns.json))
|
||||||
|
|
||||||
|
Bank movements that have no Dolibarr counterpart fall into two groups:
|
||||||
|
|
||||||
|
1. **Intentional gaps** — operational expenses or one-off events the operator knows about (URSSAF mensuel, AI subs, capital deposit, Wise plan fees). These keep recurring but their accounting treatment is well-understood.
|
||||||
|
2. **Real action items** — incoming payments not yet entered, expenses missing a supplier invoice, anomalies.
|
||||||
|
|
||||||
|
Without a catalog, both look identical in the BANK-ONLY bucket — noise drowns the signal. The catalog is an operator-curated list of patterns; `bank-match.sh` reads it and splits BANK-ONLY into two sub-buckets:
|
||||||
|
|
||||||
|
- **BANK-ONLY — known patterns** : annotated with `[classification]` + a one-line note (which Dolibarr account to use, etc.). Don't action; just verify.
|
||||||
|
- **BANK-ONLY — unknown** : the real signal. Each entry is either a missing supplier invoice, an unrecorded payment, or a new pattern to add to the catalog.
|
||||||
|
|
||||||
|
**Schema** (JSON, see [known-patterns.json](known-patterns.json) for current entries):
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"patterns": [
|
||||||
|
{
|
||||||
|
"pattern": "regex (case-insensitive, matched against bank label + operation type)",
|
||||||
|
"classification": "capital_deposit | social_charges | ai_subscription | bank_fee | internal_topup | personal_apport | needs_classification",
|
||||||
|
"bank": "qonto | wise (optional, default both)",
|
||||||
|
"side": "credit | debit (optional, default both)",
|
||||||
|
"amount_min": 0.0, "amount_max": 99999.0, // optional numeric bounds
|
||||||
|
"note": "human-readable context — what this is, which Dolibarr account, recurring schedule, etc."
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Editing workflow:**
|
||||||
|
1. Run `bin/arcodange bank match` → look at BANK-ONLY unknown.
|
||||||
|
2. For each recurring entry that's "expected", add a pattern to `known-patterns.json`.
|
||||||
|
3. Re-run match → the entry should now appear in the known sub-bucket.
|
||||||
|
4. For one-off action items (e.g. "+2147 € KM May 29 not in Dolibarr"), don't add a pattern — enter it in Dolibarr instead.
|
||||||
|
|
||||||
|
**V6.1 catalog** ships with these patterns for the current Arcodange baseline:
|
||||||
|
- `FOUREZ Quentin` → capital_deposit (initial 1000 € apport via notaire, 2026-01-21)
|
||||||
|
- `URSSAF` → social_charges
|
||||||
|
- `MISTRAL.AI` / `CLAUDE.AI` → ai_subscription
|
||||||
|
- `Wise *Plan` → bank_fee (Wise account plan billed via Qonto card)
|
||||||
|
- `qonto_fee` → bank_fee
|
||||||
|
- `BALANCE_DEPOSIT|For your account plan` → internal_topup (the Wise +50/-50 self-funding pair)
|
||||||
|
|
||||||
|
After applying the catalog to the V6 baseline, the **only remaining BANK-UNKNOWN** is the **+2147 € KissMetrics payment on 2026-05-29** that hasn't been entered in Dolibarr — the actual signal.
|
||||||
|
|
||||||
|
## V7 bucket structure
|
||||||
|
|
||||||
|
V7 adds three improvements that reshape the output buckets:
|
||||||
|
|
||||||
|
| Bucket | Meaning | Counts toward exit-1? |
|
||||||
|
|---|---|---|
|
||||||
|
| **MATCHED** | Bank ↔ Dolibarr paired. Annotated with match kind: `[wire-ref]` (strong, via `--enrich`) or `[amt+date]` (loose). | No |
|
||||||
|
| **INTERNAL** | Wise↔Qonto consolidations (5000€ moved between Arcodange's own accounts). | No |
|
||||||
|
| **AVOIR-NETTED** | Dolibarr AVC + FAC cancellation cycles paired and excluded (the bank only saw the net). | No |
|
||||||
|
| **BANK-ONLY — known patterns** | Bank movement with a `known-patterns.json` annotation. Intentional gap. | No |
|
||||||
|
| **BANK-ONLY — unknown** | Bank movement with no Dolibarr counterpart AND no catalog pattern. **Real action item**. | Yes |
|
||||||
|
| **DOLIBARR-ONLY — on API-tracked accounts** (QON*/WIS*) | Dolibarr payment that the bank should have shown. **Real gap**. | Yes |
|
||||||
|
| **DOLIBARR-ONLY — not in API scope** (CCA1 perso etc.) | Expected gap — we have no API on those accounts. | No |
|
||||||
|
|
||||||
|
Exit code 0 iff the two "real gap" buckets are empty.
|
||||||
|
|
||||||
|
### `--enrich` — wire-reference strong matching
|
||||||
|
|
||||||
|
`bank-match.sh --enrich` fetches `/v1/transfers/{id}` for each Wise TRANSFER and reads the `reference` field (the wire memo from the sender, e.g. `FROM KISSMETRICS HOLDINGS INC FOR INVOICE FAC002CL0001002/ VENDOR:DEV`). When the reference contains a `FAC\d+(CL\d+)?` pattern matching a Dolibarr customer invoice, that pairing takes precedence over the loose date+amount match. Only the strong-matched ones get `[wire-ref]`; the rest fall through to `[amt+date]`. Cost: 1 extra HTTP call per Wise transfer.
|
||||||
|
|
||||||
|
### Avoir cycle netting
|
||||||
|
|
||||||
|
When Arcodange cancels and reissues an invoice (FAC001 → AVC001 + FAC001-NEW), the bank sees one net credit but Dolibarr stores 3 payment entries. V7 pairs AVC entries of -X with FAC entries of +X for the same socid within ±5d, surfaces them in **AVOIR-NETTED**, and excludes them from `dolibarr-only`. Removes the V6.1 noise where AVC001 + FAC001-CL00001 appeared as fake gaps.
|
||||||
|
|
||||||
|
### fk_account context
|
||||||
|
|
||||||
|
`bank-match.sh` now fetches `/bankaccounts` and tags `dolibarr-only` entries with their account ref + label. Splits into API-tracked (QON*/WIS* — real gaps) vs not-in-scope (everything else — expected). The 7 CCA1 personal-account entries that used to look like failures are now correctly classified as expected gaps.
|
||||||
|
|
||||||
|
### Effect on the baseline
|
||||||
|
|
||||||
|
| | V6 | V6.1 | V7 |
|
||||||
|
|---|---|---|---|
|
||||||
|
| MATCHED | 6 (all amt+date) | 6 | 6 (1 wire-ref strong + 5 amt+date when --enrich) |
|
||||||
|
| BANK-ONLY total | 8 mixed | 7 known + 1 UNKNOWN | 7 known + 1 UNKNOWN |
|
||||||
|
| AVOIR-NETTED | — | — | 2 (silently absorbed) |
|
||||||
|
| DOL-only TRUE GAP | 9 (noisy) | 9 (noisy) | **0** |
|
||||||
|
| DOL-only EXPECTED | — | — | 7 (CCA1 personal) |
|
||||||
|
| Exit-1 signal count | 17 (noise) | 10 (less noise) | **1** (just the +2147€ KM) |
|
||||||
|
|
||||||
|
## Matching heuristic — what's in v1 and what's V7
|
||||||
|
|
||||||
|
Today's match logic:
|
||||||
|
- **Amount equality** within 0.01 € (absolute value).
|
||||||
|
- **Date proximity** within `--window-days` (default 7 — covers most settlement drift).
|
||||||
|
- **Direction-aware** (bank credit ↔ Dolibarr customer payment; bank debit ↔ Dolibarr supplier payment).
|
||||||
|
- **Smallest date delta wins** when multiple candidates qualify.
|
||||||
|
- **Internal consolidation detection** by equal-amount opposite-sign cross-bank within ±3d.
|
||||||
|
|
||||||
|
V7 improvements to consider:
|
||||||
|
- **Reference-based matching** using the `--enrich` wire reference: search the Wise reference text for `FAC\d+` patterns and match by ref string. Stronger than date+amount when wire ref is informative.
|
||||||
|
- **Multi-row aggregation** for Dolibarr sub-payments: if invoice FAFXXX has two sub-payments on different dates summing to one bank movement, aggregate Dolibarr-side before matching.
|
||||||
|
- **Avoir cycle handling**: the V1 AVC001/FAC001-CL00001/FAC001-CL0001001 dance produces 3 Dolibarr-only rows for 1 bank credit. A smarter matcher would net the AVOIR before matching.
|
||||||
|
|
||||||
|
These are deliberately deferred — V1's accuracy is "good enough to surface the real issues" and the simple heuristic is easy to reason about.
|
||||||
|
|
||||||
|
## Out of scope
|
||||||
|
|
||||||
|
- **Writes**: the API tokens are read-only; payment recording happens in Dolibarr UI.
|
||||||
|
- **Wise BUSINESS balance statements via API** — region-blocked for EU personal tokens (documented above).
|
||||||
|
- **Wise transactions routed through Qonto** — Qonto's API doesn't expose them; the integration is UI-only.
|
||||||
|
- **fk_account=3 (CCA1 personal account)** — not API-accessible. Movements there must be reconciled manually against personal bank statements.
|
||||||
|
- **Bank statement export to CSV** — possible V8 fallback if any of the APIs go away; out of scope today.
|
||||||
|
- **Currency conversion** — everything is EUR. Multi-currency would need adapting the amount parser.
|
||||||
24
.claude/skills/arcodange-bank-reco/examples/bank-balance.txt
Normal file
24
.claude/skills/arcodange-bank-reco/examples/bank-balance.txt
Normal file
@@ -0,0 +1,24 @@
|
|||||||
|
==========================================================================================
|
||||||
|
BANK-SIDE balances (live)
|
||||||
|
==========================================================================================
|
||||||
|
bank ref currency balance details
|
||||||
|
--------------------------------------------------------------------------------------
|
||||||
|
Qonto Compte principal EUR 4191.54 iban=...25639215 id=019b93fe-3cb8-
|
||||||
|
Wise STANDARD EUR 5308.25 id=147831931
|
||||||
|
|
||||||
|
Qonto total: 4191.54 EUR
|
||||||
|
Wise total : 5308.25 EUR
|
||||||
|
|
||||||
|
==========================================================================================
|
||||||
|
DOLIBARR-SIDE cumulative payments per fk_account
|
||||||
|
==========================================================================================
|
||||||
|
fk_account ref label cust paid in sup paid out net
|
||||||
|
--------------------------------------------------------------------------------------
|
||||||
|
1 QON1 QONTO 0.00 968.00 -968.00
|
||||||
|
2 WIS2 WISE EURO 8160.00 0.00 8160.00
|
||||||
|
3 CCA1 G.RADUREAU Compte Courant Asso 0.00 429.75 -429.75
|
||||||
|
|
||||||
|
# Note: Dolibarr-side numbers are CUMULATIVE since the account started in Dolibarr,
|
||||||
|
# not the current bank balance. Mismatch with the bank-side is expected when
|
||||||
|
# the account predates Dolibarr or has movements not recorded in Dolibarr
|
||||||
|
# (e.g. URSSAF, AI subscriptions — see bank-match.sh BANK-ONLY bucket).
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
# Bank reconciliation: 2026-01-01 → 2026-05-31 (window ±7d, fees: off, enrich: on)
|
||||||
|
|
||||||
|
=== MATCHED (6 bank ↔ Dolibarr) ===
|
||||||
|
Qonto 2026-01-27 - 50.00 card Wise *Plan ↔[amt+date] supplier FAF2026001 (2026-01-26, Δ-1d)
|
||||||
|
Wise 2026-02-05 + 510.00 TRANSFER Kissmetrics Holdings Inc ↔[amt+date] customer FAC001-CL0001001 (2026-02-05, Δ+0d)
|
||||||
|
Wise 2026-03-06 + 5100.00 TRANSFER Kissmetrics Holdings Inc ↔[wire-ref] customer FAC002-CL0001002 (2026-03-12, Δ+6d)
|
||||||
|
Qonto 2026-03-13 - 612.00 transfer DARNIS OPERATIONS ↔[amt+date] supplier FAF2026008 (2026-03-13, Δ+0d)
|
||||||
|
Wise 2026-04-20 + 2550.00 TRANSFER Kissmetrics Holdings Inc ↔[amt+date] customer FAC003-CL0001003 (2026-04-20, Δ+0d)
|
||||||
|
Qonto 2026-05-10 - 306.00 transfer DARNIS OPERATIONS ↔[amt+date] supplier FAF2026009 (2026-05-10, Δ+0d)
|
||||||
|
|
||||||
|
=== INTERNAL (Wise↔Qonto consolidations, 1) ===
|
||||||
|
Wise 2026-03-13 - 5000.00 TRANSFER ARCODANGE ↔ Qonto 2026-03-13 +5000.00
|
||||||
|
|
||||||
|
=== AVOIR-NETTED (2 Dolibarr entries pairing AVC↔FAC cancellation cycles) ===
|
||||||
|
customer 2026-02-05 -510.00 AVC001-CL0001001 ↔ netted against FAC001-CL00001
|
||||||
|
customer 2026-02-05 510.00 FAC001-CL00001 ↔ netted against AVC001-CL0001001
|
||||||
|
|
||||||
|
=== BANK-ONLY — known patterns (7, intentional gaps documented in known-patterns.json) ===
|
||||||
|
Qonto 2026-01-16 + 5.22 qonto_fee Qonto [bank_fee]
|
||||||
|
└─ Qonto fees ou refunds. Petites valeurs. Dolibarr: account 627.
|
||||||
|
Qonto 2026-01-21 + 1000.00 income FOUREZ Quentin [capital_deposit]
|
||||||
|
└─ Apport en capital social initial 1000 €. Maître FOUREZ Quentin, notaire centralisateur du dépôt. Date typique : 2026-01-21. Dolibarr: account 1013.
|
||||||
|
Wise 2026-01-26 - 50.00 FEATURE_CHARGE For your account plan [internal_topup]
|
||||||
|
└─ Solde Wise rechargé pour couvrir un frais immédiat (souvent net zéro avec le FEATURE_CHARGE du même jour).
|
||||||
|
Wise 2026-01-26 + 50.00 BALANCE_DEPOSIT To EUR [internal_topup]
|
||||||
|
└─ Solde Wise rechargé pour couvrir un frais immédiat (souvent net zéro avec le FEATURE_CHARGE du même jour).
|
||||||
|
Qonto 2026-04-03 - 172.68 card MISTRAL.AI [ai_subscription]
|
||||||
|
└─ Mistral AI API subscription. Récurrent mensuel. Dolibarr: account 6262 + supplier 'Mistral AI'.
|
||||||
|
Qonto 2026-04-13 - 180.00 card CLAUDE.AI SUBSCRIPTION [ai_subscription]
|
||||||
|
└─ Claude AI subscription (Anthropic). Récurrent mensuel. Dolibarr: account 6262 + supplier 'Anthropic'.
|
||||||
|
Qonto 2026-05-22 - 493.00 direct_debit URSSAF D ILE DE FRANCE [social_charges]
|
||||||
|
└─ Cotisations sociales URSSAF (régime mensuel/trimestriel). Dolibarr: account 645100 (charges de sécurité sociale).
|
||||||
|
|
||||||
|
=== BANK-ONLY — unknown (1, NEEDS attention: missing supplier invoice / unrecorded payment / new pattern) ===
|
||||||
|
Wise 2026-05-29 + 2147.00 TRANSFER Kissmetrics Holdings Inc
|
||||||
|
|
||||||
|
=== DOLIBARR-ONLY — on API-tracked accounts (0, REAL GAP: bank should have shown this) ===
|
||||||
|
|
||||||
|
=== DOLIBARR-ONLY — on accounts NOT in API scope (7, expected gap: CCA1 perso etc.) ===
|
||||||
|
supplier 2026-01-04 1.99 FAF2026003 (CCA1 (G.RADUREAU Compte Courant Asso))
|
||||||
|
supplier 2026-01-06 202.80 FAF2026005 (CCA1 (G.RADUREAU Compte Courant Asso))
|
||||||
|
supplier 2026-01-09 55.93 FAF2026002 (CCA1 (G.RADUREAU Compte Courant Asso))
|
||||||
|
supplier 2026-01-09 148.80 FAF2026004 (CCA1 (G.RADUREAU Compte Courant Asso))
|
||||||
|
supplier 2026-01-12 8.43 FAF2026006 (CCA1 (G.RADUREAU Compte Courant Asso))
|
||||||
|
supplier 2026-01-15 1.30 FAF2026002 (CCA1 (G.RADUREAU Compte Courant Asso))
|
||||||
|
supplier 2026-01-17 3.20 FAF2026007 (CCA1 (G.RADUREAU Compte Courant Asso))
|
||||||
|
|
||||||
|
----------------------------------------------------------------------------------------------------
|
||||||
|
# 6 matched, 1 internal, 2 avoir-netted, 7 bank-known, 1 bank-UNKNOWN, 0 dol-only-API, 7 dol-only-personal
|
||||||
|
# patterns loaded from /Users/gabrielradureau/Work/Arcodange/erp/.claude/worktrees/happy-wilson-ee5645/.claude/skills/arcodange-bank-reco/scripts/../known-patterns.json: 7 pattern(s)
|
||||||
37
.claude/skills/arcodange-bank-reco/examples/bank-probe.txt
Normal file
37
.claude/skills/arcodange-bank-reco/examples/bank-probe.txt
Normal file
@@ -0,0 +1,37 @@
|
|||||||
|
================================================================================
|
||||||
|
bank-probe — auth + discovery
|
||||||
|
================================================================================
|
||||||
|
|
||||||
|
--- QONTO ---
|
||||||
|
base : https://thirdparty.qonto.com
|
||||||
|
auth shape : Authorization: <login>:<secret>
|
||||||
|
env vars : QONTO_LOGIN=<set> QONTO_SECRET_KEY=<set> QONTO_ORG_SLUG=arcodange-1246
|
||||||
|
|
||||||
|
[OK] auth succeeded
|
||||||
|
slug : arcodange-1246
|
||||||
|
legal_name : ARCODANGE
|
||||||
|
legal_country : FR
|
||||||
|
1 bank account(s):
|
||||||
|
id=019b93fe-3cb8-7e28-9404-ee637f7aff27 name=Compte principal ...iban=25639215 status=active balance=4191.54
|
||||||
|
|
||||||
|
--- WISE ---
|
||||||
|
base : https://api.wise.com
|
||||||
|
auth shape : Authorization: Bearer <token>
|
||||||
|
env vars : WISE_API_TOKEN=<set> WISE_PROFILE_ID=82958299
|
||||||
|
|
||||||
|
[OK] auth succeeded
|
||||||
|
2 profile(s):
|
||||||
|
id=82958299 type=BUSINESS name=ARCODANGE ← .env WISE_PROFILE_ID
|
||||||
|
id=82958415 type=PERSONAL name=Gabriel Jonathan Marc Radureau
|
||||||
|
|
||||||
|
--- WISE balances (for the BUSINESS profile) ---
|
||||||
|
1 balance(s):
|
||||||
|
id=147831931 type=STANDARD currency=EUR balance=5308.25 EUR name=None
|
||||||
|
|
||||||
|
--- summary ---
|
||||||
|
.env should ultimately contain:
|
||||||
|
QONTO_LOGIN=<set>
|
||||||
|
QONTO_SECRET_KEY=<set>
|
||||||
|
QONTO_ORG_SLUG=arcodange-1246
|
||||||
|
WISE_API_TOKEN=<set>
|
||||||
|
WISE_PROFILE_ID=<the BUSINESS id from above>
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
# Qonto transactions on account 019b93fe... (2026-01-01 → 2026-05-31)
|
||||||
|
|
||||||
|
date side amount cur op label
|
||||||
|
----------------------------------------------------------------------------------------------------
|
||||||
|
2026-01-16 credit 5.22 EUR qonto_fee Qonto
|
||||||
|
2026-01-21 credit 1000.00 EUR income FOUREZ Quentin
|
||||||
|
2026-01-27 debit 50.00 EUR card Wise *Plan
|
||||||
|
2026-03-13 credit 5000.00 EUR income ARCODANGE
|
||||||
|
2026-03-13 debit 612.00 EUR transfer DARNIS OPERATIONS
|
||||||
|
2026-04-03 debit 172.68 EUR card MISTRAL.AI
|
||||||
|
2026-04-13 debit 180.00 EUR card CLAUDE.AI SUBSCRIPTION
|
||||||
|
2026-05-10 debit 306.00 EUR transfer DARNIS OPERATIONS
|
||||||
|
2026-05-22 debit 493.00 EUR direct_debit URSSAF D ILE DE FRANCE
|
||||||
|
----------------------------------------------------------------------------------------------------
|
||||||
|
# 9 txn(s) — credit total: 6005.22, debit total: 1813.68, net: +4191.54
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
# Wise activities for profile (window 2026-01-01 → 2026-05-31)
|
||||||
|
|
||||||
|
date type status sign amount cur title
|
||||||
|
-------------------------------------------------------------------------------------------
|
||||||
|
2026-01-26 BALANCE_DEPOSIT COMPLETED + 50.00 EUR To EUR
|
||||||
|
2026-01-26 FEATURE_CHARGE COMPLETED - 50.00 EUR For your account plan
|
||||||
|
2026-02-05 TRANSFER COMPLETED + 510.00 EUR Kissmetrics Holdings Inc
|
||||||
|
2026-03-05 BALANCE_CASHBACK COMPLETED + 0.19 EUR Cashback
|
||||||
|
2026-03-06 TRANSFER COMPLETED + 5100.00 EUR Kissmetrics Holdings Inc
|
||||||
|
2026-03-13 TRANSFER COMPLETED - 5000.00 EUR ARCODANGE
|
||||||
|
2026-04-06 BALANCE_CASHBACK COMPLETED + 0.35 EUR Cashback
|
||||||
|
2026-04-20 TRANSFER COMPLETED + 2550.00 EUR Kissmetrics Holdings Inc
|
||||||
|
2026-05-07 BALANCE_CASHBACK COMPLETED + 0.71 EUR Cashback
|
||||||
|
2026-05-29 TRANSFER COMPLETED + 2147.00 EUR Kissmetrics Holdings Inc
|
||||||
|
-------------------------------------------------------------------------------------------
|
||||||
|
# 10 activity(ies) — credit: +10358.25, debit: -5050.00, net: +5308.25
|
||||||
60
.claude/skills/arcodange-bank-reco/known-patterns.json
Normal file
60
.claude/skills/arcodange-bank-reco/known-patterns.json
Normal file
@@ -0,0 +1,60 @@
|
|||||||
|
{
|
||||||
|
"_schema": "v1",
|
||||||
|
"_description": "Operator-curated catalogue of known recurring/intentional bank movements. Used by bank-match.sh to annotate the BANK-ONLY bucket so the operator can immediately tell 'needs Dolibarr entry' from 'documented intentional gap'. Edit this file as new recurring patterns emerge.",
|
||||||
|
"_match_rules": "Pattern matched case-insensitively as a regex against the bank label. Optional filters: bank (qonto|wise), side (credit|debit), amount_min, amount_max, type (Wise activity type). All present filters must match.",
|
||||||
|
"_classifications": {
|
||||||
|
"capital_deposit": "Apport en capital social. Dolibarr account 1013 (capital souscrit appelé versé).",
|
||||||
|
"social_charges": "URSSAF, retraite complémentaire, etc. Dolibarr account 645x.",
|
||||||
|
"ai_subscription": "Claude / Mistral / OpenAI / similar. Dolibarr account 6262 (frais télécom / abonnements logiciels).",
|
||||||
|
"bank_fee": "Plan bancaire, frais d'opération, refunds. Dolibarr account 627 (services bancaires).",
|
||||||
|
"internal_topup": "Solde Wise/Qonto rechargé pour couvrir un frais immédiat. Often nets out.",
|
||||||
|
"personal_apport": "Apport en compte courant d'associé (Gabriel finançant Arcodange depuis son perso). Dolibarr account 4551.",
|
||||||
|
"needs_classification": "Pattern catched but no Dolibarr account assignment defined yet; surface for review."
|
||||||
|
},
|
||||||
|
"patterns": [
|
||||||
|
{
|
||||||
|
"pattern": "FOUREZ.*Quentin",
|
||||||
|
"classification": "capital_deposit",
|
||||||
|
"bank": "qonto",
|
||||||
|
"side": "credit",
|
||||||
|
"note": "Apport en capital social initial 1000 €. Maître FOUREZ Quentin, notaire centralisateur du dépôt. Date typique : 2026-01-21. Dolibarr: account 1013."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"pattern": "URSSAF",
|
||||||
|
"classification": "social_charges",
|
||||||
|
"bank": "qonto",
|
||||||
|
"side": "debit",
|
||||||
|
"note": "Cotisations sociales URSSAF (régime mensuel/trimestriel). Dolibarr: account 645100 (charges de sécurité sociale)."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"pattern": "MISTRAL\\.AI",
|
||||||
|
"classification": "ai_subscription",
|
||||||
|
"side": "debit",
|
||||||
|
"note": "Mistral AI API subscription. Récurrent mensuel. Dolibarr: account 6262 + supplier 'Mistral AI'."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"pattern": "CLAUDE\\.AI",
|
||||||
|
"classification": "ai_subscription",
|
||||||
|
"side": "debit",
|
||||||
|
"note": "Claude AI subscription (Anthropic). Récurrent mensuel. Dolibarr: account 6262 + supplier 'Anthropic'."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"pattern": "Wise.*Plan",
|
||||||
|
"classification": "bank_fee",
|
||||||
|
"side": "debit",
|
||||||
|
"note": "Wise account plan billed via card. Wise's internal fee for keeping the BUSINESS profile active."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"pattern": "qonto_fee",
|
||||||
|
"classification": "bank_fee",
|
||||||
|
"bank": "qonto",
|
||||||
|
"note": "Qonto fees ou refunds. Petites valeurs. Dolibarr: account 627."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"pattern": "BALANCE_DEPOSIT|For your account plan",
|
||||||
|
"classification": "internal_topup",
|
||||||
|
"bank": "wise",
|
||||||
|
"note": "Solde Wise rechargé pour couvrir un frais immédiat (souvent net zéro avec le FEATURE_CHARGE du même jour)."
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
108
.claude/skills/arcodange-bank-reco/scripts/bank-balance.sh
Executable file
108
.claude/skills/arcodange-bank-reco/scripts/bank-balance.sh
Executable file
@@ -0,0 +1,108 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Live balances per bank account, plus a Dolibarr cross-check by fk_account.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# bank-balance.sh
|
||||||
|
#
|
||||||
|
# Outputs each bank account (Qonto + Wise) with:
|
||||||
|
# - current live balance (from each bank's API)
|
||||||
|
# - sum of all Dolibarr payments touching it (per fk_account)
|
||||||
|
#
|
||||||
|
# The sum-of-payments isn't the bank balance — it's just the cumulative
|
||||||
|
# operations Dolibarr has recorded on that account. Useful as a sanity check
|
||||||
|
# (e.g. "did I record everything?"). The bank-side current balance is
|
||||||
|
# authoritative.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
BANK_CURL="${SCRIPT_DIR}/bank-curl.sh"
|
||||||
|
DOL_CURL="${SCRIPT_DIR}/../../dolibarr/scripts/dol-curl.sh"
|
||||||
|
|
||||||
|
set -a; source "${SCRIPT_DIR}/../../dolibarr/.env"; set +a
|
||||||
|
: "${WISE_PROFILE_ID:?bank-balance.sh: WISE_PROFILE_ID not set}"
|
||||||
|
|
||||||
|
WORK="$(mktemp -d -t bankbal.XXXXXX)"
|
||||||
|
trap 'rm -rf "${WORK}"' EXIT
|
||||||
|
|
||||||
|
# Bank side
|
||||||
|
"${BANK_CURL}" qonto /v2/organization > "${WORK}/qonto.json"
|
||||||
|
"${BANK_CURL}" wise "/v4/profiles/${WISE_PROFILE_ID}/balances?types=STANDARD" > "${WORK}/wise.json"
|
||||||
|
|
||||||
|
# Dolibarr side: bank accounts + payments per invoice
|
||||||
|
"${DOL_CURL}" /bankaccounts > "${WORK}/dol_acct.json"
|
||||||
|
"${DOL_CURL}" '/invoices?limit=500' > "${WORK}/dol_inv.json"
|
||||||
|
"${DOL_CURL}" '/supplierinvoices?limit=500' > "${WORK}/dol_sup.json"
|
||||||
|
|
||||||
|
mkdir -p "${WORK}/dol_pay" "${WORK}/dol_supay"
|
||||||
|
for id in $(python3 -c "import json,sys; print(' '.join(str(r['id']) for r in json.load(open(sys.argv[1])) if r.get('id')))" "${WORK}/dol_inv.json"); do
|
||||||
|
"${DOL_CURL}" "/invoices/${id}/payments" > "${WORK}/dol_pay/${id}.json" 2>/dev/null || echo "[]" > "${WORK}/dol_pay/${id}.json"
|
||||||
|
done
|
||||||
|
for id in $(python3 -c "import json,sys; print(' '.join(str(r['id']) for r in json.load(open(sys.argv[1])) if r.get('id')))" "${WORK}/dol_sup.json"); do
|
||||||
|
"${DOL_CURL}" "/supplierinvoices/${id}/payments" > "${WORK}/dol_supay/${id}.json" 2>/dev/null || echo "[]" > "${WORK}/dol_supay/${id}.json"
|
||||||
|
done
|
||||||
|
|
||||||
|
python3 - "${WORK}" <<'PY'
|
||||||
|
import json, os, sys, collections
|
||||||
|
work = sys.argv[1]
|
||||||
|
|
||||||
|
# Bank-side balances
|
||||||
|
q = json.load(open(os.path.join(work, "qonto.json")))
|
||||||
|
qonto_accs = (q.get("organization") or {}).get("bank_accounts") or []
|
||||||
|
wise_bals = json.load(open(os.path.join(work, "wise.json")))
|
||||||
|
|
||||||
|
print("=" * 90)
|
||||||
|
print(" BANK-SIDE balances (live)")
|
||||||
|
print("=" * 90)
|
||||||
|
print(f" {'bank':<8} {'ref':<20} {'currency':<3} {'balance':>12} details")
|
||||||
|
print(" " + "-" * 86)
|
||||||
|
qonto_total = 0.0
|
||||||
|
for a in qonto_accs:
|
||||||
|
bal = float(a.get("balance") or 0); qonto_total += bal
|
||||||
|
iban = (a.get("iban") or "")[-8:]
|
||||||
|
print(f" {'Qonto':<8} {a.get('name','-')[:20]:<20} {a.get('balance_cents_currency','EUR'):<3} {bal:>12.2f} iban=...{iban} id={a.get('id','-')[:14]}")
|
||||||
|
for b in wise_bals:
|
||||||
|
amt = (b.get("amount") or {})
|
||||||
|
print(f" {'Wise':<8} {b.get('type','-'):<20} {amt.get('currency','-'):<3} {float(amt.get('value') or 0):>12.2f} id={b.get('id','-')}")
|
||||||
|
print()
|
||||||
|
print(f" Qonto total: {qonto_total:.2f} EUR")
|
||||||
|
print(f" Wise total : {sum(float((b.get('amount') or {}).get('value') or 0) for b in wise_bals):.2f} EUR")
|
||||||
|
print()
|
||||||
|
|
||||||
|
# Dolibarr-side: sum of all payments per fk_account
|
||||||
|
dol_accts = {str(a["id"]): a for a in json.load(open(os.path.join(work, "dol_acct.json")))}
|
||||||
|
sums = collections.defaultdict(lambda: {"customer": 0.0, "supplier": 0.0})
|
||||||
|
|
||||||
|
inv = {str(r["id"]): r for r in json.load(open(os.path.join(work, "dol_inv.json")))}
|
||||||
|
for fn in os.listdir(os.path.join(work, "dol_pay")):
|
||||||
|
iid = fn[:-5]; i = inv.get(iid)
|
||||||
|
if not i: continue
|
||||||
|
fka = str(i.get("fk_account") or "?")
|
||||||
|
for p in json.load(open(os.path.join(work, "dol_pay", fn))):
|
||||||
|
sums[fka]["customer"] += float(p.get("amount") or 0)
|
||||||
|
|
||||||
|
sup = {str(r["id"]): r for r in json.load(open(os.path.join(work, "dol_sup.json")))}
|
||||||
|
for fn in os.listdir(os.path.join(work, "dol_supay")):
|
||||||
|
iid = fn[:-5]; s = sup.get(iid)
|
||||||
|
if not s: continue
|
||||||
|
fka = str(s.get("fk_account") or "?")
|
||||||
|
for p in json.load(open(os.path.join(work, "dol_supay", fn))):
|
||||||
|
sums[fka]["supplier"] += float(p.get("amount") or 0)
|
||||||
|
|
||||||
|
print("=" * 90)
|
||||||
|
print(" DOLIBARR-SIDE cumulative payments per fk_account")
|
||||||
|
print("=" * 90)
|
||||||
|
print(f" {'fk_account':<10} {'ref':<10} {'label':<30} {'cust paid in':>12} {'sup paid out':>12} {'net':>10}")
|
||||||
|
print(" " + "-" * 86)
|
||||||
|
for fka in sorted(sums, key=lambda k: int(k) if k.isdigit() else 99):
|
||||||
|
s = sums[fka]
|
||||||
|
net = s["customer"] - s["supplier"]
|
||||||
|
a = dol_accts.get(fka, {})
|
||||||
|
ref = a.get("ref","-")[:10]; label = (a.get("label") or "-")[:30]
|
||||||
|
print(f" {fka:<10} {ref:<10} {label:<30} {s['customer']:>12.2f} {s['supplier']:>12.2f} {net:>10.2f}")
|
||||||
|
print()
|
||||||
|
print("# Note: Dolibarr-side numbers are CUMULATIVE since the account started in Dolibarr,")
|
||||||
|
print("# not the current bank balance. Mismatch with the bank-side is expected when")
|
||||||
|
print("# the account predates Dolibarr or has movements not recorded in Dolibarr")
|
||||||
|
print("# (e.g. URSSAF, AI subscriptions — see bank-match.sh BANK-ONLY bucket).")
|
||||||
|
PY
|
||||||
107
.claude/skills/arcodange-bank-reco/scripts/bank-curl.sh
Executable file
107
.claude/skills/arcodange-bank-reco/scripts/bank-curl.sh
Executable file
@@ -0,0 +1,107 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Read-only curl wrapper for the two banks Arcodange uses: Qonto + Wise.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# bank-curl.sh qonto <path> # e.g. bank-curl.sh qonto /v2/organization
|
||||||
|
# bank-curl.sh wise <path> # e.g. bank-curl.sh wise /v2/profiles
|
||||||
|
# bank-curl.sh -i <bank> <path> # include curl's -i (response headers)
|
||||||
|
#
|
||||||
|
# Reads credentials from ../../dolibarr/.env (the shared canonical file
|
||||||
|
# used by every dolibarr-* and arcodange-* skill). Required vars:
|
||||||
|
# QONTO_LOGIN, QONTO_SECRET_KEY (for qonto)
|
||||||
|
# WISE_API_TOKEN (for wise)
|
||||||
|
#
|
||||||
|
# Exits non-zero on HTTP >=400 and writes the body to stdout + a short
|
||||||
|
# "bank-curl.sh: HTTP <code>" message to stderr — same shape as dol-curl.sh.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
ENV_FILE="${SCRIPT_DIR}/../../dolibarr/.env"
|
||||||
|
|
||||||
|
if [[ ! -f "${ENV_FILE}" ]]; then
|
||||||
|
echo "bank-curl.sh: missing ${ENV_FILE}" >&2
|
||||||
|
echo " See dolibarr/README.md for the .env schema; arcodange-bank-reco extends it" >&2
|
||||||
|
echo " with QONTO_LOGIN, QONTO_SECRET_KEY, WISE_API_TOKEN, WISE_PROFILE_ID." >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
set -a; source "${ENV_FILE}"; set +a
|
||||||
|
|
||||||
|
PASSTHRU=()
|
||||||
|
while [[ $# -gt 2 ]]; do
|
||||||
|
PASSTHRU+=("$1"); shift
|
||||||
|
done
|
||||||
|
|
||||||
|
if [[ $# -lt 2 ]]; then
|
||||||
|
echo "bank-curl.sh: usage: bank-curl.sh [curl-opts] <qonto|wise> <api-path>" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
BANK="$1"; API_PATH="$2"
|
||||||
|
|
||||||
|
case "${BANK}" in
|
||||||
|
qonto)
|
||||||
|
: "${QONTO_LOGIN:?bank-curl.sh: QONTO_LOGIN not set in .env}"
|
||||||
|
: "${QONTO_SECRET_KEY:?bank-curl.sh: QONTO_SECRET_KEY not set in .env}"
|
||||||
|
BASE="https://thirdparty.qonto.com"
|
||||||
|
AUTH_HEADER="Authorization: ${QONTO_LOGIN}:${QONTO_SECRET_KEY}"
|
||||||
|
;;
|
||||||
|
wise)
|
||||||
|
: "${WISE_API_TOKEN:?bank-curl.sh: WISE_API_TOKEN not set in .env}"
|
||||||
|
BASE="https://api.wise.com"
|
||||||
|
AUTH_HEADER="Authorization: Bearer ${WISE_API_TOKEN}"
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "bank-curl.sh: unknown bank '${BANK}' (use qonto or wise)" >&2
|
||||||
|
exit 2
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
BODY_FILE="$(mktemp -t bankcurl.XXXXXX)"
|
||||||
|
HEADERS_FILE="$(mktemp -t bankcurlhdr.XXXXXX)"
|
||||||
|
trap 'rm -f "${BODY_FILE}" "${HEADERS_FILE}"' EXIT
|
||||||
|
|
||||||
|
do_call() {
|
||||||
|
local extra_header_1="${1:-}" extra_header_2="${2:-}"
|
||||||
|
local extra_args=()
|
||||||
|
[[ -n "${extra_header_1}" ]] && extra_args+=("-H" "${extra_header_1}")
|
||||||
|
[[ -n "${extra_header_2}" ]] && extra_args+=("-H" "${extra_header_2}")
|
||||||
|
curl -sS \
|
||||||
|
-H "${AUTH_HEADER}" \
|
||||||
|
-H "Accept: application/json" \
|
||||||
|
--max-time 30 \
|
||||||
|
-o "${BODY_FILE}" \
|
||||||
|
-D "${HEADERS_FILE}" \
|
||||||
|
-w "%{http_code}" \
|
||||||
|
${PASSTHRU[@]+"${PASSTHRU[@]}"} \
|
||||||
|
${extra_args[@]+"${extra_args[@]}"} \
|
||||||
|
"${BASE}${API_PATH}"
|
||||||
|
}
|
||||||
|
|
||||||
|
HTTP_CODE=$(do_call)
|
||||||
|
|
||||||
|
# Wise SCA flow: a 403 with x-2fa-approval-result: REJECTED + x-2fa-approval: <token>
|
||||||
|
# header means the endpoint is sensitive and the call must be signed with our
|
||||||
|
# registered RSA private key. We sign the one-time token and retry.
|
||||||
|
if [[ "${BANK}" == "wise" && "${HTTP_CODE}" == "403" ]]; then
|
||||||
|
ONE_TIME=$(awk 'tolower($1) == "x-2fa-approval:" { gsub(/[\r\n]/, "", $2); print $2; exit }' "${HEADERS_FILE}")
|
||||||
|
if [[ -n "${ONE_TIME}" ]]; then
|
||||||
|
KEY_PATH="${WISE_SCA_KEY_PATH:-}"
|
||||||
|
# Expand ~ if used
|
||||||
|
KEY_PATH="${KEY_PATH/#\~/$HOME}"
|
||||||
|
if [[ -z "${KEY_PATH}" || ! -f "${KEY_PATH}" ]]; then
|
||||||
|
echo "bank-curl.sh: Wise endpoint requires SCA but WISE_SCA_KEY_PATH is missing." >&2
|
||||||
|
echo " Generate a keypair, upload the public key to Wise, set WISE_SCA_KEY_PATH in .env." >&2
|
||||||
|
echo " See arcodange-bank-reco/SKILL.md for the setup steps." >&2
|
||||||
|
cat "${BODY_FILE}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
SIGNATURE=$(printf '%s' "${ONE_TIME}" | openssl dgst -sha256 -sign "${KEY_PATH}" | base64 | tr -d '\n')
|
||||||
|
HTTP_CODE=$(do_call "x-2fa-approval: ${ONE_TIME}" "X-Signature: ${SIGNATURE}")
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
cat "${BODY_FILE}"
|
||||||
|
if [[ "${HTTP_CODE}" -ge 400 ]]; then
|
||||||
|
echo "bank-curl.sh: HTTP ${HTTP_CODE} on ${BANK}${API_PATH}" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
363
.claude/skills/arcodange-bank-reco/scripts/bank-match.sh
Executable file
363
.claude/skills/arcodange-bank-reco/scripts/bank-match.sh
Executable file
@@ -0,0 +1,363 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Match bank movements (Qonto + Wise) against Dolibarr payments.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# bank-match.sh [--month YYYY-MM | --since YYYY-MM-DD --until YYYY-MM-DD]
|
||||||
|
# [--window-days N] # date tolerance, default 7
|
||||||
|
# [--include-fees] # include Wise cashback / charges (default off)
|
||||||
|
#
|
||||||
|
# Output: three buckets
|
||||||
|
# - MATCHED bank movement ↔ Dolibarr payment
|
||||||
|
# - BANK-ONLY bank movement with no Dolibarr counterpart (potential
|
||||||
|
# missing supplier invoice or unrecorded incoming payment)
|
||||||
|
# - DOLIBARR-ONLY Dolibarr payment with no bank movement (timing or error)
|
||||||
|
#
|
||||||
|
# Internal Wise↔Qonto consolidations (e.g. 5000 € moved Wise→Qonto same day)
|
||||||
|
# are auto-detected and excluded from matching against Dolibarr.
|
||||||
|
#
|
||||||
|
# Exit 0 if everything in the window matches cleanly, 1 if there's any bank-only
|
||||||
|
# or dolibarr-only entry.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
BANK_CURL="${SCRIPT_DIR}/bank-curl.sh"
|
||||||
|
DOL_CURL="${SCRIPT_DIR}/../../dolibarr/scripts/dol-curl.sh"
|
||||||
|
|
||||||
|
SINCE=""; UNTIL=""; MONTH=""; WINDOW=7; INCLUDE_FEES=0; ENRICH=0
|
||||||
|
while [[ $# -gt 0 ]]; do
|
||||||
|
case "$1" in
|
||||||
|
--since) SINCE="$2"; shift 2 ;;
|
||||||
|
--until) UNTIL="$2"; shift 2 ;;
|
||||||
|
--month) MONTH="$2"; shift 2 ;;
|
||||||
|
--window-days) WINDOW="$2"; shift 2 ;;
|
||||||
|
--include-fees) INCLUDE_FEES=1; shift ;;
|
||||||
|
--enrich) ENRICH=1; shift ;;
|
||||||
|
-h|--help) sed -n '2,18p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||||
|
*) echo "bank-match.sh: unknown arg: $1" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
if [[ -n "${MONTH}" ]]; then
|
||||||
|
SINCE="${MONTH}-01"
|
||||||
|
UNTIL="$(python3 -c "import calendar; y,m=map(int,'${MONTH}'.split('-')); print(f'{y:04d}-{m:02d}-{calendar.monthrange(y,m)[1]:02d}')")"
|
||||||
|
fi
|
||||||
|
[[ -z "${SINCE}" ]] && SINCE="$(python3 -c "import datetime; print((datetime.date.today()-datetime.timedelta(days=365)).strftime('%Y-%m-%d'))")"
|
||||||
|
[[ -z "${UNTIL}" ]] && UNTIL="$(python3 -c "import datetime; print(datetime.date.today().strftime('%Y-%m-%d'))")"
|
||||||
|
|
||||||
|
set -a; source "${SCRIPT_DIR}/../../dolibarr/.env"; set +a
|
||||||
|
: "${WISE_PROFILE_ID:?bank-match.sh: WISE_PROFILE_ID not set}"
|
||||||
|
|
||||||
|
WORK="$(mktemp -d -t bankmatch.XXXXXX)"
|
||||||
|
trap 'rm -rf "${WORK}"' EXIT
|
||||||
|
|
||||||
|
# --- 1. Pull Qonto transactions ---
|
||||||
|
TMP_ORG=$(mktemp -t qontoorg.XXXXXX.json)
|
||||||
|
"${BANK_CURL}" qonto /v2/organization > "${TMP_ORG}"
|
||||||
|
QONTO_ACCT=$(python3 -c "
|
||||||
|
import json, sys
|
||||||
|
d = json.load(open(sys.argv[1]))
|
||||||
|
accs = (d.get('organization') or {}).get('bank_accounts') or []
|
||||||
|
print([a for a in accs if a.get('status')=='active'][0]['id'])" "${TMP_ORG}")
|
||||||
|
rm -f "${TMP_ORG}"
|
||||||
|
|
||||||
|
QURL="/v2/transactions?bank_account_id=${QONTO_ACCT}&settled_at_from=${SINCE}T00:00:00Z&settled_at_to=${UNTIL}T23:59:59Z&per_page=100"
|
||||||
|
"${BANK_CURL}" qonto "${QURL}" > "${WORK}/qonto.json"
|
||||||
|
|
||||||
|
# --- 2. Pull Wise activities ---
|
||||||
|
"${BANK_CURL}" wise "/v1/profiles/${WISE_PROFILE_ID}/activities?size=100&since=${SINCE}T00:00:00.000Z&until=${UNTIL}T23:59:59.999Z" > "${WORK}/wise.json"
|
||||||
|
|
||||||
|
# --- 3. Pull Dolibarr customer + supplier invoices, payments, and bank accounts ---
|
||||||
|
"${DOL_CURL}" '/invoices?limit=500&sortfield=t.datef&sortorder=ASC' > "${WORK}/dol_inv.json"
|
||||||
|
"${DOL_CURL}" '/supplierinvoices?limit=500' > "${WORK}/dol_sup.json"
|
||||||
|
"${DOL_CURL}" '/bankaccounts' > "${WORK}/dol_acct.json"
|
||||||
|
|
||||||
|
mkdir -p "${WORK}/dol_pay" "${WORK}/dol_supay"
|
||||||
|
for id in $(python3 -c "import json,sys; print(' '.join(str(r['id']) for r in json.load(open(sys.argv[1])) if r.get('id')))" "${WORK}/dol_inv.json"); do
|
||||||
|
"${DOL_CURL}" "/invoices/${id}/payments" > "${WORK}/dol_pay/${id}.json" 2>/dev/null || echo "[]" > "${WORK}/dol_pay/${id}.json"
|
||||||
|
done
|
||||||
|
for id in $(python3 -c "import json,sys; print(' '.join(str(r['id']) for r in json.load(open(sys.argv[1])) if r.get('id')))" "${WORK}/dol_sup.json"); do
|
||||||
|
"${DOL_CURL}" "/supplierinvoices/${id}/payments" > "${WORK}/dol_supay/${id}.json" 2>/dev/null || echo "[]" > "${WORK}/dol_supay/${id}.json"
|
||||||
|
done
|
||||||
|
|
||||||
|
# --- 3b. Optional: enrich Wise TRANSFER activities with wire references ---
|
||||||
|
if [[ "${ENRICH}" == "1" ]]; then
|
||||||
|
mkdir -p "${WORK}/wise_refs"
|
||||||
|
for tid in $(python3 -c "
|
||||||
|
import json, sys
|
||||||
|
acts = json.load(open(sys.argv[1])).get('activities') or []
|
||||||
|
for a in acts:
|
||||||
|
r = a.get('resource') or {}
|
||||||
|
if r.get('type')=='TRANSFER' and r.get('id'): print(r['id'])
|
||||||
|
" "${WORK}/wise.json"); do
|
||||||
|
"${BANK_CURL}" wise "/v1/transfers/${tid}" > "${WORK}/wise_refs/${tid}.json" 2>/dev/null || true
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
|
# --- 4. Match in python ---
|
||||||
|
PATTERNS_FILE="${SCRIPT_DIR}/../known-patterns.json"
|
||||||
|
python3 - "${WORK}" "${SINCE}" "${UNTIL}" "${WINDOW}" "${INCLUDE_FEES}" "${PATTERNS_FILE}" "${ENRICH}" <<'PY'
|
||||||
|
import json, sys, os, re, datetime, collections
|
||||||
|
work, since, until, window_days, include_fees, patterns_file, enrich = sys.argv[1:8]
|
||||||
|
window = int(window_days); include_fees = include_fees == "1"; enrich = enrich == "1"
|
||||||
|
since_d = datetime.date.fromisoformat(since); until_d = datetime.date.fromisoformat(until)
|
||||||
|
|
||||||
|
def strip(s): return re.sub(r'<[^>]+>', '', s or '').strip()
|
||||||
|
|
||||||
|
# 4a. Normalize Qonto
|
||||||
|
qonto_movs = []
|
||||||
|
for t in (json.load(open(os.path.join(work,"qonto.json"))).get("transactions") or []):
|
||||||
|
dt = datetime.date.fromisoformat((t.get("settled_at") or "")[:10])
|
||||||
|
if dt < since_d or dt > until_d: continue
|
||||||
|
amt = float(t.get("amount") or 0)
|
||||||
|
sign = "+" if t.get("side") == "credit" else "-"
|
||||||
|
label = t.get("label") or t.get("operation_type") or "-"
|
||||||
|
qonto_movs.append({"bank":"Qonto", "date":dt, "sign":sign, "amount":amt, "label":label[:40], "op":t.get("operation_type",""), "matched_dol":None, "matched_internal":False})
|
||||||
|
|
||||||
|
# 4b. Normalize Wise
|
||||||
|
wise_movs = []
|
||||||
|
for a in (json.load(open(os.path.join(work,"wise.json"))).get("activities") or []):
|
||||||
|
dt = datetime.date.fromisoformat((a.get("createdOn") or "")[:10])
|
||||||
|
if dt < since_d or dt > until_d: continue
|
||||||
|
typ = a.get("type","-")
|
||||||
|
if not include_fees and typ in ("BALANCE_CASHBACK", "BALANCE_INTEREST"):
|
||||||
|
continue
|
||||||
|
pa = strip(a.get("primaryAmount") or "")
|
||||||
|
sign = "+" if pa.startswith("+") else "-"
|
||||||
|
m = re.search(r'([\d,.]+)\s*([A-Z]{3})', pa)
|
||||||
|
amt = float(m.group(1).replace(",", "")) if m else 0.0
|
||||||
|
title = strip(a.get("title") or "")[:40]
|
||||||
|
res = a.get("resource") or {}
|
||||||
|
resource_id = str(res.get("id")) if res.get("type") == "TRANSFER" else None
|
||||||
|
wise_movs.append({"bank":"Wise", "date":dt, "sign":sign, "amount":amt, "label":title, "op":typ, "matched_dol":None, "matched_internal":False, "wise_resource_id":resource_id, "wire_ref":""})
|
||||||
|
|
||||||
|
# 4b'. If --enrich, load per-transfer wire references and attach to Wise movs
|
||||||
|
if enrich:
|
||||||
|
ref_dir = os.path.join(work, "wise_refs")
|
||||||
|
if os.path.isdir(ref_dir):
|
||||||
|
for m in wise_movs:
|
||||||
|
if not m["wise_resource_id"]: continue
|
||||||
|
p = os.path.join(ref_dir, f"{m['wise_resource_id']}.json")
|
||||||
|
if not os.path.isfile(p): continue
|
||||||
|
try:
|
||||||
|
t = json.load(open(p))
|
||||||
|
m["wire_ref"] = (t.get("reference") or "")
|
||||||
|
except Exception: pass
|
||||||
|
|
||||||
|
bank_movs = qonto_movs + wise_movs
|
||||||
|
|
||||||
|
# 4c. Detect internal Wise<->Qonto consolidations: same date, equal amount, opposite signs, one Wise + one Qonto
|
||||||
|
for w in [m for m in bank_movs if m["bank"]=="Wise" and m["sign"]=="-"]:
|
||||||
|
for q in [m for m in bank_movs if m["bank"]=="Qonto" and m["sign"]=="+" and not m["matched_internal"]]:
|
||||||
|
if abs(w["amount"] - q["amount"]) < 0.01 and abs((w["date"] - q["date"]).days) <= 3:
|
||||||
|
w["matched_internal"] = q; q["matched_internal"] = w
|
||||||
|
break
|
||||||
|
|
||||||
|
# 4d. Normalize Dolibarr payments — carry socid too for avoir netting
|
||||||
|
dol_pays = []
|
||||||
|
inv_by_id = {str(r["id"]): r for r in json.load(open(os.path.join(work,"dol_inv.json")))}
|
||||||
|
for fn in os.listdir(os.path.join(work,"dol_pay")):
|
||||||
|
iid = fn[:-5]; inv = inv_by_id.get(iid)
|
||||||
|
if not inv: continue
|
||||||
|
for p in json.load(open(os.path.join(work,"dol_pay",fn))):
|
||||||
|
d = datetime.datetime.strptime(p["date"], "%Y-%m-%d %H:%M:%S").date()
|
||||||
|
if d < since_d or d > until_d: continue
|
||||||
|
amt = float(p.get("amount") or 0)
|
||||||
|
dol_pays.append({"side":"customer", "ref":inv["ref"], "date":d, "amount":amt,
|
||||||
|
"fk_account":inv.get("fk_account"), "socid":inv.get("socid"),
|
||||||
|
"matched_bank":None, "netted_against":None})
|
||||||
|
|
||||||
|
sup_by_id = {str(r["id"]): r for r in json.load(open(os.path.join(work,"dol_sup.json")))}
|
||||||
|
for fn in os.listdir(os.path.join(work,"dol_supay")):
|
||||||
|
iid = fn[:-5]; sup = sup_by_id.get(iid)
|
||||||
|
if not sup: continue
|
||||||
|
for p in json.load(open(os.path.join(work,"dol_supay",fn))):
|
||||||
|
d = datetime.datetime.strptime(p["date"], "%Y-%m-%d %H:%M:%S").date()
|
||||||
|
if d < since_d or d > until_d: continue
|
||||||
|
amt = float(p.get("amount") or 0)
|
||||||
|
dol_pays.append({"side":"supplier", "ref":sup["ref"], "date":d, "amount":amt,
|
||||||
|
"fk_account":sup.get("fk_account"), "socid":sup.get("socid"),
|
||||||
|
"matched_bank":None, "netted_against":None})
|
||||||
|
|
||||||
|
# 4d.1. AVOIR cycle netting: an AVC (credit note) for -X on socid S cancels out
|
||||||
|
# a FAC for +X on the same socid, within a small date window. Bank sees the NET
|
||||||
|
# of the cycle (typically +X for the reissued FAC with the new ref scheme).
|
||||||
|
# Pair an AVC with a FAC of opposite sign + equal abs(amount) + same socid +
|
||||||
|
# within ±5d. Mark both as "netted" so they're excluded from matching and
|
||||||
|
# excluded from the dolibarr-only failure count.
|
||||||
|
avcs = [p for p in dol_pays if p["side"]=="customer" and p["ref"].startswith("AVC") and p["amount"] < 0]
|
||||||
|
for avc in avcs:
|
||||||
|
candidates = [p for p in dol_pays
|
||||||
|
if p is not avc
|
||||||
|
and p["side"]=="customer"
|
||||||
|
and p["socid"] == avc["socid"]
|
||||||
|
and abs(p["amount"] + avc["amount"]) < 0.01 # opposite signs equal magnitude
|
||||||
|
and abs((p["date"] - avc["date"]).days) <= 5
|
||||||
|
and p["netted_against"] is None
|
||||||
|
and p["matched_bank"] is None]
|
||||||
|
if candidates:
|
||||||
|
# Prefer the OLDEST (the original cancelled FAC), not the reissue.
|
||||||
|
# Heuristic: refs with shorter / older numbering scheme. If multiple,
|
||||||
|
# pick smallest date delta.
|
||||||
|
candidates.sort(key=lambda p: (abs((p["date"] - avc["date"]).days), p["ref"]))
|
||||||
|
partner = candidates[0]
|
||||||
|
avc["netted_against"] = partner["ref"]
|
||||||
|
partner["netted_against"] = avc["ref"]
|
||||||
|
|
||||||
|
# 4e. Match — two-pass:
|
||||||
|
# PASS 1 (strong) : Wise transfers with an --enrich'd wire reference containing
|
||||||
|
# a "FAC***" pattern try to match the Dolibarr invoice with
|
||||||
|
# that exact ref. This is the highest-confidence match.
|
||||||
|
# PASS 2 (loose) : remaining bank movements use the date+amount heuristic.
|
||||||
|
# Netted Dolibarr entries (avoir cycle) are excluded from both passes.
|
||||||
|
|
||||||
|
# Build customer ref -> dol payment index (only un-netted, un-matched entries)
|
||||||
|
ref_index = collections.defaultdict(list)
|
||||||
|
for p in dol_pays:
|
||||||
|
if p["matched_bank"] is None and p["netted_against"] is None:
|
||||||
|
# Strip trailing dash/suffix variants — FAC002CL0001002 vs FAC002-CL0001002 are equivalent
|
||||||
|
normalized = re.sub(r'[^A-Z0-9]', '', p["ref"].upper())
|
||||||
|
ref_index[normalized].append(p)
|
||||||
|
|
||||||
|
# Pass 1: strong match on wire references
|
||||||
|
for m in [x for x in bank_movs if not x["matched_internal"] and x.get("wire_ref")]:
|
||||||
|
refs_in_wire = re.findall(r'FAC\d+(?:CL\d+)?', (m["wire_ref"] or "").upper().replace("-",""))
|
||||||
|
for r in refs_in_wire:
|
||||||
|
if r in ref_index and ref_index[r]:
|
||||||
|
p = ref_index[r].pop(0)
|
||||||
|
m["matched_dol"] = p; m["match_kind"] = "wire-ref"
|
||||||
|
p["matched_bank"] = m
|
||||||
|
break
|
||||||
|
|
||||||
|
# Pass 2: loose date+amount match for remaining bank movements
|
||||||
|
for m in [x for x in bank_movs if not x["matched_internal"] and not x["matched_dol"]]:
|
||||||
|
candidates = [p for p in dol_pays
|
||||||
|
if p["matched_bank"] is None and p["netted_against"] is None
|
||||||
|
and abs(p["amount"] - m["amount"]) < 0.01
|
||||||
|
and abs((p["date"] - m["date"]).days) <= window]
|
||||||
|
if candidates:
|
||||||
|
candidates.sort(key=lambda p: abs((p["date"] - m["date"]).days))
|
||||||
|
p = candidates[0]
|
||||||
|
m["matched_dol"] = p; m["match_kind"] = "amt+date"
|
||||||
|
p["matched_bank"] = m
|
||||||
|
|
||||||
|
# 4f. Annotate non-matched movements with known-patterns catalog
|
||||||
|
patterns = []
|
||||||
|
if os.path.isfile(patterns_file):
|
||||||
|
try: patterns = json.load(open(patterns_file)).get("patterns", [])
|
||||||
|
except Exception as e: print(f" /!\\ failed to load {patterns_file}: {e}", file=sys.stderr)
|
||||||
|
|
||||||
|
def match_pattern(mov):
|
||||||
|
# Match against both the bank label AND the operation type — different
|
||||||
|
# banks surface useful info in different fields (Qonto puts the operation
|
||||||
|
# type in `op`, e.g. "qonto_fee"; Wise puts the activity type in `op`,
|
||||||
|
# e.g. "BALANCE_DEPOSIT", and the human title in `label`).
|
||||||
|
haystack = (mov.get("label") or "") + " | " + (mov.get("op") or "")
|
||||||
|
for pat in patterns:
|
||||||
|
if pat.get("bank") and pat["bank"] != mov["bank"].lower(): continue
|
||||||
|
if pat.get("side") and pat["side"] != ("credit" if mov["sign"]=="+" else "debit"): continue
|
||||||
|
amin = pat.get("amount_min"); amax = pat.get("amount_max")
|
||||||
|
if amin is not None and mov["amount"] < amin: continue
|
||||||
|
if amax is not None and mov["amount"] > amax: continue
|
||||||
|
if re.search(pat["pattern"], haystack, re.IGNORECASE):
|
||||||
|
return pat
|
||||||
|
return None
|
||||||
|
|
||||||
|
for m in bank_movs:
|
||||||
|
if m["matched_dol"] or m["matched_internal"]: continue
|
||||||
|
m["known"] = match_pattern(m)
|
||||||
|
|
||||||
|
# --- 5. Render ---
|
||||||
|
|
||||||
|
# Load Dolibarr bank accounts (for fk_account context on dolibarr-only)
|
||||||
|
dol_accts = {}
|
||||||
|
try:
|
||||||
|
for a in json.load(open(os.path.join(work, "dol_acct.json"))):
|
||||||
|
dol_accts[str(a["id"])] = {"ref": a.get("ref","-"), "label": a.get("label","-"), "country": a.get("country_code","")}
|
||||||
|
except Exception: pass
|
||||||
|
|
||||||
|
# Heuristic: which Dolibarr accounts are NOT covered by Qonto/Wise API today?
|
||||||
|
# Convention: CCA = Compte Courant d'Associé (personal). Anything not QON*/WIS*
|
||||||
|
# is treated as "API-invisible" and tagged as such.
|
||||||
|
def account_kind(fk_account):
|
||||||
|
if not fk_account: return ("unknown", "fk_account=None")
|
||||||
|
a = dol_accts.get(str(fk_account))
|
||||||
|
if not a: return ("unknown", f"fk_account={fk_account} (not in /bankaccounts)")
|
||||||
|
ref = (a["ref"] or "").upper()
|
||||||
|
if ref.startswith(("QON", "WIS")):
|
||||||
|
return ("api_tracked", f"{a['ref']} ({a['label']})")
|
||||||
|
return ("personal_or_other", f"{a['ref']} ({a['label']})")
|
||||||
|
|
||||||
|
def fmt_bank(m):
|
||||||
|
return f" {m['bank']:<5} {m['date']} {m['sign']:<2}{m['amount']:>9.2f} {m['op'][:18]:<18} {m['label']}"
|
||||||
|
|
||||||
|
print(f"# Bank reconciliation: {since} → {until} (window ±{window}d, fees: {'on' if include_fees else 'off'}, enrich: {'on' if enrich else 'off'})")
|
||||||
|
print()
|
||||||
|
matched = [m for m in bank_movs if m["matched_dol"]]
|
||||||
|
internal = [m for m in bank_movs if m["matched_internal"] and m["sign"]=="-"]
|
||||||
|
bank_only = [m for m in bank_movs if not m["matched_dol"] and not m["matched_internal"]]
|
||||||
|
netted_dol_pairs = [p for p in dol_pays if p["netted_against"]]
|
||||||
|
dol_only = [p for p in dol_pays if p["matched_bank"] is None and p["netted_against"] is None]
|
||||||
|
|
||||||
|
print(f"=== MATCHED ({len(matched)} bank ↔ Dolibarr) ===")
|
||||||
|
for m in sorted(matched, key=lambda m: m["date"]):
|
||||||
|
p = m["matched_dol"]
|
||||||
|
delta = (p["date"] - m["date"]).days
|
||||||
|
kind = m.get("match_kind", "?")
|
||||||
|
print(fmt_bank(m) + f" ↔[{kind}] {p['side']:<8} {p['ref']:<24} ({p['date']}, Δ{delta:+d}d)")
|
||||||
|
print()
|
||||||
|
|
||||||
|
print(f"=== INTERNAL (Wise↔Qonto consolidations, {len(internal)}) ===")
|
||||||
|
for m in sorted(internal, key=lambda m: m["date"]):
|
||||||
|
other = m["matched_internal"]
|
||||||
|
print(fmt_bank(m) + f" ↔ {other['bank']} {other['date']} {other['sign']}{other['amount']:.2f}")
|
||||||
|
print()
|
||||||
|
|
||||||
|
# Avoir cycles netted out (informational; bank correctly sees only the net)
|
||||||
|
if netted_dol_pairs:
|
||||||
|
print(f"=== AVOIR-NETTED ({len(netted_dol_pairs)} Dolibarr entries pairing AVC↔FAC cancellation cycles) ===")
|
||||||
|
for p in sorted(netted_dol_pairs, key=lambda p: (p["date"], p["ref"])):
|
||||||
|
print(f" {p['side']:<8} {p['date']} {p['amount']:>9.2f} {p['ref']:<24} ↔ netted against {p['netted_against']}")
|
||||||
|
print()
|
||||||
|
|
||||||
|
bank_known = [m for m in bank_only if m.get("known")]
|
||||||
|
bank_unknown = [m for m in bank_only if not m.get("known")]
|
||||||
|
|
||||||
|
print(f"=== BANK-ONLY — known patterns ({len(bank_known)}, intentional gaps documented in known-patterns.json) ===")
|
||||||
|
for m in sorted(bank_known, key=lambda m: m["date"]):
|
||||||
|
k = m["known"]
|
||||||
|
cls = k.get("classification","?")
|
||||||
|
print(fmt_bank(m) + f" [{cls}]")
|
||||||
|
print(f" └─ {k.get('note','')}")
|
||||||
|
print()
|
||||||
|
|
||||||
|
print(f"=== BANK-ONLY — unknown ({len(bank_unknown)}, NEEDS attention: missing supplier invoice / unrecorded payment / new pattern) ===")
|
||||||
|
for m in sorted(bank_unknown, key=lambda m: m["date"]):
|
||||||
|
print(fmt_bank(m))
|
||||||
|
print()
|
||||||
|
|
||||||
|
# Split dolibarr-only by whether the fk_account is API-tracked (real gap)
|
||||||
|
# or personal_or_other (expected gap — we have no API on those accounts)
|
||||||
|
dol_only_api = [p for p in dol_only if account_kind(p["fk_account"])[0] == "api_tracked"]
|
||||||
|
dol_only_personal = [p for p in dol_only if account_kind(p["fk_account"])[0] != "api_tracked"]
|
||||||
|
|
||||||
|
print(f"=== DOLIBARR-ONLY — on API-tracked accounts ({len(dol_only_api)}, REAL GAP: bank should have shown this) ===")
|
||||||
|
for p in sorted(dol_only_api, key=lambda p: p["date"]):
|
||||||
|
_, ctx = account_kind(p["fk_account"])
|
||||||
|
print(f" {p['side']:<8} {p['date']} {p['amount']:>9.2f} {p['ref']:<24} ({ctx})")
|
||||||
|
print()
|
||||||
|
|
||||||
|
print(f"=== DOLIBARR-ONLY — on accounts NOT in API scope ({len(dol_only_personal)}, expected gap: CCA1 perso etc.) ===")
|
||||||
|
for p in sorted(dol_only_personal, key=lambda p: p["date"]):
|
||||||
|
_, ctx = account_kind(p["fk_account"])
|
||||||
|
print(f" {p['side']:<8} {p['date']} {p['amount']:>9.2f} {p['ref']:<24} ({ctx})")
|
||||||
|
print()
|
||||||
|
|
||||||
|
# Verdict: only UNKNOWN bank-only AND dol-only-on-API-tracked count as failures.
|
||||||
|
# Avoir-netted pairs and personal-account dolibarr entries are intentional/expected.
|
||||||
|
fails = len(bank_unknown) + len(dol_only_api)
|
||||||
|
print("-" * 100)
|
||||||
|
print(f"# {len(matched)} matched, {len(internal)} internal, {len(netted_dol_pairs)} avoir-netted, {len(bank_known)} bank-known, {len(bank_unknown)} bank-UNKNOWN, {len(dol_only_api)} dol-only-API, {len(dol_only_personal)} dol-only-personal")
|
||||||
|
print(f"# patterns loaded from {patterns_file}: {len(patterns)} pattern(s)")
|
||||||
|
sys.exit(0 if fails == 0 else 1)
|
||||||
|
PY
|
||||||
140
.claude/skills/arcodange-bank-reco/scripts/bank-probe.sh
Executable file
140
.claude/skills/arcodange-bank-reco/scripts/bank-probe.sh
Executable file
@@ -0,0 +1,140 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Auth + discovery probe for Qonto + Wise. Run this once after dropping
|
||||||
|
# fresh tokens into .env. It confirms auth works and prints the IDs you
|
||||||
|
# need (Qonto bank account ids, Wise business profile id + balance ids).
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# bank-probe.sh
|
||||||
|
#
|
||||||
|
# Token values are NEVER printed. Only metadata + slugs / ids are.
|
||||||
|
# Output is safe to commit as an examples/ baseline.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
BANK_CURL="${SCRIPT_DIR}/bank-curl.sh"
|
||||||
|
|
||||||
|
set -a; source "${SCRIPT_DIR}/../../dolibarr/.env"; set +a
|
||||||
|
|
||||||
|
# A redactor that masks anything that looks like a token / secret / key
|
||||||
|
# in case the API ever echoes credentials back to us. Defense in depth.
|
||||||
|
redact() {
|
||||||
|
python3 - <<'PY'
|
||||||
|
import sys, re
|
||||||
|
s = sys.stdin.read()
|
||||||
|
# Mask long alphanumeric runs (typical tokens) and any explicit "token"/"secret"/"key" values
|
||||||
|
s = re.sub(r'([A-Fa-f0-9]{32,})', '<REDACTED-HEX>', s)
|
||||||
|
s = re.sub(r'([A-Za-z0-9_-]{40,})', '<REDACTED-TOKEN>', s)
|
||||||
|
s = re.sub(r'("(?:token|secret|key|password)"\s*:\s*")[^"]*(")',
|
||||||
|
r'\1<REDACTED>\2', s, flags=re.IGNORECASE)
|
||||||
|
print(s, end='')
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
|
||||||
|
echo "================================================================================"
|
||||||
|
echo " bank-probe — auth + discovery"
|
||||||
|
echo "================================================================================"
|
||||||
|
echo
|
||||||
|
|
||||||
|
# ---------------------------- Qonto ----------------------------
|
||||||
|
echo "--- QONTO ---"
|
||||||
|
echo " base : https://thirdparty.qonto.com"
|
||||||
|
echo " auth shape : Authorization: <login>:<secret>"
|
||||||
|
echo " env vars : QONTO_LOGIN=${QONTO_LOGIN:+<set>} QONTO_SECRET_KEY=${QONTO_SECRET_KEY:+<set>} QONTO_ORG_SLUG=${QONTO_ORG_SLUG:-<unset>}"
|
||||||
|
echo
|
||||||
|
|
||||||
|
if QONTO_ORG_RAW="$("${BANK_CURL}" qonto /v2/organization 2>&1)"; then
|
||||||
|
python3 - <<PY
|
||||||
|
import json
|
||||||
|
d = json.loads(${QONTO_ORG_RAW@Q}) if False else json.loads("""${QONTO_ORG_RAW}""")
|
||||||
|
PY
|
||||||
|
fi 2>/dev/null || true
|
||||||
|
|
||||||
|
# Robust version: write to tmp file, parse from there (avoids quoting hell)
|
||||||
|
TMP_QONTO=$(mktemp -t qonto.XXXXXX.json)
|
||||||
|
trap 'rm -f "${TMP_QONTO}"' EXIT
|
||||||
|
if "${BANK_CURL}" qonto /v2/organization > "${TMP_QONTO}"; then
|
||||||
|
python3 - "${TMP_QONTO}" <<'PY'
|
||||||
|
import json, sys
|
||||||
|
d = json.load(open(sys.argv[1]))
|
||||||
|
org = d.get("organization") or {}
|
||||||
|
print(f" [OK] auth succeeded")
|
||||||
|
print(f" slug : {org.get('slug')}")
|
||||||
|
print(f" legal_name : {org.get('legal_name')}")
|
||||||
|
print(f" legal_country : {org.get('legal_country')}")
|
||||||
|
accs = org.get("bank_accounts") or []
|
||||||
|
print(f" {len(accs)} bank account(s):")
|
||||||
|
for a in accs:
|
||||||
|
iban = a.get("iban") or ""
|
||||||
|
iban_tail = iban[-8:] if iban else "-"
|
||||||
|
print(f" id={a.get('id')} name={a.get('name','-')} ...iban={iban_tail} status={a.get('status','-')} balance={a.get('balance','-')} {a.get('balance_cents_currency','')}")
|
||||||
|
PY
|
||||||
|
else
|
||||||
|
echo " [XX] Qonto auth FAILED — check QONTO_LOGIN / QONTO_SECRET_KEY in .env"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo
|
||||||
|
# ---------------------------- Wise ----------------------------
|
||||||
|
echo "--- WISE ---"
|
||||||
|
echo " base : https://api.wise.com"
|
||||||
|
echo " auth shape : Authorization: Bearer <token>"
|
||||||
|
echo " env vars : WISE_API_TOKEN=${WISE_API_TOKEN:+<set>} WISE_PROFILE_ID=${WISE_PROFILE_ID:-<unset>}"
|
||||||
|
echo
|
||||||
|
|
||||||
|
TMP_WISE_PROFILES=$(mktemp -t wise.XXXXXX.json)
|
||||||
|
trap 'rm -f "${TMP_QONTO}" "${TMP_WISE_PROFILES}"' EXIT
|
||||||
|
if "${BANK_CURL}" wise /v2/profiles > "${TMP_WISE_PROFILES}"; then
|
||||||
|
python3 - "${TMP_WISE_PROFILES}" "${WISE_PROFILE_ID:-}" <<'PY'
|
||||||
|
import json, sys
|
||||||
|
profiles = json.load(open(sys.argv[1]))
|
||||||
|
env_pid = sys.argv[2]
|
||||||
|
print(f" [OK] auth succeeded")
|
||||||
|
print(f" {len(profiles)} profile(s):")
|
||||||
|
business_pid = None
|
||||||
|
for p in profiles:
|
||||||
|
name = p.get("fullName") or p.get("businessName") or (p.get("details") or {}).get("name", "-")
|
||||||
|
marker = ""
|
||||||
|
if str(p.get("id")) == env_pid:
|
||||||
|
marker = " ← .env WISE_PROFILE_ID"
|
||||||
|
if p.get("type") == "BUSINESS" and not business_pid:
|
||||||
|
business_pid = p.get("id")
|
||||||
|
print(f" id={p.get('id')} type={p.get('type','-'):<10} name={name}{marker}")
|
||||||
|
if env_pid and business_pid and str(business_pid) != env_pid:
|
||||||
|
print(f" [!!] WISE_PROFILE_ID in .env ({env_pid}) is NOT the BUSINESS profile ({business_pid}).")
|
||||||
|
print(f" Use the BUSINESS one for Arcodange.")
|
||||||
|
elif not env_pid and business_pid:
|
||||||
|
print(f" [→] Set WISE_PROFILE_ID={business_pid} in .env (BUSINESS profile).")
|
||||||
|
PY
|
||||||
|
else
|
||||||
|
echo " [XX] Wise auth FAILED — check WISE_API_TOKEN in .env"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo
|
||||||
|
# Wise balances + balance ids (needed for /balance-statements queries)
|
||||||
|
if [[ -n "${WISE_PROFILE_ID:-}" ]]; then
|
||||||
|
echo "--- WISE balances (for the BUSINESS profile) ---"
|
||||||
|
TMP_WISE_BAL=$(mktemp -t wisebal.XXXXXX.json)
|
||||||
|
trap 'rm -f "${TMP_QONTO}" "${TMP_WISE_PROFILES}" "${TMP_WISE_BAL}"' EXIT
|
||||||
|
if "${BANK_CURL}" wise "/v4/profiles/${WISE_PROFILE_ID}/balances?types=STANDARD" > "${TMP_WISE_BAL}"; then
|
||||||
|
python3 - "${TMP_WISE_BAL}" <<'PY'
|
||||||
|
import json, sys
|
||||||
|
bal = json.load(open(sys.argv[1]))
|
||||||
|
print(f" {len(bal)} balance(s):")
|
||||||
|
for b in bal:
|
||||||
|
amt = (b.get('amount') or {}).get('value', '?')
|
||||||
|
cur = (b.get('amount') or {}).get('currency', '?')
|
||||||
|
print(f" id={b.get('id')} type={b.get('type','-'):<8} currency={cur:<3} balance={amt} {cur} name={b.get('name','-')}")
|
||||||
|
PY
|
||||||
|
else
|
||||||
|
echo " [XX] /balances fetch failed — token may not have balances scope, or profile id is wrong."
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "--- summary ---"
|
||||||
|
echo " .env should ultimately contain:"
|
||||||
|
echo " QONTO_LOGIN=<set>"
|
||||||
|
echo " QONTO_SECRET_KEY=<set>"
|
||||||
|
echo " QONTO_ORG_SLUG=$(grep -E '^QONTO_ORG_SLUG' "${SCRIPT_DIR}/../../dolibarr/.env" | cut -d= -f2 || echo '<from probe>')"
|
||||||
|
echo " WISE_API_TOKEN=<set>"
|
||||||
|
echo " WISE_PROFILE_ID=<the BUSINESS id from above>"
|
||||||
116
.claude/skills/arcodange-bank-reco/scripts/qonto-transactions.sh
Executable file
116
.claude/skills/arcodange-bank-reco/scripts/qonto-transactions.sh
Executable file
@@ -0,0 +1,116 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# List Qonto transactions for a period, on one bank account.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# qonto-transactions.sh [--since YYYY-MM-DD] [--until YYYY-MM-DD]
|
||||||
|
# [--account <id>] # default: first active account
|
||||||
|
# [--month YYYY-MM]
|
||||||
|
# [--side credit|debit] # filter
|
||||||
|
# [--json] # raw JSON, no table
|
||||||
|
#
|
||||||
|
# Output: a compact table — date | side | amount | currency | op type | label.
|
||||||
|
# Pagination is followed automatically.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
BANK_CURL="${SCRIPT_DIR}/bank-curl.sh"
|
||||||
|
|
||||||
|
SINCE=""; UNTIL=""; MONTH=""; ACCOUNT=""; SIDE=""; FMT="table"
|
||||||
|
while [[ $# -gt 0 ]]; do
|
||||||
|
case "$1" in
|
||||||
|
--since) SINCE="$2"; shift 2 ;;
|
||||||
|
--until) UNTIL="$2"; shift 2 ;;
|
||||||
|
--month) MONTH="$2"; shift 2 ;;
|
||||||
|
--account) ACCOUNT="$2"; shift 2 ;;
|
||||||
|
--side) SIDE="$2"; shift 2 ;;
|
||||||
|
--json) FMT="json"; shift ;;
|
||||||
|
-h|--help) sed -n '2,12p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||||
|
*) echo "qonto-transactions.sh: unknown arg: $1" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
# Default window: --month overrides --since/--until; otherwise last 90 days.
|
||||||
|
if [[ -n "${MONTH}" ]]; then
|
||||||
|
SINCE="${MONTH}-01"
|
||||||
|
# Last day of month — works for any month using python
|
||||||
|
UNTIL="$(python3 -c "import datetime,calendar; y,m=map(int,'${MONTH}'.split('-')); print(f'{y:04d}-{m:02d}-{calendar.monthrange(y,m)[1]:02d}')")"
|
||||||
|
fi
|
||||||
|
[[ -z "${SINCE}" ]] && SINCE="$(python3 -c "import datetime; print((datetime.date.today()-datetime.timedelta(days=90)).strftime('%Y-%m-%d'))")"
|
||||||
|
[[ -z "${UNTIL}" ]] && UNTIL="$(python3 -c "import datetime; print(datetime.date.today().strftime('%Y-%m-%d'))")"
|
||||||
|
|
||||||
|
# Discover default account if not specified
|
||||||
|
if [[ -z "${ACCOUNT}" ]]; then
|
||||||
|
TMP_ORG=$(mktemp -t qontoorg.XXXXXX.json)
|
||||||
|
trap 'rm -f "${TMP_ORG}"' EXIT
|
||||||
|
"${BANK_CURL}" qonto /v2/organization > "${TMP_ORG}"
|
||||||
|
ACCOUNT=$(python3 -c "
|
||||||
|
import json, sys
|
||||||
|
d = json.load(open(sys.argv[1]))
|
||||||
|
accs = (d.get('organization') or {}).get('bank_accounts') or []
|
||||||
|
active = [a for a in accs if a.get('status') == 'active']
|
||||||
|
if not active: sys.exit('no active account')
|
||||||
|
print(active[0]['id'])
|
||||||
|
" "${TMP_ORG}")
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Paginate
|
||||||
|
TMP_ALL=$(mktemp -t qontoall.XXXXXX.json)
|
||||||
|
trap 'rm -f "${TMP_ORG:-/dev/null}" "${TMP_ALL}"' EXIT
|
||||||
|
echo '[]' > "${TMP_ALL}"
|
||||||
|
|
||||||
|
page=1
|
||||||
|
while :; do
|
||||||
|
TMP_PAGE=$(mktemp -t qontop.XXXXXX.json)
|
||||||
|
Q="bank_account_id=${ACCOUNT}&settled_at_from=${SINCE}T00:00:00Z&settled_at_to=${UNTIL}T23:59:59Z&per_page=100¤t_page=${page}"
|
||||||
|
"${BANK_CURL}" qonto "/v2/transactions?${Q}" > "${TMP_PAGE}"
|
||||||
|
# Merge transactions into the all-array
|
||||||
|
python3 - "${TMP_ALL}" "${TMP_PAGE}" <<'PY'
|
||||||
|
import json, sys
|
||||||
|
all_arr = json.load(open(sys.argv[1]))
|
||||||
|
page = json.load(open(sys.argv[2]))
|
||||||
|
all_arr.extend(page.get("transactions") or [])
|
||||||
|
json.dump(all_arr, open(sys.argv[1], "w"))
|
||||||
|
PY
|
||||||
|
NEXT=$(python3 -c "
|
||||||
|
import json, sys
|
||||||
|
d = json.load(open(sys.argv[1]))
|
||||||
|
m = d.get('meta') or {}
|
||||||
|
print(m.get('next_page') or '')
|
||||||
|
" "${TMP_PAGE}")
|
||||||
|
rm -f "${TMP_PAGE}"
|
||||||
|
[[ -z "${NEXT}" || "${NEXT}" == "None" ]] && break
|
||||||
|
page=$((page+1))
|
||||||
|
done
|
||||||
|
|
||||||
|
if [[ "${FMT}" == "json" ]]; then
|
||||||
|
cat "${TMP_ALL}"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
python3 - "${TMP_ALL}" "${SIDE}" "${SINCE}" "${UNTIL}" "${ACCOUNT}" <<'PY'
|
||||||
|
import json, sys
|
||||||
|
txs = json.load(open(sys.argv[1]))
|
||||||
|
side_filter, since, until, account = sys.argv[2:6]
|
||||||
|
if side_filter:
|
||||||
|
txs = [t for t in txs if t.get("side") == side_filter]
|
||||||
|
txs.sort(key=lambda t: t.get("settled_at") or "")
|
||||||
|
|
||||||
|
print(f"# Qonto transactions on account {account[:8]}... ({since} → {until})")
|
||||||
|
print()
|
||||||
|
print(f"{'date':<10} {'side':<6} {'amount':>10} {'cur':<3} {'op':<14} {'label':<40}")
|
||||||
|
print("-" * 100)
|
||||||
|
total_credit = total_debit = 0.0
|
||||||
|
for t in txs:
|
||||||
|
dt = (t.get("settled_at") or "")[:10]
|
||||||
|
amt = float(t.get("amount") or 0)
|
||||||
|
side = t.get("side") or "-"
|
||||||
|
op = (t.get("operation_type") or "-")[:14]
|
||||||
|
label = (t.get("label") or "-")[:40]
|
||||||
|
cur = t.get("currency") or "-"
|
||||||
|
print(f"{dt:<10} {side:<6} {amt:>10.2f} {cur:<3} {op:<14} {label:<40}")
|
||||||
|
if side == "credit": total_credit += amt
|
||||||
|
if side == "debit": total_debit += amt
|
||||||
|
print("-" * 100)
|
||||||
|
print(f"# {len(txs)} txn(s) — credit total: {total_credit:.2f}, debit total: {total_debit:.2f}, net: {total_credit - total_debit:+.2f}")
|
||||||
|
PY
|
||||||
154
.claude/skills/arcodange-bank-reco/scripts/wise-transactions.sh
Executable file
154
.claude/skills/arcodange-bank-reco/scripts/wise-transactions.sh
Executable file
@@ -0,0 +1,154 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# List Wise activities (incoming + outgoing) for a period.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# wise-transactions.sh [--since YYYY-MM-DD] [--until YYYY-MM-DD]
|
||||||
|
# [--month YYYY-MM]
|
||||||
|
# [--type TRANSFER|CARD_ORDER|BALANCE_CASHBACK|FEATURE_CHARGE|BALANCE_DEPOSIT|...]
|
||||||
|
# [--status COMPLETED|CANCELLED|IN_PROGRESS|UPCOMING|REQUIRES_ATTENTION]
|
||||||
|
# [--enrich] # also fetch /v1/transfers/{id} for the wire reference
|
||||||
|
# [--json] # raw list, no table
|
||||||
|
#
|
||||||
|
# Backed by GET /v1/profiles/{WISE_PROFILE_ID}/activities — does NOT require
|
||||||
|
# SCA and DOES expose incoming transfers (unlike /balance-statements which is
|
||||||
|
# region-restricted for EU personal tokens, see SKILL.md).
|
||||||
|
#
|
||||||
|
# Pagination is followed automatically via the cursor.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
BANK_CURL="${SCRIPT_DIR}/bank-curl.sh"
|
||||||
|
|
||||||
|
SINCE=""; UNTIL=""; MONTH=""; TYPE=""; STATUS=""; ENRICH=0; FMT="table"
|
||||||
|
while [[ $# -gt 0 ]]; do
|
||||||
|
case "$1" in
|
||||||
|
--since) SINCE="$2"; shift 2 ;;
|
||||||
|
--until) UNTIL="$2"; shift 2 ;;
|
||||||
|
--month) MONTH="$2"; shift 2 ;;
|
||||||
|
--type) TYPE="$2"; shift 2 ;;
|
||||||
|
--status) STATUS="$2"; shift 2 ;;
|
||||||
|
--enrich) ENRICH=1; shift ;;
|
||||||
|
--json) FMT="json"; shift ;;
|
||||||
|
-h|--help) sed -n '2,15p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||||
|
*) echo "wise-transactions.sh: unknown arg: $1" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
set -a; source "${SCRIPT_DIR}/../../dolibarr/.env"; set +a
|
||||||
|
: "${WISE_PROFILE_ID:?wise-transactions.sh: WISE_PROFILE_ID not set in .env}"
|
||||||
|
|
||||||
|
# Date handling — Wise wants full ISO 8601 with millis + Z
|
||||||
|
if [[ -n "${MONTH}" ]]; then
|
||||||
|
SINCE="${MONTH}-01"
|
||||||
|
UNTIL="$(python3 -c "import calendar; y,m=map(int,'${MONTH}'.split('-')); print(f'{y:04d}-{m:02d}-{calendar.monthrange(y,m)[1]:02d}')")"
|
||||||
|
fi
|
||||||
|
[[ -z "${SINCE}" ]] && SINCE="$(python3 -c "import datetime; print((datetime.date.today()-datetime.timedelta(days=365)).strftime('%Y-%m-%d'))")"
|
||||||
|
[[ -z "${UNTIL}" ]] && UNTIL="$(python3 -c "import datetime; print(datetime.date.today().strftime('%Y-%m-%d'))")"
|
||||||
|
|
||||||
|
SINCE_ISO="${SINCE}T00:00:00.000Z"
|
||||||
|
UNTIL_ISO="${UNTIL}T23:59:59.999Z"
|
||||||
|
|
||||||
|
WORK="$(mktemp -d -t wiseact.XXXXXX)"
|
||||||
|
trap 'rm -rf "${WORK}"' EXIT
|
||||||
|
|
||||||
|
# Paginate
|
||||||
|
echo '[]' > "${WORK}/all.json"
|
||||||
|
CURSOR=""
|
||||||
|
PAGE=1
|
||||||
|
while :; do
|
||||||
|
Q="size=100&since=${SINCE_ISO}&until=${UNTIL_ISO}"
|
||||||
|
[[ -n "${TYPE}" ]] && Q="${Q}&monetaryResourceType=${TYPE}"
|
||||||
|
[[ -n "${STATUS}" ]] && Q="${Q}&status=${STATUS}"
|
||||||
|
[[ -n "${CURSOR}" ]] && Q="${Q}&nextCursor=$(python3 -c "import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1]))" "${CURSOR}")"
|
||||||
|
TMP_PAGE=$(mktemp -t wisepg.XXXXXX.json)
|
||||||
|
"${BANK_CURL}" wise "/v1/profiles/${WISE_PROFILE_ID}/activities?${Q}" > "${TMP_PAGE}"
|
||||||
|
python3 - "${WORK}/all.json" "${TMP_PAGE}" <<'PY'
|
||||||
|
import json, sys
|
||||||
|
all_arr = json.load(open(sys.argv[1]))
|
||||||
|
page = json.load(open(sys.argv[2]))
|
||||||
|
all_arr.extend(page.get("activities") or [])
|
||||||
|
json.dump(all_arr, open(sys.argv[1], "w"))
|
||||||
|
PY
|
||||||
|
CURSOR=$(python3 -c "import json,sys; d=json.load(open(sys.argv[1])); print(d.get('cursor') or '')" "${TMP_PAGE}")
|
||||||
|
rm -f "${TMP_PAGE}"
|
||||||
|
if [[ -z "${CURSOR}" || "${CURSOR}" == "None" ]]; then
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
PAGE=$((PAGE+1))
|
||||||
|
done
|
||||||
|
|
||||||
|
# Optional enrichment: for each TRANSFER, fetch /v1/transfers/{id} and merge `reference`
|
||||||
|
if [[ "${ENRICH}" == "1" ]]; then
|
||||||
|
python3 - "${WORK}/all.json" <<'PY' > "${WORK}/ids.txt"
|
||||||
|
import json, sys
|
||||||
|
acts = json.load(open(sys.argv[1]))
|
||||||
|
for a in acts:
|
||||||
|
r = a.get("resource") or {}
|
||||||
|
if r.get("type") == "TRANSFER" and r.get("id"):
|
||||||
|
print(r["id"])
|
||||||
|
PY
|
||||||
|
> "${WORK}/refs.json"
|
||||||
|
echo '{}' > "${WORK}/refs.json"
|
||||||
|
while read -r tid; do
|
||||||
|
[[ -z "${tid}" ]] && continue
|
||||||
|
TMP_T=$(mktemp -t wiset.XXXXXX.json)
|
||||||
|
if "${BANK_CURL}" wise "/v1/transfers/${tid}" > "${TMP_T}" 2>/dev/null; then
|
||||||
|
python3 - "${WORK}/refs.json" "${TMP_T}" "${tid}" <<'PY'
|
||||||
|
import json, sys
|
||||||
|
refs = json.load(open(sys.argv[1]))
|
||||||
|
t = json.load(open(sys.argv[2]))
|
||||||
|
refs[sys.argv[3]] = t.get("reference") or ""
|
||||||
|
json.dump(refs, open(sys.argv[1], "w"))
|
||||||
|
PY
|
||||||
|
fi
|
||||||
|
rm -f "${TMP_T}"
|
||||||
|
done < "${WORK}/ids.txt"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "${FMT}" == "json" ]]; then
|
||||||
|
cat "${WORK}/all.json"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
python3 - "${WORK}/all.json" "${WORK}/refs.json" "${ENRICH}" "${SINCE}" "${UNTIL}" <<'PY'
|
||||||
|
import json, sys, re, os
|
||||||
|
acts_path, refs_path, enrich, since, until = sys.argv[1:6]
|
||||||
|
acts = json.load(open(acts_path))
|
||||||
|
refs = json.load(open(refs_path)) if os.path.exists(refs_path) else {}
|
||||||
|
def strip(s): return re.sub(r'<[^>]+>', '', s or '').strip()
|
||||||
|
def parse_amount(s):
|
||||||
|
# "<positive>+ 5,100.00 EUR</positive>" -> (+, 5100.00, EUR)
|
||||||
|
s = strip(s)
|
||||||
|
sign = "+" if s.startswith("+") else "-"
|
||||||
|
m = re.search(r'([\d,.]+)\s*([A-Z]{3})', s)
|
||||||
|
if not m: return sign, 0.0, "?"
|
||||||
|
return sign, float(m.group(1).replace(",", "")), m.group(2)
|
||||||
|
|
||||||
|
# Sort by date (ascending)
|
||||||
|
acts.sort(key=lambda a: a.get("createdOn") or "")
|
||||||
|
|
||||||
|
print(f"# Wise activities for profile (window {since} → {until})")
|
||||||
|
print()
|
||||||
|
header = f"{'date':<10} {'type':<18} {'status':<10} {'sign':<4} {'amount':>10} {'cur':<3} {'title':<28}"
|
||||||
|
if int(enrich): header += " reference"
|
||||||
|
print(header)
|
||||||
|
print("-" * (len(header) + 30 if int(enrich) else len(header)))
|
||||||
|
total_credit = total_debit = 0.0
|
||||||
|
for a in acts:
|
||||||
|
dt = (a.get("createdOn") or "")[:10]
|
||||||
|
sign, amt, cur = parse_amount(a.get("primaryAmount") or "")
|
||||||
|
typ = a.get("type", "-")
|
||||||
|
st = a.get("status", "-")
|
||||||
|
title = strip(a.get("title") or "")[:28]
|
||||||
|
line = f"{dt:<10} {typ:<18} {st:<10} {sign:<4} {amt:>10.2f} {cur:<3} {title:<28}"
|
||||||
|
if int(enrich):
|
||||||
|
r = a.get("resource") or {}
|
||||||
|
ref = refs.get(str(r.get("id", "")), "") if r.get("type") == "TRANSFER" else ""
|
||||||
|
line += f" {ref}"
|
||||||
|
print(line)
|
||||||
|
if sign == "+": total_credit += amt
|
||||||
|
if sign == "-": total_debit += amt
|
||||||
|
print("-" * (len(header) + 30 if int(enrich) else len(header)))
|
||||||
|
print(f"# {len(acts)} activity(ies) — credit: +{total_credit:.2f}, debit: -{total_debit:.2f}, net: {total_credit - total_debit:+.2f}")
|
||||||
|
PY
|
||||||
113
.claude/skills/arcodange-email-ingest/SKILL.md
Normal file
113
.claude/skills/arcodange-email-ingest/SKILL.md
Normal file
@@ -0,0 +1,113 @@
|
|||||||
|
---
|
||||||
|
name: arcodange-email-ingest
|
||||||
|
description: Scrape supplier-invoice emails from the Arcodange Zoho mailbox (`gabrielradureau@arcodange.fr` + its `books@arcodange.fr` alias + forwarded Gmail) via the Zoho Mail OAuth API, list candidates matching supplier patterns, download PDF attachments, run pdftotext + heuristic extract, and emit Dolibarr-ready supplier-invoice draft JSON for the operator to paste into the Dolibarr UI. Two workflows — (1) list candidates in a folder (default `/Inbox/books` where the alias auto-routes mail); (2) inspect one message by id, download + parse PDFs, propose draft entries. Surfaces concrete data: supplier name guess (first PDF line), invoice ref, invoice date, total HT/TVA/TTC, VAT rate. Read-only at every layer (Zoho scopes are READ-only; no write to Dolibarr). Use when the user asks "list pending supplier invoices in mail", "ingest invoices from email", "draft Dolibarr entry from this email", "audit cohort supplier docs from mail". Depends on `dolibarr` for the shared `.env`. SKIP for write-side Dolibarr operations (V9 candidate), for non-Zoho mailboxes (use IMAP fallback in a future skill if needed), and for attachments that aren't PDFs (only PDF text extraction is wired today).
|
||||||
|
requires:
|
||||||
|
bins: ["curl", "jq", "python3", "pdftotext"]
|
||||||
|
auth: true
|
||||||
|
---
|
||||||
|
|
||||||
|
# arcodange-email-ingest — supplier-invoice emails → Dolibarr draft
|
||||||
|
|
||||||
|
Close the inbound side of the accounting loop: bills land in `books@arcodange.fr`, this skill turns them into Dolibarr-ready draft entries for the operator to validate + create.
|
||||||
|
|
||||||
|
Depends on the [dolibarr](../dolibarr/SKILL.md) base skill (shared `.env`).
|
||||||
|
|
||||||
|
**CLI shortcuts:** `bin/arcodange email list | inspect | curl`
|
||||||
|
|
||||||
|
## Architecture choice — Zoho API, not IMAP
|
||||||
|
|
||||||
|
We chose the Zoho Mail OAuth API over IMAP because:
|
||||||
|
- **Richer metadata** — folder paths, attachment IDs, search operators, threads.
|
||||||
|
- **One account covers everything** — `books@arcodange.fr` is an alias of `gabrielradureau@arcodange.fr`. One refresh_token + the `/accounts` endpoint exposes both, plus all the other aliases (`contact@`, `bonjour@`, etc.).
|
||||||
|
- **Gmail folded in via forwarding** — `arcodange@gmail.com` forwards incoming to `books@` (configured in Gmail UI). No Google API setup, no app-passwords, no second OAuth flow.
|
||||||
|
- **Token-only auth** — no app-password fragility, no SCA dance (unlike Wise).
|
||||||
|
|
||||||
|
The single canonical inbox path: **`/Inbox/books`** — Zoho's auto-filter routes incoming mail to the `books@` alias into this sub-folder. Scan it first; widen with `--all-folders` only if needed.
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
1. Base skill set up ([dolibarr/README.md](../dolibarr/README.md)).
|
||||||
|
2. Zoho OAuth Self-Client created and a refresh_token generated. The `.env` extension:
|
||||||
|
```
|
||||||
|
ZOHO_CLIENT_ID=<from api-console.zoho.com self-client>
|
||||||
|
ZOHO_CLIENT_SECRET=<same>
|
||||||
|
ZOHO_REFRESH_TOKEN=<exchanged from one-time code>
|
||||||
|
ZOHO_DC=eu # eu | com | in | au
|
||||||
|
```
|
||||||
|
Setup walkthrough is in the V8 prep section of the cohort review notes.
|
||||||
|
3. Gmail forwarding to `books@arcodange.fr` enabled (Gmail Settings → Forwarding and POP/IMAP).
|
||||||
|
4. `pdftotext` (`brew install poppler` on macOS).
|
||||||
|
|
||||||
|
## Workflows
|
||||||
|
|
||||||
|
### 1. List candidates
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bin/arcodange email list # default: /Inbox/books, last 30 msgs, no filter
|
||||||
|
bin/arcodange email list --candidates-only # filter to subjects/attachments matching supplier patterns
|
||||||
|
bin/arcodange email list --folder /Inbox/contact --limit 50
|
||||||
|
bin/arcodange email list --all-folders --candidates-only # scan everything (slower, more API calls)
|
||||||
|
```
|
||||||
|
|
||||||
|
Captured at [examples/email-list.txt](examples/email-list.txt). The candidate filter matches subjects against `facture|invoice|receipt|reçu|payment|paiement|abonnement|subscription|order|commande|bill` OR any message with an attachment.
|
||||||
|
|
||||||
|
**Hard exclusions** (V8.1) — applied before the candidate test, regardless of attachments:
|
||||||
|
- Subjects starting with `Invitation:` / `Updated invitation:` / `Canceled event:` / `Accepted:` / `Declined:` / `Tentative:` / `Maybe:` (after stripping `Re:` / `Fwd:` / `Tr:` prefixes) → filters calendar events that always carry an `.ics` attachment.
|
||||||
|
- Senders matching newsletter/marketing patterns (`updates.<domain>`, `noreply@*calendar*`, `news@`, `newsletter@`, etc.).
|
||||||
|
|
||||||
|
The `[*]` column marks candidates, `[Y]` marks emails with attachments. Compared to V8.0, V8.1 cuts the `--all-folders --candidates-only` baseline from ~27 noisy entries down to ~12 actionable ones.
|
||||||
|
|
||||||
|
### 2. Inspect one email + draft Dolibarr entry
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bin/arcodange email inspect 1775141901205014300
|
||||||
|
bin/arcodange email inspect 1775141901205014300 --folder /Inbox/books # default
|
||||||
|
bin/arcodange email inspect 1775141901205014300 --save-pdf ~/Documents/factures-2026-Q2/
|
||||||
|
bin/arcodange email inspect 1775141901205014300 --json # machine-readable
|
||||||
|
```
|
||||||
|
|
||||||
|
The script:
|
||||||
|
1. Fetches the email metadata (subject / from / date) via `/messages/view`.
|
||||||
|
2. Lists attachments via `/messages/{mid}/attachmentinfo`.
|
||||||
|
3. Downloads each attachment via `/messages/{mid}/attachments/{aid}`.
|
||||||
|
4. For each `.pdf`, runs `pdftotext -layout`, applies regex heuristics to extract:
|
||||||
|
- Supplier name guess (first non-empty PDF line — often the supplier letterhead).
|
||||||
|
- Invoice reference (`facture/invoice n° XXX`).
|
||||||
|
- Invoice date.
|
||||||
|
- Total HT / TVA / TTC + VAT rate %.
|
||||||
|
5. Emits a draft JSON record per attachment — paste into the Dolibarr UI manually.
|
||||||
|
|
||||||
|
Heuristics are intentionally conservative (regex-based, no LLM dependency). For PDF templates where the heuristic fails, the raw `pdftotext` output is on disk in the work dir; rerun with `--save-pdf` to grab the PDF for manual entry.
|
||||||
|
|
||||||
|
Captured at [examples/email-inspect.txt](examples/email-inspect.txt) for the V8 baseline (Mistral AI receipt).
|
||||||
|
|
||||||
|
## What it doesn't do (V8.0 scope)
|
||||||
|
|
||||||
|
- **Does not write to Dolibarr.** The supplier invoice is still created manually in the Dolibarr UI from the draft JSON. V9 candidate: automate via `/supplierinvoices` POST.
|
||||||
|
- **Does not mark emails as ingested.** Each run re-emits the same candidates. Implementing this requires extending the OAuth scope: the current refresh_token only has READ scopes (`ZohoMail.messages.READ` etc.). The flag-set endpoint (`PUT /api/accounts/{aid}/updatemessage`) requires `ZohoMail.messages.UPDATE`, which would force the user to regenerate the refresh_token. **V8.2 candidate** — once the user opts in to the wider scope, `--mark-ingested` becomes a one-line flag on `email-inspect.sh` and `is_candidate()` in `email-list.sh` learns to skip messages with `flagid == flag_info`.
|
||||||
|
- **No body extraction yet.** We only parse PDF attachments. Inline-HTML invoices (rare — most suppliers send PDFs) would need body fetch via `/content`.
|
||||||
|
- **Heuristic extraction is best-effort.** Different supplier PDF templates yield different field-extraction reliability. The draft JSON is a starting point, not ground truth.
|
||||||
|
|
||||||
|
## Token cache
|
||||||
|
|
||||||
|
`zoho-curl.sh` caches the OAuth access_token in `$TMPDIR/zoho-access-$USER` (mode 600, TTL 50 min). Avoids hitting Zoho's OAuth refresh rate-limit on every invocation. On 401, the wrapper auto-refreshes once and retries.
|
||||||
|
|
||||||
|
## API endpoints used (Zoho Mail)
|
||||||
|
|
||||||
|
| Endpoint | Purpose |
|
||||||
|
|---|---|
|
||||||
|
| `POST /oauth/v2/token` (accounts.zoho.{dc}) | Refresh access_token from refresh_token |
|
||||||
|
| `GET /accounts` | Discover accountId + aliases on the account |
|
||||||
|
| `GET /accounts/{aid}/folders` | List folders (with paths like `/Inbox/books`) |
|
||||||
|
| `GET /accounts/{aid}/messages/view?folderId=&limit=&start=` | List messages in a folder |
|
||||||
|
| `GET /accounts/{aid}/folders/{fid}/messages/{mid}/attachmentinfo` | List attachments metadata |
|
||||||
|
| `GET /accounts/{aid}/folders/{fid}/messages/{mid}/attachments/{aid}` | Download attachment bytes |
|
||||||
|
|
||||||
|
## Out of scope
|
||||||
|
|
||||||
|
- **Writing to Dolibarr** (V9 candidate — would lift the read-only constraint on the API key, or use a separate write-scoped key).
|
||||||
|
- **Marking ingested emails** (V8.1 trivial follow-up).
|
||||||
|
- **Non-PDF attachments** (heuristics are PDF-specific).
|
||||||
|
- **Body-text extraction** (would need `/content` endpoint, deferred).
|
||||||
|
- **IMAP fallback** for non-Zoho mailboxes (deferred — Gmail forwarding to books@ covers the only known external mailbox today).
|
||||||
|
- **LLM-based extraction** (deferred — regex covers the current set of supplier templates well enough).
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
================================================================================
|
||||||
|
Email 1775141901205014300
|
||||||
|
================================================================================
|
||||||
|
subject : Votre facture nº MSTRL-API-814045-001 de Mistral AI SAS
|
||||||
|
from : no-reply@mistral.ai
|
||||||
|
date : 2026-04-02
|
||||||
|
attached : True
|
||||||
|
|
||||||
|
-- Attachment 1: invoice-MSTRL-API-814045-001.pdf (74377 bytes, 1771 chars extracted) --
|
||||||
|
pdf_top_line = 'Facture'
|
||||||
|
invoice_ref = 'API-814045-001'
|
||||||
|
invoice_date_raw = None
|
||||||
|
total_ht = None
|
||||||
|
total_tva = None
|
||||||
|
total_ttc = None
|
||||||
|
vat_rate_pct = '20.0'
|
||||||
|
|
||||||
|
Suggested Dolibarr supplier-invoice draft entries:
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"supplier_hint": "Facture",
|
||||||
|
"invoice_ref": "API-814045-001",
|
||||||
|
"invoice_date": null,
|
||||||
|
"total_ht": null,
|
||||||
|
"total_tva": null,
|
||||||
|
"total_ttc": null,
|
||||||
|
"vat_rate_pct": "20.0",
|
||||||
|
"source_email": "1775141901205014300",
|
||||||
|
"source_attachment": "invoice-MSTRL-API-814045-001.pdf"
|
||||||
|
}
|
||||||
|
]
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
date cand att messageId folder from subject
|
||||||
|
----------------------------------------------------------------------------------------------------------------------------------
|
||||||
|
2026-05-20 [*] [Y] 1779312401677014300 /clients/KissMetrics rsirvent@digitalocean.com Re: VM not running despite status=active, after volume
|
||||||
|
2026-05-20 [*] [Y] 1779298419301014300 /clients/KissMetrics tdziuba@kissmetrics.io Re: VM not running despite status=active, after volume
|
||||||
|
2026-05-20 [*] [Y] 1779285954272004400 /clients/KissMetrics tdziuba@kissmetrics.io Re: VM not running despite status=active, after volume
|
||||||
|
2026-05-05 [*] [ ] 1777970798248014300 /Inbox/abonnements freemobile@free-mobile.fr Votre facture mobile Free est disponible
|
||||||
|
2026-04-21 [*] [Y] 1776785469477004300 /Notification noreply@hiway.fr Darnis Operations - Facture F1042
|
||||||
|
2026-04-12 [*] [Y] 1776017238960014300 /Inbox/books arcodange@gmail.com Fwd: Your receipt from Anthropic Ireland, Limited #2109
|
||||||
|
2026-04-04 [*] [ ] 1775264759983014300 /Inbox/abonnements freemobile@free-mobile.fr Votre facture mobile Free est disponible
|
||||||
|
2026-04-02 [*] [Y] 1775141901205014300 /Inbox/books no-reply@mistral.ai Votre facture nº MSTRL-API-814045-001 de Mistral AI SAS
|
||||||
|
2026-03-05 [*] [ ] 1772689535069004400 /Inbox/helloworld freemobile@free-mobile.fr Votre facture mobile Free est disponible
|
||||||
|
2026-02-08 [*] [Y] 1770582421208004400 /Inbox/bureaux ne-pas-repondre@portailpro.gouv.fr Valider votre espace personnel sur Portailpro.gouv
|
||||||
|
2026-01-09 [*] [ ] 1767989744791004400 /Inbox/books gabrielradureau@gmail.com Fwd: INPI - Votre paiement pour la commande Réf. 181876
|
||||||
|
2026-01-06 [*] [Y] 1767710535894005600 /Inbox gabrielradureau@gmail.com Statuts
|
||||||
|
----------------------------------------------------------------------------------------------------------------------------------
|
||||||
|
# 12 message(s) (candidates only)
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
date cand att messageId folder from subject
|
||||||
|
----------------------------------------------------------------------------------------------------------------------------------
|
||||||
|
2026-04-12 [*] [Y] 1776017238960014300 /Inbox/books arcodange@gmail.com Fwd: Your receipt from Anthropic Ireland, Limited #2109
|
||||||
|
2026-04-02 [*] [Y] 1775141901205014300 /Inbox/books no-reply@mistral.ai Votre facture nº MSTRL-API-814045-001 de Mistral AI SAS
|
||||||
|
2026-01-09 [*] [ ] 1767989744791004400 /Inbox/books gabrielradureau@gmail.com Fwd: INPI - Votre paiement pour la commande Réf. 181876
|
||||||
|
----------------------------------------------------------------------------------------------------------------------------------
|
||||||
|
# 3 message(s) (candidates only)
|
||||||
256
.claude/skills/arcodange-email-ingest/scripts/email-inspect.sh
Executable file
256
.claude/skills/arcodange-email-ingest/scripts/email-inspect.sh
Executable file
@@ -0,0 +1,256 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Inspect one email by id and propose a Dolibarr supplier-invoice draft.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# email-inspect.sh <messageId> [--folder PATH] # default folder: /Inbox/books
|
||||||
|
# [--save-pdf DIR] # save PDF attachments under DIR/
|
||||||
|
# [--json] # emit a single JSON object on stdout
|
||||||
|
#
|
||||||
|
# Pipeline (read-only):
|
||||||
|
# 1. Find the message (in the given folder, default /Inbox/books).
|
||||||
|
# 2. List attachments via /attachmentinfo.
|
||||||
|
# 3. For each PDF attachment: download, run pdftotext, extract supplier-side
|
||||||
|
# heuristics (name, totals, dates, ref).
|
||||||
|
# 4. Emit a draft "Dolibarr-ready" record per attachment so the operator can
|
||||||
|
# hand-create the supplier invoice in the Dolibarr UI.
|
||||||
|
#
|
||||||
|
# This skill DOES NOT write to Dolibarr. Auto-creation of supplier invoices is
|
||||||
|
# V9 candidate.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
ZOHO_CURL="${SCRIPT_DIR}/zoho-curl.sh"
|
||||||
|
|
||||||
|
if [[ $# -lt 1 ]]; then
|
||||||
|
echo "email-inspect.sh: missing <messageId>" >&2
|
||||||
|
echo " Hint: bin/arcodange email list to see candidate ids." >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
MID="$1"; shift || true
|
||||||
|
FOLDER="/Inbox/books"; SAVE_PDF_DIR=""; FMT="text"
|
||||||
|
while [[ $# -gt 0 ]]; do
|
||||||
|
case "$1" in
|
||||||
|
--folder) FOLDER="$2"; shift 2 ;;
|
||||||
|
--save-pdf) SAVE_PDF_DIR="$2"; shift 2 ;;
|
||||||
|
--json) FMT="json"; shift ;;
|
||||||
|
-h|--help) sed -n '2,18p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||||
|
*) echo "email-inspect.sh: unknown arg: $1" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
command -v pdftotext >/dev/null || { echo "email-inspect.sh: pdftotext not found (brew install poppler)" >&2; exit 2; }
|
||||||
|
|
||||||
|
WORK="$(mktemp -d -t emailinspect.XXXXXX)"
|
||||||
|
trap 'rm -rf "${WORK}"' EXIT
|
||||||
|
|
||||||
|
# 1. accountId + folderId
|
||||||
|
"${ZOHO_CURL}" /accounts > "${WORK}/accounts.json"
|
||||||
|
AID=$(python3 -c "import json,sys; d=json.load(open(sys.argv[1])); print((d.get('data') or [{}])[0].get('accountId',''))" "${WORK}/accounts.json")
|
||||||
|
"${ZOHO_CURL}" "/accounts/${AID}/folders" > "${WORK}/folders.json"
|
||||||
|
FID=$(python3 -c "
|
||||||
|
import json, sys
|
||||||
|
d = json.load(open(sys.argv[1]))
|
||||||
|
target = sys.argv[2]
|
||||||
|
for f in (d.get('data') or []):
|
||||||
|
if f.get('path') == target:
|
||||||
|
print(f.get('folderId')); break" "${WORK}/folders.json" "${FOLDER}")
|
||||||
|
[[ -z "${FID}" ]] && { echo "email-inspect.sh: folder '${FOLDER}' not found" >&2; exit 2; }
|
||||||
|
|
||||||
|
# 2. Find the message in the folder listing (to grab metadata: subject, from, date)
|
||||||
|
"${ZOHO_CURL}" "/accounts/${AID}/messages/view?folderId=${FID}&limit=100&sortorder=false&start=1" > "${WORK}/folder_msgs.json"
|
||||||
|
python3 - "${WORK}/folder_msgs.json" "${MID}" > "${WORK}/meta.json" <<'PY'
|
||||||
|
import json, sys
|
||||||
|
d = json.load(open(sys.argv[1]))
|
||||||
|
mid = sys.argv[2]
|
||||||
|
for m in (d.get("data") or []):
|
||||||
|
if str(m.get("messageId")) == mid:
|
||||||
|
json.dump(m, sys.stdout); sys.exit(0)
|
||||||
|
sys.exit(f"messageId {mid} not found in this folder")
|
||||||
|
PY
|
||||||
|
|
||||||
|
# 3. Attachment metadata
|
||||||
|
"${ZOHO_CURL}" "/accounts/${AID}/folders/${FID}/messages/${MID}/attachmentinfo" > "${WORK}/attachinfo.json"
|
||||||
|
|
||||||
|
# 4. Download each attachment — needs raw bytes (Accept: */*), not the JSON
|
||||||
|
# wrapper's default. We bypass zoho-curl.sh for the attachment download but
|
||||||
|
# reuse the cached access_token it wrote.
|
||||||
|
set -a; source "${SCRIPT_DIR}/../../dolibarr/.env"; set +a
|
||||||
|
: "${ZOHO_DC:=eu}"
|
||||||
|
TOKEN_CACHE="${TMPDIR:-/tmp}/zoho-access-$(whoami)"
|
||||||
|
if [[ ! -s "${TOKEN_CACHE}" ]]; then
|
||||||
|
echo "email-inspect.sh: missing access token cache — run any zoho-curl call first to populate it" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
ACCESS_TOKEN=$(cat "${TOKEN_CACHE}")
|
||||||
|
MAIL_BASE="https://mail.zoho.${ZOHO_DC}/api"
|
||||||
|
|
||||||
|
mkdir -p "${WORK}/atts" "${WORK}/text"
|
||||||
|
ATT_IDS=$(python3 -c "
|
||||||
|
import json, sys
|
||||||
|
d = json.load(open(sys.argv[1]))
|
||||||
|
data = d.get('data') or {}
|
||||||
|
for a in (data.get('attachments') or []):
|
||||||
|
print(f\"{a.get('attachmentId')}|{a.get('attachmentName','-')}\")" "${WORK}/attachinfo.json")
|
||||||
|
while IFS='|' read -r aid aname; do
|
||||||
|
[[ -z "${aid}" ]] && continue
|
||||||
|
outpath="${WORK}/atts/${aname}"
|
||||||
|
curl -sS \
|
||||||
|
-H "Authorization: Zoho-oauthtoken ${ACCESS_TOKEN}" \
|
||||||
|
-H "Accept: */*" \
|
||||||
|
--max-time 60 \
|
||||||
|
-o "${outpath}" \
|
||||||
|
"${MAIL_BASE}/accounts/${AID}/folders/${FID}/messages/${MID}/attachments/${aid}" || true
|
||||||
|
# If pdf, extract text (bash 3.2 compatible — no ${var,,})
|
||||||
|
aname_lc=$(echo "${aname}" | tr '[:upper:]' '[:lower:]')
|
||||||
|
if [[ "${aname_lc}" == *.pdf ]]; then
|
||||||
|
pdftotext -layout "${outpath}" "${WORK}/text/${aname%.pdf}.txt" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
done <<< "${ATT_IDS}"
|
||||||
|
|
||||||
|
# Optional save
|
||||||
|
if [[ -n "${SAVE_PDF_DIR}" ]]; then
|
||||||
|
mkdir -p "${SAVE_PDF_DIR}"
|
||||||
|
cp "${WORK}/atts/"*.pdf "${SAVE_PDF_DIR}/" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 5. Heuristic extract + render
|
||||||
|
python3 - "${WORK}" "${FMT}" <<'PY'
|
||||||
|
import json, sys, os, re, datetime, glob
|
||||||
|
work, fmt = sys.argv[1:3]
|
||||||
|
|
||||||
|
meta = json.load(open(os.path.join(work,"meta.json")))
|
||||||
|
ts = int(meta.get("sentDateInGMT") or meta.get("receivedTime") or 0) // 1000
|
||||||
|
mail_date = datetime.datetime.fromtimestamp(ts).strftime("%Y-%m-%d") if ts else None
|
||||||
|
mail_from = (meta.get("fromAddress") or meta.get("sender") or "-").replace("<","<").replace(">",">").replace("<","").replace(">","")
|
||||||
|
mail_subject = meta.get("subject") or "-"
|
||||||
|
|
||||||
|
# Heuristics on PDF text
|
||||||
|
def extract(text):
|
||||||
|
out = {}
|
||||||
|
# First non-empty line is often the supplier name (or the address block first line)
|
||||||
|
lines = [l.strip() for l in text.splitlines() if l.strip()]
|
||||||
|
out["pdf_top_line"] = lines[0] if lines else None
|
||||||
|
|
||||||
|
# Total TTC / HT / TVA — try multiple French/English patterns
|
||||||
|
def first_match(*patterns):
|
||||||
|
for p in patterns:
|
||||||
|
for line in lines:
|
||||||
|
m = re.search(p, line, re.IGNORECASE)
|
||||||
|
if m: return m.group(1).replace(",", ".").replace(" ", "")
|
||||||
|
return None
|
||||||
|
|
||||||
|
def parse_amount(s):
|
||||||
|
if not s: return None
|
||||||
|
clean = s.replace(",", ".").replace(" ", "")
|
||||||
|
try:
|
||||||
|
v = float(clean)
|
||||||
|
# Money amounts < 1M EUR; filters out VAT-number false positives (FR12345678901)
|
||||||
|
return v if 0 <= v < 1_000_000 else None
|
||||||
|
except: return None
|
||||||
|
|
||||||
|
def first_amount(*patterns):
|
||||||
|
for p in patterns:
|
||||||
|
for line in lines:
|
||||||
|
m = re.search(p, line, re.IGNORECASE)
|
||||||
|
if m:
|
||||||
|
v = parse_amount(m.group(1))
|
||||||
|
if v is not None: return f"{v:.2f}"
|
||||||
|
return None
|
||||||
|
|
||||||
|
out["total_ht"] = first_amount(r'(?:total\s*ht|montant\s*ht|net\s*amount|subtotal)[^\d-]*([\d \.,]+)')
|
||||||
|
# TVA: require currency suffix to avoid matching VAT-number digits
|
||||||
|
out["total_tva"] = first_amount(r'(?:tva|vat)[^\d-]*([\d \.,]+)\s*(?:€|eur)\b')
|
||||||
|
out["total_ttc"] = first_amount(r'(?:total\s*ttc|amount\s*due|total\s*due|grand\s*total|montant\s*total|amount\s*paid)[^\d-]*([\d \.,]+)')
|
||||||
|
|
||||||
|
# Invoice ref — must contain a digit (filters "umber", "Invoice", etc.)
|
||||||
|
m = re.search(r'(?:facture|invoice|receipt|reçu)\s*(?:n[°o]?|number|#|:)\s*([A-Za-z0-9][\w\d/-]{2,})', text, re.IGNORECASE)
|
||||||
|
if m and any(c.isdigit() for c in m.group(1)):
|
||||||
|
out["invoice_ref"] = m.group(1)
|
||||||
|
else:
|
||||||
|
# Fallback: any reasonable ref-shaped token after "Invoice" / "Facture" header
|
||||||
|
m = re.search(r'\b([A-Z]{2,}[-/]?\d[\w\d/-]{2,})\b', text)
|
||||||
|
out["invoice_ref"] = m.group(1) if m else None
|
||||||
|
|
||||||
|
# Invoice date — try ISO, French DD/MM/YYYY, English MM/DD/YYYY, French long form
|
||||||
|
out["invoice_date_raw"] = None
|
||||||
|
for p in (
|
||||||
|
r'\b(\d{4}-\d{2}-\d{2})\b',
|
||||||
|
r'(?:date|émise\s*le|invoice\s*date|date\s*de\s*facturation)[:\s]*(\d{1,2}[\s/.-]\d{1,2}[\s/.-]\d{2,4})',
|
||||||
|
r'(?:date|émise\s*le|invoice\s*date)[:\s]*(\d{1,2}\s+\w{3,9}\.?\s+\d{4})',
|
||||||
|
):
|
||||||
|
m = re.search(p, text, re.IGNORECASE)
|
||||||
|
if m: out["invoice_date_raw"] = m.group(1).strip(); break
|
||||||
|
|
||||||
|
# VAT rate (e.g. "20%") — restrict to 0-25% so "100%" / page footers don't match.
|
||||||
|
vrate = None
|
||||||
|
for line in lines:
|
||||||
|
m = re.search(r'\b(\d{1,2}([.,]\d+)?)\s*%', line)
|
||||||
|
if m:
|
||||||
|
v = float(m.group(1).replace(",", "."))
|
||||||
|
if 0 <= v <= 25:
|
||||||
|
vrate = m.group(1).replace(",", "."); break
|
||||||
|
out["vat_rate_pct"] = vrate
|
||||||
|
|
||||||
|
return out
|
||||||
|
|
||||||
|
pdfs = []
|
||||||
|
for pdf in sorted(glob.glob(os.path.join(work,"atts","*.pdf")) +
|
||||||
|
glob.glob(os.path.join(work,"atts","*.PDF"))):
|
||||||
|
name = os.path.basename(pdf)
|
||||||
|
txt_path = os.path.join(work,"text", os.path.splitext(name)[0] + ".txt")
|
||||||
|
text = open(txt_path).read() if os.path.isfile(txt_path) else ""
|
||||||
|
h = extract(text)
|
||||||
|
h["attachment_name"] = name
|
||||||
|
h["pdf_size_bytes"] = os.path.getsize(pdf)
|
||||||
|
h["pdf_text_len"] = len(text)
|
||||||
|
pdfs.append(h)
|
||||||
|
|
||||||
|
result = {
|
||||||
|
"email": {
|
||||||
|
"messageId": meta.get("messageId"),
|
||||||
|
"subject": mail_subject,
|
||||||
|
"from": mail_from,
|
||||||
|
"date": mail_date,
|
||||||
|
"hasAttachment": str(meta.get("hasAttachment","")) == "1",
|
||||||
|
},
|
||||||
|
"attachments": pdfs,
|
||||||
|
"dolibarr_draft_suggestions": [
|
||||||
|
{
|
||||||
|
"supplier_hint": p.get("pdf_top_line"),
|
||||||
|
"invoice_ref": p.get("invoice_ref"),
|
||||||
|
"invoice_date": p.get("invoice_date_raw"),
|
||||||
|
"total_ht": p.get("total_ht"),
|
||||||
|
"total_tva": p.get("total_tva"),
|
||||||
|
"total_ttc": p.get("total_ttc"),
|
||||||
|
"vat_rate_pct": p.get("vat_rate_pct"),
|
||||||
|
"source_email": meta.get("messageId"),
|
||||||
|
"source_attachment": p.get("attachment_name"),
|
||||||
|
} for p in pdfs
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
if fmt == "json":
|
||||||
|
print(json.dumps(result, indent=2, ensure_ascii=False))
|
||||||
|
sys.exit(0)
|
||||||
|
|
||||||
|
print("=" * 80)
|
||||||
|
print(f" Email {meta.get('messageId')}")
|
||||||
|
print("=" * 80)
|
||||||
|
print(f" subject : {mail_subject}")
|
||||||
|
print(f" from : {mail_from}")
|
||||||
|
print(f" date : {mail_date}")
|
||||||
|
print(f" attached : {result['email']['hasAttachment']}")
|
||||||
|
print()
|
||||||
|
if not pdfs:
|
||||||
|
print(" (no PDF attachments — try inspecting body or other types)")
|
||||||
|
for i, p in enumerate(pdfs, 1):
|
||||||
|
print(f" -- Attachment {i}: {p['attachment_name']} ({p['pdf_size_bytes']} bytes, {p['pdf_text_len']} chars extracted) --")
|
||||||
|
for k in ("pdf_top_line","invoice_ref","invoice_date_raw","total_ht","total_tva","total_ttc","vat_rate_pct"):
|
||||||
|
v = p.get(k)
|
||||||
|
print(f" {k:<16} = {v!r}")
|
||||||
|
print()
|
||||||
|
|
||||||
|
print(" Suggested Dolibarr supplier-invoice draft entries:")
|
||||||
|
print(json.dumps(result["dolibarr_draft_suggestions"], indent=4, ensure_ascii=False))
|
||||||
|
PY
|
||||||
141
.claude/skills/arcodange-email-ingest/scripts/email-list.sh
Executable file
141
.claude/skills/arcodange-email-ingest/scripts/email-list.sh
Executable file
@@ -0,0 +1,141 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# List candidate supplier-invoice emails from the books@ Zoho mailbox.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# email-list.sh [--folder PATH] # default: /Inbox/books (the books@ alias-filtered folder)
|
||||||
|
# [--limit N] # default: 30
|
||||||
|
# [--candidates-only] # filter by subject pattern OR attachment
|
||||||
|
# [--all-folders] # scan every folder (slow, lots of API calls)
|
||||||
|
#
|
||||||
|
# Output: table with mid, date, from, subject, hasAttachment.
|
||||||
|
# A "candidate" is a message whose subject matches a supplier-like pattern
|
||||||
|
# (facture/invoice/receipt/reçu/payment/paiement/abonnement/order/commande)
|
||||||
|
# OR which has an attachment.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
ZOHO_CURL="${SCRIPT_DIR}/zoho-curl.sh"
|
||||||
|
|
||||||
|
FOLDER="/Inbox/books"
|
||||||
|
LIMIT=30
|
||||||
|
CANDIDATES_ONLY=0
|
||||||
|
ALL_FOLDERS=0
|
||||||
|
while [[ $# -gt 0 ]]; do
|
||||||
|
case "$1" in
|
||||||
|
--folder) FOLDER="$2"; shift 2 ;;
|
||||||
|
--limit) LIMIT="$2"; shift 2 ;;
|
||||||
|
--candidates-only) CANDIDATES_ONLY=1; shift ;;
|
||||||
|
--all-folders) ALL_FOLDERS=1; shift ;;
|
||||||
|
-h|--help) sed -n '2,12p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||||
|
*) echo "email-list.sh: unknown arg: $1" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
WORK="$(mktemp -d -t emailist.XXXXXX)"
|
||||||
|
trap 'rm -rf "${WORK}"' EXIT
|
||||||
|
|
||||||
|
# 1. Discover accountId
|
||||||
|
"${ZOHO_CURL}" /accounts > "${WORK}/accounts.json"
|
||||||
|
AID=$(python3 -c "import json,sys; d=json.load(open(sys.argv[1])); print((d.get('data') or [{}])[0].get('accountId',''))" "${WORK}/accounts.json")
|
||||||
|
[[ -z "${AID}" ]] && { echo "email-list.sh: no accountId in /accounts response" >&2; exit 1; }
|
||||||
|
|
||||||
|
# 2. Resolve folder path → folderId
|
||||||
|
"${ZOHO_CURL}" "/accounts/${AID}/folders" > "${WORK}/folders.json"
|
||||||
|
|
||||||
|
# Build list of (folderId, path) tuples to scan
|
||||||
|
if [[ "${ALL_FOLDERS}" == "1" ]]; then
|
||||||
|
FOLDER_IDS=$(python3 -c "
|
||||||
|
import json, sys
|
||||||
|
d = json.load(open(sys.argv[1]))
|
||||||
|
for f in (d.get('data') or []):
|
||||||
|
fid = f.get('folderId'); path = f.get('path') or f.get('folderName','-')
|
||||||
|
# Skip noisy system folders
|
||||||
|
if path in ('/Drafts','/Templates','/Snoozed','/Sent','/Spam','/Trash','/Outbox'): continue
|
||||||
|
print(f\"{fid}|{path}\")" "${WORK}/folders.json")
|
||||||
|
else
|
||||||
|
FOLDER_IDS=$(python3 -c "
|
||||||
|
import json, sys
|
||||||
|
d = json.load(open(sys.argv[1]))
|
||||||
|
target = sys.argv[2]
|
||||||
|
for f in (d.get('data') or []):
|
||||||
|
if f.get('path') == target:
|
||||||
|
print(f\"{f.get('folderId')}|{f.get('path')}\")
|
||||||
|
break" "${WORK}/folders.json" "${FOLDER}")
|
||||||
|
if [[ -z "${FOLDER_IDS}" ]]; then
|
||||||
|
echo "email-list.sh: folder '${FOLDER}' not found. Available:" >&2
|
||||||
|
python3 -c "import json,sys; [print(f' {f.get(\"path\",\"-\")}') for f in json.load(open(sys.argv[1])).get('data',[])]" "${WORK}/folders.json" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 3. Fetch messages per folder
|
||||||
|
mkdir -p "${WORK}/msgs"
|
||||||
|
COUNT=0
|
||||||
|
while IFS='|' read -r fid fpath; do
|
||||||
|
[[ -z "${fid}" ]] && continue
|
||||||
|
COUNT=$((COUNT+1))
|
||||||
|
out="${WORK}/msgs/$(printf '%03d' "${COUNT}").json"
|
||||||
|
"${ZOHO_CURL}" "/accounts/${AID}/messages/view?folderId=${fid}&limit=${LIMIT}&sortorder=false&start=1" > "${out}" 2>/dev/null || echo '{"data":[]}' > "${out}"
|
||||||
|
echo "${fpath}" > "${out}.path"
|
||||||
|
done <<< "${FOLDER_IDS}"
|
||||||
|
|
||||||
|
# 4. Render
|
||||||
|
python3 - "${WORK}/msgs" "${CANDIDATES_ONLY}" <<'PY'
|
||||||
|
import json, sys, os, re, datetime, glob
|
||||||
|
msgs_dir, candidates_only_str = sys.argv[1:3]
|
||||||
|
candidates_only = candidates_only_str == "1"
|
||||||
|
|
||||||
|
CANDIDATE_PATTERN = re.compile(
|
||||||
|
r'facture|invoice|receipt|re[cç]u|payment|paiement|abonnement|subscription|order|commande|invoice|bill',
|
||||||
|
re.IGNORECASE,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Subjects that look like calendar invites / event updates / generic notifications
|
||||||
|
# get filtered out of --candidates-only — they always have a .ics attachment so
|
||||||
|
# the "has-attachment" heuristic alone catches them as false positives.
|
||||||
|
EXCLUDE_PATTERN = re.compile(
|
||||||
|
r'^(?:re:\s*|fwd:\s*|tr:\s*)*' # strip Re:/Fwd:/Tr: prefixes
|
||||||
|
r'(?:invitation|updated\s+invitation|canceled\s+event|accepted|declined|tentative|maybe)\s*:',
|
||||||
|
re.IGNORECASE,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Senders that are pure noise — newsletter/marketing patterns.
|
||||||
|
EXCLUDE_SENDER = re.compile(
|
||||||
|
r'(updates\.|noreply@.*calendar|@calendar\.|news@|newsletter@|@updates\.)',
|
||||||
|
re.IGNORECASE,
|
||||||
|
)
|
||||||
|
|
||||||
|
def is_candidate(m):
|
||||||
|
subj = m.get("subject","") or ""
|
||||||
|
sender = m.get("fromAddress","") or m.get("sender","") or ""
|
||||||
|
# Hard exclusions take precedence over inclusions
|
||||||
|
if EXCLUDE_PATTERN.match(subj.strip()): return False
|
||||||
|
if EXCLUDE_SENDER.search(sender): return False
|
||||||
|
if str(m.get("hasAttachment","")) == "1": return True
|
||||||
|
if CANDIDATE_PATTERN.search(subj): return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
rows = []
|
||||||
|
for f in sorted(glob.glob(os.path.join(msgs_dir, "*.json"))):
|
||||||
|
fpath = open(f + ".path").read().strip()
|
||||||
|
try: data = json.load(open(f)).get("data") or []
|
||||||
|
except: continue
|
||||||
|
for m in data:
|
||||||
|
if candidates_only and not is_candidate(m): continue
|
||||||
|
ts = int(m.get("sentDateInGMT") or m.get("receivedTime") or 0) // 1000
|
||||||
|
dt = datetime.datetime.fromtimestamp(ts).strftime("%Y-%m-%d") if ts else "-"
|
||||||
|
frm = (m.get("fromAddress") or m.get("sender") or "-").replace("<","<").replace(">",">").replace("<","").replace(">","")[:36]
|
||||||
|
subj = (m.get("subject") or "-")[:55]
|
||||||
|
has = "Y" if str(m.get("hasAttachment","")) == "1" else " "
|
||||||
|
cand = "*" if is_candidate(m) else " "
|
||||||
|
rows.append((dt, fpath, cand, has, m.get("messageId","-"), frm, subj))
|
||||||
|
|
||||||
|
rows.sort(key=lambda r: r[0], reverse=True)
|
||||||
|
print(f"{'date':<10} {'cand':<4} {'att':<3} {'messageId':<22} {'folder':<22} {'from':<36} subject")
|
||||||
|
print("-" * 130)
|
||||||
|
for dt, fpath, cand, has, mid, frm, subj in rows:
|
||||||
|
print(f"{dt:<10} [{cand}] [{has}] {mid:<22} {fpath[:22]:<22} {frm:<36} {subj}")
|
||||||
|
print("-" * 130)
|
||||||
|
print(f"# {len(rows)} message(s)" + (" (candidates only)" if candidates_only else ""))
|
||||||
|
PY
|
||||||
126
.claude/skills/arcodange-email-ingest/scripts/zoho-curl.sh
Executable file
126
.claude/skills/arcodange-email-ingest/scripts/zoho-curl.sh
Executable file
@@ -0,0 +1,126 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Read-only curl wrapper for the Zoho Mail API.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# zoho-curl.sh <path> # e.g. zoho-curl.sh /accounts
|
||||||
|
# zoho-curl.sh -i <path> # include curl's -i (response headers)
|
||||||
|
# zoho-curl.sh -o file.json <path> # write body to file
|
||||||
|
#
|
||||||
|
# Reads credentials from ../../dolibarr/.env (the shared canonical file).
|
||||||
|
# Required vars:
|
||||||
|
# ZOHO_CLIENT_ID, ZOHO_CLIENT_SECRET, ZOHO_REFRESH_TOKEN, ZOHO_DC
|
||||||
|
#
|
||||||
|
# Token strategy: each invocation refreshes a short-lived access_token from
|
||||||
|
# the refresh_token (Zoho access_tokens live 1h; the cost of refreshing on
|
||||||
|
# every call is ~150 ms and avoids state on disk). On 401 from the mail API
|
||||||
|
# we re-refresh once and retry (covers refresh-token rotation cases).
|
||||||
|
#
|
||||||
|
# Exits non-zero on HTTP >= 400 and writes body to stdout + a short message
|
||||||
|
# to stderr — same shape as dol-curl.sh / bank-curl.sh.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
ENV_FILE="${SCRIPT_DIR}/../../dolibarr/.env"
|
||||||
|
|
||||||
|
if [[ ! -f "${ENV_FILE}" ]]; then
|
||||||
|
echo "zoho-curl.sh: missing ${ENV_FILE}" >&2
|
||||||
|
echo " Required vars: ZOHO_CLIENT_ID, ZOHO_CLIENT_SECRET, ZOHO_REFRESH_TOKEN, ZOHO_DC." >&2
|
||||||
|
echo " See arcodange-email-ingest/SKILL.md for the OAuth setup." >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
set -a; source "${ENV_FILE}"; set +a
|
||||||
|
|
||||||
|
: "${ZOHO_CLIENT_ID:?zoho-curl.sh: ZOHO_CLIENT_ID not set in .env}"
|
||||||
|
: "${ZOHO_CLIENT_SECRET:?zoho-curl.sh: ZOHO_CLIENT_SECRET not set in .env}"
|
||||||
|
: "${ZOHO_REFRESH_TOKEN:?zoho-curl.sh: ZOHO_REFRESH_TOKEN not set in .env}"
|
||||||
|
: "${ZOHO_DC:=eu}"
|
||||||
|
|
||||||
|
ACCOUNTS_BASE="https://accounts.zoho.${ZOHO_DC}"
|
||||||
|
MAIL_BASE="https://mail.zoho.${ZOHO_DC}/api"
|
||||||
|
|
||||||
|
# Parse pass-through curl args (everything before the last positional)
|
||||||
|
PASSTHRU=()
|
||||||
|
while [[ $# -gt 1 ]]; do
|
||||||
|
PASSTHRU+=("$1"); shift
|
||||||
|
done
|
||||||
|
if [[ $# -lt 1 ]]; then
|
||||||
|
echo "zoho-curl.sh: missing API path. Example: zoho-curl.sh /accounts" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
API_PATH="$1"
|
||||||
|
|
||||||
|
# Cache access_token in tmpfs to avoid hitting OAuth rate limits on every
|
||||||
|
# zoho-curl invocation. Zoho access_tokens live 1h; we refresh after 50 min.
|
||||||
|
CACHE_FILE="${TMPDIR:-/tmp}/zoho-access-$(whoami)"
|
||||||
|
CACHE_TTL_SECONDS=$((50 * 60))
|
||||||
|
|
||||||
|
get_access_token() {
|
||||||
|
if [[ -f "${CACHE_FILE}" ]]; then
|
||||||
|
local age
|
||||||
|
age=$(( $(date +%s) - $(stat -f %m "${CACHE_FILE}" 2>/dev/null || stat -c %Y "${CACHE_FILE}") ))
|
||||||
|
if [[ ${age} -lt ${CACHE_TTL_SECONDS} ]]; then
|
||||||
|
cat "${CACHE_FILE}"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
local token
|
||||||
|
if ! token=$(curl -sS -X POST "${ACCOUNTS_BASE}/oauth/v2/token" \
|
||||||
|
--max-time 15 \
|
||||||
|
-d "grant_type=refresh_token" \
|
||||||
|
-d "client_id=${ZOHO_CLIENT_ID}" \
|
||||||
|
-d "client_secret=${ZOHO_CLIENT_SECRET}" \
|
||||||
|
-d "refresh_token=${ZOHO_REFRESH_TOKEN}" \
|
||||||
|
| python3 -c "
|
||||||
|
import json, sys
|
||||||
|
try: d = json.load(sys.stdin)
|
||||||
|
except: sys.exit('failed to parse OAuth response')
|
||||||
|
if 'access_token' not in d:
|
||||||
|
sys.exit(f'OAuth refresh failed: {d}')
|
||||||
|
print(d['access_token'])"); then
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
if [[ -z "${token}" ]]; then
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
# Store cache (mode 600) only on success
|
||||||
|
printf '%s' "${token}" > "${CACHE_FILE}"
|
||||||
|
chmod 600 "${CACHE_FILE}"
|
||||||
|
printf '%s' "${token}"
|
||||||
|
}
|
||||||
|
|
||||||
|
do_call() {
|
||||||
|
local token="$1"
|
||||||
|
local body_file="$2"
|
||||||
|
local headers_file="$3"
|
||||||
|
curl -sS \
|
||||||
|
-H "Authorization: Zoho-oauthtoken ${token}" \
|
||||||
|
-H "Accept: application/json" \
|
||||||
|
--max-time 30 \
|
||||||
|
-o "${body_file}" \
|
||||||
|
-D "${headers_file}" \
|
||||||
|
-w "%{http_code}" \
|
||||||
|
${PASSTHRU[@]+"${PASSTHRU[@]}"} \
|
||||||
|
"${MAIL_BASE}${API_PATH}"
|
||||||
|
}
|
||||||
|
|
||||||
|
ACCESS_TOKEN=$(get_access_token)
|
||||||
|
[[ -z "${ACCESS_TOKEN}" ]] && { echo "zoho-curl.sh: empty access_token" >&2; exit 1; }
|
||||||
|
|
||||||
|
BODY_FILE="$(mktemp -t zohocurl.XXXXXX)"
|
||||||
|
HEADERS_FILE="$(mktemp -t zohohdr.XXXXXX)"
|
||||||
|
trap 'rm -f "${BODY_FILE}" "${HEADERS_FILE}"' EXIT
|
||||||
|
|
||||||
|
HTTP_CODE=$(do_call "${ACCESS_TOKEN}" "${BODY_FILE}" "${HEADERS_FILE}")
|
||||||
|
|
||||||
|
# Retry once on 401 with a fresh token (handles edge cases of refresh-token rotation)
|
||||||
|
if [[ "${HTTP_CODE}" == "401" ]]; then
|
||||||
|
ACCESS_TOKEN=$(get_access_token)
|
||||||
|
HTTP_CODE=$(do_call "${ACCESS_TOKEN}" "${BODY_FILE}" "${HEADERS_FILE}")
|
||||||
|
fi
|
||||||
|
|
||||||
|
cat "${BODY_FILE}"
|
||||||
|
if [[ "${HTTP_CODE}" -ge 400 ]]; then
|
||||||
|
echo "zoho-curl.sh: HTTP ${HTTP_CODE} on ${API_PATH}" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
@@ -10,6 +10,21 @@ DOLIBARR_URL=https://erp.arcodange.lab
|
|||||||
DOLIBARR_API_KEY=<get from Dolibarr UI: Users → ai_agent → API key>
|
DOLIBARR_API_KEY=<get from Dolibarr UI: Users → ai_agent → API key>
|
||||||
DOLIBARR_USER=ai_agent
|
DOLIBARR_USER=ai_agent
|
||||||
DOLIBARR_PASSWORD=<the ai_agent password, only needed for occasional UI login>
|
DOLIBARR_PASSWORD=<the ai_agent password, only needed for occasional UI login>
|
||||||
|
|
||||||
|
# Required by arcodange-bank-reco only (omit if you only use dolibarr-* skills)
|
||||||
|
QONTO_LOGIN=arcodange-XXXXX
|
||||||
|
QONTO_SECRET_KEY=<from Qonto Settings → Integrations → API>
|
||||||
|
QONTO_ORG_SLUG=arcodange-XXXXX # same as login in most cases
|
||||||
|
WISE_API_TOKEN=<from wise.com/settings/api-tokens>
|
||||||
|
WISE_PROFILE_ID=<numeric id of the BUSINESS profile — bank probe prints it>
|
||||||
|
# Optional: only needed if Wise ever opens the EU statement endpoint
|
||||||
|
WISE_SCA_KEY_PATH=~/.config/arcodange-erp/wise-sca-private.pem
|
||||||
|
|
||||||
|
# Required by arcodange-email-ingest only
|
||||||
|
ZOHO_CLIENT_ID=<from api-console.zoho.com self-client>
|
||||||
|
ZOHO_CLIENT_SECRET=<same>
|
||||||
|
ZOHO_REFRESH_TOKEN=<exchanged from one-time code via /oauth/v2/token>
|
||||||
|
ZOHO_DC=eu # eu | com | in | au
|
||||||
EOF
|
EOF
|
||||||
chmod 600 .claude/skills/dolibarr/.env
|
chmod 600 .claude/skills/dolibarr/.env
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -150,7 +150,9 @@ Not available on this account (intentionally): `/setup/modules` (admin-only), `/
|
|||||||
- Workflow skill for thirdparty completeness audit (any client / supplier): [dolibarr-thirdparty-completeness](../dolibarr-thirdparty-completeness/SKILL.md).
|
- Workflow skill for thirdparty completeness audit (any client / supplier): [dolibarr-thirdparty-completeness](../dolibarr-thirdparty-completeness/SKILL.md).
|
||||||
- Workflow skill for supplier-side TVA déductible (CA3 lignes 19 / 20 / 17+24): [dolibarr-tva-deductible](../dolibarr-tva-deductible/SKILL.md).
|
- Workflow skill for supplier-side TVA déductible (CA3 lignes 19 / 20 / 17+24): [dolibarr-tva-deductible](../dolibarr-tva-deductible/SKILL.md).
|
||||||
- Workflow skill for composite CA3-ready TVA summary (collectée + déductible + net): [dolibarr-tva-summary](../dolibarr-tva-summary/SKILL.md).
|
- Workflow skill for composite CA3-ready TVA summary (collectée + déductible + net): [dolibarr-tva-summary](../dolibarr-tva-summary/SKILL.md).
|
||||||
- Future workflow skills follow the `dolibarr-<topic>` convention. Each one depends on this skill for connection + permissions + endpoint reference; each one keeps its triggers focused on its specific business workflow.
|
- **Bank-side reconciliation** (Qonto + Wise ↔ Dolibarr matching): [arcodange-bank-reco](../arcodange-bank-reco/SKILL.md).
|
||||||
|
- **Email ingestion** (Zoho Mail → supplier-invoice draft for Dolibarr): [arcodange-email-ingest](../arcodange-email-ingest/SKILL.md).
|
||||||
|
- Future workflow skills follow the `dolibarr-<topic>` convention (ERP-internal) or `arcodange-<topic>` (cross-system). Each one depends on this skill for connection + permissions + endpoint reference; each one keeps its triggers focused on its specific business workflow.
|
||||||
|
|
||||||
## Out of scope
|
## Out of scope
|
||||||
|
|
||||||
|
|||||||
@@ -22,7 +22,7 @@ concurrency:
|
|||||||
url: https://vault.arcodange.lab
|
url: https://vault.arcodange.lab
|
||||||
caCertificate: ${{ secrets.HOMELAB_CA_CERT }}
|
caCertificate: ${{ secrets.HOMELAB_CA_CERT }}
|
||||||
jwtGiteaOIDC: ${{ needs.gitea_vault_auth.outputs.gitea_vault_jwt }}
|
jwtGiteaOIDC: ${{ needs.gitea_vault_auth.outputs.gitea_vault_jwt }}
|
||||||
role: gitea_cicd_webapp
|
role: gitea_cicd_erp
|
||||||
method: jwt
|
method: jwt
|
||||||
path: gitea_jwt
|
path: gitea_jwt
|
||||||
secrets: |
|
secrets: |
|
||||||
|
|||||||
@@ -68,6 +68,19 @@ COMMANDS
|
|||||||
|
|
||||||
snapshot [--out FILE|--print-only] Bundle full read-only state into one JSON
|
snapshot [--out FILE|--print-only] Bundle full read-only state into one JSON
|
||||||
|
|
||||||
|
bank Bank-side data (Qonto + Wise) + Dolibarr reconciliation
|
||||||
|
probe Auth + discovery (org slug, profile id, balance ids)
|
||||||
|
qonto-transactions [--month|--since|--until] Qonto transactions table (incoming + outgoing)
|
||||||
|
wise-transactions [--month|--since|--until|--type|--enrich] Wise activities (incoming + outgoing)
|
||||||
|
match [--month|--since|--until|--window-days N|--enrich] Match bank ↔ Dolibarr (split buckets)
|
||||||
|
balance Live balances + Dolibarr cross-check per fk_account
|
||||||
|
curl <qonto|wise> <path> Raw read-only curl through bank-curl.sh
|
||||||
|
|
||||||
|
email Supplier-invoice emails from the Zoho mailbox
|
||||||
|
list [--folder|--limit|--candidates-only|--all-folders] List candidates
|
||||||
|
inspect <messageId> [--folder|--save-pdf|--json] Parse PDFs + draft Dolibarr entry
|
||||||
|
curl <path> Raw read-only curl through zoho-curl.sh
|
||||||
|
|
||||||
whoami GET /users/info — confirm auth
|
whoami GET /users/info — confirm auth
|
||||||
ping GET /status — liveness + Dolibarr version
|
ping GET /status — liveness + Dolibarr version
|
||||||
curl <path> Raw read-only curl through dol-curl.sh
|
curl <path> Raw read-only curl through dol-curl.sh
|
||||||
@@ -195,6 +208,63 @@ EOF
|
|||||||
exec "${SKILLS}/dolibarr-data-snapshot/scripts/snapshot.sh" "$@"
|
exec "${SKILLS}/dolibarr-data-snapshot/scripts/snapshot.sh" "$@"
|
||||||
;;
|
;;
|
||||||
|
|
||||||
|
bank)
|
||||||
|
sub="${1:-help}"; shift || true
|
||||||
|
case "${sub}" in
|
||||||
|
probe) exec "${SKILLS}/arcodange-bank-reco/scripts/bank-probe.sh" "$@" ;;
|
||||||
|
qonto-transactions) exec "${SKILLS}/arcodange-bank-reco/scripts/qonto-transactions.sh" "$@" ;;
|
||||||
|
wise-transactions) exec "${SKILLS}/arcodange-bank-reco/scripts/wise-transactions.sh" "$@" ;;
|
||||||
|
match) exec "${SKILLS}/arcodange-bank-reco/scripts/bank-match.sh" "$@" ;;
|
||||||
|
balance) exec "${SKILLS}/arcodange-bank-reco/scripts/bank-balance.sh" "$@" ;;
|
||||||
|
curl)
|
||||||
|
if [[ $# -lt 2 ]]; then
|
||||||
|
echo "arcodange bank curl: usage: bank curl <qonto|wise> <path>" >&2; exit 2
|
||||||
|
fi
|
||||||
|
exec "${SKILLS}/arcodange-bank-reco/scripts/bank-curl.sh" "$@"
|
||||||
|
;;
|
||||||
|
help|-h|--help)
|
||||||
|
cat <<'EOF'
|
||||||
|
arcodange bank — bank-side data (Qonto + Wise) and Dolibarr reconciliation.
|
||||||
|
|
||||||
|
probe Auth + discovery (org slug, profile id, balance ids)
|
||||||
|
qonto-transactions [--month|--since|--until] Qonto transactions table
|
||||||
|
wise-transactions [--month|--since|--until|--type|--enrich] Wise activities (incoming + outgoing)
|
||||||
|
match [--month|--since|--until|--window-days N] Match bank ↔ Dolibarr (3 buckets)
|
||||||
|
balance Live balances + Dolibarr cross-check per fk_account
|
||||||
|
curl <qonto|wise> <path> Raw read-only curl through bank-curl.sh
|
||||||
|
|
||||||
|
Requires QONTO_LOGIN, QONTO_SECRET_KEY, QONTO_ORG_SLUG, WISE_API_TOKEN,
|
||||||
|
WISE_PROFILE_ID in .env. See arcodange-bank-reco/SKILL.md for setup.
|
||||||
|
EOF
|
||||||
|
;;
|
||||||
|
*) echo "arcodange bank: unknown subcommand '${sub}' (try 'arcodange bank help')" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
;;
|
||||||
|
|
||||||
|
email)
|
||||||
|
sub="${1:-help}"; shift || true
|
||||||
|
case "${sub}" in
|
||||||
|
list) exec "${SKILLS}/arcodange-email-ingest/scripts/email-list.sh" "$@" ;;
|
||||||
|
inspect) exec "${SKILLS}/arcodange-email-ingest/scripts/email-inspect.sh" "$@" ;;
|
||||||
|
curl) exec "${SKILLS}/arcodange-email-ingest/scripts/zoho-curl.sh" "$@" ;;
|
||||||
|
help|-h|--help)
|
||||||
|
cat <<'EOF'
|
||||||
|
arcodange email — supplier-invoice ingestion from the Zoho mailbox.
|
||||||
|
|
||||||
|
list [--folder PATH|--limit N|--candidates-only|--all-folders]
|
||||||
|
List messages (default: /Inbox/books)
|
||||||
|
inspect <messageId> [--folder PATH|--save-pdf DIR|--json]
|
||||||
|
Parse PDF attachments, propose Dolibarr supplier-invoice draft
|
||||||
|
curl <path> Raw read-only call through zoho-curl.sh
|
||||||
|
|
||||||
|
Requires ZOHO_CLIENT_ID, ZOHO_CLIENT_SECRET, ZOHO_REFRESH_TOKEN, ZOHO_DC in .env.
|
||||||
|
See arcodange-email-ingest/SKILL.md for OAuth setup.
|
||||||
|
EOF
|
||||||
|
;;
|
||||||
|
*) echo "arcodange email: unknown subcommand '${sub}' (try 'arcodange email help')" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
;;
|
||||||
|
|
||||||
whoami)
|
whoami)
|
||||||
exec "${DOLC}" /users/info
|
exec "${DOLC}" /users/info
|
||||||
;;
|
;;
|
||||||
|
|||||||
@@ -10,12 +10,12 @@ BEGIN
|
|||||||
WHERE schemaname = 'public'
|
WHERE schemaname = 'public'
|
||||||
LIMIT 1;
|
LIMIT 1;
|
||||||
|
|
||||||
-- Si le propriétaire actuel est différent de erp_role
|
-- Si le propriétaire actuel est différent de {{ .Values.db.ownerRole }}
|
||||||
IF current_schema_owner <> 'erp_role' THEN
|
IF current_schema_owner <> '{{ .Values.db.ownerRole }}' THEN
|
||||||
-- Construire et exécuter la requête REASSIGN OWNED BY
|
-- Construire et exécuter la requête REASSIGN OWNED BY
|
||||||
EXECUTE format('REASSIGN OWNED BY %I TO %I', current_schema_owner, 'erp_role');
|
EXECUTE format('REASSIGN OWNED BY %I TO %I', current_schema_owner, '{{ .Values.db.ownerRole }}');
|
||||||
RAISE NOTICE 'Ownership of all objects in schema "public" has been reassigned from % to %', current_schema_owner, 'erp_role';
|
RAISE NOTICE 'Ownership of all objects in schema "public" has been reassigned from % to %', current_schema_owner, '{{ .Values.db.ownerRole }}';
|
||||||
ELSE
|
ELSE
|
||||||
RAISE NOTICE 'No change needed; the owner of schema "public" is already %', 'erp_role';
|
RAISE NOTICE 'No change needed; the owner of schema "public" is already %', '{{ .Values.db.ownerRole }}';
|
||||||
END IF;
|
END IF;
|
||||||
END $$;
|
END $$;
|
||||||
|
|||||||
@@ -10,8 +10,8 @@ data:
|
|||||||
DOLI_DB_HOST_PORT: !!str 5432
|
DOLI_DB_HOST_PORT: !!str 5432
|
||||||
# DOLI_DB_USER: root
|
# DOLI_DB_USER: root
|
||||||
# DOLI_DB_PASSWORD: root
|
# DOLI_DB_PASSWORD: root
|
||||||
DOLI_DB_NAME: erp
|
DOLI_DB_NAME: {{ .Values.db.name }}
|
||||||
DOLI_URL_ROOT: 'https://erp.arcodange.lab'
|
DOLI_URL_ROOT: 'https://{{ .Values.host }}'
|
||||||
# DOLI_ADMIN_LOGIN: 'admin'
|
# DOLI_ADMIN_LOGIN: 'admin'
|
||||||
# DOLI_ADMIN_PASSWORD: 'admininitialpassword'
|
# DOLI_ADMIN_PASSWORD: 'admininitialpassword'
|
||||||
DOLI_ENABLE_MODULES: Societe,Facture
|
DOLI_ENABLE_MODULES: Societe,Facture
|
||||||
|
|||||||
@@ -7,4 +7,4 @@ data:
|
|||||||
{{- .Files.Get "scripts/update_conf_db_credentials.sh" | nindent 4 }}
|
{{- .Files.Get "scripts/update_conf_db_credentials.sh" | nindent 4 }}
|
||||||
|
|
||||||
update_table_ownership.sql: |
|
update_table_ownership.sql: |
|
||||||
{{- .Files.Get "scripts/update_ownership.sql" | nindent 4 }}
|
{{- tpl (.Files.Get "scripts/update_ownership.sql") . | nindent 4 }}
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ spec:
|
|||||||
method: kubernetes
|
method: kubernetes
|
||||||
mount: kubernetes
|
mount: kubernetes
|
||||||
kubernetes:
|
kubernetes:
|
||||||
role: erp
|
role: {{ .Values.vault.k8sRole }}
|
||||||
serviceAccount: {{ include "erp.serviceAccountName" . }}
|
serviceAccount: {{ include "erp.serviceAccountName" . }}
|
||||||
audiences:
|
audiences:
|
||||||
- vault
|
- vault
|
||||||
@@ -9,7 +9,7 @@ spec:
|
|||||||
mount: postgres
|
mount: postgres
|
||||||
|
|
||||||
# Path to the secret
|
# Path to the secret
|
||||||
path: creds/erp
|
path: {{ .Values.vault.dynamicPath }}
|
||||||
|
|
||||||
# Where to store the secrets, VSO will create the secret
|
# Where to store the secrets, VSO will create the secret
|
||||||
destination:
|
destination:
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ spec:
|
|||||||
mount: kvv2
|
mount: kvv2
|
||||||
|
|
||||||
# path of the secret
|
# path of the secret
|
||||||
path: erp/config
|
path: {{ .Values.vault.staticPath }}
|
||||||
|
|
||||||
# dest k8s secret
|
# dest k8s secret
|
||||||
destination:
|
destination:
|
||||||
|
|||||||
40
chart/values-sandbox.yaml
Normal file
40
chart/values-sandbox.yaml
Normal file
@@ -0,0 +1,40 @@
|
|||||||
|
# Sandbox overlay — to be combined with values.yaml:
|
||||||
|
# helm install erp-sandbox chart/ -f chart/values.yaml -f chart/values-sandbox.yaml \
|
||||||
|
# --namespace erp-sandbox --create-namespace
|
||||||
|
#
|
||||||
|
# Activates Phase D of the multi-env evolution (cf. PR thread). Prerequisites:
|
||||||
|
# - factory/postgres/iac/terraform.tfvars: erp has envs = ["prod", "sandbox"]
|
||||||
|
# - tools/hashicorp-vault/iac/modules/app_roles: env parameter applied
|
||||||
|
# - arcodange-org/erp/iac/main.tf: for_each over local.envs (Phase D commit)
|
||||||
|
# - ArgoCD: Application "erp-sandbox" registered (Phase E)
|
||||||
|
#
|
||||||
|
# Derived names follow the elision rule: env=sandbox → suffix "-sandbox".
|
||||||
|
|
||||||
|
env: sandbox
|
||||||
|
instance: erp-sandbox
|
||||||
|
host: erp-sandbox.arcodange.lab
|
||||||
|
|
||||||
|
db:
|
||||||
|
name: erp-sandbox
|
||||||
|
ownerRole: erp_sandbox_role
|
||||||
|
|
||||||
|
vault:
|
||||||
|
k8sRole: erp-sandbox
|
||||||
|
dynamicPath: creds/erp-sandbox
|
||||||
|
staticPath: erp-sandbox/config
|
||||||
|
|
||||||
|
# Ingress annotations + hosts — override to point at the sandbox FQDN
|
||||||
|
ingress:
|
||||||
|
enabled: true
|
||||||
|
annotations:
|
||||||
|
traefik.ingress.kubernetes.io/router.entrypoints: websecure
|
||||||
|
traefik.ingress.kubernetes.io/router.tls: "true"
|
||||||
|
traefik.ingress.kubernetes.io/router.tls.certresolver: letsencrypt
|
||||||
|
traefik.ingress.kubernetes.io/router.tls.domains.0.main: arcodange.lab
|
||||||
|
traefik.ingress.kubernetes.io/router.tls.domains.0.sans: erp-sandbox.arcodange.lab
|
||||||
|
traefik.ingress.kubernetes.io/router.middlewares: localIp@file
|
||||||
|
hosts:
|
||||||
|
- host: erp-sandbox.arcodange.lab
|
||||||
|
paths:
|
||||||
|
- path: /
|
||||||
|
pathType: Prefix
|
||||||
@@ -2,6 +2,27 @@
|
|||||||
# This is a YAML-formatted file.
|
# This is a YAML-formatted file.
|
||||||
# Declare variables to be passed into your templates.
|
# Declare variables to be passed into your templates.
|
||||||
|
|
||||||
|
# ----------------------------------------------------------------------------
|
||||||
|
# Multi-environment coordinates (default = prod, elision rule applies).
|
||||||
|
# Override in values-<env>.yaml for any non-prod instance — see SKILL.md
|
||||||
|
# of the factory runbook (doc/runbooks/new-web-app/conventions.md).
|
||||||
|
# By the elision rule, env=prod produces names identical to single-env apps;
|
||||||
|
# env=sandbox produces "<app>-sandbox" everywhere except the Postgres owner
|
||||||
|
# role which uses snake-case "<app>_sandbox_role".
|
||||||
|
# ----------------------------------------------------------------------------
|
||||||
|
env: prod
|
||||||
|
instance: erp # derived id: env=prod → erp, else <app>-<env>
|
||||||
|
host: erp.arcodange.lab # internal hostname for this instance
|
||||||
|
|
||||||
|
db:
|
||||||
|
name: erp # PostgreSQL database name (matches factory tfvars)
|
||||||
|
ownerRole: erp_role # Postgres owner role; snake-case <app>_role for prod / <app>_<env>_role for non-prod (matches factory/postgres/iac)
|
||||||
|
|
||||||
|
vault:
|
||||||
|
k8sRole: erp # VaultAuth role (postgres/iac issues this per instance)
|
||||||
|
dynamicPath: creds/erp # path under postgres/ mount for short-lived DB creds
|
||||||
|
staticPath: erp/config # path under kvv2/ mount for the static admin config
|
||||||
|
|
||||||
replicaCount: 1
|
replicaCount: 1
|
||||||
|
|
||||||
image:
|
image:
|
||||||
|
|||||||
58
iac/main.tf
58
iac/main.tf
@@ -1,32 +1,66 @@
|
|||||||
locals {
|
locals {
|
||||||
app = {
|
app = {
|
||||||
name = "erp"
|
name = "erp"
|
||||||
product_name = "dolibarr" # unused
|
product_name = "dolibarr" # unused
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Environments this app is deployed to. By the elision rule (factory runbook
|
||||||
|
# conventions.md / ADR-0002) env=prod renders identical to the single-env
|
||||||
|
# baseline; non-prod envs get a "<name>-<env>" instance id from the module.
|
||||||
|
envs = toset(["prod", "sandbox"])
|
||||||
}
|
}
|
||||||
|
|
||||||
module "app_roles" {
|
module "app_roles" {
|
||||||
source = "git::ssh://git@192.168.1.202:2222/arcodange-org/tools.git//hashicorp-vault/iac/modules/app_roles?depth=1&ref=main"
|
source = "git::ssh://git@192.168.1.202:2222/arcodange-org/tools.git//hashicorp-vault/iac/modules/app_roles?depth=1&ref=main"
|
||||||
name = local.app.name
|
for_each = local.envs
|
||||||
|
name = local.app.name
|
||||||
|
env = each.key
|
||||||
}
|
}
|
||||||
|
|
||||||
resource "random_password" "admin_initial_password" {
|
resource "random_password" "admin_initial_password" {
|
||||||
length = 32
|
for_each = local.envs
|
||||||
|
length = 32
|
||||||
}
|
}
|
||||||
resource "random_uuid" "dolibarr_id" { # used for encryption as well as when buying modules
|
resource "random_uuid" "dolibarr_id" { # used for encryption as well as when buying modules
|
||||||
|
for_each = local.envs
|
||||||
lifecycle {
|
lifecycle {
|
||||||
prevent_destroy = true
|
prevent_destroy = true
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
resource "vault_kv_secret_v2" "dolibarr_admin_setup" {
|
resource "vault_kv_secret_v2" "dolibarr_admin_setup" {
|
||||||
mount = module.app_roles.mount_paths.kvv2
|
for_each = local.envs
|
||||||
name = format("%sconfig", module.app_roles.kvv2_path_prefix)
|
mount = module.app_roles[each.key].mount_paths.kvv2
|
||||||
data_json = jsonencode(
|
name = format("%sconfig", module.app_roles[each.key].kvv2_path_prefix)
|
||||||
{
|
data_json = jsonencode(
|
||||||
DOLI_ADMIN_LOGIN = "admin",
|
{
|
||||||
DOLI_ADMIN_PASSWORD = random_password.admin_initial_password.result
|
DOLI_ADMIN_LOGIN = "admin",
|
||||||
DOLI_INSTANCE_UNIQUE_ID = random_uuid.dolibarr_id.result
|
DOLI_ADMIN_PASSWORD = random_password.admin_initial_password[each.key].result
|
||||||
}
|
DOLI_INSTANCE_UNIQUE_ID = random_uuid.dolibarr_id[each.key].result
|
||||||
|
}
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# State migration (ADR-0002 Phase D): re-key the pre-existing single-env resources
|
||||||
|
# into the for_each map under the "prod" key so that introducing for_each does NOT
|
||||||
|
# plan a destroy+create. This is critical for random_uuid.dolibarr_id — it carries
|
||||||
|
# prevent_destroy = true (it is the prod Dolibarr encryption id + paid-module
|
||||||
|
# binding), so a missing moved block would HARD-FAIL the apply rather than silently
|
||||||
|
# losing it. The module's own internal moved (role -> role[0]) chains with the
|
||||||
|
# module re-key here.
|
||||||
|
moved {
|
||||||
|
from = module.app_roles
|
||||||
|
to = module.app_roles["prod"]
|
||||||
|
}
|
||||||
|
moved {
|
||||||
|
from = random_password.admin_initial_password
|
||||||
|
to = random_password.admin_initial_password["prod"]
|
||||||
|
}
|
||||||
|
moved {
|
||||||
|
from = random_uuid.dolibarr_id
|
||||||
|
to = random_uuid.dolibarr_id["prod"]
|
||||||
|
}
|
||||||
|
moved {
|
||||||
|
from = vault_kv_secret_v2.dolibarr_admin_setup
|
||||||
|
to = vault_kv_secret_v2.dolibarr_admin_setup["prod"]
|
||||||
|
}
|
||||||
|
|||||||
60
ops/sandbox/README.md
Normal file
60
ops/sandbox/README.md
Normal file
@@ -0,0 +1,60 @@
|
|||||||
|
# erp-sandbox lifecycle ops
|
||||||
|
|
||||||
|
Tooling to make `erp-sandbox` **iso-prod** and to reset it, implementing
|
||||||
|
[ADR-0003](https://gitea.arcodange.lab/arcodange-org/factory/src/branch/main/vibe/ADR/0003-sandbox-state-lifecycle.md)
|
||||||
|
(sandbox state lifecycle). The sandbox exists so AI agents can rehearse Dolibarr
|
||||||
|
**write** operations against a faithful copy of prod, with a structural guarantee
|
||||||
|
that the rehearsal path can never mutate prod.
|
||||||
|
|
||||||
|
## The prod-integrity guarantee (why this is safe)
|
||||||
|
|
||||||
|
| Layer | Enforcement |
|
||||||
|
| --- | --- |
|
||||||
|
| prod is **read-only** during a refresh | `pg_dump` runs with `default_transaction_read_only=on` |
|
||||||
|
| the restore can only write the sandbox | it uses the sandbox's own dynamic creds — a member of `erp_sandbox_role`, which **owns only `erp-sandbox`** |
|
||||||
|
| no database is dropped/created | wipe is `DROP OWNED BY erp_sandbox_role CASCADE`; reload is `pg_restore` (no `CREATEDB`, no superuser) |
|
||||||
|
| prod is structurally undroppable | `DROP DATABASE` needs ownership; `erp_sandbox_role` does not own prod `erp` (owned by `erp_role`) |
|
||||||
|
|
||||||
|
The only prod-capable credential on the platform is the `superuser=true` provider
|
||||||
|
in `factory postgres/iac/providers.tf`, used **only** in the human-gated
|
||||||
|
`postgres.yaml` CI. This tooling never touches it.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
```sh
|
||||||
|
./sandbox-lifecycle.sh refresh-from-prod # clone prod DB (data + config) into erp-sandbox
|
||||||
|
./sandbox-lifecycle.sh sync-documents # copy mycompany/ uploads (company logo, PDFs)
|
||||||
|
./sandbox-lifecycle.sh refresh # both, in order
|
||||||
|
```
|
||||||
|
|
||||||
|
`refresh-from-prod` scales the sandbox pod to 0, dumps the full prod `public`
|
||||||
|
schema (read-only), wipes the sandbox's app objects, restores, and scales back
|
||||||
|
up. It dumps the **whole** schema (not just `llx_*`) so app helper functions and
|
||||||
|
their triggers (e.g. `update_modified_column_tms()`) come over; it filters out
|
||||||
|
the provisioner-owned `user_lookup` pgbouncer function from the restore TOC
|
||||||
|
because that object already exists per-environment and is not app data.
|
||||||
|
|
||||||
|
## Two fidelity caveats (by design — see ADR-0003)
|
||||||
|
|
||||||
|
1. **Encryption.** Dolibarr ties some encrypted fields (notably API keys) to
|
||||||
|
`DOLI_INSTANCE_UNIQUE_ID`. The sandbox has its **own** uuid, so prod-encrypted
|
||||||
|
values won't decrypt there. This is why the write-scoped `ai_agent_sandbox`
|
||||||
|
API key must be **generated inside the sandbox** (see `../../test/` POC),
|
||||||
|
not copied from prod. Most data is plaintext and unaffected.
|
||||||
|
2. **Uploaded files live on the PVC, not the DB.** A DB refresh copies the logo
|
||||||
|
*const* (`MAIN_INFO_SOCIETE_LOGO`) but not the image; `sync-documents` copies
|
||||||
|
the `documents/mycompany` tree so the logo + attachments actually render.
|
||||||
|
|
||||||
|
## BDD reset loop (E4)
|
||||||
|
|
||||||
|
For repeated rehearsals, `refresh-from-prod` is the "reset to prod state". A
|
||||||
|
faster checkpoint/reset that avoids re-reading prod each time (cache a golden
|
||||||
|
dump on a small PVC, then `DROP OWNED + pg_restore` from it) is the documented
|
||||||
|
next optimization — see ADR-0003 §Decision/Consequences.
|
||||||
|
|
||||||
|
## Hardening backlog
|
||||||
|
|
||||||
|
- Replace the transient copy of prod's read+write creds with a **dedicated
|
||||||
|
read-only Postgres role** (issued via a Vault dynamic role) so the dump path is
|
||||||
|
least-privilege by construction, not just by `default_transaction_read_only`.
|
||||||
|
- Provision a golden-cache PVC for fast BDD resets.
|
||||||
134
ops/sandbox/sandbox-lifecycle.sh
Executable file
134
ops/sandbox/sandbox-lifecycle.sh
Executable file
@@ -0,0 +1,134 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
#
|
||||||
|
# sandbox-lifecycle.sh — seed / refresh the erp-sandbox Dolibarr from prod, and
|
||||||
|
# sync its uploaded documents, with prod integrity guaranteed structurally.
|
||||||
|
#
|
||||||
|
# Implements ADR-0003 (factory vibe/ADR/0003-sandbox-state-lifecycle.md):
|
||||||
|
# - prod is read ONLY (pg_dump runs in a default_transaction_read_only session);
|
||||||
|
# - the restore writes ONLY to erp-sandbox, using the sandbox's own dynamic
|
||||||
|
# credentials (a member of erp_sandbox_role, which owns only the sandbox DB),
|
||||||
|
# so it is structurally incapable of touching prod 'erp' (owned by erp_role);
|
||||||
|
# - no DROP/CREATE DATABASE, no CREATEDB, no superuser — wipe is
|
||||||
|
# `DROP OWNED BY erp_sandbox_role CASCADE`, reload is `pg_restore`.
|
||||||
|
#
|
||||||
|
# The only prod-capable credential on the platform is the superuser provider in
|
||||||
|
# factory postgres/iac, exercised solely in the human-gated postgres.yaml CI —
|
||||||
|
# this script never uses it.
|
||||||
|
#
|
||||||
|
# Requires: kubectl (context on the lab cluster), and a postgres:16 image
|
||||||
|
# reachable by the cluster. Run from anywhere.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ./sandbox-lifecycle.sh refresh-from-prod # iso-prod seed: prod DB -> erp-sandbox
|
||||||
|
# ./sandbox-lifecycle.sh sync-documents # copy mycompany/ uploads (logo, PDFs)
|
||||||
|
# ./sandbox-lifecycle.sh refresh # refresh-from-prod + sync-documents
|
||||||
|
#
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
PROD_NS="erp"
|
||||||
|
SB_NS="erp-sandbox"
|
||||||
|
PROD_DB="erp"
|
||||||
|
SB_DB="erp-sandbox"
|
||||||
|
SB_ROLE="erp_sandbox_role" # snake-case owner role (ADR-0002 elision rule)
|
||||||
|
PGHOST="192.168.1.202" # direct Postgres (NOT pgbouncer — pooler breaks pg_dump)
|
||||||
|
PG_IMAGE="postgres:16-alpine"
|
||||||
|
DOC_ROOT="/var/www/documents" # dolibarr_main_data_root
|
||||||
|
TMP_PROD_SECRET="prod-db-ro-temp" # transient copy of prod creds, deleted on exit
|
||||||
|
|
||||||
|
log() { printf '\033[1;36m==>\033[0m %s\n' "$*"; }
|
||||||
|
die() { printf '\033[1;31mABORT:\033[0m %s\n' "$*" >&2; exit 1; }
|
||||||
|
|
||||||
|
sb_pod() { kubectl get pod -n "$SB_NS" -l app.kubernetes.io/instance=erp-sandbox -o name 2>/dev/null | head -1; }
|
||||||
|
prod_pod() { kubectl get pod -n "$PROD_NS" -l app.kubernetes.io/instance=erp -o name 2>/dev/null | head -1; }
|
||||||
|
|
||||||
|
cleanup_secret() { kubectl delete secret "$TMP_PROD_SECRET" -n "$SB_NS" --ignore-not-found >/dev/null 2>&1 || true; }
|
||||||
|
|
||||||
|
refresh_from_prod() {
|
||||||
|
command -v python3 >/dev/null || die "python3 required to copy the prod secret without exposing it"
|
||||||
|
trap cleanup_secret EXIT
|
||||||
|
|
||||||
|
log "Copying prod DB creds into a transient, read-only-intent secret in $SB_NS (values stay base64)"
|
||||||
|
kubectl get secret vso-db-credentials -n "$PROD_NS" -o json \
|
||||||
|
| python3 -c "import json,sys; d=json.load(sys.stdin); d['metadata']={'name':'$TMP_PROD_SECRET','namespace':'$SB_NS'}; d.pop('status',None); d['data']={k:d['data'][k] for k in ('username','password')}; print(json.dumps(d))" \
|
||||||
|
| kubectl apply -f - >/dev/null
|
||||||
|
|
||||||
|
log "Scaling erp-sandbox to 0 (exclusive DB access for the restore)"
|
||||||
|
kubectl scale deploy erp-sandbox -n "$SB_NS" --replicas=0 >/dev/null
|
||||||
|
kubectl wait --for=delete pod -l app.kubernetes.io/instance=erp-sandbox -n "$SB_NS" --timeout=120s >/dev/null 2>&1 || true
|
||||||
|
|
||||||
|
log "Running the seed Job (pg_dump prod read-only -> DROP OWNED -> pg_restore into sandbox)"
|
||||||
|
kubectl delete job sandbox-seed -n "$SB_NS" --ignore-not-found >/dev/null 2>&1 || true
|
||||||
|
kubectl apply -f - >/dev/null <<EOF
|
||||||
|
apiVersion: batch/v1
|
||||||
|
kind: Job
|
||||||
|
metadata: { name: sandbox-seed, namespace: $SB_NS }
|
||||||
|
spec:
|
||||||
|
backoffLimit: 0
|
||||||
|
ttlSecondsAfterFinished: 900
|
||||||
|
template:
|
||||||
|
spec:
|
||||||
|
restartPolicy: Never
|
||||||
|
containers:
|
||||||
|
- name: seed
|
||||||
|
image: $PG_IMAGE
|
||||||
|
env:
|
||||||
|
- { name: PROD_PGUSER, valueFrom: { secretKeyRef: { name: $TMP_PROD_SECRET, key: username } } }
|
||||||
|
- { name: PROD_PGPASSWORD, valueFrom: { secretKeyRef: { name: $TMP_PROD_SECRET, key: password } } }
|
||||||
|
- { name: SB_PGUSER, valueFrom: { secretKeyRef: { name: vso-db-credentials, key: username } } }
|
||||||
|
- { name: SB_PGPASSWORD, valueFrom: { secretKeyRef: { name: vso-db-credentials, key: password } } }
|
||||||
|
- { name: PGHOST, value: "$PGHOST" }
|
||||||
|
- { name: PGSSLMODE, value: "disable" }
|
||||||
|
command: ["/bin/sh","-c"]
|
||||||
|
args:
|
||||||
|
- |
|
||||||
|
set -eu
|
||||||
|
SBDB=\$(PGPASSWORD=\$SB_PGPASSWORD psql -h "\$PGHOST" -U "\$SB_PGUSER" -d $SB_DB -tAc 'select current_database()')
|
||||||
|
[ "\$SBDB" = "$SB_DB" ] || { echo "ABORT: target is '\$SBDB' not $SB_DB"; exit 1; }
|
||||||
|
echo "source=$PROD_DB (read-only) target=$SB_DB ok"
|
||||||
|
# 1. dump prod — full public schema (incl. helper functions + triggers), read-only session
|
||||||
|
PGPASSWORD=\$PROD_PGPASSWORD PGOPTIONS='-c default_transaction_read_only=on' \\
|
||||||
|
pg_dump -h "\$PGHOST" -U "\$PROD_PGUSER" -d $PROD_DB -n public -Fc -f /tmp/golden.dump
|
||||||
|
# drop provisioner-owned infra (pgbouncer user_lookup) from the TOC: it already
|
||||||
|
# exists in the sandbox and is not app data, so restoring it conflicts.
|
||||||
|
pg_restore -l /tmp/golden.dump | grep -vi 'user_lookup' > /tmp/golden.toc
|
||||||
|
echo "dump: \$(ls -l /tmp/golden.dump | awk '{print \$5}') bytes, tables=\$(grep -c 'TABLE DATA ' /tmp/golden.toc)"
|
||||||
|
# 2. wipe sandbox app objects (everything owned by the app role; infra untouched)
|
||||||
|
PGPASSWORD=\$SB_PGPASSWORD psql -h "\$PGHOST" -U "\$SB_PGUSER" -d $SB_DB -v ON_ERROR_STOP=1 \\
|
||||||
|
-c "DROP OWNED BY $SB_ROLE CASCADE;"
|
||||||
|
# 3. restore golden, owned by the sandbox role
|
||||||
|
PGPASSWORD=\$SB_PGPASSWORD \\
|
||||||
|
pg_restore -L /tmp/golden.toc --no-owner --role=$SB_ROLE -d $SB_DB /tmp/golden.dump \\
|
||||||
|
&& echo "restore: clean" || echo "restore: completed with ignorable warnings"
|
||||||
|
# 4. verify
|
||||||
|
Q() { PGPASSWORD=\$SB_PGPASSWORD psql -h "\$PGHOST" -U "\$SB_PGUSER" -d $SB_DB -tAc "\$1"; }
|
||||||
|
echo "llx tables=\$(Q "select count(*) from pg_tables where schemaname='public' and tablename like 'llx_%'") company=\$(Q "select value from llx_const where name='MAIN_INFO_SOCIETE_NOM'") lang=\$(Q "select value from llx_const where name='MAIN_LANG_DEFAULT'") owner=\$(Q "select tableowner from pg_tables where tablename='llx_societe'")"
|
||||||
|
echo "DONE."
|
||||||
|
EOF
|
||||||
|
kubectl wait --for=condition=complete job/sandbox-seed -n "$SB_NS" --timeout=300s >/dev/null 2>&1 \
|
||||||
|
|| die "seed Job did not complete — see: kubectl logs -n $SB_NS job/sandbox-seed"
|
||||||
|
kubectl logs -n "$SB_NS" job/sandbox-seed | sed 's/^/ /'
|
||||||
|
kubectl delete job sandbox-seed -n "$SB_NS" --ignore-not-found >/dev/null 2>&1 || true
|
||||||
|
|
||||||
|
log "Scaling erp-sandbox back to 1"
|
||||||
|
kubectl scale deploy erp-sandbox -n "$SB_NS" --replicas=1 >/dev/null
|
||||||
|
cleanup_secret; trap - EXIT
|
||||||
|
log "Refresh complete. Run 'sync-documents' to also copy the company logo + uploads."
|
||||||
|
}
|
||||||
|
|
||||||
|
sync_documents() {
|
||||||
|
local pp sp
|
||||||
|
pp=$(prod_pod); sp=$(sb_pod)
|
||||||
|
[ -n "$pp" ] || die "no prod erp pod found"
|
||||||
|
[ -n "$sp" ] || die "no erp-sandbox pod found"
|
||||||
|
log "Syncing $DOC_ROOT/mycompany (logo + uploads) ${pp##*/} -> ${sp##*/} via tar pipe"
|
||||||
|
kubectl exec -n "$PROD_NS" "${pp#pod/}" -- tar -C "$DOC_ROOT" -cf - mycompany 2>/dev/null \
|
||||||
|
| kubectl exec -i -n "$SB_NS" "${sp#pod/}" -- tar -C "$DOC_ROOT" -xf -
|
||||||
|
log "Documents synced. (For a one-shot logo only, scope the tar to mycompany/logos.)"
|
||||||
|
}
|
||||||
|
|
||||||
|
case "${1:-}" in
|
||||||
|
refresh-from-prod) refresh_from_prod ;;
|
||||||
|
sync-documents) sync_documents ;;
|
||||||
|
refresh) refresh_from_prod; sync_documents ;;
|
||||||
|
*) echo "usage: $0 {refresh-from-prod|sync-documents|refresh}" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
@@ -1,5 +1,27 @@
|
|||||||
|
# Copy this template to one of:
|
||||||
|
# .env — production target, loaded by main.ts
|
||||||
|
# .env.sandbox — sandbox target, loaded by provisionSandbox.ts
|
||||||
|
# Both are gitignored. Never commit real secret values.
|
||||||
|
|
||||||
|
# --- Target ---
|
||||||
|
# prod: https://erp.arcodange.lab (.env)
|
||||||
|
# sandbox: https://erp-sandbox.arcodange.lab (.env.sandbox)
|
||||||
DOLIBARR_ADDRESS=https://erp.arcodange.lab
|
DOLIBARR_ADDRESS=https://erp.arcodange.lab
|
||||||
DOLI_DB_PASSWORD=
|
|
||||||
DOLI_ADMIN_LOGIN=admin
|
DOLI_ADMIN_LOGIN=admin
|
||||||
DOLI_ADMIN_PASSWORD=""
|
DOLI_ADMIN_PASSWORD=""
|
||||||
|
DOLI_DB_PASSWORD=""
|
||||||
ROOT_FOLDER=$HOME/erp
|
ROOT_FOLDER=$HOME/erp
|
||||||
|
|
||||||
|
# Populate the passwords from the cluster secrets, e.g. (prod shown):
|
||||||
|
# DOLI_ADMIN_PASSWORD <- kubectl get secret secretkv -n erp -o jsonpath='{.data.DOLI_ADMIN_PASSWORD}' | base64 -d
|
||||||
|
# DOLI_DB_PASSWORD <- kubectl get secret vso-db-credentials -n erp -o jsonpath='{.data.password}' | base64 -d
|
||||||
|
#
|
||||||
|
# NOTE for a sandbox SEEDED from prod (ops/sandbox/sandbox-lifecycle.sh): the seed
|
||||||
|
# clones prod's admin password into the sandbox, so .env.sandbox's
|
||||||
|
# DOLI_ADMIN_PASSWORD must be PROD's admin password (-n erp), not the sandbox
|
||||||
|
# secretkv. The DB password is the sandbox's own (-n erp-sandbox).
|
||||||
|
|
||||||
|
# Optional: fix the provisioned user's password (else one is generated and only
|
||||||
|
# the API key is emitted to .ai_agent_sandbox.key).
|
||||||
|
# AI_AGENT_SANDBOX_PASSWORD=""
|
||||||
|
|||||||
6
test/.gitignore
vendored
Normal file
6
test/.gitignore
vendored
Normal file
@@ -0,0 +1,6 @@
|
|||||||
|
# Secrets — never commit. Covers .env (prod, main.ts) and .env.sandbox
|
||||||
|
# (sandbox, provisionSandbox.ts), plus any generated *.key.
|
||||||
|
.env*
|
||||||
|
!.env.example
|
||||||
|
.ai_agent_sandbox.key
|
||||||
|
*.key
|
||||||
104
test/README.md
Normal file
104
test/README.md
Normal file
@@ -0,0 +1,104 @@
|
|||||||
|
# test — Dolibarr UI automation (Deno + Playwright)
|
||||||
|
|
||||||
|
A small Deno + Playwright POC that drives the Dolibarr admin UI in the `fr-FR`
|
||||||
|
locale. Playwright fills the same forms a human admin would, so the automation
|
||||||
|
works even where the REST API can't (e.g. generating an API key, which is
|
||||||
|
encrypted with the instance's own `DOLI_INSTANCE_UNIQUE_ID`).
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
- `main.ts` — original entrypoint (first install, company/display/module setup).
|
||||||
|
- `provisionSandbox.ts` — entrypoint that provisions the `erp-sandbox` instance
|
||||||
|
for the AI agent (enable REST API, create a write-scoped user, generate its
|
||||||
|
API key).
|
||||||
|
- `scripts/login.ts` — admin login / logout / whoami helpers.
|
||||||
|
- `scripts/forms.ts` — `fillForm`, `toggleOnOff`, CKEditor/ACE helpers.
|
||||||
|
- `scripts/admin/moduleSetup.ts` — `configureModule`, `enableApiModule`.
|
||||||
|
- `scripts/admin/userSetup.ts` — `createUser`, `assignRights`, `generateApiKey`.
|
||||||
|
|
||||||
|
## Configure
|
||||||
|
|
||||||
|
Copy `.env.example` to `.env` and fill it in. `.env`, `*.key`, and
|
||||||
|
`.ai_agent_sandbox.key` are gitignored — never commit secrets.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cp .env.example .env
|
||||||
|
```
|
||||||
|
|
||||||
|
## Lock the installer (after a fresh install via `main.ts`)
|
||||||
|
|
||||||
|
Dolibarr keeps its web installer reachable until an `install.lock` file exists.
|
||||||
|
After a fresh install (the `main.ts` flow), create it in the target pod — for the
|
||||||
|
sandbox:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
kubectl -n erp-sandbox exec \
|
||||||
|
"$(kubectl get pod -n erp-sandbox -l app.kubernetes.io/instance=erp-sandbox -o name)" -- \
|
||||||
|
/bin/sh -c 'touch /var/www/html/install.lock && chown www-data:www-data /var/www/html/install.lock'
|
||||||
|
```
|
||||||
|
|
||||||
|
For prod, swap to `-n erp -l app.kubernetes.io/instance=erp`. Not needed when the
|
||||||
|
instance was seeded from a prod dump instead of freshly installed — see
|
||||||
|
`../ops/sandbox/`.
|
||||||
|
|
||||||
|
## Provision the sandbox
|
||||||
|
|
||||||
|
Provisions `erp-sandbox.arcodange.lab`: enables the REST API module, creates the
|
||||||
|
write-scoped `ai_agent_sandbox` user, grants it its write rights, and has
|
||||||
|
Dolibarr generate the user's API key. The key is written to
|
||||||
|
`test/.ai_agent_sandbox.key` (gitignored) — it is never printed.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cd test
|
||||||
|
deno run --allow-all provisionSandbox.ts
|
||||||
|
```
|
||||||
|
|
||||||
|
Populate `.env` from the `erp-sandbox` namespace secrets first. `secretkv`
|
||||||
|
carries the app env (including `DOLI_ADMIN_PASSWORD`); `vso-db-credentials`
|
||||||
|
carries the database password:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# Admin password (key DOLI_ADMIN_PASSWORD inside the secretkv secret)
|
||||||
|
kubectl get secret secretkv -n erp-sandbox \
|
||||||
|
-o jsonpath='{.data.DOLI_ADMIN_PASSWORD}' | base64 -d
|
||||||
|
|
||||||
|
# Database password (key `password` inside vso-db-credentials)
|
||||||
|
kubectl get secret vso-db-credentials -n erp-sandbox \
|
||||||
|
-o jsonpath='{.data.password}' | base64 -d
|
||||||
|
```
|
||||||
|
|
||||||
|
Set in `.env`:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
DOLIBARR_ADDRESS=https://erp-sandbox.arcodange.lab
|
||||||
|
DOLI_ADMIN_LOGIN=admin
|
||||||
|
DOLI_ADMIN_PASSWORD="<from secretkv above>"
|
||||||
|
DOLI_DB_PASSWORD="<from vso-db-credentials above>"
|
||||||
|
# Optional — otherwise a random password is generated and only the API key emitted:
|
||||||
|
# AI_AGENT_SANDBOX_PASSWORD="<choose one>"
|
||||||
|
```
|
||||||
|
|
||||||
|
### After it runs
|
||||||
|
|
||||||
|
The generated API key lands in `test/.ai_agent_sandbox.key`. Next step (not
|
||||||
|
automated by this POC): load it into the `dolibarr` skill's sandbox config /
|
||||||
|
Vault at `kvv2/erp-sandbox/ai_agent`.
|
||||||
|
|
||||||
|
> [!IMPORTANT]
|
||||||
|
> The sandbox Dolibarr is not installed/provisioned yet (empty DB, fresh install
|
||||||
|
> wizard). Until the install wizard has been completed against the sandbox,
|
||||||
|
> `provisionSandbox.ts` will not have a UI to drive, and the selectors in
|
||||||
|
> `moduleSetup.ts` / `userSetup.ts` are best-effort (Dolibarr 22 conventions,
|
||||||
|
> not verified live). Confirm them on the first real run.
|
||||||
|
|
||||||
|
### Write rights granted
|
||||||
|
|
||||||
|
The `ai_agent_sandbox` user is created non-admin and granted read + create on:
|
||||||
|
|
||||||
|
| Module | rights ids |
|
||||||
|
| ---------------- | ---------------------------------- |
|
||||||
|
| facture | lire=11, creer=12 |
|
||||||
|
| societe | lire=121, creer=122 |
|
||||||
|
| societe contact | lire=281, creer=282 |
|
||||||
|
| fournisseur | lire=1181, facture lire=1231, facture creer=1232 |
|
||||||
|
| produit | lire=31, creer=32 |
|
||||||
26
test/deno.lock
generated
26
test/deno.lock
generated
@@ -1,7 +1,8 @@
|
|||||||
{
|
{
|
||||||
"version": "4",
|
"version": "5",
|
||||||
"specifiers": {
|
"specifiers": {
|
||||||
"jsr:@std/dotenv@*": "0.225.2",
|
"jsr:@std/dotenv@*": "0.225.2",
|
||||||
|
"npm:@types/node@*": "24.2.0",
|
||||||
"npm:playwright@^1.48.2": "1.48.2"
|
"npm:playwright@^1.48.2": "1.48.2"
|
||||||
},
|
},
|
||||||
"jsr": {
|
"jsr": {
|
||||||
@@ -10,18 +11,33 @@
|
|||||||
}
|
}
|
||||||
},
|
},
|
||||||
"npm": {
|
"npm": {
|
||||||
|
"@types/node@24.2.0": {
|
||||||
|
"integrity": "sha512-3xyG3pMCq3oYCNg7/ZP+E1ooTaGB4cG8JWRsqqOYQdbWNY4zbaV0Ennrd7stjiJEFZCaybcIgpTjJWHRfBSIDw==",
|
||||||
|
"dependencies": [
|
||||||
|
"undici-types"
|
||||||
|
]
|
||||||
|
},
|
||||||
"fsevents@2.3.2": {
|
"fsevents@2.3.2": {
|
||||||
"integrity": "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA=="
|
"integrity": "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==",
|
||||||
|
"os": ["darwin"],
|
||||||
|
"scripts": true
|
||||||
},
|
},
|
||||||
"playwright-core@1.48.2": {
|
"playwright-core@1.48.2": {
|
||||||
"integrity": "sha512-sjjw+qrLFlriJo64du+EK0kJgZzoQPsabGF4lBvsid+3CNIZIYLgnMj9V6JY5VhM2Peh20DJWIVpVljLLnlawA=="
|
"integrity": "sha512-sjjw+qrLFlriJo64du+EK0kJgZzoQPsabGF4lBvsid+3CNIZIYLgnMj9V6JY5VhM2Peh20DJWIVpVljLLnlawA==",
|
||||||
|
"bin": true
|
||||||
},
|
},
|
||||||
"playwright@1.48.2": {
|
"playwright@1.48.2": {
|
||||||
"integrity": "sha512-NjYvYgp4BPmiwfe31j4gHLa3J7bD2WiBz8Lk2RoSsmX38SVIARZ18VYjxLjAcDsAhA+F4iSEXTSGgjua0rrlgQ==",
|
"integrity": "sha512-NjYvYgp4BPmiwfe31j4gHLa3J7bD2WiBz8Lk2RoSsmX38SVIARZ18VYjxLjAcDsAhA+F4iSEXTSGgjua0rrlgQ==",
|
||||||
"dependencies": [
|
"dependencies": [
|
||||||
"fsevents",
|
|
||||||
"playwright-core"
|
"playwright-core"
|
||||||
]
|
],
|
||||||
|
"optionalDependencies": [
|
||||||
|
"fsevents"
|
||||||
|
],
|
||||||
|
"bin": true
|
||||||
|
},
|
||||||
|
"undici-types@7.10.0": {
|
||||||
|
"integrity": "sha512-t5Fy/nfn+14LuOc2KNYg75vZqClpAiqscVvMygNnlsHBFpSXdJaYtXMcdNLpl/Qvc3P2cB3s6lOV51nqsFq4ag=="
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"workspace": {
|
"workspace": {
|
||||||
|
|||||||
149
test/provisionSandbox.ts
Normal file
149
test/provisionSandbox.ts
Normal file
@@ -0,0 +1,149 @@
|
|||||||
|
import { loadSync } from "jsr:@std/dotenv";
|
||||||
|
// Sandbox provisioning loads its OWN .env.sandbox; prod config stays in .env (main.ts).
|
||||||
|
loadSync({ envPath: ".env.sandbox", export: true });
|
||||||
|
import { chromium } from "playwright";
|
||||||
|
import path from "node:path";
|
||||||
|
import login from "./scripts/login.ts";
|
||||||
|
import moduleSetup from "./scripts/admin/moduleSetup.ts";
|
||||||
|
import userSetup from "./scripts/admin/userSetup.ts";
|
||||||
|
|
||||||
|
/*
|
||||||
|
provisionSandbox.ts — separate entrypoint (does NOT touch main.ts).
|
||||||
|
|
||||||
|
Provisions the erp-sandbox Dolibarr so the AI agent can write to it:
|
||||||
|
1. enable the REST API / Web services module,
|
||||||
|
2. create a write-scoped `ai_agent_sandbox` user (non-admin),
|
||||||
|
3. grant it the write rights it needs,
|
||||||
|
4. have Dolibarr generate its API key and emit it safely to
|
||||||
|
test/.ai_agent_sandbox.key (gitignored).
|
||||||
|
|
||||||
|
Run:
|
||||||
|
cd test && deno run --allow-all provisionSandbox.ts
|
||||||
|
|
||||||
|
CANNOT be run end-to-end yet: the sandbox Dolibarr is not installed/provisioned
|
||||||
|
(empty DB, fresh install wizard). Selector correctness in moduleSetup.ts /
|
||||||
|
userSetup.ts is therefore best-effort and must be verified on the first real
|
||||||
|
run, AFTER the install wizard has been completed against the sandbox.
|
||||||
|
|
||||||
|
NEXT STEP (not implemented here): load the generated key into the dolibarr
|
||||||
|
skill's sandbox config / Vault at kvv2/erp-sandbox/ai_agent. We intentionally
|
||||||
|
do NOT write to Vault from this POC.
|
||||||
|
*/
|
||||||
|
|
||||||
|
/*
|
||||||
|
Write rights to grant `ai_agent_sandbox` (stable Dolibarr rights ids, verified
|
||||||
|
against prod Dolibarr 22.0.4). Each module's read (lire) + create (creer):
|
||||||
|
facture: lire=11, creer=12
|
||||||
|
societe: lire=121, creer=122
|
||||||
|
societe contact: lire=281, creer=282
|
||||||
|
fournisseur: lire=1181, facture lire=1231, facture creer=1232
|
||||||
|
produit: lire=31, creer=32
|
||||||
|
*/
|
||||||
|
const WRITE_IDS = [
|
||||||
|
11, // facture lire
|
||||||
|
12, // facture creer
|
||||||
|
121, // societe lire
|
||||||
|
122, // societe creer
|
||||||
|
281, // societe contact lire
|
||||||
|
282, // societe contact creer
|
||||||
|
1181, // fournisseur lire
|
||||||
|
1231, // fournisseur facture lire
|
||||||
|
1232, // fournisseur facture creer
|
||||||
|
31, // produit lire
|
||||||
|
32, // produit creer
|
||||||
|
];
|
||||||
|
|
||||||
|
const KEY_FILE = ".ai_agent_sandbox.key";
|
||||||
|
|
||||||
|
/*
|
||||||
|
Initialisation — mirrors main.ts but targeted at the sandbox. The default
|
||||||
|
address points at the sandbox; admin creds come from env (the same
|
||||||
|
DOLI_ADMIN_LOGIN / DOLI_ADMIN_PASSWORD vars main.ts uses, populated from the
|
||||||
|
erp-sandbox namespace secrets — see test/README.md "Provision the sandbox").
|
||||||
|
*/
|
||||||
|
const dolibarrAddress = Deno.env.get("DOLIBARR_ADDRESS") ||
|
||||||
|
"https://erp-sandbox.arcodange.lab";
|
||||||
|
const debug = true;
|
||||||
|
const DBpassword = Deno.env.get("DOLI_DB_PASSWORD") || "undefined";
|
||||||
|
const adminCredentials = {
|
||||||
|
username: Deno.env.get("DOLI_ADMIN_LOGIN") || "admin",
|
||||||
|
password: Deno.env.get("DOLI_ADMIN_PASSWORD") || "undefined",
|
||||||
|
};
|
||||||
|
|
||||||
|
// The new user's login is fixed; its password comes from env or is generated.
|
||||||
|
const AI_AGENT_LOGIN = "ai_agent_sandbox";
|
||||||
|
const aiAgentPassword = Deno.env.get("AI_AGENT_SANDBOX_PASSWORD") ||
|
||||||
|
generatePassword();
|
||||||
|
|
||||||
|
const rootFolderPath = Deno.env.get("ROOT_FOLDER") ||
|
||||||
|
path.join(Deno.cwd(), "..");
|
||||||
|
const imgFolderPath = Deno.env.get("IMG_FOLDER") ||
|
||||||
|
path.join(rootFolderPath, "static/img");
|
||||||
|
const configFolderPath = Deno.env.get("CONFIG_FOLDER") ||
|
||||||
|
path.join(rootFolderPath, "static/config");
|
||||||
|
|
||||||
|
/* Generate a reasonably strong random password (used only if none provided). */
|
||||||
|
function generatePassword(): string {
|
||||||
|
const bytes = new Uint8Array(18);
|
||||||
|
crypto.getRandomValues(bytes);
|
||||||
|
// URL-safe base64 → no shell-hostile characters.
|
||||||
|
return btoa(String.fromCharCode(...bytes))
|
||||||
|
.replace(/\+/g, "-")
|
||||||
|
.replace(/\//g, "_")
|
||||||
|
.replace(/=+$/, "");
|
||||||
|
}
|
||||||
|
|
||||||
|
const browser = await chromium.launch({
|
||||||
|
headless: false,
|
||||||
|
logger: {
|
||||||
|
isEnabled: (_name, _severity) => debug,
|
||||||
|
log: (name, severity, message, args) =>
|
||||||
|
console.warn(`${severity}| ${name} :: ${message} __ ${args}`),
|
||||||
|
},
|
||||||
|
});
|
||||||
|
const context = await browser.newContext({ locale: "fr-FR" });
|
||||||
|
const page = await context.newPage();
|
||||||
|
|
||||||
|
const globalCtx = {
|
||||||
|
dolibarrAddress,
|
||||||
|
DBpassword,
|
||||||
|
adminCredentials,
|
||||||
|
|
||||||
|
debug,
|
||||||
|
rootFolderPath,
|
||||||
|
imgFolderPath,
|
||||||
|
configFolderPath,
|
||||||
|
|
||||||
|
browser,
|
||||||
|
page,
|
||||||
|
context,
|
||||||
|
};
|
||||||
|
|
||||||
|
try {
|
||||||
|
await login.doAdminLogin(globalCtx);
|
||||||
|
console.log(`connected as ${await login.whoAmI(globalCtx)}`);
|
||||||
|
|
||||||
|
await moduleSetup.enableApiModule(globalCtx);
|
||||||
|
console.log("REST API module enabled");
|
||||||
|
|
||||||
|
const userId = await userSetup.createUser(globalCtx, {
|
||||||
|
login: AI_AGENT_LOGIN,
|
||||||
|
password: aiAgentPassword,
|
||||||
|
lastname: "AI Agent (sandbox)",
|
||||||
|
admin: false,
|
||||||
|
});
|
||||||
|
console.log(`created user '${AI_AGENT_LOGIN}' (id=${userId})`);
|
||||||
|
|
||||||
|
await userSetup.assignRights(globalCtx, userId, WRITE_IDS);
|
||||||
|
console.log(`granted ${WRITE_IDS.length} write rights to id=${userId}`);
|
||||||
|
|
||||||
|
const apiKey = await userSetup.generateApiKey(globalCtx, userId);
|
||||||
|
|
||||||
|
// Emit the key safely: write it to a gitignored file rather than printing it.
|
||||||
|
await Deno.writeTextFile(KEY_FILE, apiKey + "\n");
|
||||||
|
console.log(`AI_AGENT_SANDBOX API KEY WRITTEN TO test/${KEY_FILE}`);
|
||||||
|
} catch (error) {
|
||||||
|
console.error(error);
|
||||||
|
} finally {
|
||||||
|
await browser.close();
|
||||||
|
}
|
||||||
@@ -1,19 +1,20 @@
|
|||||||
|
|
||||||
// info-box
|
// info-box
|
||||||
|
|
||||||
import { Page } from "playwright";
|
import { Locator, Page } from "playwright";
|
||||||
import forms from "../forms.ts";
|
import forms from "../forms.ts";
|
||||||
import login from "../login.ts";
|
import login from "../login.ts";
|
||||||
|
|
||||||
// span.info-box-title has-text /moduleName/
|
// span.info-box-title has-text /moduleName/
|
||||||
// span.fa-cog[title="Configuration"]
|
// span.fa-cog[title="Configuration"]
|
||||||
|
|
||||||
|
type ModuleCtx = {
|
||||||
|
page: Page;
|
||||||
|
dolibarrAddress: string;
|
||||||
|
adminCredentials: { username: string; password: string };
|
||||||
|
};
|
||||||
|
|
||||||
async function configureModule(
|
async function configureModule(
|
||||||
globalCtx: {
|
globalCtx: ModuleCtx,
|
||||||
page: Page;
|
|
||||||
dolibarrAddress: string;
|
|
||||||
adminCredentials: { username: string; password: string };
|
|
||||||
},
|
|
||||||
{
|
{
|
||||||
moduleName,
|
moduleName,
|
||||||
enabled,
|
enabled,
|
||||||
@@ -42,6 +43,61 @@ async function configureModule(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/*
|
||||||
|
Enable Dolibarr's REST API / Web services module.
|
||||||
|
|
||||||
|
Activation maps to llx_const.MAIN_MODULE_API=1; in the UI this is the module
|
||||||
|
card on /admin/modules.php (kanban mode). In fr_FR the card title is
|
||||||
|
"API/Web services REST (serveur)" (family "Interfaces").
|
||||||
|
|
||||||
|
NOTE: confirm the exact card title on the first real run against an installed
|
||||||
|
sandbox — the label has not been verified live here. To stay resilient we try
|
||||||
|
the known fr_FR label first (anchored exact match, same as configureModule),
|
||||||
|
and if that card is not present we fall back to matching ANY module card whose
|
||||||
|
title contains "API ... REST" / "REST ... API" in either order.
|
||||||
|
*/
|
||||||
|
async function enableApiModule(globalCtx: ModuleCtx): Promise<void> {
|
||||||
|
const {page, dolibarrAddress}= globalCtx;
|
||||||
|
|
||||||
|
// fr_FR label as observed in the module catalogue. Confirm on first real run.
|
||||||
|
const knownLabel = "API/Web services REST (serveur)";
|
||||||
|
|
||||||
|
await login.doAdminLogin(globalCtx);
|
||||||
|
await page.goto(`${dolibarrAddress}/admin/modules.php?mode=commonkanban`);
|
||||||
|
|
||||||
|
const exactCard = page.locator('.info-box', {
|
||||||
|
has: page.locator('.info-box-title',{
|
||||||
|
hasText: new RegExp('^'+knownLabel+'\n?$','i')
|
||||||
|
})
|
||||||
|
});
|
||||||
|
|
||||||
|
if(await exactCard.count() > 0) {
|
||||||
|
await forms.toggleOnOff(exactCard, true);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Fallback: the exact fr_FR label was not found (different Dolibarr version,
|
||||||
|
// locale, or wording). Match any card whose title looks like the REST API
|
||||||
|
// module regardless of word order.
|
||||||
|
const fuzzyCard = page.locator('.info-box', {
|
||||||
|
has: page.locator('.info-box-title',{
|
||||||
|
hasText: /API.*REST|REST.*API/i
|
||||||
|
})
|
||||||
|
});
|
||||||
|
|
||||||
|
if(await fuzzyCard.count() === 0) {
|
||||||
|
throw new Error(
|
||||||
|
`REST API module card not found on ${dolibarrAddress}/admin/modules.php `+
|
||||||
|
`(tried exact label "${knownLabel}" and /API.*REST|REST.*API/i). `+
|
||||||
|
`Confirm the card title in the running sandbox.`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// If several cards match the fuzzy pattern, toggle the first one only.
|
||||||
|
await forms.toggleOnOff(fuzzyCard.first() as Locator, true);
|
||||||
|
}
|
||||||
|
|
||||||
export default {
|
export default {
|
||||||
configureModule,
|
configureModule,
|
||||||
|
enableApiModule,
|
||||||
}
|
}
|
||||||
|
|||||||
255
test/scripts/admin/userSetup.ts
Normal file
255
test/scripts/admin/userSetup.ts
Normal file
@@ -0,0 +1,255 @@
|
|||||||
|
/*
|
||||||
|
User provisioning for the Dolibarr admin UI, driven by Playwright.
|
||||||
|
|
||||||
|
Why the UI and not raw SQL:
|
||||||
|
- Passwords use MAIN_SECURITY_HASH_ALGO=password_hash (bcrypt), so we let
|
||||||
|
Dolibarr hash them by submitting the create form.
|
||||||
|
- API keys are stored ENCRYPTED with the instance key DOLI_INSTANCE_UNIQUE_ID.
|
||||||
|
Each instance (incl. the sandbox) has its own uuid, so a key can only be
|
||||||
|
produced correctly by that instance — we MUST generate it via the UI and
|
||||||
|
read back the resulting value, never INSERT a raw key.
|
||||||
|
|
||||||
|
IMPORTANT — selectors below are best-effort and have NOT been verified against
|
||||||
|
a live, installed sandbox (the sandbox Dolibarr is not provisioned yet). Field
|
||||||
|
names / icon controls are taken from Dolibarr 22 conventions and must be
|
||||||
|
confirmed on the first real run. Spots that are guessed are marked GUESS:.
|
||||||
|
*/
|
||||||
|
import type { Page } from "playwright";
|
||||||
|
import forms from "../forms.ts";
|
||||||
|
import login from "../login.ts";
|
||||||
|
|
||||||
|
type UserCtx = {
|
||||||
|
page: Page;
|
||||||
|
dolibarrAddress: string;
|
||||||
|
adminCredentials: { username: string; password: string };
|
||||||
|
// imgFolderPath is required by forms.fillForm's signature even though no
|
||||||
|
// file input is used here; provisionSandbox.ts supplies it from globalCtx.
|
||||||
|
imgFolderPath: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
type NewUser = {
|
||||||
|
login: string;
|
||||||
|
password: string;
|
||||||
|
lastname?: string;
|
||||||
|
admin?: boolean;
|
||||||
|
};
|
||||||
|
|
||||||
|
// The text fields filled via forms.fillForm (the admin flag is a checkbox we
|
||||||
|
// handle separately, so it is intentionally not part of this type).
|
||||||
|
type UserFormFields = {
|
||||||
|
login: string;
|
||||||
|
password: string;
|
||||||
|
lastname?: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
// fr_FR Dolibarr user create form (/user/card.php?action=create&mode=create).
|
||||||
|
// GUESS: confirm these input names on first real run. `login` and `lastname`
|
||||||
|
// are stable across Dolibarr versions; the password field has been `password`
|
||||||
|
// for the manual-entry case (the "generate automatically" option is a separate
|
||||||
|
// radio/checkbox we leave untouched so our explicit value is used).
|
||||||
|
const newUserInputNames = new Map<keyof UserFormFields, string>([
|
||||||
|
["login", "login"],
|
||||||
|
["lastname", "lastname"],
|
||||||
|
["password", "password"],
|
||||||
|
]);
|
||||||
|
|
||||||
|
/*
|
||||||
|
Create a user via /user/card.php?action=create and return its numeric id.
|
||||||
|
|
||||||
|
The admin flag is a checkbox (name="admin"); fillForm has no checkbox handler
|
||||||
|
so we toggle it explicitly. After submit Dolibarr redirects to the user card
|
||||||
|
whose URL carries ?id=<n> — we parse the id from there (falling back to a
|
||||||
|
hidden input on the page).
|
||||||
|
*/
|
||||||
|
async function createUser(
|
||||||
|
globalCtx: UserCtx,
|
||||||
|
{ login: userLogin, password, lastname, admin = false }: NewUser,
|
||||||
|
): Promise<number> {
|
||||||
|
const { page, dolibarrAddress, imgFolderPath } = globalCtx;
|
||||||
|
await login.doAdminLogin(globalCtx);
|
||||||
|
|
||||||
|
await page.goto(`${dolibarrAddress}/user/card.php?action=create&mode=create`);
|
||||||
|
|
||||||
|
// Fill text inputs (login, lastname, password) without submitting yet.
|
||||||
|
const formFields: UserFormFields = {
|
||||||
|
login: userLogin,
|
||||||
|
password,
|
||||||
|
lastname,
|
||||||
|
};
|
||||||
|
await forms.fillForm(
|
||||||
|
{ page, imgFolderPath },
|
||||||
|
formFields,
|
||||||
|
newUserInputNames,
|
||||||
|
0,
|
||||||
|
);
|
||||||
|
|
||||||
|
// Admin checkbox — only present when the logged-in user is themselves admin.
|
||||||
|
// GUESS: input[name="admin"]; on some setups it's a <select> instead.
|
||||||
|
const adminCheckbox = page.locator('input[name="admin"]');
|
||||||
|
if (await adminCheckbox.count() > 0) {
|
||||||
|
if (admin) {
|
||||||
|
await adminCheckbox.check();
|
||||||
|
} else {
|
||||||
|
// Leave unchecked (default), but be explicit/defensive.
|
||||||
|
if (await adminCheckbox.isChecked()) await adminCheckbox.uncheck();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Submit the create form. The create page has a single primary submit button
|
||||||
|
// ("Créer utilisateur"); :nth-match index 1 mirrors the fillForm convention.
|
||||||
|
await page.click(':nth-match(input[type="submit"], 1)');
|
||||||
|
|
||||||
|
// After creation Dolibarr lands on the user card. Parse the id.
|
||||||
|
await page.waitForLoadState("domcontentloaded");
|
||||||
|
const id = await readUserIdFromPage(page);
|
||||||
|
if (id === undefined) {
|
||||||
|
throw new Error(
|
||||||
|
`Could not determine the new user id for login "${userLogin}" after ` +
|
||||||
|
`submitting the create form (current URL: ${page.url()}).`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return id;
|
||||||
|
}
|
||||||
|
|
||||||
|
/*
|
||||||
|
Read the numeric user id from the current user card: first from the URL
|
||||||
|
(?id=<n>), then from a hidden input[name="id"] as a fallback.
|
||||||
|
*/
|
||||||
|
async function readUserIdFromPage(page: Page): Promise<number | undefined> {
|
||||||
|
const fromUrl = new URL(page.url()).searchParams.get("id");
|
||||||
|
if (fromUrl && /^\d+$/.test(fromUrl)) return Number(fromUrl);
|
||||||
|
|
||||||
|
// GUESS: a hidden input carrying the id (common on Dolibarr card pages).
|
||||||
|
const hidden = page.locator('input[name="id"]');
|
||||||
|
if (await hidden.count() > 0) {
|
||||||
|
const val = await hidden.first().getAttribute("value");
|
||||||
|
if (val && /^\d+$/.test(val)) return Number(val);
|
||||||
|
}
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
/*
|
||||||
|
Grant a set of rights to a user on /user/perms.php?id=<userId>.
|
||||||
|
|
||||||
|
Dolibarr renders each grantable permission as a link whose href carries
|
||||||
|
action=addrights&rights=<id> (and a matching action=delrights to revoke).
|
||||||
|
A right that is already granted shows the delrights link instead, so we treat
|
||||||
|
the presence of an addrights link as "not yet granted" and click it; if it's
|
||||||
|
absent we assume the right is already on and skip it (defensive/idempotent).
|
||||||
|
*/
|
||||||
|
async function assignRights(
|
||||||
|
globalCtx: UserCtx,
|
||||||
|
userId: number,
|
||||||
|
rightIds: number[],
|
||||||
|
): Promise<void> {
|
||||||
|
const { page, dolibarrAddress } = globalCtx;
|
||||||
|
await login.doAdminLogin(globalCtx);
|
||||||
|
|
||||||
|
for (const rightId of rightIds) {
|
||||||
|
// Re-navigate per right: Dolibarr reloads the perms table after each grant,
|
||||||
|
// which would otherwise stale the locators.
|
||||||
|
await page.goto(`${dolibarrAddress}/user/perms.php?id=${userId}`);
|
||||||
|
|
||||||
|
// GUESS: the grant control is an <a> linking to action=addrights&rights=<id>.
|
||||||
|
// We match on the href substring to avoid depending on row layout / icons.
|
||||||
|
const addLink = page.locator(
|
||||||
|
`a[href*="action=addrights"][href*="rights=${rightId}"]`,
|
||||||
|
);
|
||||||
|
|
||||||
|
if (await addLink.count() === 0) {
|
||||||
|
// Either already granted (only a delrights link is shown) or the row is
|
||||||
|
// not visible for this user. Skip rather than fail the whole run.
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
await addLink.first().click();
|
||||||
|
await page.waitForLoadState("domcontentloaded");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/*
|
||||||
|
Generate (or read) the user's API key on /user/card.php?id=<userId> in edit
|
||||||
|
mode and RETURN it as a string.
|
||||||
|
|
||||||
|
With the API module enabled the user card edit form shows an api_key text
|
||||||
|
input plus a "generate" control (a dice/refresh icon-link, typically
|
||||||
|
id="generate_api_key" or an <a> with action=...generate...apikey). We trigger
|
||||||
|
generation, then read the value back out of the api_key input. If a key is
|
||||||
|
already present we read and return it without regenerating (idempotent).
|
||||||
|
|
||||||
|
The caller is responsible for handling the returned secret safely — DO NOT log
|
||||||
|
it at debug verbosity.
|
||||||
|
*/
|
||||||
|
async function generateApiKey(
|
||||||
|
globalCtx: UserCtx,
|
||||||
|
userId: number,
|
||||||
|
): Promise<string> {
|
||||||
|
const { page, dolibarrAddress } = globalCtx;
|
||||||
|
await login.doAdminLogin(globalCtx);
|
||||||
|
|
||||||
|
await page.goto(`${dolibarrAddress}/user/card.php?id=${userId}&action=edit`);
|
||||||
|
|
||||||
|
// GUESS: the API key field is input[name="api_key"].
|
||||||
|
const apiKeyInput = page.locator('input[name="api_key"]');
|
||||||
|
if (await apiKeyInput.count() === 0) {
|
||||||
|
throw new Error(
|
||||||
|
`API key input (input[name="api_key"]) not found on the user card for ` +
|
||||||
|
`id=${userId}. Is the REST API module enabled? Confirm the field name ` +
|
||||||
|
`on the running sandbox.`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// If a key is already set, reuse it.
|
||||||
|
const existing = (await apiKeyInput.first().inputValue()).trim();
|
||||||
|
if (existing.length > 0) return existing;
|
||||||
|
|
||||||
|
// Trigger generation. GUESS: a generate control next to the field. We try a
|
||||||
|
// few likely selectors in order and click the first that exists.
|
||||||
|
const generateSelectors = [
|
||||||
|
"#generate_api_key",
|
||||||
|
'a[href*="action=generateapikey"]',
|
||||||
|
'a[href*="generate"][href*="apikey"]',
|
||||||
|
// Icon-only fallbacks commonly used by Dolibarr "generate" buttons.
|
||||||
|
"span.fa-dice",
|
||||||
|
"a.fa-refresh",
|
||||||
|
];
|
||||||
|
|
||||||
|
let clicked = false;
|
||||||
|
for (const sel of generateSelectors) {
|
||||||
|
const ctrl = page.locator(sel);
|
||||||
|
if (await ctrl.count() > 0) {
|
||||||
|
await ctrl.first().click();
|
||||||
|
clicked = true;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!clicked) {
|
||||||
|
throw new Error(
|
||||||
|
`No API-key generate control found near input[name="api_key"] for ` +
|
||||||
|
`id=${userId} (tried ${generateSelectors.join(", ")}). Confirm the ` +
|
||||||
|
`generate control on the running sandbox.`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// The generate control fills the input client-side; poll the input value
|
||||||
|
// (via the locator, no browser globals) until it is non-empty or we time out.
|
||||||
|
let key = "";
|
||||||
|
const deadline = Date.now() + 10_000;
|
||||||
|
while (Date.now() < deadline) {
|
||||||
|
key = (await apiKeyInput.first().inputValue()).trim();
|
||||||
|
if (key.length > 0) break;
|
||||||
|
await page.waitForTimeout(200);
|
||||||
|
}
|
||||||
|
if (key.length === 0) {
|
||||||
|
throw new Error(
|
||||||
|
`API key field was still empty after triggering generation for ` +
|
||||||
|
`id=${userId}.`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return key;
|
||||||
|
}
|
||||||
|
|
||||||
|
export default {
|
||||||
|
createUser,
|
||||||
|
assignRights,
|
||||||
|
generateApiKey,
|
||||||
|
};
|
||||||
Reference in New Issue
Block a user