feat(ops): dedicated Dolibarr backup (DB + documents → offsite GCS, 10y retention)

The accounting data + issued documents are legally retained 10 years and warrant a
backup dedicated to Dolibarr. An audit found the generic Longhorn external backup
NEVER covered the erp volume (its Longhorn volume sits in the orphaned `default`
recurring-job group; the only job has groups=[] → serves nothing; lastBackupAt=never).
So /var/www/documents (invoice PDFs, supplier pieces, contracts, ECM) had zero
offsite copy — only in-cluster replicas.

ops/backup/dolibarr-backup.sh (orchestrator) + ops/backup/backup-job.sh (in-container
logic, env-driven, single source of truth):
- pg_dump -Fc of the DB + tar of the documents PVC (RWX, read-only mount) ->
  s3://arcodange-backup/erp/<env>/{db,docs}/<ts>, then tiered prune (daily 30d /
  monthly 12m / yearly 10y).
- prod is READ-only (dump+tar read; writes go only to the backup bucket); the DB is
  read with the env's own dynamic creds; the GCS HMAC secret is copied transiently
  (base64, deleted on exit) and never printed; the whole script ships base64.
- fixes the aws-cli v2.23+ default-checksum incompatibility with GCS/S3-compat
  (SignatureDoesNotMatch) via AWS_*_CHECKSUM_*=when_required.

Proven live: sandbox end-to-end (dump+tar+upload+prune, verified in GCS, cleaned up)
and retention logic unit-tested (1100 daily -> 46 kept). The FIRST real prod backup
was taken (erp/prod/db 1.2 MB + erp/prod/docs 12.5 MB) — closing the gap now.

Automation (recurring CronJob in the chart + a dedicated erp Vault policy for its
own S3 creds) is the documented next step; the orchestrator works today on demand.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

ops/backup/backup-job.sh

@@ -0,0 +1,56 @@
+#!/bin/sh
+# In-container backup logic for Dolibarr — the single source of truth shared by the
+# manual orchestrator (ops/backup/dolibarr-backup.sh) and the scheduled CronJob
+# (chart/templates/backup-cronjob.yaml). Driven entirely by environment:
+#   BUCKET PREFIX DB PGHOST       (config)
+#   PGUSER PGPASSWORD             (DB creds, from vso-db-credentials)
+#   AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_ENDPOINTS   (S3 creds)
+# It dumps the DB (pg_dump -Fc) + tars the documents mounted at /docs, pushes both
+# to s3://$BUCKET/$PREFIX/{db,docs}/, then prunes to a tiered retention.
+set -eu
+apk add --no-cache aws-cli tar gzip >/dev/null 2>&1 || { echo "ABORT apk add"; exit 1; }
+: "${BUCKET:?}"; : "${PREFIX:?}"; : "${DB:?}"; : "${PGHOST:?}"
+export AWS_DEFAULT_REGION="${AWS_DEFAULT_REGION:-us-east-1}"
+# GCS / S3-compatible stores reject aws-cli v2.23+ default integrity checksums
+# ("SignatureDoesNotMatch / Invalid argument") — only sign/validate when required.
+export AWS_REQUEST_CHECKSUM_CALCULATION=when_required
+export AWS_RESPONSE_CHECKSUM_VALIDATION=when_required
+S3() { aws --endpoint-url "$AWS_ENDPOINTS" s3 "$@"; }
+
+TS=$(date -u +%Y-%m-%dT%H-%M-%SZ)
+echo "timestamp=$TS  db=$DB  -> s3://$BUCKET/$PREFIX"
+pg_dump -h "$PGHOST" -U "$PGUSER" -d "$DB" -Fc -f /tmp/db.dump
+echo "db.dump $(wc -c < /tmp/db.dump) bytes"
+tar -C /docs -czf /tmp/docs.tar.gz . 2>/dev/null
+echo "docs.tar.gz $(wc -c < /tmp/docs.tar.gz) bytes"
+S3 cp /tmp/db.dump     "s3://$BUCKET/$PREFIX/db/$TS.dump"
+S3 cp /tmp/docs.tar.gz "s3://$BUCKET/$PREFIX/docs/$TS.tar.gz"
+echo "uploaded to s3://$BUCKET/$PREFIX/{db,docs}/$TS.*"
+
+# tiered retention: daily 30d / monthly 12m (latest per month) / yearly ~10y
+cat > /tmp/prune.py <<'PY'
+import sys, datetime
+keys=[k.strip() for k in open(sys.argv[1]) if k.strip()]
+now=datetime.datetime.strptime(sys.argv[2][:10], "%Y-%m-%d").date()
+def d(k):
+    try: return datetime.datetime.strptime(k[:10], "%Y-%m-%d").date()
+    except Exception: return None
+dated=sorted([(d(k),k) for k in keys if d(k)], key=lambda x:x[0])
+keep=set(); bymonth={}; byyear={}
+for dt,k in dated:
+    age=(now-dt).days
+    if age <= 30: keep.add(k)
+    elif age <= 365: bymonth[(dt.year,dt.month)]=k
+    elif age <= 3660: byyear[dt.year]=k
+keep |= set(bymonth.values()) | set(byyear.values())
+for dt,k in dated:
+    if k not in keep: print(k)
+PY
+for SUB in db docs; do
+  S3 ls "s3://$BUCKET/$PREFIX/$SUB/" | awk '{print $4}' > /tmp/keys.$SUB || true
+  python3 /tmp/prune.py "/tmp/keys.$SUB" "$TS" > /tmp/del.$SUB || true
+  while read -r DK; do
+    [ -n "$DK" ] && S3 rm "s3://$BUCKET/$PREFIX/$SUB/$DK" && echo "pruned $SUB/$DK"
+  done < /tmp/del.$SUB
+done
+echo "DONE."