arcodangeandClaude Opus 5
a3540982a9
DEUX CHOSES, dont une inachevée et dite comme telle. 1. LA SAUVEGARDE N'AVAIT AUCUNE GARDE DE CLUSTER. 24 appels kubectl nus, aucun contexte épinglé — alors que le script lit des secrets, crée des Jobs et sait RESTAURER une base. Lancé sur le contexte courant d'une station de travail, il serait parti chercher les secrets Longhorn du cluster d'un client. Même patron que ops/sandbox : ERP_KUBE_CONTEXT, wrapper K(), empreinte positive vérifiée avant tout dispatch. Testé contre le cluster client et contre un contexte inexistant : il refuse les deux. Sauvegarde de production passée dans la foulée. La base a été dédupliquée à juste titre — rien n'avait été écrit depuis la sauvegarde automatique de 01:00. Au passage, le CronJob quotidien tourne bien depuis juillet ; il était encore noté comme à faire. 2. LE MODE VIERGE PURGE MAIS NE RECONSTRUIT PAS. `blank --yes` vide la base du bac à sable (295 tables -> 0, vérifié) avec trois protections empilées : refus sans --yes, garde de cluster, et une relecture de current_database() DANS le Job lui-même — une purge sur la mauvaise base ne se rattrape pas par un refresh, contrairement à la bonne. Mais l'instance ne se reconstruit pas. Premier essai : install.lock vit sur le volume documents, que la purge ne touche pas, si bien que Dolibarr servait un login sur un schéma inexistant. Correctif appliqué — retrait du verrou puis redémarrage. Second essai : le verrou reste absent, et le schéma reste à ZÉRO table. L'entrypoint de l'image ne lance aucune installation automatique. CE QU'IL MANQUE est donc nommé dans le script : un « golden empty », pg_dump d'une instance fraîchement installée — schéma et données de référence, aucune donnée métier. `blank` le restaurerait au lieu de laisser la base vide, comme refresh-from-prod restaure le dump de production. Seule la source change. En l'état `blank` laisse le bac à sable inutilisable, et le script le dit. Le bac à sable a été remis iso-prod avant de rendre la main. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
290 lines
16 KiB
Bash
Executable File
290 lines
16 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
#
|
|
# sandbox-lifecycle.sh — seed / refresh the erp-sandbox Dolibarr from prod, and
|
|
# sync its uploaded documents, with prod integrity guaranteed structurally.
|
|
#
|
|
# Implements ADR-0003 (factory vibe/ADR/0003-sandbox-state-lifecycle.md):
|
|
# - prod is read ONLY (pg_dump runs in a default_transaction_read_only session);
|
|
# - the restore writes ONLY to erp-sandbox, using the sandbox's own dynamic
|
|
# credentials (a member of erp_sandbox_role, which owns only the sandbox DB),
|
|
# so it is structurally incapable of touching prod 'erp' (owned by erp_role);
|
|
# - no DROP/CREATE DATABASE, no CREATEDB, no superuser — wipe is
|
|
# `DROP OWNED BY erp_sandbox_role CASCADE`, reload is `pg_restore`.
|
|
#
|
|
# The only prod-capable credential on the platform is the superuser provider in
|
|
# factory postgres/iac, exercised solely in the human-gated postgres.yaml CI —
|
|
# this script never uses it.
|
|
#
|
|
# Requires: kubectl (context on the lab cluster), and a postgres:16 image
|
|
# reachable by the cluster. Run from anywhere.
|
|
#
|
|
# Usage:
|
|
# ./sandbox-lifecycle.sh refresh-from-prod # iso-prod seed: prod DB -> erp-sandbox
|
|
# ./sandbox-lifecycle.sh sync-documents # copy mycompany/ uploads (logo, PDFs)
|
|
# ./sandbox-lifecycle.sh refresh # refresh-from-prod + sync-documents
|
|
#
|
|
set -euo pipefail
|
|
|
|
PROD_NS="erp"
|
|
SB_NS="erp-sandbox"
|
|
PROD_DB="erp"
|
|
SB_DB="erp-sandbox"
|
|
SB_ROLE="erp_sandbox_role" # snake-case owner role (ADR-0002 elision rule)
|
|
PGHOST="192.168.1.202" # direct Postgres (NOT pgbouncer — pooler breaks pg_dump)
|
|
PG_IMAGE="postgres:16-alpine"
|
|
DOC_ROOT="/var/www/documents" # dolibarr_main_data_root
|
|
TMP_PROD_SECRET="prod-db-ro-temp" # transient copy of prod creds, deleted on exit
|
|
|
|
log() { printf '\033[1;36m==>\033[0m %s\n' "$*"; }
|
|
die() { printf '\033[1;31mABORT:\033[0m %s\n' "$*" >&2; exit 1; }
|
|
|
|
sb_pod() { K get pod -n "$SB_NS" -l app.kubernetes.io/instance=erp-sandbox -o name 2>/dev/null | head -1; }
|
|
prod_pod() { K get pod -n "$PROD_NS" -l app.kubernetes.io/instance=erp -o name 2>/dev/null | head -1; }
|
|
|
|
cleanup_secret() { K delete secret "$TMP_PROD_SECRET" -n "$SB_NS" --ignore-not-found >/dev/null 2>&1 || true; }
|
|
|
|
# erp-sandbox is ArgoCD-managed with self-heal ON, which reverts `kubectl scale
|
|
# --replicas=0` within seconds — so without pausing it the seed runs while Dolibarr
|
|
# is still connected, and the restore collides with the app re-creating tables.
|
|
# Pause self-heal for the duration so the scale-down holds; always re-arm it.
|
|
ARGOCD_NS="argocd"; ARGOCD_APP="erp-sandbox"
|
|
|
|
# --- Cluster guard -----------------------------------------------------------
|
|
# This script scales deployments to zero, patches the ArgoCD Application and runs
|
|
# `DROP OWNED ... CASCADE`. Until now every one of those ran against whatever
|
|
# kube-context happened to be current — and this workstation also carries a
|
|
# CLIENT production cluster (observed 2026-07-25: the current context was the
|
|
# client's, and refresh only failed because that cluster has no `erp` namespace).
|
|
# Never trust the ambient context: resolve an explicit one and prove it is the
|
|
# Arcodange homelab before touching anything.
|
|
ERP_KUBE_CONTEXT="${ERP_KUBE_CONTEXT:-default}"
|
|
K() { kubectl --context "$ERP_KUBE_CONTEXT" "$@"; }
|
|
|
|
assert_arcodange_cluster() {
|
|
kubectl config get-contexts -o name 2>/dev/null | grep -qx "$ERP_KUBE_CONTEXT" \
|
|
|| die "kube-context '$ERP_KUBE_CONTEXT' does not exist (set ERP_KUBE_CONTEXT)"
|
|
# Positive fingerprint: the three namespaces AND the ArgoCD Application this
|
|
# script drives. A client cluster cannot match all four by accident.
|
|
for ns in "$PROD_NS" "$SB_NS" "$ARGOCD_NS"; do
|
|
K get ns "$ns" >/dev/null 2>&1 \
|
|
|| die "kube-context '$ERP_KUBE_CONTEXT' has no '$ns' namespace — refusing to run against it.
|
|
This script is destructive (scale-to-0, DROP OWNED CASCADE, ArgoCD patch) and must only
|
|
ever target the Arcodange homelab. Current context is '$(kubectl config current-context 2>/dev/null)'.
|
|
Set ERP_KUBE_CONTEXT to the homelab context and retry."
|
|
done
|
|
K get application "$ARGOCD_APP" -n "$ARGOCD_NS" >/dev/null 2>&1 \
|
|
|| die "kube-context '$ERP_KUBE_CONTEXT' has no ArgoCD Application '$ARGOCD_APP' — refusing to run."
|
|
log "cluster guard OK — context '$ERP_KUBE_CONTEXT' (namespaces $PROD_NS/$SB_NS/$ARGOCD_NS + app $ARGOCD_APP)"
|
|
}
|
|
set_selfheal() { K patch application "$ARGOCD_APP" -n "$ARGOCD_NS" --type merge \
|
|
-p "{\"spec\":{\"syncPolicy\":{\"automated\":{\"selfHeal\":$1,\"prune\":true}}}}" >/dev/null 2>&1 || true; }
|
|
# Safety net (EXIT trap): whatever happens, bring the app back, re-arm self-heal, drop the secret.
|
|
restore_state() { set_selfheal true; K scale deploy erp-sandbox -n "$SB_NS" --replicas=1 >/dev/null 2>&1 || true; cleanup_secret; }
|
|
|
|
refresh_from_prod() {
|
|
command -v python3 >/dev/null || die "python3 required to copy the prod secret without exposing it"
|
|
trap restore_state EXIT
|
|
|
|
log "Pausing ArgoCD self-heal so the scale-to-0 holds (else it is reverted in seconds)"
|
|
set_selfheal false
|
|
|
|
log "Copying prod DB creds into a transient, read-only-intent secret in $SB_NS (values stay base64)"
|
|
K get secret vso-db-credentials -n "$PROD_NS" -o json \
|
|
| python3 -c "import json,sys; d=json.load(sys.stdin); d['metadata']={'name':'$TMP_PROD_SECRET','namespace':'$SB_NS'}; d.pop('status',None); d['data']={k:d['data'][k] for k in ('username','password')}; print(json.dumps(d))" \
|
|
| K apply -f - >/dev/null
|
|
|
|
log "Scaling erp-sandbox to 0 (exclusive DB access for the restore)"
|
|
K scale deploy erp-sandbox -n "$SB_NS" --replicas=0 >/dev/null
|
|
K wait --for=delete pod -l app.kubernetes.io/instance=erp-sandbox -n "$SB_NS" --timeout=120s >/dev/null 2>&1 || true
|
|
|
|
log "Running the seed Job (pg_dump prod read-only -> DROP OWNED -> pg_restore into sandbox)"
|
|
K delete job sandbox-seed -n "$SB_NS" --ignore-not-found >/dev/null 2>&1 || true
|
|
K apply -f - >/dev/null <<EOF
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata: { name: sandbox-seed, namespace: $SB_NS }
|
|
spec:
|
|
backoffLimit: 0
|
|
ttlSecondsAfterFinished: 900
|
|
template:
|
|
spec:
|
|
restartPolicy: Never
|
|
containers:
|
|
- name: seed
|
|
image: $PG_IMAGE
|
|
env:
|
|
- { name: PROD_PGUSER, valueFrom: { secretKeyRef: { name: $TMP_PROD_SECRET, key: username } } }
|
|
- { name: PROD_PGPASSWORD, valueFrom: { secretKeyRef: { name: $TMP_PROD_SECRET, key: password } } }
|
|
- { name: SB_PGUSER, valueFrom: { secretKeyRef: { name: vso-db-credentials, key: username } } }
|
|
- { name: SB_PGPASSWORD, valueFrom: { secretKeyRef: { name: vso-db-credentials, key: password } } }
|
|
- { name: PGHOST, value: "$PGHOST" }
|
|
- { name: PGSSLMODE, value: "disable" }
|
|
command: ["/bin/sh","-c"]
|
|
args:
|
|
- |
|
|
set -eu
|
|
SBDB=\$(PGPASSWORD=\$SB_PGPASSWORD psql -h "\$PGHOST" -U "\$SB_PGUSER" -d $SB_DB -tAc 'select current_database()')
|
|
[ "\$SBDB" = "$SB_DB" ] || { echo "ABORT: target is '\$SBDB' not $SB_DB"; exit 1; }
|
|
echo "source=$PROD_DB (read-only) target=$SB_DB ok"
|
|
# 1. dump prod — full public schema (incl. helper functions + triggers), read-only session
|
|
PGPASSWORD=\$PROD_PGPASSWORD PGOPTIONS='-c default_transaction_read_only=on' \\
|
|
pg_dump -h "\$PGHOST" -U "\$PROD_PGUSER" -d $PROD_DB -n public -Fc -f /tmp/golden.dump
|
|
# drop provisioner-owned infra (pgbouncer user_lookup) from the TOC: it already
|
|
# exists in the sandbox and is not app data, so restoring it conflicts.
|
|
pg_restore -l /tmp/golden.dump | grep -vi 'user_lookup' > /tmp/golden.toc
|
|
echo "dump: \$(ls -l /tmp/golden.dump | awk '{print \$5}') bytes, tables=\$(grep -c 'TABLE DATA ' /tmp/golden.toc)"
|
|
# 2. wipe sandbox app objects (everything owned by the app role; infra untouched)
|
|
PGPASSWORD=\$SB_PGPASSWORD psql -h "\$PGHOST" -U "\$SB_PGUSER" -d $SB_DB -v ON_ERROR_STOP=1 \\
|
|
-c "DROP OWNED BY $SB_ROLE CASCADE;"
|
|
# 3. restore golden, owned by the sandbox role. MUST pass -U: without it
|
|
# pg_restore connects as the container's OS user (root) and auth-fails.
|
|
# pg_restore also exits non-zero on the harmless "schema public already
|
|
# exists" notice, so its exit code is NOT trustworthy — verify by count below.
|
|
Q() { PGPASSWORD=\$SB_PGPASSWORD psql -h "\$PGHOST" -U "\$SB_PGUSER" -d $SB_DB -tAc "\$1"; }
|
|
PGPASSWORD=\$SB_PGPASSWORD \\
|
|
pg_restore -h "\$PGHOST" -U "\$SB_PGUSER" -L /tmp/golden.toc --no-owner --role=$SB_ROLE -d $SB_DB /tmp/golden.dump 2>/tmp/restore.err \\
|
|
&& echo "restore: clean" || echo "restore: pg_restore rc=\$? — verifying by table count, not exit code"
|
|
# 4. verify — FAIL the Job if the restore did not actually populate the schema
|
|
N=\$(Q "select count(*) from pg_tables where schemaname='public' and tablename like 'llx_%'")
|
|
[ "\$N" -ge 250 ] || { echo "ABORT: only \$N llx tables after restore — restore failed. Last errors:"; tail -5 /tmp/restore.err; exit 1; }
|
|
echo "llx tables=\$N company=\$(Q "select value from llx_const where name='MAIN_INFO_SOCIETE_NOM'") lang=\$(Q "select value from llx_const where name='MAIN_LANG_DEFAULT'") owner=\$(Q "select tableowner from pg_tables where tablename='llx_societe'")"
|
|
echo "DONE."
|
|
EOF
|
|
K wait --for=condition=complete job/sandbox-seed -n "$SB_NS" --timeout=300s >/dev/null 2>&1 \
|
|
|| die "seed Job did not complete — see: K logs -n $SB_NS job/sandbox-seed"
|
|
K logs -n "$SB_NS" job/sandbox-seed | sed 's/^/ /'
|
|
K delete job sandbox-seed -n "$SB_NS" --ignore-not-found >/dev/null 2>&1 || true
|
|
|
|
log "Restoring app (replicas=1) + re-arming ArgoCD self-heal"
|
|
set_selfheal true
|
|
K scale deploy erp-sandbox -n "$SB_NS" --replicas=1 >/dev/null
|
|
cleanup_secret; trap - EXIT
|
|
log "Refresh complete. Run 'sync-documents' to also copy the company logo + uploads."
|
|
}
|
|
|
|
sync_documents() {
|
|
local pp sp
|
|
pp=$(prod_pod); sp=$(sb_pod)
|
|
[ -n "$pp" ] || die "no prod erp pod found"
|
|
[ -n "$sp" ] || die "no erp-sandbox pod found"
|
|
log "Syncing $DOC_ROOT/mycompany (logo + uploads) ${pp##*/} -> ${sp##*/} via tar pipe"
|
|
K exec -n "$PROD_NS" "${pp#pod/}" -- tar -C "$DOC_ROOT" -cf - mycompany 2>/dev/null \
|
|
| K exec -i -n "$SB_NS" "${sp#pod/}" -- tar -C "$DOC_ROOT" -xf -
|
|
log "Documents synced. (For a one-shot logo only, scope the tar to mycompany/logos.)"
|
|
}
|
|
|
|
# --- mode vierge -------------------------------------------------------------
|
|
# ÉTAT : INCOMPLET. La purge fonctionne, l'instance ne se reconstruit PAS.
|
|
#
|
|
# Vérifié le 2026-08-14 : après DROP OWNED et retrait de install.lock, Dolibarr
|
|
# sert une page de login sur un schéma à ZÉRO table et l'API répond « Module Api
|
|
# must be enabled ». L'entrypoint de l'image NE LANCE PAS d'installation
|
|
# automatique — retirer le verrou ne suffit pas.
|
|
#
|
|
# CE QU'IL MANQUE : un « golden empty », c'est-à-dire un pg_dump d'une instance
|
|
# Dolibarr fraîchement installée — schéma + données de référence (dictionnaires,
|
|
# pays, plan comptable) — sans aucune donnée métier. `blank` restaurerait ce dump
|
|
# au lieu de laisser la base vide, exactement comme `refresh-from-prod` restaure
|
|
# le dump de production. Seule la source change.
|
|
#
|
|
# En l'état, `blank` laisse le bac à sable INUTILISABLE : ne l'employer que suivi
|
|
# de `refresh-from-prod`, ou une fois le golden empty constitué.
|
|
#
|
|
# Vide la base du bac à sable SANS rien restaurer, pour qu'un rejeu d'exercice
|
|
# reparte de zéro. `refresh-from-prod` réimporterait précisément les défauts que
|
|
# le rejeu doit corriger : reconstruire suppose donc une base vide, pas iso-prod.
|
|
#
|
|
# CE QUI REND L'OPÉRATION SÛRE : elle est ANNULABLE. `refresh-from-prod` restaure
|
|
# un bac à sable iso-prod en trois minutes. On peut donc casser sans regret — et
|
|
# c'est la seule raison pour laquelle une commande qui détruit une base entière
|
|
# est acceptable ici.
|
|
#
|
|
# Elle ne vise QUE $SB_NS. Le namespace de production n'apparaît nulle part dans
|
|
# cette fonction, et la garde de cluster refuse tout contexte qui n'est pas le
|
|
# homelab.
|
|
blank_sandbox() {
|
|
[ "${1:-}" = "--yes" ] || die "blank détruit toutes les données du bac à sable — relancer avec --yes"
|
|
trap restore_state EXIT
|
|
|
|
log "Pause du self-heal ArgoCD (sinon le scale-to-0 est repris en quelques secondes)"
|
|
set_selfheal false
|
|
|
|
log "erp-sandbox à 0 réplique (accès exclusif à la base)"
|
|
K scale deploy erp-sandbox -n "$SB_NS" --replicas=0 >/dev/null
|
|
K wait --for=delete pod -l app.kubernetes.io/instance=erp-sandbox -n "$SB_NS" --timeout=120s >/dev/null 2>&1 || true
|
|
|
|
log "Job de purge (DROP OWNED, aucune restauration)"
|
|
K delete job sandbox-blank -n "$SB_NS" --ignore-not-found >/dev/null 2>&1 || true
|
|
K apply -f - >/dev/null <<EOF
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata: { name: sandbox-blank, namespace: $SB_NS }
|
|
spec:
|
|
backoffLimit: 0
|
|
ttlSecondsAfterFinished: 900
|
|
template:
|
|
spec:
|
|
restartPolicy: Never
|
|
containers:
|
|
- name: blank
|
|
image: $PG_IMAGE
|
|
env:
|
|
- { name: SB_PGUSER, valueFrom: { secretKeyRef: { name: vso-db-credentials, key: username } } }
|
|
- { name: SB_PGPASSWORD, valueFrom: { secretKeyRef: { name: vso-db-credentials, key: password } } }
|
|
- { name: PGHOST, value: "$PGHOST" }
|
|
- { name: PGSSLMODE, value: "disable" }
|
|
command: ["/bin/sh","-c"]
|
|
args:
|
|
- |
|
|
set -eu
|
|
Q() { PGPASSWORD=\$SB_PGPASSWORD psql -h "\$PGHOST" -U "\$SB_PGUSER" -d $SB_DB -tAc "\$1"; }
|
|
# Ceinture ET bretelles : on refuse d'agir si la base courante n'est
|
|
# pas celle du bac à sable. Une purge sur la mauvaise base ne se
|
|
# rattrape pas par un refresh.
|
|
CUR=\$(Q 'select current_database()')
|
|
[ "\$CUR" = "$SB_DB" ] || { echo "ABORT: base courante '\$CUR' au lieu de $SB_DB"; exit 1; }
|
|
echo "cible=\$CUR role=$SB_ROLE"
|
|
AVANT=\$(Q "select count(*) from pg_tables where schemaname='public' and tablename like 'llx_%'")
|
|
echo "tables llx_ avant purge : \$AVANT"
|
|
PGPASSWORD=\$SB_PGPASSWORD psql -h "\$PGHOST" -U "\$SB_PGUSER" -d $SB_DB -v ON_ERROR_STOP=1 \\
|
|
-c "DROP OWNED BY $SB_ROLE CASCADE;"
|
|
APRES=\$(Q "select count(*) from pg_tables where schemaname='public' and tablename like 'llx_%'")
|
|
echo "tables llx_ après purge : \$APRES"
|
|
[ "\$APRES" -eq 0 ] || { echo "ABORT: \$APRES tables subsistent — purge incomplète"; exit 1; }
|
|
echo "DONE."
|
|
EOF
|
|
K wait --for=condition=complete job/sandbox-blank -n "$SB_NS" --timeout=180s >/dev/null 2>&1 \
|
|
|| die "le Job de purge n'a pas abouti — voir : K logs -n $SB_NS job/sandbox-blank"
|
|
K logs -n "$SB_NS" job/sandbox-blank | sed 's/^/ /'
|
|
K delete job sandbox-blank -n "$SB_NS" --ignore-not-found >/dev/null 2>&1 || true
|
|
|
|
# Le verrou d'installation vit sur le VOLUME DOCUMENTS, que la purge de la base
|
|
# ne touche pas. Tant qu'il est là, Dolibarr saute l'installeur et sert une page
|
|
# de login sur un schéma inexistant : l'instance a l'air vivante et ne l'est pas.
|
|
# Le retirer laisse l'entrypoint de l'image réinstaller un schéma neuf.
|
|
log "Retrait du verrou d'installation (sinon Dolibarr sert un login sur une base sans schéma)"
|
|
K scale deploy erp-sandbox -n "$SB_NS" --replicas=1 >/dev/null
|
|
K wait --for=condition=ready pod -l app.kubernetes.io/instance=erp-sandbox -n "$SB_NS" --timeout=180s >/dev/null 2>&1 || true
|
|
POD=$(sb_pod)
|
|
[ -n "$POD" ] && K exec -n "$SB_NS" "$POD" -- rm -f /var/www/documents/install.lock >/dev/null 2>&1 || true
|
|
|
|
log "Redémarrage pour déclencher l'installation"
|
|
K rollout restart deploy erp-sandbox -n "$SB_NS" >/dev/null
|
|
K rollout status deploy erp-sandbox -n "$SB_NS" --timeout=300s >/dev/null 2>&1 || true
|
|
|
|
log "Réarmement du self-heal ArgoCD"
|
|
set_selfheal true
|
|
|
|
log "Base vidée et verrou retiré. Vérifier que le schéma se reconstruit :"
|
|
log " curl -s https://erp-sandbox.arcodange.lab/api/index.php/status"
|
|
log "Annulation à tout moment : $0 refresh-from-prod"
|
|
}
|
|
|
|
case "${1:-}" in
|
|
refresh-from-prod) assert_arcodange_cluster; refresh_from_prod ;;
|
|
blank) assert_arcodange_cluster; shift; blank_sandbox "${1:-}" ;;
|
|
sync-documents) assert_arcodange_cluster; sync_documents ;;
|
|
refresh) assert_arcodange_cluster; refresh_from_prod; sync_documents ;;
|
|
*) echo "usage: $0 {refresh-from-prod|sync-documents|refresh|blank --yes}" >&2; exit 2 ;;
|
|
esac
|