fix(sandbox-poc): regenerate api_key when the existing one is garbage; note platform backup gaps

After an iso-prod refresh the instance unique-id changes, so an api_key encrypted
with the OLD id can't be decrypted — Dolibarr renders non-UTF-8 bytes in the field.
The POC's generateApiKey reused any non-empty value, so it copied that garbage into
test/.ai_agent_sandbox.key (corrupt key, 401s). Now it reuses ONLY a clean key
(^[A-Za-z0-9_-]{24,}$); otherwise it clears the field and regenerates. So
`checkpoint provision` after a refresh yields a fresh, working key.

Also documents the open PLATFORM follow-ups in ops/backup/README.md (easy to find
when revisiting ERP backups): the orphaned Longhorn `default` recurring-job group
(other cluster volumes have no offsite backup), and verifying the factory
pg_dumpall host cron.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

ops/backup/README.md

@@ -82,18 +82,24 @@ was reverted to the backup's value by `restore --db`.
 The recurring form ships in the chart (`chart/templates/backup-cronjob.yaml`,
 `backup.enabled=false` by default): a daily **CronJob** (ConfigMap-mounted
 `backup-job.sh`) with its **own** S3 creds via a `VaultStaticSecret` — no
-cross-namespace borrowing of the Longhorn secret. To activate:
+cross-namespace borrowing of the Longhorn secret.
 
-1. store the GCS HMAC creds (`AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` /
-   `AWS_ENDPOINTS`, same shape as `longhorn-gcs-backup-credentials`) at
-   `kvv2/<backup.vaultS3Path>` (default `erp/backup`);
-2. grant the erp `auth` Vault role read on that path (a `tools` change) if its
-   policy doesn't already cover it;
-3. set `backup.enabled: true` (+ tune `schedule`).
+**Status: LIVE (2026-06-30).** tools#5 granted the erp prod Vault policy read on
+`kvv2/data/longhorn/gcs-backup` (via the module's `kv_read_paths`), and the chart
+sets `backup.enabled: true` + `vaultS3Path: longhorn/gcs-backup`. The CronJob runs
+daily at 03:00 UTC; a manual run was verified (db + docs → GCS). Sandbox keeps it off.
 
-Until then, run the orchestrator above on demand / from a host cron — it works
-today by borrowing the Longhorn creds transiently.
+## Platform follow-ups — NOT erp (open; revisit when we look at ERP backups)
 
-> The generic Longhorn gap (the orphaned `default` group) should be fixed too, as a
-> platform concern — but this dedicated, offsite, 10-year-retention backup is the
-> one that matches Dolibarr's legal criticality.
+1. **Longhorn `default` group is orphaned → other volumes have NO offsite backup.**
+   The cluster's only Longhorn recurring-backup job (`thrice-a-month-backup`) has
+   `groups=[]`, so it serves no group — yet volumes are enrolled in the `default`
+   group. Result found 2026-06-30: erp + most volumes had `lastBackupAt=never`. This
+   dedicated backup covers Dolibarr, but every OTHER cluster volume is still exposed.
+   Fix at the platform layer (factory): attach a recurring job to the `default` group
+   (or set the storageclass `recurringJobSelector`).
+2. **Verify the factory `pg_dumpall` host cron actually runs.** `factory` ships
+   `ansible/.../playbooks/backup/postgres.yml` that `pg_dumpall`s daily on the
+   Postgres host (192.168.1.202) — unverified from the k8s side. Confirm it's live
+   and where it ships; it's the platform-level DB safety net beneath this app-level
+   one.

test/scripts/admin/userSetup.ts

@@ -228,9 +228,16 @@ async function generateApiKey(
     );
   }
 
-  // If a key is already set, reuse it.
+  // Reuse the existing key ONLY if it is a clean plaintext key. After an iso-prod
+  // refresh the instance unique-id changes, so a key encrypted with the OLD id can
+  // no longer be decrypted — Dolibarr then renders non-UTF-8 bytes in the field.
+  // Reusing that (the old bug) wrote a corrupt key file. Treat anything that isn't
+  // a plausible key (≥24 url-safe chars) as garbage: clear it and regenerate.
   const existing = (await apiKeyInput.first().inputValue()).trim();
-  if (existing.length > 0) return existing;
+  if (/^[A-Za-z0-9_-]{24,}$/.test(existing)) return existing;
+  if (existing.length > 0) {
+    await apiKeyInput.first().fill(""); // drop the undecryptable/garbage value
+  }
 
   // Trigger generation. GUESS: a generate control next to the field. We try a
   // few likely selectors in order and click the first that exists.