feat(ops): backup freshness watchdog + monthly restore drill (T14) #59

Open
opened 2026-07-12 13:59:09 +02:00 by arcodange · 0 comments
Owner

Part of the AI back-office PRDT14 + challenge C7 (silence is an alert). Split out of #48 (backlog sharpening 2026-07-12) so each watchdog ships as its own small PR — #48 keeps the ERP snapshot/drift side.

Context. ops/backup runs daily on prod (CronJob + Vault creds; per-run fingerprints). Nothing watches the watcher: a silently stopped backup is invisible until the day it's needed — the exact failure mode that motivated the backup chantier.

Deliverables

  • Freshness check: daily job asserting the newest prod backup fingerprint is < 26 h old and the db+documents pair is complete; stale/missing → alert (interim mail/ntfy; gateway digest when live — arcodange/telegram-gateway#2).
  • Monthly restore drill, automated: restore the latest prod backup into the sandbox (the restore subcommand exists — proven on the sandbox 2026-07) → smoke-check (table count, company name, latest invoice ref) → report journaled.
  • Chart pattern: reuse chart/templates/backup-cronjob.yaml conventions (Values-gated, env-aware, OFF on sandbox by default).

Acceptance: backup CronJob suspended in a test window → alarm fires within a day; one automated drill green end-to-end with its report archived.


Execution — Blocked by: none · Blocks: P3 milestone closure · Human gates: none.
Start: claude --worktree t14-backup-watchdog from the erp repo root; read ops/backup/ and chart/templates/backup-cronjob.yaml first.
Done means: acceptance evidence posted as a comment here (alarm log + drill report — numbers/links, not claims) → PR merged closing this issue → SKILL.md/AGENTS.md/PRD-STATUS updated in the same PR wherever their claims change.

Part of the **AI back-office PRD** — [T14](https://gitea.arcodange.lab/arcodange-org/factory/src/branch/main/vibe/PRD/ai-back-office/task-inventory.md) + [challenge C7](https://gitea.arcodange.lab/arcodange-org/factory/src/branch/main/vibe/PRD/ai-back-office/challenges.md) (silence is an alert). **Split out of #48** (backlog sharpening 2026-07-12) so each watchdog ships as its own small PR — #48 keeps the ERP snapshot/drift side. **Context.** `ops/backup` runs daily on prod (CronJob + Vault creds; per-run fingerprints). Nothing watches the watcher: a silently stopped backup is invisible until the day it's needed — the exact failure mode that motivated the backup chantier. **Deliverables** - **Freshness check**: daily job asserting the newest prod backup fingerprint is < 26 h old and the db+documents pair is complete; stale/missing → alert (interim mail/ntfy; gateway digest when live — arcodange/telegram-gateway#2). - **Monthly restore drill, automated**: restore the latest prod backup into the sandbox (the `restore` subcommand exists — proven on the sandbox 2026-07) → smoke-check (table count, company name, latest invoice ref) → report journaled. - Chart pattern: reuse `chart/templates/backup-cronjob.yaml` conventions (Values-gated, env-aware, OFF on sandbox by default). **Acceptance:** backup CronJob suspended in a test window → alarm fires within a day; one automated drill green end-to-end with its report archived. --- **Execution** — Blocked by: none · Blocks: P3 milestone closure · Human gates: none. **Start:** `claude --worktree t14-backup-watchdog` from the erp repo root; read `ops/backup/` and `chart/templates/backup-cronjob.yaml` first. **Done means:** acceptance evidence posted as a comment here (alarm log + drill report — numbers/links, not claims) → PR merged closing this issue → SKILL.md/AGENTS.md/PRD-STATUS updated in the same PR wherever their claims change.
arcodange added this to the P3 — Standing fleet milestone 2026-07-12 13:59:09 +02:00
Sign in to join this conversation.
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: arcodange-org/erp#59