arcodange-org/factory
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(longhorn): document 2026-04-13 power-cut recovery + add data-recovery tooling

Browse Source 
Captures the post-mortem of the April 13 power-cut: incident timeline,
retrospective, and architecture/role diagrams. Adds an ADR explaining why
Longhorn cannot re-associate orphaned replica directories after a nuclear
reinstall (engine-id naming), plus block-device recovery runbooks and the
`playbooks/recover/longhorn_data.yml` automation that wires `merge-longhorn-layers.py`
to rebuild PVCs from raw `volume-head-*.img` chains.

Also extends the k3s_pvc backup to capture Longhorn `volumes`/`settings` CRDs
(needed for the fast-path restore) and rewrites the restore script with a
fallback dir + English messages.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
arcodange
2026-05-06 12:55:18 +02:00
co-authored by Claude Opus 4.7
parent 934b62d922
commit 1ae28cb944
20 changed files with 5939 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ansible/arcodange/factory/docs/adr/20260414-longhorn-pvc-recovery.md
+550
View File
@@ -0,0 +1,550 @@
# ADR 20260414: Longhorn PVC Recovery When Reinstalled
---
## 📋 **Executive Summary**
After the April 13, 2026 power cut incident and subsequent cluster recovery, we discovered a **critical gap** in Longhorn volume restoration. While the **raw replica data files** (`volume-head-*.img`) remain intact on disk across all nodes, Longhorn cannot automatically **re-associate** them with new Volume CRDs due to its internal engine ID naming scheme. This document explains the problem and provides three recovery approaches.
---
---
## 🔍 **The Root Problem**
### **What Happened**
1. **Power cut** → Longhorn CSI driver lost connection
2. **Force-deletion of Longhorn pods** → Webhook circular dependency
3. **Nuclear cleanup** → All Longhorn CRDs (Volume, Engine, Replica) were deleted
4. **Reinstallation** → New Volume CRDs created with new engine IDs
### **Directory Structure Issue**
Longhorn stores replica data in directories named by **volume name + engine ID**:
```
/mnt/arcodange/longhorn/replicas/
├── pvc-cc8a3cbb-dbc2-47a2-a0cc-a02136122b90-cd16e459/ # ← OLD (orphaned)
│ ├── volume-head-002.img # ← Actual Traefik data (128Mi)
│ ├── volume-head-002.img.meta
│ └── volume-snap-*.img
│
├── pvc-cc8a3cbb-dbc2-47a2-a0cc-a02136122b90-8c7d8ab4/ # ← NEW (empty)
│ ├── volume-head-002.img # ← Empty 128Mi
│ └── volume-head-002.img.meta
└── ...
```
**The Problem:** When you recreate a Volume CRD, Longhorn generates a **new engine ID** (e.g., `8c7d8ab4`), creating a **new empty directory** instead of adopting the existing one (`cd16e459`).
### **Why This Matters**
| Component | Persistence | Recovery Path |
|-----------|-------------|---------------|
| **Replica `.img` files** | ✅ **Survives** on disk | Manual intervention required |
| **Volume CRD** | ❌ **Deleted** | Must recreate |
| **Engine/Replica CRDs** | ❌ **Deleted** | Auto-recreated by Longhorn |
| **Engine ID** | ❌ **Changes** | ** Cannot be recovered without backup ** |
**Without the original Volume CRD backup, Longhorn cannot match orphaned replica directories to new Volume CRDs.**
---
---
## 🎯 **Recovery Methods Comparison**
| Method | Complexity | Data Safety | Downtime | Best For |
|--------|------------|-------------|----------|----------|
| **[A: Manual `dd` Copy](#method-a-manual-dd-copy)** | ⭐⭐⭐⭐ | ✅✅✅✅ | Medium | Critical data, no app backup |
| **[B: Directory Rename](#method-b-directory-rename)** | ⭐⭐⭐ | ✅✅ | Low | Small volumes, no Rebuilding replicas |
| **[C: Fresh Volume + App Restore](#method-c-fresh-volume--app-restore)** | ⭐⭐ | ✅✅✅✅✅ | Low | Non-critical data, app backups exist |
| **[D: Block-Device Injection (Automated)](#method-d-block-device-injection-automated)** | ⭐⭐⭐ | ✅✅✅✅ | Medium | **Recommended — any volume, no dir swap needed** |
| **[E: Longhorn Google Storage Restore](#method-e-longhorn-google-storage-restore)** | ⭐⭐ | ✅✅✅✅✅ | Low | Volumes with Longhorn backup configured |
**Method B was proven risky** (2026-04-13 recovery): Longhorn reconciliation finds `Dirty: true`
metadata + a clean empty pi1 replica → silently rebuilds from the empty source, destroying data.
Use Method D for any volume larger than ~128Mi or with Rebuilding replicas.
---
---
## 🛠️ **Method A: Manual `dd` Copy**
### **Concept**
Manually copy the data from the orphaned `.img` file to the new replica directory that Longhorn created for the new Volume CRD.
### **Prerequisites**
- Root access to all nodes
- Volume CRD already recreated (with new engine ID)
- Longhorn has created new empty replica directories
- `dd` and `qemu-img` tools available
### **Steps**
```bash
# 1. Identify source (old data) and destination (new empty)
SOURCE_NODE=pi2
SOURCE_DIR=/mnt/arcodange/longhorn/replicas/pvc-cc8a3cbb-dbc2-47a2-a0cc-a02136122b90-cd16e459
SOURCE_IMG=$(ssh $SOURCE_NODE "ls $SOURCE_DIR/volume-head-*.img | head -1")
DEST_DIRS=(
pi1:/mnt/arcodange/longhorn/pvc-cc8a3cbb-dbc2-47a2-a0cc-a02136122b90-8c7d8ab4
pi2:/mnt/arcodange/longhorn/pvc-cc8a3cbb-dbc2-47a2-a0cc-a02136122b90-8c7d8ab4
pi3:/mnt/arcodange/longhorn/pvc-cc8a3cbb-dbc2-47a2-a0cc-a02136122b90-8c7d8ab4
)
# 2. Copy data to each node
for DEST in "${DEST_DIRS[@]}"; do
NODE=${DEST%%:*}
PATH=${DEST#*:}
ssh $NODE "sudo mkdir -p $PATH && sudo dd if=$SOURCE_IMG of=$PATH/volume-head-002.img bs=4M"
done
# 3. Restart Longhorn engine pods to pick up new data
kubectl delete pod -n longhorn-system -l longhorn.io/component=engine
# 4. Verify data is accessible
kubectl get volume -n longhorn-system pvc-cc8a3cbb-dbc2-47a2-a0cc-a02136122b90
# Should show: state=attached, robustness=healthy
```
### **Pros**
- ✅ Guaranteed data recovery
- ✅ Works for any volume size
- ✅ Preserves all snapshots and metadata
### **Cons**
- ⚠️ Requires manual intervention on each node
- ⚠️ Must know source and destination paths
- ⚠️ Risk of data corruption if `dd` fails mid-copy
- ⚠️ Volume must be in detached state during copy
### **Risk Mitigation**
- Verify checksums after copy: `sha256sum /path/to/image.img`
- Copy to one node at a time, verify between each
- Use `pv` for progress: `pv $SOURCE_IMG | ssh $NODE "sudo dd of=$PATH/volume-head-002.img bs=4M"`
---
---
## 🏷️ **Method B: Directory Rename**
### **Concept**
Rename the orphaned replica directory to match the **engine ID** that Longhorn expects for the new Volume CRD.
### **Prerequisites**
- Volume CRD already recreated
- Longhorn has created engine CRDs (check: `kubectl get engines -n longhorn-system`)
- Must act quickly before Longhorn initializes new empty replicas
### **Steps**
```bash
# 1. Find the new engine ID for the volume
ENGINE=$(kubectl get engines -n longhorn-system -l longhorn.io/volume=pvc-cc8a3cbb-dbc2-47a2-a0cc-a02136122b90 -o jsonpath='{.items[0].metadata.name}')
# Example: pvc-cc8a3cbb-dbc2-47a2-a0cc-a02136122b90-e-0
ENGINE_ID=${ENGINE#*-} # Extract suffix: e-0
# But the directory uses a different format...
# 2. Check actual directory names
kubectl get replicas -n longhorn-system | grep pvc-cc8a
# Output: pvc-cc8a3cbb-dbc2-47a2-a0cc-a02136122b90-r-8c7d8ab4
# 3. Rename on the node where orphaned data exists
NEW_DIR_SUFFIX=$(kubectl get replicas -n longhorn-system pvc-cc8a3cbb-dbc2-47a2-a0cc-a02136122b90-r-8c7d8ab4 -o jsonpath='{.metadata.labels.longhorn\.io/last-attached-node}')
ssh $NEW_DIR_SUFFIX "sudo mv /mnt/arcodange/longhorn/replicas/pvc-cc8a3cbb-dbc2-47a2-a0cc-a02136122b90-cd16e459 \
/mnt/arcodange/longhorn/replicas/pvc-cc8a3cbb-dbc2-47a2-a0cc-a02136122b90-8c7d8ab4"
# 4. Restart the replica pod
kubectl delete pod -n longhorn-system $(kubectl get pods -n longhorn-system -o jsonpath='{.items[?(@.metadata.labels.longhorn\.io/replica)=pvc-cc8a3cbb-dbc2-47a2-a0cc-a02136122b90].metadata.name}')
```
### **Pros**
- ✅ Fastest method
- ✅ No data copying required
- ✅ Preserves all existing data and snapshots
### **Cons**
- ⚠️ **High risk of mismatch** - wrong directory rename = data loss
- ⚠️ Must identify correct engine ID for each node
- ⚠️ Replica directories exist on multiple nodes - must rename on ALL
- ⚠️ Longhorn may have already initialized new empty replicas
### **Critical Warning**
**Each volume has replicas on ALL nodes.** You must:
1. Identify which node has which orphaned directory
2. Rename each to match the corresponding new engine's expected path
3. Ensure consistency across all nodes
**Example for pvc-cc8a:**
```bash
# Orphaned dirs:
# pi2: pvc-cc8a...-cd16e459
# pi3: pvc-cc8a...-011b54b3
# New engine paths (from kubectl get replicas):
# pi1: pvc-cc8a...-r-8c7d8ab4
# pi2: pvc-cc8a...-r-32aa3e1e
# pi3: pvc-cc8a...-r-3e84c460
# Must rename EACH orphaned dir to match new engine on SAME node
```
---
---
## 🆕 **Method C: Fresh Volume + App Restore** *(Recommended for Traefik)*
### **Concept**
1. Let Longhorn create a **new empty volume** for the PVC
2. Restore the **application data** (Traefik's `acme.json`) from application-level backups
### **Prerequisites**
- Application-level backup exists (e.g., Traefik config, certificates)
- Data is non-critical or easily restorable
- Storage requirements are small (128Mi for Traefik)
### **Steps**
```bash
# 1. Delete the problematic Volume CRD (if any)
kubectl delete volume -n longhorn-system pvc-cc8a3cbb-dbc2-47a2-a0cc-a02136122b90 --ignore-not-found
# 2. Delete the PVC
kubectl delete pvc -n kube-system traefik
# 3. Let StorageClass provision a fresh volume
kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: traefik
namespace: kube-system
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 128Mi}}
storageClassName: longhorn
volumeMode: Filesystem
EOF
# 4. Wait for PV to be provisioned
kubectl wait --for=jsonpath='{.status.phase}'=Bound pvc -n kube-system traefik
# 5. Restore Traefik data from backup
BACKUP_FILE="/path/to/traefik-backup/acme.json"
kubectl cp $BACKUP_FILE kube-system/traefik-XXXXXX-XXXX:/data/acme.json
kubectl exec -n kube-system traefik-XXXXXX-XXXX -- chown 65532:65532 /data/acme.json
kubectl exec -n kube-system traefik-XXXXXX-XXXX -- chmod 600 /data/acme.json
```
### **Traefik-Specific Recovery**
For Traefik, the critical data is:
- `/data/acme.json` - TLS certificates obtained from Let's Encrypt
- `/data/tls.yml` - (if used)
- Secrets in Kubernetes (separate from PVC)
**Backup locations to check:**
```bash
# Check if we have Traefik data backups
ssh pi1 "ls -la /home/pi/arcodange/backups/traefik/ 2>/dev/null || echo 'No backup found'"
# Check ArgoCD apps (if Traefik was deployed via GitOps)
kubectl get app -n argocd | grep traefik
```
### **Pros**
- ✅ **Simplest and safest** method
- ✅ No risk of Longhorn directory mismatches
- ✅ Works even without Longhorn CRD backups
- ✅ Verifiable - you can confirm data was restored
- ✅ Clean state - no orphaned directories
### **Cons**
- ⚠️ Requires application-level backups
- ⚠️ TLS certificates may have expired (need to re-issue)
---
---
## 🏆 **Recommendation: Method C for Traefik**
### **Why Method C is Best for This Case**
| Factor | Assessment |
|--------|------------|
| **Volume Size** | 128Mi (small) |
| **Data Criticality** | TLS certs can be re-generated |
| **Backup Availability** | Likely exists in ArgoCD/Git |
| **Complexity** | Low |
| **Risk** | Minimal |
| **Time Required** | ~5 minutes |
### **Data Loss Assessment for Traefik**
The **worst case** (no Traefik backup):
- TLS certificates will be **re-issued** automatically by cert-manager + Let's Encrypt
- No permanent data loss - certificates are ephemeral
- Client impact: Brief TLS warning during re-issuance (~1-2 minutes)
**Verdict:** 🟢 **Method C is the safest and most practical approach.**
---
## 🔧 **Prevention: What We Must Fix**
### **1. Update Backup Playbook** (`playbooks/backup/k3s_pvc.yml`) ✅ Done 2026-04-16
`backup_cmd` now captures:
1. All PersistentVolumes (PV)
2. All PersistentVolumeClaims (PVC)
3. **All Longhorn Volumes** (critical — enables fast restore via `kubectl apply` instead of block-device injection)
4. All Longhorn Settings (backup target configuration)
### **2. Test Backups Regularly**
```bash
# Monthly test: Restore a non-critical volume
# Pick a test volume, delete it, restore from backup
kubectl delete volume -n longhorn-system <test-volume>
kubectl apply -f <backup-file>
kubectl get volume -n longhorn-system <test-volume> -w
```
### **3. Validate Backup Files**
```bash
# Check backup contains Longhorn resources
grep "longhorn.io/v1beta2" /path/to/backup-*.volumes
grep "kind: Volume" /path/to/backup-*.volumes
```
### **4. Document Recovery Procedure**
- [ ] Create `docs/admin/longhorn-recovery.md` with these steps
- [ ] Add to team runbook
- [ ] Include in incident response training
---
## 📊 **Test Scenario: Battle Testing PVC Recovery**
### **Test Setup**
```bash
# 1. Create a test namespace
kubectl create ns longhorn-test
# 2. Create a test PVC
kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-longhorn-recovery
namespace: longhorn-test
labels:
purpose: test
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 1Gi}}
storageClassName: longhorn
EOF
# 3. Deploy a test pod to write data
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: test-writer
namespace: longhorn-test
spec:
containers:
- name: writer
image: alpine
command: [sh, -c, "echo 'test data for recovery' > /data/testfile.txt && echo 'more data' >> /data/testfile.txt && tail -f /dev/null"]
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: test-longhorn-recovery
EOF
# 4. Write and verify data
kubectl exec -n longhorn-test test-writer -- cat /data/testfile.txt
# Should show: "test data for recovery\nmore data"
# 5. Backup everything
kubectl get -A pv,pvc -o yaml > /tmp/test-backup-pv-pvc.yaml
kubectl get -A volumes.longhorn.io -o yaml >> /tmp/test-backup-pv-pvc.yaml
echo '---' >> /tmp/test-backup-pv-pvc.yaml
kubectl get -A settings.longhorn.io -o yaml >> /tmp/test-backup-pv-pvc.yaml
```
### **Test Execution: Simulate Disaster**
```bash
# 6. Simulate disaster - delete everything
kubectl delete pvc -n longhorn-test test-longhorn-recovery
kubectl delete pod -n longhorn-test test-writer
kubectl delete volume -n longhorn-system pvc-$(kubectl get pvc -n longhorn-test test-longhorn-recovery -o jsonpath='{.spec.volumeName}')
# 7. Restore from backup
kubectl apply -f /tmp/test-backup-pv-pvc.yaml
# 8. Verify recovery
kubectl get pvc -n longhorn-test test-longhorn-recovery
kubectl get volumes -n longhorn-system | grep test-longhorn-recovery
# 9. Deploy test reader pod
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: test-reader
namespace: longhorn-test
spec:
containers:
- name: reader
image: alpine
command: [sh, -c, "cat /data/testfile.txt && tail -f /dev/null"]
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: test-longhorn-recovery
EOF
# 10. Check if data is recovered
kubectl logs -n longhorn-test test-reader
# Should show: "test data for recovery\nmore data"
```
### **Expected Results**
| Test Step | Pass Criteria |
|-----------|---------------|
| Volume CRD restored | `kubectl get volumes` shows the test volume |
| PVC bound | `kubectl get pvc` shows status=Bound |
| Data accessible | Test reader pod shows original data |
### **Test Cleanup**
```bash
kubectl delete ns longhorn-test
```
---
---
---
## 🛠️ **Method D: Block-Device Injection (Automated)**
### **Concept**
Bypass Longhorn's replica reconciliation entirely. Create a fresh Volume CRD, attach it in
maintenance mode, then inject the recovered filesystem directly into the live block device via
`rsync`. The old replica dirs are never renamed or touched — the data is copied into the new
Longhorn-managed volume.
### **Implementation**
See `playbooks/recover/longhorn_data.yml` — a 9-phase Ansible playbook that automates the full
sequence for one or more volumes in a single run.
### **Key Steps**
```
Phase 0: Auto-discover best replica dir (skip Rebuilding:true, rank by actual disk usage)
Phase 1: Backup untouched replica dir
Phase 2: Merge sparse snapshot+head layers → single flat image (merge-longhorn-layers.py)
Phase 3: Create Longhorn Volume CRD, wait for replicas
Phase 4: Scale down workload
Phase 5: Attach via VolumeAttachment maintenance ticket
Phase 6: mkfs.ext4 + mount + rsync from merged image
Phase 7: Remove maintenance ticket
Phase 8: Recreate PV (Retain, no claimRef) + PVC (volumeName pinned)
Phase 9: Scale up, wait readyReplicas ≥ 1
```
### **Usage**
```bash
ansible-playbook -i inventory/hosts.yml playbooks/recover/longhorn_data.yml \
-e @playbooks/recover/longhorn_data_vars.yml
```
Vars file format:
```yaml
longhorn_recovery_volumes:
- pv_name: pvc-abc123
pvc_name: myapp-data
namespace: myapp
size_bytes: "134217728"
size_human: 128Mi
access_mode: ReadWriteOnce
workload_kind: Deployment
workload_name: myapp
# source_node and source_dir are auto-discovered if omitted
verify_cmd: ""
```
### **Pros**
- ✅ Fully automated — handles all phases including PV/PVC recreation
- ✅ Auto-discovers best replica (skips Rebuilding dirs)
- ✅ Idempotent — safe to re-run (skips backup/merge if already done)
- ✅ Works for RWO and RWX volumes
### **Cons**
- ⚠️ Requires ~2× volume size in temporary disk space for merged image
- ⚠️ The new volume has 3 fresh replicas (not the original topology) — Longhorn will resync
---
---
## 🗄️ **Method E: Longhorn Google Storage Restore**
### **Concept**
Some volumes are configured with Longhorn's built-in backup feature targeting a Google Storage
bucket. For those volumes, a Longhorn backup can be restored into a new volume without needing
the raw replica files.
### **Applicable Volumes**
- `backups-rwx` (`pvc-efda1d2f`) — the cluster backup volume itself has a Longhorn GCS backup configured
### **When to use**
Use when:
- The local replica dirs are missing or corrupted (Method D cannot be used)
- A clean point-in-time restore is preferred over a raw replica merge
### **Status**
A playbook for this method (`playbooks/recover/longhorn_gcs_restore.yml`) is **planned but not
yet implemented**. In the 2026-04-13 incident, `backups-rwx` was successfully recovered via
Method D (local replica merge), so Method E was not needed.
When the playbook is implemented, it will use `kubectl apply` of a `BackupVolume` + `Backup`
restore CR pointing to the GCS bucket configured in Longhorn settings.
---
---
## 📚 **References**
- [Longhorn Documentation: Disaster Recovery](https://longhorn.io/docs/1.6.0/deploy/uninstall/disaster-recovery/)
- [Longhorn Volume CRD Spec](https://github.com/longhorn/longhorn/blob/master/types/types.go)
- [Original Issue: Longhorn GitHub #4837](https://github.com/longhorn/longhorn/issues/4837) (Replica orphan handling)
- [Related ADR: Internal DNS Architecture](./20260414-internal-dns-architecture.md)
- [Related Incident: 2026-04-13 Power Cut](../incidents/2026-04-13-power-cut/README.md)
---
---
*Document created: 2026-04-14*
*Last updated: 2026-04-15*
*Status: Method D (block-device injection) implemented and battle-tested on 5 volumes (2026-04-14/15)*