diff --git a/ansible/arcodange/factory/playbooks/system/system_k3s.yml b/ansible/arcodange/factory/playbooks/system/system_k3s.yml index 57d55fe..31fce7d 100644 --- a/ansible/arcodange/factory/playbooks/system/system_k3s.yml +++ b/ansible/arcodange/factory/playbooks/system/system_k3s.yml @@ -42,14 +42,40 @@ # redémarré depuis avril, donc jamais régénéré par la version actuelle du # script), révélé par le premier restart forcé par ce playbook. La forme # SANS guillemets (`--kubelet-arg=k=v`) traverse la génération intacte. + # + # Ramasse-miettes d'images du kubelet : haut 85 → 65 %, bas 80 → 55 %. + # Les images Docker (data-root /mnt/arcodange/docker) partagent le disque + # externe avec Longhorn (/mnt/arcodange/longhorn), qui retire le disque de + # l'ordonnancement sous 25 % libres, soit dès 75 % occupés. Au défaut, le + # kubelet ne nettoie qu'à 85 % et s'arrête à 80 % : il laisse le disque + # EN PERMANENCE au-dessus de la ligne de Longhorn. Relevé le 2026-09-25 : + # pi3 à 82 % (491 Go), disque Longhorn « Schedulable=False », 272 + # anciennes images du front Kadans (~450 Mo chacune) jamais retirées. + # 65 % laisse 10 points de marge sous la ligne de Longhorn. + # + # ⚠ Ça s'applique BIEN au runtime Docker : `--docker` fait passer le + # kubelet par cri-dockerd (containerRuntimeEndpoint + # unix:///run/k3s/cri-dockerd/cri-dockerd.sock), qui lui rend le système + # de fichiers des images (491 Go = le disque externe) et exécute ses + # suppressions — `kubelet_image_garbage_collected_total{reason="space"}` + # valait 371 sur pi3 le 2026-09-25. Une image référencée par un conteneur, + # même arrêté (épingle de ci-node-playwright, runner, Gitea et Postgres en + # compose), est refusée par Docker : le kubelet passe à la suivante. + # + # Ne dépend PAS du disque externe : sans lui, les images tombent sur la + # carte SD, et le même seuil la protège. extra_server_args: >- --docker --disable traefik --kubelet-arg=container-log-max-files=5 --kubelet-arg=container-log-max-size=10Mi + --kubelet-arg=image-gc-high-threshold=65 + --kubelet-arg=image-gc-low-threshold=55 extra_agent_args: >- --docker --kubelet-arg=container-log-max-files=5 --kubelet-arg=container-log-max-size=10Mi + --kubelet-arg=image-gc-high-threshold=65 + --kubelet-arg=image-gc-low-threshold=55 {{ kubelet_reserved_args | default('') }} api_endpoint: "{{ hostvars[groups['server'][0]]['ansible_host'] | default(groups['server'][0]) }}" diff --git a/vibe/guidebooks/factory-provisioning/ansible/01-system.md b/vibe/guidebooks/factory-provisioning/ansible/01-system.md index 8e4b03f..9975b14 100644 --- a/vibe/guidebooks/factory-provisioning/ansible/01-system.md +++ b/vibe/guidebooks/factory-provisioning/ansible/01-system.md @@ -3,7 +3,7 @@ # 01 · System — base OS, Docker, K3s, Longhorn, DNS, SSL > [!NOTE] -> **Status:** ✅ active · **Last Updated:** 2026-06-23 +> **Status:** ✅ active · **Last Updated:** 2026-09-25 > **Upstream:** [Ansible sub-hub](README.md) · [Factory provisioning hub](../README.md) > **Downstream:** [02 · Setup](02-setup.md) · [03 · CI/CD](03-cicd.md) > **Related:** [Storage & recovery](../../lab-ecosystem/storage-and-recovery.md) · [Secrets & Vault](../../lab-ecosystem/secrets-and-vault.md) · [Naming conventions](../../lab-ecosystem/naming-conventions.md) · [ADR-0001 safe prod-like environment](../../../ADR/0001-safe-prod-like-environment.md) @@ -24,7 +24,7 @@ All host-facing plays target `raspberries:&local` — the intersection of the `r | 4 | [`system/prepare_disks.yml`](../../../../ansible/arcodange/factory/playbooks/system/prepare_disks.yml) | Auto-detect the largest external (non-`mmcblk0`) USB partition, format it **ext4 with label `arcodange_500`**, mount at `/mnt/arcodange`, and persist in `fstab`. Skips format if the label already exists. **`pause` confirm before any format.** | `mount_point: /mnt/arcodange`, `disk_label: arcodange_500` | | 5 | [`system/system_docker.yml`](../../../../ansible/arcodange/factory/playbooks/system/system_docker.yml) | Install Docker via `geerlingguy.docker`; write `daemon.json` with **json-file logging** (`max-size 10m`, `max-file 5`) and **`data-root: /mnt/arcodange/docker`** (only when the external disk is mounted). | `tags: never`; `storage-driver: overlay2` | | 6 | [`system/iscsi_longhorn.yml`](../../../../ansible/arcodange/factory/playbooks/system/iscsi_longhorn.yml) | Install `open-iscsi` (+ enable `iscsid`) and `cryptsetup`, and load the **`dm_crypt`** kernel module (persisted in `/etc/modules`) — Longhorn's encrypted-volume prerequisites. Creates `/mnt/arcodange/longhorn`. | module `dm_crypt` | -| 7 | [`system/system_k3s.yml`](../../../../ansible/arcodange/factory/playbooks/system/system_k3s.yml) | Build the K3s inventory dynamically (first sorted host → `server`, rest → `agent`), install the `k3s-ansible` content, run `k3s.orchestration.site`, then **fetch the kubeconfig** to `~/.kube/config` (rewriting `127.0.0.1` → server IP). | **k3s `v1.34.3+k3s1`**; server args `--docker --disable traefik` | +| 7 | [`system/system_k3s.yml`](../../../../ansible/arcodange/factory/playbooks/system/system_k3s.yml) | Build the K3s inventory dynamically (first sorted host → `server`, rest → `agent`), install the `k3s-ansible` content, run `k3s.orchestration.site`, then **fetch the kubeconfig** to `~/.kube/config` (rewriting `127.0.0.1` → server IP). | **k3s `v1.34.3+k3s1`**; server args `--docker --disable traefik`; kubelet args on server **and** agents: container logs capped (`5 × 10Mi`), **image GC at 65 % → 55 %** (below Longhorn's 75 % disk-pressure line) | | 8 | [`system/k3s_dns.yml`](../../../../ansible/arcodange/factory/playbooks/system/k3s_dns.yml) | Create the **`coredns-custom`** ConfigMap so cluster DNS forwards `arcodange.lab:53` to the Pi-hole IPs; also patch the main CoreDNS Corefile to forward to the same HA Pi-holes. | `pihole_ips` (extracted from hostvars) | | 9 | [`system/k3s_ssl.yml`](../../../../ansible/arcodange/factory/playbooks/system/k3s_ssl.yml) | Deploy **cert-manager** + **step-issuer** as k3s static HelmCharts; create the `StepClusterIssuer` `step-ca` wired to the JWK provisioner and root CA. | cert-manager `v1.19.2`, step-issuer `1.9.11`, `caUrl: https://ssl-ca.arcodange.lab:8443`, **ARM64 `kube-rbac-proxy` override** | | 10 | [`system/k3s_config.yml`](../../../../ansible/arcodange/factory/playbooks/system/k3s_config.yml) | Deploy **Longhorn** + **Traefik** as HelmCharts; issue the wildcard cert, set the default `TLSStore`, wire Gitea, the IP-allow-list middleware, and the CrowdSec bouncer plugin; then **delete the old Traefik** to force a redeploy. | Longhorn `v1.9.1`, Traefik `v37.4.0` (see detail below) | @@ -58,7 +58,7 @@ flowchart TD 4. **`prepare_disks.yml`** formats and mounts the external USB disk at `/mnt/arcodange` (with a confirmation pause). 5. **Docker** installs with its data-root pointed at that disk and capped logging. 6. **iSCSI + dm_crypt** prerequisites land so Longhorn can attach (and encrypt) volumes. -7. **K3s** installs with the first host as server, Docker as the container runtime, and Traefik disabled. +7. **K3s** installs with the first host as server, Docker as the container runtime (through k3s's embedded cri-dockerd), and Traefik disabled; the kubelet garbage-collects unused images from 65 % disk use down to 55 %, before Longhorn stops scheduling on the shared external disk at 75 %. 8. **CoreDNS** is reconfigured to forward `arcodange.lab` to the Pi-holes. 9. **cert-manager + step-issuer** wire the in-cluster issuer to step-ca. 10. **`k3s_config.yml`** deploys Longhorn and a fully-customized Traefik, then deletes the old Traefik so the helm-controller redeploys with the new config.