Incident 2026-07-23: an uncapped nuxt generate (3.5G RSS) on pi1 starved the k3s control-plane and traefik (load >150, no swap, no OOM-kill) — every *.arcodange.lab endpoint went dark, Gitea included, while Gitea itself was healthy on pi2. Job containers are spawned via the host docker socket, so cgroup caps on the job container are the only guardrail. Applied live on pi1+pi3 via 03_cicd.yml on 2026-07-24 (both runners re-registered; pi3's runner was down and is back in service). Co-Authored-By: Claude Fable 5 <[email protected]>
Use Ansible
Run locally (uv)
A project-local venv is defined in pyproject.toml at the repo root (ansible-core + the kubernetes, jmespath, dnspython libraries that kubernetes.core and friends need at runtime).
uv sync # creates .venv/ and installs ansible-core + python deps
uv run ansible-galaxy collection install -r ansible/requirements.yml
uv run ansible-playbook -i ansible/arcodange/factory/inventory ansible/arcodange/factory/playbooks/<playbook>.yml
The localhost entry in the inventory uses ansible_python_interpreter: "{{ ansible_playbook_python }}", so uv run is enough — Ansible picks up the venv's Python automatically without any hardcoded path.
Run with docker ssh agent side proxy
build docker images
git clone -q --depth 1 --branch master https://github.com/arcodange/ssh-agent.git /tmp/ssh-agent
(cd /tmp/ssh-agent ; docker build -t docker-ssh-agent:latest -f Dockerfile . ; rm -rf /tmp/ssh-agent)
(cd ansible; docker build -t arcodange-ansible:0.0.0 .)
run in container
# git clone -q --depth 1 --branch master https://github.com/arcodange/ssh-agent.git /tmp/ssh-agent
# (cd /tmp/ssh-agent ; docker build -t docker-ssh-agent:latest -f Dockerfile . ; rm -rf /tmp/ssh-agent)
# (cd ansible; docker build -t arcodange-ansible:0.0.0 .)
docker run -d --name=ssh-agent docker-ssh-agent:latest
docker run --rm --volumes-from=ssh-agent -v ~/.ssh:/.ssh -it docker-ssh-agent:latest ssh-add /root/.ssh/id_rsa
docker run --rm -u root --name test --volumes-from=ssh-agent -v $PWD:/home/arcodange/code \
-v "$HOME"/.kube/config:/home/arcodange/.kube/config \
-e ANSIBLE_VAULT_PASSWORD_FILE=$ANSIBLE_VAULT_PASSWORD_FILE -v $ANSIBLE_VAULT_PASSWORD_FILE:$ANSIBLE_VAULT_PASSWORD_FILE \
arcodange-ansible:0.0.0 \
ansible-playbook ansible/arcodange/factory/playbooks/03_cicd.yml -i ansible/arcodange/factory/inventory -vv
use vault with single password
Important
Required for gitea mailer
kubectl create secret generic arcodange-ansible-vault --from-literal="pass=<ansible_vault_password>" -n kube-system`to be set as a file variable for gitea runners
ANSIBLE_VAULT_PASSWORD_FILE=~/.local/bin/read-vault-key.sh;
mkdir -p `dirname $ANSIBLE_VAULT_PASSWORD_FILE`; set +o histexpand;
echo -e "#!/bin/bash\nkubectl get secret -n kube-system arcodange-ansible-vault --template='{{index .data.pass | base64decode}}'" > $ANSIBLE_VAULT_PASSWORD_FILE;
set -o histexpand; chmod +x $ANSIBLE_VAULT_PASSWORD_FILE; echo "export ANSIBLE_VAULT_PASSWORD_FILE=$ANSIBLE_VAULT_PASSWORD_FILE" >> `find ~ -maxdepth 1 -type f -name '\.*profile' -or -name '\.bashrc' -or -name '\.zshenv' | head -n1`
export ANSIBLE_VAULT_PASSWORD_FILE
a tool to reuse a ssh agent (not required)
FIND_SSH_AGENT=$HOME/.local/bin/ssh-find-agent
curl -s https://raw.githubusercontent.com/wwalker/ssh-find-agent/master/ssh-find-agent.sh > $FIND_SSH_AGENT
chmod +x $FIND_SSH_AGENT
echo 'ssh_find_agent "$@"' >> $FIND_SSH_AGENT
which brew && brew install coreutils # if on macos
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_rsa
dev
test an expression
ansible -i ,localhost -c local localhost -m raw -a "echo hello world {{ inventory_hostname }} : {{ hostvars | to_nice_json | regex_replace(\"['\n]\",' ') }}"
local python environment with uv
Install UV (one-time)
python3 -m pip install uv
python3 -m uv python install 3.12
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshenv
Bootstrap the project venv
uv sync # honors .python-version (3.12) and pyproject.toml
uv run ansible-galaxy collection install -r ansible/requirements.yml
# `--token <token>` is only needed if you hit galaxy.ansible.com rate limits
Run
uv run ansible-galaxy collection install ./ansible/arcodange/factory -f
uv run ansible-playbook -i ansible/arcodange/factory/inventory ansible/arcodange/factory/playbooks/02_setup.yml