factory/ansible/arcodange/factory/docs/adr/20260407-cicd-architecture.md

ADR 20260407: CI/CD Architecture with ArgoCD, Gitea, and Vault

Status

Proposed

Context

The home lab requires a secure and automated CI/CD pipeline to deploy applications to the k3s cluster. The pipeline must integrate with:

• Gitea: For Git repository management and CI runners.
• ArgoCD: For GitOps-based continuous deployment.
• Vault: For secrets management and OIDC authentication.
• Gitea Act Runner: For executing CI jobs.

Decision

We will implement a GitOps-driven CI/CD pipeline with the following components:

1. Gitea OIDC Authentication with Vault

• Gitea is registered as an OIDC application in Vault.
• Vault issues short-lived tokens for Gitea users.
• The gitea_oidc_auth.yml playbook automates this setup using Playwright and OpenTofu.
• OIDC Workflow:
  1. The oidc_jwt_token.sh script (base64-encoded in secrets.vault_oauth__sh_b64) handles the OIDC flow.
  2. Gitea Act Runner executes the script to obtain an ID token from Gitea.
  3. The ID token is used to authenticate with Vault and retrieve secrets.

2. Gitea Act Runner

• Deployed on pi1 and pi3 (not on the Gitea host, which is pi2).
• Uses Docker-in-Docker for job execution.
• Custom Runner Image (ubuntu-latest-ca): Required due to the self-signed .lab domain. The custom image includes the local CA certificate to trust the Gitea instance (gitea.arcodange.lab).
• Managed via Docker Compose (03_cicd.yml).

3. ArgoCD

• Deployed on the k3s cluster (via HelmChart in /var/lib/rancher/k3s/server/manifests/argocd.yaml).
• Uses Gitea as the source of truth for GitOps.
• Synchronizes the factory repository to deploy applications.
• Configured with Traefik for TLS termination.

4. Vault Secrets Operator

• Deployed in the tools namespace.
• Manages secrets for applications deployed via ArgoCD.
• Integrates with Gitea OIDC for authentication.
• Helm Chart Integration:
  • VaultAuth: Authenticates with Vault using Kubernetes service accounts.
  • VaultStaticSecret: Retrieves static secrets (e.g., kvv2/webapp/config).
  • VaultDynamicSecret: Generates dynamic secrets (e.g., PostgreSQL credentials).

5. Security

• TLS: Traefik terminates TLS using Let's Encrypt.
• OIDC: Gitea authentication via Vault.
• Secrets: Stored in Vault, injected via the Vault Secrets Operator.

Architecture Diagram

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#333333', 'edgeLabelBackground':'#f0f0f0', 'tertiaryColor': '#e67e22'}}}%%
graph TD
    %% Styles
    classDef gitea fill:#ffcc99,stroke:#cc9966,color:#333;
    classDef argocd fill:#99ffcc,stroke:#66cc99,color:#333;
    classDef vault fill:#ccccff,stroke:#6666cc,color:#333;
    classDef k3s fill:#ff9999,stroke:#cc0000,color:#333;
    classDef runner fill:#ffff99,stroke:#cccc00,color:#333;

    %% Components
    Gitea["Gitea (pi2)"]:::gitea
    ArgoCD["ArgoCD (k3s)"]:::argocd
    Vault["Vault (k3s/tools)"]:::vault
    Runner1["Gitea Act Runner (pi1)"]:::runner
    Runner2["Gitea Act Runner (pi3)"]:::runner
    VaultOperator["Vault Secrets Operator (k3s/tools)"]:::vault
    k3s["k3s Cluster"]:::k3s

    %% Workflow
    Gitea -->|OIDC Auth| Vault
    Gitea -->|Trigger CI| Runner1
    Gitea -->|Trigger CI| Runner2
    Runner1 -->|Deploy to| k3s
    Runner2 -->|Deploy to| k3s
    ArgoCD -->|GitOps Sync| Gitea
    ArgoCD -->|Deploy Apps| k3s
    VaultOperator -->|Inject Secrets| k3s
    Vault -->|Secrets| VaultOperator

    %% Annotations
    linkStyle 0,1,2,3,4,5,6,7 stroke:#999,stroke-width:1px;

Consequences

Positive

• Automated Deployments: ArgoCD ensures the cluster state matches Git.
• Secure Secrets: Vault centralizes secret management.
• Scalable CI: Gitea Act Runners can be added to any host.
• OIDC Integration: Secure authentication via Vault.

Negative

• Complexity: Multiple moving parts (Gitea, ArgoCD, Vault).
• Dependency on Vault: If Vault fails, CI/CD may be disrupted.
• Learning Curve: Requires familiarity with GitOps and Vault.

Alternatives Considered

Alternative 1: GitHub Actions

• Rejected: Self-hosted Gitea aligns better with the home lab's privacy goals.

Alternative 2: Jenkins

• Rejected: ArgoCD + Gitea Act Runner is lighter and more GitOps-native.

Alternative 3: No CI/CD

• Rejected: Manual deployments are error-prone and unscalable.

Sequence Diagrams

1. CI/CD Workflow for OpenTofu/Terraform

sequenceDiagram
    participant Gitea
    participant Runner as Gitea Act Runner (pi1/pi3)
    participant Vault
    participant WebApp as WebApp (k3s)

    Gitea->>Runner: Trigger vault.yaml workflow
    Runner->>Gitea: Execute vault_oauth__sh_b64 (OIDC)
    Gitea-->>Runner: Return ID Token
    Runner->>Vault: Authenticate with ID Token
    Vault-->>Runner: Return Vault Token
    Runner->>Runner: Run OpenTofu/Terraform
    Runner->>Vault: Fetch Secrets (via Vault Action)
    Vault-->>Runner: Return Secrets
    Runner->>WebApp: Deploy Changes

2. Vault Secrets Operator Workflow

sequenceDiagram
    participant ArgoCD
    participant WebApp as WebApp (k3s)
    participant VaultOperator as Vault Secrets Operator
    participant Vault

    ArgoCD->>WebApp: Deploy Helm Chart
    WebApp->>VaultOperator: Create VaultAuth (K8s Auth)
    VaultOperator->>Vault: Authenticate (K8s Service Account)
    Vault-->>VaultOperator: Return Vault Token
    WebApp->>VaultOperator: Create VaultStaticSecret (kvv2/webapp/config)
    VaultOperator->>Vault: Fetch Static Secret
    Vault-->>VaultOperator: Return Secret
    VaultOperator->>WebApp: Inject Secret (secretkv)
    WebApp->>VaultOperator: Create VaultDynamicSecret (postgres/creds/webapp)
    VaultOperator->>Vault: Generate Dynamic Secret
    Vault-->>VaultOperator: Return Credentials
    VaultOperator->>WebApp: Inject Credentials (vso-db-credentials)
    WebApp->>WebApp: Restart Pods (Rollout)

Success Metrics

• Gitea Act Runners successfully execute CI jobs.
• ArgoCD synchronizes the factory repository without errors.
• Vault Secrets Operator injects secrets into deployed applications.