Helm Charts / Detect changed charts (pull_request) Successful in 1m27s
Helm Charts / Library charts tool (pull_request) Skipped
Helm Charts / Application charts alloy (pull_request) Skipped
Helm Charts / Application charts chart (pull_request) Skipped
Helm Charts / Application charts crowdsec (pull_request) Skipped
Helm Charts / Application charts grafana (pull_request) Skipped
Helm Charts / Application charts loki (pull_request) Skipped
Helm Charts / Application charts minio (pull_request) Skipped
Helm Charts / Application charts pgbouncer (pull_request) Skipped
Helm Charts / Application charts pgcat (pull_request) Skipped
Helm Charts / Application charts redis (pull_request) Skipped
Helm Charts / Application charts hashicorp-vault (pull_request) Successful in 18s
Helm Charts / Application charts prometheus (pull_request) Successful in 20s
INCIDENT
Le 30/08 à 11:35, pi3 passe de 3,6 Gio de mémoire disponible à 1,7 en cinq
minutes. À 11:45 node-exporter ne répond plus, à 11:48 le nœud est NotReady, et
300 s plus tard — la tolérance `node.kubernetes.io/unreachable:NoExecute` — le
node-lifecycle controller supprime quatre pods, dont hashicorp-vault-0.
Vault repart SCELLÉ, comme le prévoit Shamir 1/1 sans auto-unseal. Personne ne
le redescelle. Pendant onze jours le VSO répond 503 « Vault is sealed » et plus
aucun Secret n'est réconcilié — découvert par hasard, en enquêtant sur autre
chose.
Ce qui a fonctionné ce jour-là : `NoeudInjoignable` a tiré à 11:46. L'infra a
parlé. Ce qui a manqué : la CONSÉQUENCE. Le nœud est revenu, les pods ont été
recréés, l'alerte s'est éteinte, et personne n'a su que Vault, lui, resterait
scellé.
Ce n'était ni un crash ni un OOMKill : `lastState: {}` et `restartCount: 0` sur
un pod vieux de 11 jours. Et le StatefulSet n'a aucune livenessProbe — seulement
une readiness `exec: vault status`, dont l'échec ne redémarre rien. D'où 196 906
events Unhealthy pour zéro redémarrage.
1. L'ALERTE QUI MANQUAIT — VaultIndisponible
`kube_pod_status_ready{namespace="tools", pod=~"hashicorp-vault-[0-9]+"} == 0`
pendant 10 min, avec `or absent(...)` pour couvrir la disparition du pod.
Pourquoi kube-state-metrics et pas `vault_core_unsealed` : Vault n'expose ses
métriques que via /v1/sys/metrics, qui exige un token ou l'ouverture d'un
endpoint non authentifié. Or la readinessProbe du chart est littéralement
`vault status` : un Vault scellé est NotReady. kube-state-metrics est déjà
scrapé, ça ne coûte rien et ça ne touche pas à la configuration de Vault.
⚠ Le sélecteur est ancré sur `[0-9]+`, et c'est une correction, pas un détail :
ma première version utilisait `hashicorp-vault-.*`, qui attrape aussi les pods
du Vault Secrets Operator — dont un exemplaire terminé traîne en permanence. La
règle tirait donc en continu. Vérifié contre le Prometheus du cluster : avec
`[0-9]+` elle ne rend que hashicorp-vault-0 et ne tire pas.
2. VAULT N'EST PLUS LE POD LE PLUS FAIBLE DU CLUSTER
Mesuré avant : `priorityClassName=""`, `priority=0`, QoS `BestEffort`. Dernier
servi par le scheduler, premier évincé par le kubelet. Sur 114 pods, 78 sont
BestEffort — Vault était noyé dedans, alors que tout le cluster dépend de lui
et qu'il exige une intervention humaine pour revenir.
- PriorityClass `vault-critical` à 900000000. Sous `longhorn-critical`
(1000000000) parce que Vault stocke sur un PVC : le stockage doit survivre à
Vault. Au-dessus de tout le reste.
- `requests: {memory: 512Mi, cpu: 100m}`. Vault consomme 175 Mio : très en
dessous de sa requête, donc tout en bas de la liste d'éviction.
AUCUNE LIMITE, DÉLIBÉRÉMENT. Une limite trop basse déclenche un OOMKill, et un
OOMKill sur ce pod coûte un descellement manuel. Or je n'ai pas pu établir de
pic mémoire fiable : les séries cAdvisor de ce cluster rendent huit valeurs
contradictoires pour ce pod, jusqu'à 2,3 Gio, ce qui est invraisemblable pour un
Vault en storage "file". On ne pose pas une limite sur un chiffre auquel on ne
croit pas.
3. LE PDB EXISTANT : GARDÉ, MAIS SON PIÈGE EST MAINTENANT ÉCRIT
`maxUnavailable: 0` sur un StatefulSet à un réplica donne ALLOWED DISRUPTIONS: 0
en permanence. Un `kubectl drain` du nœud qui héberge Vault ne se terminera donc
jamais. C'est le comportement voulu — il force à traiter Vault consciemment —
mais la sortie de secours (`--disable-eviction`, ou suppression manuelle du pod)
n'était écrite nulle part. Elle l'est.
CE QUE ÇA NE RÉSOUT PAS, ET IL FAUT LE DIRE
Ni la priorité ni les requests n'empêchent la suppression par le node-lifecycle
controller quand un nœud devient injoignable — c'est exactement ce qui est
arrivé le 30/08, et aucun réglage de pod ne l'évite. Ce cas-là est traité par
l'alerte, pas par la protection.
L'auto-unseal reste écarté : décision assumée, documentée dans factory
vibe/guidebooks/lab-ecosystem/secrets-and-vault.md. Le descellement restera
manuel ; c'est le délai de détection qui passe de onze jours à dix minutes.
⚠ APPLICATION : le StatefulSet est en `updateStrategy: OnDelete`. Les réglages
du point 2 ne prendront effet qu'à la prochaine suppression du pod — laquelle
rescellera Vault. À faire au moment choisi, clé sous la main.
Vérifié : `helm template` rend la PriorityClass, le PDB et un StatefulSet
portant `priorityClassName: vault-critical` et les requests ; `helm lint` passe ;
la requête de l'alerte testée contre le Prometheus du cluster.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1564 lines
59 KiB
YAML
1564 lines
59 KiB
YAML
prometheus: &prometheus_config
|
|
# yaml-language-server: $schema=values.schema.json
|
|
# Default values for prometheus.
|
|
# This is a YAML-formatted file.
|
|
# Declare variables to be passed into your templates.
|
|
rbac:
|
|
create: true
|
|
|
|
imagePullSecrets: []
|
|
# - name: "image-pull-secret"
|
|
|
|
## Define serviceAccount names for components. Defaults to component's fully qualified name.
|
|
##
|
|
serviceAccounts:
|
|
server:
|
|
create: true
|
|
name: ""
|
|
annotations: {}
|
|
|
|
## Opt out of automounting Kubernetes API credentials.
|
|
## It will be overriden by server.automountServiceAccountToken value, if set.
|
|
# automountServiceAccountToken: false
|
|
|
|
## Additional labels to attach to all resources
|
|
commonMetaLabels: {}
|
|
|
|
## Monitors ConfigMap changes and POSTs to a URL
|
|
## Ref: https://github.com/prometheus-operator/prometheus-operator/tree/main/cmd/prometheus-config-reloader
|
|
##
|
|
configmapReload:
|
|
## URL for configmap-reload to use for reloads
|
|
##
|
|
reloadUrl: ""
|
|
|
|
## env sets environment variables to pass to the container. Can be set as name/value pairs,
|
|
## read from secrets or configmaps.
|
|
env: []
|
|
# - name: SOMEVAR
|
|
# value: somevalue
|
|
# - name: PASSWORD
|
|
# valueFrom:
|
|
# secretKeyRef:
|
|
# name: mysecret
|
|
# key: password
|
|
# optional: false
|
|
|
|
prometheus:
|
|
## If false, the configmap-reload container will not be deployed
|
|
##
|
|
enabled: true
|
|
|
|
## configmap-reload container name
|
|
##
|
|
name: configmap-reload
|
|
|
|
## configmap-reload container image
|
|
##
|
|
image:
|
|
repository: quay.io/prometheus-operator/prometheus-config-reloader
|
|
tag: v0.89.0
|
|
# When digest is set to a non-empty value, images will be pulled by digest (regardless of tag value).
|
|
digest: ""
|
|
pullPolicy: IfNotPresent
|
|
|
|
## config-reloader's container port and port name for probes and metrics
|
|
containerPort: 8080
|
|
containerPortName: metrics
|
|
|
|
## Additional configmap-reload container arguments
|
|
## Set to null for argumentless flags
|
|
##
|
|
extraArgs: {}
|
|
|
|
## Additional configmap-reload volume directories
|
|
##
|
|
extraVolumeDirs: []
|
|
|
|
## Additional configmap-reload volume mounts
|
|
##
|
|
extraVolumeMounts: []
|
|
|
|
## Additional configmap-reload mounts
|
|
##
|
|
extraConfigmapMounts: []
|
|
# - name: prometheus-alerts
|
|
# mountPath: /etc/alerts.d
|
|
# subPath: ""
|
|
# configMap: prometheus-alerts
|
|
# readOnly: true
|
|
|
|
## Security context to be added to configmap-reload container
|
|
containerSecurityContext: {}
|
|
|
|
## Settings for Prometheus reloader's readiness, liveness and startup probes
|
|
## Ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
|
|
##
|
|
|
|
livenessProbe:
|
|
httpGet:
|
|
path: /healthz
|
|
port: metrics
|
|
scheme: HTTP
|
|
periodSeconds: 10
|
|
initialDelaySeconds: 2
|
|
|
|
readinessProbe:
|
|
httpGet:
|
|
path: /healthz
|
|
port: metrics
|
|
scheme: HTTP
|
|
periodSeconds: 10
|
|
|
|
startupProbe:
|
|
enabled: false
|
|
httpGet:
|
|
path: /healthz
|
|
port: metrics
|
|
scheme: HTTP
|
|
periodSeconds: 10
|
|
|
|
## configmap-reload resource requests and limits
|
|
## Ref: http://kubernetes.io/docs/user-guide/compute-resources/
|
|
##
|
|
resources: {}
|
|
|
|
server:
|
|
## Prometheus server container name
|
|
##
|
|
name: server
|
|
|
|
## Opt out of automounting Kubernetes API credentials.
|
|
## If set it will override serviceAccounts.server.automountServiceAccountToken value for ServiceAccount.
|
|
# automountServiceAccountToken: false
|
|
|
|
## If set it will override prometheus.server.fullname value for ClusterRole and ClusterRoleBinding
|
|
##
|
|
clusterRoleNameOverride: ""
|
|
|
|
## Name of an existing cluster role to use in a role binding in namespaces set in
|
|
## namespaces and releaseNamespace for namespaced discovery.
|
|
##
|
|
useExistingClusterRoleName: ""
|
|
|
|
## releaseNamespace to enable only the release namespace for service discovery.
|
|
## By default all namespaces are included in service discovery.
|
|
## If releaseNamespace and namespaces are both set, a merged list will be created.
|
|
## Note that kubernetes_sd_configs.namespaces in scrape configs must be specified
|
|
## if namespaced service discovery is desired. Setting useExistingClusterRoleName is required.
|
|
##
|
|
releaseNamespace: false
|
|
|
|
## namespaces to include in service discovery instead of clusterwide discovery. Needed if you want to run
|
|
## Prometheus without cluster-admin privileges (namespaced configuration). See also releaseNamespace.
|
|
## Setting useExistingClusterRoleName is required.
|
|
##
|
|
namespaces: []
|
|
# - yournamespace
|
|
|
|
# sidecarContainers - add more containers to prometheus server
|
|
# Key/Value where Key is the sidecar `- name: <Key>`
|
|
# Example:
|
|
# sidecarContainers:
|
|
# webserver:
|
|
# image: nginx
|
|
# OR for adding OAuth authentication to Prometheus
|
|
# sidecarContainers:
|
|
# oauth-proxy:
|
|
# image: quay.io/oauth2-proxy/oauth2-proxy
|
|
# args:
|
|
# - --upstream=http://127.0.0.1:9090
|
|
# - --http-address=0.0.0.0:8081
|
|
# - ...
|
|
# ports:
|
|
# - containerPort: 8081
|
|
# name: oauth-proxy
|
|
# protocol: TCP
|
|
# resources: {}
|
|
sidecarContainers: {}
|
|
|
|
# sidecarTemplateValues - context to be used in template for sidecarContainers
|
|
# Example:
|
|
# sidecarTemplateValues: *your-custom-globals
|
|
# sidecarContainers:
|
|
# webserver: |-
|
|
# {{ include "webserver-container-template" . }}
|
|
# Template for `webserver-container-template` might looks like this:
|
|
# image: "{{ .Values.server.sidecarTemplateValues.repository }}:{{ .Values.server.sidecarTemplateValues.tag }}"
|
|
# ...
|
|
#
|
|
sidecarTemplateValues: {}
|
|
|
|
## Prometheus server container image
|
|
##
|
|
image:
|
|
repository: quay.io/prometheus/prometheus
|
|
# If not set appVersion field from Chart.yaml is used. Supports templating.
|
|
tag: ""
|
|
# When digest is set to a non-empty value, images will be pulled by digest (regardless of tag value).
|
|
digest: ""
|
|
# Use the distroless image variant.
|
|
distroless: false
|
|
pullPolicy: IfNotPresent
|
|
|
|
## Prometheus server command
|
|
##
|
|
command: []
|
|
|
|
## prometheus server priorityClassName
|
|
##
|
|
priorityClassName: ""
|
|
## prometheus server runtimeClassName
|
|
##
|
|
runtimeClassName: ""
|
|
|
|
## EnableServiceLinks indicates whether information about services should be injected
|
|
## into pod's environment variables, matching the syntax of Docker links.
|
|
## WARNING: the field is unsupported and will be skipped in K8s prior to v1.13.0.
|
|
##
|
|
enableServiceLinks: true
|
|
|
|
## The URL prefix at which the container can be accessed. Useful in the case the '-web.external-url' includes a slug
|
|
## so that the various internal URLs are still able to access as they are in the default case.
|
|
## (Optional)
|
|
prefixURL: ""
|
|
|
|
## External URL which can access prometheus
|
|
## Maybe same with Ingress host name
|
|
baseURL: ""
|
|
|
|
## Additional server container environment variables
|
|
##
|
|
## You specify this manually like you would a raw deployment manifest.
|
|
## This means you can bind in environment variables from secrets.
|
|
##
|
|
## e.g. static environment variable:
|
|
## - name: DEMO_GREETING
|
|
## value: "Hello from the environment"
|
|
##
|
|
## e.g. secret environment variable:
|
|
## - name: USERNAME
|
|
## valueFrom:
|
|
## secretKeyRef:
|
|
## name: mysecret
|
|
## key: username
|
|
env: []
|
|
|
|
# List of flags to override default parameters, e.g:
|
|
# - --enable-feature=agent
|
|
# - --storage.agent.retention.max-time=30m
|
|
# - --config.file=/etc/config/prometheus.yml
|
|
defaultFlagsOverride: []
|
|
|
|
extraFlags:
|
|
- web.enable-lifecycle
|
|
## web.enable-admin-api flag controls access to the administrative HTTP API which includes functionality such as
|
|
## deleting time series. This is disabled by default.
|
|
# - web.enable-admin-api
|
|
##
|
|
## storage.tsdb.no-lockfile flag controls BD locking
|
|
# - storage.tsdb.no-lockfile
|
|
##
|
|
## storage.tsdb.wal-compression flag enables compression of the write-ahead log (WAL)
|
|
# - storage.tsdb.wal-compression
|
|
|
|
## Path to a configuration file on prometheus server container FS
|
|
configPath: /etc/config/prometheus.yml
|
|
|
|
### The data directory used by prometheus to set --storage.tsdb.path
|
|
### When empty server.persistentVolume.mountPath is used instead
|
|
storagePath: ""
|
|
|
|
global:
|
|
## How frequently to scrape targets by default
|
|
##
|
|
scrape_interval: 1m
|
|
## How long until a scrape request times out
|
|
##
|
|
scrape_timeout: 10s
|
|
## How frequently to evaluate rules
|
|
##
|
|
evaluation_interval: 1m
|
|
## https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write
|
|
##
|
|
remoteWrite: []
|
|
## https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_read
|
|
##
|
|
remoteRead: []
|
|
|
|
## https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tsdb
|
|
##
|
|
tsdb: {}
|
|
# out_of_order_time_window: 0s
|
|
|
|
## https://prometheus.io/docs/guides/opentelemetry
|
|
##
|
|
otlp: {}
|
|
# promote_resource_attributes: []
|
|
# keep_identifying_resource_attributes: false
|
|
# translation_strategy: NoUTF8EscapingWithSuffixes
|
|
|
|
## https://prometheus.io/docs/prometheus/latest/configuration/configuration/#exemplars
|
|
## Must be enabled via --enable-feature=exemplar-storage
|
|
##
|
|
exemplars: {}
|
|
# max_exemplars: 100000
|
|
|
|
## Custom HTTP headers for Liveness/Readiness/Startup Probe
|
|
##
|
|
## Useful for providing HTTP Basic Auth to healthchecks
|
|
probeHeaders: []
|
|
# - name: "Authorization"
|
|
# value: "Bearer ABCDEabcde12345"
|
|
|
|
## Additional Prometheus server container arguments
|
|
## Set to null for argumentless flags
|
|
##
|
|
extraArgs: {}
|
|
# web.enable-remote-write-receiver: null
|
|
|
|
## Additional InitContainers to initialize the pod
|
|
##
|
|
extraInitContainers: []
|
|
|
|
## Additional Prometheus server Volume mounts
|
|
##
|
|
extraVolumeMounts: []
|
|
|
|
## Additional Prometheus server Volumes
|
|
##
|
|
extraVolumes: []
|
|
|
|
## Additional Prometheus server hostPath mounts
|
|
##
|
|
extraHostPathMounts: []
|
|
# - name: certs-dir
|
|
# mountPath: /etc/kubernetes/certs
|
|
# subPath: ""
|
|
# hostPath: /etc/kubernetes/certs
|
|
# readOnly: true
|
|
|
|
extraConfigmapMounts: []
|
|
# - name: certs-configmap
|
|
# mountPath: /prometheus
|
|
# subPath: ""
|
|
# configMap: certs-configmap
|
|
# readOnly: true
|
|
|
|
## Additional Prometheus server Secret mounts
|
|
# Defines additional mounts with secrets. Secrets must be manually created in the namespace.
|
|
extraSecretMounts: []
|
|
# - name: secret-files
|
|
# mountPath: /etc/secrets
|
|
# subPath: ""
|
|
# secretName: prom-secret-files
|
|
# readOnly: true
|
|
|
|
## Prometheus server configuration from a secret
|
|
## Do not set both `configMapOverrideName` and `configFromSecret` simultaneously.
|
|
## Use either `configMapOverrideName` or `configFromSecret`.
|
|
## If `configFromSecret` is defined, a ConfigMap resource will NOT be generated.
|
|
configFromSecret: ""
|
|
|
|
## ConfigMap override where fullname is {{.Release.Name}}-{{.Values.server.configMapOverrideName}}
|
|
## Defining configMapOverrideName will cause templates/server-configmap.yaml
|
|
## to NOT generate a ConfigMap resource
|
|
##
|
|
configMapOverrideName: ""
|
|
|
|
## Extra labels for Prometheus server ConfigMap (ConfigMap that holds serverFiles)
|
|
extraConfigmapLabels: {}
|
|
|
|
## Override the prometheus.server.fullname for all objects related to the Prometheus server
|
|
fullnameOverride: ""
|
|
|
|
ingress:
|
|
## If true, Prometheus server Ingress will be created
|
|
##
|
|
enabled: false
|
|
|
|
ingressClassName: ""
|
|
|
|
## Prometheus server Ingress annotations
|
|
##
|
|
annotations: {}
|
|
# kubernetes.io/ingress.class: nginx
|
|
# kubernetes.io/tls-acme: 'true'
|
|
|
|
## Prometheus server Ingress additional labels
|
|
##
|
|
extraLabels: {}
|
|
|
|
## Redirect ingress to an additional defined port on the service
|
|
# servicePort: 8081
|
|
|
|
## Prometheus server Ingress hostnames with optional path (passed through tpl)
|
|
## Must be provided if Ingress is enabled
|
|
##
|
|
hosts: []
|
|
# - prometheus.domain.com
|
|
# - domain.com/prometheus
|
|
|
|
path: /
|
|
|
|
# pathType determines the interpretation of the path matching
|
|
pathType: Prefix
|
|
|
|
## Extra paths to prepend to every host configuration. This is useful when working with annotation based services. (passed through tpl)
|
|
extraPaths: []
|
|
# - path: /*
|
|
# backend:
|
|
# serviceName: ssl-redirect
|
|
# servicePort: use-annotation
|
|
|
|
## Prometheus server Ingress TLS configuration (hosts passed through tpl)
|
|
## Secrets must be manually created in the namespace
|
|
##
|
|
tls: []
|
|
# - secretName: prometheus-server-tls
|
|
# hosts:
|
|
# - prometheus.domain.com
|
|
|
|
## route (map) allows configuration of HTTPRoute resources
|
|
## Requires Gateway API resources and suitable controller installed within the cluster
|
|
## Ref. https://gateway-api.sigs.k8s.io/guides/http-routing/
|
|
route:
|
|
main:
|
|
## Enable this route
|
|
enabled: false
|
|
|
|
## ApiVersion set by default to "gateway.networking.k8s.io/v1"
|
|
apiVersion: ""
|
|
## kind set by default to HTTPRoute
|
|
kind: ""
|
|
|
|
## Annotations to attach to the HTTPRoute resource
|
|
annotations: {}
|
|
## Labels to attach to the HTTPRoute resource
|
|
labels: {}
|
|
|
|
## ParentRefs refers to resources this HTTPRoute is to be attached to (Gateways)
|
|
parentRefs: []
|
|
# - name: contour
|
|
# sectionName: http
|
|
|
|
## Hostnames (templated) defines a set of hostnames that should match against the HTTP Host
|
|
## header to select a HTTPRoute used to process the request
|
|
hostnames: []
|
|
# - my.example.com
|
|
|
|
## additionalRules (templated) allows adding custom rules to the route
|
|
additionalRules: []
|
|
|
|
## Filters define the filters that are applied to requests that match
|
|
## this rule
|
|
filters: []
|
|
|
|
## Matches define conditions used for matching the rule against incoming
|
|
## HTTP requests
|
|
matches:
|
|
- path:
|
|
type: PathPrefix
|
|
value: /
|
|
|
|
## httpsRedirect adds a filter for redirecting to https (HTTP 301 Moved Permanently).
|
|
## To redirect HTTP traffic to HTTPS, you need to have a Gateway with both HTTP and HTTPS listeners.
|
|
## Matches and filters do not take effect if enabled.
|
|
## Ref. https://gateway-api.sigs.k8s.io/guides/http-redirect-rewrite/
|
|
httpsRedirect: false
|
|
|
|
## Server Deployment Strategy type
|
|
strategy:
|
|
type: Recreate
|
|
|
|
## hostAliases allows adding entries to /etc/hosts inside the containers
|
|
hostAliases: []
|
|
# - ip: "127.0.0.1"
|
|
# hostnames:
|
|
# - "example.com"
|
|
|
|
## Node tolerations for server scheduling to nodes with taints
|
|
## Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/
|
|
##
|
|
tolerations: []
|
|
# - key: "key"
|
|
# operator: "Equal|Exists"
|
|
# value: "value"
|
|
# effect: "NoSchedule|PreferNoSchedule|NoExecute(1.6 only)"
|
|
|
|
## Node labels for Prometheus server pod assignment
|
|
## Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/
|
|
##
|
|
nodeSelector: {}
|
|
|
|
## Pod affinity
|
|
##
|
|
affinity: {}
|
|
|
|
## Pod anti-affinity can prevent the scheduler from placing Prometheus server replicas on the same node.
|
|
## The value "soft" means that the scheduler should *prefer* to not schedule two replica pods onto the same node but no guarantee is provided.
|
|
## The value "hard" means that the scheduler is *required* to not schedule two replica pods onto the same node.
|
|
## The default value "" will disable pod anti-affinity so that no anti-affinity rules will be configured (unless set in `server.affinity`).
|
|
##
|
|
podAntiAffinity: ""
|
|
|
|
## If anti-affinity is enabled sets the topologyKey to use for anti-affinity.
|
|
## This can be changed to, for example, failure-domain.beta.kubernetes.io/zone
|
|
##
|
|
podAntiAffinityTopologyKey: kubernetes.io/hostname
|
|
|
|
## Pod topology spread constraints
|
|
## ref. https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/
|
|
topologySpreadConstraints: []
|
|
|
|
## PodDisruptionBudget settings
|
|
## ref: https://kubernetes.io/docs/concepts/workloads/pods/disruptions/
|
|
##
|
|
podDisruptionBudget:
|
|
enabled: false
|
|
# maxUnavailable: 1
|
|
# minAvailable: 1
|
|
## unhealthyPodEvictionPolicy is available since 1.27.0 (beta)
|
|
## https://kubernetes.io/docs/tasks/run-application/configure-pdb/#unhealthy-pod-eviction-policy
|
|
# unhealthyPodEvictionPolicy: IfHealthyBudget
|
|
|
|
## Use an alternate scheduler, e.g. "stork".
|
|
## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
|
|
##
|
|
# schedulerName:
|
|
|
|
persistentVolume:
|
|
## If true, Prometheus server will create/use a Persistent Volume Claim
|
|
## If false, use emptyDir
|
|
##
|
|
enabled: true
|
|
|
|
## If set it will override the name of the created persistent volume claim
|
|
## generated by the stateful set.
|
|
##
|
|
statefulSetNameOverride: ""
|
|
|
|
## Prometheus server data Persistent Volume access modes
|
|
## Must match those of existing PV or dynamic provisioner
|
|
## Ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
|
|
##
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
|
|
## Prometheus server data Persistent Volume labels
|
|
##
|
|
labels: {}
|
|
|
|
## Prometheus server data Persistent Volume annotations
|
|
##
|
|
annotations: {}
|
|
|
|
## Prometheus server data Persistent Volume existing claim name
|
|
## Requires server.persistentVolume.enabled: true
|
|
## If defined, PVC must be created manually before volume will be bound
|
|
existingClaim: ""
|
|
|
|
## Prometheus server data Persistent Volume mount root path
|
|
##
|
|
mountPath: /data
|
|
|
|
## Prometheus server data Persistent Volume size
|
|
##
|
|
size: 8Gi
|
|
|
|
## Prometheus server data Persistent Volume Storage Class
|
|
## If defined, storageClassName: <storageClass>
|
|
## If set to "-", storageClassName: "", which disables dynamic provisioning
|
|
## If undefined (the default) or set to null, no storageClassName spec is
|
|
## set, choosing the default provisioner. (gp2 on AWS, standard on
|
|
## GKE, AWS & OpenStack)
|
|
##
|
|
# storageClass: "-"
|
|
|
|
## Subdirectory of Prometheus server data Persistent Volume to mount
|
|
## Useful if the volume's root directory is not empty
|
|
##
|
|
subPath: ""
|
|
|
|
## Persistent Volume Claim Selector
|
|
## Useful if Persistent Volumes have been provisioned in advance
|
|
## Ref: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#selector
|
|
##
|
|
# selector:
|
|
# matchLabels:
|
|
# release: "stable"
|
|
# matchExpressions:
|
|
# - { key: environment, operator: In, values: [ dev ] }
|
|
|
|
## Persistent Volume Name
|
|
## Useful if Persistent Volumes have been provisioned in advance and you want to use a specific one
|
|
##
|
|
# volumeName: ""
|
|
|
|
emptyDir:
|
|
## Prometheus server emptyDir volume
|
|
## Configure size limit and medium
|
|
##
|
|
medium: ""
|
|
sizeLimit: ""
|
|
|
|
## Annotations to be added to Prometheus server pods
|
|
##
|
|
podAnnotations: {}
|
|
# iam.amazonaws.com/role: prometheus
|
|
|
|
## Labels to be added to Prometheus server pods
|
|
##
|
|
podLabels: {}
|
|
|
|
## Prometheus AlertManager configuration
|
|
## Lien Prometheus -> Alertmanager (service du sous-chart, ns tools).
|
|
alertmanagers:
|
|
- static_configs:
|
|
- targets:
|
|
- prometheus-alertmanager:9093
|
|
|
|
## Use a StatefulSet if replicaCount needs to be greater than 1 (see below)
|
|
##
|
|
replicaCount: 1
|
|
|
|
## Number of old history to retain to allow rollback
|
|
## Default Kubernetes value is set to 10
|
|
##
|
|
revisionHistoryLimit: 10
|
|
|
|
## Annotations to be added to ConfigMap
|
|
##
|
|
configMapAnnotations: {}
|
|
|
|
## Annotations to be added to deployment
|
|
##
|
|
deploymentAnnotations: {}
|
|
|
|
statefulSet:
|
|
## If true, use a statefulset instead of a deployment for pod management.
|
|
## This allows to scale replicas to more than 1 pod
|
|
##
|
|
enabled: false
|
|
|
|
annotations: {}
|
|
labels: {}
|
|
podManagementPolicy: OrderedReady
|
|
|
|
## Alertmanager headless service to use for the statefulset
|
|
##
|
|
headless:
|
|
annotations: {}
|
|
labels: {}
|
|
servicePort: 80
|
|
## Enable gRPC port on service to allow auto discovery with thanos-querier
|
|
gRPC:
|
|
enabled: false
|
|
servicePort: 10901
|
|
# nodePort: 10901
|
|
|
|
## Statefulset's persistent volume claim retention policy
|
|
## pvcDeleteOnStsDelete and pvcDeleteOnStsScale determine whether
|
|
## statefulset's PVCs are deleted (true) or retained (false) on scaling down
|
|
## and deleting statefulset, respectively. Requires 1.27.0+.
|
|
## Ref: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#persistentvolumeclaim-retention
|
|
##
|
|
pvcDeleteOnStsDelete: false
|
|
pvcDeleteOnStsScale: false
|
|
|
|
daemonSet:
|
|
## If true, use a daemonset instead of a deployment for pod management.
|
|
## This allows to run prometheus agent on every node in the cluster.
|
|
##
|
|
enabled: false
|
|
annotations: {}
|
|
labels: {}
|
|
|
|
## Prometheus server readiness and liveness probe initial delay and timeout
|
|
## Ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
|
|
##
|
|
tcpSocketProbeEnabled: false
|
|
probeScheme: HTTP
|
|
readinessProbeInitialDelay: 30
|
|
readinessProbePeriodSeconds: 5
|
|
readinessProbeTimeout: 4
|
|
readinessProbeFailureThreshold: 3
|
|
readinessProbeSuccessThreshold: 1
|
|
livenessProbeInitialDelay: 30
|
|
livenessProbePeriodSeconds: 15
|
|
livenessProbeTimeout: 10
|
|
livenessProbeFailureThreshold: 3
|
|
livenessProbeSuccessThreshold: 1
|
|
startupProbe:
|
|
enabled: false
|
|
periodSeconds: 5
|
|
failureThreshold: 30
|
|
timeoutSeconds: 10
|
|
|
|
## Prometheus server resource requests and limits
|
|
## Ref: http://kubernetes.io/docs/user-guide/compute-resources/
|
|
##
|
|
resources: {}
|
|
# limits:
|
|
# cpu: 500m
|
|
# memory: 512Mi
|
|
# requests:
|
|
# cpu: 500m
|
|
# memory: 512Mi
|
|
|
|
# Required for use in managed kubernetes clusters (such as AWS EKS) with custom CNI (such as calico),
|
|
# because control-plane managed by AWS cannot communicate with pods' IP CIDR and admission webhooks are not working
|
|
##
|
|
hostNetwork: false
|
|
|
|
# When hostNetwork is enabled, this will set to ClusterFirstWithHostNet automatically
|
|
dnsPolicy: ClusterFirst
|
|
|
|
# Use hostPort
|
|
# hostPort: 9090
|
|
|
|
# Use portName
|
|
portName: ""
|
|
|
|
## Vertical Pod Autoscaler config
|
|
## Ref: https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler
|
|
verticalAutoscaler:
|
|
## If true a VPA object will be created for the controller (either StatefulSet or Deployemnt, based on above configs)
|
|
enabled: false
|
|
# updateMode: "Auto"
|
|
# containerPolicies:
|
|
# - containerName: 'prometheus-server'
|
|
|
|
# Custom DNS configuration to be added to prometheus server pods
|
|
dnsConfig: {}
|
|
# nameservers:
|
|
# - 1.2.3.4
|
|
# searches:
|
|
# - ns1.svc.cluster-domain.example
|
|
# - my.dns.search.suffix
|
|
# options:
|
|
# - name: ndots
|
|
# value: "2"
|
|
# - name: edns0
|
|
|
|
## Security context to be added to server pods
|
|
##
|
|
securityContext:
|
|
runAsUser: 65534
|
|
runAsNonRoot: true
|
|
runAsGroup: 65534
|
|
fsGroup: 65534
|
|
|
|
## Security context to be added to server container
|
|
##
|
|
containerSecurityContext: {}
|
|
|
|
service:
|
|
## If false, no Service will be created for the Prometheus server
|
|
##
|
|
enabled: true
|
|
|
|
annotations: {}
|
|
labels: {}
|
|
clusterIP: ""
|
|
|
|
## List of IP addresses at which the Prometheus server service is available
|
|
## Ref: https://kubernetes.io/docs/concepts/services-networking/service/#external-ips
|
|
##
|
|
externalIPs: []
|
|
|
|
loadBalancerIP: ""
|
|
loadBalancerSourceRanges: []
|
|
loadBalancerClass: ""
|
|
servicePort: 80
|
|
sessionAffinity: None
|
|
type: ClusterIP
|
|
## externalTrafficPolicy is applicable to service with externally-facing addresses (NodePorts, ExternalIPs, and LoadBalancer IPs)
|
|
externalTrafficPolicy: ""
|
|
|
|
## Enable gRPC port on service to allow auto discovery with thanos-querier
|
|
gRPC:
|
|
enabled: false
|
|
servicePort: 10901
|
|
# nodePort: 10901
|
|
|
|
## If using a statefulSet (statefulSet.enabled=true), configure the
|
|
## service to connect to a specific replica to have a consistent view
|
|
## of the data.
|
|
statefulsetReplica:
|
|
enabled: false
|
|
replica: 0
|
|
|
|
## Additional port to define in the Service
|
|
additionalPorts: []
|
|
# additionalPorts:
|
|
# - name: authenticated
|
|
# port: 8081
|
|
# targetPort: 8081
|
|
|
|
## Prometheus server pod termination grace period
|
|
##
|
|
terminationGracePeriodSeconds: 300
|
|
|
|
## Prometheus data retention period (default if not specified is 15 days)
|
|
##
|
|
retention: "15d"
|
|
|
|
## Prometheus' data retention size. Supported units: B, KB, MB, GB, TB, PB, EB.
|
|
##
|
|
retentionSize: ""
|
|
|
|
## scrapeConfigs (map) defines Prometheus' default scrape_configs.
|
|
## Each can be disabled by setting "enabled" to "false" or leaving it empty. The key sets the default "job_name".
|
|
## Further scrapeConfigs can be added as new keys, these are then enabled by default.
|
|
## ref. https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config
|
|
scrapeConfigs:
|
|
prometheus:
|
|
enabled: true
|
|
job_name: ""
|
|
static_configs:
|
|
- targets:
|
|
- localhost:9090
|
|
# Le worker d'analyse Kadans sur le LAPTOP du fondateur (tier 1 VOLATILE,
|
|
# kadans-jobs cmd/worker, /metrics stdlib sur :9105). Un scrape en échec
|
|
# n'est PAS un incident : le Mac dort — `up == 0` raconte exactement ça
|
|
# (latence des analyses, jamais une indisponibilité produit). Ne PAS
|
|
# alerter sur cette cible.
|
|
# ⚠ IP DHCP : réserver 192.168.1.103 pour le Mac au routeur (ou remplacer
|
|
# par un nom résolvable du LAN) — sinon la cible dérive au renouvellement.
|
|
kadans-worker-mac:
|
|
enabled: true
|
|
job_name: "kadans-worker-mac"
|
|
static_configs:
|
|
- targets:
|
|
- 192.168.1.103:9105
|
|
labels:
|
|
tier: laptop
|
|
app: kadans-worker
|
|
kubernetes-api-servers:
|
|
enabled: true
|
|
job_name: ""
|
|
kubernetes_sd_configs:
|
|
- role: endpoints
|
|
scheme: https
|
|
tls_config:
|
|
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
|
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
|
|
relabel_configs:
|
|
- source_labels:
|
|
- __meta_kubernetes_namespace
|
|
- __meta_kubernetes_service_name
|
|
- __meta_kubernetes_endpoint_port_name
|
|
action: keep
|
|
regex: default;kubernetes;https
|
|
kubernetes-nodes:
|
|
enabled: true
|
|
job_name: ""
|
|
scheme: https
|
|
tls_config:
|
|
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
|
insecure_skip_verify: true
|
|
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
|
|
kubernetes_sd_configs:
|
|
- role: node
|
|
relabel_configs:
|
|
- action: labelmap
|
|
regex: __meta_kubernetes_node_label_(.+)
|
|
kubernetes-nodes-cadvisor:
|
|
enabled: true
|
|
job_name: ""
|
|
scheme: https
|
|
tls_config:
|
|
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
|
insecure_skip_verify: true
|
|
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
|
|
metrics_path: /metrics/cadvisor
|
|
kubernetes_sd_configs:
|
|
- role: node
|
|
relabel_configs:
|
|
- action: labelmap
|
|
regex: __meta_kubernetes_node_label_(.+)
|
|
- source_labels: [__metrics_path__]
|
|
target_label: metrics_path
|
|
kubernetes-service-endpoints:
|
|
enabled: true
|
|
job_name: ""
|
|
honor_labels: true
|
|
kubernetes_sd_configs:
|
|
- role: endpoints
|
|
relabel_configs:
|
|
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
|
|
action: keep
|
|
regex: true
|
|
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape_slow]
|
|
action: drop
|
|
regex: true
|
|
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
|
|
action: replace
|
|
target_label: __scheme__
|
|
regex: (https?)
|
|
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
|
|
action: replace
|
|
target_label: __metrics_path__
|
|
regex: (.+)
|
|
- source_labels:
|
|
- __address__
|
|
- __meta_kubernetes_service_annotation_prometheus_io_port
|
|
action: replace
|
|
target_label: __address__
|
|
regex: (.+?)(?::\d+)?;(\d+)
|
|
replacement: $1:$2
|
|
- action: labelmap
|
|
regex: __meta_kubernetes_service_annotation_prometheus_io_param_(.+)
|
|
replacement: __param_$1
|
|
- action: labelmap
|
|
regex: __meta_kubernetes_service_label_(.+)
|
|
- source_labels: [__meta_kubernetes_namespace]
|
|
action: replace
|
|
target_label: namespace
|
|
- source_labels: [__meta_kubernetes_service_name]
|
|
action: replace
|
|
target_label: service
|
|
- source_labels: [__meta_kubernetes_pod_node_name]
|
|
action: replace
|
|
target_label: node
|
|
kubernetes-service-endpoints-slow:
|
|
enabled: true
|
|
job_name: ""
|
|
honor_labels: true
|
|
scrape_interval: 5m
|
|
scrape_timeout: 30s
|
|
kubernetes_sd_configs:
|
|
- role: endpoints
|
|
relabel_configs:
|
|
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape_slow]
|
|
action: keep
|
|
regex: true
|
|
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
|
|
action: replace
|
|
target_label: __scheme__
|
|
regex: (https?)
|
|
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
|
|
action: replace
|
|
target_label: __metrics_path__
|
|
regex: (.+)
|
|
- source_labels:
|
|
- __address__
|
|
- __meta_kubernetes_service_annotation_prometheus_io_port
|
|
action: replace
|
|
target_label: __address__
|
|
regex: (.+?)(?::\d+)?;(\d+)
|
|
replacement: $1:$2
|
|
- action: labelmap
|
|
regex: __meta_kubernetes_service_annotation_prometheus_io_param_(.+)
|
|
replacement: __param_$1
|
|
- action: labelmap
|
|
regex: __meta_kubernetes_service_label_(.+)
|
|
- source_labels: [__meta_kubernetes_namespace]
|
|
action: replace
|
|
target_label: namespace
|
|
- source_labels: [__meta_kubernetes_service_name]
|
|
action: replace
|
|
target_label: service
|
|
- source_labels: [__meta_kubernetes_pod_node_name]
|
|
action: replace
|
|
target_label: node
|
|
prometheus-pushgateway:
|
|
enabled: true
|
|
job_name: ""
|
|
honor_labels: true
|
|
kubernetes_sd_configs:
|
|
- role: service
|
|
relabel_configs:
|
|
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
|
|
action: keep
|
|
regex: pushgateway
|
|
kubernetes-services:
|
|
enabled: true
|
|
job_name: ""
|
|
honor_labels: true
|
|
metrics_path: /probe
|
|
params:
|
|
module: [http_2xx]
|
|
kubernetes_sd_configs:
|
|
- role: service
|
|
relabel_configs:
|
|
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
|
|
action: keep
|
|
regex: true
|
|
- source_labels: [__address__]
|
|
target_label: __param_target
|
|
- target_label: __address__
|
|
replacement: blackbox
|
|
- source_labels: [__param_target]
|
|
target_label: instance
|
|
- action: labelmap
|
|
regex: __meta_kubernetes_service_label_(.+)
|
|
- source_labels: [__meta_kubernetes_namespace]
|
|
target_label: namespace
|
|
- source_labels: [__meta_kubernetes_service_name]
|
|
target_label: service
|
|
kubernetes-pods:
|
|
enabled: true
|
|
job_name: ""
|
|
honor_labels: true
|
|
kubernetes_sd_configs:
|
|
- role: pod
|
|
relabel_configs:
|
|
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
|
|
action: keep
|
|
regex: true
|
|
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape_slow]
|
|
action: drop
|
|
regex: true
|
|
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
|
|
action: replace
|
|
regex: (https?)
|
|
target_label: __scheme__
|
|
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
|
|
action: replace
|
|
target_label: __metrics_path__
|
|
regex: (.+)
|
|
- source_labels:
|
|
- __meta_kubernetes_pod_annotation_prometheus_io_port
|
|
- __meta_kubernetes_pod_ip
|
|
action: replace
|
|
regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
|
|
replacement: '[$2]:$1'
|
|
target_label: __address__
|
|
- source_labels:
|
|
- __meta_kubernetes_pod_annotation_prometheus_io_port
|
|
- __meta_kubernetes_pod_ip
|
|
action: replace
|
|
regex: (\d+);((([0-9]+?)(\.|$)){4})
|
|
replacement: $2:$1
|
|
target_label: __address__
|
|
- action: labelmap
|
|
regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)
|
|
replacement: __param_$1
|
|
- action: labelmap
|
|
regex: __meta_kubernetes_pod_label_(.+)
|
|
- source_labels: [__meta_kubernetes_namespace]
|
|
action: replace
|
|
target_label: namespace
|
|
- source_labels: [__meta_kubernetes_pod_name]
|
|
action: replace
|
|
target_label: pod
|
|
- source_labels: [__meta_kubernetes_pod_phase]
|
|
regex: Pending|Succeeded|Failed|Completed
|
|
action: drop
|
|
- source_labels: [__meta_kubernetes_pod_node_name]
|
|
action: replace
|
|
target_label: node
|
|
kubernetes-pods-slow:
|
|
enabled: true
|
|
job_name: ""
|
|
honor_labels: true
|
|
scrape_interval: 5m
|
|
scrape_timeout: 30s
|
|
kubernetes_sd_configs:
|
|
- role: pod
|
|
relabel_configs:
|
|
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape_slow]
|
|
action: keep
|
|
regex: true
|
|
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
|
|
action: replace
|
|
regex: (https?)
|
|
target_label: __scheme__
|
|
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
|
|
action: replace
|
|
target_label: __metrics_path__
|
|
regex: (.+)
|
|
- source_labels:
|
|
- __meta_kubernetes_pod_annotation_prometheus_io_port
|
|
- __meta_kubernetes_pod_ip
|
|
action: replace
|
|
regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
|
|
replacement: '[$2]:$1'
|
|
target_label: __address__
|
|
- source_labels:
|
|
- __meta_kubernetes_pod_annotation_prometheus_io_port
|
|
- __meta_kubernetes_pod_ip
|
|
action: replace
|
|
regex: (\d+);((([0-9]+?)(\.|$)){4})
|
|
replacement: $2:$1
|
|
target_label: __address__
|
|
- action: labelmap
|
|
regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)
|
|
replacement: __param_$1
|
|
- action: labelmap
|
|
regex: __meta_kubernetes_pod_label_(.+)
|
|
- source_labels: [__meta_kubernetes_namespace]
|
|
action: replace
|
|
target_label: namespace
|
|
- source_labels: [__meta_kubernetes_pod_name]
|
|
action: replace
|
|
target_label: pod
|
|
- source_labels: [__meta_kubernetes_pod_phase]
|
|
regex: Pending|Succeeded|Failed|Completed
|
|
action: drop
|
|
- source_labels: [__meta_kubernetes_pod_node_name]
|
|
action: replace
|
|
target_label: node
|
|
|
|
# extraScrapeConfigs adds additional scrape configs to prometheus.yml
|
|
# must be a string so you have to add a | after extraScrapeConfigs:
|
|
# example adds prometheus-blackbox-exporter scrape config
|
|
extraScrapeConfigs: ""
|
|
# - job_name: 'prometheus-blackbox-exporter'
|
|
# metrics_path: /probe
|
|
# params:
|
|
# module: [http_2xx]
|
|
# static_configs:
|
|
# - targets:
|
|
# - https://example.com
|
|
# relabel_configs:
|
|
# - source_labels: [__address__]
|
|
# target_label: __param_target
|
|
# - source_labels: [__param_target]
|
|
# target_label: instance
|
|
# - target_label: __address__
|
|
# replacement: prometheus-blackbox-exporter:9115
|
|
|
|
## Prometheus server ConfigMap entries for rule files (allow prometheus labels interpolation)
|
|
ruleFiles: {}
|
|
|
|
## Prometheus server ConfigMap entries for scrape_config_files
|
|
## (allows scrape configs defined in additional files)
|
|
##
|
|
scrapeConfigFiles: []
|
|
|
|
## Prometheus server ConfigMap entries
|
|
##
|
|
serverFiles:
|
|
## Alerts configuration
|
|
## Ref: https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/
|
|
alerting_rules.yml:
|
|
groups:
|
|
# Pipeline prospection (métriques poussées au Pushgateway job=prospection en fin de
|
|
# run). Livraison : Alertmanager → receiver telegram (chaîne testée live 2026-07-10,
|
|
# firing + resolved reçus). Visibles aussi dans Prometheus /alerts + le dashboard
|
|
# Grafana « Prospection » (panneau Alertes actives).
|
|
- name: prospection
|
|
rules:
|
|
- alert: ProspectionRunStale
|
|
expr: time() - prospection_run_timestamp_seconds > 90000 # > 25 h (cron quotidien)
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
app: prospection
|
|
annotations:
|
|
summary: "Prospection — aucun run réussi depuis plus de 25 h"
|
|
description: "Dernier run réussi il y a {{ $value | humanizeDuration }} ; le CronJob quotidien (~06:30 UTC) n'a pas abouti."
|
|
- alert: ProspectionRunFailed
|
|
expr: prospection_run_success == 0
|
|
for: 5m
|
|
labels:
|
|
severity: warning
|
|
app: prospection
|
|
annotations:
|
|
summary: "Prospection — le dernier run a échoué"
|
|
description: "prospection_run_success=0 : toutes les collectes ont échoué au dernier run."
|
|
- alert: ProspectionStepError
|
|
expr: prospection_step_status == 0
|
|
for: 5m
|
|
labels:
|
|
severity: info
|
|
app: prospection
|
|
annotations:
|
|
summary: "Prospection — étape {{ $labels.step }} en erreur"
|
|
description: "L'étape {{ $labels.step }} du pipeline a fini en erreur au dernier run."
|
|
- alert: ProspectionNoOffers
|
|
expr: prospection_offers_total == 0
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
app: prospection
|
|
annotations:
|
|
summary: "Prospection — 0 offre (mission) collectée"
|
|
description: "Aucune offre au dernier run : collecte France Travail / Free-Work potentiellement cassée."
|
|
# Vidéo quotidienne : deux modes d'échec distincts, tous deux gardés contre le
|
|
# skip volontaire — metrics.py met brief_rendered=0 aussi quand l'étape est
|
|
# SAUTÉE (cadence/kill-switch), seul step_status distingue skip(2) d'erreur(0).
|
|
- alert: ProspectionBriefFailed
|
|
# PRODUCTION en échec : l'étape brief a tourné mais n'a pas rendu la vidéo.
|
|
expr: prospection_brief_rendered == 0 and on(job) prospection_step_status{step="brief"} != 2
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
app: prospection
|
|
annotations:
|
|
summary: "Prospection — la vidéo du brief quotidien n'a pas été produite"
|
|
description: "L'étape brief a échoué au dernier run (rendu vidéo KO : TTS/ffmpeg/PIL ou erreur amont). Voir les logs du CronJob prospection."
|
|
- alert: ProspectionBriefNotSent
|
|
# LIVRAISON en échec : vidéo produite mais pas poussée sur Telegram.
|
|
expr: prospection_brief_telegram_pushed == 0 and prospection_brief_rendered == 1
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
app: prospection
|
|
annotations:
|
|
summary: "Prospection — brief produit mais non poussé sur Telegram"
|
|
description: "La vidéo du brief a été rendue mais l'envoi Telegram a échoué (token/chat_id/API). Voir les logs du CronJob prospection."
|
|
# Santé du socle : être prévenu quand (ou juste avant que) le homelab tombe.
|
|
# Incident 2026-07-23 : build CI sans limites sur pi1 → RAM épuisée (0 swap),
|
|
# load15 >100, traefik + apiserver k3s affamés → tout *.arcodange.lab injoignable
|
|
# (Gitea compris, pourtant sain sur pi2). Prometheus vit sur pi3 et Alertmanager
|
|
# sur pi2 : cette chaîne d'alerte survit donc à la perte de pi1.
|
|
- name: homelab
|
|
rules:
|
|
- alert: NoeudInjoignable
|
|
# node-exporter ne répond plus : nœud éteint, réseau HS, ou surcharge telle
|
|
# que plus rien n'y répond (le cas de l'incident).
|
|
expr: up{job="kubernetes-service-endpoints", app_kubernetes_io_name="prometheus-node-exporter"} == 0
|
|
for: 3m
|
|
labels:
|
|
severity: critical
|
|
app: homelab
|
|
annotations:
|
|
summary: "Homelab — {{ $labels.node }} est injoignable"
|
|
description: "node-exporter de {{ $labels.node }} ({{ $labels.instance }}) ne répond plus depuis 3 min : nœud down ou en surcharge sévère."
|
|
- alert: IngressLabIndisponible
|
|
# Plus aucun replica traefik dispo — ou kube-state-metrics muet (il vit sur
|
|
# pi1 : quand pi1 tombe, la métrique disparaît au lieu de passer à 0).
|
|
expr: >-
|
|
kube_deployment_status_replicas_available{namespace="kube-system",deployment="traefik"} < 1
|
|
or absent(kube_deployment_status_replicas_available{namespace="kube-system",deployment="traefik"})
|
|
for: 3m
|
|
labels:
|
|
severity: critical
|
|
app: homelab
|
|
annotations:
|
|
summary: "Homelab — ingress traefik indisponible : *.arcodange.lab est HS"
|
|
description: "Aucun replica traefik disponible (ou métrique absente = kube-state-metrics muet). Gitea, ArgoCD, Grafana, Vault… sont injoignables via leurs URLs .lab. Gitea reste accessible en direct : http://192.168.1.202:3000."
|
|
- alert: NoeudPressionMemoire
|
|
# Précurseur direct de l'incident : <500 Mo dispo sur un Pi 8 Go — sans
|
|
# swap, le kernel part en thrash bien avant d'atteindre 0.
|
|
expr: node_memory_MemAvailable_bytes < 500 * 1024 * 1024
|
|
for: 5m
|
|
labels:
|
|
severity: warning
|
|
app: homelab
|
|
annotations:
|
|
summary: "Homelab — mémoire critique sur {{ $labels.node }}"
|
|
description: "{{ $labels.node }} n'a plus que {{ $value | humanize1024 }}B de mémoire disponible depuis 5 min : risque imminent de thrash (pas de swap sur les Pis). Suspect n°1 : un job CI trop gourmand."
|
|
- alert: NoeudEnSurcharge
|
|
expr: node_load15 > 8
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
app: homelab
|
|
annotations:
|
|
summary: "Homelab — {{ $labels.node }} en surcharge (load15 = {{ $value | humanize }})"
|
|
description: "load15 > 8 depuis 10 min sur {{ $labels.node }} (4 cœurs) : quelque chose sature la machine, l'ingress et l'API k3s sont en danger si c'est pi1."
|
|
- alert: CertificatNonRenouvele
|
|
# Les certs .lab durent 24 h et cert-manager renouvelle à ~16 h d'âge : s'il
|
|
# reste <4 h, le renouvellement a raté plusieurs fois (incident 2026-07-24 :
|
|
# step-issuer ne résolvait plus ssl-ca.arcodange.lab, wildcard expiré au matin
|
|
# → « le https ne fonctionne plus » sur tout *.arcodange.lab).
|
|
expr: certmanager_certificate_expiration_timestamp_seconds - time() < 4 * 3600
|
|
for: 15m
|
|
labels:
|
|
severity: critical
|
|
app: homelab
|
|
annotations:
|
|
summary: "Homelab — cert {{ $labels.namespace }}/{{ $labels.name }} expire dans {{ $value | humanizeDuration }}"
|
|
description: "Le renouvellement automatique (cert-manager → step-issuer → step-ca) est en échec. Vérifier : kubectl get certificaterequest -A, logs step-issuer (résolution DNS de ssl-ca.arcodange.lab), santé de step-ca sur pi1:8443."
|
|
# ---- VAULT SCELLÉ ------------------------------------------------------
|
|
# Incident 2026-08-30 → 2026-09-10 : Vault est resté SCELLÉ ONZE JOURS.
|
|
# Le VSO ne pouvait plus s'authentifier (503 « Vault is sealed »), donc les
|
|
# Secrets de plusieurs applications ont cessé d'être réconciliés — en silence.
|
|
# Découvert par hasard, en enquêtant sur tout autre chose.
|
|
#
|
|
# CE QUI A MARCHÉ CE JOUR-LÀ : `NoeudInjoignable` a bien tiré à 11:46 quand pi3
|
|
# a cessé de répondre. L'infra a parlé. CE QUI A MANQUÉ : la CONSÉQUENCE. Le
|
|
# nœud est revenu, les pods ont été recréés, l'alerte s'est éteinte — et
|
|
# personne n'a su que Vault, lui, était reparti scellé et le resterait.
|
|
#
|
|
# POURQUOI `kube_pod_status_ready` ET PAS `vault_core_unsealed` : Vault n'expose
|
|
# ses métriques que via /v1/sys/metrics, qui exige soit un token, soit
|
|
# `unauthenticated_metrics_access = true` dans la stanza telemetry. Le second
|
|
# ouvre un endpoint non authentifié pour gagner… le même signal. Or la
|
|
# readinessProbe du chart est littéralement `vault status` : un Vault scellé est
|
|
# NotReady, point. kube-state-metrics est déjà scrapé, ça ne coûte rien, et ça
|
|
# ne touche pas à la configuration de Vault.
|
|
#
|
|
# ⚠ Cette alerte ne distingue pas « scellé » de « en train de démarrer » ou
|
|
# « planté ». C'est VOULU : dans les trois cas Vault ne sert plus de secrets, et
|
|
# le geste de l'opérateur commence pareil — aller voir `vault status`.
|
|
- alert: VaultIndisponible
|
|
# ⚠ LE SÉLECTEUR EST ANCRÉ SUR `[0-9]+`, ET CE N'EST PAS COSMÉTIQUE : un
|
|
# `hashicorp-vault-.*` attrape aussi les pods du Vault Secrets Operator
|
|
# (`hashicorp-vault-vault-secrets-operator-controller-manager-xxxxx`), dont un
|
|
# exemplaire terminé traîne en permanence — la règle aurait donc tiré sans
|
|
# arrêt, et une alerte qui crie tout le temps ne se lit plus. Vérifié contre le
|
|
# Prometheus du cluster : `[0-9]+` ne rend que hashicorp-vault-0. Toute
|
|
# évolution de ce sélecteur se re-teste de la même façon, sur le vrai serveur.
|
|
#
|
|
# `absent()` : si le pod disparaît (évincé, nœud perdu), la série n'existe plus
|
|
# et une simple comparaison `== 0` serait muette — exactement le mode de panne
|
|
# qu'on essaie de couvrir. Même motif que IngressLabIndisponible ci-dessus.
|
|
expr: >-
|
|
kube_pod_status_ready{namespace="tools", pod=~"hashicorp-vault-[0-9]+", condition="true"} == 0
|
|
or absent(kube_pod_status_ready{namespace="tools", pod=~"hashicorp-vault-[0-9]+", condition="true"})
|
|
# 10 min : large devant un redémarrage normal (le pod est Ready en ~30 s),
|
|
# court devant onze jours.
|
|
for: 10m
|
|
labels:
|
|
severity: critical
|
|
app: homelab
|
|
annotations:
|
|
summary: "Homelab — Vault ne sert plus de secrets (probablement scellé)"
|
|
description: >-
|
|
hashicorp-vault n'est plus Ready depuis 10 min. Le cas le plus fréquent est
|
|
un Vault SCELLÉ : il repart toujours scellé après recréation du pod (Shamir
|
|
1/1, pas d'auto-unseal — c'est une décision assumée, cf. factory
|
|
vibe/guidebooks/lab-ecosystem/secrets-and-vault.md). Tant qu'il l'est, le
|
|
Vault Secrets Operator ne réconcilie plus AUCUN Secret : les applications
|
|
tournent sur leurs Secrets existants et cassent dès qu'un pod redémarre.
|
|
Vérifier - kubectl -n tools exec hashicorp-vault-0 -- vault status.
|
|
Desceller - le rôle ansible arcodange.factory/hashicorp_vault, tâche
|
|
unseal.yml (clé dans ~/.arcodange/cluster-keys.json sur le poste).
|
|
# Le veilleur (arcodange-org/tools#36). Du 2026-09-04 02:41 au
|
|
# 2026-09-07 07:37, Prometheus n'a rien enregistré et les 11 règles
|
|
# ci-dessus se sont TUES — sans échantillon frais, une règle ne se
|
|
# déclenche pas. Sur Telegram, ce silence est indiscernable du bon
|
|
# fonctionnement.
|
|
#
|
|
# ⚠⚠ CES DEUX RÈGLES NE SONT PAS LE REMÈDE : elles vivent dans le
|
|
# Prometheus surveillé, donc elles sont muettes exactement quand il
|
|
# faudrait qu'elles parlent. Le remède est le CronJob
|
|
# `veilleur-prometheus` (templates/veilleur.yaml), qui interroge
|
|
# Prometheus depuis l'EXTÉRIEUR et pousse lui-même sur Telegram.
|
|
# Ces deux règles-ci sont ce que le CronJob observe et ce qui le
|
|
# complète — rien de plus.
|
|
- name: veilleur
|
|
rules:
|
|
- alert: Veilleur
|
|
# Témoin toujours allumé (dead man's switch). Il ne dit rien de
|
|
# l'état du système : il est TOUJOURS vrai. C'est son ABSENCE
|
|
# dans Alertmanager qui est le signal, et c'est le CronJob
|
|
# `veilleur-prometheus` qui la constate depuis dehors.
|
|
#
|
|
# ⚠ `vector(1)` ne lit AUCUNE série : ce témoin aurait continué
|
|
# de tirer pendant les 3 j 5 h de panne. Il prouve que la chaîne
|
|
# évaluation → Alertmanager vit, jamais que le TSDB enregistre.
|
|
# C'est le contrôle de fraîcheur du CronJob qui garde ça.
|
|
#
|
|
# Il ne part PAS sur Telegram : la route `veilleur` d'Alertmanager
|
|
# l'envoie au récepteur vide (cf. alertmanager.config plus bas).
|
|
# Un témoin qui sonne toutes les 3 h serait un témoin qu'on coupe.
|
|
expr: vector(1)
|
|
labels:
|
|
severity: none
|
|
app: veilleur
|
|
annotations:
|
|
summary: "Témoin toujours allumé — c'est son absence qui est le signal"
|
|
description: "Si cette alerte disparaît d'Alertmanager, l'évaluation des règles de Prometheus ou sa livraison vers Alertmanager est arrêtée : AUCUNE alerte ne partirait plus."
|
|
- alert: VeilleurMuet
|
|
# L'auto-plainte du veilleur, deuxième moitié : il pousse un
|
|
# battement au Pushgateway à chaque exécution menée à son terme.
|
|
# 1200 s = 4 tours de CronJob (*/5).
|
|
#
|
|
# ⚠ Cette règle est DANS le Prometheus surveillé : elle ne peut
|
|
# rien dire d'un Prometheus mort — ce n'est pas son travail, c'est
|
|
# celui du CronJob. Elle couvre le cas inverse : Prometheus en
|
|
# pleine forme et le veilleur, lui, à l'arrêt (CronJob suspendu,
|
|
# image introuvable, Telegram injoignable). Le cas « les deux à la
|
|
# fois » reste NON COUVERT : il faudrait un second canal,
|
|
# indépendant de ce homelab.
|
|
expr: >-
|
|
time() - veilleur_prometheus_derniere_reussite_timestamp_seconds > 1200
|
|
or absent(veilleur_prometheus_derniere_reussite_timestamp_seconds)
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
app: veilleur
|
|
annotations:
|
|
summary: "Veilleur — le guetteur extérieur de Prometheus ne rapporte plus"
|
|
description: "Le CronJob veilleur-prometheus (ns tools) n'a pas mené une exécution à son terme depuis plus de 20 min. Tant qu'il est muet, une panne d'enregistrement de Prometheus redeviendrait invisible. Vérifier : kubectl -n tools get cronjob,jobs -l app.kubernetes.io/name=veilleur-prometheus puis les journaux du dernier Job."
|
|
# groups:
|
|
# - name: Instances
|
|
# rules:
|
|
# - alert: InstanceDown
|
|
# expr: up == 0
|
|
# for: 5m
|
|
# labels:
|
|
# severity: page
|
|
# annotations:
|
|
# description: '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.'
|
|
# summary: 'Instance {{ $labels.instance }} down'
|
|
## DEPRECATED DEFAULT VALUE, unless explicitly naming your files, please use alerting_rules.yml
|
|
alerts: {}
|
|
|
|
## Records configuration
|
|
## Ref: https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/
|
|
recording_rules.yml: {}
|
|
## DEPRECATED DEFAULT VALUE, unless explicitly naming your files, please use recording_rules.yml
|
|
rules: {}
|
|
|
|
prometheus.yml:
|
|
rule_files:
|
|
- /etc/config/recording_rules.yml
|
|
- /etc/config/alerting_rules.yml
|
|
## Below two files are DEPRECATED will be removed from this default values file
|
|
- /etc/config/rules
|
|
- /etc/config/alerts
|
|
|
|
# Adds option to add alert_relabel_configs to avoid duplicate alerts in alertmanager
|
|
# useful in H/A prometheus with different external labels but the same alerts
|
|
alertRelabelConfigs: {}
|
|
# alert_relabel_configs:
|
|
# - source_labels: [dc]
|
|
# regex: (.+)\d+
|
|
# target_label: dc
|
|
|
|
networkPolicy:
|
|
## Enable creation of NetworkPolicy resources.
|
|
##
|
|
enabled: false
|
|
|
|
# Force namespace of namespaced resources
|
|
forceNamespace: ""
|
|
|
|
# Extra manifests to deploy as an array
|
|
extraManifests: []
|
|
# - |
|
|
# apiVersion: v1
|
|
# kind: ConfigMap
|
|
# metadata:
|
|
# labels:
|
|
# name: prometheus-extra
|
|
# data:
|
|
# extra-data: "value"
|
|
|
|
# Configuration of subcharts defined in Chart.yaml
|
|
|
|
## alertmanager sub-chart configurable values
|
|
## Please see https://github.com/prometheus-community/helm-charts/tree/main/charts/alertmanager
|
|
##
|
|
alertmanager:
|
|
## If false, alertmanager will not be installed
|
|
##
|
|
enabled: true
|
|
|
|
## Configuration Alertmanager : livraison native Telegram (bot prospection).
|
|
## Le token est lu depuis le fichier monté via extraSecretMounts (Secret
|
|
## alertmanager-telegram, synchronisé par VSO — cf. templates/vault-telegram.yaml).
|
|
## chat_id est public (non sensible), donc inline.
|
|
config:
|
|
enabled: true
|
|
global: {}
|
|
templates:
|
|
- /etc/alertmanager/*.tmpl
|
|
route:
|
|
group_by: ["alertname", "app"]
|
|
group_wait: 30s
|
|
group_interval: 5m
|
|
repeat_interval: 3h
|
|
receiver: telegram
|
|
routes:
|
|
# Le témoin toujours allumé ne dérange personne : il est là pour être
|
|
# CONSTATÉ par le CronJob veilleur-prometheus via /api/v2/alerts, pas
|
|
# pour être lu. Sans cette route il sonnerait toutes les 3 h, et un
|
|
# témoin qui sonne est un témoin qu'on finit par couper.
|
|
- matchers:
|
|
- alertname="Veilleur"
|
|
receiver: neant
|
|
group_wait: 0s
|
|
repeat_interval: 8760h
|
|
receivers:
|
|
# Récepteur vide = trou noir. C'est la forme prévue par Alertmanager.
|
|
- name: neant
|
|
- name: telegram
|
|
telegram_configs:
|
|
- bot_token_file: /etc/alertmanager/telegram/BOT_TOKEN
|
|
chat_id: 7497777082
|
|
parse_mode: HTML
|
|
send_resolved: true
|
|
|
|
## Montage du token du bot dans le pod Alertmanager (fichier BOT_TOKEN).
|
|
extraSecretMounts:
|
|
- name: telegram
|
|
mountPath: /etc/alertmanager/telegram
|
|
secretName: alertmanager-telegram
|
|
readOnly: true
|
|
|
|
persistence:
|
|
## If true, storage will create or use Persistence Volume
|
|
## If false, storage will use emptyDir
|
|
##
|
|
enabled: true
|
|
|
|
## Custom annotations for the PVC created by the alertmanager StatefulSet.
|
|
## Useful for configuring storage provider options such as disk type, KMS encryption keys, or custom volume name prefixes.
|
|
annotations: {}
|
|
|
|
## Custom labels for the PVC created by the alertmanager StatefulSet.
|
|
## Useful for selecting, grouping, and organizing so that they can be queried or targeted in deployments, policies, etc.
|
|
labels: {}
|
|
|
|
## Persistent Volume Storage Class
|
|
## If defined, storageClassName: <storageClass>
|
|
## If set to "-", storageClassName: "", which disables dynamic provisioning
|
|
## If undefined (the default) or set to null, no storageClassName spec is
|
|
## set, choosing the default provisioner.
|
|
##
|
|
# storageClass: "-"
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
size: 2Gi
|
|
|
|
## Configure emptyDir volume
|
|
##
|
|
emptyDir: {}
|
|
|
|
podSecurityContext:
|
|
runAsUser: 65534
|
|
runAsNonRoot: true
|
|
runAsGroup: 65534
|
|
fsGroup: 65534
|
|
|
|
## kube-state-metrics sub-chart configurable values
|
|
## Please see https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-state-metrics
|
|
##
|
|
kube-state-metrics:
|
|
## If false, kube-state-metrics sub-chart will not be installed
|
|
##
|
|
enabled: true
|
|
|
|
## prometheus-node-exporter sub-chart configurable values
|
|
## Please see https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-node-exporter
|
|
##
|
|
prometheus-node-exporter:
|
|
## If false, node-exporter will not be installed
|
|
##
|
|
enabled: true
|
|
|
|
rbac:
|
|
pspEnabled: false
|
|
|
|
containerSecurityContext:
|
|
allowPrivilegeEscalation: false
|
|
|
|
## prometheus-pushgateway sub-chart configurable values
|
|
## Please see https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-pushgateway
|
|
##
|
|
prometheus-pushgateway:
|
|
## If false, pushgateway will not be installed
|
|
##
|
|
enabled: true
|
|
|
|
# Optional service annotations
|
|
serviceAnnotations:
|
|
prometheus.io/probe: pushgateway
|
|
|
|
|
|
tool:
|
|
# kind: 'SubChart' or 'HelmChart', if subchart then uncomment Chart.yaml dependency, else comment and use tool library with helm chart template
|
|
kind: 'SubChart'
|
|
repo: https://prometheus-community.github.io/helm-charts
|
|
chart: prometheus
|
|
version: 28.13.0
|
|
values: *prometheus_config |