Rules

guix

6.022s ago

2.063ms

Rule State Error Last Evaluation Evaluation Time
alert: GuixTargetMissing expr: up{job=~"bordeaux.guix.gnu.org-nar-herders|coordinator.bayfront.guix.gnu.org|data.qa.guix.gnu.org-guix-data-service|data.guix.gnu.org-guix-data-service|monokuma.cbaines.net/node-exporter|hatysa.cbaines.net/node-exporter|hamal.cbaines.net/node-exporter|milano-guix-1/node-exporter"} == 0 for: 10m labels: severity: critical annotations: description: |- A Prometheus target has disappeared. An exporter might be crashed. VALUE = {{ $value }} LABELS = {{ $labels }} summary: Prometheus target missing (instance {{ $labels.instance }}) ok 6.025s ago 941.8us
alert: MirrorMissingNars expr: narherder_nar_files_total{instance=~"hydra-guix-129.guix.gnu.org:443|hatysa.cbaines.net:443",stored="false"} > 250 for: 10m labels: severity: critical annotations: description: Mirror missing {{ $value }} nars summary: Mirror missing nars (instance {{ $labels.instance }}) ok 6.024s ago 305.8us
alert: TooManyNarsOnBordeaux expr: narherder_nar_files_total{instance="bordeaux.guix.gnu.org:443",stored="true"} > 250 for: 10m labels: severity: critical annotations: description: Bordeaux has {{ $value }} nars summary: Too many nars on bordeaux ok 6.025s ago 124.6us
alert: BuildCoordinatorHookEventsNotBeingProcessed expr: rate(guixbuildcoordinator_hook_duration_seconds_count[30m]) < 0.001 and rate(guixbuildcoordinator_unprocessed_hook_events_total[30m]) > 0.001 for: 10m labels: severity: critical annotations: description: '{{ $labels.event }} events are not being processed' summary: Build coordinator events not being processed ok 6.026s ago 673us

node

10.697s ago

4.14ms

Rule State Error Last Evaluation Evaluation Time
alert: HostOutOfDiskSpace expr: (node_filesystem_avail_bytes{instance!="capella.cbaines.net:9100"} * 100) / node_filesystem_size_bytes < 10 for: 5m labels: severity: warning annotations: description: |- Disk is almost full (< 10% left) VALUE = {{ $value }} LABELS: {{ $labels }} summary: Host out of disk space (instance {{ $labels.instance }}) ok 10.698s ago 4.107ms