chore(observability): NoVirtHandlerPodsRunning (#1263)

Add prometheus alert if no virt-handler during 10m

Signed-off-by: Nikita Korolev <[email protected]>

---------

Signed-off-by: Nikita Korolev <[email protected]>
Co-authored-by: Ivan Mikheykin <[email protected]>

Author: universal-itengineer

monitoring/prometheus-rules/internal-virtualization-virt-hander.yaml

@@ -19,7 +19,7 @@
           2. Or check the Pod logs: `kubectl -n d8-virtualization logs daemonsets/virt-handler`
 
     - alert: D8InternalVirtualizationVirtHandlerTargetAbsent
-      expr: absent(up{job="internal-virtualization-kubevirt-prometheus-metrics") == 1
+      expr: absent(up{job="internal-virtualization-kubevirt-prometheus-metrics"}) == 1
       labels:
         severity_level: "6"
         tier: cluster
@@ -70,3 +70,19 @@
           The recommended course of action:
           1. Retrieve details of the Deployment: `kubectl -n d8-virtualization describe daemonset virt-handler`
           2. View the status of the Pod and try to figure out why it is not running: `kubectl -n d8-virtualization describe pod --field-selector=spec.nodeName={{ $labels.node }} -l kubevirt.internal.virtualization.deckhouse.io=virt-handler`
+
+    - alert: D8InternalVirtualizationVirtHandlerPodsAbsent
+      expr: absent(kube_pod_info{namespace="d8-virtualization", pod=~"virt-handler-.+"})
+      for: 10m
+      labels:
+        severity_level: "6"
+        tier: cluster
+      annotations:
+        plk_protocol_version: "1"
+        plk_markup_format: "markdown"
+        plk_create_group_if_not_exists__d8_internal_virtualization_virt_handler_health: "D8InternalVirtualizationVirtHandlerHealth,tier=~tier,prometheus=deckhouse,kubernetes=~kubernetes"
+        plk_grouped_by__d8_internal_virtualization_virt_handler_health: "D8InternalVirtualizationVirtHandlerHealth,tier=~tier,prometheus=deckhouse,kubernetes=~kubernetes"
+        summary: "No virt-handler pods found in d8-virtualization namespace"
+        description: |
+          There are currently no virt-handler pods (matching 'virt-handler-.*') in the 'd8-virtualization' namespace.
+          Make sure that kvm is enabled on the cluster nodes intended for virtualization and at least one node is assigned to start virtual machines.

monitoring/prometheus-rules/virtualization-controller.yaml

@@ -19,7 +19,7 @@
           2. Or check the Pod logs: `kubectl -n d8-virtualization logs deploy/virtualization-controller`
 
     - alert: D8VirtualizationControllerTargetAbsent
-      expr: absent(up{job="scrapeconfig/d8-monitoring/virtualization-controller") == 1
+      expr: absent(up{job="scrapeconfig/d8-monitoring/virtualization-controller"}) == 1
       labels:
         severity_level: "6"
         tier: cluster

werf.yaml

@@ -98,7 +98,6 @@ shell:
 ---
 image: bundle
 fromImage: builder/scratch
-fromCacheVersion: "2024-11-07.1"
 import:
   - image: images-digests
     add: /images_digests.json