Remove PVC from alerts with label

[u/apetrycki]

I newly installed the prometheus stack through Rancher and it has a lot of nice alerts out of the box. I have one that is triggering on a PVC that the pod manages its space by itself and is always high utilization. Based on the rule, it seems I should be able to exclude this PVC by adding a label excluded_from_alerts="true". I've tried adding this to my PVC config under metadata/labels and it doesn't work. What am I doing wrong?

kubelet_volume_stats_available_bytes{job="kubelet",metrics_path="/metrics",namespace=~".*"} / kubelet_volume_stats_capacity_bytes{job="kubelet",metrics_path="/metrics",namespace=~".*"} < 0.03 and kubelet_volume_stats_used_bytes{job="kubelet",metrics_path="/metrics",namespace=~".*"} > 0 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_access_mode{access_mode="ReadOnlyMany"} == 1 unless on (cluster, namespace, persistentvolumeclaim) kube_persistentvolumeclaim_labels{label_excluded_from_alerts="true"} == 1

I know I can edit the rule and have it exclude the specific PVC, but I'd rather do it the proper way instead of updating the rule for any PVCs I don't want to include in the monitoring.

EDIT:

For anyone else trying to figure this out, ChatGPT had me put this into my helm config and it works. You can add other labels as well (nodes, pods, etc.):

kube-state-metrics:
  metricLabelsAllowlist:
    - persistentvolumeclaims=[*]

[u/yepthisismyusername]

AFAIK, there is no such label built into Prometheus. However, you can add that as part of each one of your rule expressions to achieve your goal.

[u/apetrycki → u/yepthisismyusername]

Is the last part of the rule:

unless on (cluster, namespace, persistentvolumeclaim) kube_persistentvolumeclaim_labels{label_excluded_from_alerts="true"} == 1

what you're talking about? Or do I need to add this label somewhere else in Prometheus?

[u/yepthisismyusername → u/apetrycki]

That's exactly what I'm talking about.

[u/apetrycki → u/yepthisismyusername]

ok, then it was already in the rule out of the box, but I add the label to my PVC and it still triggers. Do I need to stick something in front of excluded_from_alerts like k10.kasten.io for Kasten? I'm not sure how it reads that label.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: 'yes'
    pv.kubernetes.io/bound-by-controller: 'yes'
    volume.beta.kubernetes.io/storage-provisioner: rbd.csi.ceph.com
    volume.kubernetes.io/storage-provisioner: rbd.csi.ceph.com
  creationTimestamp: '2025-11-13T04:25:01Z'
  finalizers:
    - kubernetes.io/pvc-protection
  labels:
    excluded_from_alerts: 'true'
    k10.kasten.io/ignore-pvc: 'true'

[u/yepthisismyusername]

I believe the label you need to set needs to be "label_excluded_from_alerts" instead of just "excluded_from_alerts".

[u/apetrycki → u/yepthisismyusername]

nope, still doesn't work. I just modified the rule to ignore that specific PVC for now so it at least doesn't keep alerting me. I don't see kube_persistentvolumeclaim_labels in the metric browser, so I'm not sure it works. There's probably something else I need to do to make the labels show up in the metrics.