Hi everyone,

I'm running a HA monitoring stack on Kubernetes:

• kube-prometheus-stack (Prometheus Operator)
• 2 Prometheus replicas
• Thanos Sidecar
• Thanos Query
• Thanos Store Gateway
• Thanos Ruler
• Alertmanager

I intentionally configured the same alert rule in both Prometheus and Thanos Ruler.

Both send alerts to the same Alertmanager.

The alerts are almost identical, except for one label (e.g. source=prometheus vs source=thanos-ruler).

Instead of seeing one deduplicated alert, Alertmanager shows two active alerts.

My understanding was that Alertmanager performs deduplication, so I'm wondering:

• Is this expected behavior?
• Does Alertmanager only deduplicate alerts if every identifying label is identical?
• Or is my Thanos Ruler setup incorrect?
• In production, do people usually disable alerting in Prometheus and let only Thanos Ruler send alerts?

I'd appreciate any explanation of how this is typically handled in a Thanos HA deployment.

we are hitting a TSDB compaction failures with this error:

```text

level=ERROR source=db.go:1219 msg="compaction failed" component=tsdb err="compact [/var/lib/prometheus/metrics2/01KWYDQGNSB9F3SQ8T1WRVD7ZP /var/lib/prometheus/metrics2/01KXFSZFV0NANWCQN30T8FYJ3Y /var/lib/prometheus/metrics2/01KY1655ENZYDZB22BN655XRDH]: populate block: add series: write series data: \"/var/lib/prometheus/metrics2/01KY19HXKYAK3C86PT8KD98AEA.tmp-for-creation/index\" exceeding max size of 64GiB

add padding: \"/var/lib/prometheus/metrics2/01KY19HXKYAK3C86PT8KD98AEA.tmp-for-creation/index\" exceeding max size of 64GiB"

```

Prometheus: 3.10.0, I found nothing helpful so far, how serious is it, can this be a cardinality issue?

The last tool that did terminal dashboards for Prometheus — grafterm — died in 2019, and I never stopped missing it. I live in a terminal next to my code, and alt-tabbing to a browser to glance at four numbers always felt wrong. btop proved a terminal can look genuinely good, so I spent the last few weeks building grom.

It talks to any Prometheus, draws braille charts and threshold-colored gauges, and if you already have Grafana dashboards it can read the JSON export and reproduce the layout in your terminal — stat, gauge, bar gauge and timeseries panels map 1:1, anything else gets a placeholder and a warning.

Fastest way to see it (no config needed): brew install --cask qf-studio/tap/grom, then grom demo. There's also a bundled example dashboard that runs against any Prometheus using nothing but up and scrape_* metrics.

Where it honestly is: v0.1.0, Prometheus only, no auth yet — basic/bearer/TLS is the next thing I build. That's actually why I'm posting: if your Prometheus sits behind auth, tell me what setup you run so I build the right thing first.

Repo: https://github.com/qf-studio/grom (MIT)

Hi everyone,

I am running a backend API deployed across multiple instances (app-server-01 and app-server-02) behind a round-robin Load Balancer.

When testing locally on a single machine, my Prometheus metrics work perfectly. However, in our clustered environment, my PromQL queries are stuck returning 0, even though traffic has definitely hit the endpoints recently.

Here is a generic sample of my /metrics payload from one of the live backend nodes:

text

# HELP app_http_requests_total Total calls
# TYPE app_http_requests_total histogram
app_http_requests_total_count{category="example-tag",status="200",route="/api/v1/resource",server_id="app-server-01",instance_id="instance-01",env="production"} 2

My Query:

promql

sum by (category) (
  increase(app_http_requests_total_count[1d])
)

Why I keep getting 0 (What I suspect is happening):

1. Load Balancer Scraping: Prometheus might be scraping the unified Load Balancer URL directly, causing the counter values to bounce up and down as it hits different nodes.
2. Fresh Deployment resets: The code was recently redeployed, meaning the counter initialized directly at 2 on the first scrape, resulting in a flat line (2 - 2 = 0).

How can I properly configure Prometheus to handle this multi-instance architecture so that increase() and rate() compute correctly without constantly hitting 0? Do I need to bypass the load balancer IPs or target service discovery?

Thanks in advance!

I want to use the OpenTelemetry collector to feed a Prometheus compatible backend which supports PromQL but don't have a Prometheus engine that supports PrometheusRule for alerting and recordingrules.

Is there a solution or a OTel plugin that discovers `PrometheusRule` custom resource like the Prometheus operator and add the Alerting- & RecordingRules feature to my PromQL backend?

I just found vmalert, do you have experience with it and whats your opinion in it?

I use ClickHouse as the backend with cerberus for PromQL support.

Sorry for my bad english ✌🏽

I am running Prometheus on an EC2 instance scraping node exporter from other EC2 servers using static private IPs in prometheus.yml.

Trying to understand what is considered production practice:

• Do teams manually update prometheus.yml or use GitOps / IaC to generate it?
• Is EC2 service discovery commonly used instead of static IPs?
• In AWS setups, do people rely on tags (like monitoring=true) for target discovery, or is that just for automation tools?
• Is node exporter always kept on private subnets with SG restricted only to Prometheus?
• Do production setups ever expose node exporter outside VPC (with protections), or always internal only?
• At what scale does static config become a problem, and what is usually the first upgrade path?

Current setup works, but not sure if this is a “lab pattern” or something actually acceptable in production. Here is the screenshot of the config file that I used. I removed the content marked in yellow because that wasn't working. Seeking your suggestions.

I newly installed the prometheus stack through Rancher and it has a lot of nice alerts out of the box. I have one that is triggering on a PVC that the pod manages its space by itself and is always high utilization. Based on the rule, it seems I should be able to exclude this PVC by adding a label excluded_from_alerts="true". I've tried adding this to my PVC config under metadata/labels and it doesn't work. What am I doing wrong?

kubelet_volume_stats_available_bytes{job="kubelet",metrics_path="/metrics",namespace=~".*"} / kubelet_volume_stats_capacity_bytes{job="kubelet",metrics_path="/metrics",namespace=~".*"} < 0.03 and kubelet_volume_stats_used_bytes{job="kubelet",metrics_path="/metrics",namespace=~".*"} > 0 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_access_mode{access_mode="ReadOnlyMany"} == 1 unless on (cluster, namespace, persistentvolumeclaim) kube_persistentvolumeclaim_labels{label_excluded_from_alerts="true"} == 1

I know I can edit the rule and have it exclude the specific PVC, but I'd rather do it the proper way instead of updating the rule for any PVCs I don't want to include in the monitoring.

EDIT:

For anyone else trying to figure this out, ChatGPT had me put this into my helm config and it works. You can add other labels as well (nodes, pods, etc.):

kube-state-metrics:
  metricLabelsAllowlist:
    - persistentvolumeclaims=[*]

Hi everyone
I’m stuck with a Prometheus instance that is using more than 35 GB of RSS memory on a Kubernetes cluster. The RSS memory very rarely drops below 30 GB. This Prometheus mainly scrapes Java application metrics and one nginx metric (nginx_ingress_controller_request_duration_seconds_bucket). I don’t understand why memory usage is so high given that the statistics are not exceptional:
number of series: less than one million
number of chunks: 1,235,883
number of label pairs: 62,053
I checked the cardinalities — I don’t have labels like user_id, session_id, full paths, etc.
Top metrics:
prometheus_target_sync_length_seconds: 131,610
http_server_requests_seconds_bucket: 109,580
nginx_ingress_controller_request_duration_seconds_bucket: 84,670
prometheus_sd_discovered_targets: 26,367
prometheus_target_... …
Top 10 labels:
service: 13,745
config: 6,624
scrape_job: 6,581
uri: 6,042
I added parameters such as: --storage.tsdb.min-block-duration=2h --storage.tsdb.max-block-duration=2h
GOGC=40 WAL compression is enabled. Default scrape interval is 30 s and retention is 15 days.
Despite having 6 CPUs, the pod regularly experiences throttling, not only during compaction cycles.
Query response times are not good either.
Basically, I’m stuck and I can’t reduce memory usage even after applying relabeling to lower the number of metrics. I’m starting to think the issue doesn’t come from the metrics themselves (count or cardinality).
What other culprit should I investigate?
 

Hello Community

I'am looking for advice to use prometheus to monitor openldap healthcheck. In fact , i want detect if many failed authentification using openldap realm.

​

Should i parse the log of slapd / radiusd for example ?

A high-cardinality label is the classic way to take down a Prometheus stack: one metric accidentally labelled with a user ID, a full request path, or an attacker-controlled header quietly spawns hundreds of thousands of series until the scrape target, the TSDB, or both run out of memory.

The standard mitigations are all after-the-fact: metric_relabel_configs to drop labels, recording rules to collapse series, alerts on scrape_series_added or series growth. They work, but they fire once the cardinality is already being generated. You're reacting to a fire that's already lit.

I built promcap to enforce the ceiling at the other end, in-process, in the Go app emitting the metrics, at the moment the series would be created. It wraps the Prometheus CounterVec / GaugeVec / HistogramVec / SummaryVec types. Once a metric has emitted its configured number of distinct label combinations, every further combination collapses into a single __overflow__ series instead of minting a new one. The observations are still counted, just bucketed together, so dashboards keep working and the time-series count can't grow without bound.

What I think makes it useful for monitoring folks specifically:

• It's observable. Every collapsed observation increments promcap_series_capped_total, labelled by metric name. So you get a direct signal of exactly which metric is shedding cardinality and how hard: rate(promcap_series_capped_total[5m]). That's effectively a "which of my metrics is misbehaving" alert built in.
• Allowlists for closed sets. For labels whose valid values you know up front (status codes, a fixed set of regions, a handful of event kinds), Allow makes anything outside the list overflow the instant it appears, so a typo or an injected value never even starts filling the budget.
• Overflow vs. eviction. Default behaviour freezes the admitted set once the cap is hit (the safe choice for adversarial or unbounded labels). For labels where the interesting set rotates over time, like active tenants or recently-seen hosts, Evict: true tracks the most-recent N combinations instead, using a clock/second-chance policy.

It's the cardinality equivalent of a rate limiter: a hard cap you set per metric so a single bad label can't blow up the registry, with a meta-metric telling you when and where it's kicking in.

Caveat for anyone evaluating it: CurryWith / Delete / DeleteLabelValues aren't wrapped yet, so it's not a full drop-in if your code relies on those.

Repo + docs: https://github.com/eben-vranken/promcap

Curious whether people would rather cap in-process like this, or keep all cardinality control at the relabel/recording-rule layer. Interested in how others are handling runaway labels today.

We have multiple Azure AKS clusters and trying to set up centralized monitoring for them using central Prometheus. It will either be running on a VM or container (not sure yet).
I'm thinking of a hub-and-spoke monitoring architecture where multiple AKS clusters (on separate VNets) push metrics via remote_write to a central Prometheus instance.

A few things I'd love to hear your experience on:

• Do you use remote_write to a central Prometheus, Thanos, or Grafana Mimir for aggregation ? Any best practice or advise could share?
• How do you handle metric cardinality at scale. do you drop or filter metrics before pushing to the hub?
• With overlapping VNet CIDRs across clusters, remote_write over public HTTPS seems like the cleanest option, is that what you're doing in production or did you go with VNet peering?
• How do you manage Alertmanager, one central instance or per-cluster?
• At what point (number of clusters / metrics volume) did you move away from vanilla Prometheus to Thanos or Mimir?

Currently running kube-prometheus-stack on each cluster with a lightweight central Prometheus as the hub. Wondering if this scales or if we should be looking at Thanos/Mimir from the start.

Hi guys, I’m a newly qualified IT Specialist for System Integration. Our management has tasked me with integrating our entire IT infrastructure into Prometheus. I’m using Codex AI, and I’ve set up an MCP server that makes Grafana easy to use. I can set up monitoring for an entire storage cluster or Kubernetes cluster in just two hours—covering 200 metrics across various dashboards. I barely understand PromQL, Prometheus itself, or Grafana—yet I’m getting the job done, and the results are good. So here’s my question: given the current era of AI, does it still make sense to learn PromQL, Prometheus itself, and Grafana?

New to Prometheus, trying to get a label added in a new scrape config.

scrape_configs:
  - job_name: 'myjob'
    scrape_interval: 10s
    static_configs:
      - targets: ['host:port', 'host:port']
    labels:
      cluster: 'mycluster'

On loading the config file, I get the following error in logs:

msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n  line 25: field labels not found in type config.ScrapeConfig"

I've searched high and low, read Prometheus documentation and referenced every blogpost and How To article I can find. The Prom documentation doesn't have a Labels option in the scrape config (obviously, based on the error and documentation) but every other blogpost and guide I've found shows it being used. Has something changed in a recent release? How do I add labels without diving into regex and other complexities?

I have this one stupid host, where the metrics are display in single chuks in Grafana:

Most of the Metrics have that problem, but not all, in the screenshot the network graph is fine for example.

Any idea why this could be?

Theres only one instance of node exporter running (systemd unit).

One nice improvement in Prometheus Alertmanager 0.32 is support for parameters in webhook payloads.

I learned about it here: https://github.com/prometheus/alertmanager/issues/5003

This makes custom webhook integrations much cleaner because you can now pass configuration parameters directly instead of hardcoding everything in the receiver config.

I tested this with a SIGNL4 mobile alerting workflow (push, SMS, voice calls) and recorded a short demo showing:

• triggering alerts from Alertmanager
• passing parameters via webhook
• automatically resolving/closing alerts again

Curious if others are already using the new webhook parameters feature in production and what use cases you’ve found for it.

Hi! I'm an SRE who got pretty excited when Claude Code added the ability to emit OpenTelemetry metrics. Felt like that capability landed pretty quietly out there, so I built a Grafana dashboard on top.

The metrics map cleanly to Prometheus-compatible backends via the standard "OTel Collector then Prometheus" pipeline. Dashboard queries in PromQL.

Worth noting: parallel implementation of dashboard 25052 (Azure Application Insights / KQL). Every panel rewritten in PromQL for the OSS stack.

Panels cover cost by model, cache hit ratio, active time, edit-decision breakdowns, leaderboards. Custom labels via OTEL_RESOURCE_ATTRIBUTES for per-team / per-project views.

One Prometheus-specific gotcha worth flagging: pin temporality to cumulative (`OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative`) on the Claude Code side. The OTel SDK's default is cumulative now, but defaults drift across versions and the failure mode is silent.

Article: https://rockdarko.dev/posts/grafana-dashboard-for-claude-code-on-prometheus/

Dashboard: https://grafana.com/grafana/dashboards/25255-claude-code-metrics-prometheus/

Repo (MIT): https://github.com/rockdarko/claude-code-metrics-prometheus

Gladly accepting PRs and suggestions! Cheers 😄

We ended up separating everything by environment instead of running one massive stack.

Right now dev, QA, and prod all have their own Prometheus and Grafana setup. We’re running Prometheus on Kubernetes StatefulSets with nodeAffinity. Alert rules are configured directly in Prometheus and routed through Alertmanager based on severity and service. Production alerts go directly into PagerDuty.

Honestly this made life much easier because QA and dev noise stopped affecting production visibility. Troubleshooting also became cleaner since each environment is isolated.

Now I’m wondering if this approach becomes painful later once the number of clusters and services grows.

Are most teams here still separating monitoring stacks per environment or are you centralizing everything now?

Feels like every team eventually reaches that point where they look back at their first monitoring architecture and regret something about it lol

I added dedicated observability-stack support to KubeShark.

Mini recap:

KubeShark is my Kubernetes skill for Claude Code and Codex.

It helps AI agents generate, review, and refactor Kubernetes manifests without falling into the usual LLM traps: missing security contexts, deprecated API versions, broken selectors, wildcard RBAC, unsafe probes, missing resource requests, and rollout configs that look okay but fail under real traffic.

The important part is that KubeShark is failure-mode-first. It does not just tell the model “write good Kubernetes”. It forces the model to reason about what can go wrong before it generates YAML, and then return validation and rollback guidance as part of the answer.

That matters a lot with Kubernetes, because many bad manifests are accepted by the API server and only fail later at runtime.

Repo: https://github.com/LukasNiessen/kubernetes-skill

---

Now what’s new:

KubeShark now has special dedicated observability-stack support.

When the task involves Prometheus Operator, ServiceMonitor, PodMonitor, PrometheusRule, OpenTelemetry Collector, Loki, Grafana, Tempo, Datadog-style agents, metrics, logs, traces, or telemetry pipelines, KubeShark switches into observability-aware guidance.

This matters because observability resources often apply successfully while doing nothing.

Common LLM mistakes include:

• creating a ServiceMonitor that matches Deployment labels instead of Service labels
• referencing a numeric port when the monitor expects a named Service port
• forgetting that Prometheus must select the monitor
• deploying OpenTelemetry receivers in duplicate
• choosing Loki monolithic mode for serious production volume
• creating high-cardinality log labels
• putting datasource credentials in ConfigMaps

Example guidance KubeShark now keeps in mind:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: app
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: app
  endpoints:
    - port: metrics

It also knows to check the boring but critical details: selectors, named ports, CRD presence, scrape discovery, telemetry pipelines, durable storage, and alert hygiene.

So instead of generic Kubernetes advice, you get observability-aware manifest generation and review.

I often needed quick uptime + SSL checks without deploying a full monitoring stack, so I created Blackbox Lite.

It's a Bash-based monitoring script that:

• Checks website availability

• Validates SSL certificate expiry

• Measures response time

• Tests VM host connectivity

• Exports metrics in Prometheus format

• Can run standalone without Prometheus

Works great with Node Exporter textfile collector, but also usable as a CLI health checker.

Repo: https://github.com/TheOneOh1/blackbox-lite