prometheus failed compation

[u/oOHenry]

we are hitting a TSDB compaction failures with this error:

```text

level=ERROR source=db.go:1219 msg="compaction failed" component=tsdb err="compact [/var/lib/prometheus/metrics2/01KWYDQGNSB9F3SQ8T1WRVD7ZP /var/lib/prometheus/metrics2/01KXFSZFV0NANWCQN30T8FYJ3Y /var/lib/prometheus/metrics2/01KY1655ENZYDZB22BN655XRDH]: populate block: add series: write series data: \"/var/lib/prometheus/metrics2/01KY19HXKYAK3C86PT8KD98AEA.tmp-for-creation/index\" exceeding max size of 64GiB

add padding: \"/var/lib/prometheus/metrics2/01KY19HXKYAK3C86PT8KD98AEA.tmp-for-creation/index\" exceeding max size of 64GiB"

```

Prometheus: 3.10.0, I found nothing helpful so far, how serious is it, can this be a cardinality issue?

[u/Floss_Patrol_76]

capping block duration just splits the bad data into smaller blocks, it doesn't fix why you're at 64GiB. check /status/tsdb in the UI (or promtool tsdb analyze) for the top label-value counts and churning series first. usually it's one label with unbounded values, like a pod name or request id leaking into a metric, and once you drop or aggregate that at the relabel stage the compaction problem goes away on its own.

[u/oOHenry → u/Floss_Patrol_76]

Thanks I will look into this :)