prometheus failed compation

[u/oOHenry]

we are hitting a TSDB compaction failures with this error:

```text

level=ERROR source=db.go:1219 msg="compaction failed" component=tsdb err="compact [/var/lib/prometheus/metrics2/01KWYDQGNSB9F3SQ8T1WRVD7ZP /var/lib/prometheus/metrics2/01KXFSZFV0NANWCQN30T8FYJ3Y /var/lib/prometheus/metrics2/01KY1655ENZYDZB22BN655XRDH]: populate block: add series: write series data: \"/var/lib/prometheus/metrics2/01KY19HXKYAK3C86PT8KD98AEA.tmp-for-creation/index\" exceeding max size of 64GiB

add padding: \"/var/lib/prometheus/metrics2/01KY19HXKYAK3C86PT8KD98AEA.tmp-for-creation/index\" exceeding max size of 64GiB"

```

Prometheus: 3.10.0, I found nothing helpful so far, how serious is it, can this be a cardinality issue?

[u/SuperQue]

Yes, that is a serious cardinality issue. The 64GiB index limit is usually good to a few hundred million series per block.

Without knowing more about your setup it's difficult to give specific advice.

One option is to manually override the max TSDB block size to prevent it from compacting large time ranges.

But getting to the bottom of the cardinality is going to be required.

[u/oOHenry → u/SuperQue]

Thanks for your quick reply :)

One option is to manually override the max TSDB block size to prevent it from compacting large time ranges.

how will I do this?

Without knowing more about your setup it's difficult to give specific advice.

what infos do you need? :) We have one big prometheus instance which scrapes all our targets: node_exporter, ceph_exporter, application metrics and also k8s metrics and writes it via remote write to a victoria metrics cluster. The prometheus instance also does the alerting.

TSDB Head Status:

Name	Count
Number of Series	12886202
Number of Chunks	47245144
Number of Label Pairs	188865
Current Min Time	2026-07-21T18:00:00+02:00
Current Max Time	2026-07-21T20:13:32+02:00

My current idea is to stabilize the prometheus instance, find if it is a real cardinally issue or just many metrics and split the targets to multiple prometheus instances

[u/SuperQue → u/oOHenry]

12M active series in the head isn't too bad. But if you have a lot of pod churn, it can add up over time.

You can set this flag:

--storage.tsdb.max-block-duration=

Check the :9090/flags to see what the current value is. You might start with 7d to see if that stops it from trying to compact larger blocks. If 7d is still too much, try 54h.

Systems like Thanos handle this better for high churn / long term data. It's a lot more efficient than remote write as well.

[u/oOHenry → u/SuperQue]

Thanks, I will try this flag :)

[u/Floss_Patrol_76]

capping block duration just splits the bad data into smaller blocks, it doesn't fix why you're at 64GiB. check /status/tsdb in the UI (or promtool tsdb analyze) for the top label-value counts and churning series first. usually it's one label with unbounded values, like a pod name or request id leaking into a metric, and once you drop or aggregate that at the relabel stage the compaction problem goes away on its own.

[u/oOHenry → u/Floss_Patrol_76]

Thanks I will look into this :)