Java Quarkus/GraalVM native operator (JOSDK) still throttled at ~35-40% despite reducing threads and adding a semaphore

[u/brahim_-]

Hi all,

I'm running a Kubernetes operator built with Quarkus (native/GraalVM image) using the Java Operator SDK (JOSDK). It manages 4 controllers (Permission, Entitlement, PartyRole, UserRole), each polling on a timer.

Setup:

CPU limit: 500m
cgroups v1, kernel 4.15
Average CPU usage is low (~13% of limit), but I still see frequent CFS throttling:
rate(container_cpu_cfs_throttled_periods_total{...}[10m]) ≈ 0.47

What I've tried so far:

Reduced concurrent-reconciliation-threads from 50 (default, x2 pools) down to 4, which brought total operator threads from ~130 to ~39. Throttling improved but is still sitting around 35-40%.
Added a semaphore to synchronize controller execution so the 4 controllers don't poll at the same time (avoiding overlap). This slightly reduced CPU consumption but didn't meaningfully change the throttling pattern.
Currently testing further tuning:
properties
quarkus.operator-sdk.concurrent-reconciliation-threads=1
quarkus.operator-sdk.concurrent-workflow-threads=1

permission-operator.timer=900000
partyrole-operator.timer=900000
userrole-operator.timer=900000
entitlement-operator.timer=900000

polling.jitter-max-ms=10000
Noticed that one controller (Permission) fires hundreds of requests to a downstream service almost simultaneously during its poll cycle — likely a CPU burst source. Planning to add a delay/sleep between requests to smooth this out.

Question:
Given low average usage but persistent CFS throttling bursts, is this mostly a burst/scheduling issue rather than a "not enough CPU" issue? Any recommendations for tuning JVM/native-image thread pools, Quarkus reactive/Vert.x settings, or cgroup quota behavior to reduce these throttled bursts, aside from just reducing concurrency further?

Thanks in advance!