You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Give the exporter's HttpServer an explicit bounded executor (httpServer.setExecutor(Executors.newFixedThreadPool(2))) so one slow scrape can't serialize/queue all others.
Add a short TTL/in-flight guard around updateMetrics() (e.g., skip recompute if last run was < N seconds ago, or synchronize so concurrent scrapes share one in-progress computation) so scrape frequency can never multiply backend load.
Instrument: log/measure updateMetrics() wall-clock time so the reporter (and CI) can confirm which sub-metric collector is actually slow and verify the fix closes the growth.
additional comments:
Stale dynamic config (CONFIRMED) — capacity.calculate.workers is a runtime-dynamic setting, but the new shared executor only reads it once at first creation; live changes are silently ignored until a restart.
Swallowed capacity-recalculation abort (CONFIRMED per the extra verify pass) — shutdown racing an in-flight recalculation throws RejectedExecutionException, caught by the blanket catch(Throwable) in recalculateCapacity(), silently skipping storage/IP/VLAN updates for that cycle.
Unsynchronized race on _capacityExecutorService (CONFIRMED per the extra verify pass) — can leak a freshly-recreated pool that's never shut down again.
Shared fixed-size pool serializes previously-independent callers (PLAUSIBLE) — rolling-maintenance host-drain gating can now queue behind the hourly timer or API-triggered recalculations.
Pool no longer bounded to actual task count, so it can stay oversized/stale relative to fleet size (efficiency).
@DaanHoogland I have switched the base branch of PR #13678 to your ghi13586-prometheusDrainage-20 branch, and GitHub confirms there are zero merge conflicts.
@DaanHoogland could you please confirm that this is addressed with: #13696, instead of the originally linked closed PR: #13678 - so we can close this issue? Thank you!
Activity
PrashantBhanage commentedon Jul 23, 2026
@DaanHoogland , I'd like to work on this could you assign it to me?
PrashantBhanage commentedon Jul 23, 2026
Opened #13678 with the three changes from the issue, scoped to the prometheus exporter plugin only. PTAL when you get a chance.
DaanHoogland commentedon Jul 24, 2026
@PrashantBhanage , happy to assign you, but let’s make sure we don’t create code conflicts (see my question #13678 (comment))
PrashantBhanage commentedon Jul 24, 2026
@DaanHoogland I have switched the base branch of PR #13678 to your ghi13586-prometheusDrainage-20 branch, and GitHub confirms there are zero merge conflicts.
prometheus: fix scrape duration growing unbounded (#13667) (#13696)
prometheus: fix scrape duration growing unbounded (#13667) (#13696)
RosiKyu commentedon Oct 5, 2026
@DaanHoogland could you please confirm that this is addressed with: #13696, instead of the originally linked closed PR: #13678 - so we can close this issue? Thank you!
DaanHoogland commentedon Oct 5, 2026
@RosiKyu , #13650 needs to be merged for this.