Skip to content

prometheus improvements #13667

Open
Open

Description

@DaanHoogland

problem

  1. Give the exporter's HttpServer an explicit bounded executor (httpServer.setExecutor(Executors.newFixedThreadPool(2))) so one slow scrape can't serialize/queue all others.
  2. Add a short TTL/in-flight guard around updateMetrics() (e.g., skip recompute if last run was < N seconds ago, or synchronize so concurrent scrapes share one in-progress computation) so scrape frequency can never multiply backend load.
  3. Instrument: log/measure updateMetrics() wall-clock time so the reporter (and CI) can confirm which sub-metric collector is actually slow and verify the fix closes the growth.

additional comments:

  1. Stale dynamic config (CONFIRMED) — capacity.calculate.workers is a runtime-dynamic setting, but the new shared executor only reads it once at first creation; live changes are silently ignored until a restart.
  2. Swallowed capacity-recalculation abort (CONFIRMED per the extra verify pass) — shutdown racing an in-flight recalculation throws RejectedExecutionException, caught by the blanket catch(Throwable) in recalculateCapacity(), silently skipping storage/IP/VLAN updates for that cycle.
  3. Unsynchronized race on _capacityExecutorService (CONFIRMED per the extra verify pass) — can leak a freshly-recreated pool that's never shut down again.
  4. Shared fixed-size pool serializes previously-independent callers (PLAUSIBLE) — rolling-maintenance host-drain gating can now queue behind the hourly timer or API-triggered recalculations.
  5. Pool no longer bounded to actual task count, so it can stay oversized/stale relative to fleet size (efficiency).
  6. Bundling this executor-lifecycle rewrite into what the reported bug (Scrape duration using prometheus is increasing continuously until restart #13586) only needed a one-line fix for (altitude/scope creep).
  7. Inconsistent lazy-vs-eager thread-pool lifecycle pattern within the same class (reuse/convention).
  8. Minor: the synchronized getter is called per-loop-iteration instead of hoisted once (efficiency).

Activity

  1. added this to the 4.24.0 milestone on Jul 22, 2026
  2. PrashantBhanage commented on Jul 23, 2026

    @PrashantBhanage

    @DaanHoogland , I'd like to work on this could you assign it to me?

  3. PrashantBhanage commented on Jul 23, 2026

    @PrashantBhanage

    Opened #13678 with the three changes from the issue, scoped to the prometheus exporter plugin only. PTAL when you get a chance.

  4. DaanHoogland commented on Jul 24, 2026

    @DaanHoogland
    ContributorAuthor

    @PrashantBhanage , happy to assign you, but let’s make sure we don’t create code conflicts (see my question #13678 (comment))

  5. PrashantBhanage commented on Jul 24, 2026

    @PrashantBhanage

    @PrashantBhanage , happy to assign you, but let’s make sure we don’t create code conflicts (see my question #13678 (comment))

    @DaanHoogland I have switched the base branch of PR #13678 to your ghi13586-prometheusDrainage-20 branch, and GitHub confirms there are zero merge conflicts.

  6. RosiKyu commented on Oct 5, 2026

    @RosiKyu
    Collaborator

    @DaanHoogland could you please confirm that this is addressed with: #13696, instead of the originally linked closed PR: #13678 - so we can close this issue? Thank you!

  7. DaanHoogland commented on Oct 5, 2026

    @DaanHoogland
    ContributorAuthor

    @RosiKyu , #13650 needs to be merged for this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions