[v1r3n]

Pull Request type

• Bugfix
• Feature
• Refactoring (no functional changes, no api changes)
• Build related changes
• WHOSUSING.md
• Other (please describe):

Changes in this PR

Adaptive worker concurrency — an opt-in mode where workers size themselves instead of using fixed thread counts.

Today you guess a threadCount per worker and hope it's right: too low leaves backlog with an idle machine, too high risks OOM or CPU saturation. In adaptive mode, one controller per process continuously answers three questions:

1. How much work is waiting? Queue depths are sampled once a second (one batch GET /tasks/queue/sizes for all queues) and the process's capacity is divided among workers in proportion to their backlog — a queue with 600 waiting tasks gets ~6x the slots of one with 100.
2. Can this JVM handle more? Capacity grows ~20% per 5s tick while CPU (smoothed, container-quota-aware) and post-GC heap have headroom; it shrinks 25% under sustained CPU pressure, and pauses new work entirely when post-collection heap is near full — running tasks always finish and report.
3. Can we take this task right now? Capacity is reserved before every poll and the poll asks for at most that many tasks, so a polled task always has a free thread — nothing waits in an in-memory queue (a Conductor poll accepts work; it is not a preview).

Also: idle queues hold no threads but stay alive via standby long-polls (instant pickup) or 5s heartbeats (keeps the server's 10s active-poller window for domain routing); empty polls back off from the worker's configured polling interval up to 5s; an idle process decays back to minimum capacity.

Usage:

var controller = new AdaptiveConcurrencyController(new AdaptiveConcurrencyConfig.Builder().build());
new TaskRunnerConfigurer.Builder(taskClient, workers)
        .withAdaptiveConcurrency(controller)   // share one controller per process
        .build().init();

Spring: conductor.client.adaptive-concurrency.enabled=true; per-worker ceiling via conductor.worker.<task>.adaptive_max_concurrency.

Included: controller + config + JVM resource sampler, TaskRunner integration (fixed mode unchanged), Spring/annotated-worker wiring, new AdaptiveConcurrencyChanged event + Prometheus gauges (including adaptive_worker_task_slot_age_seconds, the wedged-worker signal), design doc (design/adaptive-worker-concurrency.md), unit tests, e2e suites, and a harness overhead scenario (controller lock p99 was 15µs with 305 queues).

Self-monitoring hardenings: a stalled GC-notification pipeline falls back to collection usage instead of freezing the heap estimate; standby polls require a free process slot; slot counters that would go negative are clamped and reported.

Measured (local server): 30 min at 8 wf/s — adaptive completed 98.7% of 14,400 workflows self-adjusting between 17 and 250 slots with zero client errors and flat memory; fixed 10-threads/worker completed 7% (13,331 of 14,400 workflows still queued or mid-pipeline at cutoff) with a linearly growing backlog, using 1.8x the poll requests. Full curves in the harness logs.

Behavior notes: disabled by default — fixed mode is untouched. In adaptive mode N pollers for a task share one pool (poller count no longer multiplies capacity). withTaskPollCount is deprecated. Virtual threads remain opt-in and use a thread-per-task executor.

Issue #

Alternatives considered

• Fixed threads with better defaults — still a static guess; cannot respond to backlog shifts or memory pressure at runtime.
• Used-heap instead of post-GC heap as the memory signal — rejected: used heap routinely spikes above 90% before a normal young collection; post-collection usage (incl. the fullest old-gen pool) is the JVM's real live-data estimate. Validated by load tests that OOM'd under the naive signal.
• Server-side admission control — would require protocol changes; this design works with existing APIs and keeps the server authoritative over queues while the client governs only its own capacity.