Repository navigation
feat(ha): add production scaling signals and graceful gateway redistribution #3528
Description
User Story
As an OpenShell cluster operator, I want multi-replica gateways to expose actionable capacity signals and redistribute supervisor ownership during planned changes, so that scaling and rolling upgrades do not leave one replica overloaded while new replicas remain idle.
Problem Statement
PR #1868 makes multi-replica gateway routing possible, but several production-scale behaviors remain incomplete:
- Supervisor ownership stays with the original connection. Rolling restarts and failover can leave the last surviving replica holding most sessions until sandboxes naturally reconnect.
- The gateway does not expose held-session, pending-relay, or peer-relay rate and latency metrics suitable for capacity planning or autoscaling.
- Cross-object mutations use one fleet-wide PostgreSQL advisory-lock key, so unrelated sandbox and provider mutations serialize across all replicas.
WatchSandboxperforms one database lookup per actively watched sandbox, per polling interval, per replica.
Impact / Why This Matters
Operators can add gateway replicas, but cannot tell when capacity is exhausted or rely on replicas receiving a balanced share of established supervisor sessions. The current workaround is to overprovision, manually inspect logs, and wait for sandbox churn or force reconnects after a rollout. That does not prevent pending-relay exhaustion, database amplification, or fleet-wide mutation contention at larger scale.
Proposed Design
Provide an observable, documented HA lifecycle with graceful draining and bounded shared-database load:
- A terminating gateway stops accepting new ownership, releases or transfers existing ownership, and prompts affected supervisors to reconnect before the termination grace period expires.
- Gateway metrics expose held supervisor sessions, pending relays versus capacity, and peer-relay request rate, failure rate, and latency.
- The Helm chart documents those signals and provides an optional autoscaling configuration, or a documented integration contract for a Kubernetes metrics adapter.
- Unrelated object mutations do not share one fleet-wide serialization point when their invariants cannot overlap.
- Cross-replica sandbox watches use a batched or notification-based mechanism whose database work does not grow as one query per watched sandbox on every replica.
Acceptance Criteria
- During a graceful gateway rollout, established supervisor ownership redistributes without waiting for natural sandbox churn, within the configured termination grace period.
- Operators can observe per-replica held sessions, pending relays/capacity, and peer-relay rate, failures, and latency through the supported metrics endpoint.
- The Helm documentation explains how to use these signals for horizontal scaling; any chart-provided HPA behavior is optional and configurable.
- Mutations for unrelated sandboxes or providers can proceed concurrently across replicas unless they share a documented cross-object invariant.
- Cross-replica watch synchronization avoids one database query per watched sandbox per interval per replica.
- Load or integration tests cover rollout redistribution, relay saturation signals, concurrent unrelated mutations, and large watch sets.
- HA architecture and Kubernetes operator documentation describe the lifecycle, capacity signals, and database-scaling behavior.
Alternatives Considered
Keeping ownership sticky is simple and preserves correctness, but leaves rollout skew entirely dependent on sandbox churn. Relying only on logs does not provide stable autoscaling inputs. Increasing the PostgreSQL connection pool masks neither fleet-wide lock serialization nor query amplification.
Agent Investigation
This groups the remaining production-scale review discussions from PR #1868:
- Graceful redistribution and sticky ownership: feat(kubernetes): support HA gateway rebalancing #1868 (comment)
- Scaling metrics and HPA signals: feat(kubernetes): support HA gateway rebalancing #1868 (comment)
- Fleet-wide advisory-lock serialization: feat(kubernetes): support HA gateway rebalancing #1868 (comment)
- Watch-poller query amplification: feat(kubernetes): support HA gateway rebalancing #1868 (comment)
Related umbrella issue: #1021.
Activity
feat(server): add HA capacity metrics, scoped locks, and graceful drain
feat(server): add HA capacity metrics, scoped locks, and graceful drain
feat(server): add HA capacity metrics, scoped locks, and graceful drain
feat(server): add HA capacity metrics, scoped locks, and graceful drain
feat(server): add HA capacity metrics, scoped locks, and graceful drain
feat(server): add HA capacity metrics, scoped locks, and graceful drain
feat(server): add HA capacity metrics, scoped locks, and graceful drain
feat(server): add gateway capacity metrics and optional HPA
feat(server): add gateway capacity metrics and optional HPA
feat(server): add gateway capacity metrics and optional HPA
88 remaining items
fix(server): reconcile endpoint status on startup one sandbox at a time
perf(server): skip the endpoint-status lock when a session was replaced
test(server): cover staged credential cleanup on a lock timeout
feat(server): export mutation lock wait and timeout metrics
test(server): add PostgreSQL mutation lock integration tests
ci: run the PostgreSQL-backed server tests
docs: document scoped mutation locks and lock-pool sizing
feat(server): check mutation lock ordering in debug builds
fix(server): do not hold gateway shutdown for the endpoint-status retry
refactor(server): make the platform profile lock scope explicit
perf(server): amortize local mutation lock table sweeps
feat(server): make the mutation lock pool size configurable
feat(server): export mutation lock errors and lock connections in use
feat(server): time mutation lock holds and test a stalled holder
feat(server): scope mutation locks and bound lock waits (NVIDIA#4147)