Document WebSocket relay HA, NATS vs AMQP backend selection, and operator troubleshooting.

Update architecture, ws-sessions guide, swagger WS/NATS creds paths, and changelog for relay production and NATS platform account naming.

Author: emirhandurmus

docs/architecture.md

@@ -206,15 +206,45 @@ Full spec: [`.cursor/controllerv3.8/docs/15-fog-platform-reconcile.md`](../.curs
 
 ## WebSocket exec & log sessions
 
-Interactive **exec** and **log streaming** use paired WebSocket sessions between operators (Bearer JWT), Controller, and Edgelet agents (fog token). Plan 16 hardens log sessions and shared WS infra (HA, drain, OTEL). **Plan 17** redesigns **microservice exec** to log-style multi-session flow (3 concurrent per MS, agent poll + session-scoped WS) — **Edgelet agent wire change required** for exec (see [edgelet-invariants.md §10.1](../.cursor/controllerv3.8/docs/edgelet-invariants.md)).
+Interactive **exec** and **log streaming** use paired WebSocket sessions between operators (Bearer JWT), Controller, and Edgelet agents (fog token). Plan 16 hardens log sessions and shared WS infra (HA, drain, OTEL). **Plan 17** redesigns **microservice exec** to log-style multi-session flow (3 concurrent per MS, agent poll + session-scoped WS). **Plan 18** production-hardens cross-replica relay via **`WsRelayTransport`** — AMQP pool + recovery when `nats.enabled=false`, NATS Core when `nats.enabled=true` (R102–R113). **Edgelet agent wire change required** for exec only (see [edgelet-invariants.md §10.1](../.cursor/controllerv3.8/docs/edgelet-invariants.md)).
+
+```mermaid
+flowchart TB
+  subgraph ws [WebSocketServer]
+    U[User WS]
+    A[Agent WS]
+  end
+
+  subgraph factory [WsRelayTransportFactory]
+    SEL{nats.enabled?}
+  end
+
+  subgraph amqp [AmqpRelayTransport]
+    POOL[RouterConnectionManager pool x8]
+    Q[agent/user queues per sessionId]
+  end
+
+  subgraph nats [NatsRelayTransport]
+    NC[NatsRelayConnectionManager]
+    SUB[Core pub/sub per sessionId]
+  end
+
+  U --> ws
+  A --> ws
+  ws --> factory
+  SEL -->|false| amqp
+  SEL -->|true| nats
+  POOL --> Q
+  NC --> SUB
+```
 
 ```mermaid
 sequenceDiagram
   participant U as User WS
   participant C as Controller
   participant DB as MicroserviceExecSessions
   participant CT as Change tracking
-  participant Q as AMQP
+  participant R as WsRelayTransport
   participant A as Edgelet agent
 
   Note over U,A: MS exec (R92–R98) — log-style
@@ -228,11 +258,11 @@ sequenceDiagram
   A->>C: GET /agent/exec/sessions
   A->>C: WS /agent/exec/microservice/:uuid/:sessionId
   C->>DB: ACTIVE agentConnected
-  C->>Q: agent-{sessionId} user-{sessionId}
+  C->>R: enable bridge if cross-replica
   U->>C: STDIN
-  C->>Q->>A: relay
+  C->>R->>A: relay
   A->>C: STDOUT
-  C->>Q->>U: relay
+  C->>R->>U: relay
   U-->>C: close
   C->>DB: DELETE row
   C->>CT: execSessions if needed
@@ -242,12 +272,12 @@ sequenceDiagram
   C->>C: provision debug system MS
   U->>C: WS /microservices/system/exec/:debugMsUuid
 
-  Note over U,A: Logs (R82–R83, R84) — unchanged Plan 16
+  Note over U,A: Logs (R82–R83, R84/R112)
   U->>C: WS logs + tail params
   C->>DB: PENDING sessionId
   A->>C: WS agent/logs/:sessionId
   A->>C: LOG_LINE
-  C->>Q->>U: logs-user-{sessionId}
+  C->>R->>U: relay
 ```
 
 | Topic | Normative value |
@@ -263,21 +293,23 @@ sequenceDiagram
 | Log concurrency | **3** user log WS per microservice (or per fog for node logs) |
 | Log limits | Tail max **5,000** lines; **120s** pending; **2h** idle |
 | Log content | Live relay only — no log line persistence; audit connect/disconnect |
-| HA relay | Cross-replica sessions **require** AMQP (`WebSocketQueueService`); same-replica may use direct WS; **fail fast** when router down |
+| HA relay | Cross-replica sessions **require** a **relay backend** (R112): **AMQP** router queues when `nats.enabled=false`; **NATS Core** subjects on hub when `nats.enabled=true`. Same-replica may use direct WS; **fail fast** close **1013** when active backend unavailable |
 | Graceful drain | **30s** on SIGTERM / k8s `preStop` — CLOSE frames, queue cleanup, session row delete |
 | Security | Agent handlers validate fog token **before** message processing; **50** upgrades/min/IP; **100** active WS/IP; JWT in `?token=` (ingress log redaction required) |
 | Scale SLO | **500** concurrent WS per replica; **p99 pairing < 5s** |
 | Observability | OpenTelemetry: active/pending sessions, pairing latency, AMQP failures, router connectivity |
 
 **OTEL metric names (R87):** `ws_exec_sessions_active`, `ws_log_sessions_active`, `ws_pending_pairings`, `ws_pairing_duration_ms` (histogram), `ws_amqp_publish_errors`, `ws_router_connected` (gauge). Emitted when `ENABLE_TELEMETRY=true`; see `src/websocket/ws-metrics.js`.
 
-**HA config (`server.webSocket.ha`):** `crossReplicaRequiresAmqp` (default `true`), `failFastOnRouterUnavailable` (default `true`). Env: `WS_HA_CROSS_REPLICA_REQUIRES_AMQP`, `WS_HA_FAIL_FAST_ON_ROUTER_UNAVAILABLE`. Graceful drain timeout: `server.webSocket.session.drainTimeoutMs` (default **30s**, env `WS_DRAIN_TIMEOUT_MS`).
+**Relay transport (Plan 18, R102):** Selected once at startup from existing `nats.enabled` / `NATS_ENABLED` — **no new relay env var**. `false` → AMQP pool (8 connections, sticky by `sessionId`); `true` → NATS Core on hub with **`controller`** NATS account. Relay connect is **lazy** — does not block Controller startup.
+
+**HA config (`server.webSocket.ha`):** `crossReplicaRequiresAmqp` (default `true`; semantics: cross-replica requires **active relay backend** per R112), `failFastOnRouterUnavailable` (default `true`; applies to selected backend). Env: `WS_HA_CROSS_REPLICA_REQUIRES_AMQP`, `WS_HA_FAIL_FAST_ON_ROUTER_UNAVAILABLE`. Graceful drain timeout: `server.webSocket.session.drainTimeoutMs` (default **30s**, env `WS_DRAIN_TIMEOUT_MS`).
 
-**Core modules:** `src/websocket/server.js`, `exec-session-manager.js` (Plan 17), `log-session-manager.js`, `src/services/websocket-queue-service.js`, `src/services/router-connection-service.js`.
+**Core modules:** `src/websocket/server.js`, `exec-session-manager.js` (Plan 17), `log-session-manager.js`, `ws-relay-transport-factory.js` / `amqp-relay-transport.js` / `nats-relay-transport.js` (Plan 18), `src/services/websocket-queue-service.js`, `src/services/router-connection-manager.js` (Plan 18), `src/services/nats-relay-connection-manager.js` (Plan 18).
 
-**Operator guide:** [operations/ws-sessions.md](operations/ws-sessions.md) — ingress `?token=` log redaction, HTTPS/WSS, multi-replica AMQP requirement, k8s preStop drain, load SLO probe.
+**Operator guide:** [operations/ws-sessions.md](operations/ws-sessions.md) — ingress `?token=` log redaction, HTTPS/WSS, multi-replica relay backend (`nats.enabled`), k8s preStop drain, load SLO probe.
 
-Full spec: Plan 16 [logs + shared infra](../.cursor/controllerv3.8/docs/16-ws-exec-log-hardening.md) · Plan 17 [MS exec](../.cursor/controllerv3.8/docs/17-multi-exec-sessions.md) · RFC R80–R91, R92–R101 · Edgelet contract: [edgelet-invariants.md §10–§10.1](../.cursor/controllerv3.8/docs/edgelet-invariants.md).
+Full spec: Plan 16 [logs + shared infra](../.cursor/controllerv3.8/docs/16-ws-exec-log-hardening.md) · Plan 17 [MS exec](../.cursor/controllerv3.8/docs/17-multi-exec-sessions.md) · Plan 18 [WS relay production](../.cursor/controllerv3.8/docs/18-ws-relay-production.md) · RFC R80–R91, R92–R101, R102–R113 · Edgelet contract: [edgelet-invariants.md §10–§10.1](../.cursor/controllerv3.8/docs/edgelet-invariants.md).
 
 ---
 

docs/operations/ws-sessions.md

@@ -6,7 +6,7 @@
 
 ## Overview
 
-Controller exposes **interactive exec** and **log streaming** over WebSocket on the API port (default **51121**). Sessions pair an operator browser/CLI client (Bearer JWT) with an Edgelet agent (fog token). In multi-replica deployments, cross-replica relay requires the **Skupper-style AMQP router** microservice.
+Controller exposes **interactive exec** and **log streaming** over WebSocket on the API port (default **51121**). Sessions pair an operator browser/CLI client (Bearer JWT) with an Edgelet agent (fog token). In multi-replica deployments, cross-replica relay uses a **relay backend** selected at startup by **`nats.enabled`** (Plan 18, R102): **AMQP** router queues when `false`, **NATS Core** on the platform hub when `true`.
 
 ---
 
@@ -38,18 +38,41 @@ Without redaction, long-lived bearer tokens may appear in load balancer logs.
 
 ## Multi-replica HA
 
+Relay transport is selected **once at startup** from existing platform config — **no separate relay env var** (R102):
+
+| `nats.enabled` | Cross-replica relay backend |
+|----------------|----------------------------|
+| `false` (default) | **AMQP** — Skupper-style router queues via `WebSocketQueueService` |
+| `true` | **NATS Core** — hub pub/sub subjects `controller.relay.v1.*` via `NatsRelayTransport` |
+
+Set `NATS_ENABLED=true` only when the platform NATS hub is deployed and all Controller replicas share the same value.
+
 | Setting | Default | Env |
 |---------|---------|-----|
-| Cross-replica requires AMQP | `true` | `WS_HA_CROSS_REPLICA_REQUIRES_AMQP` |
-| Fail fast when router down | `true` | `WS_HA_FAIL_FAST_ON_ROUTER_UNAVAILABLE` |
+| Cross-replica requires relay backend | `true` | `WS_HA_CROSS_REPLICA_REQUIRES_AMQP` |
+| Fail fast when relay backend down | `true` | `WS_HA_FAIL_FAST_ON_ROUTER_UNAVAILABLE |
 
-**Requirements:**
+> Env names retain `AMQP`/`ROUTER` for backward compatibility; semantics apply to the **active relay backend** (AMQP or NATS) per R112.
 
-1. Deploy the **router** system microservice and ensure Controller can reach AMQP (`RouterConnectionService`).
+### AMQP relay (`nats.enabled=false`)
+
+1. Deploy the **router** system microservice and ensure Controller can reach AMQP (`RouterConnectionManager` pool).
 2. Run **2+ Controller replicas** behind a load balancer with **sticky sessions optional** — cross-replica exec/log uses AMQP queues (`agent-{sessionId}`, `user-{sessionId}`, `logs-user-{sessionId}`).
-3. When the router is unavailable, new cross-replica sessions close with WebSocket code **1013** (`Router unavailable for cross-replica session`).
+3. When the router/AMQP backend is unavailable, new cross-replica sessions close with WebSocket code **1013** (`Router unavailable for cross-replica session`).
+
+Plan 18 adds an **8-connection AMQP pool** per replica with overflow recovery — intense log streams must not poison other sessions (no router restart required). **Remote CP** resolves **`router.default.svc.bridge.local`** then default router `host`; **Kubernetes CP** resolves **`router.{namespace}.svc.cluster.local`** then default router `host`. Port from `Routers.messagingPort` (default **5671**).
 
-Same-replica sessions may relay directly without AMQP when both user and agent land on the same pod.
+### NATS relay (`nats.enabled=true`)
+
+1. Platform NATS hub must be running with `NatsInstances.isHub=true`.
+2. Controller provisions dedicated **`controller`** NATS account/user (not SYS / `admin-hub`) via NATS auth reconcile.
+3. Cross-replica exec uses subjects `controller.relay.v1.exec.{sessionId}.agent` / `.user`; logs use `controller.relay.v1.log.{sessionId}.user`. Plain TCP to hub — port from `NatsInstances.serverPort` (default **4222**) for every host in the resolver list.
+4. **Remote CP:** Controller resolves **`nats.default.svc.bridge.local`** (Edgelet internal DNS) then hub `host`; both use hub `serverPort`.
+5. **Kubernetes CP:** Controller resolves **`nats-server.{namespace}.svc.cluster.local`** then hub `host`.
+6. Remote ControlPlane replicas connect to the **hub** NATS only — not local fog NATS leaf.
+7. When NATS relay is unavailable, fail-fast semantics match AMQP (close **1013** when configured).
+
+Same-replica sessions may relay directly without AMQP or NATS when both user and agent land on the same pod.
 
 ---
 
@@ -59,7 +82,7 @@ On shutdown, Controller drains WebSocket sessions for up to **`WS_DRAIN_TIMEOUT_
 
 1. Reject new upgrades (`verifyClient` → draining).
 2. Close pending users with code **1001** (`Server draining`).
-3. Send CLOSE frames, clean exec/log session DB rows, tear down AMQP bridges.
+3. Send CLOSE frames, clean exec/log session DB rows, tear down relay bridges (AMQP or NATS).
 
 ### Kubernetes manifest example
 
@@ -113,6 +136,15 @@ node test/load/ws-pairing-load.js --multi-ms 100
 
 The `--multi-ms` mode creates **3 exec sessions per microservice** (100 MS × 3 = 300 pairs) to validate multi-session pairing latency under the same p99 SLO.
 
+**AMQP profile** (`nats.enabled=false`): run the probe above on a dev machine — it exercises in-process `ExecSessionManager` pairing only (no router required). Record p99 from stdout; target **< 5000 ms**.
+
+**NATS profile** (`nats.enabled=true`): the same probe validates session-manager pairing latency (transport-agnostic SLO). For end-to-end NATS relay validation in staging:
+
+1. Deploy Controller with **`NATS_ENABLED=true`** on **2+ replicas** and a platform NATS hub (`NatsInstances.isHub=true`).
+2. Confirm **`controller`** NATS account reconcile succeeded (NATS auth logs).
+3. Run cross-replica exec/log sessions (user on replica A, agent on replica B) while recording OTEL **`ws_pairing_duration_ms`** p99.
+4. Optionally repeat `node test/load/ws-pairing-load.js --pairs 500` against staging API with agent simulators — same **p99 < 5s** SLO applies.
+
 For production validation, repeat against a staging cluster with real agent simulators and record p99 from Controller OTEL histogram `ws_pairing_duration_ms`.
 
 ---
@@ -128,7 +160,9 @@ Enable `ENABLE_TELEMETRY=true`. Key metrics (`src/websocket/ws-metrics.js`):
 | `ws_pending_pairings` | gauge |
 | `ws_pairing_duration_ms` | histogram |
 | `ws_amqp_publish_errors` | counter |
-| `ws_router_connected` | gauge |
+| `ws_amqp_session_saturated` | counter (Plan 18 overflow/backpressure) |
+| `ws_router_pool_connections` | gauge (Plan 18 AMQP pool health) |
+| `ws_router_pool_unsettled` | gauge (Plan 18 AMQP unsettled deliveries) |
 
 ---
 

docs/swagger.yaml

@@ -2774,8 +2774,7 @@ paths:
         `{ sessionId, microserviceUuid }` in the `data` field and top-level `sessionId`,
         followed by STDERR "waiting for agent…". Relay frames use `execId` equal to `sessionId`.
 
-        **HA:** Multi-replica deployments require AMQP router (`WebSocketQueueService`).
-        Cross-replica sessions fail fast with close code **1013** when router is unavailable.
+        **HA (R112):** Multi-replica deployments require a **relay backend** selected at startup by **`nats.enabled`**: AMQP router queues when `false` (default), NATS Core pub/sub on the platform hub when `true`. Cross-replica sessions fail fast with close code **1013** when the active relay backend is unavailable. See `docs/operations/ws-sessions.md`.
 
         See `#/components/schemas/WsExecMessageTypes` and `#/components/schemas/WsCloseCodes`.
       operationId: userMicroserviceExecWebSocket
@@ -6593,11 +6592,16 @@ paths:
       tags:
         - NATS
       summary: Gets NATS creds for specific account user
+      description: |
+        Returns base64-encoded `.creds` file content for the user.
+
+        Path `{appName}` is an **application name** for app-linked accounts, or a **NATS account name** for platform/system accounts (`SYS`, leaf accounts, **`controller`**).
       operationId: getNatsUserCreds
       parameters:
         - in: path
           name: appName
           required: true
+          description: Application name or NATS account name (e.g. `SYS`, `controller`)
           schema:
             type: string
         - in: path
@@ -7033,7 +7037,7 @@ tags:
   - name: WebSocketSessions
     description: |
       Interactive exec and log streaming WebSocket endpoints (MessagePack binary).
-      Multi-replica HA requires AMQP router — see operations/ws-sessions.md.
+      Multi-replica HA requires a relay backend per **`nats.enabled`** (AMQP when `false`, NATS Core when `true`) — see operations/ws-sessions.md.
   - name: User
     description: Manage your users
   - name: Secrets
@@ -8276,7 +8280,11 @@ components:
           description: True when microservice is the ControlPlane controller workload (DB column)
     ControllerRegisterRequest:
       type: object
-      description: Slim register body for Edgelet ControlPlane controller workload
+      description: >-
+        Edgelet ControlPlane controller workload register body. Accepts the same
+        container/workload fields as user microservice deploy (excluding application,
+        serviceAccount, and natsConfig). Ownership fields (iofogUuid, application)
+        are derived from the authenticated system fog.
       required:
         - uuid
         - images
@@ -8298,6 +8306,31 @@ components:
             $ref: "#/components/schemas/MicroserviceContainerImage"
         registryId:
           type: integer
+        config:
+          type: string
+        annotations:
+          type: string
+        hostNetworkMode:
+          type: boolean
+        isPrivileged:
+          type: boolean
+        logSize:
+          type: integer
+          minimum: 0
+        runtime:
+          type: string
+        pidMode:
+          type: string
+        ipcMode:
+          type: string
+        runAsUser:
+          type: string
+        platform:
+          type: string
+        cpuSetCpus:
+          type: string
+        memoryLimit:
+          type: integer
         ports:
           type: array
           items:
@@ -8306,16 +8339,57 @@ components:
           type: array
           items:
             $ref: "#/components/schemas/VolumeMappingRequest"
+        extraHosts:
+          type: array
+          items:
+            type: object
+            required:
+              - name
+              - address
+            properties:
+              name:
+                type: string
+              address:
+                type: string
         env:
           type: array
           items:
             $ref: "#/components/schemas/AgentEnvRequest"
-        config:
-          type: string
-        hostNetworkMode:
-          type: boolean
-        runtime:
-          type: string
+        cmd:
+          type: array
+          items:
+            type: string
+        cdiDevices:
+          type: array
+          items:
+            type: string
+        capAdd:
+          type: array
+          items:
+            type: string
+        capDrop:
+          type: array
+          items:
+            type: string
+        healthCheck:
+          type: object
+          required:
+            - test
+          properties:
+            test:
+              type: array
+              items:
+                type: string
+            interval:
+              type: integer
+            timeout:
+              type: integer
+            startPeriod:
+              type: integer
+            startInterval:
+              type: integer
+            retries:
+              type: integer
         schedule:
           type: integer
           enum: