From ac83d05cb47f95cc979d7a064b44a501e74ef9b9 Mon Sep 17 00:00:00 2001 From: "warp-agent-staging[bot]" <240773466+warp-agent-staging[bot]@users.noreply.github.com> Date: Sat, 19 Sep 2026 18:55:42 +0000 Subject: [PATCH 1/3] docs: clarify Kubernetes resource failures Co-Authored-By: Oz --- .../self-hosting/managed-kubernetes.mdx | 19 ++++- .../docs/platform/self-hosting/monitoring.mdx | 13 ++++ .../self-hosting/security-and-networking.mdx | 2 +- .../platform/self-hosting/troubleshooting.mdx | 78 +++++++++++++++++-- 4 files changed, 101 insertions(+), 11 deletions(-) diff --git a/src/content/docs/platform/self-hosting/managed-kubernetes.mdx b/src/content/docs/platform/self-hosting/managed-kubernetes.mdx index f13bb13b3..1b5902e43 100644 --- a/src/content/docs/platform/self-hosting/managed-kubernetes.mdx +++ b/src/content/docs/platform/self-hosting/managed-kubernetes.mdx @@ -8,10 +8,10 @@ sidebar: --- import { VARS } from '@data/vars'; -Deploy the `oz-agent-worker` daemon into a Kubernetes cluster using the included Helm chart. Each agent task runs as a **Kubernetes Job** in your cluster. The {VARS.WARP_AUTOMATION_PLATFORM} orchestrates runs end to end (Slack, Linear, schedules, API, `oz agent run-cloud`); your cluster provides the compute, scheduling, and policy enforcement. +Deploy the `oz-agent-worker` daemon into a Kubernetes cluster using the included Helm chart. Each agent task runs as a Kubernetes Job in your cluster. The {VARS.WARP_AUTOMATION_PLATFORM} orchestrates runs end to end (Slack, Linear, schedules, API, `oz agent run-cloud`); your cluster provides the compute, scheduling, and policy enforcement. :::note -This page covers the [managed architecture](/platform/self-hosting/#managed-architecture) with the Kubernetes backend. For the default Docker backend, see [Managed: Docker](/platform/self-hosting/managed-docker/). For host execution without a container runtime, see [Managed: Direct](/platform/self-hosting/managed-direct/). To route runs to a connected worker, see [Routing runs to this worker](/platform/self-hosting/managed-docker/#routing-runs-to-this-worker). +Use the Kubernetes backend with the [managed architecture](/platform/self-hosting/#managed-architecture). For the default Docker backend, see [Managed: Docker](/platform/self-hosting/managed-docker/). For host execution without a container runtime, see [Managed: Direct](/platform/self-hosting/managed-direct/). To route runs to a connected worker, see [Routing runs to this worker](/platform/self-hosting/managed-docker/#routing-runs-to-this-worker). ::: ## When to use the Kubernetes backend @@ -28,7 +28,7 @@ This page covers the [managed architecture](/platform/self-hosting/#managed-arch 2. On startup, the worker runs a short-lived **preflight Job** to verify that cluster permissions, admission policies, and Pod Security Standards are compatible. If the preflight fails, the worker exits with a diagnostic error before accepting any tasks. 3. For each assigned task, the worker creates a Kubernetes Job in the configured namespace. 4. The worker monitors the Job and Pod status via Kubernetes Watch (with a 30-second safety-net poll for watch disconnects). -5. After the task completes, the Job is cleaned up (unless `--no-cleanup` is set). +5. After a successful task completes, the worker deletes its Job. Failed Jobs remain available temporarily for diagnosis before Kubernetes TTL cleanup. Set `--no-cleanup` to retain task Jobs without worker-managed TTL cleanup. --- @@ -117,7 +117,7 @@ To scale horizontally, deploy multiple Helm releases with distinct worker IDs ra * `worker.cleanup` — Whether to clean up task Jobs after execution. Defaults to `true`. * `worker.maxConcurrentTasks` — Maximum concurrent tasks. Defaults to `0` (unlimited). * `worker.idleOnComplete` — Duration to keep the oz process alive after task completion. -* `worker.resources` — Resource requests/limits for the worker Deployment. Defaults to `100m` CPU and `128Mi` memory. +* `worker.resources` — Resource requests and limits for the long-running worker Deployment, not task Jobs. The chart requests `100m` CPU and `128Mi` memory by default and sets no default limits. * `worker.livenessProbe` — Liveness probe for the worker Deployment. Defaults to an `exec` probe (`kill -0 1`). Override with a custom probe or set to `null` to disable. * `worker.nodeSelector`, `worker.tolerations`, `worker.affinity` — Scheduling constraints for the worker Deployment pod. @@ -165,6 +165,17 @@ The `pod_template` field accepts standard Kubernetes PodSpec YAML and is the dec When using `pod_template`, define a container named `task` to customize the main task container directly. Otherwise, the worker appends its own `task` container to the PodSpec. +### Size task workloads + +The worker does not set default CPU or memory resources on task containers. Configure workload resources in either of these places: + +* **Pod template** - Set `resources.requests` and `resources.limits` on the container named `task`. Use this for the worker's task baseline and use the rest of the PodSpec for scheduling controls such as node selectors, affinity, and tolerations. +* **Runner instance shape** - Assign a [runner](/platform/runners/) with the vCPUs and memory needed for a specific workload. On the Kubernetes backend, an explicit instance shape sets the `task` container's CPU and memory requests equal to its limits for that run. These values override matching `task` container resources from `pod_template`. + +Runner instance shapes size only the `task` container. Configure resources for setup init containers or other containers directly in `pod_template`. + +Requests determine whether the scheduler can place a Pod, while limits constrain a running container. Account for concurrent task Jobs and leave node headroom for system and other workloads. Use workload-specific runners for occasional heavy builds or tests instead of increasing every task's baseline. + Use `valueFrom.secretKeyRef` to inject Kubernetes Secret values into task container environment variables: ```yaml diff --git a/src/content/docs/platform/self-hosting/monitoring.mdx b/src/content/docs/platform/self-hosting/monitoring.mdx index d9263e2bf..15d8e1c94 100644 --- a/src/content/docs/platform/self-hosting/monitoring.mdx +++ b/src/content/docs/platform/self-hosting/monitoring.mdx @@ -143,9 +143,12 @@ All metrics use the `oz_worker_` prefix. Each worker process with metrics enable * **`oz_worker_tasks_rejected_total{reason}`** (counter) — Tasks the worker declined (e.g., `reason="at_capacity"`). * **`oz_worker_tasks_completed_total{result}`** (counter) — Completed tasks labeled `result="succeeded"` or `result="failed"`. * **`oz_worker_task_duration_seconds{result}`** (histogram) — Wall-clock task duration on the worker, labeled by result. +* **`oz_worker_task_failures_total{phase,reason}`** (counter) — Task failures by execution phase and bounded reason, including `container_oom`, `evicted`, and `unschedulable` for Kubernetes tasks. * **`oz_worker_websocket_reconnects_total{reason}`** (counter) — WebSocket reconnect attempts (e.g., `reason="dial_failed"`, `reason="remote_close"`). Spikes indicate flapping workers. * **`oz_worker_info{version,backend,worker_id}`** (gauge, constant `1`) — Build and runtime metadata. Useful for joining other series by labels. +Worker metrics identify failure categories and concurrency. Use Kubernetes Pod events, termination states, scheduler messages, node conditions, and resource metrics to diagnose the underlying OOM, eviction, or scheduling constraint. + --- ## Sample PromQL queries @@ -191,6 +194,16 @@ Direct mappings for common operational questions: sum(rate(oz_worker_tasks_completed_total{result="failed"}[5m])) ``` +* **Kubernetes resource failures by reason:** + + ```promql + sum by (reason) ( + rate(oz_worker_task_failures_total{ + reason=~"container_oom|evicted|unschedulable" + }[5m]) + ) + ``` + * **Reconnect storms (alert threshold):** ```promql diff --git a/src/content/docs/platform/self-hosting/security-and-networking.mdx b/src/content/docs/platform/self-hosting/security-and-networking.mdx index 8935cab8d..1bd81f384 100644 --- a/src/content/docs/platform/self-hosting/security-and-networking.mdx +++ b/src/content/docs/platform/self-hosting/security-and-networking.mdx @@ -83,7 +83,7 @@ The exception runs the other way. [Team-managed inference endpoints](/enterprise * **Kubernetes RBAC** — The worker needs namespaced permissions to create, get, list, watch, and delete Jobs and Pods. The Helm chart creates a minimal Role/RoleBinding scoped to a single namespace. The task namespace must allow creating Jobs with a root init container, as sidecar materialization currently depends on that pattern. Review your Pod Security Standards and admission policies accordingly. * **Kubernetes service accounts** — The worker Deployment's ServiceAccount (used by the long-lived worker process) is separate from the optional task Job `serviceAccountName` you may configure in `pod_template`. Scope each appropriately. * **API key management** — Store `WARP_API_KEY` in a Kubernetes Secret. Avoid hardcoding it in scripts or config files. If your organization uses an external secrets manager (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, etc.), you can inject secrets into task pods via the CSI Secrets Store Driver or a similar operator — configure the required `volumes`, `volumeMounts`, and annotations in `pod_template`. -* **Task isolation** — Each task runs as a separate Kubernetes Job/Pod. Jobs are removed after execution by default (disable with `--no-cleanup` for debugging). +* **Task isolation** — Each task runs as a separate Kubernetes Job/Pod. Successful Jobs are removed after execution. Failed Jobs remain temporarily available for diagnosis before Kubernetes TTL cleanup. Set `--no-cleanup` to retain task Jobs without worker-managed TTL cleanup. ### Direct backend diff --git a/src/content/docs/platform/self-hosting/troubleshooting.mdx b/src/content/docs/platform/self-hosting/troubleshooting.mdx index 2a53f2b64..4c5115af6 100644 --- a/src/content/docs/platform/self-hosting/troubleshooting.mdx +++ b/src/content/docs/platform/self-hosting/troubleshooting.mdx @@ -115,12 +115,78 @@ See [Monitoring](/platform/self-hosting/monitoring/) for the full setup guide. ### Kubernetes backend (task failures) -1. Check task Job and Pod status: `kubectl get jobs,pods -n `. -2. Common issues: - * **Unschedulable pods** — Check node selectors, tolerations, and resource requests in `pod_template`. - * **Image pull failures** — Check `imagePullSecrets` in `pod_template`. - * **Admission policy rejections** — Review Pod Security Standards, OPA Gatekeeper, Kyverno, or similar admission controllers. -3. The worker fails a task early if its pod remains unschedulable beyond `unschedulable_timeout` (default `30s`). Raise the timeout or fix the scheduling issue. +First determine whether the task Pod started. A running container that exceeds its memory limit and a Pending Pod that cannot fit on a node require different fixes. + +The Helm chart's `worker.resources` configures the long-running worker Deployment only. Task containers have no worker-defined CPU or memory defaults unless you set resources in `pod_template` or assign an explicit [runner instance shape](/platform/self-hosting/managed-kubernetes/#size-task-workloads). + +#### Task container was terminated with `OOMKilled` + +`OOMKilled` means Kubernetes reports that a container started and then encountered an out-of-memory condition. Confirm the termination reason before changing resources. + +1. Find the failed task Pod: + + ```bash + kubectl get jobs,pods -n NAMESPACE + ``` + + Replace `NAMESPACE` with the task namespace. Use the returned task Pod name as `POD_NAME` below. + +2. Inspect the `task` container's terminated state, resource settings, and Pod events: + + ```bash + kubectl describe pod POD_NAME -n NAMESPACE + kubectl get pod POD_NAME -n NAMESPACE -o yaml + ``` + +3. Compare peak memory usage with the configured limit using your cluster metrics. Check the node for `MemoryPressure` and eviction events. +4. Reduce the task's peak memory use or increase its memory limit. For a workload-specific [runner](/platform/runners/), increase the instance shape. For a baseline shared by all tasks on the worker, change the `task` container's resources in `pod_template`. + +Increasing a runner's memory also increases the task container's memory request to the same value. Confirm that a compatible node has enough allocatable memory, or the replacement Pod can remain Pending. + +If the Pod reason is `Evicted` instead, diagnose node pressure rather than a container limit. Restore node headroom, add compatible capacity, or adjust scheduling and concurrency before rerunning the task. + +#### Task remains `Pending` with `FailedScheduling` + +A Pending Pod with a `PodScheduled=False` condition and `FailedScheduling` events has not started. Messages such as `Insufficient cpu` or `Insufficient memory` mean no eligible node has enough allocatable capacity for the Pod's requests. + +1. Read the scheduler message and recent events: + + ```bash + kubectl describe pod POD_NAME -n NAMESPACE + kubectl get events -n NAMESPACE --sort-by=.lastTimestamp + ``` + +2. Compare the Pod's requests with node allocatable capacity and, when resource metrics are available, current usage: + + ```bash + kubectl describe nodes + kubectl top nodes + kubectl top pods -n NAMESPACE --containers + ``` + +3. Review the Pod's `nodeSelector`, affinity, tolerations, and taints. A node with free resources is not eligible if another scheduling constraint excludes it. +4. Check how many task Jobs run concurrently. Set `max_concurrent_tasks` to keep aggregate requests within cluster capacity when needed. +5. Right-size requests only if the task can run reliably at the lower values. Otherwise, add compatible node capacity or configure cluster autoscaling for nodes that satisfy the Pod's scheduling constraints. + +Raising only a memory limit does not help an unschedulable Pod because the scheduler places Pods from requests. The worker stops waiting after the configured `unschedulable_timeout`; fix the scheduling constraint rather than extending the timeout when the cluster lacks capacity. + +#### Task exits with code `143` + +Exit code `143` generally indicates `SIGTERM`; it does not prove that a container ran out of memory. Check the container termination reason, Pod conditions, and events before choosing a remediation. + +```bash +kubectl describe pod POD_NAME -n NAMESPACE +kubectl get events -n NAMESPACE --sort-by=.lastTimestamp +``` + +Look for eviction, preemption, node drain, `activeDeadlineSeconds`, or manual deletion. Correlate the event timeline with node pressure and resource metrics. Treat the failure as OOM only when Kubernetes reports `OOMKilled`. + +#### Other Kubernetes task failures + +* **Image pull failures** - Inspect `imagePullSecrets` in `pod_template`. +* **Admission policy rejections** - Review Pod Security Standards, OPA Gatekeeper, Kyverno, or similar admission controllers. + +With cleanup enabled, failed Jobs and Pods remain temporarily available for diagnosis before Kubernetes TTL cleanup. Use `--no-cleanup` when you need to retain them longer. ### Direct backend (task failures) From 542cb9507315f388321d1c9ab74d4438179f3e61 Mon Sep 17 00:00:00 2001 From: "warp-agent-staging[bot]" <240773466+warp-agent-staging[bot]@users.noreply.github.com> Date: Sat, 19 Sep 2026 19:13:10 +0000 Subject: [PATCH 2/3] docs: refine Kubernetes resource guidance Co-Authored-By: Oz --- .../self-hosting/managed-kubernetes.mdx | 86 +++---------------- .../self-hosting/security-and-networking.mdx | 4 - 2 files changed, 13 insertions(+), 77 deletions(-) diff --git a/src/content/docs/platform/self-hosting/managed-kubernetes.mdx b/src/content/docs/platform/self-hosting/managed-kubernetes.mdx index 1b5902e43..ca25cb2ec 100644 --- a/src/content/docs/platform/self-hosting/managed-kubernetes.mdx +++ b/src/content/docs/platform/self-hosting/managed-kubernetes.mdx @@ -10,9 +10,6 @@ import { VARS } from '@data/vars'; Deploy the `oz-agent-worker` daemon into a Kubernetes cluster using the included Helm chart. Each agent task runs as a Kubernetes Job in your cluster. The {VARS.WARP_AUTOMATION_PLATFORM} orchestrates runs end to end (Slack, Linear, schedules, API, `oz agent run-cloud`); your cluster provides the compute, scheduling, and policy enforcement. -:::note -Use the Kubernetes backend with the [managed architecture](/platform/self-hosting/#managed-architecture). For the default Docker backend, see [Managed: Docker](/platform/self-hosting/managed-docker/). For host execution without a container runtime, see [Managed: Direct](/platform/self-hosting/managed-direct/). To route runs to a connected worker, see [Routing runs to this worker](/platform/self-hosting/managed-docker/#routing-runs-to-this-worker). -::: ## When to use the Kubernetes backend @@ -28,7 +25,7 @@ Use the Kubernetes backend with the [managed architecture](/platform/self-hostin 2. On startup, the worker runs a short-lived **preflight Job** to verify that cluster permissions, admission policies, and Pod Security Standards are compatible. If the preflight fails, the worker exits with a diagnostic error before accepting any tasks. 3. For each assigned task, the worker creates a Kubernetes Job in the configured namespace. 4. The worker monitors the Job and Pod status via Kubernetes Watch (with a 30-second safety-net poll for watch disconnects). -5. After a successful task completes, the worker deletes its Job. Failed Jobs remain available temporarily for diagnosis before Kubernetes TTL cleanup. Set `--no-cleanup` to retain task Jobs without worker-managed TTL cleanup. +5. The worker deletes successful Jobs. Failed Jobs remain until Kubernetes TTL cleanup. Set `--no-cleanup` to disable worker-managed cleanup. --- @@ -106,44 +103,13 @@ To scale horizontally, deploy multiple Helm releases with distinct worker IDs ra ## Key chart values -**Required:** - -* `worker.workerId` — The worker ID (same as `--worker-id`). -* `image.tag` — The worker image tag to deploy. - -**Worker configuration:** - -* `worker.logLevel` — Log verbosity (`debug`, `info`, `warn`, `error`). Defaults to `info`. -* `worker.cleanup` — Whether to clean up task Jobs after execution. Defaults to `true`. -* `worker.maxConcurrentTasks` — Maximum concurrent tasks. Defaults to `0` (unlimited). -* `worker.idleOnComplete` — Duration to keep the oz process alive after task completion. -* `worker.resources` — Resource requests and limits for the long-running worker Deployment, not task Jobs. The chart requests `100m` CPU and `128Mi` memory by default and sets no default limits. -* `worker.livenessProbe` — Liveness probe for the worker Deployment. Defaults to an `exec` probe (`kill -0 1`). Override with a custom probe or set to `null` to disable. -* `worker.nodeSelector`, `worker.tolerations`, `worker.affinity` — Scheduling constraints for the worker Deployment pod. - -**Kubernetes backend:** +Set `worker.workerId` and `image.tag` for each installation. These values affect capacity and task placement: -* `kubernetesBackend.namespace` — Namespace for task Jobs. Defaults to the release namespace. -* `kubernetesBackend.defaultImage` — Default Docker image for task pods when no [Warp environment](/platform/environments/) has been supplied. Leave empty (default) to fall back to `ubuntu:22.04`. -* `kubernetesBackend.imagePullPolicy` — Image pull policy for task pods. Defaults to `IfNotPresent`. -* `kubernetesBackend.preflightImage` — Image for the startup preflight Job. Set this if your cluster restricts allowed registries. -* `kubernetesBackend.unschedulableTimeout` — How long a pod may remain unschedulable before failing. Defaults to `30s`. -* `kubernetesBackend.setupCommand` — Shell command to run before each task. -* `kubernetesBackend.teardownCommand` — Shell command to run after each task. -* `kubernetesBackend.extraLabels` — Additional labels for task Jobs and Pods. -* `kubernetesBackend.extraAnnotations` — Additional annotations for task Jobs and Pods. -* `kubernetesBackend.activeDeadlineSeconds` — Maximum task Job lifetime. -* `kubernetesBackend.workspaceSizeLimit` — Size limit for workspace `emptyDir` volume. -* `kubernetesBackend.podTemplate` — Raw PodSpec YAML for task Jobs (same as `backend.kubernetes.pod_template` in the [config file](/platform/self-hosting/reference/#config-file)). +* `worker.resources` — CPU and memory for the worker Deployment, not task Jobs. The chart requests `100m` CPU and `128Mi` memory by default and sets no limits. +* `worker.maxConcurrentTasks` — Maximum concurrent tasks. The default `0` allows unlimited tasks. +* `kubernetesBackend.podTemplate` — Raw PodSpec YAML for task Jobs. -**API key Secret:** - -* `warp.apiKeySecret.create` — Set to `true` to have the chart create a Secret from `warp.apiKeySecret.value`. Defaults to `false` (expects a pre-existing Secret). -* `warp.apiKeySecret.value` — The API key value to store in the chart-managed Secret. Only used when `warp.apiKeySecret.create` is `true`. -* `warp.apiKeySecret.name` — Name of the Secret containing `WARP_API_KEY`. Defaults to `oz-agent-worker`. -* `warp.apiKeySecret.key` — Key within the Secret. Defaults to `WARP_API_KEY`. - -See the [self-hosted worker reference](/platform/self-hosting/reference/#kubernetes-backend-config) for the full config file schema. +See the [self-hosted worker reference](/platform/self-hosting/reference/#kubernetes-backend-config) for every worker, backend, API key, and metrics value. --- @@ -167,14 +133,14 @@ When using `pod_template`, define a container named `task` to customize the main ### Size task workloads -The worker does not set default CPU or memory resources on task containers. Configure workload resources in either of these places: +The worker does not set default CPU or memory resources on task containers. Configure task resources in either of these places: -* **Pod template** - Set `resources.requests` and `resources.limits` on the container named `task`. Use this for the worker's task baseline and use the rest of the PodSpec for scheduling controls such as node selectors, affinity, and tolerations. -* **Runner instance shape** - Assign a [runner](/platform/runners/) with the vCPUs and memory needed for a specific workload. On the Kubernetes backend, an explicit instance shape sets the `task` container's CPU and memory requests equal to its limits for that run. These values override matching `task` container resources from `pod_template`. +* **Pod template** — Set `resources.requests` and `resources.limits` on the container named `task`. Use the rest of the PodSpec for node selectors, affinity, and tolerations. +* **Runner instance shape** — Assign a [runner](/platform/runners/) with the vCPUs and memory needed for a workload. An explicit shape sets the `task` container's CPU and memory requests equal to its limits and overrides matching `pod_template` values. -Runner instance shapes size only the `task` container. Configure resources for setup init containers or other containers directly in `pod_template`. +A runner instance shape sizes only `task`. A `pod_template` resource setting applies only to a container you define in the template; it does not alter worker-generated setup or materialization init containers. -Requests determine whether the scheduler can place a Pod, while limits constrain a running container. Account for concurrent task Jobs and leave node headroom for system and other workloads. Use workload-specific runners for occasional heavy builds or tests instead of increasing every task's baseline. +Requests determine Pod placement, while limits constrain a running container. Account for concurrent Jobs and node headroom. Use workload-specific runners for occasional heavy builds or tests instead of increasing every task's baseline. Use `valueFrom.secretKeyRef` to inject Kubernetes Secret values into task container environment variables: @@ -204,9 +170,7 @@ pod_template: effect: "NoSchedule" ``` -:::note -The worker Deployment's ServiceAccount is separate from the task Job `serviceAccountName` you configure in `pod_template`. The Deployment ServiceAccount needs RBAC to manage Jobs and Pods. The task ServiceAccount (if any) controls what the agent process can access at runtime. -::: +The task Job's `serviceAccountName` controls runtime access. It is separate from the worker Deployment's ServiceAccount, which manages Jobs and Pods. --- @@ -247,31 +211,7 @@ Use `kubernetesBackend.setupCommand` (Helm value) or `backend.kubernetes.setup_c ## Metrics -The Helm chart includes built-in support for exporting OpenTelemetry metrics from the worker. Enable metrics by setting `metrics.enabled=true`: - -```bash -helm install oz-agent-worker ./charts/oz-agent-worker \ - --namespace warp-oz \ - --set worker.workerId=oz-k8s-worker \ - --set image.tag=VERSION \ - --set metrics.enabled=true -``` - -With the default `metrics.exporter=prometheus`, the chart creates a `Service` with Prometheus scrape annotations and exposes port `9464`. For clusters using the Prometheus Operator, set `metrics.podMonitor.create=true` to create a `PodMonitor`. - -To push metrics to an OTLP collector instead, set `metrics.exporter=otlp` and configure the endpoint via `metrics.extraEnv`. - -See [Monitoring](/platform/self-hosting/monitoring/) for the full list of Helm values, the metric catalog, and sample PromQL queries. - ---- - -## Operational notes - -* **Scaling** — The chart always deploys a single replica for a given `worker.workerId`. To run multiple workers, deploy multiple Helm releases with distinct worker IDs rather than scaling a single release horizontally. -* **Security context** — The Deployment defaults to a non-root security context (`runAsUser: 10001`) with `allowPrivilegeEscalation: false` and all capabilities dropped. -* **Liveness probe** — The Deployment includes a default `exec` liveness probe (`kill -0 1`). Override `worker.livenessProbe` for a custom probe, or set it to `null` to disable. -* **In-cluster auth** — The chart assumes the worker runs inside the target cluster and uses in-cluster Kubernetes auth by default. -* **Root init containers** — The worker Deployment itself is non-root, but task Jobs require a root init container for sidecar materialization. Ensure the task namespace's Pod Security Standards allow this. +Enable worker OpenTelemetry metrics with `metrics.enabled=true`. See [Monitoring](/platform/self-hosting/monitoring/) for Helm values, the metric catalog, and sample PromQL queries. --- diff --git a/src/content/docs/platform/self-hosting/security-and-networking.mdx b/src/content/docs/platform/self-hosting/security-and-networking.mdx index 1bd81f384..4a324c538 100644 --- a/src/content/docs/platform/self-hosting/security-and-networking.mdx +++ b/src/content/docs/platform/self-hosting/security-and-networking.mdx @@ -7,10 +7,6 @@ description: >- Self-hosting uses a split-plane architecture. Understanding which data stays on your infrastructure and which data routes through Warp is critical for security evaluation. This page summarizes the data model, network egress requirements, and backend-specific security considerations for self-hosted workers. -:::note -This page applies to both the [managed](/platform/self-hosting/#managed-architecture) and [unmanaged](/platform/self-hosting/unmanaged/) architectures. Backend-specific notes call out Docker-, Kubernetes-, and Direct-only considerations. -::: - ## Data boundaries Self-hosted execution keeps repository clones, source files, build artifacts, runtime secrets, environment variables, and agent workspaces on infrastructure you control. Agents can also reach internal systems that your host can reach, such as VPN-only services, private databases, or self-hosted source control. From f6689fbe248e38ea027bda3847e17139b7fe9454 Mon Sep 17 00:00:00 2001 From: "warp-agent-staging[bot]" <240773466+warp-agent-staging[bot]@users.noreply.github.com> Date: Sun, 20 Sep 2026 17:26:49 +0000 Subject: [PATCH 3/3] Polish stacked troubleshooting guidance Co-Authored-By: Oz --- src/content/docs/platform/self-hosting/troubleshooting.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/content/docs/platform/self-hosting/troubleshooting.mdx b/src/content/docs/platform/self-hosting/troubleshooting.mdx index 26c1f7d96..bc0b5df39 100644 --- a/src/content/docs/platform/self-hosting/troubleshooting.mdx +++ b/src/content/docs/platform/self-hosting/troubleshooting.mdx @@ -91,7 +91,7 @@ See [Security and networking](/platform/self-hosting/security-and-networking/#ne 1. For Prometheus, confirm `OTEL_METRICS_EXPORTER=prometheus`, bind to `0.0.0.0` in Docker or Kubernetes, and run `curl -s localhost:9464/metrics`. 2. Confirm no firewall or network policy blocks the metrics port. 3. For OTLP, confirm the collector endpoint is reachable and the protocol is correct. -4. For Helm, confirm `metrics.enabled=true` and check the `Service` or `PodMonitor`. A `PodMonitor` requires the Prometheus Operator CRDs. +4. For Helm, confirm `metrics.enabled=true` and check the Kubernetes Service or PodMonitor. A PodMonitor requires the Prometheus Operator CRDs. 5. Restart the worker with `--log-level debug` and check for metrics errors. See [Monitoring](/platform/self-hosting/monitoring/) for the full setup guide.