docs: Improve self-hosted Kubernetes troubleshooting

src/content/docs/platform/self-hosting/managed-kubernetes.mdx

@@ -8,11 +8,7 @@ sidebar:
 ---
 import { VARS } from '@data/vars';
 
-Deploy the `oz-agent-worker` daemon into a Kubernetes cluster using the included Helm chart. Each agent task runs as a **Kubernetes Job** in your cluster. The {VARS.WARP_AUTOMATION_PLATFORM} orchestrates runs end to end (Slack, Linear, schedules, API, `oz agent run-cloud`); your cluster provides the compute, scheduling, and policy enforcement.
-
-:::note
-This page covers the [managed architecture](/platform/self-hosting/#managed-architecture) with the Kubernetes backend. For the default Docker backend, see [Managed: Docker](/platform/self-hosting/managed-docker/). For host execution without a container runtime, see [Managed: Direct](/platform/self-hosting/managed-direct/). To route runs to a connected worker, see [Routing runs to this worker](/platform/self-hosting/managed-docker/#routing-runs-to-this-worker).
-:::
+Deploy the `oz-agent-worker` daemon into a Kubernetes cluster with the included Helm chart. Each agent task runs as a Kubernetes Job. The {VARS.WARP_AUTOMATION_PLATFORM} orchestrates runs; your cluster handles compute, scheduling, and policy enforcement.
 
 ## When to use the Kubernetes backend
 
@@ -25,21 +21,21 @@ This page covers the [managed architecture](/platform/self-hosting/#managed-arch
 ## How it works
 
 1. The worker connects to the Kubernetes API server (using in-cluster auth by default, or an explicit kubeconfig).
-2. On startup, the worker runs a short-lived **preflight Job** to verify that cluster permissions, admission policies, and Pod Security Standards are compatible. If the preflight fails, the worker exits with a diagnostic error before accepting any tasks.
+2. On startup, the worker runs a preflight Job with the configured task pod shape.
 3. For each assigned task, the worker creates a Kubernetes Job in the configured namespace.
-4. The worker monitors the Job and Pod status via Kubernetes Watch (with a 30-second safety-net poll for watch disconnects).
-5. After the task completes, the Job is cleaned up (unless `--no-cleanup` is set).
+4. The worker monitors the Job and Pod status.
+5. After the task completes, the worker removes successful Jobs. Failed Jobs remain available for diagnosis for 24 hours by default.
 
 ---
 
 ## Prerequisites
 
 * **Enterprise plan with self-hosting enabled** — [Contact sales](https://www.warp.dev/contact-sales) if self-hosting is not yet enabled for your team.
 * **A Kubernetes cluster** with the worker process able to reach the API server. The cluster must:
-  * Allow the worker's namespace to create Jobs with a **root init container** (sidecar materialization depends on this pattern).
+  * Allow the task namespace to create Jobs with a **root init container**, unless you enable native image volumes with `kubernetesBackend.useImageVolumes=true`.
   * Grant the worker these namespace-scoped permissions: `create`, `get`, `list`, `watch`, `delete` on `jobs`; `get`, `list`, `watch` on `pods`; `get` on `pods/log`; `list` on `events`.
 * **[Helm](https://helm.sh/docs/intro/install/)** installed locally, plus `kubectl` authenticated against the target cluster.
-* **An agent API key** — Create one in the <a href={`${VARS.WEB_APP_URL}/settings`}>{VARS.WEB_APP}</a> so the worker can authenticate to the {VARS.WARP_AUTOMATION_PLATFORM}. You can bind the key to any cloud agent — that choice doesn't restrict which agents can run on the worker. See [API Keys](/reference/cli/api-keys/) for the full creation flow.
+* **An agent API key** — Create one in the <a href={`${VARS.WEB_APP_URL}/settings`}>{VARS.WEB_APP}</a> so the worker can authenticate to the {VARS.WARP_AUTOMATION_PLATFORM}. Binding the key to a cloud agent doesn't restrict which agents can run on the worker. See [API Keys](/reference/cli/api-keys/).
 
 ---
 
@@ -98,7 +94,7 @@ helm install oz-agent-worker ./oz-agent-worker/charts/oz-agent-worker \
 Set `image.tag` explicitly to pin the worker image. Check the [oz-agent-worker releases](https://github.com/warpdotdev/oz-agent-worker/releases) for the latest version. Do not rely on `latest`.
 :::
 
-**Expected outcome:** `kubectl get pods -n warp-oz` shows the worker Deployment pod as `Running`, and the worker logs show `Connected to Oz` / `Listening for tasks`.
+**Expected outcome:** `kubectl get pods -n warp-oz` shows the worker Deployment pod as `Running`, and the worker logs include `Successfully connected to server`.
 
 To scale horizontally, deploy multiple Helm releases with distinct worker IDs rather than increasing replicas on a single release.
 
@@ -119,20 +115,25 @@ To scale horizontally, deploy multiple Helm releases with distinct worker IDs ra
 * `worker.idleOnComplete` — Duration to keep the oz process alive after task completion.
 * `worker.resources` — Resource requests/limits for the worker Deployment. Defaults to `100m` CPU and `128Mi` memory.
 * `worker.livenessProbe` — Liveness probe for the worker Deployment. Defaults to an `exec` probe (`kill -0 1`). Override with a custom probe or set to `null` to disable.
+* `worker.terminationGracePeriodSeconds` — Grace period for worker Deployment shutdown. Defaults to `30`.
 * `worker.nodeSelector`, `worker.tolerations`, `worker.affinity` — Scheduling constraints for the worker Deployment pod.
 
 **Kubernetes backend:**
 
 * `kubernetesBackend.namespace` — Namespace for task Jobs. Defaults to the release namespace.
 * `kubernetesBackend.defaultImage` — Default Docker image for task pods when no [Warp environment](/platform/environments/) has been supplied. Leave empty (default) to fall back to `ubuntu:22.04`.
 * `kubernetesBackend.imagePullPolicy` — Image pull policy for task pods. Defaults to `IfNotPresent`.
+* `kubernetesBackend.useImageVolumes` — Use native Kubernetes image volumes instead of root init containers to materialize sidecars. Defaults to `false`.
 * `kubernetesBackend.preflightImage` — Image for the startup preflight Job. Set this if your cluster restricts allowed registries.
+* `kubernetesBackend.preflightResources` — CPU and memory requests and limits for preflight containers.
+* `kubernetesBackend.sidecarImage` — Internal-registry override for the Warp agent sidecar image.
 * `kubernetesBackend.unschedulableTimeout` — How long a pod may remain unschedulable before failing. Defaults to `30s`.
 * `kubernetesBackend.setupCommand` — Shell command to run before each task.
 * `kubernetesBackend.teardownCommand` — Shell command to run after each task.
 * `kubernetesBackend.extraLabels` — Additional labels for task Jobs and Pods.
 * `kubernetesBackend.extraAnnotations` — Additional annotations for task Jobs and Pods.
-* `kubernetesBackend.activeDeadlineSeconds` — Maximum task Job lifetime.
+* `kubernetesBackend.activeDeadlineSeconds` — Maximum task Job lifetime. Defaults to eight hours.
+* `kubernetesBackend.ttlSecondsAfterFinished` — Retention period for failed Jobs and Jobs orphaned by worker disruption. Defaults to 24 hours when cleanup is enabled.
 * `kubernetesBackend.workspaceSizeLimit` — Size limit for workspace `emptyDir` volume.
 * `kubernetesBackend.podTemplate` — Raw PodSpec YAML for task Jobs (same as `backend.kubernetes.pod_template` in the [config file](/platform/self-hosting/reference/#config-file)).
 
@@ -193,23 +194,17 @@ pod_template:
       effect: "NoSchedule"
 ```
 
-:::note
-The worker Deployment's ServiceAccount is separate from the task Job `serviceAccountName` you configure in `pod_template`. The Deployment ServiceAccount needs RBAC to manage Jobs and Pods. The task ServiceAccount (if any) controls what the agent process can access at runtime.
-:::
+The worker Deployment ServiceAccount needs RBAC to manage Jobs and Pods. The task Job `serviceAccountName` in `pod_template` controls the agent process's runtime access.
+
+A run's instance shape overrides the `task` container's CPU and memory values in `pod_template`.
 
 ---
 
 ## Preflight check
 
-On startup, the worker creates a short-lived preflight Job to verify that:
-
-* The worker has sufficient RBAC permissions in the target namespace.
-* Cluster admission policies (Pod Security Standards, OPA Gatekeeper, Kyverno, etc.) allow the worker's task pod shape.
-* The preflight image can be pulled.
-
-If the preflight fails, the worker logs a diagnostic error and exits before accepting any tasks. This surfaces policy and configuration issues at deploy time rather than at task execution time.
+On startup, the worker runs a preflight Job with the configured task PodSpec. It catches insufficient RBAC, admission-policy failures, and sidecar-loading problems. If preflight fails, the worker exits before accepting tasks. A successful preflight does not validate task-specific images, Secrets, setup commands, or network access.
 
-The preflight image defaults to `busybox:1.36`. If your cluster restricts allowed registries or images, set `preflight_image` to an allowlisted image. When `imagePullSecrets` is configured in `pod_template`, those secrets apply to the preflight Job as well, so you can point `preflight_image` at an image in your private registry.
+The preflight image defaults to `busybox:1.36`. For an allowlisted or private image, set `kubernetesBackend.preflightImage`. The `imagePullSecrets` in `kubernetesBackend.podTemplate` also apply to preflight.
 
 ---
 
@@ -222,9 +217,7 @@ There are two ways to pass environment variables to Kubernetes task containers:
 
 When configuring the Kubernetes backend via YAML or Helm, declarative task-container env belongs in `pod_template` rather than a separate top-level list.
 
-:::note
-If your organization uses an external secrets manager (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, etc.), you can inject secrets into task pods via the CSI Secrets Store Driver or a similar operator. Configure the required `volumes`, `volumeMounts`, and annotations in `pod_template` just as you would for any other Kubernetes workload. See your secrets provider's documentation for details.
-:::
+For an external secrets manager, inject task secrets through a CSI driver or operator. Add the provider's `volumes`, `volumeMounts`, and annotations to `pod_template`.
 
 ---
 
@@ -234,6 +227,38 @@ Use `kubernetesBackend.setupCommand` (Helm value) or `backend.kubernetes.setup_c
 
 ---
 
+## Protect active task pods from disruption
+
+Terminating the worker pod normally leaves active task Jobs running. Evicting a task pod interrupts the run and deletes its pod-local `emptyDir` workspace. Configure node lifecycle tooling to avoid voluntary disruption of active task pods.
+
+For Karpenter, add its pod-level disruption annotation to every task Job through the Helm values:
+
+```yaml title="values.yaml"
+kubernetesBackend:
+  extraAnnotations:
+    karpenter.sh/do-not-disrupt: "true"
+```
+
+The annotation blocks Karpenter consolidation. It blocks drift only when the NodePool omits `terminationGracePeriod`. Expiration, interruption, node repair, and manual deletion can still terminate the node. With `terminationGracePeriod`, Karpenter can terminate blocking pods when the period ends. Review [Karpenter's pod-level disruption controls](https://karpenter.sh/docs/concepts/disruption/#pod-level-controls).
+
+A PodDisruptionBudget (PDB) constrains tools that use the Kubernetes Eviction API. It protects a group of pods, not an individual task's process or workspace. Direct deletion, kubelet pressure eviction, node failure, and controllers that bypass the Eviction API can still terminate a task.
+
+For other node lifecycle tools, use the equivalent protection and verify which disruption paths bypass it. A replacement pod cannot resume an interrupted run.
+
+---
+
+## Plan capacity and scheduling
+
+Task pods need capacity for their configured requests before the worker's unschedulable timeout expires.
+
+* Set `worker.maxConcurrentTasks` to a finite value that matches cluster capacity. The default, `0`, does not cap concurrency.
+* Set task CPU and memory through a runner instance shape or the `task` container in `kubernetesBackend.podTemplate`. Reserve capacity for init containers, DaemonSets, and workload spikes.
+* Set `kubernetesBackend.unschedulableTimeout` longer than the slowest expected node provisioning time. The default is `30s`; `0s` disables the check.
+* Use `worker.nodeSelector`, `worker.tolerations`, and `worker.affinity` for the worker Deployment. Use the equivalent fields in `kubernetesBackend.podTemplate` for task pods.
+* A toleration makes a pod eligible for a tainted node; it does not reserve capacity. Pair dedicated-node tolerations with matching selectors or affinity and autoscaler capacity.
+
+---
+
 ## Metrics
 
 The Helm chart includes built-in support for exporting OpenTelemetry metrics from the worker. Enable metrics by setting `metrics.enabled=true`:
@@ -260,7 +285,6 @@ See [Monitoring](/platform/self-hosting/monitoring/) for the full list of Helm v
 * **Security context** — The Deployment defaults to a non-root security context (`runAsUser: 10001`) with `allowPrivilegeEscalation: false` and all capabilities dropped.
 * **Liveness probe** — The Deployment includes a default `exec` liveness probe (`kill -0 1`). Override `worker.livenessProbe` for a custom probe, or set it to `null` to disable.
 * **In-cluster auth** — The chart assumes the worker runs inside the target cluster and uses in-cluster Kubernetes auth by default.
-* **Root init containers** — The worker Deployment itself is non-root, but task Jobs require a root init container for sidecar materialization. Ensure the task namespace's Pod Security Standards allow this.
 
 ---
 

src/content/docs/platform/self-hosting/monitoring.mdx

@@ -8,18 +8,14 @@ description: >-
 ---
 import { VARS } from '@data/vars';
 
-When enabled, the `oz-agent-worker` daemon exports infrastructure-level metrics over [OpenTelemetry](https://opentelemetry.io/), giving your team real-time visibility into worker health, task throughput, and capacity. Combine these metrics with the <a href={VARS.WEB_APP_URL}>{VARS.DASHBOARD}</a> for full observability across both the orchestration plane and your self-hosted compute.
-
-:::note
-Metrics export is opt-in. Set `OTEL_METRICS_EXPORTER` to `prometheus`, `otlp`, or `console` to enable it. When the variable is unset, empty, whitespace-only, or `none`, the worker does not initialize a metrics exporter. The Helm chart also enables export only when `metrics.enabled=true`.
-:::
+The `oz-agent-worker` daemon exports [OpenTelemetry](https://opentelemetry.io/) metrics for worker health, task throughput, and capacity. Use these metrics with the <a href={VARS.WEB_APP_URL}>{VARS.DASHBOARD}</a> to monitor self-hosted compute and cloud runs.
 
 ## Key features
 
 * **Prometheus scrape** — Expose a `/metrics` endpoint for Prometheus to scrape, with optional `PodMonitor` support for the Prometheus Operator.
 * **OTLP push** — Push metrics to any OpenTelemetry-compatible collector (Grafana Alloy, Datadog Agent, New Relic, etc.).
-* **Standard configuration** — Exporter selection uses the standard [OpenTelemetry environment variables](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/), so the worker integrates with your existing observability stack without custom configuration.
-* **Pre-seeded series** — When metrics export is enabled, all metric series appear at startup (before any tasks run), so dashboards and alerts can reference them immediately.
+* **Standard configuration** — Select an exporter with [OpenTelemetry environment variables](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/).
+* **Pre-seeded series** — Metric series appear at startup before any tasks run.
 
 ## How it works
 
@@ -30,7 +26,7 @@ The worker uses the [OpenTelemetry autoexport package](https://github.com/open-t
 * `console` — Writes metrics to stdout (useful for debugging).
 * `none` — Disables metrics export entirely.
 
-The worker initializes autoexport only when `OTEL_METRICS_EXPORTER` names an exporter. Unset, empty, whitespace-only, and `none` values disable metrics initialization.
+When `OTEL_METRICS_EXPORTER` is unset, autoexport uses OTLP. Set it to `none` to disable metrics.
 
 All metrics carry resource attributes (`service.name=oz-agent-worker`, `service.version`, `worker.id`, `worker.backend`) so each worker process shows up as a distinct series in your monitoring system.
 
@@ -141,17 +137,16 @@ All metrics use the `oz_worker_` prefix. Each worker process with metrics enable
 * **`oz_worker_tasks_max_concurrent`** (gauge) — Configured concurrency limit (`0` means unlimited).
 * **`oz_worker_tasks_claimed_total`** (counter) — Total tasks accepted since process start.
 * **`oz_worker_tasks_rejected_total{reason}`** (counter) — Tasks the worker declined (e.g., `reason="at_capacity"`).
-* **`oz_worker_tasks_completed_total{result}`** (counter) — Completed tasks labeled `result="succeeded"` or `result="failed"`.
+* **`oz_worker_tasks_completed_total{result}`** (counter) — Completed tasks labeled `result="succeeded"`, `result="failed"`, `result="cancelled"`, or `result="dispatched"`.
 * **`oz_worker_task_duration_seconds{result}`** (histogram) — Wall-clock task duration on the worker, labeled by result.
+* **`oz_worker_task_failures_total{phase,reason}`** (counter) — Task failures classified by execution phase and a bounded reason such as `image_pull`, `unschedulable`, `container_oom`, or `evicted`.
 * **`oz_worker_websocket_reconnects_total{reason}`** (counter) — WebSocket reconnect attempts (e.g., `reason="dial_failed"`, `reason="remote_close"`). Spikes indicate flapping workers.
 * **`oz_worker_info{version,backend,worker_id}`** (gauge, constant `1`) — Build and runtime metadata. Useful for joining other series by labels.
 
 ---
 
 ## Sample PromQL queries
 
-Direct mappings for common operational questions:
-
 * **Workers available:**
 
   ```promql
@@ -191,6 +186,12 @@ Direct mappings for common operational questions:
   sum(rate(oz_worker_tasks_completed_total{result="failed"}[5m]))
   ```
 
+* **Failure modes:**
+
+  ```promql
+  sum by (phase, reason) (rate(oz_worker_task_failures_total[5m]))
+  ```
+
 * **Reconnect storms (alert threshold):**
 
   ```promql
@@ -201,14 +202,14 @@ Direct mappings for common operational questions:
 
 ## Disabling metrics
 
-Metrics export is disabled when `OTEL_METRICS_EXPORTER` is unset, empty, whitespace-only, or `none`. To override an exporter inherited from the worker's environment, set it to `none` explicitly:
+Set `OTEL_METRICS_EXPORTER=none` to disable metrics export:
 
 ```bash
 export OTEL_METRICS_EXPORTER=none
 oz-agent-worker --api-key "$WARP_API_KEY" --worker-id "my-worker"
 ```
 
-The Helm chart keeps this explicit override when metrics are disabled:
+The Helm chart sets the same override when metrics are disabled:
 
 ```yaml
 metrics: