From 74349f77cdc639987e2d4219a6ddf68232df5b83 Mon Sep 17 00:00:00 2001 From: "warp-agent-staging[bot]" <240773466+warp-agent-staging[bot]@users.noreply.github.com> Date: Sat, 19 Sep 2026 18:51:26 +0000 Subject: [PATCH 1/3] docs: improve Kubernetes worker troubleshooting Co-Authored-By: Oz --- .../self-hosting/managed-kubernetes.mdx | 74 +++++++++++++------ .../docs/platform/self-hosting/monitoring.mdx | 17 +++-- .../docs/platform/self-hosting/reference.mdx | 16 ++-- .../self-hosting/security-and-networking.mdx | 14 +--- .../platform/self-hosting/troubleshooting.mdx | 47 ++++++++---- 5 files changed, 107 insertions(+), 61 deletions(-) diff --git a/src/content/docs/platform/self-hosting/managed-kubernetes.mdx b/src/content/docs/platform/self-hosting/managed-kubernetes.mdx index f13bb13b3..a6c84e86b 100644 --- a/src/content/docs/platform/self-hosting/managed-kubernetes.mdx +++ b/src/content/docs/platform/self-hosting/managed-kubernetes.mdx @@ -8,11 +8,7 @@ sidebar: --- import { VARS } from '@data/vars'; -Deploy the `oz-agent-worker` daemon into a Kubernetes cluster using the included Helm chart. Each agent task runs as a **Kubernetes Job** in your cluster. The {VARS.WARP_AUTOMATION_PLATFORM} orchestrates runs end to end (Slack, Linear, schedules, API, `oz agent run-cloud`); your cluster provides the compute, scheduling, and policy enforcement. - -:::note -This page covers the [managed architecture](/platform/self-hosting/#managed-architecture) with the Kubernetes backend. For the default Docker backend, see [Managed: Docker](/platform/self-hosting/managed-docker/). For host execution without a container runtime, see [Managed: Direct](/platform/self-hosting/managed-direct/). To route runs to a connected worker, see [Routing runs to this worker](/platform/self-hosting/managed-docker/#routing-runs-to-this-worker). -::: +Deploy the `oz-agent-worker` daemon into a Kubernetes cluster using the included Helm chart. Each agent task runs as a Kubernetes Job in your cluster. The {VARS.WARP_AUTOMATION_PLATFORM} orchestrates runs end to end (Slack, Linear, schedules, API, `oz agent run-cloud`); your cluster provides the compute, scheduling, and policy enforcement. ## When to use the Kubernetes backend @@ -25,10 +21,10 @@ This page covers the [managed architecture](/platform/self-hosting/#managed-arch ## How it works 1. The worker connects to the Kubernetes API server (using in-cluster auth by default, or an explicit kubeconfig). -2. On startup, the worker runs a short-lived **preflight Job** to verify that cluster permissions, admission policies, and Pod Security Standards are compatible. If the preflight fails, the worker exits with a diagnostic error before accepting any tasks. +2. On startup, the worker creates a short-lived **preflight Job** with the configured task pod shape. This catches Job creation, admission policy, Pod Security, and some image-volume failures before the worker accepts tasks. 3. For each assigned task, the worker creates a Kubernetes Job in the configured namespace. 4. The worker monitors the Job and Pod status via Kubernetes Watch (with a 30-second safety-net poll for watch disconnects). -5. After the task completes, the Job is cleaned up (unless `--no-cleanup` is set). +5. After the task completes, the worker removes successful Jobs. Failed Jobs remain available for diagnosis for 24 hours by default. --- @@ -36,7 +32,7 @@ This page covers the [managed architecture](/platform/self-hosting/#managed-arch * **Enterprise plan with self-hosting enabled** — [Contact sales](https://www.warp.dev/contact-sales) if self-hosting is not yet enabled for your team. * **A Kubernetes cluster** with the worker process able to reach the API server. The cluster must: - * Allow the worker's namespace to create Jobs with a **root init container** (sidecar materialization depends on this pattern). + * Allow the task namespace to create Jobs with a **root init container**, unless you enable native image volumes with `kubernetesBackend.useImageVolumes=true`. * Grant the worker these namespace-scoped permissions: `create`, `get`, `list`, `watch`, `delete` on `jobs`; `get`, `list`, `watch` on `pods`; `get` on `pods/log`; `list` on `events`. * **[Helm](https://helm.sh/docs/intro/install/)** installed locally, plus `kubectl` authenticated against the target cluster. * **An agent API key** — Create one in the {VARS.WEB_APP} so the worker can authenticate to the {VARS.WARP_AUTOMATION_PLATFORM}. You can bind the key to any cloud agent — that choice doesn't restrict which agents can run on the worker. See [API Keys](/reference/cli/api-keys/) for the full creation flow. @@ -98,7 +94,7 @@ helm install oz-agent-worker ./oz-agent-worker/charts/oz-agent-worker \ Set `image.tag` explicitly to pin the worker image. Check the [oz-agent-worker releases](https://github.com/warpdotdev/oz-agent-worker/releases) for the latest version. Do not rely on `latest`. ::: -**Expected outcome:** `kubectl get pods -n warp-oz` shows the worker Deployment pod as `Running`, and the worker logs show `Connected to Oz` / `Listening for tasks`. +**Expected outcome:** `kubectl get pods -n warp-oz` shows the worker Deployment pod as `Running`, and the worker logs include `Successfully connected to server`. To scale horizontally, deploy multiple Helm releases with distinct worker IDs rather than increasing replicas on a single release. @@ -119,6 +115,7 @@ To scale horizontally, deploy multiple Helm releases with distinct worker IDs ra * `worker.idleOnComplete` — Duration to keep the oz process alive after task completion. * `worker.resources` — Resource requests/limits for the worker Deployment. Defaults to `100m` CPU and `128Mi` memory. * `worker.livenessProbe` — Liveness probe for the worker Deployment. Defaults to an `exec` probe (`kill -0 1`). Override with a custom probe or set to `null` to disable. +* `worker.terminationGracePeriodSeconds` — Grace period for worker Deployment shutdown. Defaults to `30`. * `worker.nodeSelector`, `worker.tolerations`, `worker.affinity` — Scheduling constraints for the worker Deployment pod. **Kubernetes backend:** @@ -126,13 +123,17 @@ To scale horizontally, deploy multiple Helm releases with distinct worker IDs ra * `kubernetesBackend.namespace` — Namespace for task Jobs. Defaults to the release namespace. * `kubernetesBackend.defaultImage` — Default Docker image for task pods when no [Warp environment](/platform/environments/) has been supplied. Leave empty (default) to fall back to `ubuntu:22.04`. * `kubernetesBackend.imagePullPolicy` — Image pull policy for task pods. Defaults to `IfNotPresent`. +* `kubernetesBackend.useImageVolumes` — Use native Kubernetes image volumes instead of root init containers to materialize sidecars. Defaults to `false`. * `kubernetesBackend.preflightImage` — Image for the startup preflight Job. Set this if your cluster restricts allowed registries. +* `kubernetesBackend.preflightResources` — CPU and memory requests and limits for preflight containers. +* `kubernetesBackend.sidecarImage` — Internal-registry override for the Warp agent sidecar image. * `kubernetesBackend.unschedulableTimeout` — How long a pod may remain unschedulable before failing. Defaults to `30s`. * `kubernetesBackend.setupCommand` — Shell command to run before each task. * `kubernetesBackend.teardownCommand` — Shell command to run after each task. * `kubernetesBackend.extraLabels` — Additional labels for task Jobs and Pods. * `kubernetesBackend.extraAnnotations` — Additional annotations for task Jobs and Pods. -* `kubernetesBackend.activeDeadlineSeconds` — Maximum task Job lifetime. +* `kubernetesBackend.activeDeadlineSeconds` — Maximum task Job lifetime. Defaults to eight hours. +* `kubernetesBackend.ttlSecondsAfterFinished` — Retention period for failed Jobs and Jobs orphaned by worker disruption. Defaults to 24 hours when cleanup is enabled. * `kubernetesBackend.workspaceSizeLimit` — Size limit for workspace `emptyDir` volume. * `kubernetesBackend.podTemplate` — Raw PodSpec YAML for task Jobs (same as `backend.kubernetes.pod_template` in the [config file](/platform/self-hosting/reference/#config-file)). @@ -193,23 +194,23 @@ pod_template: effect: "NoSchedule" ``` -:::note The worker Deployment's ServiceAccount is separate from the task Job `serviceAccountName` you configure in `pod_template`. The Deployment ServiceAccount needs RBAC to manage Jobs and Pods. The task ServiceAccount (if any) controls what the agent process can access at runtime. -::: + +When a run specifies an instance shape, the worker sets the `task` container's CPU and memory requests and limits from that shape. Those values override CPU and memory configured on the `task` container in `pod_template`. --- ## Preflight check -On startup, the worker creates a short-lived preflight Job to verify that: +On startup, the worker creates a short-lived preflight Job that uses the configured task PodSpec. The preflight confirms that the worker can create the Job and that the cluster can create the expected pod shape. It surfaces early failures from: -* The worker has sufficient RBAC permissions in the target namespace. -* Cluster admission policies (Pod Security Standards, OPA Gatekeeper, Kyverno, etc.) allow the worker's task pod shape. -* The preflight image can be pulled. +* Insufficient worker RBAC permissions in the target namespace. +* Cluster admission policies such as Pod Security Standards, OPA Gatekeeper, and Kyverno. +* Sidecar-loading requirements, including native image-volume support when enabled. -If the preflight fails, the worker logs a diagnostic error and exits before accepting any tasks. This surfaces policy and configuration issues at deploy time rather than at task execution time. +If the preflight fails, the worker logs a diagnostic error and exits before accepting tasks. A successful preflight does not validate every task image, task Secret, setup command, or network dependency. -The preflight image defaults to `busybox:1.36`. If your cluster restricts allowed registries or images, set `preflight_image` to an allowlisted image. When `imagePullSecrets` is configured in `pod_template`, those secrets apply to the preflight Job as well, so you can point `preflight_image` at an image in your private registry. +The preflight image defaults to `busybox:1.36`. If your cluster restricts allowed registries or images, set `kubernetesBackend.preflightImage` to an allowlisted image. When `imagePullSecrets` is configured in `kubernetesBackend.podTemplate`, those secrets apply to the preflight Job as well, so you can point `kubernetesBackend.preflightImage` at an image in your private registry. --- @@ -222,9 +223,7 @@ There are two ways to pass environment variables to Kubernetes task containers: When configuring the Kubernetes backend via YAML or Helm, declarative task-container env belongs in `pod_template` rather than a separate top-level list. -:::note If your organization uses an external secrets manager (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, etc.), you can inject secrets into task pods via the CSI Secrets Store Driver or a similar operator. Configure the required `volumes`, `volumeMounts`, and annotations in `pod_template` just as you would for any other Kubernetes workload. See your secrets provider's documentation for details. -::: --- @@ -234,6 +233,40 @@ Use `kubernetesBackend.setupCommand` (Helm value) or `backend.kubernetes.setup_c --- +## Protect active task pods from disruption + +The worker Deployment and task Jobs have different disruption requirements. When Kubernetes terminates the worker pod normally, the worker leaves active task Jobs running. Replacing the worker pod does not intentionally stop their task pods. + +The task pod holds the live agent process and an `emptyDir` workspace. Evicting that pod interrupts the run and loses pod-local state. Configure node lifecycle tooling to avoid voluntary disruption of active task pods. + +For Karpenter, add its pod-level disruption annotation to every task Job through the Helm values: + +```yaml title="values.yaml" +kubernetesBackend: + extraAnnotations: + karpenter.sh/do-not-disrupt: "true" +``` + +Task pods exist only for the life of their Jobs. The annotation blocks Karpenter consolidation and, depending on your NodePool configuration, drift while a run is active. It does not prevent forceful expiration, interruption, node repair, or manual deletion. A configured NodePool `terminationGracePeriod` can also make a node eligible for drift and eventually terminate blocking pods. Review [Karpenter's disruption controls](https://karpenter.sh/docs/concepts/disruption/) against your NodePool policy. + +A PodDisruptionBudget only constrains tools that use the Kubernetes Eviction API. It does not preserve the task process or workspace after deletion or protect against involuntary node loss. If your platform requires a PDB, validate its selector and availability rule against concurrent task Jobs. Keep the disruption protection required by your node lifecycle tool. + +For other autoscalers or node lifecycle controllers, use their equivalent protection for active task pods. Do not assume that a replacement pod can resume an interrupted run. + +--- + +## Plan capacity and scheduling + +Task pods need capacity for their configured requests before the worker's unschedulable timeout expires. + +* Set `worker.maxConcurrentTasks` to a finite value based on the task capacity your cluster can provide. The default, `0`, does not cap concurrency. +* Set task CPU and memory requests through a runner instance shape or the `task` container in `kubernetesBackend.podTemplate`. Leave headroom for init containers, node DaemonSets, and workload spikes. +* Set `kubernetesBackend.unschedulableTimeout` longer than the slowest expected node provisioning time. The default is `30s`; set it to `0s` only when you want to disable the fail-fast check. +* Use `worker.nodeSelector`, `worker.tolerations`, and `worker.affinity` for the worker Deployment. Use `kubernetesBackend.podTemplate` for task-pod selectors, tolerations, affinity, and topology constraints. +* A toleration makes a pod eligible for a tainted node; it does not reserve capacity. Pair dedicated-node tolerations with matching selectors or affinity and autoscaler capacity. + +--- + ## Metrics The Helm chart includes built-in support for exporting OpenTelemetry metrics from the worker. Enable metrics by setting `metrics.enabled=true`: @@ -260,7 +293,6 @@ See [Monitoring](/platform/self-hosting/monitoring/) for the full list of Helm v * **Security context** — The Deployment defaults to a non-root security context (`runAsUser: 10001`) with `allowPrivilegeEscalation: false` and all capabilities dropped. * **Liveness probe** — The Deployment includes a default `exec` liveness probe (`kill -0 1`). Override `worker.livenessProbe` for a custom probe, or set it to `null` to disable. * **In-cluster auth** — The chart assumes the worker runs inside the target cluster and uses in-cluster Kubernetes auth by default. -* **Root init containers** — The worker Deployment itself is non-root, but task Jobs require a root init container for sidecar materialization. Ensure the task namespace's Pod Security Standards allow this. --- diff --git a/src/content/docs/platform/self-hosting/monitoring.mdx b/src/content/docs/platform/self-hosting/monitoring.mdx index d9263e2bf..da46a9c68 100644 --- a/src/content/docs/platform/self-hosting/monitoring.mdx +++ b/src/content/docs/platform/self-hosting/monitoring.mdx @@ -8,10 +8,10 @@ description: >- --- import { VARS } from '@data/vars'; -When enabled, the `oz-agent-worker` daemon exports infrastructure-level metrics over [OpenTelemetry](https://opentelemetry.io/), giving your team real-time visibility into worker health, task throughput, and capacity. Combine these metrics with the {VARS.DASHBOARD} for full observability across both the orchestration plane and your self-hosted compute. +The `oz-agent-worker` daemon exports infrastructure-level metrics over [OpenTelemetry](https://opentelemetry.io/), giving your team real-time visibility into worker health, task throughput, and capacity. Combine these metrics with the {VARS.DASHBOARD} for full observability across both the orchestration plane and your self-hosted compute. :::note -Metrics export is opt-in. Set `OTEL_METRICS_EXPORTER` to `prometheus`, `otlp`, or `console` to enable it. When the variable is unset, empty, whitespace-only, or `none`, the worker does not initialize a metrics exporter. The Helm chart also enables export only when `metrics.enabled=true`. +When `OTEL_METRICS_EXPORTER` is unset, the worker uses the OpenTelemetry autoexport default: OTLP. Set the variable to `prometheus`, `otlp`, or `console` to choose an exporter, or to `none` to disable metrics. The Helm chart sets it to `none` when `metrics.enabled=false`. ::: ## Key features @@ -30,7 +30,7 @@ The worker uses the [OpenTelemetry autoexport package](https://github.com/open-t * `console` — Writes metrics to stdout (useful for debugging). * `none` — Disables metrics export entirely. -The worker initializes autoexport only when `OTEL_METRICS_EXPORTER` names an exporter. Unset, empty, whitespace-only, and `none` values disable metrics initialization. +When `OTEL_METRICS_EXPORTER` is unset, autoexport defaults to OTLP and uses the standard OTLP endpoint defaults. Set the variable to `none` to disable metrics initialization. All metrics carry resource attributes (`service.name=oz-agent-worker`, `service.version`, `worker.id`, `worker.backend`) so each worker process shows up as a distinct series in your monitoring system. @@ -141,8 +141,9 @@ All metrics use the `oz_worker_` prefix. Each worker process with metrics enable * **`oz_worker_tasks_max_concurrent`** (gauge) — Configured concurrency limit (`0` means unlimited). * **`oz_worker_tasks_claimed_total`** (counter) — Total tasks accepted since process start. * **`oz_worker_tasks_rejected_total{reason}`** (counter) — Tasks the worker declined (e.g., `reason="at_capacity"`). -* **`oz_worker_tasks_completed_total{result}`** (counter) — Completed tasks labeled `result="succeeded"` or `result="failed"`. +* **`oz_worker_tasks_completed_total{result}`** (counter) — Completed tasks labeled `result="succeeded"`, `result="failed"`, `result="cancelled"`, or `result="dispatched"`. * **`oz_worker_task_duration_seconds{result}`** (histogram) — Wall-clock task duration on the worker, labeled by result. +* **`oz_worker_task_failures_total{phase,reason}`** (counter) — Task failures classified by execution phase and a bounded reason such as `image_pull`, `unschedulable`, `container_oom`, or `evicted`. * **`oz_worker_websocket_reconnects_total{reason}`** (counter) — WebSocket reconnect attempts (e.g., `reason="dial_failed"`, `reason="remote_close"`). Spikes indicate flapping workers. * **`oz_worker_info{version,backend,worker_id}`** (gauge, constant `1`) — Build and runtime metadata. Useful for joining other series by labels. @@ -191,6 +192,12 @@ Direct mappings for common operational questions: sum(rate(oz_worker_tasks_completed_total{result="failed"}[5m])) ``` +* **Failure modes:** + + ```promql + sum by (phase, reason) (rate(oz_worker_task_failures_total[5m])) + ``` + * **Reconnect storms (alert threshold):** ```promql @@ -201,7 +208,7 @@ Direct mappings for common operational questions: ## Disabling metrics -Metrics export is disabled when `OTEL_METRICS_EXPORTER` is unset, empty, whitespace-only, or `none`. To override an exporter inherited from the worker's environment, set it to `none` explicitly: +Set `OTEL_METRICS_EXPORTER=none` to disable metrics export: ```bash export OTEL_METRICS_EXPORTER=none diff --git a/src/content/docs/platform/self-hosting/reference.mdx b/src/content/docs/platform/self-hosting/reference.mdx index c5ced7a2d..9b0d8bc4a 100644 --- a/src/content/docs/platform/self-hosting/reference.mdx +++ b/src/content/docs/platform/self-hosting/reference.mdx @@ -7,10 +7,6 @@ description: >- Reference for the `oz-agent-worker` daemon: CLI flags and the full YAML config-file schema for all three [managed backends](/platform/self-hosting/#managed-architecture) — Docker, Kubernetes, and Direct. For installation instructions, see [Install and run the worker](/platform/self-hosting/managed-docker/#install-and-run-the-worker). -:::note -This page documents every flag and config option. For installation and backend-specific setup walkthroughs, see [Managed: Docker](/platform/self-hosting/managed-docker/), [Managed: Kubernetes](/platform/self-hosting/managed-kubernetes/), or [Managed: Direct](/platform/self-hosting/managed-direct/). This reference applies to the managed architecture only; the [unmanaged architecture](/platform/self-hosting/unmanaged/) uses `oz agent run` instead. -::: - --- ## Worker flags @@ -33,9 +29,7 @@ The following flags are available when starting the worker. * `--max-concurrent-tasks` — Maximum number of tasks to run concurrently. Defaults to `0` (unlimited). When set, additional tasks wait until a slot is available. * `--idle-on-complete` — How long to keep the `oz` process alive after a task's conversation finishes, allowing follow-up interactions via session sharing. Uses duration format (e.g. `45m`, `10m`, `0s`). Defaults to `45m` when not set. Set to `0s` to disable. -:::note Worker IDs starting with `warp` are reserved and cannot be used. The worker refuses to start if `--worker-id` begins with `warp`. -::: ### Example with all flags @@ -151,12 +145,16 @@ backend: * `kubeconfig` — Path to an explicit kubeconfig file. If omitted, the worker uses in-cluster config when running inside Kubernetes, or falls back to the default kubeconfig loading rules. * `default_image` — Default Docker image for task Jobs when the run has no Warp environment image. Precedence: Warp environment image > `default_image` > `ubuntu:22.04`. Set this to skip creating a Warp environment when all your tasks use the same base image. * `image_pull_policy` — One of `Always`, `Never`, or `IfNotPresent`. Defaults to `IfNotPresent`. +* `use_image_volumes` — Use native Kubernetes image volumes instead of root init containers to materialize sidecars. Defaults to `false`. Enable this only on a cluster and container runtime that support image volumes. * `preflight_image` — Image used for the startup preflight Job. Defaults to `busybox:1.36`. Override this if your cluster only allows pulling from an internal or allowlisted registry. +* `preflight_resources` — CPU and memory requests and limits for startup preflight containers. +* `sidecar_image` — Registry override for the Warp agent sidecar image. * `setup_command` — Shell command to run before each task. * `teardown_command` — Shell command to run after each task completes. * `extra_labels` — Map of additional labels to add to task Jobs and Pods. * `extra_annotations` — Map of additional annotations to add to task Jobs and Pods. -* `active_deadline_seconds` — Maximum lifetime for a task Job (Kubernetes `activeDeadlineSeconds`). +* `active_deadline_seconds` — Maximum lifetime for a task Job (Kubernetes `activeDeadlineSeconds`). Defaults to eight hours. +* `ttl_seconds_after_finished` — Retention period for failed Jobs and Jobs orphaned by worker disruption. Defaults to 24 hours when cleanup is enabled. * `workspace_size_limit` — Size limit for the workspace `emptyDir` volume (e.g., `10Gi`). * `unschedulable_timeout` — How long a pod may remain unschedulable before the task is failed early. Defaults to `30s`. Set to `0s` to disable the fail-fast behavior. * `pod_template` — Raw Kubernetes PodSpec YAML merged with the worker's required fields at runtime. Use this to configure task pod scheduling, `serviceAccountName`, `imagePullSecrets`, `nodeSelector`, `tolerations`, resources, and environment variables (including `valueFrom.secretKeyRef` for Kubernetes Secrets). Define a container named `task` to customize the main task container directly; otherwise the worker appends its own. @@ -169,9 +167,7 @@ backend: * `teardown_command` — Shell command to run after each task completes. * `environment` — List of environment variables (same format as the Docker backend). -:::note Only one backend can be configured at a time. Specifying more than one of `docker`, `kubernetes`, and `direct` in the same config file is an error. -::: --- @@ -179,7 +175,7 @@ Only one backend can be configured at a time. Specifying more than one of `docke The worker exports metrics over OpenTelemetry when configured. Exporter selection is controlled by standard environment variables, not CLI flags or config file fields. Set these variables on the worker process (or the worker container via Docker `-e` / Kubernetes `env`). -* `OTEL_METRICS_EXPORTER` — Exporter to use: `prometheus`, `otlp`, or `console`. Unset, empty, whitespace-only, and `none` values disable metrics initialization. +* `OTEL_METRICS_EXPORTER` — Exporter to use: `prometheus`, `otlp`, `console`, or `none`. When unset, the worker uses the OpenTelemetry autoexport default, OTLP. Set it to `none` to disable metrics initialization. * `OTEL_EXPORTER_PROMETHEUS_HOST` — Bind address for the Prometheus exporter. Defaults to `localhost`. Set to `0.0.0.0` when running in Docker or Kubernetes. * `OTEL_EXPORTER_PROMETHEUS_PORT` — Port for the Prometheus exporter. Defaults to `9464`. * `OTEL_EXPORTER_OTLP_ENDPOINT` — OTLP collector endpoint (e.g., `http://otel-collector.observability.svc:4318`). diff --git a/src/content/docs/platform/self-hosting/security-and-networking.mdx b/src/content/docs/platform/self-hosting/security-and-networking.mdx index 8935cab8d..b34518555 100644 --- a/src/content/docs/platform/self-hosting/security-and-networking.mdx +++ b/src/content/docs/platform/self-hosting/security-and-networking.mdx @@ -5,11 +5,7 @@ description: >- cloud agents — including per-backend considerations and BYOLLM. --- -Self-hosting uses a split-plane architecture. Understanding which data stays on your infrastructure and which data routes through Warp is critical for security evaluation. This page summarizes the data model, network egress requirements, and backend-specific security considerations for self-hosted workers. - -:::note -This page applies to both the [managed](/platform/self-hosting/#managed-architecture) and [unmanaged](/platform/self-hosting/unmanaged/) architectures. Backend-specific notes call out Docker-, Kubernetes-, and Direct-only considerations. -::: +Self-hosting uses a split-plane architecture. Repository clones and agent workloads stay on your infrastructure, while orchestration and inference route through Warp. The boundaries below apply to both the [managed](/platform/self-hosting/#managed-architecture) and [unmanaged](/platform/self-hosting/unmanaged/) architectures. ## Data boundaries @@ -47,7 +43,7 @@ Self-hosted agents **do not require any network ingress**. They require outbound * `sessions.app.warp.dev` — port 443 * `oz.warp.dev` — port 443 (managed architecture only) -**Docker Hub** — for pulling task images (managed architecture only). +Docker Hub — for pulling task images (managed architecture only). Tasks use the following Docker images: * [`warpdotdev/warp-agent:latest`](https://hub.docker.com/r/warpdotdev/warp-agent) @@ -62,9 +58,7 @@ Tasks use the following Docker images: **Linux distribution-specific package repositories** — only with the managed architecture, when using a Warp environment whose base image does not have Git pre-installed. The exact repositories depend on the package manager configuration in the environment's base image. -:::note All traffic uses HTTPS (port 443). No inbound ports need to be opened. -::: The exception runs the other way. [Team-managed inference endpoints](/enterprise/enterprise-features/team-managed-keys-and-endpoints/) and URL-backed MCP servers that you host must be reachable from Warp's servers. @@ -80,10 +74,10 @@ The exception runs the other way. [Team-managed inference endpoints](/enterprise ### Kubernetes backend -* **Kubernetes RBAC** — The worker needs namespaced permissions to create, get, list, watch, and delete Jobs and Pods. The Helm chart creates a minimal Role/RoleBinding scoped to a single namespace. The task namespace must allow creating Jobs with a root init container, as sidecar materialization currently depends on that pattern. Review your Pod Security Standards and admission policies accordingly. +* **Kubernetes RBAC** — The worker needs namespaced permissions to create, get, list, watch, and delete Jobs; get, list, and watch Pods; get pod logs; and list events. The Helm chart creates a Role and RoleBinding scoped to one namespace. The task namespace must allow Jobs with a root init container unless native image volumes are enabled. * **Kubernetes service accounts** — The worker Deployment's ServiceAccount (used by the long-lived worker process) is separate from the optional task Job `serviceAccountName` you may configure in `pod_template`. Scope each appropriately. * **API key management** — Store `WARP_API_KEY` in a Kubernetes Secret. Avoid hardcoding it in scripts or config files. If your organization uses an external secrets manager (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, etc.), you can inject secrets into task pods via the CSI Secrets Store Driver or a similar operator — configure the required `volumes`, `volumeMounts`, and annotations in `pod_template`. -* **Task isolation** — Each task runs as a separate Kubernetes Job/Pod. Jobs are removed after execution by default (disable with `--no-cleanup` for debugging). +* **Task isolation** — Each task runs as a separate Kubernetes Job/Pod. Successful Jobs are removed after completion. With cleanup enabled, failed Jobs and Jobs orphaned by worker disruption remain for 24 hours by default; use `--no-cleanup` to keep them without a TTL for debugging. ### Direct backend diff --git a/src/content/docs/platform/self-hosting/troubleshooting.mdx b/src/content/docs/platform/self-hosting/troubleshooting.mdx index 2a53f2b64..d157fa54c 100644 --- a/src/content/docs/platform/self-hosting/troubleshooting.mdx +++ b/src/content/docs/platform/self-hosting/troubleshooting.mdx @@ -30,15 +30,18 @@ The steps below apply to the [managed architecture](/platform/self-hosting/#mana ### Kubernetes backend -**Cause:** The startup preflight Job failed. Common reasons include insufficient RBAC, restrictive Pod Security policies, or an unreachable Kubernetes API server. +**Cause:** The worker Deployment could not start, reach the Kubernetes API, or create its startup preflight Job. Missing Secrets, insufficient RBAC, and admission policies are common causes. **Fix:** -1. Check the worker logs for the preflight diagnostic message. -2. Confirm the worker's namespace has these permissions: `create`, `get`, `list`, `watch`, `delete` on `jobs`; `get`, `list`, `watch` on `pods`; `get` on `pods/log`; `list` on `events`. -3. Confirm the task namespace allows pods with a **root init container** (required for sidecar materialization). -4. If your cluster restricts image sources, set `preflight_image` in the worker config to an allowlisted image (default is `busybox:1.36`). -5. To pull the preflight image from a private registry, configure `imagePullSecrets` in `pod_template` — these secrets also apply to the preflight Job. +1. Run `kubectl describe pod -n NAMESPACE WORKER_POD`. For Helm installs, if the pod reports `CreateContainerConfigError`, confirm the Secret configured by `warp.apiKeySecret.name` exists and contains the key configured by `warp.apiKeySecret.key`. +2. Check the worker logs for Kubernetes API or preflight diagnostics: `kubectl logs -n NAMESPACE WORKER_POD`. +3. Confirm the worker's namespace has these permissions: `create`, `get`, `list`, `watch`, `delete` on `jobs`; `get`, `list`, `watch` on `pods`; `get` on `pods/log`; `list` on `events`. +4. Confirm the task namespace allows pods with a root init container, unless you enabled native image volumes with `kubernetesBackend.useImageVolumes=true`. +5. If your cluster restricts image sources, set `kubernetesBackend.preflightImage` to an allowlisted image. The default is `busybox:1.36`. +6. To pull the preflight image from a private registry, configure `imagePullSecrets` in `kubernetesBackend.podTemplate`. + +A successful preflight confirms that the cluster can create the configured pod shape. It does not validate every task image, task Secret, setup command, or runtime network dependency. ### Direct backend @@ -73,7 +76,7 @@ See [Security and networking](/platform/self-hosting/security-and-networking/#ne **Fix:** -1. Confirm the worker is running and connected. Check the worker logs for `Listening for tasks` or similar. +1. Confirm the worker is running and connected. Check the worker logs for `Successfully connected to server`. 2. Verify the `--host` (or `worker_host`) value you passed matches your `--worker-id` exactly. Case-sensitive. 3. Ensure the worker's team matches the team creating the task. @@ -89,7 +92,7 @@ See [Security and networking](/platform/self-hosting/security-and-networking/#ne 2. For Prometheus scrape mode, confirm the bind address is `0.0.0.0` (not `localhost`) when running in Docker or Kubernetes. `localhost` is only reachable from inside the container. 3. Confirm no firewall or network policy blocks the metrics port (default `9464`). 4. For OTLP push mode, verify `OTEL_EXPORTER_OTLP_ENDPOINT` points to a reachable collector and that the protocol matches (`http/protobuf` vs `grpc`). -5. When using the Helm chart, confirm `metrics.enabled=true` is set. Check that the `Service` and (optionally) `PodMonitor` were created: `kubectl get svc,podmonitor -n `. +5. When using the Helm chart, confirm `metrics.enabled=true` is set. Check that the `Service` and (optionally) `PodMonitor` were created: `kubectl get svc,podmonitor -n NAMESPACE`. 6. If using `metrics.podMonitor.create=true`, verify the `monitoring.coreos.com` CRDs are installed in the cluster. The `PodMonitor` resource requires the Prometheus Operator. 7. Restart the worker with `--log-level debug` and look for metrics-related error messages at startup. @@ -104,7 +107,7 @@ See [Monitoring](/platform/self-hosting/monitoring/) for the full setup guide. **Fix (all backends):** 1. Review task logs in the {VARS.DASHBOARD} or via [session sharing](/agents/local-agents/session-sharing/). -2. Use `--no-cleanup` to keep the container, Job, or workspace around for inspection after failure. +2. Use `--no-cleanup` to keep the container, Job, or workspace indefinitely for inspection. With Kubernetes cleanup enabled, failed Jobs remain for 24 hours by default. 3. Use `--log-level debug` to see detailed execution logs. 4. Ensure the worker machine or cluster has sufficient resources (CPU, memory, disk). @@ -115,12 +118,26 @@ See [Monitoring](/platform/self-hosting/monitoring/) for the full setup guide. ### Kubernetes backend (task failures) -1. Check task Job and Pod status: `kubectl get jobs,pods -n `. -2. Common issues: - * **Unschedulable pods** — Check node selectors, tolerations, and resource requests in `pod_template`. - * **Image pull failures** — Check `imagePullSecrets` in `pod_template`. - * **Admission policy rejections** — Review Pod Security Standards, OPA Gatekeeper, Kyverno, or similar admission controllers. -3. The worker fails a task early if its pod remains unschedulable beyond `unschedulable_timeout` (default `30s`). Raise the timeout or fix the scheduling issue. +Start with the pod phase and recent events: + +```bash +kubectl get jobs,pods -n NAMESPACE +kubectl describe pod -n NAMESPACE TASK_POD +kubectl logs -n NAMESPACE TASK_POD -c CONTAINER_NAME +kubectl logs -n NAMESPACE TASK_POD -c CONTAINER_NAME --previous +``` + +Then match the reported reason to these checks: + +* **`Pending` or `Unschedulable`:** Read the pod's scheduling events. Confirm that at least one eligible node has enough unallocated CPU and memory for the complete pod request. Also check task-pod selectors, affinity, taints and tolerations, topology constraints, resource quotas, and persistent volume requirements. If you depend on a node autoscaler, allow for provisioning latency and keep enough headroom to start task pods before the configured unschedulable timeout. +* **`OOMKilled`:** Increase the memory in the agent profile's instance shape. The selected instance shape sets the task container's CPU and memory requests and limits, overriding those fields for the task container in the pod template. Also confirm that sidecars and init containers have appropriate resources. +* **`ErrImagePull`, `ImagePullBackOff`, or `InvalidImageName`:** Confirm the image exists for the node architecture, the registry is reachable, and the task pod or service account references the required `imagePullSecrets`. A successful startup preflight does not validate a later task image. +* **`CreateContainerConfigError` or `FailedMount`:** Inspect the pod events for the named Secret, ConfigMap, service account, or volume. Confirm the object exists in the task namespace and that every referenced key is present. +* **Init container failure:** Inspect each init container's status and logs. Admission policies must allow the worker's sidecar-materialization init container to run as root unless native image volumes are enabled. Custom init containers must finish successfully before the task containers start. +* **`Evicted`, `DeadlineExceeded`, or exit code `143`:** Check node pressure, lifecycle-tool events, the Job's active deadline, and termination events. A task pod stores its workspace in pod-local ephemeral storage. Replacing an interrupted pod does not resume that run, so protect active task pods from voluntary node disruption. See [Protect active task pods from disruption](/platform/self-hosting/managed-kubernetes/#protect-active-task-pods-from-disruption). +* **The task starts but cannot reach a dependency:** Test DNS resolution, TLS, and the destination from a task pod, not only from the worker pod. Worker connectivity to Warp does not prove that task-pod network policies, service mesh rules, proxies, or egress controls allow the task's own dependencies. + +The worker API key authenticates the worker to Warp. It is not injected as a general-purpose task credential. Provide repository, registry, and application credentials to task pods through your approved Secret integration and pod template. ### Direct backend (task failures) From fb9d5393034c695f61d866a3f7777e4f48b47303 Mon Sep 17 00:00:00 2001 From: "warp-agent-staging[bot]" <240773466+warp-agent-staging[bot]@users.noreply.github.com> Date: Sun, 20 Sep 2026 17:19:57 +0000 Subject: [PATCH 2/3] docs: tighten self-hosted troubleshooting guidance Co-Authored-By: Oz --- .../self-hosting/managed-kubernetes.mdx | 42 ++++++++----------- .../docs/platform/self-hosting/monitoring.mdx | 16 +++---- .../docs/platform/self-hosting/reference.mdx | 10 ++--- .../self-hosting/security-and-networking.mdx | 14 +++---- .../platform/self-hosting/troubleshooting.mdx | 30 ++++++------- 5 files changed, 49 insertions(+), 63 deletions(-) diff --git a/src/content/docs/platform/self-hosting/managed-kubernetes.mdx b/src/content/docs/platform/self-hosting/managed-kubernetes.mdx index a6c84e86b..4903ff0d4 100644 --- a/src/content/docs/platform/self-hosting/managed-kubernetes.mdx +++ b/src/content/docs/platform/self-hosting/managed-kubernetes.mdx @@ -8,7 +8,7 @@ sidebar: --- import { VARS } from '@data/vars'; -Deploy the `oz-agent-worker` daemon into a Kubernetes cluster using the included Helm chart. Each agent task runs as a Kubernetes Job in your cluster. The {VARS.WARP_AUTOMATION_PLATFORM} orchestrates runs end to end (Slack, Linear, schedules, API, `oz agent run-cloud`); your cluster provides the compute, scheduling, and policy enforcement. +Deploy the `oz-agent-worker` daemon into a Kubernetes cluster with the included Helm chart. Each agent task runs as a Kubernetes Job. The {VARS.WARP_AUTOMATION_PLATFORM} orchestrates runs; your cluster handles compute, scheduling, and policy enforcement. ## When to use the Kubernetes backend @@ -21,9 +21,9 @@ Deploy the `oz-agent-worker` daemon into a Kubernetes cluster using the included ## How it works 1. The worker connects to the Kubernetes API server (using in-cluster auth by default, or an explicit kubeconfig). -2. On startup, the worker creates a short-lived **preflight Job** with the configured task pod shape. This catches Job creation, admission policy, Pod Security, and some image-volume failures before the worker accepts tasks. +2. On startup, the worker runs a preflight Job with the configured task pod shape. 3. For each assigned task, the worker creates a Kubernetes Job in the configured namespace. -4. The worker monitors the Job and Pod status via Kubernetes Watch (with a 30-second safety-net poll for watch disconnects). +4. The worker monitors the Job and Pod status. 5. After the task completes, the worker removes successful Jobs. Failed Jobs remain available for diagnosis for 24 hours by default. --- @@ -35,7 +35,7 @@ Deploy the `oz-agent-worker` daemon into a Kubernetes cluster using the included * Allow the task namespace to create Jobs with a **root init container**, unless you enable native image volumes with `kubernetesBackend.useImageVolumes=true`. * Grant the worker these namespace-scoped permissions: `create`, `get`, `list`, `watch`, `delete` on `jobs`; `get`, `list`, `watch` on `pods`; `get` on `pods/log`; `list` on `events`. * **[Helm](https://helm.sh/docs/intro/install/)** installed locally, plus `kubectl` authenticated against the target cluster. -* **An agent API key** — Create one in the {VARS.WEB_APP} so the worker can authenticate to the {VARS.WARP_AUTOMATION_PLATFORM}. You can bind the key to any cloud agent — that choice doesn't restrict which agents can run on the worker. See [API Keys](/reference/cli/api-keys/) for the full creation flow. +* **An agent API key** — Create one in the {VARS.WEB_APP} so the worker can authenticate to the {VARS.WARP_AUTOMATION_PLATFORM}. Binding the key to a cloud agent doesn't restrict which agents can run on the worker. See [API Keys](/reference/cli/api-keys/). --- @@ -194,23 +194,17 @@ pod_template: effect: "NoSchedule" ``` -The worker Deployment's ServiceAccount is separate from the task Job `serviceAccountName` you configure in `pod_template`. The Deployment ServiceAccount needs RBAC to manage Jobs and Pods. The task ServiceAccount (if any) controls what the agent process can access at runtime. +The worker Deployment ServiceAccount needs RBAC to manage Jobs and Pods. The task Job `serviceAccountName` in `pod_template` controls the agent process's runtime access. -When a run specifies an instance shape, the worker sets the `task` container's CPU and memory requests and limits from that shape. Those values override CPU and memory configured on the `task` container in `pod_template`. +A run's instance shape overrides the `task` container's CPU and memory values in `pod_template`. --- ## Preflight check -On startup, the worker creates a short-lived preflight Job that uses the configured task PodSpec. The preflight confirms that the worker can create the Job and that the cluster can create the expected pod shape. It surfaces early failures from: +On startup, the worker runs a preflight Job with the configured task PodSpec. It catches insufficient RBAC, admission-policy failures, and sidecar-loading problems. If preflight fails, the worker exits before accepting tasks. A successful preflight does not validate task-specific images, Secrets, setup commands, or network access. -* Insufficient worker RBAC permissions in the target namespace. -* Cluster admission policies such as Pod Security Standards, OPA Gatekeeper, and Kyverno. -* Sidecar-loading requirements, including native image-volume support when enabled. - -If the preflight fails, the worker logs a diagnostic error and exits before accepting tasks. A successful preflight does not validate every task image, task Secret, setup command, or network dependency. - -The preflight image defaults to `busybox:1.36`. If your cluster restricts allowed registries or images, set `kubernetesBackend.preflightImage` to an allowlisted image. When `imagePullSecrets` is configured in `kubernetesBackend.podTemplate`, those secrets apply to the preflight Job as well, so you can point `kubernetesBackend.preflightImage` at an image in your private registry. +The preflight image defaults to `busybox:1.36`. For an allowlisted or private image, set `kubernetesBackend.preflightImage`. The `imagePullSecrets` in `kubernetesBackend.podTemplate` also apply to preflight. --- @@ -223,7 +217,7 @@ There are two ways to pass environment variables to Kubernetes task containers: When configuring the Kubernetes backend via YAML or Helm, declarative task-container env belongs in `pod_template` rather than a separate top-level list. -If your organization uses an external secrets manager (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, etc.), you can inject secrets into task pods via the CSI Secrets Store Driver or a similar operator. Configure the required `volumes`, `volumeMounts`, and annotations in `pod_template` just as you would for any other Kubernetes workload. See your secrets provider's documentation for details. +For an external secrets manager, inject task secrets through a CSI driver or operator. Add the provider's `volumes`, `volumeMounts`, and annotations to `pod_template`. --- @@ -235,9 +229,7 @@ Use `kubernetesBackend.setupCommand` (Helm value) or `backend.kubernetes.setup_c ## Protect active task pods from disruption -The worker Deployment and task Jobs have different disruption requirements. When Kubernetes terminates the worker pod normally, the worker leaves active task Jobs running. Replacing the worker pod does not intentionally stop their task pods. - -The task pod holds the live agent process and an `emptyDir` workspace. Evicting that pod interrupts the run and loses pod-local state. Configure node lifecycle tooling to avoid voluntary disruption of active task pods. +Terminating the worker pod normally leaves active task Jobs running. Evicting a task pod interrupts the run and deletes its pod-local `emptyDir` workspace. Configure node lifecycle tooling to avoid voluntary disruption of active task pods. For Karpenter, add its pod-level disruption annotation to every task Job through the Helm values: @@ -247,11 +239,11 @@ kubernetesBackend: karpenter.sh/do-not-disrupt: "true" ``` -Task pods exist only for the life of their Jobs. The annotation blocks Karpenter consolidation and, depending on your NodePool configuration, drift while a run is active. It does not prevent forceful expiration, interruption, node repair, or manual deletion. A configured NodePool `terminationGracePeriod` can also make a node eligible for drift and eventually terminate blocking pods. Review [Karpenter's disruption controls](https://karpenter.sh/docs/concepts/disruption/) against your NodePool policy. +The annotation blocks Karpenter consolidation. It blocks drift only when the NodePool omits `terminationGracePeriod`. Expiration, interruption, node repair, and manual deletion can still terminate the node. With `terminationGracePeriod`, Karpenter can terminate blocking pods when the period ends. Review [Karpenter's pod-level disruption controls](https://karpenter.sh/docs/concepts/disruption/#pod-level-controls). -A PodDisruptionBudget only constrains tools that use the Kubernetes Eviction API. It does not preserve the task process or workspace after deletion or protect against involuntary node loss. If your platform requires a PDB, validate its selector and availability rule against concurrent task Jobs. Keep the disruption protection required by your node lifecycle tool. +A PodDisruptionBudget (PDB) constrains tools that use the Kubernetes Eviction API. It protects a group of pods, not an individual task's process or workspace. Direct deletion, kubelet pressure eviction, node failure, and controllers that bypass the Eviction API can still terminate a task. -For other autoscalers or node lifecycle controllers, use their equivalent protection for active task pods. Do not assume that a replacement pod can resume an interrupted run. +For other node lifecycle tools, use the equivalent protection and verify which disruption paths bypass it. A replacement pod cannot resume an interrupted run. --- @@ -259,10 +251,10 @@ For other autoscalers or node lifecycle controllers, use their equivalent protec Task pods need capacity for their configured requests before the worker's unschedulable timeout expires. -* Set `worker.maxConcurrentTasks` to a finite value based on the task capacity your cluster can provide. The default, `0`, does not cap concurrency. -* Set task CPU and memory requests through a runner instance shape or the `task` container in `kubernetesBackend.podTemplate`. Leave headroom for init containers, node DaemonSets, and workload spikes. -* Set `kubernetesBackend.unschedulableTimeout` longer than the slowest expected node provisioning time. The default is `30s`; set it to `0s` only when you want to disable the fail-fast check. -* Use `worker.nodeSelector`, `worker.tolerations`, and `worker.affinity` for the worker Deployment. Use `kubernetesBackend.podTemplate` for task-pod selectors, tolerations, affinity, and topology constraints. +* Set `worker.maxConcurrentTasks` to a finite value that matches cluster capacity. The default, `0`, does not cap concurrency. +* Set task CPU and memory through a runner instance shape or the `task` container in `kubernetesBackend.podTemplate`. Reserve capacity for init containers, DaemonSets, and workload spikes. +* Set `kubernetesBackend.unschedulableTimeout` longer than the slowest expected node provisioning time. The default is `30s`; `0s` disables the check. +* Use `worker.nodeSelector`, `worker.tolerations`, and `worker.affinity` for the worker Deployment. Use the equivalent fields in `kubernetesBackend.podTemplate` for task pods. * A toleration makes a pod eligible for a tainted node; it does not reserve capacity. Pair dedicated-node tolerations with matching selectors or affinity and autoscaler capacity. --- diff --git a/src/content/docs/platform/self-hosting/monitoring.mdx b/src/content/docs/platform/self-hosting/monitoring.mdx index da46a9c68..1f9cd60cf 100644 --- a/src/content/docs/platform/self-hosting/monitoring.mdx +++ b/src/content/docs/platform/self-hosting/monitoring.mdx @@ -8,18 +8,14 @@ description: >- --- import { VARS } from '@data/vars'; -The `oz-agent-worker` daemon exports infrastructure-level metrics over [OpenTelemetry](https://opentelemetry.io/), giving your team real-time visibility into worker health, task throughput, and capacity. Combine these metrics with the {VARS.DASHBOARD} for full observability across both the orchestration plane and your self-hosted compute. - -:::note -When `OTEL_METRICS_EXPORTER` is unset, the worker uses the OpenTelemetry autoexport default: OTLP. Set the variable to `prometheus`, `otlp`, or `console` to choose an exporter, or to `none` to disable metrics. The Helm chart sets it to `none` when `metrics.enabled=false`. -::: +The `oz-agent-worker` daemon exports [OpenTelemetry](https://opentelemetry.io/) metrics for worker health, task throughput, and capacity. Use these metrics with the {VARS.DASHBOARD} to monitor self-hosted compute and cloud runs. ## Key features * **Prometheus scrape** — Expose a `/metrics` endpoint for Prometheus to scrape, with optional `PodMonitor` support for the Prometheus Operator. * **OTLP push** — Push metrics to any OpenTelemetry-compatible collector (Grafana Alloy, Datadog Agent, New Relic, etc.). -* **Standard configuration** — Exporter selection uses the standard [OpenTelemetry environment variables](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/), so the worker integrates with your existing observability stack without custom configuration. -* **Pre-seeded series** — When metrics export is enabled, all metric series appear at startup (before any tasks run), so dashboards and alerts can reference them immediately. +* **Standard configuration** — Select an exporter with [OpenTelemetry environment variables](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/). +* **Pre-seeded series** — Metric series appear at startup before any tasks run. ## How it works @@ -30,7 +26,7 @@ The worker uses the [OpenTelemetry autoexport package](https://github.com/open-t * `console` — Writes metrics to stdout (useful for debugging). * `none` — Disables metrics export entirely. -When `OTEL_METRICS_EXPORTER` is unset, autoexport defaults to OTLP and uses the standard OTLP endpoint defaults. Set the variable to `none` to disable metrics initialization. +When `OTEL_METRICS_EXPORTER` is unset, autoexport uses OTLP. Set it to `none` to disable metrics. All metrics carry resource attributes (`service.name=oz-agent-worker`, `service.version`, `worker.id`, `worker.backend`) so each worker process shows up as a distinct series in your monitoring system. @@ -151,8 +147,6 @@ All metrics use the `oz_worker_` prefix. Each worker process with metrics enable ## Sample PromQL queries -Direct mappings for common operational questions: - * **Workers available:** ```promql @@ -215,7 +209,7 @@ export OTEL_METRICS_EXPORTER=none oz-agent-worker --api-key "$WARP_API_KEY" --worker-id "my-worker" ``` -The Helm chart keeps this explicit override when metrics are disabled: +The Helm chart sets the same override when metrics are disabled: ```yaml metrics: diff --git a/src/content/docs/platform/self-hosting/reference.mdx b/src/content/docs/platform/self-hosting/reference.mdx index 9b0d8bc4a..e9ec4baa4 100644 --- a/src/content/docs/platform/self-hosting/reference.mdx +++ b/src/content/docs/platform/self-hosting/reference.mdx @@ -5,7 +5,7 @@ description: >- file schema for the Docker, Kubernetes, and Direct backends. --- -Reference for the `oz-agent-worker` daemon: CLI flags and the full YAML config-file schema for all three [managed backends](/platform/self-hosting/#managed-architecture) — Docker, Kubernetes, and Direct. For installation instructions, see [Install and run the worker](/platform/self-hosting/managed-docker/#install-and-run-the-worker). +Look up CLI flags and YAML config fields for the `oz-agent-worker` daemon's [managed backends](/platform/self-hosting/#managed-architecture). For installation, see [Install and run the worker](/platform/self-hosting/managed-docker/#install-and-run-the-worker). --- @@ -145,8 +145,8 @@ backend: * `kubeconfig` — Path to an explicit kubeconfig file. If omitted, the worker uses in-cluster config when running inside Kubernetes, or falls back to the default kubeconfig loading rules. * `default_image` — Default Docker image for task Jobs when the run has no Warp environment image. Precedence: Warp environment image > `default_image` > `ubuntu:22.04`. Set this to skip creating a Warp environment when all your tasks use the same base image. * `image_pull_policy` — One of `Always`, `Never`, or `IfNotPresent`. Defaults to `IfNotPresent`. -* `use_image_volumes` — Use native Kubernetes image volumes instead of root init containers to materialize sidecars. Defaults to `false`. Enable this only on a cluster and container runtime that support image volumes. -* `preflight_image` — Image used for the startup preflight Job. Defaults to `busybox:1.36`. Override this if your cluster only allows pulling from an internal or allowlisted registry. +* `use_image_volumes` — Use native image volumes instead of root init containers. Defaults to `false` and requires cluster and runtime support. +* `preflight_image` — Startup preflight Job image. Defaults to `busybox:1.36`. * `preflight_resources` — CPU and memory requests and limits for startup preflight containers. * `sidecar_image` — Registry override for the Warp agent sidecar image. * `setup_command` — Shell command to run before each task. @@ -154,7 +154,7 @@ backend: * `extra_labels` — Map of additional labels to add to task Jobs and Pods. * `extra_annotations` — Map of additional annotations to add to task Jobs and Pods. * `active_deadline_seconds` — Maximum lifetime for a task Job (Kubernetes `activeDeadlineSeconds`). Defaults to eight hours. -* `ttl_seconds_after_finished` — Retention period for failed Jobs and Jobs orphaned by worker disruption. Defaults to 24 hours when cleanup is enabled. +* `ttl_seconds_after_finished` — Retention period for failed or orphaned Jobs. Defaults to 24 hours when cleanup is enabled. * `workspace_size_limit` — Size limit for the workspace `emptyDir` volume (e.g., `10Gi`). * `unschedulable_timeout` — How long a pod may remain unschedulable before the task is failed early. Defaults to `30s`. Set to `0s` to disable the fail-fast behavior. * `pod_template` — Raw Kubernetes PodSpec YAML merged with the worker's required fields at runtime. Use this to configure task pod scheduling, `serviceAccountName`, `imagePullSecrets`, `nodeSelector`, `tolerations`, resources, and environment variables (including `valueFrom.secretKeyRef` for Kubernetes Secrets). Define a container named `task` to customize the main task container directly; otherwise the worker appends its own. @@ -173,7 +173,7 @@ Only one backend can be configured at a time. Specifying more than one of `docke ## Metrics configuration -The worker exports metrics over OpenTelemetry when configured. Exporter selection is controlled by standard environment variables, not CLI flags or config file fields. Set these variables on the worker process (or the worker container via Docker `-e` / Kubernetes `env`). +Configure OpenTelemetry metrics with environment variables on the worker process or container: * `OTEL_METRICS_EXPORTER` — Exporter to use: `prometheus`, `otlp`, `console`, or `none`. When unset, the worker uses the OpenTelemetry autoexport default, OTLP. Set it to `none` to disable metrics initialization. * `OTEL_EXPORTER_PROMETHEUS_HOST` — Bind address for the Prometheus exporter. Defaults to `localhost`. Set to `0.0.0.0` when running in Docker or Kubernetes. diff --git a/src/content/docs/platform/self-hosting/security-and-networking.mdx b/src/content/docs/platform/self-hosting/security-and-networking.mdx index b34518555..023c85f47 100644 --- a/src/content/docs/platform/self-hosting/security-and-networking.mdx +++ b/src/content/docs/platform/self-hosting/security-and-networking.mdx @@ -5,7 +5,7 @@ description: >- cloud agents — including per-backend considerations and BYOLLM. --- -Self-hosting uses a split-plane architecture. Repository clones and agent workloads stay on your infrastructure, while orchestration and inference route through Warp. The boundaries below apply to both the [managed](/platform/self-hosting/#managed-architecture) and [unmanaged](/platform/self-hosting/unmanaged/) architectures. +Repository clones and agent workloads stay on your infrastructure, while orchestration and inference route through Warp. These boundaries apply to [managed](/platform/self-hosting/#managed-architecture) and [unmanaged](/platform/self-hosting/unmanaged/) deployments. ## Data boundaries @@ -60,7 +60,7 @@ Tasks use the following Docker images: All traffic uses HTTPS (port 443). No inbound ports need to be opened. -The exception runs the other way. [Team-managed inference endpoints](/enterprise/enterprise-features/team-managed-keys-and-endpoints/) and URL-backed MCP servers that you host must be reachable from Warp's servers. +[Team-managed inference endpoints](/enterprise/enterprise-features/team-managed-keys-and-endpoints/) and URL-backed MCP servers that you host must be reachable from Warp's servers. --- @@ -74,10 +74,10 @@ The exception runs the other way. [Team-managed inference endpoints](/enterprise ### Kubernetes backend -* **Kubernetes RBAC** — The worker needs namespaced permissions to create, get, list, watch, and delete Jobs; get, list, and watch Pods; get pod logs; and list events. The Helm chart creates a Role and RoleBinding scoped to one namespace. The task namespace must allow Jobs with a root init container unless native image volumes are enabled. -* **Kubernetes service accounts** — The worker Deployment's ServiceAccount (used by the long-lived worker process) is separate from the optional task Job `serviceAccountName` you may configure in `pod_template`. Scope each appropriately. -* **API key management** — Store `WARP_API_KEY` in a Kubernetes Secret. Avoid hardcoding it in scripts or config files. If your organization uses an external secrets manager (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, etc.), you can inject secrets into task pods via the CSI Secrets Store Driver or a similar operator — configure the required `volumes`, `volumeMounts`, and annotations in `pod_template`. -* **Task isolation** — Each task runs as a separate Kubernetes Job/Pod. Successful Jobs are removed after completion. With cleanup enabled, failed Jobs and Jobs orphaned by worker disruption remain for 24 hours by default; use `--no-cleanup` to keep them without a TTL for debugging. +* **Kubernetes RBAC** — The worker uses namespace-scoped permissions to manage Jobs and read Pods, logs, and events. The task namespace must allow a root init container unless native image volumes are enabled. +* **Kubernetes service accounts** — The worker Deployment ServiceAccount is separate from the task Job `serviceAccountName` in `pod_template`. Scope each to its workload. +* **API key management** — Store `WARP_API_KEY` in a Kubernetes Secret and inject it only into the worker. Add task credentials through `pod_template` or an external secrets operator. +* **Task isolation** — Each task runs as a Kubernetes Job and Pod. Successful Jobs are removed. With cleanup enabled, failed or orphaned Jobs remain for 24 hours; `--no-cleanup` keeps them without a TTL. ### Direct backend @@ -94,7 +94,7 @@ The exception runs the other way. [Team-managed inference endpoints](/enterprise ## VPN and on-premises access -Since self-hosted agents run on your infrastructure, they inherit your network access. Self-hosted agents can reach services behind VPNs, self-hosted GitLab/Bitbucket instances, databases, and any other internal resources your host can reach. This is one of the primary reasons teams choose self-hosting. +Self-hosted agents inherit the host's network access, including VPN services, private source control, and databases. See [GitLab](/platform/integrations/gitlab/) and [Bitbucket](/platform/integrations/bitbucket/) setup guides for SCM integration details. diff --git a/src/content/docs/platform/self-hosting/troubleshooting.mdx b/src/content/docs/platform/self-hosting/troubleshooting.mdx index d157fa54c..5c640bad1 100644 --- a/src/content/docs/platform/self-hosting/troubleshooting.mdx +++ b/src/content/docs/platform/self-hosting/troubleshooting.mdx @@ -8,7 +8,7 @@ sidebar: --- import { VARS } from '@data/vars'; -Diagnostic guides for the `oz-agent-worker` daemon and its task execution. Use this page when a worker won't start, won't connect, tasks stay queued, or tasks fail. +Use these checks when the `oz-agent-worker` daemon won't start or connect, tasks stay queued, or tasks fail. :::note The steps below apply to the [managed architecture](/platform/self-hosting/#managed-architecture) (`oz-agent-worker` daemon). For [unmanaged](/platform/self-hosting/unmanaged/) deployments, refer to the documentation for the environment running `oz agent run` (e.g., GitHub Actions, Kubernetes). @@ -30,18 +30,18 @@ The steps below apply to the [managed architecture](/platform/self-hosting/#mana ### Kubernetes backend -**Cause:** The worker Deployment could not start, reach the Kubernetes API, or create its startup preflight Job. Missing Secrets, insufficient RBAC, and admission policies are common causes. +**Cause:** The worker Deployment couldn't start, reach the Kubernetes API, or create its preflight Job. **Fix:** -1. Run `kubectl describe pod -n NAMESPACE WORKER_POD`. For Helm installs, if the pod reports `CreateContainerConfigError`, confirm the Secret configured by `warp.apiKeySecret.name` exists and contains the key configured by `warp.apiKeySecret.key`. +1. Run `kubectl describe pod -n NAMESPACE WORKER_POD`. Replace `NAMESPACE` with the chart namespace and `WORKER_POD` with the worker pod name. For `CreateContainerConfigError`, verify the Secret and key configured by `warp.apiKeySecret`. 2. Check the worker logs for Kubernetes API or preflight diagnostics: `kubectl logs -n NAMESPACE WORKER_POD`. 3. Confirm the worker's namespace has these permissions: `create`, `get`, `list`, `watch`, `delete` on `jobs`; `get`, `list`, `watch` on `pods`; `get` on `pods/log`; `list` on `events`. 4. Confirm the task namespace allows pods with a root init container, unless you enabled native image volumes with `kubernetesBackend.useImageVolumes=true`. 5. If your cluster restricts image sources, set `kubernetesBackend.preflightImage` to an allowlisted image. The default is `busybox:1.36`. 6. To pull the preflight image from a private registry, configure `imagePullSecrets` in `kubernetesBackend.podTemplate`. -A successful preflight confirms that the cluster can create the configured pod shape. It does not validate every task image, task Secret, setup command, or runtime network dependency. +A successful preflight validates the configured pod shape, not task-specific images, Secrets, setup commands, or network access. ### Direct backend @@ -92,7 +92,7 @@ See [Security and networking](/platform/self-hosting/security-and-networking/#ne 2. For Prometheus scrape mode, confirm the bind address is `0.0.0.0` (not `localhost`) when running in Docker or Kubernetes. `localhost` is only reachable from inside the container. 3. Confirm no firewall or network policy blocks the metrics port (default `9464`). 4. For OTLP push mode, verify `OTEL_EXPORTER_OTLP_ENDPOINT` points to a reachable collector and that the protocol matches (`http/protobuf` vs `grpc`). -5. When using the Helm chart, confirm `metrics.enabled=true` is set. Check that the `Service` and (optionally) `PodMonitor` were created: `kubectl get svc,podmonitor -n NAMESPACE`. +5. When using the Helm chart, confirm `metrics.enabled=true` is set. Check that the `Service` and optional `PodMonitor` were created: `kubectl get svc,podmonitor -n NAMESPACE`. Replace `NAMESPACE` with the chart namespace. 6. If using `metrics.podMonitor.create=true`, verify the `monitoring.coreos.com` CRDs are installed in the cluster. The `PodMonitor` resource requires the Prometheus Operator. 7. Restart the worker with `--log-level debug` and look for metrics-related error messages at startup. @@ -102,7 +102,7 @@ See [Monitoring](/platform/self-hosting/monitoring/) for the full setup guide. ## Task failures -**Cause:** A variety of reasons depending on backend. Start with the diagnostic steps common to all backends, then follow the backend-specific checks. +**Cause:** The backend logs identify why the task failed. **Fix (all backends):** @@ -127,17 +127,17 @@ kubectl logs -n NAMESPACE TASK_POD -c CONTAINER_NAME kubectl logs -n NAMESPACE TASK_POD -c CONTAINER_NAME --previous ``` -Then match the reported reason to these checks: +Replace `NAMESPACE` with the task namespace, `TASK_POD` with the task pod name, and `CONTAINER_NAME` with the failed container name. Match the reported reason: -* **`Pending` or `Unschedulable`:** Read the pod's scheduling events. Confirm that at least one eligible node has enough unallocated CPU and memory for the complete pod request. Also check task-pod selectors, affinity, taints and tolerations, topology constraints, resource quotas, and persistent volume requirements. If you depend on a node autoscaler, allow for provisioning latency and keep enough headroom to start task pods before the configured unschedulable timeout. -* **`OOMKilled`:** Increase the memory in the agent profile's instance shape. The selected instance shape sets the task container's CPU and memory requests and limits, overriding those fields for the task container in the pod template. Also confirm that sidecars and init containers have appropriate resources. -* **`ErrImagePull`, `ImagePullBackOff`, or `InvalidImageName`:** Confirm the image exists for the node architecture, the registry is reachable, and the task pod or service account references the required `imagePullSecrets`. A successful startup preflight does not validate a later task image. -* **`CreateContainerConfigError` or `FailedMount`:** Inspect the pod events for the named Secret, ConfigMap, service account, or volume. Confirm the object exists in the task namespace and that every referenced key is present. -* **Init container failure:** Inspect each init container's status and logs. Admission policies must allow the worker's sidecar-materialization init container to run as root unless native image volumes are enabled. Custom init containers must finish successfully before the task containers start. -* **`Evicted`, `DeadlineExceeded`, or exit code `143`:** Check node pressure, lifecycle-tool events, the Job's active deadline, and termination events. A task pod stores its workspace in pod-local ephemeral storage. Replacing an interrupted pod does not resume that run, so protect active task pods from voluntary node disruption. See [Protect active task pods from disruption](/platform/self-hosting/managed-kubernetes/#protect-active-task-pods-from-disruption). -* **The task starts but cannot reach a dependency:** Test DNS resolution, TLS, and the destination from a task pod, not only from the worker pod. Worker connectivity to Warp does not prove that task-pod network policies, service mesh rules, proxies, or egress controls allow the task's own dependencies. +* **`Pending` or `Unschedulable`:** Read the scheduling events. Check free CPU and memory, selectors and affinity, taints and tolerations, topology constraints, quotas, and volume binding. Allow enough time for node provisioning and reserve capacity for init containers, DaemonSets, and spikes. +* **`OOMKilled`:** Increase memory in the agent profile's instance shape. The instance shape overrides the task container's CPU and memory values in the pod template. Check sidecar and init-container resources separately. +* **`ErrImagePull`, `ImagePullBackOff`, or `InvalidImageName`:** Follow [Image pull failures](#image-pull-failures). Preflight does not pull every task image. +* **`CreateContainerConfigError` or `FailedMount`:** Events name the missing Secret, ConfigMap, service account, key, or volume. Verify that it exists in the task namespace. +* **Init container failure:** Check each init container's status and logs. The Warp sidecar-loading init container runs as root unless native image volumes are enabled. Custom init containers must finish before the task starts. +* **`Evicted`, `DeadlineExceeded`, or exit code `143`:** Check node pressure, lifecycle-tool events, the Job deadline, and termination events. A replacement pod cannot recover the task's `emptyDir` workspace. See [protecting active task pods from voluntary disruption](/platform/self-hosting/managed-kubernetes/#protect-active-task-pods-from-disruption). +* **The task cannot reach a dependency:** Test DNS, TLS, and the destination from the task pod. Worker connectivity to Warp does not test the task's network policies, service mesh, proxy, or dependency egress. -The worker API key authenticates the worker to Warp. It is not injected as a general-purpose task credential. Provide repository, registry, and application credentials to task pods through your approved Secret integration and pod template. +The worker API key authenticates the worker to Warp; it is not a task credential. Provide task credentials through your Secret integration and pod template. ### Direct backend (task failures) From e568ac4251d1411d25271edf0724fb4701a03d96 Mon Sep 17 00:00:00 2001 From: "warp-agent-staging[bot]" <240773466+warp-agent-staging[bot]@users.noreply.github.com> Date: Sun, 20 Sep 2026 17:29:25 +0000 Subject: [PATCH 3/3] docs: correct Kubernetes deadline default Co-Authored-By: Oz --- src/content/docs/platform/self-hosting/reference.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/content/docs/platform/self-hosting/reference.mdx b/src/content/docs/platform/self-hosting/reference.mdx index e9ec4baa4..ea8258c75 100644 --- a/src/content/docs/platform/self-hosting/reference.mdx +++ b/src/content/docs/platform/self-hosting/reference.mdx @@ -153,7 +153,7 @@ backend: * `teardown_command` — Shell command to run after each task completes. * `extra_labels` — Map of additional labels to add to task Jobs and Pods. * `extra_annotations` — Map of additional annotations to add to task Jobs and Pods. -* `active_deadline_seconds` — Maximum lifetime for a task Job (Kubernetes `activeDeadlineSeconds`). Defaults to eight hours. +* `active_deadline_seconds` — Maximum task Job lifetime (Kubernetes `activeDeadlineSeconds`). The worker has no default; the Helm chart defaults `kubernetesBackend.activeDeadlineSeconds` to eight hours. * `ttl_seconds_after_finished` — Retention period for failed or orphaned Jobs. Defaults to 24 hours when cleanup is enabled. * `workspace_size_limit` — Size limit for the workspace `emptyDir` volume (e.g., `10Gi`). * `unschedulable_timeout` — How long a pod may remain unschedulable before the task is failed early. Defaults to `30s`. Set to `0s` to disable the fail-fast behavior.