Skip to content

docs: Improve self-hosted Kubernetes troubleshooting - #770

Draft
warp-agent-staging[bot] wants to merge 1 commit into
mainfrom
docs/self-hosted-kubernetes-troubleshooting
Draft

warp-agent-staging[bot] wants to merge 1 commit into
mainfrom
docs/self-hosted-kubernetes-troubleshooting

Conversation

@warp-agent-staging

Copy link
Copy Markdown
Contributor

What this feature does

Self-hosted Kubernetes workers run each task as a Kubernetes Job. These docs show platform engineers how to protect active task pods, plan capacity, and diagnose startup, scheduling, image, Secret, networking, initialization, and eviction failures. Shipped in oz-agent-worker v2026-09-18-23-15-02 (2026-09-18).

Summary

This PR updates the existing self-hosting documentation with production-backed Kubernetes troubleshooting and lifecycle guidance. It keeps operational procedures on the managed Kubernetes and troubleshooting pages, with focused corrections to the related reference, monitoring, and security pages.

Changes

  • Adds Karpenter disruption protection, PDB limitations, task-pod lifecycle behavior, resource headroom, scheduling, taint, and toleration guidance.
  • Adds a Kubernetes task-failure decision tree for unschedulable pods, OOM kills, image pulls, missing Secrets and volumes, init failures, task-pod egress, deadlines, and eviction.
  • Corrects preflight, RBAC, Job cleanup, connection log, and OpenTelemetry exporter behavior against the current worker implementation.
  • Documents current Kubernetes config fields and the classified task-failure metric.

Content design plan

Audience and JTBD: External platform and DevOps engineers operating self-hosted cloud agent workers on Kubernetes who need to keep active runs stable and diagnose failures from standard Kubernetes signals.

Problem: Existing pages did not explain the production failure modes operators see across pod scheduling, resources, images, initialization, credentials, egress, and node disruption. They also contained stale lifecycle and metrics defaults.

Goals:

  • Help operators map Kubernetes pod states and events to a specific remediation.
  • Explain which pods require disruption protection and why an interrupted task does not resume.
  • Keep configuration defaults and cleanup behavior aligned with the current worker release.

Purpose and value: Operators can diagnose failures without incident-specific context or knowledge of Warp's internal systems.

Content type: Troubleshooting, with targeted feature, reference, monitoring, and security corrections.

Skill and template: draft_troubleshooting / .agents/templates/troubleshooting.md.

High-impact scenarios:

  • Covers worker startup, preflight, capacity, unschedulable pods, CPU and memory, image pulls, Secrets, init containers, networking, Karpenter, PDB limitations, and mid-run eviction.
  • Excludes universal PDB manifests, provider-specific controls other than verified Karpenter behavior, and any claim that a replacement pod resumes task state.

Verification

  • python3 .agents/skills/style_lint/style_lint.py --changed — passed; 5 files scanned, 0 issues.
  • python3 .agents/skills/check_for_broken_links/check_links.py --internal-only — passed; 4,241 internal links checked, 0 broken.
  • npm run build — passed; Astro build, prerendering, Markdown generation, sitemap, and search index completed.
  • python3 .agents/skills/doc_quality_policy/check_compression_contract.py ... — passed for troubleshooting, monitoring, reference, and security pages. The managed Kubernetes page remains above the combined-feature budget at 2,236 words; it was already above budget at 1,791 words on main. This PR removes three callouts and keeps the requested operational guidance on the existing canonical page instead of creating a duplicate page.
  • trunk check / trunk fmt — unavailable because Trunk is not installed in the environment.

Source verification

  • oz-agent-worker@530bbe934a632910dfe8818ab7b4c6027a3473ef: internal/worker/kubernetes.go, internal/worker/worker.go, internal/config/config.go, internal/metrics/metrics.go, charts/oz-agent-worker/values.yaml, and charts/oz-agent-worker/templates/deployment.yaml.
  • warp-server@2916aa67b511e8ef1f556766aa3d2dfe42a42f74: logic/ai/ambient_agents/workers/selfhosted/websocket.go.
  • Karpenter disruption controls.

Unverified claims

None — worker behavior, config fields, defaults, failure classifications, and Kubernetes lifecycle claims were verified against the sources above. Karpenter behavior was verified against its current public documentation.

Documentation risk

Risk: engineering-review-required
Rationale: Changes commands, configuration examples and defaults, RBAC, task lifecycle and disruption, networking, security, and monitoring claims.
Source files consulted: oz-agent-worker@530bbe934a632910dfe8818ab7b4c6027a3473ef (worker, config, metrics, and Helm sources); warp-server@2916aa67b511e8ef1f556766aa3d2dfe42a42f74 (self-hosted worker failure contract); Karpenter disruption documentation
Requested engineering reviewers: none (ownership resolved to a team, not one unambiguous human owner)
Engineering review status: pending
Docs override: none

Co-Authored-By: Oz oz-agent@warp.dev

Co-Authored-By: Oz <oz-agent@warp.dev>
@cla-bot cla-bot Bot added the cla-signed label Sep 19, 2026
@vercel

vercel Bot commented Sep 19, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
docs Ready Ready Preview Sep 19, 2026 6:59pm UTC

Request Review

@warp-agent-staging warp-agent-staging Bot added warpy-factory Opened by the Warp factory agents factory:docs-factory Label associated to the "docs-factory" factory labels Sep 19, 2026
@warp-agent-staging

Copy link
Copy Markdown
Contributor Author

This PR was generated with Warp.

Comment @warp-staging-factory on this PR to send it follow-up work.

View run View conversation View origin

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-signed factory:docs-factory Label associated to the "docs-factory" factory warpy-factory Opened by the Warp factory agents

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants