Skip to content

docs: clarify Kubernetes OOM and scheduling failures - #771

Draft
warp-agent-staging[bot] wants to merge 6 commits into
docs/self-hosted-kubernetes-troubleshootingfrom
factory/k8s-resource-troubleshooting
Draft

warp-agent-staging[bot] wants to merge 6 commits into
docs/self-hosted-kubernetes-troubleshootingfrom
factory/k8s-resource-troubleshooting

Conversation

@warp-agent-staging

@warp-agent-staging warp-agent-staging Bot commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Summary

Clarifies how operators diagnose and remediate self-hosted Kubernetes task failures without confusing worker-daemon resources, task-container OOMs, and scheduler capacity failures. Closes #769.

Dependency

This PR is stacked on #770 (docs/self-hosted-kubernetes-troubleshooting at e568ac4251d1411d25271edf0724fb4701a03d96, based on main). Its GitHub base is that branch, and the #770 head is an ancestor of this branch. Merge #770 before this PR; restack if #770's head changes.

Changes

platform/self-hosting/troubleshooting.mdx

  • Replaces a dense Kubernetes failure list with short symptom, verify, and fix paths for FailedScheduling, OOMKilled, eviction, and exit code 143.
  • Preserves the distinction between SIGTERM and an OOM report, and between failures before scheduling and failures in a running container.
  • Reduces metrics troubleshooting from seven steps to five and removes detail that operators do not need to choose the next action.

platform/self-hosting/managed-kubernetes.mdx

  • Distinguishes the long-running worker Deployment's resources from task Job resources.
  • Clarifies the chart's daemon resource defaults and that it sets no daemon limits.
  • Documents task sizing with pod_template and workload-specific runner instance shapes, including task-only shape precedence and the generated-init-container limitation.

Publishable sizing guidance

  • The chart's 100m CPU and 128Mi memory defaults are requests for the worker daemon Deployment; the chart sets no daemon limits by default.
  • Task containers have no worker-defined CPU or memory defaults without pod_template resources or an explicit runner instance shape. Cluster policies can still inject defaults.
  • An explicit runner instance shape sets the task container's CPU and memory requests equal to its limits for that run and overrides matching pod_template values. A pod_template can size operator-defined containers but does not alter worker-generated setup or materialization init containers.
  • Operators should size by workload, cap concurrency when aggregate requests exceed capacity, preserve node headroom, and use compatible autoscaling rather than overprovisioning every worker.

Intentional omissions

  • No universal task size or blanket overprovisioning recommendation was added because the product defines no workload-independent baseline.
  • Internal retry counts and policies are omitted. The docs describe only the observable Job and Pod state operators can act on.

Content design plan

  • Reader and job: A Kubernetes operator diagnosing why a self-hosted task stopped or never started, then sizing the workload without changing unrelated workers.
  • Gap today: Existing troubleshooting combines resource failures, does not cover OOMKilled or exit 143, and does not distinguish daemon resources from task resources.
  • Change: Add symptom-first OOM, eviction, SIGTERM, and scheduling paths plus concise sizing guidance. Exclude universal sizing numbers, internal retry behavior, and unverified future behavior.

Source evidence

  • warpdotdev/oz-agent-worker@530bbe934a632910dfe8818ab7b4c6027a3473ef: internal/worker/kubernetes.go, internal/types/messages.go, internal/metrics/metrics.go, Helm values.yaml, and Kubernetes tests for resource merging, termination classification, scheduling, and cleanup.
  • warp-server@2916aa67b511e8ef1f556766aa3d2dfe42a42f74: runner instance-shape resolution and self-hosted assignment in runner_instance_shape.go, workers/common/task_utils.go, and workers/selfhosted/websocket.go.
  • PR docs: Improve self-hosted Kubernetes troubleshooting #770 at e568ac4251d1411d25271edf0724fb4701a03d96 for current public terminology, configuration names, and page ownership.

Verification

  • npm run build - passed.
  • python3 .agents/skills/check_for_broken_links/check_links.py --internal-only - passed; 4,244 internal links checked, 0 broken.
  • python3 .agents/skills/style_lint/style_lint.py --changed - passed; 5 branch-changed files scanned, 0 issues.
  • python3 .agents/skills/doc_quality_policy/check_compression_contract.py - passed for troubleshooting.mdx and the three parent-PR pages. managed-kubernetes.mdx is 2,010 words, above the 1,500-word feature-doc budget. The 1,941-word baseline is inherited from docs: Improve self-hosted Kubernetes troubleshooting #770; this PR keeps its focused resource guidance next to the canonical pod_template and instance-shape behavior.
  • git diff --check - passed.
  • Trunk CLI - unavailable in the runner, so trunk check was not run.

Unverified claims

None — all new behavior, configuration, default, and metric claims were verified against current source.

Documentation risk

Risk: engineering-review-required
Rationale: Adds technical claims about self-hosted Kubernetes task resources, runner shapes, and failure classification.
Source files consulted: oz-agent-worker@530bbe934a632910dfe8818ab7b4c6027a3473ef: internal/worker/kubernetes.go and charts/oz-agent-worker/values.yaml; warp-server@2916aa67b511e8ef1f556766aa3d2dfe42a42f74: logic/ai/ambient_agents/workers/common/task_utils.go and logic/ai/ambient_agents/workers/selfhosted/websocket.go
Docs override: none

Co-Authored-By: Oz oz-agent@warp.dev

Co-Authored-By: Oz <oz-agent@warp.dev>
@cla-bot cla-bot Bot added the cla-signed label Sep 19, 2026
@vercel

vercel Bot commented Sep 19, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
docs Ready Ready Preview Sep 20, 2026 5:38pm UTC

Request Review

@warp-agent-staging warp-agent-staging Bot added factory:docs-factory Label associated to the "docs-factory" factory warpy-factory Opened by the Warp factory agents labels Sep 19, 2026
@warp-agent-staging

Copy link
Copy Markdown
Contributor Author

This PR was generated with Warp.

Comment @warp-staging-factory on this PR to send it follow-up work.

View run View conversation View origin

Co-Authored-By: Oz <oz-agent@warp.dev>
warp-agent-staging Bot and others added 2 commits September 20, 2026 17:22
Co-Authored-By: Oz <oz-agent@warp.dev>
Co-Authored-By: Oz <oz-agent@warp.dev>
@warp-agent-staging
warp-agent-staging Bot changed the base branch from main to docs/self-hosted-kubernetes-troubleshooting September 20, 2026 17:28
warp-agent-staging Bot and others added 2 commits September 20, 2026 17:32
Co-Authored-By: Oz <oz-agent@warp.dev>
Co-Authored-By: Oz <oz-agent@warp.dev>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-signed factory:docs-factory Label associated to the "docs-factory" factory warpy-factory Opened by the Warp factory agents

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Document self-hosted Kubernetes OOM vs scheduling failures

0 participants