feat: warmpool support for OpenShell sandboxes #1447

Activity

1. github-actions

   added

   state:triage-neededOpened without agent diagnostics and needs triageOpened without agent diagnostics and needs triage

   on May 19, 2026
2. ryanzhang-oss commented on May 21, 2026

   

   ryanzhang-oss

   on May 21, 2026

   More actions

   second to this, the better way to integrate with sandbox is through the SandboxClaim instead of a cold one.
3. ryanzhang-oss commented on May 26, 2026

   

   ryanzhang-oss

   on May 26, 2026

   More actions

   I took a closer look at this and it seems that the change is relatively simple, here is the AI generated plan which I vetted

   Issue 1447: SandboxClaim Migration — Implementation Guide

   Invariant (do not break)

   Gateway is the sole allocator and owner of sandbox-id. Supervisor connect is rejected unless that id is already persisted in the gateway store.

   This migration changes only the Kubernetes provisioning object used by the driver. It does not change gateway, server protocol, or supervisor behavior.

   Relevant unchanged code paths:

   • ID allocation: crates/openshell-server/src/grpc/sandbox.rs#L91
   • Persistence before driver create: crates/openshell-server/src/compute/mod.rs#L439
   • Supervisor connect gate: crates/openshell-server/src/supervisor_session.rs#L572
   • Required env injection (including OPENSHELL_SANDBOX_ID): crates/openshell-driver-kubernetes/src/driver.rs#L1340

   Implementation

   Step 1: Add a provisioning mode

   File: crates/openshell-driver-kubernetes/src/config.rs and main.rs.

   Add:

   • sandbox_provisioning_mode: sandbox (default) or sandboxclaim
   • sandboxclaim_template_name: required when mode is sandboxclaim
   • sandboxclaim_warmpool: optional, default default

   No proto or server changes.

   Step 2: Mode-aware GVK in the driver

   File: crates/openshell-driver-kubernetes/src/driver.rs.

   Today api() and watch_api() hardcode agents.x-k8s.io/v1alpha1, Sandbox. Replace both with one helper that returns the correct GVK based on mode:

   • sandbox -> agents.x-k8s.io/v1alpha1, Sandbox
   • sandboxclaim -> extensions.agents.x-k8s.io/v1alpha1, SandboxClaim

   Centralize this so the rest of the driver does not branch.

   Step 3: Branch create_sandbox by mode

   Today create_sandbox always calls sandbox_to_k8s_spec(...).

   In claim mode, build a claim object instead:

   apiVersion: extensions.agents.x-k8s.io/v1alpha1
   kind: SandboxClaim
   metadata:
     name: <gateway sandbox name>
     labels:
       openshell.ai/sandbox-id: <gateway-issued id>
       openshell.ai/managed-by: openshell
   spec:
     sandboxTemplateRef:
       name: <sandboxclaim_template_name>
     warmpool: <sandboxclaim_warmpool>
     env:
       - name: OPENSHELL_SANDBOX_ID
         value: <gateway-issued id>
       # ... rest of build_env_list output, converted to claim env entries
     additionalPodMetadata:
       labels:
         openshell.ai/sandbox-id: <gateway-issued id>

   Key implementation rules:

   1. Reuse build_env_list(...) to produce env entries — do not duplicate logic.
   2. Convert each entry into the claim spec.env shape.
   3. If sandboxclaim_template_name is empty, fail fast with Precondition.
   4. Do not mutate or regenerate the id anywhere in the driver.

   Step 4: OPENSHELL_SANDBOX_ID propagation contract

   This is the part the original issue actually depends on.

   In Sandbox mode, the id flows: gateway -> driver create_sandbox -> pod env via apply_required_env.

   In SandboxClaim mode, the upstream controller materializes the pod, not the driver. So the id must ride into the pod through claim fields that the controller is guaranteed to propagate:

   1. Primary: spec.env entry for OPENSHELL_SANDBOX_ID. This is the upstream-supported per-claim env override.
   2. Secondary: spec.additionalPodMetadata.labels with the id, so the pod is identifiable in cluster tooling even if env stripping ever occurs.
   3. Tertiary: claim metadata.labels carry the id for driver-side correlation.

   Reviewer acceptance checks:

   • Pod env in claim mode contains OPENSHELL_SANDBOX_ID exactly equal to gateway store id.
   • Supervisor connect succeeds without any change in supervisor_session.rs.
   • If env propagation is dropped by a particular claim CRD version, driver fails create with a clear precondition error rather than producing an orphan pod.

   Step 5: Status, list, get

   sandbox_from_object(...) currently assumes core Sandbox status keys. Make it tolerant:

   • Read id from label openshell.ai/sandbox-id first (works for both kinds).
   • In claim mode, read status.sandbox.name when present to populate instance correlation.
   • Do not extend the DriverSandbox protobuf; keep external surface stable.

   Step 6: Watch and event correlation

   Today map_kube_event_to_platform(...) handles involved kinds Sandbox and Pod.

   Add:

   • SandboxClaim involved kind.
   • Index claim name -> sandbox-id from claim labels.
   • Index bound sandbox name / pod name -> sandbox-id from claim status.

   Without this, platform events get dropped in claim mode.

   Step 7: Delete

   delete_sandbox and sandbox_exists only need the mode-aware GVK from Step 2. No further changes.

   Tests to add

   Only the risky seams:

   1. Mode selection returns correct GVK.
   2. Claim create payload includes spec.sandboxTemplateRef.name.
   3. Claim spec.env contains OPENSHELL_SANDBOX_ID matching the gateway id.
   4. sandbox_from_object reads id from label in both kinds.
   5. Event mapper resolves SandboxClaim involved kind.

   Caveats

   • Current dev cluster context does not expose sandboxclaims. Validate against a cluster with the extensions CRDs installed before enabling claim mode.
   • Claim CRD versions across upstream releases differ. Pin to the version that supports spec.env and spec.additionalPodMetadata.
4. rohancmr commented on Jun 12, 2026

   

   rohancmr

   on Jun 12, 2026

   More actions

   +1 for this feature.

   From a Kubernetes platform perspective, this is important for multi-tenant sandbox environments where the platform owns tenant onboarding, namespace controls, quotas, NetworkPolicy, RuntimeClass/node placement, and approved sandbox profiles.

   We would like OpenShell's Kubernetes driver to support a claim-based allocation mode using the Agent Sandbox extension CRDs:

   • SandboxTemplate
   • SandboxWarmPool
   • SandboxClaim

   The desired flow is that OpenShell Gateway remains the allocator and owner of the sandbox identity, but instead of always creating a cold Sandbox, it can create a SandboxClaim against an operator-approved SandboxTemplate and optional SandboxWarmPool.

   This would let platform teams pre-warm secure sandbox capacity for approved profiles, including gVisor/Kata runtime classes, tainted node pools, GPU pools, and tenant-specific profiles, without deploying one OpenShell Gateway per runtime profile.

   Important requirements from our side:

   • OpenShell Gateway should continue to own sandbox_id allocation and supervisor identity validation.
   • Claim mode should propagate the OpenShell sandbox identity and useful tenant/request metadata onto the SandboxClaim and resulting pod.
   • Warm capacity should be operator/admin configured, not arbitrary user supplied.
   • Warm pools should only reuse clean, platform-owned base capacity. A warm sandbox may be pre-created with the approved image, runtime class, base filesystem, supervisor startup, and empty workspace, but once it is claimed by a tenant or request it must be treated as single-use.
   • After a sandbox has run customer workload code, OpenShell must not return that same sandbox, workspace, supervisor session, process tree, network namespace, or resolved credentials back to the warm pool for another allocation. The claimed sandbox should either be destroyed or reset through a mechanism that guarantees no customer-specific runtime state remains.
   • Customer-specific runtime state includes writable workspace contents, .env files, provider credentials, MCP/tool tokens, process state, background jobs, network connections, model/tool sessions, logs, caches, and the live supervisor connection.
   • If no matching warm pool exists, OpenShell should be able to fall back to the existing cold Sandbox creation path.

   This would make OpenShell much easier to use as the sandbox execution plane behind a Kubernetes platform that manages tenant isolation and placement.
5. craig-kindo commented on Jul 7, 2026

   

   craig-kindo

   on Jul 7, 2026

   Last edited by craig-kindo

   More actions

   Moving the design context from my vouch request (#2132) here at @drew's request.

   I've built and validated a warm-pool integration against a GKE test cluster, and I'd like to propose it as the implementation direction for this issue. The short version: let the Agent Sandbox extension CRDs own all pool state, and teach the supervisor to start unbound and wait for its identity to arrive as a file. No gateway pool reconciler, no new gateway↔supervisor protocol, no Kubernetes client in the supervisor.

   The design, in two independent pieces

   Part 1 — claim provisioning mode in the Kubernetes driver. An optional driver mode where CreateSandbox produces a SandboxClaim (referencing an operator-managed SandboxTemplate / SandboxWarmPool) instead of a core Sandbox. This is close to the migration plan @ryanzhang-oss posted above, and it's useful on its own: a claim with no warm capacity available just has the Agent Sandbox controller create a fresh Sandbox to satisfy it, so cold fallback comes for free. The gateway still allocates sandbox_id and persists the store record before creating the claim, so the ConnectSupervisor gate in supervisor_session.rs is untouched.

   Part 2 — a wait-for-bind startup phase in the supervisor. This is the piece that makes true pre-warming work, and it's where I'd like to correct one thing in the plan above. The issue correctly identifies static identity binding (OPENSHELL_SANDBOX_ID as a boot-time env var) as blocker # 1. The plan above routes the id through SandboxClaim.spec.env — but upstream rejects custom env on the warm path (custom environment variables are not supported when using a warm pool). I hit this in-cluster; claim env only works for claims that fall through to cold creation. So env can't be the identity carrier for warm adoption, full stop.

   Instead:

   1. The warm-pool template starts the supervisor with a flag/env var putting it in an optional unbound phase. It does structural, identity-free init and then waits. Pods in this state carry no user policy, no credentials, no session.
   2. At claim time, the driver puts the sandbox id/name on the claim's additionalPodMetadata. The Agent Sandbox controller propagates those onto the adopted pod's annotations.
   3. The template mounts a downwardAPI volume projecting those annotations as files. The supervisor watches the projected file as a sentinel; when the id appears, it proceeds through the existing, unchanged bootstrap — ConnectSupervisor with the discovered id, token, policy fetch, session — exactly as if it had booted with the env var.

   The claim's binding state lives in exactly one place (the k8s API), and the pod observes it without a k8s client or any new RBAC, because the kubelet does the projection for us.

   The projection isn't free relative to giving the supervisor its own Kubernetes watch: in a pilot benchmark of annotation-change → in-pod detection, the downwardAPI file lagged a direct pod watch by ~75ms p50 / ~120ms p95. I consider that an acceptable price for keeping a Kubernetes client, RBAC, and API-server coupling out of the sandbox pod entirely.

   How this answers the open questions in the issue

   • "Gateway: introduce a warm_idle state + pool reconciler" — not needed. SandboxWarmPool already is the reconciler; unclaimed pods never appear in the gateway store at all, so there's nothing to hide from ListSandboxes.
   • "Where does pool configuration live / runtime resizing" — in the SandboxWarmPool/SandboxTemplate objects, operator-owned, resizable with kubectl and no gateway restart. OpenShell config only names which template/pool the claim references.
   • "Update strategy: Recreate vs OnReplenish" — inherited from upstream rather than reimplemented; we don't need to pick.
   • "TTL / policy staleness of unclaimed slots" — mostly dissolves: an unbound pod has no policy yet. Policy is fetched post-bind through the normal path, so there's no version-N-policy-baked-into-a-warm-pod problem. (Baking a static base policy into the template is a possible later optimization; measured numbers below suggest it isn't needed.)
   • "Is the window between READY and policy-reload safe" — there is no such window: the sandbox isn't READY, and isn't user-visible, until after bind → bootstrap → policy completes.
   • @rohancmr's isolation requirements — I think this design satisfies them structurally. Gateway keeps id allocation and supervisor identity validation; warm capacity is operator-defined via template/pool; a pre-bind pod has never held customer state; and adoption is single-use — deleting the sandbox deletes the claim and the backing sandbox (shutdownPolicy foreground delete), nothing returns to the pool. Replenishment creates fresh pods.
   • "Non-Kubernetes drivers" — untouched. This is honestly the main tradeoff, discussed below.

   One open question I'll flag but haven't investigated: the unbound phase is a natural place for pre-claim preparation beyond just parking the process — e.g. warm-pool-level conventions could let NetworkPolicies (or similar environment shaping) be set up for whatever class of claimants a given pool serves, before any identity exists.

   Validation

   I validated this end-to-end on a GKE test cluster (Agent Sandbox core + extensions, operator-managed template and warm pool): a real SandboxClaim adopts a running warm pod, the supervisor binds via the projected files, reaches READY through the unchanged bootstrap, and serves exec. On that cluster the cold create → READY path runs ~26s p50; the warm claim path measures ~1.9s p50 / ~2.1s p95 from CreateSandbox to first successful ExecSandbox.

   The tradeoff vs. a gateway-native pool

   A gateway-owned pool (warm_idle records, reconciler loop, claim-at-CreateSandbox) is more portable — it would extend to the Docker/VM drivers and to providers with no Agent Sandbox equivalent. I don't think that's a bad design. But it mirrors pool/claim state into the gateway store and the supervisor session layer on top of the k8s custom resources that already hold it, which is a distributed-state problem I'd rather not create, and it grows the gateway↔supervisor handshake, which I think needs to stay as small as possible to remain maintainable. This proposal keeps the handshake byte-identical and confines the entire feature to the k8s driver plus one optional supervisor startup phase. If a native pool for non-k8s drivers is wanted later, nothing here blocks it — the supervisor's wait-for-bind phase would even be reusable, with a different bind-signal carrier per driver.

   Size-wise, the supervisor change is a couple hundred lines; the driver claim mode is the larger mechanical diff. The two pieces split naturally into two PRs (claim mode first, wait-for-bind second), and I'm not attached to any of my prototype's flag names or file paths — those were chosen for speed of validation and I'd expect them to change in review.

   Happy to share benchmark harnesses/artifacts or the working branches if useful.
6. rhuss

   added a commit that references this issue on Jul 10, 2026

   docs: add validation run, fix TL;DR, upstream references, mermaid dia…

   ...

   3d8cde4
7. matthewgrossman commented on Jul 29, 2026

   

   matthewgrossman

   on Jul 29, 2026

   Member

   More actions

   📋 triage-agent

   Triage Assessment

   Classification: duplicate

   Summary

   #2157 is the routed warm-pool spike and incorporates this issue’s SandboxClaim, identity-binding, lifecycle, isolation, and configuration analysis.

   Investigation

   #2157 also has active implementation PR #2460 and preserves the useful design evidence contributed here.

   Recommendation

   Closing as a duplicate of #2157. Continue design review and implementation tracking there.
8. matthewgrossman

   added

   duplicateThis issue or pull request already existsThis issue or pull request already exists

   and removed

   state:triage-neededOpened without agent diagnostics and needs triageOpened without agent diagnostics and needs triage

   on Jul 29, 2026
9. matthewgrossman

   closed this as not plannedon Jul 29, 2026