Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions apps/web/src/lib/sandbox-labels.ts
Original file line number Diff line number Diff line change
Expand Up @@ -55,5 +55,6 @@ export function sandboxConfigurationRejection(error: unknown, locale: SupportedL
export function sandboxProviderLabel(provider: SandboxProvider | "", locale: SupportedLanguage): string {
if (provider === "docker") return "Docker";
if (provider === "microsandbox") return "microsandbox";
if (provider === "smolvm") return "smolvm";
return i18n.getFixedT(locale, "sandbox")(provider === "e2b" ? "E2B cloud" : "Unknown state");
}
4 changes: 2 additions & 2 deletions contracts/agents-api/core.openapi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -659,7 +659,7 @@ definitions:
allOf:
- $ref: '#/definitions/sandbox.Resources'
description: |-
Per-sandbox limits, required for Docker and microsandbox. E2B may omit
Per-sandbox limits, required for node providers. E2B may omit
them; Core then uses the validated template build's cpus and memory_mib.
runtime:
$ref: '#/definitions/sandbox.RuntimeRelease'
Expand All @@ -680,7 +680,7 @@ definitions:
allOf:
- $ref: '#/definitions/sandbox.Resources'
description: |-
Per-sandbox limits, required for Docker and microsandbox. E2B may omit
Per-sandbox limits, required for node providers. E2B may omit
them; Core then uses the validated template build's cpus and memory_mib.
runtime:
$ref: '#/definitions/sandbox.RuntimeRelease'
Expand Down
62 changes: 31 additions & 31 deletions contracts/agents-api/sandbox-deployment.md

Large diffs are not rendered by default.

64 changes: 32 additions & 32 deletions contracts/agents-api/zh/sandbox-deployment.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion deploy/node/node_install.py
Original file line number Diff line number Diff line change
Expand Up @@ -1295,7 +1295,7 @@ def main(argv=None):
source.add_argument("--source-url", type=origin)
source.add_argument("--bundle", type=Path)
parser.add_argument("--core-url", type=origin)
parser.add_argument("--provider", choices=("docker", "microsandbox"), help="Optional assertion; Core owns provider selection")
parser.add_argument("--provider", choices=provider_assets.AUTOMATIC_PROVIDERS, help="Optional assertion; Core owns provider selection")
parser.add_argument("--installation-id", required=True)
parser.add_argument("--checkpoint-root", type=Path, help="Private microsandbox checkpoint directory, shared across compatible nodes when configured")
parser.add_argument("--enrollment-token-stdin", action="store_true", help="Read the one-time enrollment token from standard input")
Expand Down
6 changes: 4 additions & 2 deletions deploy/node/node_spec.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@
import urllib.request
import uuid

import provider_assets


class SpecificationError(Exception):
pass
Expand All @@ -27,7 +29,7 @@ def release(manifest):

# BEGIN GENERATED DEPLOYMENT CONTRACT
# Generated from sandbox/deployment_contract.go; do not edit.
_CONTRACT = json.loads("{\"workspace_fields\":[\"attachment\",\"user_xattr\",\"capacity_quota\"],\"resources\":[{\"name\":\"cpus\",\"min\":1,\"max\":255,\"omit_zero\":false},{\"name\":\"memory_mib\",\"min\":512,\"max\":1048576,\"omit_zero\":false},{\"name\":\"root_disk_mib\",\"min\":0,\"max\":4294967295,\"omit_zero\":true},{\"name\":\"environment_disk_mib\",\"min\":0,\"max\":4294967295,\"omit_zero\":true}],\"runtime\":[{\"name\":\"source_commit\",\"pattern\":\"[0-9a-f]{40}\"},{\"name\":\"image_id\",\"pattern\":\"sha256:[0-9a-f]{64}\"},{\"name\":\"image_manifest_digest\",\"pattern\":\"sha256:[0-9a-f]{64}\"},{\"name\":\"microsandbox_ref\",\"pattern\":\"oac-runtime@sha256:[0-9a-f]{64}\"},{\"name\":\"runtime_sha256\",\"pattern\":\"[0-9a-f]{64}\"},{\"name\":\"firmware_sha256\",\"pattern\":\"[0-9a-f]{64}\"}],\"providers\":{\"docker\":{\"mode\":\"nodes\",\"disk\":false,\"runtime\":true,\"default_resources\":{\"cpus\":2,\"memory_mib\":2048}},\"e2b\":{\"mode\":\"direct\",\"disk\":false,\"runtime\":false,\"default_resources\":null},\"microsandbox\":{\"mode\":\"nodes\",\"workspace\":{\"attachment\":\"host_directory\",\"user_xattr\":true},\"disk\":true,\"runtime\":true,\"default_resources\":{\"cpus\":2,\"memory_mib\":4096,\"root_disk_mib\":8192,\"environment_disk_mib\":8192}}},\"minimum_disk\":1024}")
_CONTRACT = json.loads("{\"workspace_fields\":[\"attachment\",\"user_xattr\",\"capacity_quota\"],\"resources\":[{\"name\":\"cpus\",\"min\":1,\"max\":255,\"omit_zero\":false},{\"name\":\"memory_mib\",\"min\":512,\"max\":1048576,\"omit_zero\":false},{\"name\":\"root_disk_mib\",\"min\":0,\"max\":4294967295,\"omit_zero\":true},{\"name\":\"environment_disk_mib\",\"min\":0,\"max\":4294967295,\"omit_zero\":true}],\"runtime\":[{\"name\":\"source_commit\",\"pattern\":\"[0-9a-f]{40}\"},{\"name\":\"image_id\",\"pattern\":\"sha256:[0-9a-f]{64}\"},{\"name\":\"image_manifest_digest\",\"pattern\":\"sha256:[0-9a-f]{64}\"},{\"name\":\"microsandbox_ref\",\"pattern\":\"oac-runtime@sha256:[0-9a-f]{64}\"},{\"name\":\"runtime_sha256\",\"pattern\":\"[0-9a-f]{64}\"},{\"name\":\"firmware_sha256\",\"pattern\":\"[0-9a-f]{64}\"}],\"providers\":{\"docker\":{\"mode\":\"nodes\",\"disk\":false,\"runtime\":true,\"default_resources\":{\"cpus\":2,\"memory_mib\":2048}},\"e2b\":{\"mode\":\"direct\",\"disk\":false,\"runtime\":false,\"default_resources\":null},\"microsandbox\":{\"mode\":\"nodes\",\"workspace\":{\"attachment\":\"host_directory\",\"user_xattr\":true},\"disk\":true,\"runtime\":true,\"default_resources\":{\"cpus\":2,\"memory_mib\":4096,\"root_disk_mib\":8192,\"environment_disk_mib\":8192}},\"smolvm\":{\"mode\":\"nodes\",\"disk\":true,\"runtime\":true,\"default_resources\":{\"cpus\":2,\"memory_mib\":4096,\"root_disk_mib\":8192,\"environment_disk_mib\":8192}}},\"minimum_disk\":1024}")
# END GENERATED DEPLOYMENT CONTRACT


Expand Down Expand Up @@ -89,7 +91,7 @@ def validate(data, args):
raise SpecificationError(PUBLIC_URL_CHANGED)
try:
provider, spec = data["provider"], data["specification"]
if (provider not in ("docker", "microsandbox") or data["installation_id"] != args.installation_id
if (provider not in provider_assets.AUTOMATIC_PROVIDERS or data["installation_id"] != args.installation_id
or data["core_url"] != args.core_url or type(data["generation"]) is not int or data["generation"] < 1
or getattr(args, "provider", None) not in (None, provider)
or set(spec) - {"workspace"} != {"resources", "runtime"}
Expand Down
4 changes: 3 additions & 1 deletion deploy/node/provider_assets.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
# Code generated by go run ./services/core/cmd/provider-artifacts -write; DO NOT EDIT.
import json

CATALOG = json.loads("{\n \"docker\": [\n {\n \"path\": \"native/bin/oac-node\",\n \"suffix\": \"sandbox-node\",\n \"role\": \"node\"\n },\n {\n \"path\": \"images/runtime.tar.gz\",\n \"suffix\": \"runtime.tar.gz\",\n \"role\": \"image\"\n },\n {\n \"path\": \"runtime/seccomp.json\",\n \"suffix\": \"seccomp.json\",\n \"role\": \"policy\"\n }\n ],\n \"microsandbox\": [\n {\n \"path\": \"native/bin/oac-node\",\n \"suffix\": \"sandbox-node\",\n \"role\": \"node\"\n },\n {\n \"path\": \"images/runtime.tar.gz\",\n \"suffix\": \"runtime.tar.gz\",\n \"role\": \"image\"\n },\n {\n \"path\": \"runtime/seccomp.json\",\n \"suffix\": \"seccomp.json\",\n \"role\": \"policy\"\n },\n {\n \"path\": \"native/bin/oac-microsandbox-provider\",\n \"suffix\": \"microsandbox-provider\",\n \"role\": \"runtime\"\n },\n {\n \"path\": \"native/microsandbox/msb\",\n \"suffix\": \"msb\",\n \"role\": \"runtime\"\n },\n {\n \"path\": \"native/microsandbox/libkrunfw.so.5.6.1\",\n \"suffix\": \"libkrunfw.so.5.6.1\",\n \"role\": \"runtime\"\n }\n ]\n}\n")
CATALOG = json.loads("{\n \"docker\": [\n {\n \"path\": \"native/bin/oac-node\",\n \"suffix\": \"sandbox-node\",\n \"role\": \"node\"\n },\n {\n \"path\": \"images/runtime.tar.gz\",\n \"suffix\": \"runtime.tar.gz\",\n \"role\": \"image\"\n },\n {\n \"path\": \"runtime/seccomp.json\",\n \"suffix\": \"seccomp.json\",\n \"role\": \"policy\"\n }\n ],\n \"microsandbox\": [\n {\n \"path\": \"native/bin/oac-node\",\n \"suffix\": \"sandbox-node\",\n \"role\": \"node\"\n },\n {\n \"path\": \"images/runtime.tar.gz\",\n \"suffix\": \"runtime.tar.gz\",\n \"role\": \"image\"\n },\n {\n \"path\": \"runtime/seccomp.json\",\n \"suffix\": \"seccomp.json\",\n \"role\": \"policy\"\n },\n {\n \"path\": \"native/bin/oac-microsandbox-provider\",\n \"suffix\": \"microsandbox-provider\",\n \"role\": \"runtime\"\n },\n {\n \"path\": \"native/microsandbox/msb\",\n \"suffix\": \"msb\",\n \"role\": \"runtime\"\n },\n {\n \"path\": \"native/microsandbox/libkrunfw.so.5.6.1\",\n \"suffix\": \"libkrunfw.so.5.6.1\",\n \"role\": \"runtime\"\n }\n ],\n \"smolvm\": [\n {\n \"path\": \"native/bin/oac-node\",\n \"suffix\": \"sandbox-node\",\n \"role\": \"node\"\n },\n {\n \"path\": \"images/runtime.tar.gz\",\n \"suffix\": \"runtime.tar.gz\",\n \"role\": \"image\"\n }\n ]\n}\n")

AUTOMATIC_PROVIDERS = json.loads("[\n \"docker\",\n \"microsandbox\"\n]\n")

def artifacts(provider, roles=None):
return tuple(item["path"] for item in CATALOG[provider] if roles is None or item["role"] in roles)
8 changes: 8 additions & 0 deletions deploy/node/test_node_spec.py
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,14 @@ def test_invalid_limits_digests_and_provider_assertions_are_rejected(self):
with self.assertRaises(node_spec.SpecificationError):
node_spec.validate(self.data, self.args)

def test_manual_provider_is_not_advertised_by_automatic_installer(self):
data = copy.deepcopy(self.data)
data["provider"] = "smolvm"
data["specification"]["resources"].update(root_disk_mib=8192, environment_disk_mib=8192)
data["specification_digest"] = node_spec.digest("smolvm", data["specification"])
with self.assertRaisesRegex(node_spec.SpecificationError, "configuration differs or is invalid"):
node_spec.validate(data, self.args)

def test_capacity_requires_approved_bounded_integers(self):
for key, value in (("max_active", None), ("max_active", True), ("max_active", 0),
("max_retained", 1), ("max_retained", 1000001)):
Expand Down
2 changes: 1 addition & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ Runtime settings live in Core's database. Change them in Web; scripts use the sa

| Setting | Where in Web | Core API | Notes |
| --- | --- | --- | --- |
| Sandbox backend: Docker, microsandbox or E2B | **System** → **Manage sandbox configuration**: the setup wizard, ending with **Save configuration** | `/core/v1/sandbox/deployment` | One backend per installation, chosen after the first sign-in. Another backend needs **Reset deployment** first; see [change the sandbox configuration](./getting-started/nodes.md#change-the-sandbox-configuration) |
| Sandbox backend: Docker, microsandbox, smolvm or E2B | **System** → **Manage sandbox configuration**: the setup wizard for Docker, microsandbox and E2B, ending with **Save configuration**; select smolvm through the administration API | `/core/v1/sandbox/deployment` | One backend per installation, chosen after the first sign-in. Another backend needs **Reset deployment** first; see [change the sandbox configuration](./getting-started/nodes.md#change-the-sandbox-configuration) |
| Sandbox size, Runtime release, E2B key and template build | **System** → **Manage sandbox configuration** → **Change resources** | `/core/v1/sandbox/deployment` | Web proposes the [default size the Provider declares](./sandbox-provider.md#register-the-provider-kind). Existing sandboxes keep their size and release. The E2B key is write-only and encrypted |
| Nodes and their capacity | **Nodes**: **Add node**; **Edit node** and **Remove node** on a node's page | `/core/v1/sandbox/enrollment-tokens`, `/core/v1/sandbox/nodes` | See [Node capacity](#node-capacity) and the [nodes guide](./getting-started/nodes.md) |
| Projects and API keys | **Projects and keys**: **Create project**, **Rename**, **Issue key**, **Revoke**, **Archive** | `/core/v1/projects` | Keys are shown once; Core stores digests |
Expand Down
8 changes: 6 additions & 2 deletions docs/getting-started/nodes.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
title: "Add and manage nodes"
---

A node is a Linux host that runs sandboxes for Core-hosted Sessions when the sandbox backend is Docker or microsandbox. Core places each new Session on a node with free capacity; the node creates the sandbox, and the sandbox connects back to Core. E2B needs no nodes. Machines that applications connect for their own Sessions are [self-hosted executors](./self-hosted.md), not nodes.
A node is a Linux host that runs sandboxes for Core-hosted Sessions when the sandbox backend is Docker, microsandbox or smolvm. Core places each new Session on a node with free capacity; the node creates the sandbox, and the sandbox connects back to Core. E2B needs no nodes. Machines that applications connect for their own Sessions are [self-hosted executors](./self-hosted.md), not nodes.

You add a node by generating a command in Web and running it on the host. The [sandbox deployment contract](../../contracts/agents-api/sandbox-deployment.md) defines the sandbox settings and their change rules, and the [node protocol](../../contracts/agents-api/node-generation-protocol.md) defines how nodes prepare and keep Runtime generations.
For Docker and microsandbox, add a node by generating a command in Web and running it on the host. Register smolvm nodes manually using the steps below. The [sandbox deployment contract](../../contracts/agents-api/sandbox-deployment.md) defines the sandbox settings and their change rules, and the [node protocol](../../contracts/agents-api/node-generation-protocol.md) defines how nodes prepare and keep Runtime generations.

## Before you add a node

Expand Down Expand Up @@ -47,6 +47,7 @@ The installer shows each phase as it runs and, once Core confirms the node, a su
- SELinux not enforcing. The installer does not support hosts with enforcing SELinux.
- Docker: rootful Docker Engine running, its socket `/var/run/docker.sock` owned by the `docker` group with mode `0660`, enforcing CPU and memory limits (cgroup v2).
- microsandbox: `/dev/kvm` in the `kvm` group (hardware or nested virtualization), and the libraries microsandbox links (glibc).
- smolvm (manual registration): `/dev/kvm` and a persistent smolvm API service bound to a Unix socket accessible only to the node account; the packaged Runtime OCI archive must remain on the node.
- CPUs and memory for at least one sandbox of the installation's size, and about 2 GB of disk for the Runtime image.
- Access to the console and Core at the public URL; sandboxes reach Core too.

Expand Down Expand Up @@ -126,12 +127,15 @@ It never deletes sandboxes, volumes or images. It keeps the Runtime image and pr

Use manual registration when you manage the node's files and service yourself instead of running Web's command. A manually registered node serves only the configuration it registered with: after a size or Runtime change, **Nodes** shows it as **Node software incompatible**, and it keeps serving the old configuration until you remove it and register the host again.

For smolvm, select the backend through the [Core administration API](./operations.md#script-the-core-api): POST `/sandbox/deployment` with the observed `expected_generation`, `provider: "smolvm"`, `configuration: {}`, `resources` including both disk sizes, and the complete matching [Runtime release identity](../../contracts/agents-api/sandbox-deployment.md#runtime-release). Web's configuration wizard and **Add node** command do not set up the smolvm service. Run `smolvm serve start --listen unix:///var/lib/oac/smolvm.sock` as the same account as `oac-node`, with a durable smolvm machine store; the API socket grants control over machines and must have mode `0600` for that account. Use a smolvm build that supports persistent machine labels and local OCI archive images.

1. Take `oac-node` from the same release as Core.
2. Get an enrollment token: the token in a command from **Add node**, or `POST /core/v1/sandbox/enrollment-tokens` with the Core key. It is single-use and carries the node's approved capacity; the response's `expires_at` says when it expires. Save it in a `0600` file on the host.
3. Read the node configuration with the token, which does not consume it: `GET /api/v1/sandbox-node/configuration` with `Authorization: Bearer <token>`.
4. Write a private provider file. Copy `provider`, `installation_id`, `core_url`, `generation` and `specification` from the response, and add a `native` object with the host settings of that provider. The adapter reads sandbox size, the Runtime image and artifact hashes from `specification`:
- Docker: the [Docker node configuration](../configuration.md#docker-node-configuration) fields, with `host` an explicit Unix socket, `image` the local ID of the imported Runtime image and `seccomp_file` absolute.
- microsandbox: absolute `helper_path`, `runtime_path` and `firmware_path`, a `network` policy, an explicit `checkpoint_root` for private checkpoint archives, and `runtime_home`: a private directory, which the helper creates with mode `0700` when it is missing. microsandbox places Unix sockets under it, so keep its path within 48 bytes; the installer refuses a longer one for its own nodes.
- smolvm: `socket` as a canonical local `unix:///absolute/path.sock` API URL, `image_file` as the absolute path to the Runtime release OCI archive, and `receipt_root` as a persistent private directory (mode `0700`). The node verifies the archive against Core’s selected Runtime image and manifest digests. Use manual registration; the installer does not provision a smolvm service.
Before the first microsandbox registration, initialize the empty `checkpoint_root` as the node service account using the same release's source checkout: `PYTHONPATH=deploy/node python3 -c 'from node_install import prepare_checkpoint_root; prepare_checkpoint_root("/absolute/private/checkpoints")'`. This reuses the installer's exclusive UUID-marker initialization and ownership checks. Initialize a shared store once; other nodes use the existing marker. Never replace the marker or initialize over restored or nonempty unowned storage.
5. Register, then run the node under the host's service supervisor, with real absolute paths:

Expand Down
Loading