Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions apps/web/e2e/nodes.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,13 @@ test("removes a node after confirmation", async ({ page, request }) => {
await expect(page.getByRole("heading", { name: "Nodes", level: 1 })).toBeFocused();
});

test("gives a host's uninstall command at any time from Clean up a host", async ({ page, request }) => {
await openConsole(page, request, "nodes");
await page.getByRole("button", { name: "Clean up a host" }).click();
const cleanup = page.getByRole("dialog", { name: "Clean up the host" });
await expect(cleanup.getByLabel("Uninstall command", { exact: true })).toHaveValue(/--uninstall --installation-id '7f3c2a90-5b1e-4c2d-9e3f-0a1b2c3d4e5f'\)$/);
});

test("marks a node on an old Core address in its row, beside each node's limit", async ({ page, request }) => {
await openConsole(page, request, "nodes", { installation: "stale" });
const row = page.getByRole("row", { name: /core-01/ });
Expand Down
17 changes: 10 additions & 7 deletions apps/web/src/features/sandbox/NodeCleanupDialog.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,18 @@ import { nodeSourceUrl } from "./core-origin";
import { nodeUninstallCommand } from "./enrollment-command";
import { CommandBlock } from "./node-commands";

/** A node Core has just removed, with what builds its host's uninstall command. */
/** What builds a host's uninstall command; `name` is the node Core has just removed, or null for any host. */
export interface NodeCleanup {
name: string;
name: string | null;
installationId: string;
scriptDigest: string;
/** The Core address the node enrolled with, when it is no longer the deployment's; else null. */
oldAddress: string | null;
}

/**
* After Remove: the command that removes the node's service and files from its
* After Remove, or from Clean up a host at any time (also for a host that never
* registered): the command that removes the node's service and files from its
* host (deploy/node/node_install.py `uninstall_system`).
* The installer first confirms with Core, at the node's own address, that the
* node is removed, which holds from the removal on. The command uses sudo unless the shell is already root. A node enrolled with an earlier address may find it gone; then
Expand All @@ -35,7 +36,7 @@ export function NodeCleanupDialog({ cleanup, open, onClose }: { cleanup: NodeCle
const sourceUrl = installation.data ? nodeSourceUrl(installation.data) : null;
const command = (force = false) => cleanup && sourceUrl
? nodeUninstallCommand({ sourceUrl, installationId: cleanup.installationId, scriptDigest: cleanup.scriptDigest, force }) : "";
const stays = cleanup ? t("{{name}} is removed from Core, but its service and files stay on the host.", { name: cleanup.name }) : "";
const stays = cleanup ? cleanup.name === null ? t("A node's service and files stay on its host until it is uninstalled there.") : t("{{name}} is removed from Core, but its service and files stay on the host.", { name: cleanup.name }) : "";
return <Modal open={open} title={t("Clean up the host")} onClose={onClose} footer={<button className="button primary" type="button" onClick={onClose}>{t("Done")}</button>}>
{cleanup && !installation.data ? <div className="sandbox-add-node form-stack">
{installation.isError
Expand All @@ -44,13 +45,15 @@ export function NodeCleanupDialog({ cleanup, open, onClose }: { cleanup: NodeCle
</div> : cleanup && installation.data && !sourceUrl ? <div className="sandbox-add-node form-stack">
<p>{join(stays, t("Other machines can't reach this installation's public URL, {{url}}, so no uninstall command can be given.", { url: installation.data.public_url }))}</p>
</div> : cleanup ? <div className="sandbox-add-node form-stack">
<p>{t("{{name}} is removed from Core. To remove its service and files from the host, run:", { name: cleanup.name })}</p>
<p>{cleanup.name === null
? join(t("To remove a node's service and files from its host, run this on the host."), t("Core must no longer list the node, so remove it on this page first. A host that never registered needs no removal."))
: t("{{name}} is removed from Core. To remove its service and files from the host, run:", { name: cleanup.name })}</p>
<CommandBlock key={command()} value={command()} label={t("Uninstall command")} autoFocus />
<p className="sandbox-cleanup-note">{t("It never deletes sandboxes, volumes or images.")}</p>
<p className="sandbox-cleanup-note">{t("It keeps Docker images and volumes; a microsandbox store goes once Core confirmed the removal.")}</p>
{cleanup.oldAddress !== null ? <details className="sandbox-host-requirements">
<summary>{t("Old Core address gone?")}</summary>
<div className="sandbox-cleanup-details">
<p>{join(t("{{name}} still points at the old Core address {{address}}.", { name: cleanup.name, address: cleanup.oldAddress }),
<p>{join(t("{{name}} still points at the old Core address {{address}}.", { name: cleanup.name ?? "", address: cleanup.oldAddress }),
t("If this node's old Core address no longer responds, first remove it on the Nodes page, then add --force to the uninstall command."))}</p>
<CommandBlock key={command(true)} value={command(true)} label={t("Uninstall command with --force")} copyName={t("Copy command with --force")} />
</div>
Expand Down
3 changes: 3 additions & 0 deletions apps/web/src/features/sandbox/SandboxManagerView.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,9 @@ function SandboxManager({ consoleConfig }: { consoleConfig: SandboxConsoleConfig
const actions = <>
{localOnly && hostedNodes ? <span id="add-node-blocked" className="muted">{tCommon("installationNotice.addBlocked")}</span> : null}
{hostedNodes ? <button className="button outline" type="button" onClick={() => navigate("settings", { id: "sandbox" })}>{tSandboxNav("open")}</button> : null}
{consoleConfig.node_installer && snapshot ? <button className="button outline" type="button" onClick={() => setCleanup({ node: {
name: null, installationId: snapshot.deployment.installation_id, scriptDigest: consoleConfig.node_installer_sha256, oldAddress: null,
}, open: true })}>{t("Clean up a host")}</button> : null}
{refreshButton}
{hostedNodes && snapshot ? <button type="button" className="button primary" disabled={addDisabled} aria-describedby={localOnly ? "add-node-blocked" : undefined} onClick={() => setAdding(true)}><Plus size={16} />{t("Add node")}</button> : null}
</>;
Expand Down
6 changes: 5 additions & 1 deletion apps/web/src/i18n/locales/zh-CN/sandbox.ts
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ export const sandbox = {
"Copy command with --force": "复制命令(--force)",
"Checking this installation's public URL…": "正在检查这个安装的公网地址…",
"The installation couldn't be read, so no command can be issued.": "无法读取安装信息,暂时不能生成命令。",
"It never deletes sandboxes, volumes or images.": "它不会删除任何沙箱、卷或镜像。",
"It keeps Docker images and volumes; a microsandbox store goes once Core confirmed the removal.": "它保留 Docker 镜像和卷;Core 确认移除后,microsandbox 存储会一并删除。",
"Old Core address gone?": "旧 Core 地址已失效?",
"{{name}} still points at the old Core address {{address}}.": "{{name}} 仍指向旧的 Core 地址 {{address}}。",
"If this node's old Core address no longer responds, first remove it on the Nodes page, then add --force to the uninstall command.": "如果这个节点的旧 Core 地址已无法访问,请先在“节点”页移除它,再在卸载命令后加上 --force。",
Expand All @@ -162,6 +162,10 @@ export const sandbox = {
"The command creates the oac-node service user and a system service. It installs no software; if something is missing it stops and says what to install.": "命令会创建 oac-node 服务用户和一个系统服务。它不安装任何软件;缺少什么时会停下并说明要装什么。",
"Set a public address other machines can reach before adding nodes.": "添加节点前,请先设置其他机器能访问的公开地址。",
"Clean up the host": "清理主机",
"Clean up a host": "清理主机",
"A node's service and files stay on its host until it is uninstalled there.": "节点的服务和文件会一直留在它的主机上,直到在主机上卸载。",
"To remove a node's service and files from its host, run this on the host.": "要删除节点在主机上的服务和文件,请在该主机上运行下面的命令。",
"Core must no longer list the node, so remove it on this page first. A host that never registered needs no removal.": "Core 不能再列出该节点,所以请先在本页移除它。从未注册过的主机无需移除。",
"{{name}} is removed from Core. To remove its service and files from the host, run:": "{{name}} 已从 Core 移除。要删除它在主机上的服务和文件,请运行:",
"{{name}} is removed from Core, but its service and files stay on the host.": "{{name}} 已从 Core 移除,但它的服务和文件仍留在主机上。",
"Other machines can't reach this installation's public URL, {{url}}, so no uninstall command can be given.": "其他机器无法访问本安装的公开地址 {{url}},因此无法生成卸载命令。",
Expand Down
6 changes: 4 additions & 2 deletions deploy/node/node_spec.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
import http.client
import json
import re
import shlex
import sys
import urllib.error
import urllib.request
import uuid
Expand Down Expand Up @@ -143,8 +145,8 @@ def fetch(args, token, retained, open_request, allow_enrollment=False, generatio
except urllib.error.HTTPError as error:
if error.code == 401 and retained is not None and not allow_enrollment:
raise SpecificationError("Core no longer accepts this node: it was removed on the Nodes page, or a sandbox "
"deployment change retired it. Uninstall it with node-install.pyz --uninstall "
"--installation-id " + args.installation_id + ", then add the host with a new command.") from None
"deployment change retired it. Uninstall it with: sudo python3 " + shlex.quote(sys.argv[0]) +
" --uninstall --installation-id " + shlex.quote(args.installation_id) + ", then add the host with a new command.") from None
if error.code == 404:
raise SpecificationError("Core node configuration was not found (HTTP 404); route /api/v1 on the Core origin directly to Core, not to Web") from None
if error.code == 409:
Expand Down
7 changes: 7 additions & 0 deletions deploy/node/test_node_spec.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,13 @@ def test_changed_public_url_says_to_generate_a_new_command(self):
with self.assertRaisesRegex(node_spec.SpecificationError, "public URL changed"):
node_spec.fetch(self.args, "once", None, mock.Mock(return_value=self.response()))

def test_retired_node_gets_a_runnable_uninstall_command(self):
rejected = urllib.error.HTTPError("https://core.example", 401, "", {}, None)
with mock.patch.object(node_spec.sys, "argv", ["/home/me/.oac/node-bootstrap/abc.pyz"]):
with self.assertRaises(node_spec.SpecificationError) as raised:
node_spec.fetch(self.args, "", self.retained, mock.Mock(side_effect=rejected))
self.assertIn("sudo python3 /home/me/.oac/node-bootstrap/abc.pyz --uninstall --installation-id " + self.args.installation_id, str(raised.exception))

def test_failures_name_their_cause(self):
for failure, message in ((urllib.error.HTTPError("https://core.example", 404, "", {}, None), "route /api/v1"),
(urllib.error.HTTPError("https://core.example", 409, "", {}, None), "resetting"),
Expand Down
13 changes: 2 additions & 11 deletions docs/getting-started/nodes.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,18 +112,9 @@ The [reset contract](../../contracts/agents-api/sandbox-deployment.md#generation
## Remove a node

1. In Web, open **Nodes** and choose **Remove node** on the node's page, or **Remove** in its list row, then **Confirm removal**. Core refuses while the node still holds sandboxes, snapshots or pending cleanup; let them finish, or archive their Sessions. Removal is permanent: the host can come back only as a new node. Removing every node keeps the sandbox configuration.
2. Web then shows **Clean up the host** with the uninstall command. Run it on the host:
2. Web then shows **Clean up the host** with the uninstall command. Run it on the host. **Clean up a host** on **Nodes** shows the same command at any time, for a host you removed earlier, after a deployment reset retired its node, or for a host that never registered.

```sh
(umask 077; d=$(mktemp -d) || exit; trap 'rm -rf "$d"' EXIT; s=; [ "$(id -u)" -eq 0 ] || s=sudo
printf '\n==> Downloading node installer...\n' &&
curl -fsS --max-time 30 --max-filesize 1048576 'https://core.example/node-install/node-install.pyz' -o "$d/node-install.pyz" &&
printf '==> Verifying node installer...\n' &&
printf '%s %s\n' '<installer-sha256>' "$d/node-install.pyz" | sha256sum -c --status &&
$s python3 "$d/node-install.pyz" ${NO_COLOR+--no-color} --uninstall --installation-id '<installation-id>')
```

Uninstall first asks Core, at the address the node enrolled with, whether the node was removed, and refuses while Core still lists it. When the node enrolled with an address other than the current public URL, the dialog also shows **Old Core address gone?**: if that address no longer responds, it gives the command with `--force`, which skips the check; remove the node on the Nodes page first. Without the dialog, take the installer's SHA-256 from the `node-install.pyz` line of `https://core.example/node-install/SHA256SUMS`. Uninstall stops and removes the service, the node state, the records, the proxy settings and the Docker network. It deletes the `oac-node` account only if the installer created it and no node remains; an adopted account only loses the groups the installer added.
Uninstall first asks Core, at the address the node enrolled with, whether the node was removed, and refuses while Core still lists it. When the node enrolled with an address other than the current public URL, the dialog also shows **Old Core address gone?**: if that address no longer responds, it gives the command with `--force`, which skips the check; remove the node on the Nodes page first. If Core is gone, so is Web: run `sudo python3 ~/.oac/node-bootstrap/<sha256>.pyz --uninstall --installation-id <installation-id> --force` on the host, with the installer the original command left in that directory. The installation ID is in the unit name `oac-node-<installation-id>.service` and in `/etc/oac-node/<installation-id>.json`; Web shows it under **Settings** → **Service and deployment** → **Installation details**. Uninstall stops and removes the service, the node state, the records, the proxy settings and the Docker network. It deletes the `oac-node` account only if the installer created it and no node remains; an adopted account only loses the groups the installer added.

It never deletes Docker images or volumes. It keeps the Runtime image and prints the `sudo docker image rm` command. For microsandbox, once Core has confirmed the removal, it also deletes the node's store under `/var/lib/oac-node/.oac/m/` (about 1.5 GB), and with it the created account. With `--force` Core isn't asked, so the store is kept and the output prints the command that deletes it (`sudo -u oac-node rm -rf <store>`); microVMs may still use it, so check `pgrep -u oac-node` first, then rerun uninstall to finish removing the account. The output lists what was deleted and what was kept. Uninstall can be rerun until it completes.

Expand Down
4 changes: 2 additions & 2 deletions docs/getting-started/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,7 @@ The upgrade leaves the sandbox configuration unchanged: nodes keep serving the s

Three connections require the same protocol version on both sides: Core and its nodes, a node and its microsandbox helper, and Core and the Runtime. The upgrade changes no node host and no running sandbox. A release that changes one of these protocols says so in its release notes and names which of these actions it requires:

- reinstall each node: [remove it](./nodes.md#remove-a-node), clean up the host, then add the host again with **Add node**;
- reinstall each node: [remove it](./nodes.md#remove-a-node), clean up the host, then add the host again with **Add node**. Core refuses to remove a node that still holds sandboxes, so archive those Sessions before you upgrade;
- apply the release's Runtime right after the upgrade; sandboxes on earlier generations stop working;
- [build a new E2B template](https://github.com/MiniMax-AI/OpenAgentCore/blob/main/services/core/deploy/e2b/README.md#build-a-template) and select it in [the sandbox configuration](./nodes.md#change-the-sandbox-configuration);
- reconnect [self-hosted machines](./self-hosted.md#reconnect-an-existing-session).
Expand All @@ -253,7 +253,7 @@ All data goes with it: Projects and API keys, Session history, stored credential

Uninstall stops no sandbox: node sandboxes keep running on their nodes, and E2B sandboxes keep running, and billing, at E2B. While Core is still up, archive their Sessions or [reset the deployment](./nodes.md#change-the-sandbox-configuration) and let it complete.

Nodes on other hosts keep running. To uninstall them the usual way, remove them in Web first, as in [Remove a node](./nodes.md#remove-a-node). After the installation is gone, their Core is gone: on each node host, run the node uninstall command with `--force`, using `node-install.pyz` from the release that installed them and the installation ID that `oac uninstall` printed. The data volume held the ID, so if you didn't note it, read it from the `oac-node-<installation-id>.service` unit on a node host.
Nodes on other hosts keep running. To uninstall them the usual way, remove them in Web first, as in [Remove a node](./nodes.md#remove-a-node). After the installation is gone, their Core is gone: on each node host, run the node uninstall command with `--force`, using the installer the original command left in `~/.oac/node-bootstrap/`, as in [Remove a node](./nodes.md#remove-a-node), and the installation ID that `oac uninstall` printed. If you didn't note it, read it from the node's unit name `oac-node-<installation-id>.service`.

If the installation directory was deleted by hand, its containers keep running and keep the Web port. Find the project and remove it without any installation file:

Expand Down
Loading
Loading