diff --git a/.gitignore b/.gitignore index 82616f0..ce8da70 100644 --- a/.gitignore +++ b/.gitignore @@ -38,3 +38,8 @@ node_modules/ .DS_Store *.log out/ + +# LevelCode in your browser: the release directory scripts/build-web.mjs writes, and the bundle's +# generated assets module (web/ai-extension/build.mjs removes it after bundling). +/dist-web/ +/web/ai-extension/assets.generated.js diff --git a/CLAUDE.md b/CLAUDE.md index 6f15df8..1bc3149 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -40,8 +40,9 @@ extensions/levelcode-hackability/ user init script + Atom/NPP keymap presets + p extensions/levelcode-sync/ 'levelcode' auth provider that lights up the built-in Settings Sync (LevelCode Sync, S0) extensions/levelcode-updater/ notify-only update checker (polls the update feed; never auto-applies) patches/levelcode-core.patch our core source edits, applied on bootstrap -scripts/ bootstrap.sh, apply-branding.mjs, run-dev.sh, editor-identity.mjs, build-macos.sh, make-dmg.sh, make-icon.sh; atom (CLI launcher) + install-level.sh +scripts/ bootstrap.sh, apply-branding.mjs, run-dev.sh, editor-identity.mjs, build-macos.sh, make-dmg.sh, make-icon.sh, build-web.mjs; atom (CLI launcher) + install-level.sh tools/ dependency-free reference servers: sync-server (/v1 Settings-Sync), update-server (/api/update feed) +web/ LevelCode in a browser tab: page entry + callback, scratch-workspace extension, browser build of levelcode-ai, dev server, tests (docs/WEB.md) vscode/ GITIGNORED upstream Code-OSS checkout (generated) ``` @@ -82,6 +83,30 @@ To macOS a run from source and the installed LevelCode used to be ONE app — sa - A LaunchServices handler must live outside temp folders — a bundle under `/tmp` is registered but never chosen. Tests therefore run on fixtures and do not register anything (`test/editorIdentity.test.js`). +## LevelCode in a tab (`web/`) — read `docs/WEB.md` first + +The same `extensions/levelcode-ai` source runs in a browser tab (web-worker extension host). Rules that keep +the desktop app exactly as it was: + +- **A workspace read or write goes through `extensions/levelcode-ai/host.js`**, never `fs` on a workspace path: + in a tab the workspace is whatever the editor's file-system provider says, and every read is asynchronous. + On the desktop each `host` function does what the call site did before, behind `await`. +- **Nothing outside `host.js` asks "am I in a browser?"** — a call site asks for a capability (`host.caps.shell`, + `.mcpStdio`, `.ripgrep`) or calls a function that already knows. A new tool that needs a shell/disk/process is + listed in `NEEDS_SHELL` (agent.js) so the browser neither offers it nor promises it. +- The browser build is `web/ai-extension/` (esbuild + shims for `fs`/`path`/`os`/`crypto`/`child_process`). + Strings that are true on a Mac and false in a tab are rewritten by `web/ai-extension/copy.js`, which fails the + build when the desktop text it expects is gone — change both together. `manifest.js` drops the commands and + settings a tab cannot honour. +- `web/main.js` is a plain ES module with a **top-level `await`**: a `const` declared lower in the file is not + initialised until that await finishes (this cleared a saved session once). Declare constants above the + `try`, and use function declarations for helpers it calls. +- The session store (`createSecretStorage`) belongs to every tab: no cache, one-key read-modify-write under a + Web Lock. The refresh token is rotated on use; do not reintroduce a start-up snapshot. +- Build: `node scripts/build-web.mjs --account ` → `dist-web/`; serve with `node web/serve.mjs --dist dist-web`. + Tests: the unit gate (`scripts/test-extensions.sh` includes `web/test/unit/*.test.js`) and the end-to-end checks in + `web/test/e2e*.mjs` (headless Chrome, stand-in backend; Node 22+). What was not run is listed in `docs/WEB.md`. + ## Toolchain (hard requirements — these bit us) - **Node = `vscode/.nvmrc` (currently 24.15.0)**. Older majors fail to compile native modules. @@ -213,7 +238,7 @@ big-file mode badge. Files: extension.js + fileOps/lineOps/columnOps/encodingEol is not in the editor until you load it: `./scripts/run-dev.sh --extensionDevelopmentPath=/extensions/levelcode-ai` (from the main checkout; the dev extension replaces the built-in one). Uncommitted work is not "on the branch" — checking the branch out somewhere else gets none of it. Run it in the editor before telling anyone to try it. -- Commit `extensions/`, `patches/`, `branding/`, `scripts/`, `docs/`, `PLAN.md`, `CLAUDE.md`. Never commit `vscode/`. +- Commit `extensions/`, `patches/`, `branding/`, `scripts/`, `web/`, `docs/`, `PLAN.md`, `CLAUDE.md`. Never commit `vscode/` or `dist-web/`. - `extensions/levelcode-ai/diagram/` modules listed in `bundle.FILES` are pasted INTO a script block in `chat.html`. They must never contain the text of a script tag or an HTML comment opener — not even in a comment — or the block ends early; `bundle.js` refuses to build if one does. Keep them dependency-free and free of `require('vscode')`/`fs`. diff --git a/README.md b/README.md index f731685..95d2807 100644 --- a/README.md +++ b/README.md @@ -35,7 +35,8 @@ LevelCode is a **clean overlay on top of Code-OSS**, not a vendored copy of the - `extensions/` — the first-party extensions (native AI, Notepad++ pack, themes, hackability, sync, updater), plain JS with no build step. - `branding/` — the LevelCode identity (`product.overlay.json`) + icons, deep-merged onto upstream `product.json`; the extension gallery wired to **Open VSX**. - `patches/` — the small set of core source patches (each tagged `// [LevelCode]`). -- `scripts/` — bootstrap / run-dev / build / dmg / icon. +- `scripts/` — bootstrap / run-dev / build / dmg / icon, and `build-web.mjs` (LevelCode in a browser tab). +- `web/` — LevelCode in your browser: the same workbench and AI extension as a static site you can host; sign in with a LevelCode account or use your own key. See [`docs/WEB.md`](docs/WEB.md). - `tools/` — dependency-free reference servers (Settings-Sync feed, update feed). - `PLAN.md`, `docs/`, `CLAUDE.md` — the vision, roadmap, and repo map. diff --git a/docs/WEB.md b/docs/WEB.md new file mode 100644 index 0000000..f0e8efa --- /dev/null +++ b/docs/WEB.md @@ -0,0 +1,342 @@ +# LevelCode in your browser + +The same workbench, the same `levelcode-ai` extension, in a browser tab: sign in with a LevelCode +account and chat with the agent against the gateway, or bring a provider key. There is nothing to +install and no server of ours that runs code — the editor is a **static directory**. + +This page is the engineering reference: how it fits together, how to build and serve it, what the +host has to provide, the security model, and — kept separate on purpose — what has and has not been +run. For the product's own page and copy see the account site (`/web`, `/docs/browser`). + +- [What it is, and what it is not](#what-it-is-and-what-it-is-not) +- [How it fits together](#how-it-fits-together) +- [Signing in](#signing-in) +- [Build and run](#build-and-run) +- [Deploying](#deploying) — origins, headers, backend settings +- [Security model](#security-model) +- [Where state lives](#where-state-lives) +- [Keeping the desktop and the browser in step](#keeping-the-desktop-and-the-browser-in-step) +- [Tests](#tests) — and [what was not run](#what-was-not-run) +- [Decisions that are the owner's](#decisions-that-are-the-owners) + +## What it is, and what it is not + +A Code-OSS **web client** (the `vscode-web-min-ci` build of the pinned checkout) with a +**web-worker extension host**, plus LevelCode's own extensions loaded as built-ins. + +| Works in a tab | Stays in the Mac app | +| --- | --- | +| The workbench: explorer, editor tabs, themes, search, Quick Open | A terminal | +| Chat, inline completion, the agent: read / search / create / edit / delete files | `run_command`, background commands, auto-verify commands | +| Keep / Undo review of the agent's edits | MCP servers that start local processes | +| Sign in with a LevelCode account (gateway), **or** your own provider key | Installing extensions (no marketplace in the page) | +| A scratch workspace saved in the browser (every modern browser) | Importing another editor's settings, Settings Sync, the update check | +| A folder from your computer (File System Access API: Chrome, Edge) | The Notepad++ pack and the customization pack (not loaded in the page), Agent Sketch | + +The browser build offers the agent **only the tools it can honour**: with no shell the shell tools +are withheld from the model rather than offered and failing, and the system prompt is derived from +the same lines so it never promises one (`host.caps`, see below). + +## How it fits together + +``` + account site (levelcode.ai) editor origin (any static host) + ┌───────────────────────────┐ tab goes to ┌──────────────────────────────────────────────┐ + │ /ai/login (sign-in) │◀─────────────────│ index.html ← build-addressed /_//… │ + │ /api/levelcode/v1/… │──── ?code= ─────▶│ main.js embedder: config, callbacks, │ + │ auth/exchange │ /callback.html │ secrets, workspace, commands │ + │ account/models │ │ callback.html hands the result back (storage)│ + │ ai/chat/completions │◀──── fetch ──────│ ┌─ workbench (Code-OSS web client) ────────┐ │ + └───────────────────────────┘ (CORS) │ │ web-worker extension host (iframe) │ │ + │ │ levelcode-ai esbuild bundle + shims │ │ + provider you chose ◀──── fetch (BYOK) ──────│ │ levelcode-web scratch workspace │ │ + │ │ chat webview (own origin, see below) │ │ + │ └──────────────────────────────────────────┘ │ + └──────────────────────────────────────────────┘ +``` + +| Path | What it is | +| --- | --- | +| `scripts/build-web.mjs` | One command → `dist-web/`: the client, the extensions, `index.html` with the configuration baked in, `_headers`, an nginx fragment, `build.json`. Changes nothing outside `--out`. | +| `web/main.js`, `index.html`, `boot.css` | The page. Plain ES module, no build step: reads `#levelcode-web-config`, creates the workbench, supplies the URL-callback provider, the secret store, the workspace provider and the `levelcode.web.openAuthUrl` command. | +| `web/callback.html`, `callback.js` | The page a sign-in returns to. | +| `web/lib/config.mjs`, `headers.mjs`, `extensions.mjs` | The configuration, the HTTP policy and the extension staging — shared by the dev server and the release, so what is tested is what ships. | +| `web/ai-extension/` | The browser build of `levelcode-ai`: `build.mjs` (esbuild, `platform: browser`), `shims/` (`fs`, `path`, `os`, `crypto`, `child_process`, `process`), `manifest.js` (the desktop manifest minus what a tab cannot do), `copy.js` (the few strings that are true on a Mac and false in a tab). | +| `extensions/levelcode-ai/host.js` | The one place the desktop/browser difference lives in the extension (see below). | +| `web/workspace/` | The `levelcode-web` extension: the **scratch workspace** (`levelcode-scratch:` file system in IndexedDB, with file and text search), and the commands to open a local folder or the scratch. | +| `web/serve.mjs` | Dev / end-to-end server. Serves a build (`--dist`) with its own `_headers`, or the pieces (`--static`, `--extensions`). | +| `web/test/` | Unit tests (`unit/`) and the end-to-end checks (`e2e*.mjs`) with a hand-written CDP driver and a stand-in backend. | + +### The extension runs unchanged — through `host.js` + +`extensions/levelcode-ai` is the **same source** on both. Its Node-only assumptions are handled in +two places only: + +1. **`web/ai-extension/shims/`** — the build swaps `fs`, `path`, `os`, `crypto`, `child_process` + for small in-memory/IndexedDB stand-ins for the extension's *private* state (sessions, memory, + images). `path` and `crypto` are compared against Node's over many inputs. +2. **`extensions/levelcode-ai/host.js`** — the user's *workspace* is not in the shim and must never + be: in a tab it is whatever the editor's file-system provider says, and every read is + asynchronous. `host` gives `readText`, `readBytes`, `stat`, `exists`, `isBinary`, `findFiles`, + `withReads`, `openAuth` and the capability flags `caps.shell`, `caps.mcpStdio`, `caps.ripgrep`. + On the desktop each function does exactly what the call site did before, behind `await`. + +The two rules that keep the desktop app exactly as it was: + +- **On the desktop every `host` function does what the call site did before** — the same `fs` + call, the same error. +- **Nothing outside `host.js` asks "am I in a browser?".** A call site asks for a capability + (`host.caps.shell`) or calls a function that already knows. + +## Signing in + +The extension's existing PKCE flow, with the browser's pieces in place of the OS's: + +``` +1 extension accountSignIn(): verifier → SecretStorage; builds + /ai/login?redirect_uri=&code_challenge= +2 extension host.openAuth(uri) → command levelcode.web.openAuthUrl (registered by main.js) +3 main.js checks the address is /ai/…, marks the tab (sessionStorage), location.assign +4 account already signed in there → mints a one-time code bound to the challenge, 302 to + /callback.html?vscode-reqid=N&vscode-scheme=levelcode + &vscode-authority=levelcode.levelcode-ai&vscode-path=%2Fauth%2Fcallback&code=… + (not signed in → the login page first) +5 callback writes localStorage "vscode-web.url-callbacks[N]" (dated), sees the mark → goes back to / +6 main.js at start-up, once the workbench is listening, delivers a fresh result as + levelcode://levelcode.levelcode-ai/auth/callback?code=… to the URL service +7 extension handleAuthCallback → POST /api/levelcode/v1/auth/exchange {code, verifier} → tokens +``` + +Why the tab leaves instead of opening a pop-up: a window opened from a chain that began in a +webview is the first thing a strict browser (Safari, a phone) refuses. Leaving the tab and +returning needs no permission. The PKCE verifier lives in SecretStorage, so it survives the +navigation. + +- **`?signin=1`** (the account site's "Open LevelCode" link) makes step 1 happen by itself, once: + `main.js` removes the parameter, then delivers `levelcode://…/launch`, which focuses the chat and + starts the sign-in only when there is no session. +- **A result is delivered only if it is fresh** (written in the last two minutes — the one-time + code lives for a minute) and is removed either way. One left by a callback page whose editor + tab is gone is delivered when the editor next starts. +- **`levelcode.web.openAuthUrl` is an open-redirect if it is loose**, and any extension in the page + can call it. It goes only to `/ai/…`. +- The address handed over is `encodeURI(uri.toString(true))` — the string the editor's own opener + produces — so the server sees the same address however the sign-in was started. The backend + decodes `redirect_uri` until stable (three rounds at most) and then applies the exact rule in + `Levelcode::EditorCallback`. + +## Build and run + +Needs the bootstrapped Code-OSS checkout (`scripts/bootstrap.sh`, Node 24.x as the pin requires). + +```bash +# the release: one static directory (about a minute for the client, then bundling) +node scripts/build-web.mjs --account https://levelcode.ai --out dist-web + +# …serve it exactly as a static host would, with its own _headers +node web/serve.mjs --dist dist-web # http://127.0.0.1:8800 +``` + +| `build-web.mjs` flag | Meaning | +| --- | --- | +| `--account ` (`LEVELCODE_WEB_ACCOUNT`) | The LevelCode Cloud origin: where sign-in happens, and where the gateway is. Baked into `index.html`. | +| `--api-url ` (`LEVELCODE_WEB_API_URL`) | Only when the API is not on the account origin (production has one host for both): a local account site with the backend behind a tunnel, say. The gateway must be `https`. | +| `--webview-origin 'https://{{uuid}}.view.example.com'` (`LEVELCODE_WEB_WEBVIEW_ORIGIN`) | Self-host the webview origin. Without it the page uses Code-OSS's CDN at this build's own commit. Needs wildcard DNS and TLS — see below. | +| `--ext-host-origin 'https://{{uuid}}.ext.example.com'` (`LEVELCODE_WEB_EXTHOST_ORIGIN`) | Run the extension host on an origin of its own. Optional while no third-party code is loaded. | +| `--static ` | Use a prebuilt client (skips the gulp build). | +| `--id ` | Override the build id (default: a fingerprint of the client, the page and the extension). | + +**Dev server** (`web/serve.mjs`): `--static --extensions --extension-names +levelcode-ai,levelcode-web --account `, plus `--product ` and `--config-defaults +` to override the product configuration. The extension is built with +`node web/ai-extension/build.mjs --out --vscode `. + +**Release layout.** `index.html` and the callback page are at the root, un-cached. Everything +else is under `/_//` and cached for a year (`immutable`): `static/` (the client), `extensions/`, +`main.js`, `boot.css`. A cold first load fetched about 20 MB uncompressed (5 MB gzip) over 147 +files from the editor's origin, plus the webview host page from the CDN; the whole directory is +about 195 MB, most of it grammars and extensions the page does not request until they are used. +(Measured as the size of the files the page asked for, not as bytes on the wire.) + +## Deploying + +It is a static site. What the host must supply: + +### Three origins, and which is optional + +| Origin | Default | Why it is separate | +| --- | --- | --- | +| **Editor** — `index.html`, `callback.html`, `/_/…` | the host you deploy to | The only origin that holds the session. | +| **Webview** — where the chat panel (and every webview) runs | `https://{{uuid}}.vscode-cdn.net`, Code-OSS's CDN at this build's own commit | Webview content must not share the editor's origin. **Each webview needs its own subdomain**, so a self-hosted origin has to be a wildcard (`*.view.example.com` with a certificate for it); a single host fails with "Expected '' as hostname or subdomain". | +| **Extension host** — the iframe the worker runs in | the editor's own origin | Isolating it (`--ext-host-origin`, a wildcard likewise) puts third-party extension code on a different origin than the session. Not needed while only LevelCode's extensions load. When it is on, **every** `fetch` an extension makes carries that origin, so the API's CORS must allow it. | + +### Headers + +`dist-web/_headers` (Netlify, Cloudflare Pages) and `dist-web/deploy/nginx.conf` carry the same +policy; `web/serve.mjs --dist` reads the same file, so the end-to-end checks run under it. The +Content-Security-Policy for the editor is the tightest the workbench allows: + +- `script-src 'self' 'unsafe-eval' blob:` — Monaco and the TextMate/WASM engines compile code at runtime; +- `connect-src 'self' https: http://localhost:* http://127.0.0.1:*` — a provider key is used from the tab, to the + provider the user chose; loopback is the one place a browser allows plain http from an https page, for a + model server on the user's own machine (an OpenAI-compatible endpoint through the custom provider — *not run*); +- `frame-src 'self' [] data:`; `object-src 'none'`, `base-uri 'none'`, + `form-action 'none'`, `frame-ancestors 'none'`. + +The callback page has `default-src 'none'` and only its own script and style. + +### On the account site + +Three settings on the Rails app (all default **off**; with none set nothing changes): + +| Variable | Meaning | +| --- | --- | +| `LEVELCODE_WEB_EDITOR_ORIGINS` | Comma list of origins the editor is served from, exactly (`https://editor.example.com`). A sign-in code is only ever sent to `/callback.html` with the right parameters. Also the CORS list. | +| `LEVELCODE_WEB_EDITOR_URL` | Where the account site's "Open LevelCode" goes: one of those origins and, optionally, a path. | +| `LEVELCODE_WEB_EXTENSION_HOST_ORIGINS` | Comma list (one wildcard each) of origins the extension host runs on, when isolated. CORS only — never a callback. | + +`GET /api/levelcode/v1/web_editor` answers `{ enabled, url }`; the account SPA shows its entry +points only when it is on. + +**Do not host the editor on a subdomain of the account host.** The account session cookie is +`SameSite=Lax`, which is a site boundary, not an origin one: it is sent on a request from a sibling +subdomain, and the API takes that cookie when there is no bearer token. A page that runs other +people's extensions there could cause credentialed requests (CORS stops it *reading* the answer, not +causing the effect). Use another registrable domain. + +## Security model + +- **The session is the user's tokens.** They are in SecretStorage, which in a tab is sealed with + AES-GCM under a **non-extractable** key kept in IndexedDB, and stored in localStorage. That keeps + tokens out of a localStorage dump, a backup or a copied profile. It is **not** a defence against + script running on the editor's origin, which can use the same key — so **the page loads no + third-party extension** (`extensionsGallery: null`) and nothing is installed into its origin. +- **The store belongs to every tab.** The refresh token is rotated on use; a tab that cached it + would present a token another tab had rotated away and end a session the other just renewed. Reads + go to storage each time and a change is read-modify-write of one key under a Web Lock. +- **One command, one destination** — `levelcode.web.openAuthUrl` (above). +- **A sign-in code is only handed to a listed origin, to the exact callback page**, by the backend; + the page then takes it out of the address bar (`history.replaceState`) and storage. +- **Keys of the user's own** (BYOK) are stored the same way and used from the tab to the provider the + user chose; no LevelCode call is involved (the end-to-end check asserts it). The provider must + accept requests from a web page; Anthropic's is called with its browser-access header. +- **The scratch workspace and the extension's private state are on the extension host's origin**, in + IndexedDB. With the host isolated they are not readable from the editor's page, by design. + +## Where state lives + +| What | Where | Notes | +| --- | --- | --- | +| Session tokens, BYOK keys | localStorage `levelcode-web.secrets.v1` (sealed); the key in IndexedDB `levelcode-web` → `keys` | editor origin; shared by tabs | +| Sign-in result in flight | localStorage `vscode-web.url-callbacks[N]`; sessionStorage `levelcode-web.return` | removed when read | +| Scratch workspace | IndexedDB `levelcode-scratch` → `nodes` | extension host origin; changes are broadcast to other tabs | +| Chat sessions, memory, images | IndexedDB `levelcode-ai-fs` → `files`, loaded at start | extension host origin; a change made in another tab shows after a reload | +| Editor settings, layout | the workbench's own storage | editor origin | + +Clearing the browser's site data removes all of it. A private window may discard it on close; the +secret store then falls back to memory for the session. + +## Keeping the desktop and the browser in step + +- **A new tool that needs a shell, a disk or a local process** declares it: add the tool name to + `NEEDS_SHELL` (or gate on `host.caps`) in `agent.js` so the browser neither offers nor promises it. + The browser system prompt is built from the same `SYSTEM_LINES`. +- **A new workspace read or write goes through `host`.** `fs` on a workspace path would work on the + desktop and quietly do nothing in a tab. `extensions/levelcode-ai/test/` and `web/test/unit/` check + the places this has bitten. +- **Copy that is true on a Mac and false in a tab** (keychain, "your machine") is rewritten by + `web/ai-extension/copy.js`, which **fails the build** when the desktop text it expects is gone. Update + the desktop and `copy.js` together. +- **`web/ai-extension/manifest.js`** removes the commands and settings a tab cannot honour. A new + command that shells out belongs in `REMOVED_COMMANDS`. +- **A Code-OSS bump** can change the web client's packaging and the webview service worker's version; + rebuild and run the end-to-end checks. The webview template names this build's own commit + (`{{quality}}/{{commit}}`) for that reason. + +## Tests + +| Check | Command | What it proves | +| --- | --- | --- | +| Unit gate | `./scripts/test-extensions.sh` | Includes `web/test/unit/*.test.js`: the shims against Node, `host.js` on both sides, `main.js`'s functions (address check, delivery, the shared secret store, `?signin=1`), the manifest, the copy rewrites. Runs in CI on every PR (`test.yml`). Also run on Linux / Node 18 in a container with the network refused. | +| End to end | `node web/test/e2e.mjs --dist dist-web --stub-port ` | Real headless Chrome: boot from static files, the extension activates in the worker, sign in through the real callback page, agent tool calls (write / list / search / read / edit) against the scratch workspace, Quick Open, Keep and Undo, a reload that keeps the session and files. | +| Same-tab sign-in | `node web/test/e2e-signin-tab.mjs --dist dist-web --stub-port ` | No second window; verifier survives the reload; no repeat; no sign-in for a signed-in editor; a second tab uses the same session; a stranded result is delivered, an old one dropped. | +| Bring your own key | `node web/test/e2e-byok.mjs --static … --extensions …` | The request goes from the tab to the provider with the user's key and nothing of LevelCode's. | + +The end-to-end checks use a **stand-in backend** (`web/test/stub-backend.mjs`, HTTPS with a +throwaway certificate) that applies the same redirect and CORS rules the real one must. They prove +the editor's half of the contract; the backend's half is its own specs. The hand-written CDP driver +needs Node 22+ and a Chrome (`CHROME=` to point at one). + +The checks were run against a release served with its own `_headers` (same-origin extension host, the +webview from Code-OSS's CDN), and against the development server with the webview and the extension +host on origins of their own (`--product` with `webviewContentExternalBaseUrlTemplate` and +`webEndpointUrlTemplate`). + +### Trying it yourself, against a real backend + +The repo's checks use a stand-in for the account site. To use the real one, locally: + +1. **Backend** — the thin.ly branch with the web edition, its usual dev server and https tunnel, plus + `LEVELCODE_WEB_EDITOR_ORIGINS=http://localhost:8800` and `LEVELCODE_WEB_EDITOR_URL=http://localhost:8800/`. + (The gateway refuses plain http, so the editor's API host has to be the tunnel.) +2. **Account site** — the onetime branch: `npm run dev` (it proxies to the backend), at `http://localhost:5173/ai`. +3. **Editor** (from the levelcode repo root; the build needs the bootstrapped `vscode/` checkout, and a git + worktree has none, so pass `--vscode ` there): + + ```bash + export API_URL=https://your-tunnel.example.com # the https address of your backend + node scripts/build-web.mjs --account http://localhost:5173 --api-url "$API_URL" --out dist-web + node web/serve.mjs --dist dist-web # http://localhost:8800 + ``` + +4. **What to do, and what you should see:** + - Signed out everywhere: open `http://localhost:8800/?signin=1`. The tab goes to the account site's + sign-in page ("Connect the editor to your account…"). Sign in with an email code. The tab comes back to + the editor with a clean address, a "Signed in to LevelCode." message, and the model and plan in the chat footer. + - Signed in on the account site first: open `http://localhost:5173/ai/account` and use **Open in browser**. + The editor opens, the tab flashes through the login page ("Opening LevelCode…") and is back signed in, + with nothing typed. + - Ask the agent: *Create a file named hello.txt containing: hello.* The file appears in the Explorer. + - **LevelCode: Open Folder from Your Computer…** (palette) in Chrome or Edge: pick a folder, ask the agent + to read a file of it and write another. + - A second tab of the editor opens already signed in (it shares the session). + +Look at the browser's network panel while you do it: every call to the API should be `200`, with no CORS +errors in the console. A CORS error means the editor's origin is not in `LEVELCODE_WEB_EDITOR_ORIGINS`. + +This was run once, by a hand-driven script, against Rails from the thin.ly branch in test mode (email and +Stripe stubbed, a throwaway database): email-code sign-in from the editor, sign-in from an already +signed-in browser, a gateway chat, the agent writing a file, and a folder opened through a real +`FileSystemDirectoryHandle` (the origin-private file system stands in for the OS picker). That script needs +the other repository's internals and is not part of this repo's checks. + +### What was not run + +Say these out loud rather than discover them in production: + +- **A real browser other than headless Chrome.** Safari and Firefox are untested; so are phones. +- **The operating system's folder picker itself.** It needs a person. Everything after it — the command, the + workbench's `file` provider on a real `FileSystemDirectoryHandle`, the agent's list, read and write — is run + (`web/test/e2e.mjs` section 5), with the origin-private file system standing in for the picker. Permission + handling on a *persisted* handle after a reload (the browser asks again) is the browser's and was not run. +- **A production account site.** The real sign-in page and Rails were run locally (above), not the deployed + ones; the backend's rules are its own specs. +- **Another provider's CORS.** Anthropic's browser-access header is set; whether a given provider + answers a page's request is that provider's. +- **Production hosting:** DNS, a wildcard certificate, the CDN in front, the real headers on the real host. +- **Long sessions across the 8-hour access-token boundary in a tab** (the logic is the desktop's, + unchanged; the shared store was added for it and is unit-tested, not soaked). + +## Decisions that are the owner's + +1. **The editor's domain.** Not a subdomain of the account host (see Deploying). Which registrable + domain, and who holds it. +2. **The webview origin.** Keep Code-OSS's CDN (no setup; a dependency on Microsoft's hosting of the + published page for this build's commit) or self-host a wildcard origin. +3. **Isolating the extension host.** Worth it before any third-party extension is ever allowed; not + needed while the page loads only LevelCode's. +4. **Where the entry points point.** Today every "Open LevelCode" link carries `?signin=1`, so the + editor starts the sign-in itself. Whether visitors who are not signed in at the account site + should see the editor first instead. +5. **Support statement.** Chrome and Edge for folders; scratch workspace everywhere else; phones + unstated until run. diff --git a/extensions/levelcode-ai/agent.js b/extensions/levelcode-ai/agent.js index de92aac..8568893 100644 --- a/extensions/levelcode-ai/agent.js +++ b/extensions/levelcode-ai/agent.js @@ -14,6 +14,7 @@ const fs = require('fs'); const path = require('path'); const cp = require('child_process'); const os = require('os'); // MCP servers need a cwd even when no folder is open +const host = require('./host'); // the machine we run on: workspace I/O is async here, and capabilities differ (browser) const providers = require('./providers/index'); const { formatVerifyFeedback, verifyOutcome, looksUnrunnable, sniffPort, sniffPreviewUrl, looksReady } = require('./verify'); const { classifyCommand, dangerLabel } = require('./commandSafety'); @@ -24,7 +25,7 @@ const { connectAll, getServer } = require('./mcpClient'); const diagramTool = require('./diagram/tool'); const diagramRepair = require('./diagram/repair'); -const SYSTEM_BASE = [ +const SYSTEM_LINES = [ "You are LevelCode's built-in autonomous coding agent. You accomplish the user's goal in their", 'workspace using the provided tools. Rules:', '- Be DECISIVE and FAST. Read only the file you are changing (plus at most 1 other if truly needed), then ACT.', @@ -44,7 +45,24 @@ const SYSTEM_BASE = [ '- You have SKILLS — short expert playbooks for common task types, listed under "Available skills" below with a name and a one-line description. When the user\'s goal clearly matches a skill\'s description (e.g. reviewing a diff, fixing one reported bug, writing tests), call use_skill with that exact name FIRST, before other tools, and follow the steps it returns. Pick at most one; if nothing clearly fits, just proceed normally. Do not mention skills to the user.', '- When the goal is finished, end with a line starting with "Done:" — one crisp sentence of what was accomplished. For a substantial run (several files, a tricky diagnosis, or a decision the user should know about), follow that line with a SHORT wrap-up in plain lines: what landed, how you verified it, anything left open or worth pushing on. For a trivial goal the "Done:" line alone is right. Then STOP (no more tools).', '- After you finish, your work is verified automatically: editor diagnostics for the files you changed (plus a verify command, if the user configured one) are checked. If problems are found you will get them back as a follow-up message — fix the ones you introduced, then finish again with "Done:". If a reported problem is clearly pre-existing and unrelated to the goal, do not chase it: note it in one line and finish.' -].join('\n'); +]; +const SYSTEM_BASE = SYSTEM_LINES.join('\n'); + +// The same prompt for a host that cannot run programs (LevelCode in the browser). Derived from +// SYSTEM_LINES rather than written out again so the two cannot drift: lines about commands go, the +// two mixed lines lose their command sentences, and one line says plainly what is missing. +const BROWSER_NOTE = '- You are running in LevelCode in the user\'s browser. There is no terminal: you cannot run commands, tests, installs or servers, and there is no run_command tool. Read, search, create, edit and delete files, and when something must be run, say so in one line and tell the user the exact command to run on their own computer.'; +function browserLines(lines) { + return lines + .filter((l) => !/run_command background:true/.test(l)) + .map((l) => l + .replace(' Only run_command still needs approval; if the user skips a command, adapt or stop.', '') + .replace('EVERY run_command, read_file and search MUST include', 'EVERY read_file and search MUST include') + .replace(' ("Run the extension unit tests", "Find the insertion point in section 10", "Read the runAgent call site")', ' ("Find the insertion point in section 10", "Read the runAgent call site")') + .replace(' run_command accepts an optional "folder" to pick which folder it runs in.', '')) + .concat([BROWSER_NOTE]); +} +const SYSTEM_BASE_HOST = host.caps.shell ? SYSTEM_BASE : browserLines(SYSTEM_LINES).join('\n'); const TOOLS = [ { name: 'list_files', description: 'List workspace files (optional glob like "**/*.js"). Excludes node_modules/.git/build dirs.', input_schema: { type: 'object', properties: { glob: { type: 'string' } } } }, @@ -62,14 +80,14 @@ const TOOLS = [ // Rough token estimates (chars/4) for the static prompt segments, so the context popover can break // down "what's filling the window" — system + tools are sent on every request, the rest is messages. -const SYSTEM_TOKENS_EST = Math.round(SYSTEM_BASE.length / 4); +const SYSTEM_TOKENS_EST = Math.round(SYSTEM_BASE_HOST.length / 4); /** Append an "Available skills" name+description menu to the base prompt. Bodies NEVER go here — * progressive disclosure means only name+description are ever in the prompt (~100 words for a dozen). */ function buildSystem(menu) { - if (!menu || !menu.length) { return SYSTEM_BASE; } + if (!menu || !menu.length) { return SYSTEM_BASE_HOST; } const list = menu.map((s) => '- ' + s.name + ': ' + s.description).join('\n'); - return SYSTEM_BASE + '\n\nAvailable skills (call use_skill with the name):\n' + list; + return SYSTEM_BASE_HOST + '\n\nAvailable skills (call use_skill with the name):\n' + list; } // The tools that resolve a PATH or a CWD against the workspace root. With no folder open they have // nothing to resolve against, so they are withheld from the model rather than offered and left to fail @@ -87,7 +105,12 @@ const NEEDS_ROOT = new Set([ ]); const PORTABLE_TOOLS = TOOLS.filter((t) => !NEEDS_ROOT.has(t.name)); -const TOOLS_TOKENS_EST = Math.round(JSON.stringify(TOOLS).length / 4); +// A host that cannot start programs (LevelCode in the browser) never offers the shell tools: a tool +// that is present but always errors is worse than one that is absent. The tool list is narrowed where +// a run assembles it (see `withheld` in runAgent); this is the estimate for the list that is sent. +// PORTABLE_TOOLS never contains them, since both are in NEEDS_ROOT. +const NEEDS_SHELL = new Set(['run_command', 'read_command_output']); +const TOOLS_TOKENS_EST = Math.round(JSON.stringify(host.caps.shell ? TOOLS : TOOLS.filter((t) => !NEEDS_SHELL.has(t.name))).length / 4); // The same estimate for the rootless list, and it has to exist separately rather than be derived at // call time: the plain path deliberately never re-stringifies (see toolsTokensEst below), so without a // second constant a rootless run reports the FULL schema cost for a list it never sent — about 1000 @@ -116,6 +139,7 @@ function rgPath() { return cands.find((c) => { try { return fs.existsSync(c); } catch { return false; } }) || null; } function rgSearch(term, cwd) { + if (!host.caps.ripgrep) { return scanSearch(term, cwd); } return new Promise((resolve) => { const bin = rgPath(); if (!bin || !cwd) { resolve(''); return; } @@ -131,6 +155,48 @@ function rgSearch(term, cwd) { }); } +/** + * The same search without ripgrep: list the workspace's files and scan them. Output matches rg's + * (`./path:line:text`, case-insensitive literal, 20 matches per file, files over 1 MB skipped, capped + * at 6000 characters) so the callers that prefix folder names and the model's habits need no change. + * Used where there is no binary to run (LevelCode in the browser). + */ +async function scanSearch(term, root) { + if (!term || !root) { return ''; } + const needle = term.toLowerCase(); + const out = []; + let size = 0; + let uris = []; + try { uris = await host.findFiles('**/*', FILE_EXCLUDES, 2000); } catch { return ''; } + const rootPrefix = root.endsWith('/') ? root : root + '/'; + for (const uri of uris) { + if (size >= 6000) { break; } + const abs = uri.path; + if (abs !== root && !abs.startsWith(rootPrefix)) { continue; } // another workspace folder's file + let text; + try { + const st = await host.stat(abs); + if (!st || !st.isFile || st.size > 1024 * 1024) { continue; } + const bytes = await host.readBytes(abs); + if (bytes.subarray(0, 8000).includes(0)) { continue; } // binary, as rg would skip it + text = new TextDecoder('utf-8').decode(bytes); + } catch { continue; } + const rel = './' + abs.slice(rootPrefix.length); + let hits = 0; + const lines = text.split('\n'); + for (let i = 0; i < lines.length && hits < 20; i++) { + if (lines[i].toLowerCase().includes(needle)) { + const line = rel + ':' + (i + 1) + ':' + lines[i].replace(/\r$/, ''); + out.push(line); + size += line.length + 1; + hits++; + if (size >= 6000) { break; } + } + } + } + return out.join('\n').slice(0, 6000); +} + // ---- readable line diff ---------------------------------------------------- /** LCS diff of two small line arrays → [{type:'ctx'|'add'|'del', text}]. */ function lcsDiff(a, b) { @@ -185,7 +251,8 @@ function workspaceRoot() { } function safeJoin(root, rel) { const pth = path.resolve(root, rel); - if (pth !== root && !pth.startsWith(root + path.sep)) { return null; } + const prefix = root.endsWith(path.sep) ? root : root + path.sep; // a root of "/" is its own prefix + if (pth !== root && !pth.startsWith(prefix)) { return null; } return pth; } /** Resolve a model-supplied workspace-relative path to an absolute path, multi-root aware. @@ -197,6 +264,22 @@ function safeJoin(root, rel) { * the primary folder, then every other folder. Returns null if nowhere. * mustExist=false (create): the folder-name prefix targets that folder; else the primary. */ function resolveWorkspacePath(rel, opts) { + const r = resolveCandidates(rel, opts); + if (!r) { return null; } + if (!r.mustExist) { return r.create; } + for (const c of r.candidates) { if (c && fs.existsSync(c)) { return c; } } + return null; +} +/** The same resolution for a host that can only ask whether a file exists asynchronously. */ +async function resolveWorkspacePathAsync(rel, opts) { + const r = resolveCandidates(rel, opts); + if (!r) { return null; } + if (!r.mustExist) { return r.create; } + for (const c of r.candidates) { if (c && await host.exists(c)) { return c; } } + return null; +} +/** Where `rel` could be: the path to create it at, and the paths that count as "exists", in order. */ +function resolveCandidates(rel, opts) { const mustExist = !!(opts && opts.mustExist); const folders = workspaceFolderList(); if (!folders.length) { return null; } @@ -205,11 +288,9 @@ function resolveWorkspacePath(rel, opts) { const named = folders.length > 1 ? folders.find((f) => f.name === seg) : null; const namedAbs = named ? safeJoin(named.root, rel.slice(seg.length).replace(/^[\\/]+/, '')) : null; const primaryAbs = safeJoin(folders[0].root, rel); - if (!mustExist) { return namedAbs || primaryAbs; } const candidates = [namedAbs, primaryAbs]; for (const f of folders.slice(1)) { candidates.push(safeJoin(f.root, rel)); } - for (const c of candidates) { if (c && fs.existsSync(c)) { return c; } } - return null; + return { mustExist, create: namedAbs || primaryAbs, candidates }; } /** "file not found" help for the model — names the folders it can address in a multi-root workspace. */ function whereHint() { @@ -259,9 +340,7 @@ function inputPreview(input, redact) { } /** Cheap binary sniff — a NUL byte in the first 8KB means it isn't text we should round-trip as UTF-8. */ -function isBinaryFile(abs) { - try { const buf = fs.readFileSync(abs); const n = Math.min(buf.length, 8000); for (let i = 0; i < n; i++) { if (buf[i] === 0) { return true; } } return false; } catch { return false; } -} +function isBinaryFile(abs) { return host.isBinary(abs); } // Run a shell command, STREAMING its output: onChunk(text, 'stdout'|'stderr') fires as it runs (live // terminal in the chat), onExit(code, ms, how) at the end. onStart(child, stop) exposes a stop() that @@ -333,10 +412,10 @@ async function runTool(tu, ctx) { // label/path ride alongside the legacy `text` so the chat can title this action with the // model's own words ("Read the runAgent call site") and still attribute it to a file. ctx.post({ type: 'agentTool', icon: 'file', text: 'read ' + input.path, label: input.explanation || '', path: input.path, kind: 'read' }); - const abs = resolveWorkspacePath(input.path || '', { mustExist: true }); + const abs = await resolveWorkspacePathAsync(input.path || '', { mustExist: true }); if (!abs) { return 'ERROR: file not found: ' + input.path + whereHint(); } - if (isBinaryFile(abs)) { return 'ERROR: ' + input.path + ' looks like a binary file — not reading it as text.'; } - let body = fs.readFileSync(abs, 'utf8').replace(/^/, ''); // drop BOM so old_str matches cleanly + if (await isBinaryFile(abs)) { return 'ERROR: ' + input.path + ' looks like a binary file — not reading it as text.'; } + let body = (await host.readText(abs)).replace(/^/, ''); // drop BOM so old_str matches cleanly if (body.length > 100 * 1024) { body = body.slice(0, 100 * 1024) + '\n…(truncated)…'; } return body; } @@ -362,10 +441,10 @@ async function runTool(tu, ctx) { return 'Plan updated (' + done + '/' + todos.length + ' done).'; } if (tu.name === 'edit_file') { - const abs = resolveWorkspacePath(input.path || '', { mustExist: true }); + const abs = await resolveWorkspacePathAsync(input.path || '', { mustExist: true }); if (!abs) { return 'ERROR: file not found: ' + input.path + whereHint() + ' (use write_file to create it)'; } - if (isBinaryFile(abs)) { return 'ERROR: ' + input.path + ' looks like a binary file — refusing to edit it as text.'; } - const cur = fs.readFileSync(abs, 'utf8'); + if (await isBinaryFile(abs)) { return 'ERROR: ' + input.path + ' looks like a binary file — refusing to edit it as text.'; } + const cur = await host.readText(abs); const oldStr = String(input.old_str || ''); if (!oldStr) { return 'ERROR: old_str is empty.'; } const res = applyStringEdit(cur, oldStr, String(input.new_str || '')); @@ -379,12 +458,12 @@ async function runTool(tu, ctx) { if (tu.name === 'write_file') { const abs = resolveWorkspacePath(input.path || ''); if (!abs) { return 'ERROR: path is outside the workspace' + whereHint(); } - const existed = fs.existsSync(abs); + const existed = await host.exists(abs); let newStr = String(input.content || ''); if (existed) { - if (isBinaryFile(abs)) { return 'ERROR: ' + input.path + ' looks like a binary file — refusing to overwrite it.'; } - try { if (fs.statSync(abs).size > 100 * 1024) { return 'ERROR: ' + input.path + ' is large (>100KB) and you only saw the first 100KB on read. Use edit_file for targeted changes — a full write_file risks dropping content you did not see.'; } } catch { /* */ } - const raw = fs.readFileSync(abs, 'utf8'); const eol = raw.includes('\r\n') ? '\r\n' : '\n'; newStr = newStr.replace(/\r\n/g, '\n').replace(/\n/g, eol); + if (await isBinaryFile(abs)) { return 'ERROR: ' + input.path + ' looks like a binary file — refusing to overwrite it.'; } + try { const st = await host.stat(abs); if (st && st.size > 100 * 1024) { return 'ERROR: ' + input.path + ' is large (>100KB) and you only saw the first 100KB on read. Use edit_file for targeted changes — a full write_file risks dropping content you did not see.'; } } catch { /* */ } + const raw = await host.readText(abs); const eol = raw.includes('\r\n') ? '\r\n' : '\n'; newStr = newStr.replace(/\r\n/g, '\n').replace(/\n/g, eol); } const ok = await ctx.applyEdit({ path: input.path, exists: existed, proposed: newStr }); if (!ok) { return 'ERROR: could not write ' + input.path + ' (path may be read-only or in conflict).'; } @@ -393,9 +472,9 @@ async function runTool(tu, ctx) { return 'Applied edit to ' + input.path + ' (pending the user\'s Keep/Undo review).'; } if (tu.name === 'delete_file') { - const abs = resolveWorkspacePath(input.path || '', { mustExist: true }); + const abs = await resolveWorkspacePathAsync(input.path || '', { mustExist: true }); if (!abs) { return 'ERROR: file not found: ' + input.path + whereHint(); } - try { if (fs.statSync(abs).isDirectory()) { return 'ERROR: ' + input.path + ' is a directory — delete_file removes a single file.'; } } catch { /* */ } + try { const st = await host.stat(abs); if (st && st.isDirectory) { return 'ERROR: ' + input.path + ' is a directory — delete_file removes a single file.'; } } catch { /* */ } // Deletion is in the autopilot danger set — ask even when autopilot runs everything else silently. // (In manual mode nothing gates here: the edit is applied and reviewed with Keep/Undo as before.) if (ctx.autopilot && typeof ctx.approve === 'function') { @@ -678,12 +757,15 @@ function resetContextAnnounce() { lastContextSig = ''; } async function setupMcp(ctx, wsFolders, dbg) { const empty = { tools: [], routes: null }; const cfg = ctx.mcp || {}; + // MCP servers here are local processes spoken to over stdio. A host that cannot start programs has + // none to offer; saying nothing is right, since nothing is configured that it could have run. + if (!host.caps.mcpStdio) { return empty; } try { - const { servers, problems } = loadServerConfig({ + const { servers, problems } = await host.withReads((readFile) => loadServerConfig({ settings: cfg.servers, folders: wsFolders, - readFile: (abs) => { try { return fs.readFileSync(abs, 'utf8'); } catch { return null; } } - }); + readFile + })); for (const p of problems) { dbg('mcp.config', p); } if (!servers.length) { return empty; } @@ -780,7 +862,7 @@ async function runAgent(ctx) { // Multi-root: name every workspace folder so the model addresses them by prefix from turn one. const wsFolders = workspaceFolderList(); const multiRootNote = wsFolders.length > 1 - ? '\n\nWorkspace folders (multi-root — prefix paths with the folder name): ' + wsFolders.map((f) => f.name).join(', ') + '. The first folder ("' + wsFolders[0].name + '") is the default for unprefixed paths and run_command.' + ? '\n\nWorkspace folders (multi-root — prefix paths with the folder name): ' + wsFolders.map((f) => f.name).join(', ') + '. The first folder ("' + wsFolders[0].name + '") is the default for unprefixed paths' + (host.caps.shell ? ' and run_command.' : '.') : ''; // Rootless: say so plainly. Without this the model sees a tool list with no read_file and improvises — // answering about files it cannot see, or apologising for a limit it cannot name. Telling it WHY the @@ -796,13 +878,13 @@ async function runAgent(ctx) { // Autopilot: act decisively and self-verify rather than pausing. Commands run without approval (the // host still gates the danger set — deletion, sudo, force-push, remote|shell, publish, system writes), // so the model should lean on verification, not on asking, when it's unsure. - const autopilotNote = ctx.autopilot + const autopilotNote = ctx.autopilot && host.caps.shell ? '\n\nAUTOPILOT IS ON. Work end-to-end without pausing for confirmation. Your run_command calls execute immediately (only irreversible ones — deleting files, sudo, force-push, piping a remote script to a shell, publishing — still ask the user). Do NOT call ask_user for anything you can reasonably decide; pick a sensible default and proceed. When you are unsure whether a change is correct, do not stop to ask — verify it: run the build/tests/linters via run_command and read editor diagnostics, then fix and re-verify until clean, and only then move on. Prefer doing and checking over asking.' : ''; // Project rules: fold a repo's own AGENTS.md (or CLAUDE.md / .cursorrules) into the cached system // block so the agent follows the project's conventions from turn one. Read once per run — an edit is // picked up on the next run. - const rules = loadProjectRules(wsFolders, (abs) => { try { return fs.readFileSync(abs, 'utf8'); } catch { return null; } }); + const rules = await host.withReads((readFile) => loadProjectRules(wsFolders, readFile)); // Project memory: a tight, verify-first digest of what earlier sessions achieved (built by extension.js // from the per-project journal). Rides the SAME cached-system channel as project rules — always-on but // small — so a new session's first reply is continuous, not amnesiac. It is untrusted context like the @@ -859,6 +941,8 @@ async function runAgent(ctx) { if (ctx.recallSessions) { extras.push(RECALL_TOOL); } if (rich) { extras.push(diagramTool.RENDER_DIAGRAM); if (ctx.diagramsStubbed) { extras.push(diagramTool.GET_DIAGRAM); } } if (extras.length) { tools = tools.concat(extras); } + // Withhold what this host cannot do: no shell, no run_command, no read_command_output. + if (!host.caps.shell) { tools = tools.filter((t) => !NEEDS_SHELL.has(t.name)); } const baseTools = extras.length ? builtins.concat(extras) : builtins; // built-ins + extras; MCP is the rest // Recomputed only when MCP or a host-gated extra actually contributed tools, so the plain path keeps the module // constant and pays nothing for a feature it isn't using — but there are now TWO plain paths, and the @@ -904,7 +988,7 @@ async function runAgent(ctx) { let feedbackUsed = 0, verifySeq = 0; async function runVerifyOnce() { const id = 'verify-' + (++verifySeq); - const cmd = ((ctx.verify && ctx.verify.command) || '').trim(); + const cmd = host.caps.shell ? ((ctx.verify && ctx.verify.command) || '').trim() : ''; ctx.post({ type: 'verifyRun', id, command: cmd }); let ran = false, exitCode = 0, cmdTail = ''; if (cmd) { @@ -929,7 +1013,7 @@ async function runAgent(ctx) { if (!v || !v.enabled) { return false; } if (ctx.signal.aborted) { return false; } if (!(ctx.editCount > 0)) { return false; } // no edits this run → nothing to verify - const cmd = (v.command || '').trim(); + const cmd = host.caps.shell ? (v.command || '').trim() : ''; const touchedCount = ctx.getTouchedUris ? ctx.getTouchedUris().length : 0; if (!cmd && !touchedCount) { return false; } // nothing to check const max = Math.max(0, v.maxRounds || 0); @@ -1174,4 +1258,4 @@ async function runAgent(ctx) { } } -module.exports = { resetContextAnnounce, runAgent, makeDiff, resolveWorkspacePath, saysDone }; +module.exports = { resetContextAnnounce, runAgent, makeDiff, resolveWorkspacePath, resolveWorkspacePathAsync, saysDone }; diff --git a/extensions/levelcode-ai/extension.js b/extensions/levelcode-ai/extension.js index b4e7e13..4c2bc6f 100644 --- a/extensions/levelcode-ai/extension.js +++ b/extensions/levelcode-ai/extension.js @@ -14,6 +14,7 @@ const fs = require('fs'); const os = require('os'); const cp = require('child_process'); const crypto = require('crypto'); +const host = require('./host'); // workspace I/O + what this machine can do (browser vs desktop) const providers = require('./providers/index'); const catalog = require('./providers/catalog'); const { resolveGateway } = require('./providers/gateway'); @@ -919,6 +920,36 @@ function rgFiles(term, cwd) { }); } +/** + * Files whose content contains each term, for every term at once. ripgrep where the editor has one; + * a bounded scan of the workspace's text files where it does not (LevelCode in the browser). Returns + * one array of absolute paths per term, in order. + */ +async function contentHits(terms, dir) { + if (host.caps.ripgrep) { return Promise.all(terms.map((kw) => rgFiles(kw, dir))); } + const hits = terms.map(() => []); + if (!dir) { return hits; } + const prefix = dir.endsWith('/') ? dir : dir + '/'; + let uris = []; + try { uris = await host.findFiles('**/*', '{**/node_modules/**,**/.git/**,**/out/**,**/dist/**,**/*.map,**/*.min.*}', 400); } catch { return hits; } + const needles = terms.map((t) => t.toLowerCase()); + const deadline = Date.now() + 4000; // the same budget rgFiles gives ripgrep + for (const uri of uris) { + if (Date.now() > deadline) { break; } + const abs = uri.path; + if (!abs.startsWith(prefix)) { continue; } + try { + const st = await host.stat(abs); + if (!st || !st.isFile || st.size > 1024 * 1024) { continue; } + const bytes = await host.readBytes(abs); + if (bytes.subarray(0, 8000).includes(0)) { continue; } + const text = new TextDecoder('utf-8').decode(bytes).toLowerCase(); + needles.forEach((n, i) => { if (text.includes(n)) { hits[i].push(abs); } }); + } catch { /* unreadable: not a hit */ } + } + return hits; +} + /** Scope retrieval to the active file's top-level sub-project, so unrelated sibling trees * (e.g. a vendored source dump) don't pollute results. Returns the search dir + rel prefix. */ function activeProjectScope() { @@ -926,14 +957,17 @@ function activeProjectScope() { const fallback = { dir: folders.length ? folders[0].uri.fsPath : '', prefix: '' }; if (!aiConfig().get('chat.scopeToActiveProject', true)) { return fallback; } const ed = vscode.window.activeTextEditor; - if (!ed || ed.document.uri.scheme !== 'file') { return fallback; } + if (!ed || (!host.isBrowser && ed.document.uri.scheme !== 'file')) { return fallback; } const wsFolder = vscode.workspace.getWorkspaceFolder(ed.document.uri); if (!wsFolder) { return fallback; } const rel = path.relative(wsFolder.uri.fsPath, ed.document.uri.fsPath); if (!rel || rel.startsWith('..')) { return fallback; } const top = rel.split(path.sep)[0]; const topPath = path.join(wsFolder.uri.fsPath, top); - try { if (!fs.statSync(topPath).isDirectory()) { return fallback; } } catch { return fallback; } + // A first segment is a directory when the file sits below it. The desktop asks the disk, as it + // always did; a browser workspace cannot be asked synchronously and the path says the same thing. + if (host.isBrowser) { if (rel.indexOf(path.sep) < 0) { return fallback; } } + else { try { if (!fs.statSync(topPath).isDirectory()) { return fallback; } } catch { return fallback; } } return { dir: topPath, prefix: top + '/' }; } @@ -965,9 +999,9 @@ async function gatherAutoContext(question, allFiles) { // 1. CONTENT search via ripgrep (primary signal — finds files by what's inside them). const terms = keywords.slice(0, 6); - const hits = await Promise.all(terms.map((kw) => rgFiles(kw, scope.dir))); + const hits = await contentHits(terms, scope.dir); for (let i = 0; i < terms.length; i++) { - for (const abs of hits[i]) { bump(vscode.Uri.file(abs), contentWeight(terms[i])); } + for (const abs of hits[i]) { bump(host.uriFor(abs), contentWeight(terms[i])); } } // 2. filename matches (scoped). @@ -1087,7 +1121,7 @@ async function openDiagramLink(id, nodeId) { if (!r.ok) { vscode.window.showWarningMessage('LevelCode: that link cannot be opened — ' + r.reason + '.'); dbg('diagram.link.refused', { reason: r.reason }); return; } recordDiagramStat({ type: 'link' }); try { - const uri = vscode.Uri.file(r.abs); + const uri = host.uriFor(r.abs); const doc = await vscode.workspace.openTextDocument(uri); let line = Number.isInteger(node.link.line) ? node.link.line : null; if (node.link.symbol) { @@ -1881,6 +1915,7 @@ async function saveMcpLaunchTrust(store) { * broken configuration it exists to explain. */ function mcpOverview() { + if (!host.caps.mcpStdio) { return { configured: 0, running: 0, awaitingTrust: 0, servers: [], problems: [] }; } try { const cfg = aiConfig(); const folders = (vscode.workspace.workspaceFolders || []).map((f) => ({ name: f.name, root: f.uri.fsPath })); @@ -2282,8 +2317,8 @@ async function agentFlow(text, imageBlocks) { applyDelete: (req) => review.applyDelete(req), // file deletions: apply-then-review verify: verifyCfg, // M5 auto-verify settings touched: runTouched, // agent.js adds each edited abs path here - getTouchedUris: () => [...runTouched].map((p) => vscode.Uri.file(p)), - getNewDiagnostics: () => collectNewDiagnostics([...runTouched].map((p) => vscode.Uri.file(p)), diagBaseline, verifyCfg.includeWarnings), + getTouchedUris: () => [...runTouched].map((p) => host.uriFor(p)), + getNewDiagnostics: () => collectNewDiagnostics([...runTouched].map((p) => host.uriFor(p)), diagBaseline, verifyCfg.includeWarnings), signal: abort.signal }); } finally { @@ -2341,7 +2376,7 @@ async function attachImagePaths(paths) { const mt = { '.png': 'image/png', '.jpg': 'image/jpeg', '.jpeg': 'image/jpeg', '.gif': 'image/gif', '.webp': 'image/webp' }[ext]; if (!mt) { vscode.window.showWarningMessage(path.basename(fsPath) + ' is not an image LevelCode can read.'); continue; } - const buf = await fs.promises.readFile(fsPath); + const buf = Buffer.from(await host.readBytes(fsPath)); // Guard before the bytes cross into the webview: a 200MB file would otherwise be // base64-ed onto the message bus before anything got a chance to refuse it. if (buf.length > 25 * 1024 * 1024) { diff --git a/extensions/levelcode-ai/host.js b/extensions/levelcode-ai/host.js new file mode 100644 index 0000000..fefc250 --- /dev/null +++ b/extensions/levelcode-ai/host.js @@ -0,0 +1,191 @@ +/*--------------------------------------------------------------------------------------------- + * LevelCode — AI: the machine this extension runs on. + * + * The extension was written for a Node extension host: the user's project is a directory it can + * read with `fs`, and programs can be started. LevelCode also runs in a browser tab (docs/WEB.md), + * where the project is whatever the editor's file system provider says it is, every read is + * asynchronous, and there is no shell. This module is the one place that difference lives. + * + * Two rules keep the desktop app exactly as it was: + * 1. On the desktop every function here does what the call site did before — the same `fs` + * call, the same error — only behind `await`. + * 2. Nothing outside this file asks "am I in a browser?". Call sites ask for a capability + * (`caps.shell`) or call a function that already knows. + * + * Paths are the absolute strings the extension has always used for workspace files + * (`Uri.fsPath` of a workspace folder, joined). In the browser they are URI paths, and `uriFor` + * maps one back to the workspace folder it belongs to, so the scheme (`file`, `levelcode-scratch`) + * is never guessed. + *--------------------------------------------------------------------------------------------*/ +// @ts-check +'use strict'; + +const vscode = require('vscode'); +const nodeFs = require('fs'); +const path = require('path'); + +/** True in the browser build (web/ai-extension/entry.js sets the flag before anything else loads). */ +const isBrowser = !!globalThis.__LEVELCODE_BROWSER_HOST__; + +/** What this host can do. Tools and prompt text that need a capability are withheld without it. */ +const caps = Object.freeze({ + /** Start programs: run_command, background commands, verify commands. */ + shell: !isBrowser, + /** Start MCP servers as local processes (stdio). */ + mcpStdio: !isBrowser, + /** Search file contents with the ripgrep binary the editor ships. */ + ripgrep: !isBrowser, +}); + +const TEXT_SNIFF_BYTES = 8000; + +/* ----- paths and URIs ------------------------------------------------------------------------ */ + +function folders() { return vscode.workspace.workspaceFolders || []; } + +/** + * The URI for a workspace path. On the desktop that is `Uri.file`, as ever. In the browser the + * workspace folder that contains the path supplies scheme and authority; a path inside none of + * them falls back to `Uri.file`, which is right for a folder opened from the user's computer. + * @param {string} abs + * @returns {vscode.Uri} + */ +function uriFor(abs) { + if (!isBrowser) { return vscode.Uri.file(abs); } + let best = null; + for (const f of folders()) { + const root = f.uri.path; + const prefix = root.endsWith('/') ? root : root + '/'; + if (abs === root || abs.startsWith(prefix)) { + if (!best || root.length > best.uri.path.length) { best = f; } + } + } + return best ? best.uri.with({ path: abs }) : vscode.Uri.file(abs); +} + +/* ----- reading and probing ------------------------------------------------------------------- */ + +/** @param {string} abs @returns {Promise} rejects when the file is missing or unreadable. */ +async function readBytes(abs) { + if (!isBrowser) { return nodeFs.promises.readFile(abs); } + return vscode.workspace.fs.readFile(uriFor(abs)); +} + +/** @param {string} abs @returns {Promise} UTF-8 text; a leading BOM is left for the caller to strip. */ +async function readText(abs) { + if (!isBrowser) { return nodeFs.promises.readFile(abs, 'utf8'); } + return new TextDecoder('utf-8').decode(await vscode.workspace.fs.readFile(uriFor(abs))); +} + +/** @param {string} abs @returns {Promise} null when it cannot be read — never throws. */ +async function readTextOrNull(abs) { + try { return await readText(abs); } catch { return null; } +} + +/** + * @param {string} abs + * @returns {Promise<{isFile: boolean, isDirectory: boolean, size: number}|null>} null when absent. + */ +async function stat(abs) { + try { + if (!isBrowser) { + const s = await nodeFs.promises.stat(abs); + return { isFile: s.isFile(), isDirectory: s.isDirectory(), size: s.size }; + } + const s = await vscode.workspace.fs.stat(uriFor(abs)); + return { isFile: (s.type & vscode.FileType.File) !== 0, isDirectory: (s.type & vscode.FileType.Directory) !== 0, size: s.size }; + } catch { return null; } +} + +/** @param {string} abs @returns {Promise} */ +async function exists(abs) { return (await stat(abs)) !== null; } + +/** A NUL byte in the first 8 KB means it is not text we should round-trip as UTF-8. Unreadable counts as text. */ +async function isBinary(abs) { + try { + const buf = await readBytes(abs); + const n = Math.min(buf.length, TEXT_SNIFF_BYTES); + for (let i = 0; i < n; i++) { if (buf[i] === 0) { return true; } } + return false; + } catch { return false; } +} + +/** + * Run a loader that reads files synchronously (project rules, MCP config) against files this host can + * only read asynchronously. The loader runs once to learn which paths it asks for, those are read, + * and it runs again with the answers. A path the loader only asks for after seeing another file's + * content is picked up by the next pass (at most four). A path never read in time is absent. + * + * On the desktop the loader simply runs once against `fs`, as before. + * @template T + * @param {(readFile: (abs: string) => string|null) => T} run + * @returns {Promise} + */ +async function withReads(run) { + if (!isBrowser) { + return run((abs) => { try { return nodeFs.readFileSync(abs, 'utf8'); } catch { return null; } }); + } + /** @type {Map} */ + const known = new Map(); + let result; + for (let pass = 0; pass < 4; pass++) { + const asked = new Set(); + result = run((abs) => { + if (known.has(abs)) { return known.get(abs); } + asked.add(abs); + return null; + }); + if (asked.size === 0) { break; } + await Promise.all([...asked].map(async (p) => { known.set(p, await readTextOrNull(p)); })); + } + return result; +} + +/* ----- sending the user to sign in ------------------------------------------------------------- */ + +/** Registered by the page (web/main.js); it takes the tab to the account site and the sign-in comes back to it. */ +const OPEN_AUTH_COMMAND = 'levelcode.web.openAuthUrl'; + +/** + * Open the account site's sign-in page. The desktop hands it to the system browser, as ever. A browser + * tab leaves the page for it: a pop-up opened from a chain that began in a webview is the first thing a + * strict browser (Safari, a phone) refuses, and the sign-in returns to the editor in the same tab. The + * page's entry script supplies the command that navigates; without it this falls back to openExternal. + * + * The string handed over is the one the editor's own opener would have opened — `encodeURI(uri.toString(true))` + * (src/vs/editor/browser/services/openerService.ts) — so the server sees the same address whichever way + * the sign-in was started, as it has from the desktop app all along. + * @param {vscode.Uri} uri + */ +async function openAuth(uri) { + if (isBrowser) { + try { + const known = await vscode.commands.getCommands(true); + if (known.includes(OPEN_AUTH_COMMAND)) { + await vscode.commands.executeCommand(OPEN_AUTH_COMMAND, encodeURI(uri.toString(true))); + return; + } + } catch { /* fall through to the pop-up */ } + } + await vscode.env.openExternal(uri); +} + +/* ----- finding files ------------------------------------------------------------------------- */ + +/** + * `vscode.workspace.findFiles`, which is what the editor's own search service answers. The browser + * build uses it too: LevelCode's scratch workspace registers a file search provider. + * @param {string} glob + * @param {string} exclude + * @param {number} max + */ +function findFiles(glob, exclude, max) { + return vscode.workspace.findFiles(glob, exclude, max); +} + +module.exports = { + isBrowser, caps, + uriFor, readBytes, readText, readTextOrNull, stat, exists, isBinary, withReads, findFiles, openAuth, + // For tests. + _path: path, +}; diff --git a/extensions/levelcode-ai/providers/anthropic.js b/extensions/levelcode-ai/providers/anthropic.js index 31c2103..8ce94ce 100644 --- a/extensions/levelcode-ai/providers/anthropic.js +++ b/extensions/levelcode-ai/providers/anthropic.js @@ -13,7 +13,11 @@ const { readLines } = require('./sse'); const ANTHROPIC_URL = 'https://api.anthropic.com/v1/messages'; function anthropicHeaders(apiKey) { - return { 'content-type': 'application/json', 'x-api-key': apiKey, 'anthropic-version': '2023-06-01' }; + const h = { 'content-type': 'application/json', 'x-api-key': apiKey, 'anthropic-version': '2023-06-01' }; + // From a browser the call is cross-origin, and Anthropic answers CORS only to a request that says it + // knows the key is being used from a page (web/ai-extension/entry.js sets the flag; the desktop never does). + if (globalThis.__LEVELCODE_BROWSER_HOST__) { h['anthropic-dangerous-direct-browser-access'] = 'true'; } + return h; } /** diff --git a/extensions/levelcode-ai/reviewSession.js b/extensions/levelcode-ai/reviewSession.js index c23ecb3..0984800 100644 --- a/extensions/levelcode-ai/reviewSession.js +++ b/extensions/levelcode-ai/reviewSession.js @@ -16,7 +16,8 @@ const vscode = require('vscode'); const fs = require('fs'); const path = require('path'); -const { makeDiff, resolveWorkspacePath } = require('./agent'); +const { makeDiff, resolveWorkspacePathAsync } = require('./agent'); +const host = require('./host'); const PERSIST_KEY = 'levelcode.ai.pendingReviews'; @@ -108,18 +109,18 @@ function registerReview(context, post, dbg, recordTouch) { if (!workspaceRoot()) { return false; } // Multi-root aware — MUST resolve exactly like the agent's tool layer (same resolver), or an // edit the agent computed against folder N would be applied into folder 0. - const abs = resolveWorkspacePath(req.path || '', { mustExist: !!req.exists }); + const abs = await resolveWorkspacePathAsync(req.path || '', { mustExist: !!req.exists }); if (!abs) { return false; } - const uri = vscode.Uri.file(abs); + const uri = host.uriFor(abs); const key = uri.toString(); const proposed = String(req.proposed != null ? req.proposed : ''); - const existsNow = fs.existsSync(abs); // re-check at apply time (TOCTOU vs the agent's earlier check) + const existsNow = await host.exists(abs); // re-check at apply time (TOCTOU vs the agent's earlier check) // Snapshot the pre-run content ONCE, from DISK (the true baseline — never a dirty buffer). let snapshot; if (pending.has(key)) { snapshot = pending.get(key).snapshot; } else if (existsNow) { - try { snapshot = fs.readFileSync(abs, 'utf8').replace(/^/, ''); } + try { snapshot = (await host.readText(abs)).replace(/^/, ''); } catch { return false; } // can't read the file we're about to edit → abort (never a destructive empty-snapshot edit) } else { snapshot = ''; } @@ -179,13 +180,13 @@ function registerReview(context, post, dbg, recordTouch) { async function applyDelete(req) { if (!workspaceRoot()) { return false; } // Multi-root aware — same resolver as the agent's tool layer (see applyEdit). - const abs = resolveWorkspacePath(req.path || '', { mustExist: true }); + const abs = await resolveWorkspacePathAsync(req.path || '', { mustExist: true }); if (!abs) { return false; } - const uri = vscode.Uri.file(abs); + const uri = host.uriFor(abs); const key = uri.toString(); let snapshot; if (pending.has(key)) { snapshot = pending.get(key).snapshot; } - else { try { snapshot = fs.readFileSync(abs, 'utf8').replace(/^/, ''); } catch { return false; } } + else { try { snapshot = (await host.readText(abs)).replace(/^/, ''); } catch { return false; } } // Checkpoint hook: the file EXISTED (created=false) → per-turn Restore recreates it from this snapshot. if (recordTouch) { try { recordTouch(key, snapshot, false); } catch (e) { /* checkpoint capture must never break editing */ } } const edit = new vscode.WorkspaceEdit(); @@ -239,7 +240,7 @@ function registerReview(context, post, dbg, recordTouch) { const fr = pending.get(key); if (!fr) { return; } const edit = new vscode.WorkspaceEdit(); - const gone = !fs.existsSync(fr.uri.fsPath); + const gone = !(await host.exists(fr.uri.fsPath)); if (fr.exists && gone) { // the agent DELETED this file → recreate it from the snapshot edit.createFile(fr.uri, { ignoreIfExists: true }); @@ -333,11 +334,11 @@ function registerReview(context, post, dbg, recordTouch) { vscode.window.onDidChangeActiveTextEditor(() => updateActiveUi()), vscode.commands.registerCommand('levelcode.ai.review.keepActive', () => { const k = activePendingKey(); if (k) { keepFile(k, 'kept'); } }), vscode.commands.registerCommand('levelcode.ai.review.undoActive', () => { const k = activePendingKey(); if (k) { undoFile(k); } }), - vscode.languages.registerCodeLensProvider({ scheme: 'file' }, lensProvider), + vscode.languages.registerCodeLensProvider({ scheme: host.isBrowser ? '*' : 'file' }, lensProvider), vscode.workspace.registerTextDocumentContentProvider(SNAPSHOT_SCHEME, snapshotProvider), vscode.window.onDidChangeVisibleTextEditors(() => { for (const fr of pending.values()) { decorate(fr); } }), vscode.workspace.onDidChangeTextDocument((e) => { - if (e.document.uri.scheme !== 'file') { return; } + if (!host.isBrowser && e.document.uri.scheme !== 'file') { return; } // a browser workspace has its own schemes const key = e.document.uri.toString(); if (!pending.has(key)) { return; } if (expected.get(key) === e.document.getText()) { return; } // our own apply — ignore @@ -361,18 +362,32 @@ function registerReview(context, post, dbg, recordTouch) { ); // Rehydrate pending reviews from a previous window (so a reload mid-review keeps Keep/Undo working). - for (const s of (context.workspaceState.get(PERSIST_KEY, []) || [])) { - try { - const uri = vscode.Uri.parse(s.uri); - let current = ''; - try { current = fs.readFileSync(uri.fsPath, 'utf8').replace(/^/, ''); } catch { /* file gone */ } - const span = changedSpan(s.snapshot, current); - const { add, del } = countDiff(makeDiff(s.snapshot, current)); - pending.set(s.uri, { uri, rel: s.rel, exists: s.exists, snapshot: s.snapshot, range: new vscode.Range(span.start, 0, span.end, 0), add, del, appliedText: current }); - expected.set(s.uri, current); - } catch { /* */ } + const saved = context.workspaceState.get(PERSIST_KEY, []) || []; + function rehydrate(readCurrent) { + for (const s of saved) { + try { + const uri = vscode.Uri.parse(s.uri); + const current = readCurrent(uri, s); + const span = changedSpan(s.snapshot, current); + const { add, del } = countDiff(makeDiff(s.snapshot, current)); + pending.set(s.uri, { uri, rel: s.rel, exists: s.exists, snapshot: s.snapshot, range: new vscode.Range(span.start, 0, span.end, 0), add, del, appliedText: current }); + expected.set(s.uri, current); + } catch { /* */ } + } + if (pending.size) { for (const fr of pending.values()) { decorate(fr); } refreshLenses(); updateActiveUi(); } + } + if (!host.isBrowser) { + rehydrate((uri) => { try { return fs.readFileSync(uri.fsPath, 'utf8').replace(/^/, ''); } catch { return ''; /* file gone */ } }); + } else { + // A browser workspace can only be read asynchronously: read every pending file, then rehydrate. + void (async () => { + const current = new Map(); + for (const s of saved) { + try { current.set(s.uri, ((await host.readTextOrNull(vscode.Uri.parse(s.uri).fsPath)) || '').replace(/^/, '')); } catch { /* */ } + } + rehydrate((uri, s) => current.get(s.uri) || ''); + })(); } - if (pending.size) { for (const fr of pending.values()) { decorate(fr); } refreshLenses(); updateActiveUi(); } /** Re-send all pending review cards + state to the webview (called when the chat view becomes ready). */ function resync() { diff --git a/extensions/levelcode-ai/test/diagramHost.test.js b/extensions/levelcode-ai/test/diagramHost.test.js index 833c1b6..5ada661 100644 --- a/extensions/levelcode-ai/test/diagramHost.test.js +++ b/extensions/levelcode-ai/test/diagramHost.test.js @@ -114,7 +114,7 @@ const HOST = ['clientRender', 'openDiagramLink', 'diagramMarkdown', 'exportDiagr // eslint-disable-next-line no-new-func const makeHost = new Function('env', [ "'use strict';", - 'const { vscode, fs, path, os, catalog, sessionMemory, diagrams, diagramLinks, diagramExport, diagramText, diagramTool, diagramBundle, diagramAscii, aiConfig, dbg,', + 'const { vscode, host, fs, path, os, catalog, sessionMemory, diagrams, diagramLinks, diagramExport, diagramText, diagramTool, diagramBundle, diagramAscii, aiConfig, dbg,', ' recordDiagramStat, agentFlow, findCompactionCut, estimateMsgTokens, meterModel, prepProviderRequest, authRetry, providers, checkpoints, ctx,', ' sessionsManager, currentContextLimit, post, replaySessionExpired, postContextFiles, refreshSessions, focusChatView } = env;', "const COMPACT_SYSTEM = 'system', COMPACT_INSTRUCTIONS = 'Summarize:\\n';", @@ -134,7 +134,10 @@ function boot(over) { const settings = Object.assign({ 'diagrams.enabled': true }, over && over.settings); const diagrams = createDiagrams({ resolveLink: (link) => diagramLinks.resolveLink(link, [{ name: 'app', root: ws }]), onStat: (ev) => stats.push(ev) }); const env = { - vscode, fs, path, os, catalog, sessionMemory, diagrams, diagramLinks, diagramExport, diagramText, diagramTool, diagramBundle: bundle, + vscode, + // The host module (host.js) as the desktop sees it: workspace paths are files on disk. + host: { isBrowser: false, caps: { shell: true, mcpStdio: true, ripgrep: true }, uriFor: (abs) => vscode.Uri.file(abs) }, + fs, path, os, catalog, sessionMemory, diagrams, diagramLinks, diagramExport, diagramText, diagramTool, diagramBundle: bundle, diagramAscii: (over && over.ascii) || ascii, aiConfig: () => ({ get: (k, d) => (k in settings ? settings[k] : d) }), dbg: (label, data) => logs.push({ label, data }), diff --git a/scripts/build-web.mjs b/scripts/build-web.mjs new file mode 100644 index 0000000..84d2b38 --- /dev/null +++ b/scripts/build-web.mjs @@ -0,0 +1,173 @@ +#!/usr/bin/env node +// Build LevelCode in your browser: one static directory, dist-web/, that any static host can serve. +// +// node scripts/build-web.mjs [--vscode vscode] [--out dist-web] [--account https://levelcode.ai] [--api-url ] +// [--webview-origin 'https://{{uuid}}.view.example.com'] +// [--ext-host-origin 'https://{{uuid}}.ext.example.com'] +// [--static ] [--id ] +// +// What it does, in order (docs/WEB.md has the reasoning): +// 1. builds Code-OSS's web client from the bootstrapped checkout (gulp vscode-web-min-ci, about a +// minute), or takes a prebuilt one with --static; +// 2. bundles levelcode-ai for the web worker extension host (web/ai-extension), copies the scratch +// workspace extension (web/workspace) and every declarative built-in (grammars, themes, icons); +// 3. lays everything out under a build-addressed prefix (/_//...), renders index.html with the +// configuration baked in, and writes the HTTP policy for the host (_headers, nginx). +// +// It changes nothing outside --out, except that building the client from the checkout leaves gulp's own output, +// vscode-web/, next to it (pass --static to use a client built earlier and skip that). It does not deploy. +import { spawnSync, execFileSync } from 'node:child_process'; +import fs from 'node:fs'; +import path from 'node:path'; +import { createHash } from 'node:crypto'; +import { fileURLToPath } from 'node:url'; +import { webConfig, renderIndex } from '../web/lib/config.mjs'; +import { stageDeclarativeExtensions, copyExtension } from '../web/lib/extensions.mjs'; +import { policy, toHeadersFile, toNginx } from '../web/lib/headers.mjs'; +import { hashTree } from '../web/lib/fingerprint.mjs'; + +const REPO = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..'); +const WEB = path.join(REPO, 'web'); + +const args = Object.fromEntries(process.argv.slice(2).reduce((a, v, i, all) => { + if (v.startsWith('--')) { a.push([v.slice(2), all[i + 1] && !all[i + 1].startsWith('--') ? all[i + 1] : 'true']); } + return a; +}, [])); +const vscodeDir = path.resolve(REPO, args.vscode || 'vscode'); +const outDir = path.resolve(REPO, args.out || 'dist-web'); +const account = String(args.account || process.env.LEVELCODE_WEB_ACCOUNT || 'https://levelcode.ai').replace(/\/+$/, ''); +// Only for a deployment whose API is not on the account origin (production has one host for both). +const apiUrl = String(args['api-url'] || process.env.LEVELCODE_WEB_API_URL || '').replace(/\/+$/, ''); +const webviewOrigin = args['webview-origin'] || process.env.LEVELCODE_WEB_WEBVIEW_ORIGIN || ''; +const extHostOrigin = args['ext-host-origin'] || process.env.LEVELCODE_WEB_EXTHOST_ORIGIN || ''; + +const log = (m) => console.log('[build-web] ' + m); +const die = (m) => { console.error('[build-web] ' + m); process.exit(1); }; +const sizeOf = (p) => { let n = 0; const walk = (d) => { for (const e of fs.readdirSync(d, { withFileTypes: true })) { const f = path.join(d, e.name); e.isDirectory() ? walk(f) : (n += fs.statSync(f).size); } }; walk(p); return n; }; +const mb = (n) => (n / 1048576).toFixed(1) + ' MB'; + +/* ----- 1. Code-OSS's web client ---------------------------------------------------------------- */ +let staticDir; +if (args.static) { + staticDir = path.resolve(args.static); + log('using the prebuilt web client at ' + staticDir); +} else { + if (!fs.existsSync(path.join(vscodeDir, 'node_modules'))) { die('no bootstrapped checkout at ' + vscodeDir + ' — run scripts/bootstrap.sh first'); } + const product = JSON.parse(fs.readFileSync(path.join(vscodeDir, 'product.json'), 'utf8')); + if (product.nameShort !== 'LevelCode') { die('the checkout is not branded (product.json nameShort is "' + product.nameShort + '") — run scripts/apply-branding.mjs'); } + log('building the web client (gulp vscode-web-min-ci) in ' + vscodeDir); + const r = spawnSync('npm', ['run', 'gulp', '--', 'vscode-web-min-ci'], { cwd: vscodeDir, stdio: 'inherit', env: { ...process.env, NODE_OPTIONS: '--max-old-space-size=8192' } }); + if (r.status !== 0) { die('the web client build failed'); } + staticDir = path.resolve(vscodeDir, '..', 'vscode-web'); +} +for (const must of ['out/vs/workbench/workbench.web.main.internal.js', 'out/vs/workbench/workbench.web.main.internal.css', 'out/nls.messages.js', + 'out/vs/workbench/services/extensions/worker/webWorkerExtensionHostIframe.html', 'out/vs/workbench/contrib/webview/browser/pre/index.html']) { + if (!fs.existsSync(path.join(staticDir, must))) { die('the web client is missing ' + must); } +} + +/* ----- 2. identity of this build ---------------------------------------------------------------- */ +const fingerprint = createHash('sha256'); +fingerprint.update(fs.readFileSync(path.join(staticDir, 'out/vs/workbench/workbench.web.main.internal.js'))); +for (const f of ['main.js', 'boot.css', 'index.html']) { fingerprint.update(fs.readFileSync(path.join(WEB, f))); } +// Everything that goes into /_// and is not the web client itself: the extension (its sources, media and skills), +// its browser build, the scratch workspace, and the page's own configuration code. +const aiDir = path.join(REPO, 'extensions', 'levelcode-ai'); +hashTree(fingerprint, aiDir, { skip: ['test', 'node_modules', 'scripts', '.DS_Store'] }); +hashTree(fingerprint, path.join(WEB, 'ai-extension'), { skip: ['assets.generated.js', 'node_modules', '.DS_Store'] }); +hashTree(fingerprint, path.join(WEB, 'workspace'), { skip: ['node_modules', '.DS_Store'] }); +hashTree(fingerprint, path.join(WEB, 'lib'), { skip: ['.DS_Store'] }); +const id = String(args.id || fingerprint.digest('hex').slice(0, 12)); +let gitHead = 'unknown'; +try { gitHead = execFileSync('git', ['-C', REPO, 'rev-parse', '--short=10', 'HEAD'], { encoding: 'utf8' }).trim(); } catch { /* not a checkout */ } +log('build id ' + id + ' (levelcode ' + gitHead + ')'); + +/* ----- 3. assemble ------------------------------------------------------------------------------- */ +fs.rmSync(outDir, { recursive: true, force: true }); +const prefix = `/_/${id}`; +const root = path.join(outDir, '_', id); +fs.mkdirSync(root, { recursive: true }); + +log('copying the web client'); +fs.cpSync(staticDir, path.join(root, 'static'), { recursive: true, filter: (src) => !/\.(js|css)\.map$/.test(src) }); +const baked = new Set(fs.existsSync(path.join(staticDir, 'extensions')) ? fs.readdirSync(path.join(staticDir, 'extensions')) : []); + +const extOut = path.join(root, 'extensions'); +fs.mkdirSync(extOut, { recursive: true }); + +log('bundling levelcode-ai for the web worker extension host'); +const r = spawnSync(process.execPath, [path.join(WEB, 'ai-extension', 'build.mjs'), '--out', extOut, '--vscode', vscodeDir, '--minify'], { stdio: 'inherit' }); +if (r.status !== 0) { die('the levelcode-ai browser build failed'); } + +log('staging the scratch workspace and the declarative built-ins'); +const scratchOut = path.join(extOut, 'levelcode-web'); +copyExtension(path.join(WEB, 'workspace'), scratchOut); +fs.writeFileSync(path.join(scratchOut, 'package.nls.json'), '{}\n'); +const extensionsSource = path.join(vscodeDir, 'extensions'); +if (!fs.existsSync(extensionsSource)) { die('no extensions/ in ' + vscodeDir + ' to take the declarative built-ins from'); } +const declarative = stageDeclarativeExtensions(extensionsSource, extOut, { exclude: [...baked, 'levelcode-ai', 'levelcode-web'] }); +// A checkout that has not had LevelCode's extensions copied in still builds; the themes are then absent. +if (!declarative.includes('levelcode-themes')) { + const themes = path.join(REPO, 'extensions', 'levelcode-themes'); + if (fs.existsSync(themes)) { copyExtension(themes, path.join(extOut, 'levelcode-themes')); declarative.push('levelcode-themes'); } +} +const extensions = ['levelcode-ai', 'levelcode-web', ...declarative.sort()]; +log(`${extensions.length} extensions: levelcode-ai, levelcode-web and ${declarative.length} declarative (${declarative.slice(0, 6).join(', ')}, ...)`); + +log('writing the page'); +// main.js and boot.css are addressed by the build id like everything else; the page names them. +for (const f of ['main.js', 'boot.css']) { fs.copyFileSync(path.join(WEB, f), path.join(root, f)); } + +const productConfiguration = { + // No gallery: nothing is installed from the Internet into the page's own origin. + extensionsGallery: null, +}; +if (!webviewOrigin) { + // Code-OSS publishes each stable commit's webview host pages to its CDN. product.json pins an older insider commit, + // whose service worker is a version behind this build's; the template below names this build's own commit. + productConfiguration.webviewContentExternalBaseUrlTemplate = 'https://{{uuid}}.vscode-cdn.net/{{quality}}/{{commit}}/out/vs/workbench/contrib/webview/browser/pre/'; + console.warn('[build-web] note: no --webview-origin, so the chat panel loads its host page from Code-OSS\'s CDN (*.vscode-cdn.net, this build\'s own commit).\n' + + ' That works with no setup. To serve it yourself give a wildcard origin, e.g. --webview-origin \'https://{{uuid}}.view.example.com\'\n' + + ' (each webview needs its own subdomain; see docs/WEB.md).'); +} +if (webviewOrigin) { productConfiguration.webviewContentExternalBaseUrlTemplate = `${webviewOrigin}${prefix}/static/out/vs/workbench/contrib/webview/browser/pre/`; } +if (extHostOrigin) { productConfiguration.webEndpointUrlTemplate = `${extHostOrigin}${prefix}/static`; } +const config = webConfig({ + account, + apiUrl, + extensions, + base: prefix, + staticBase: `${prefix}/static`, + extensionsBase: `${prefix}/extensions`, + productConfiguration, +}); +fs.writeFileSync(path.join(outDir, 'index.html'), renderIndex(config)); +for (const f of ['callback.html', 'callback.js', 'callback.css', 'icon.svg', 'icon-192.png', 'icon-512.png', 'manifest.webmanifest']) { + fs.copyFileSync(path.join(WEB, f), path.join(outDir, f)); +} + +/* ----- 4. the host's half ------------------------------------------------------------------------ */ +const toPattern = (o) => (o ? o.replace('{{uuid}}', '*') : ''); +const pol = policy({ account, webviewOrigin: toPattern(webviewOrigin), extHostOrigin: toPattern(extHostOrigin), base: '/_' }); +fs.writeFileSync(path.join(outDir, '_headers'), toHeadersFile(pol)); +fs.mkdirSync(path.join(outDir, 'deploy'), { recursive: true }); +fs.writeFileSync(path.join(outDir, 'deploy', 'nginx.conf'), toNginx(pol)); +const info = { + id, levelcode: gitHead, builtAt: new Date().toISOString(), account, apiUrl: apiUrl || '(the account origin)', + webviewOrigin: webviewOrigin || '(Code-OSS default: *.vscode-cdn.net)', + extHostOrigin: extHostOrigin || '(same origin as the editor)', + extensions, +}; +fs.writeFileSync(path.join(outDir, 'build.json'), JSON.stringify(info, null, 2) + '\n'); + +/* ----- 5. check what was written ----------------------------------------------------------------- */ +const html = fs.readFileSync(path.join(outDir, 'index.html'), 'utf8'); +const m = html.match(/ + + diff --git a/web/callback.js b/web/callback.js new file mode 100644 index 0000000..76c09e1 --- /dev/null +++ b/web/callback.js @@ -0,0 +1,47 @@ +// The page the sign-in returns to. It hands the result to the editor tab through localStorage +// (the editor listens for the `storage` event) and then closes itself. Same contract as +// Code-OSS's callback.html; everything the page needs is in its own query string. +(function () { + var title = document.getElementById('title'); + var text = document.getElementById('text'); + function say(t, s) { title.textContent = t; text.textContent = s; } + function decode(v) { return v === null ? null : decodeURIComponent(v); } + try { + var params = new URL(location.href).searchParams; + var id = decode(params.get('vscode-reqid')); + var scheme = decode(params.get('vscode-scheme')); + var authority = decode(params.get('vscode-authority')); + if (!id || !scheme || !authority) { throw new Error('This page is only used to finish a sign-in from LevelCode.'); } + var uri = { scheme: scheme, authority: authority }; + var path = decode(params.get('vscode-path')); + var query = decode(params.get('vscode-query')); + var fragment = decode(params.get('vscode-fragment')); + ['vscode-reqid', 'vscode-scheme', 'vscode-authority', 'vscode-path', 'vscode-query', 'vscode-fragment'].forEach(function (k) { params.delete(k); }); + if (path) { uri.path = path; } + if (query) { new URLSearchParams(query).forEach(function (value, key) { params.set(key, value); }); } + var rest = params.toString(); + if (rest) { uri.query = rest; } + if (fragment) { uri.fragment = fragment; } + // `at` lets the editor tell this result from one left behind long ago; the URL it is revived from ignores it. + uri.at = Date.now(); + localStorage.setItem('vscode-web.url-callbacks[' + id + ']', JSON.stringify(uri)); + // Take the one-time code out of the address bar and history. + history.replaceState(null, '', location.pathname); + var home = new URL('./', location.href).href; + var back = false; + // Set by the editor when it took THIS tab to the sign-in (web/main.js); the editor clears it at start-up. + try { back = sessionStorage.getItem('levelcode-web.return') === '1'; } catch (e) { /* no storage: the pop-up path */ } + if (back) { + // This tab left the editor for the sign-in: go back. The editor finds the result at start-up. + say('You are signed in', 'Back to LevelCode.'); + location.replace(home); + } else { + say('You are signed in', 'You can close this tab and go back to LevelCode.'); + var more = document.getElementById('more'); + if (more) { more.hidden = false; more.querySelector('a').href = home; } + setTimeout(function () { try { window.close(); } catch (e) { /* the text above still tells the user */ } }, 600); + } + } catch (e) { + say('This link does not look right', String((e && e.message) || e)); + } +})(); diff --git a/web/icon-192.png b/web/icon-192.png new file mode 100644 index 0000000..68dbc8c Binary files /dev/null and b/web/icon-192.png differ diff --git a/web/icon-512.png b/web/icon-512.png new file mode 100644 index 0000000..7e5e270 Binary files /dev/null and b/web/icon-512.png differ diff --git a/web/icon.svg b/web/icon.svg new file mode 100644 index 0000000..3b1cffd --- /dev/null +++ b/web/icon.svg @@ -0,0 +1,6 @@ + + + + + + diff --git a/web/index.html b/web/index.html new file mode 100644 index 0000000..bf444bd --- /dev/null +++ b/web/index.html @@ -0,0 +1,39 @@ + + + + + LevelCode + + + + + + + + + + + + +
+
+ +
LevelCode
+ +

Opening your editor

+
+ +
+ + + diff --git a/web/lib/config.mjs b/web/lib/config.mjs new file mode 100644 index 0000000..5f26681 --- /dev/null +++ b/web/lib/config.mjs @@ -0,0 +1,68 @@ +// The page configuration, in one place. `serve.mjs` (development) and `scripts/build-web.mjs` +// (release) both call webConfig() and renderIndex(), so what you test locally is what ships. +import fs from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; + +export const WEB_DIR = path.dirname(path.dirname(fileURLToPath(import.meta.url))); + +/** + * @param {object} o + * @param {string} [o.account] LevelCode Cloud origin the editor signs in to and calls (the gateway lives there). + * @param {string} [o.apiUrl] The API host, when it is not the account origin (the extension's levelcode.cloud.apiUrl): + * a tunnel to a local backend, say. The gateway must be https. + * @param {string[]} [o.extensions] Folder names, under /extensions, of the extensions loaded as built-ins. + * @param {string} [o.staticBase] Where the Code-OSS web build is mounted. + * @param {object} [o.productConfiguration] Overrides merged over the product the build embeds. + * @param {object} [o.configurationDefaults] + * @param {boolean} [o.development] + */ +export function webConfig(o = {}) { + const account = trimSlash(o.account || 'https://levelcode.ai'); + return { + // Where the page's own scripts and styles, the Code-OSS web build, and the extensions are mounted. + // A release puts all three under one build-addressed prefix (/_/) so they can be cached for + // a year; the development server mounts them at the root, /static and /extensions. + account, + base: o.base || '', + staticBase: o.staticBase || '/static', + extensionsBase: o.extensionsBase || '/extensions', + callbackRoute: '/callback.html', + extensions: o.extensions || ['levelcode-ai'], + // The folder is /scratch, not the provider's root: a root named "/" is what the title bar and the + // Explorer would show, and "scratch" is what it is. + scratch: { scheme: 'levelcode-scratch', path: '/scratch' }, + trustedDomains: [account], + productConfiguration: Object.assign({ + // The scratch workspace implements file and text search for its own scheme. Spelling the grant + // out here (rather than relying on "built-in extensions may declare proposals") is the + // sanctioned route and keeps the workbench from logging an error about it at start. + extensionEnabledApiProposals: { 'levelcode.levelcode-web': ['fileSearchProvider', 'textSearchProvider'] }, + }, o.productConfiguration), + configurationDefaults: Object.assign({ + 'levelcode.cloud.endpoint': account, + ...(o.apiUrl ? { 'levelcode.cloud.apiUrl': trimSlash(o.apiUrl) } : {}), + // LevelCode is the assistant here; the stock Copilot surfaces (status item, chat view, command + // center) are for an account this editor does not use. The desktop app sets the same. + 'chat.disableAIFeatures': true, + 'chat.commandCenter.enabled': false, + 'workbench.startupEditor': 'none', + 'telemetry.telemetryLevel': 'off', + 'update.mode': 'none', + 'extensions.autoUpdate': false, + 'extensions.autoCheckUpdates': false, + 'security.workspace.trust.enabled': false, + }, o.configurationDefaults), + development: !!o.development, + }; +} + +export function renderIndex(config, htmlPath = path.join(WEB_DIR, 'index.html')) { + const json = JSON.stringify(config).replace(//g, '\\u003e').replace(/&/g, '\\u0026'); + return fs.readFileSync(htmlPath, 'utf8') + .replaceAll('{{BASE}}', config.base) + .replaceAll('{{STATIC}}', config.staticBase) + .replace('{{CONFIG}}', () => json); +} + +function trimSlash(s) { return String(s).replace(/\/+$/, ''); } diff --git a/web/lib/extensions.mjs b/web/lib/extensions.mjs new file mode 100644 index 0000000..569b75e --- /dev/null +++ b/web/lib/extensions.mjs @@ -0,0 +1,62 @@ +// Which of Code-OSS's built-in extensions the browser edition carries, and how they are staged. +// +// The web build bakes into its bundle only the extensions that have a browser entry point (about +// twenty: JSON, CSS and HTML language features, Markdown, the TypeScript language service...). The +// declarative ones that give an editor its colour — TextMate grammars, language configurations, +// themes, file icons — are not packaged for the web by this checkout's tooling, so without them a +// file is plain text. Declaring them is all they need, so they are staged beside LevelCode's own +// extensions and loaded as additional built-ins (the workbench reads each package.json). +import fs from 'node:fs'; +import path from 'node:path'; + +/** Directories never copied: tests, sources, and package managers' droppings. */ +const SKIP = new Set(['node_modules', 'test', 'tests', 'src', '.vscode', '.git', 'build']); +/** Extensions that exist in a Code-OSS checkout for its own development. */ +const DEV_ONLY = new Set(['vscode-api-tests', 'vscode-colorize-tests', 'vscode-colorize-perf-tests', 'vscode-test-resolver', 'copilot']); + +/** Mirrors build/lib/extensions.ts isWebExtension() for a manifest with neither main nor browser. */ +export function isDeclarativeWebExtension(manifest) { + if (manifest.main || manifest.browser) { return false; } + const c = manifest.contributes || {}; + for (const id of ['debuggers', 'terminal', 'typescriptServerPlugins']) { if (Object.prototype.hasOwnProperty.call(c, id)) { return false; } } + return true; +} + +/** + * @param {string} extensionsDir Code-OSS's extensions/ directory + * @param {string} outDir where to stage + * @param {{ exclude?: Iterable }} [opts] names to leave out (already baked into the bundle) + * @returns {string[]} the staged extension names, sorted + */ +export function stageDeclarativeExtensions(extensionsDir, outDir, opts = {}) { + const exclude = new Set(opts.exclude || []); + const staged = []; + for (const name of fs.readdirSync(extensionsDir).sort()) { + if (exclude.has(name) || DEV_ONLY.has(name)) { continue; } + const dir = path.join(extensionsDir, name); + const pkgPath = path.join(dir, 'package.json'); + if (!fs.statSync(dir).isDirectory() || !fs.existsSync(pkgPath)) { continue; } + let manifest; + try { manifest = JSON.parse(fs.readFileSync(pkgPath, 'utf8')); } catch { continue; } + if (!isDeclarativeWebExtension(manifest)) { continue; } + copyExtension(dir, path.join(outDir, name)); + // The workbench asks every extension for a package.nls.json; an absent one is a 404 in the console. + const nls = path.join(outDir, name, 'package.nls.json'); + if (!fs.existsSync(nls)) { fs.writeFileSync(nls, '{}\n'); } + staged.push(name); + } + return staged; +} + +/** Copy an extension directory without its development files and its translations (English ships). */ +export function copyExtension(from, to) { + fs.mkdirSync(to, { recursive: true }); + for (const e of fs.readdirSync(from, { withFileTypes: true })) { + if (SKIP.has(e.name)) { continue; } + if (/^package\.nls\..+\.json$/.test(e.name)) { continue; } // localisations: the build is English-only + const src = path.join(from, e.name); + const dst = path.join(to, e.name); + if (e.isDirectory()) { copyExtension(src, dst); } + else if (e.isFile()) { fs.copyFileSync(src, dst); } + } +} diff --git a/web/lib/fingerprint.mjs b/web/lib/fingerprint.mjs new file mode 100644 index 0000000..747cde9 --- /dev/null +++ b/web/lib/fingerprint.mjs @@ -0,0 +1,29 @@ +// What identifies a build. A release puts everything under /_// and tells browsers it never changes, so the id has to +// change whenever anything under it does: a file of the extension that was edited but not named here would be served +// stale, from cache, for a year. +import fs from 'node:fs'; +import path from 'node:path'; + +/** + * Feed every file under `dir` — its path relative to `dir`, then its bytes — into `hash`, in a stable order. + * Files and folders whose NAME is in `skip` are left out wherever they are. + * @param {import('node:crypto').Hash} hash + * @param {string} dir + * @param {{ skip?: string[] }} [opts] + * @returns {import('node:crypto').Hash} + */ +export function hashTree(hash, dir, { skip = [] } = {}) { + const skipped = new Set(skip); + const walk = (d, rel) => { + const entries = fs.readdirSync(d, { withFileTypes: true }).sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0)); + for (const e of entries) { + if (skipped.has(e.name)) { continue; } + const full = path.join(d, e.name); + const r = rel ? rel + '/' + e.name : e.name; + if (e.isDirectory()) { walk(full, r); } + else if (e.isFile()) { hash.update('\0' + r + '\0'); hash.update(fs.readFileSync(full)); } + } + }; + walk(dir, ''); + return hash; +} diff --git a/web/lib/headers.mjs b/web/lib/headers.mjs new file mode 100644 index 0000000..a1602c9 --- /dev/null +++ b/web/lib/headers.mjs @@ -0,0 +1,117 @@ +// The HTTP headers a static host must send for the browser edition, in one place. +// +// They are written out as a `_headers` file (Netlify and Cloudflare Pages read it), an nginx +// snippet, and read back by web/serve.mjs so the end-to-end check runs under the same policy a +// release does. A policy that is only written down is a policy nobody has tried. +// +// What the editor needs, and why the policy is not tighter: +// script-src 'unsafe-eval' Monaco and the TextMate/WASM engines compile code at runtime. +// style-src 'unsafe-inline' the workbench sets styles from script. +// connect-src https: BYOK sends the user's key to the provider they chose, from the tab. +// frame-src webviews and (optionally) the extension host live on origins of +// their own; they are listed explicitly. + +/** + * @param {{ account: string, webviewOrigin?: string, extHostOrigin?: string, base?: string }} o + * webviewOrigin / extHostOrigin: origin patterns such as https://*.view.example.com; omitted when the + * editor uses Code-OSS's default webview host (https://*.vscode-cdn.net) or an in-page extension host. + */ +export function policy(o) { + const frames = ["'self'"]; + frames.push(o.webviewOrigin || 'https://*.vscode-cdn.net'); + if (o.extHostOrigin) { frames.push(o.extHostOrigin); } + frames.push('data:'); + const workbench = [ + "default-src 'self'", + "script-src 'self' 'unsafe-eval' blob:", + "style-src 'self' 'unsafe-inline'", + "img-src 'self' https: data: blob:", + "media-src 'self' blob: data:", + "font-src 'self' blob:", + `connect-src 'self' https: http://localhost:* http://127.0.0.1:*`, + `frame-src ${frames.join(' ')}`, + "worker-src 'self' data: blob:", + "child-src 'self' blob:", + "manifest-src 'self'", + "object-src 'none'", + "base-uri 'none'", + "form-action 'none'", + "frame-ancestors 'none'", + ].join('; '); + const callback = [ + "default-src 'none'", + "script-src 'self'", + "style-src 'self'", + "img-src 'self'", + "base-uri 'none'", + "form-action 'none'", + "frame-ancestors 'none'", + ].join('; '); + const common = { + 'X-Content-Type-Options': 'nosniff', + 'Referrer-Policy': 'no-referrer', + 'Permissions-Policy': 'camera=(), microphone=(), geolocation=(), payment=(), usb=()', + }; + return { + // Exact paths first; a later rule only adds headers the earlier ones did not set. + rules: [ + { path: '/', headers: { ...common, 'Content-Security-Policy': workbench, 'Cache-Control': 'no-cache' } }, + { path: '/index.html', headers: { ...common, 'Content-Security-Policy': workbench, 'Cache-Control': 'no-cache' } }, + { path: '/callback.html', headers: { ...common, 'Content-Security-Policy': callback, 'Cache-Control': 'no-store' } }, + { path: '/callback.js', headers: { 'Cache-Control': 'no-cache' } }, + { path: '/callback.css', headers: { 'Cache-Control': 'no-cache' } }, + { path: '/manifest.webmanifest', headers: { 'Cache-Control': 'no-cache', 'Content-Type': 'application/manifest+json' } }, + // Everything under the build prefix is addressed by its build id: it never changes. + { path: `${o.base || '/_'}/*`, headers: { 'X-Content-Type-Options': 'nosniff', 'Cache-Control': 'public, max-age=31536000, immutable', + 'Access-Control-Allow-Origin': '*', 'Cross-Origin-Resource-Policy': 'cross-origin' } }, + ], + }; +} + +/** Netlify / Cloudflare Pages `_headers` text. */ +export function toHeadersFile(p) { + return p.rules.map((r) => r.path + '\n' + Object.entries(r.headers).map(([k, v]) => ` ${k}: ${v}`).join('\n')).join('\n\n') + '\n'; +} + +/** Parse a `_headers` file back into rules (the dev server serves a built release with it). */ +export function parseHeadersFile(text) { + const rules = []; + let cur = null; + for (const raw of text.split('\n')) { + if (!raw.trim() || raw.trim().startsWith('#')) { continue; } + if (!/^\s/.test(raw)) { cur = { path: raw.trim(), headers: {} }; rules.push(cur); continue; } + const i = raw.indexOf(':'); + if (cur && i > 0) { cur.headers[raw.slice(0, i).trim()] = raw.slice(i + 1).trim(); } + } + return { rules }; +} + +/** Headers for a request path, later and more specific rules overriding earlier ones. */ +export function headersFor(p, pathname) { + const out = {}; + for (const r of p.rules) { + const star = r.path.indexOf('*'); + const match = star < 0 ? r.path === pathname : pathname.startsWith(r.path.slice(0, star)); + if (match) { for (const [k, v] of Object.entries(r.headers)) { if (!(k in out)) { out[k] = v; } } } + } + return out; +} + +/** An nginx `server` fragment with the same rules, for a host that is not Netlify or Cloudflare. */ +export function toNginx(p, root = '/srv/levelcode-web') { + const lines = ['# LevelCode in your browser: a static site, nothing else. Generated by scripts/build-web.mjs.', + 'server {', ' listen 80;', ' server_name _;', ` root ${root};`, ' index index.html;', ' default_type application/octet-stream;', ' gzip on;', + ' gzip_types text/css application/javascript text/javascript application/json image/svg+xml;', '']; + for (const r of p.rules) { + const star = r.path.indexOf('*'); + const loc = star < 0 ? `location = ${r.path}` : `location ^~ ${r.path.slice(0, star)}`; + lines.push(` ${loc} {`); + for (const [k, v] of Object.entries(r.headers)) { + if (k === 'Content-Type') { lines.push(` types { } default_type ${v};`); continue; } + lines.push(` add_header ${k} "${v.replace(/"/g, '\\"')}" always;`); + } + lines.push(' try_files $uri =404;', ' }', ''); + } + lines.push(' location / { try_files $uri $uri/ =404; }', '}'); + return lines.join('\n') + '\n'; +} diff --git a/web/main.js b/web/main.js new file mode 100644 index 0000000..584a28c --- /dev/null +++ b/web/main.js @@ -0,0 +1,413 @@ +/*--------------------------------------------------------------------------------------------- + * LevelCode in your browser — page entry. + * + * Code-OSS ships a web client (src/vs/code/browser/workbench/workbench.ts) but it is written for + * its own server: it reads its configuration from a template the server fills in and assumes a + * remote authority. A static host has neither, so this file is the small amount of embedder code + * that a static host needs. It follows workbench.ts closely on purpose (MIT, same behaviour for + * the URL callback, the workspace URL and the secret store) and differs where LevelCode differs: + * + * - configuration comes from ', + '//levelcode.example/ai/login', // no scheme: not an absolute address + '/ai/login', + 'not a url', + '', + ]; + for (const u of refused) { + const t = makeLeave(); + t.leave(u, { account: ACCOUNT }); + assert.deepStrictEqual(t.assigned, [], 'went to ' + JSON.stringify(u)); + assert.strictEqual(t.session.size, 0, 'marked the tab for ' + JSON.stringify(u)); + } + }); + await test('leaveForSignIn: arguments that are not addresses, and a page with no usable account, do nothing', () => { + for (const bad of [undefined, null, 42, {}, ['https://levelcode.example/ai/login', 'x']]) { + const t = makeLeave(); + t.leave(bad, { account: ACCOUNT }); + if (Array.isArray(bad)) { continue; } // String([..]) joins; the origin test still decides, below + assert.deepStrictEqual(t.assigned, [], 'went somewhere for ' + JSON.stringify(bad)); + } + const t = makeLeave(); + t.leave(ACCOUNT + '/ai/login', { account: '' }); + t.leave(ACCOUNT + '/ai/login', {}); + assert.deepStrictEqual(t.assigned, []); + }); + await test('leaveForSignIn: sessionStorage being unavailable does not stop the sign-in', () => { + const assigned = []; + const leave = new Function('location', 'sessionStorage', 'console', 'RETURN_KEY', 'URL', + extract('leaveForSignIn') + '\nreturn leaveForSignIn;')( + { assign: (u) => assigned.push(u) }, + { setItem() { throw new Error('denied'); } }, console, RETURN_KEY, URL); + leave(ACCOUNT + '/ai/login', { account: ACCOUNT }); + assert.strictEqual(assigned.length, 1); + }); + + /* ---- the callback provider --------------------------------------------------------------------- */ + function makeProvider(entries, now = 1_000_000) { + const store = new Map(Object.entries(entries)); + const localStorage = { + get length() { return store.size; }, + key: (i) => [...store.keys()][i] ?? null, + getItem: (k) => (store.has(k) ? store.get(k) : null), + setItem: (k, v) => store.set(k, String(v)), + removeItem: (k) => store.delete(k), + }; + const realNow = Date.now; + Date.now = () => now; + const URI = { revive: (d) => ({ ...d, revived: true }), from: (c) => c }; + const win = { addEventListener() {}, removeEventListener() {} }; + const factory = new Function('URI', 'localStorage', 'window', 'location', 'setTimeout', 'console', + extract('createUrlCallbackProvider') + '\nreturn createUrlCallbackProvider;'); + const provider = factory(URI, localStorage, win, { protocol: 'https:', host: 'editor.example' }, setTimeout, console)(URI, '/callback.html'); + return { provider, store, restore: () => { Date.now = realNow; } }; + } + const KEY = (i) => 'vscode-web.url-callbacks[' + i + ']'; + const NOW = 1_000_000; + + await test('drain: a fresh result is delivered once and removed; one with no age is delivered too', () => { + const t = makeProvider({ + [KEY(1)]: JSON.stringify({ scheme: 'levelcode', authority: 'levelcode.levelcode-ai', path: '/auth/callback', query: 'code=a', at: NOW - 1000 }), + [KEY(2)]: JSON.stringify({ scheme: 'levelcode', authority: 'levelcode.levelcode-ai', path: '/auth/callback', query: 'code=b' }), + unrelated: 'keep me', + }); + try { + const got = []; + t.provider.onCallback((u) => got.push(u)); + assert.strictEqual(t.provider.drain(FRESH_MS), 2); + assert.deepStrictEqual(got.map((u) => u.query).sort(), ['code=a', 'code=b']); + assert.ok(got.every((u) => u.revived), 'delivered as URIs'); + assert.deepStrictEqual([...t.store.keys()], ['unrelated'], 'only the sign-in results were taken'); + assert.strictEqual(t.provider.drain(FRESH_MS), 0, 'and not again'); + } finally { t.restore(); } + }); + await test('drain: a result older than the limit is removed and NOT delivered', () => { + const t = makeProvider({ + [KEY(1)]: JSON.stringify({ scheme: 'levelcode', authority: 'levelcode.levelcode-ai', path: '/auth/callback', query: 'code=old', at: NOW - FRESH_MS - 1 }), + }); + try { + const got = []; + t.provider.onCallback((u) => got.push(u)); + assert.strictEqual(t.provider.drain(FRESH_MS), 0); + assert.deepStrictEqual(got, []); + assert.strictEqual(t.store.size, 0, 'a code that cannot be exchanged any more is not kept'); + } finally { t.restore(); } + }); + await test('drain: an entry that is not JSON is dropped without stopping the rest', () => { + const t = makeProvider({ + [KEY(1)]: '{not json', + [KEY(2)]: JSON.stringify({ scheme: 'levelcode', authority: 'levelcode.levelcode-ai', path: '/auth/callback', query: 'code=ok', at: NOW }), + }); + const errors = console.error; + console.error = () => {}; + try { + const got = []; + t.provider.onCallback((u) => got.push(u)); + assert.strictEqual(t.provider.drain(FRESH_MS), 1); + assert.strictEqual(got[0].query, 'code=ok'); + assert.strictEqual(t.store.size, 0); + } finally { console.error = errors; t.restore(); } + }); + await test('whenListening: true as soon as the workbench has subscribed, false when it never does', async () => { + const t = makeProvider({}); + try { + let state = 'pending'; + const p = t.provider.whenListening().then((v) => { state = v; return v; }); + await new Promise((r) => setTimeout(r, 30)); + assert.strictEqual(state, 'pending'); + t.provider.onCallback(() => {}); + assert.strictEqual(await p, true); + } finally { t.restore(); } + }); + + /* ---- the secret store --------------------------------------------------------------------------- */ + const nodeCrypto = require('crypto'); + const STORE_KEY = 'levelcode-web.secrets.v1'; + const SECRET_SOURCE = ['b64', 'unb64', 'seal', 'unseal', 'createSecretStorage'].map(extract).join('\n'); + const memoryStorage = () => { + const m = new Map(); + return { + m, + getItem: (k) => (m.has(k) ? m.get(k) : null), + setItem: (k, v) => { m.set(k, String(v)); }, + removeItem: (k) => { m.delete(k); }, + }; + }; + /** A mutex shared by every "tab" that is handed the same one, as Web Locks is. */ + const sharedLocks = () => { + let tail = Promise.resolve(); + const held = { max: 0, now: 0 }; + return { + held, + request: (_name, fn) => { + const run = async () => { held.now++; held.max = Math.max(held.max, held.now); try { return await fn(); } finally { held.now--; } }; + const p = tail.then(run, run); + tail = p.then(() => undefined, () => undefined); + return p; + }, + }; + }; + /** One tab of the editor: its own storage object over the shared localStorage, the shared origin key. */ + const tab = async ({ storage, key, locks, failKey = false, noKey = false }) => { + const logs = { error: [], warn: [] }; + const factory = new Function('localStorage', 'navigator', 'console', 'loadOrCreateKey', 'crypto', 'TextEncoder', 'TextDecoder', 'btoa', 'atob', 'Uint8Array', + SECRET_SOURCE + '\nreturn createSecretStorage;'); + const create = factory(storage, locks ? { locks } : {}, { warn: (...a) => logs.warn.push(a), error: (...a) => logs.error.push(a) }, + async () => { if (noKey) { throw new Error('no indexedDB'); } return key; }, nodeCrypto.webcrypto, TextEncoder, TextDecoder, btoa, atob, Uint8Array); + return { store: await create(), logs }; + }; + const newKey = () => nodeCrypto.webcrypto.subtle.generateKey({ name: 'AES-GCM', length: 256 }, false, ['encrypt', 'decrypt']); + + await test('secrets: set, get, keys and delete, and they are there after a reload', async () => { + const storage = memoryStorage(); + const key = await newKey(); + const a = await tab({ storage, key }); + assert.strictEqual(a.store.type, 'persisted'); + await a.store.set('access', 'token-AAA'); + await a.store.set('refresh', 'token-RRR'); + assert.strictEqual(await a.store.get('access'), 'token-AAA'); + assert.deepStrictEqual((await a.store.keys()).sort(), ['access', 'refresh']); + await a.store.delete('access'); + assert.strictEqual(await a.store.get('access'), undefined); + const reloaded = await tab({ storage, key }); + assert.strictEqual(await reloaded.store.get('refresh'), 'token-RRR'); + assert.deepStrictEqual(await reloaded.store.keys(), ['refresh']); + }); + await test('secrets: what is stored is sealed — neither the value nor its base64 is in it', async () => { + const storage = memoryStorage(); + const a = await tab({ storage, key: await newKey() }); + await a.store.set('refresh', 'a-very-recognisable-secret-value'); + const blob = storage.getItem(STORE_KEY); + assert.ok(blob && blob.length > 20); + assert.ok(!blob.includes('recognisable') && !blob.includes(Buffer.from('a-very-recognisable-secret-value').toString('base64'))); + }); + await test('secrets: another tab\'s change is seen at once — the rotated refresh token is the one a second tab uses', async () => { + const storage = memoryStorage(); + const key = await newKey(); + const locks = sharedLocks(); + const first = await tab({ storage, key, locks }); + await first.store.set('refresh', 'R1'); + await first.store.set('access', 'A1'); + const second = await tab({ storage, key, locks }); // started while R1 was current + assert.strictEqual(await second.store.get('refresh'), 'R1'); + await first.store.set('refresh', 'R2'); // the first tab renews the session + await first.store.set('access', 'A2'); + assert.strictEqual(await second.store.get('refresh'), 'R2', 'a second tab must not present the refresh token that was rotated away'); + assert.strictEqual(await second.store.get('access'), 'A2'); + }); + await test('secrets: a change is one key — a tab does not write back what it saw at start-up', async () => { + const storage = memoryStorage(); + const key = await newKey(); + const locks = sharedLocks(); + const first = await tab({ storage, key, locks }); + await first.store.set('access', 'A1'); + const second = await tab({ storage, key, locks }); + await first.store.set('refresh', 'R2'); // after the second tab started + await second.store.set('verifier', 'V'); // an unrelated change in the second tab + assert.strictEqual(await first.store.get('refresh'), 'R2', 'the second tab\'s write did not undo the first\'s'); + assert.strictEqual(await first.store.get('verifier'), 'V'); + await second.store.delete('access'); // the second tab signs out of the access token + assert.strictEqual(await first.store.get('refresh'), 'R2'); + assert.strictEqual(await first.store.get('access'), undefined); + }); + await test('secrets: changes made at the same moment in two tabs all land, one at a time', async () => { + const storage = memoryStorage(); + const key = await newKey(); + const locks = sharedLocks(); + const a = await tab({ storage, key, locks }); + const b = await tab({ storage, key, locks }); + const work = []; + for (let i = 0; i < 12; i++) { + work.push(a.store.set('a' + i, 'x' + i)); + work.push(b.store.set('b' + i, 'y' + i)); + } + await Promise.all(work); + const keys = await a.store.keys(); + assert.strictEqual(keys.length, 24, 'lost: ' + [...Array(12).keys()].flatMap((i) => ['a' + i, 'b' + i]).filter((k) => !keys.includes(k)).join(',')); + assert.strictEqual(locks.held.max, 1, 'never two changes inside the lock at once'); + }); + await test('secrets: with no Web Locks a tab still applies its own changes one at a time', async () => { + const storage = memoryStorage(); + const a = await tab({ storage, key: await newKey() }); + await Promise.all([...Array(10).keys()].map((i) => a.store.set('k' + i, 'v' + i))); + assert.strictEqual((await a.store.keys()).length, 10); + }); + await test('secrets: a store that cannot be read is dropped, with a warning, and the tab carries on', async () => { + const storage = memoryStorage(); + storage.setItem(STORE_KEY, 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'); + const a = await tab({ storage, key: await newKey() }); + assert.strictEqual(await a.store.get('refresh'), undefined); + assert.strictEqual(storage.getItem(STORE_KEY), null); + assert.strictEqual(a.logs.warn.length >= 1, true); + await a.store.set('refresh', 'R'); + assert.strictEqual(await a.store.get('refresh'), 'R'); + }); + await test('secrets: a key from another browser profile cannot read it — that is a sign-in again, not an error', async () => { + const storage = memoryStorage(); + const mine = await tab({ storage, key: await newKey() }); + await mine.store.set('refresh', 'R'); + const other = await tab({ storage, key: await newKey() }); + assert.strictEqual(await other.store.get('refresh'), undefined); + }); + await test('secrets: where there is no key the store is in memory, writes nothing, and works for the session', async () => { + const storage = memoryStorage(); + const a = await tab({ storage, key: null, noKey: true }); + assert.strictEqual(a.store.type, 'in-memory'); + await a.store.set('access', 'A'); + assert.strictEqual(await a.store.get('access'), 'A'); + assert.deepStrictEqual(await a.store.keys(), ['access']); + await a.store.delete('access'); + assert.strictEqual(await a.store.get('access'), undefined); + assert.strictEqual(storage.m.size, 0, 'nothing was written to localStorage'); + }); + await test('secrets: when the browser refuses the write the session is kept for this tab, not lost', async () => { + const storage = memoryStorage(); + const key = await newKey(); + const a = await tab({ storage, key }); + await a.store.set('refresh', 'R1'); + const realSet = storage.setItem; + storage.setItem = () => { throw new Error('QuotaExceededError'); }; + await a.store.set('access', 'A1'); // refused: kept in memory, with R1 carried over + storage.setItem = realSet; + assert.strictEqual(await a.store.get('access'), 'A1'); + assert.strictEqual(await a.store.get('refresh'), 'R1'); + await a.store.set('refresh', 'R2'); + assert.strictEqual(await a.store.get('refresh'), 'R2'); + assert.ok(a.logs.error.length >= 1, 'said so'); + }); + + /* ---- startReturnFromSignIn --------------------------------------------------------------------- */ + function makeStart(href, { drained = 0, listening = true, sessionThrows = false } = {}) { + const calls = { replaced: [], drain: [], fired: [], removed: [] }; + const location = { href }; + const history = { replaceState: (_s, _t, url) => calls.replaced.push(url) }; + const sessionStorage = { removeItem: (k) => { if (sessionThrows) { throw new Error('denied'); } calls.removed.push(k); } }; + const callbacks = { + whenListening: async () => listening, + drain: (ms) => { calls.drain.push(ms); return drained; }, + fire: (u) => calls.fired.push(u), + }; + const URI = { from: (c) => c }; + const start = new Function('location', 'history', 'sessionStorage', 'RETURN_KEY', 'FRESH_MS', 'URL', + extract('startReturnFromSignIn') + '\nreturn startReturnFromSignIn;')(location, history, sessionStorage, RETURN_KEY, FRESH_MS, URL); + return { start: () => start(callbacks, URI), calls }; + } + const tick = () => new Promise((r) => setTimeout(r, 5)); + + await test('start: ?signin=1 asks for the sign-in once, as the launch link, and leaves no trace in the address', async () => { + const t = makeStart('https://editor.example/?signin=1&keep=this'); + t.start(); await tick(); + assert.deepStrictEqual(t.calls.replaced, ['/?keep=this']); + assert.deepStrictEqual(t.calls.fired, [{ scheme: 'levelcode', authority: 'levelcode.levelcode-ai', path: '/launch' }]); + assert.deepStrictEqual(t.calls.drain, [FRESH_MS]); + assert.deepStrictEqual(t.calls.removed, [RETURN_KEY], 'the tab is no longer one that is expected back'); + }); + await test('start: when a sign-in result was waiting, it is delivered and no second sign-in is started', async () => { + const t = makeStart('https://editor.example/?signin=1', { drained: 1 }); + t.start(); await tick(); + assert.deepStrictEqual(t.calls.fired, []); + }); + await test('start: a plain start only delivers what is waiting', async () => { + const t = makeStart('https://editor.example/', { drained: 0 }); + t.start(); await tick(); + assert.deepStrictEqual(t.calls.replaced, [], 'the address is left alone'); + assert.deepStrictEqual(t.calls.fired, []); + assert.deepStrictEqual(t.calls.drain, [FRESH_MS]); + }); + await test('start: signin with any other value is not the parameter', async () => { + for (const q of ['?signin=0', '?signin=true', '?signin=', '?Signin=1']) { + const t = makeStart('https://editor.example/' + q); + t.start(); await tick(); + assert.deepStrictEqual(t.calls.fired, [], q); + assert.deepStrictEqual(t.calls.replaced, [], q); + } + }); + await test('start: with nobody listening nothing is taken out of storage and nothing is started', async () => { + const t = makeStart('https://editor.example/?signin=1', { listening: false }); + t.start(); await tick(); + assert.deepStrictEqual(t.calls.drain, []); + assert.deepStrictEqual(t.calls.fired, []); + }); + await test('start: sessionStorage being unavailable is not an error', async () => { + const t = makeStart('https://editor.example/?signin=1', { sessionThrows: true }); + t.start(); await tick(); + assert.strictEqual(t.calls.fired.length, 1); + }); + + console.log(`\n${n} tests passed`); +})().catch((e) => { console.error(e); process.exit(1); }); diff --git a/web/test/unit/manifest.test.js b/web/test/unit/manifest.test.js new file mode 100644 index 0000000..845b15d --- /dev/null +++ b/web/test/unit/manifest.test.js @@ -0,0 +1,98 @@ +/*--------------------------------------------------------------------------------------------- + * The browser manifest of levelcode-ai — run: node web/test/unit/manifest.test.js + * + * The browser build removes what a tab cannot do. These tests keep the removal honest: nothing + * the browser manifest still contributes may point at a command it dropped, the welcome page may + * only link to commands that exist, and the desktop manifest is never modified. + *--------------------------------------------------------------------------------------------*/ +// @ts-check +'use strict'; + +const assert = require('assert'); +const fs = require('fs'); +const path = require('path'); + +const ROOT = path.join(__dirname, '..', '..', '..'); +const desktop = JSON.parse(fs.readFileSync(path.join(ROOT, 'extensions', 'levelcode-ai', 'package.json'), 'utf8')); +const web = require('../../ai-extension/manifest.js'); +const scratch = JSON.parse(fs.readFileSync(path.join(ROOT, 'web', 'workspace', 'package.json'), 'utf8')); + +let n = 0; +function test(name, fn) { fn(); n++; console.log(' ok - ' + name); } + +const before = JSON.stringify(desktop); +const m = web.toWebManifest(desktop); + +test('the desktop manifest is not modified, and the browser one has its own entry', () => { + assert.strictEqual(JSON.stringify(desktop), before); + assert.strictEqual(m.browser, './extension.web.js'); + assert.strictEqual(desktop.browser, undefined); + assert.strictEqual(m.name, desktop.name); + assert.strictEqual(m.publisher, desktop.publisher, 'the extension id (publisher.name) is what the sign-in callback names'); +}); + +test('removed commands are gone from commands, menus, keybindings and activation events', () => { + const mentioned = new Set(); + const c = m.contributes; + for (const x of c.commands) { mentioned.add(x.command); } + for (const items of Object.values(c.menus)) { for (const i of items) { mentioned.add(i.command); } } + for (const k of c.keybindings) { mentioned.add(k.command); } + for (const e of m.activationEvents || []) { if (e.startsWith('onCommand:')) { mentioned.add(e.slice(10)); } } + for (const removed of web.REMOVED_COMMANDS) { + assert.ok(!mentioned.has(removed), removed + ' is still referenced'); + assert.ok(desktop.contributes.commands.some((x) => x.command === removed), removed + ' is not a desktop command, so removing it hides nothing'); + } +}); + +test('every command the browser manifest binds is one it declares', () => { + const declared = new Set(m.contributes.commands.map((x) => x.command)); + for (const items of Object.values(m.contributes.menus)) { + for (const i of items) { assert.ok(declared.has(i.command), 'menu item for undeclared ' + i.command); } + } + for (const k of m.contributes.keybindings) { assert.ok(declared.has(k.command), 'keybinding for undeclared ' + k.command); } +}); + +test('the welcome page links only to commands that exist, here or in the editor', () => { + const known = new Set([...m.contributes.commands.map((x) => x.command), ...scratch.contributes.commands.map((x) => x.command)]); + const builtIn = /^workbench\./; + const w = m.contributes.walkthroughs; + assert.strictEqual(w.length, 1); + for (const step of w[0].steps) { + for (const [, cmd] of step.description.matchAll(/\]\(command:([\w.]+)\)/g)) { + assert.ok(known.has(cmd) || builtIn.test(cmd), `${step.id} links to ${cmd}`); + } + for (const ev of step.completionEvents || []) { + const cmd = ev.replace(/^onCommand:/, ''); + assert.ok(known.has(cmd) || builtIn.test(cmd), `${step.id} completes on ${cmd}`); + } + } +}); + +test('the welcome page says nothing the browser cannot do', () => { + const text = JSON.stringify(m.contributes.walkthroughs).toLowerCase(); + for (const banned of ['hackable', 'runs commands', 'notepad++', 'keymap', 'check for updates', 'happy hacking']) { + assert.ok(!text.includes(banned), 'the welcome page still says "' + banned + '"'); + } +}); + +test('settings that need a shell, a disk or a local model server are not offered', () => { + const props = Object.assign({}, ...(Array.isArray(m.contributes.configuration) ? m.contributes.configuration : [m.contributes.configuration]).map((s) => s.properties)); + for (const removed of web.REMOVED_SETTINGS) { + assert.ok(!(removed in props), removed + ' is still offered'); + } + const dprops = Object.assign({}, ...(Array.isArray(desktop.contributes.configuration) ? desktop.contributes.configuration : [desktop.contributes.configuration]).map((s) => s.properties)); + for (const removed of web.REMOVED_SETTINGS) { assert.ok(removed in dprops, removed + ' is not a desktop setting: stale entry in REMOVED_SETTINGS'); } + assert.ok('levelcode.cloud.endpoint' in props && 'levelcode.ai.providerMode' in props, 'the settings the sign-in depends on are still here'); +}); + +test('the browser binds a key the browser lets through', () => { + assert.ok(m.contributes.keybindings.some((k) => k.command === 'levelcode.ai.focus' && k.key === 'ctrl+alt+i' && /isWeb/.test(k.when))); +}); + +test('the scratch workspace extension declares what it uses', () => { + assert.strictEqual(scratch.browser, './extension.js'); + assert.deepStrictEqual(scratch.enabledApiProposals.slice().sort(), ['fileSearchProvider', 'textSearchProvider']); + assert.ok(scratch.activationEvents.includes('onFileSystem:levelcode-scratch')); +}); + +console.log(`\n${n} tests passed`); diff --git a/web/test/unit/shims.test.js b/web/test/unit/shims.test.js new file mode 100644 index 0000000..0899069 --- /dev/null +++ b/web/test/unit/shims.test.js @@ -0,0 +1,198 @@ +/*--------------------------------------------------------------------------------------------- + * The browser build's stand-ins for Node (path, crypto, fs, os, child_process) — run: + * node web/test/unit/shims.test.js + * + * The extension was written against Node and the browser build swaps these in, so what matters + * is that they answer the way Node does. path and crypto are compared with Node itself over many + * inputs; fs is checked for the behaviours the extension depends on (and for the errors it + * catches by code); the rest is the contract the build relies on. + *--------------------------------------------------------------------------------------------*/ +// @ts-check +'use strict'; + +const assert = require('assert'); +const nodePath = require('path'); +const nodeCrypto = require('crypto'); +// A browser always has the global; Node has had it since 19 (18 only behind a flag). The shim is for the browser. +if (!globalThis.crypto) { globalThis.crypto = nodeCrypto.webcrypto; } + +const SHIMS = nodePath.join(__dirname, '..', '..', 'ai-extension', 'shims'); +const shimPath = require(nodePath.join(SHIMS, 'path.js')); +const shimCrypto = require(nodePath.join(SHIMS, 'crypto.js')); +const shimOs = require(nodePath.join(SHIMS, 'os.js')); +const shimCp = require(nodePath.join(SHIMS, 'child_process.js')); + +let n = 0; +function test(name, fn) { fn(); n++; console.log(' ok - ' + name); } + +// ---- path: the same answers as path.posix ------------------------------------------------------ +const PATHS = ['', '.', '..', '/', '//', 'a', 'a/b', '/a/b', '/a/b/', 'a//b', './a', '../a', 'a/../b', '/a/../..', '/a/./b/../c.txt', + 'a/b/c.d.e', '.hidden', '/.hidden/file', 'dir.d/file', 'trailing/', '/x/y/z.tar.gz', 'C:\\win', '\u00e9/\u4e2d', ' spaced /name ']; +test('path: normalize, dirname, basename, extname, isAbsolute match path.posix', () => { + for (const p of PATHS) { + assert.strictEqual(shimPath.normalize(p), nodePath.posix.normalize(p), 'normalize ' + JSON.stringify(p)); + assert.strictEqual(shimPath.dirname(p), nodePath.posix.dirname(p), 'dirname ' + JSON.stringify(p)); + assert.strictEqual(shimPath.basename(p), nodePath.posix.basename(p), 'basename ' + JSON.stringify(p)); + assert.strictEqual(shimPath.basename(p, '.txt'), nodePath.posix.basename(p, '.txt'), 'basename ext ' + JSON.stringify(p)); + assert.strictEqual(shimPath.extname(p), nodePath.posix.extname(p), 'extname ' + JSON.stringify(p)); + assert.strictEqual(shimPath.isAbsolute(p), nodePath.posix.isAbsolute(p), 'isAbsolute ' + JSON.stringify(p)); + } +}); +test('path: join and resolve match path.posix over pairs and triples', () => { + const parts = ['', '/', 'a', 'b/c', '../x', './y', '/abs', 'trail/', '..']; + for (const a of parts) { + for (const b of parts) { + assert.strictEqual(shimPath.join(a, b), nodePath.posix.join(a, b), `join(${JSON.stringify(a)}, ${JSON.stringify(b)})`); + // resolve() reads the process cwd in Node; the shim's cwd is "/". Compare with an absolute first part. + assert.strictEqual(shimPath.resolve('/', a, b), nodePath.posix.resolve('/', a, b), `resolve(/, ${JSON.stringify(a)}, ${JSON.stringify(b)})`); + } + } + assert.strictEqual(shimPath.resolve('a/b'), '/a/b', 'a relative path resolves against "/"'); +}); +test('path: relative matches path.posix for absolute pairs', () => { + const abs = ['/', '/a', '/a/b', '/a/b/c', '/x', '/a/x/y']; + for (const a of abs) { for (const b of abs) { assert.strictEqual(shimPath.relative(a, b), nodePath.posix.relative(a, b), `relative(${a}, ${b})`); } } +}); +test('path: parse and format round-trip like path.posix', () => { + for (const p of ['/a/b/c.txt', 'c.txt', '/a/b/', '.gitignore', '/a/b.c/d']) { + const mine = shimPath.parse(p), node = nodePath.posix.parse(p); + assert.deepStrictEqual(mine, node, 'parse ' + p); + assert.strictEqual(shimPath.format(mine), nodePath.posix.format(node), 'format ' + p); + } +}); +test('path: sep, posix, win32 are what callers read', () => { + assert.strictEqual(shimPath.sep, '/'); + assert.strictEqual(shimPath.posix, shimPath); +}); + +// ---- crypto: SHA-256 is computed here because WebCrypto's is asynchronous --------------------- +test('crypto: sha256 matches Node for every length around the block boundaries, hex and raw', () => { + for (let len = 0; len <= 200; len++) { + const buf = Buffer.alloc(len, len % 251); + assert.strictEqual(shimCrypto.createHash('sha256').update(buf).digest('hex'), nodeCrypto.createHash('sha256').update(buf).digest('hex'), 'length ' + len); + } + const text = 'caf\u00e9 \u4e2d\u6587 \ud83d\ude80'; + assert.strictEqual(shimCrypto.createHash('sha256').update(text).digest('hex'), nodeCrypto.createHash('sha256').update(text).digest('hex')); + assert.deepStrictEqual([...shimCrypto.createHash('sha256').update('abc').digest()], [...nodeCrypto.createHash('sha256').update('abc').digest()]); +}); +test('crypto: updates may be chunked, and a PKCE challenge comes out as RFC 7636 says', () => { + const h = shimCrypto.createHash('sha256'); + h.update('hello ').update(Buffer.from('wor')).update('ld'); + assert.strictEqual(h.digest('hex'), nodeCrypto.createHash('sha256').update('hello world').digest('hex')); + // RFC 7636 appendix B + const verifier = 'dBjftJeZ4CVP-mB92K27uhbUJU1p1r_wW1gFWFOEjXk'; + const b64url = (b) => b.toString('base64').replace(/\+/g, '-').replace(/\//g, '_').replace(/=+$/, ''); + assert.strictEqual(b64url(shimCrypto.createHash('sha256').update(verifier).digest()), 'E9Melhoa2OwvFrEMTJguCHaoeK1t8URWbuGJSstw-cM'); +}); +test('crypto: a digest is single use, an unknown algorithm says so, randomness is random', () => { + const h = shimCrypto.createHash('sha256'); h.digest(); + assert.throws(() => h.digest(), /already called/); + assert.throws(() => shimCrypto.createHash('md5'), (e) => e.code === 'ERR_CRYPTO_INVALID_DIGEST'); + const a = shimCrypto.randomBytes(32), b = shimCrypto.randomBytes(32); + assert.strictEqual(a.length, 32); + assert.notDeepStrictEqual([...a], [...b]); + assert.strictEqual(shimCrypto.randomBytes(3).toString('hex').length, 6); + assert.match(shimCrypto.randomUUID(), /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/); +}); + +// ---- os and child_process -------------------------------------------------------------------- +test('os: a home directory under which private state lives, and nothing that needs a machine', () => { + assert.strictEqual(shimOs.homedir(), '/home/levelcode'); + assert.strictEqual(shimOs.EOL, '\n'); + assert.strictEqual(typeof shimOs.tmpdir(), 'string'); +}); +test('child_process: every way of starting a program fails the way an unsupported syscall does', () => { + for (const name of ['spawn', 'spawnSync', 'exec', 'execSync', 'execFile', 'execFileSync', 'fork']) { + assert.throws(() => shimCp[name]('ls'), (e) => e.code === 'ENOSYS', name); + } +}); + +// ---- fs ---------------------------------------------------------------------------------------- +const fs = require(nodePath.join(SHIMS, 'fs.js')); +const code = (fn) => { try { fn(); } catch (e) { return e.code; } return null; }; + +test('fs: write, read (string and bytes), append, overwrite', () => { + fs.writeFileSync('/home/levelcode/.levelcode/a.txt', 'one'); + assert.strictEqual(fs.readFileSync('/home/levelcode/.levelcode/a.txt', 'utf8'), 'one'); + assert.deepStrictEqual([...fs.readFileSync('/home/levelcode/.levelcode/a.txt')], [111, 110, 101]); + fs.appendFileSync('/home/levelcode/.levelcode/a.txt', '+two'); + assert.strictEqual(fs.readFileSync('/home/levelcode/.levelcode/a.txt', { encoding: 'utf8' }), 'one+two'); + fs.writeFileSync('/home/levelcode/.levelcode/a.txt', Buffer.from([0, 1, 2])); + assert.deepStrictEqual([...fs.readFileSync('/home/levelcode/.levelcode/a.txt')], [0, 1, 2]); + assert.strictEqual(fs.readFileSync('/home/levelcode/.levelcode/a.txt').length, 3); +}); +test('fs: writing creates the directories above it, as the extension\'s mkdirSync-then-write calls expect', () => { + fs.writeFileSync('/p/q/r/file.json', '{}'); + assert.strictEqual(fs.existsSync('/p/q/r'), true); + assert.strictEqual(fs.statSync('/p/q').isDirectory(), true); + assert.strictEqual(fs.statSync('/p/q/r/file.json').isFile(), true); + assert.strictEqual(fs.statSync('/p/q/r/file.json').size, 2); +}); +test('fs: errors carry the codes the extension catches', () => { + assert.strictEqual(code(() => fs.readFileSync('/nope/missing')), 'ENOENT'); + assert.strictEqual(code(() => fs.statSync('/nope/missing')), 'ENOENT'); + assert.strictEqual(code(() => fs.readdirSync('/nope')), 'ENOENT'); + assert.strictEqual(code(() => fs.readFileSync('/p/q')), 'EISDIR'); + assert.strictEqual(code(() => fs.readdirSync('/p/q/r/file.json')), 'ENOTDIR'); + assert.strictEqual(code(() => fs.mkdirSync('/p/q')), 'EEXIST'); + assert.strictEqual(code(() => fs.mkdirSync('/p/q', { recursive: true })), null); + assert.strictEqual(code(() => fs.mkdirSync('/m/n/o')), 'ENOENT'); + assert.strictEqual(code(() => fs.unlinkSync('/p/q')), 'EISDIR'); + assert.strictEqual(code(() => fs.rmdirSync('/p/q')), 'ENOTEMPTY'); +}); +test('fs: existsSync never throws, statSync can answer "absent" without throwing', () => { + assert.strictEqual(fs.existsSync('/p/q/r/file.json'), true); + assert.strictEqual(fs.existsSync('/nope'), false); + assert.strictEqual(fs.existsSync(''), false); + assert.strictEqual(fs.statSync('/nope', { throwIfNoEntry: false }), undefined); +}); +test('fs: readdir lists names sorted, with types on request', () => { + fs.writeFileSync('/d/b.txt', '1'); fs.writeFileSync('/d/a.txt', '2'); fs.mkdirSync('/d/sub', { recursive: true }); fs.writeFileSync('/d/sub/deep.txt', '3'); + assert.deepStrictEqual(fs.readdirSync('/d'), ['a.txt', 'b.txt', 'sub']); + assert.deepStrictEqual(fs.readdirSync('/d', { withFileTypes: true }).map((e) => [e.name, e.isFile(), e.isDirectory()]), + [['a.txt', true, false], ['b.txt', true, false], ['sub', false, true]]); +}); +test('fs: rename moves a file and a directory with what is under it; unlink and rm remove', () => { + fs.renameSync('/d/b.txt', '/d/c.txt'); + assert.strictEqual(fs.existsSync('/d/b.txt'), false); + assert.strictEqual(fs.readFileSync('/d/c.txt', 'utf8'), '1'); + fs.renameSync('/d/sub', '/e'); + assert.strictEqual(fs.readFileSync('/e/deep.txt', 'utf8'), '3'); + assert.strictEqual(fs.existsSync('/d/sub'), false); + fs.unlinkSync('/d/c.txt'); + assert.strictEqual(fs.existsSync('/d/c.txt'), false); + assert.strictEqual(code(() => fs.rmSync('/e')), 'ENOTEMPTY'); + fs.rmSync('/e', { recursive: true }); + assert.strictEqual(fs.existsSync('/e'), false); + assert.strictEqual(code(() => fs.rmSync('/e')), 'ENOENT'); + assert.strictEqual(code(() => fs.rmSync('/e', { force: true })), null); +}); +test('fs: copyFileSync copies bytes; a returned Buffer is a copy', () => { + fs.writeFileSync('/c/src.bin', Buffer.from([9, 8, 7])); + fs.copyFileSync('/c/src.bin', '/c/dst.bin'); + const got = fs.readFileSync('/c/dst.bin'); + got[0] = 0; + assert.deepStrictEqual([...fs.readFileSync('/c/dst.bin')], [9, 8, 7]); +}); +test('fs: the promise and callback forms agree with the sync one', async () => { + await fs.promises.writeFile('/pr/a.txt', 'promised'); + assert.strictEqual(await fs.promises.readFile('/pr/a.txt', 'utf8'), 'promised'); + await assert.rejects(fs.promises.readFile('/pr/none'), (e) => e.code === 'ENOENT'); + await new Promise((resolve, reject) => fs.writeFile('/cb/a.txt', 'called', (err) => (err ? reject(err) : resolve()))); + await new Promise((resolve, reject) => fs.stat('/cb/a.txt', (err, st) => (err ? reject(err) : (assert.strictEqual(st.size, 6), resolve())))); +}); +test('fs: mounted assets are readable and read-only, and survive being listed', () => { + fs.__levelcode.mountAssets('/ext/levelcode-ai', { 'media/chat.html': '

hi

', 'skills/a/SKILL.md': '# a', 'media/x.bin': { base64: Buffer.from([1, 2, 3]).toString('base64') } }); + assert.strictEqual(fs.readFileSync('/ext/levelcode-ai/media/chat.html', 'utf8'), '

hi

'); + assert.deepStrictEqual([...fs.readFileSync('/ext/levelcode-ai/media/x.bin')], [1, 2, 3]); + assert.deepStrictEqual(fs.readdirSync('/ext/levelcode-ai/skills'), ['a']); + assert.strictEqual(code(() => fs.writeFileSync('/ext/levelcode-ai/media/chat.html', 'x')), 'EROFS'); + assert.strictEqual(code(() => fs.unlinkSync('/ext/levelcode-ai/media/chat.html')), 'EROFS'); +}); +test('fs: where there is no IndexedDB the tree is simply session-only, and flush is harmless', async () => { + assert.strictEqual(await fs.__levelcode.ready, false); + await fs.__levelcode.flush(); + assert.strictEqual(fs.__levelcode.isPersistent(), false); +}); + +console.log(`\n${n} tests passed`); diff --git a/web/test/unit/workspace.test.js b/web/test/unit/workspace.test.js new file mode 100644 index 0000000..18ba0ff --- /dev/null +++ b/web/test/unit/workspace.test.js @@ -0,0 +1,64 @@ +/*--------------------------------------------------------------------------------------------- + * web/workspace/extension.js — run: node web/test/unit/workspace.test.js + * + * The scratch-workspace extension is evaluated as ONE file by the web extension host, so it cannot be split into + * modules; this test loads that file against a stand-in for `vscode` and runs what it exports. + * + * openLocalFolder must ask the dialog from the `file` scheme. From the scratch scheme the workbench shows its + * own file browser instead of the browser's picker (the command did nothing useful until + * it was run in a real browser). + *--------------------------------------------------------------------------------------------*/ +// @ts-check +'use strict'; + +const assert = require('assert'); +const path = require('path'); +const Module = require('module'); + +class Uri { + constructor(scheme, p) { this.scheme = scheme; this.path = p; } + static from(c) { return new Uri(c.scheme, c.path || '/'); } + toString() { return this.scheme + '://' + this.path; } +} +const calls = { dialogs: /** @type {any[]} */ ([]), commands: /** @type {any[][]} */ ([]) }; +let nextPick = /** @type {any} */ (undefined); +const vscodeMock = { + Uri, + window: { async showOpenDialog(o) { calls.dialogs.push(o); return nextPick; } }, + commands: { async executeCommand(...a) { calls.commands.push(a); } }, +}; +const origLoad = Module._load; +// @ts-ignore +Module._load = function (request, parent, isMain) { return request === 'vscode' ? vscodeMock : origLoad.call(this, request, parent, isMain); }; +const ext = require(path.join(__dirname, '..', '..', 'workspace', 'extension.js')); + +let n = 0; +async function test(name, fn) { calls.dialogs.length = 0; calls.commands.length = 0; await fn(); n++; console.log(' ok - ' + name); } + +(async () => { + await test('the picker is asked for a folder, starting from the file scheme, not the scratch scheme', async () => { + nextPick = [new Uri('file', '/my-project')]; + await ext.openLocalFolder(vscodeMock); + assert.strictEqual(calls.dialogs.length, 1); + const o = calls.dialogs[0]; + assert.strictEqual(o.defaultUri.scheme, 'file', 'a default location in another scheme gets the workbench\'s own file browser'); + assert.strictEqual(o.canSelectFolders, true); + assert.strictEqual(o.canSelectFiles, false); + assert.strictEqual(o.canSelectMany, false); + }); + await test('the folder that was picked becomes the workspace, in this tab', async () => { + const picked = new Uri('file', '/my-project'); + nextPick = [picked]; + assert.strictEqual(await ext.openLocalFolder(vscodeMock), true); + assert.deepStrictEqual(calls.commands, [['vscode.openFolder', picked, { forceReuseWindow: true }]]); + }); + await test('cancelling the picker does nothing', async () => { + for (const none of [undefined, []]) { + nextPick = none; + calls.commands.length = 0; + assert.strictEqual(await ext.openLocalFolder(vscodeMock), false); + assert.deepStrictEqual(calls.commands, []); + } + }); + console.log(`\n${n} tests passed`); +})().catch((e) => { console.error(e); process.exit(1); }); diff --git a/web/workspace/extension.js b/web/workspace/extension.js new file mode 100644 index 0000000..66fb217 --- /dev/null +++ b/web/workspace/extension.js @@ -0,0 +1,432 @@ +/*--------------------------------------------------------------------------------------------- + * LevelCode in your browser — the scratch workspace. + * + * A browser has no project folder to open until the user picks one (and only Chromium can). + * The scratch workspace is a folder that always exists: files live in this browser's IndexedDB, + * survive reloads, and are shared by every tab of this origin. A signed-in user can ask LevelCode + * to build something with no further setup, then move the files out (download, or copy into a + * real folder opened from their computer). + * + * It is a FileSystemProvider for the scheme `levelcode-scratch`. Plain JS: the web extension + * host evaluates this one file, no bundler involved. + *--------------------------------------------------------------------------------------------*/ +'use strict'; + +const vscode = require('vscode'); + +const SCHEME = 'levelcode-scratch'; +const DB_NAME = 'levelcode-scratch'; +const STORE = 'nodes'; +const CHANNEL = 'levelcode-scratch-changes'; +const FILE = 1, DIR = 2; // vscode.FileType.File / Directory + +const WELCOME = [ + '# Scratch workspace', + '', + 'This folder lives in your browser. It is saved on this computer, in this browser,', + 'and nowhere else.', + '', + '- Ask LevelCode to build something and the files appear here.', + '- To work on a real project, use **LevelCode: Open Folder from Your Computer** (Chrome and Edge).', + '- To keep these files, download them from the Explorer (right-click, Download).', + '', +].join('\n'); + +/* ----- IndexedDB -------------------------------------------------------------------------- */ + +function openDb() { + return new Promise((resolve, reject) => { + const req = indexedDB.open(DB_NAME, 1); + req.onupgradeneeded = () => { req.result.createObjectStore(STORE, { keyPath: 'path' }); }; + req.onsuccess = () => resolve(req.result); + req.onerror = () => reject(req.error); + }); +} +function request(r) { return new Promise((res, rej) => { r.onsuccess = () => res(r.result); r.onerror = () => rej(r.error); }); } +function done(tx) { return new Promise((res, rej) => { tx.oncomplete = () => res(); tx.onerror = () => rej(tx.error); tx.onabort = () => rej(tx.error); }); } + +const norm = (p) => { const s = '/' + String(p || '/').split('/').filter(Boolean).join('/'); return s; }; +const parentOf = (p) => (p === '/' ? '/' : norm(p.slice(0, p.lastIndexOf('/')))); +const under = (p) => (p === '/' ? '/' : p + '/'); + +class ScratchFileSystem { + constructor(db) { + this.db = db; + this._emitter = new vscode.EventEmitter(); + this.onDidChangeFile = this._emitter.event; + this.channel = typeof BroadcastChannel === 'function' ? new BroadcastChannel(CHANNEL) : null; + if (this.channel) { + this.channel.onmessage = (e) => { + const events = (e.data || []).map((c) => ({ type: c.type, uri: vscode.Uri.from({ scheme: SCHEME, path: c.path }) })); + if (events.length) { this._emitter.fire(events); } + }; + } + } + + _changed(list) { + this._emitter.fire(list.map((c) => ({ type: c.type, uri: vscode.Uri.from({ scheme: SCHEME, path: c.path }) }))); + if (this.channel) { this.channel.postMessage(list); } + } + + async _get(path) { + return request(this.db.transaction(STORE).objectStore(STORE).get(path)); + } + async _range(path) { + const lo = under(path); + return request(this.db.transaction(STORE).objectStore(STORE).getAll(IDBKeyRange.bound(lo, lo + '￿'))); + } + async _isDir(path) { + if (path === '/') { return true; } + const n = await this._get(path); + return !!n && n.type === DIR; + } + + watch() { return new vscode.Disposable(() => { }); } + + async stat(uri) { + const path = norm(uri.path); + if (path === '/') { const t = Date.now(); return { type: vscode.FileType.Directory, ctime: t, mtime: t, size: 0 }; } + const n = await this._get(path); + if (!n) { throw vscode.FileSystemError.FileNotFound(uri); } + return { type: n.type === DIR ? vscode.FileType.Directory : vscode.FileType.File, ctime: n.ctime, mtime: n.mtime, size: n.size || 0 }; + } + + async readDirectory(uri) { + const path = norm(uri.path); + if (!(await this._isDir(path))) { + throw (await this._get(path)) ? vscode.FileSystemError.FileNotADirectory(uri) : vscode.FileSystemError.FileNotFound(uri); + } + const prefix = under(path); + const out = new Map(); + for (const n of await this._range(path)) { + const rest = n.path.slice(prefix.length); + const i = rest.indexOf('/'); + const name = i < 0 ? rest : rest.slice(0, i); + if (!name) { continue; } + out.set(name, i >= 0 || n.type === DIR ? vscode.FileType.Directory : vscode.FileType.File); + } + return [...out.entries()]; + } + + async readFile(uri) { + const path = norm(uri.path); + const n = await this._get(path); + if (!n) { throw vscode.FileSystemError.FileNotFound(uri); } + if (n.type === DIR) { throw vscode.FileSystemError.FileIsADirectory(uri); } + return n.data instanceof Uint8Array ? n.data : new Uint8Array(n.data || 0); + } + + /** Make every missing ancestor of `path` a directory, in the same transaction. */ + async _mkdirs(store, path, created) { + const missing = []; + let cur = parentOf(path); + while (cur !== '/') { + const n = await request(store.get(cur)); + if (n) { if (n.type !== DIR) { throw vscode.FileSystemError.FileNotADirectory(vscode.Uri.from({ scheme: SCHEME, path: cur })); } break; } + missing.push(cur); + cur = parentOf(cur); + } + const t = Date.now(); + for (const d of missing.reverse()) { store.put({ path: d, type: DIR, ctime: t, mtime: t, size: 0 }); created.push({ type: vscode.FileChangeType.Created, path: d }); } + } + + async writeFile(uri, content, options) { + const path = norm(uri.path); + if (path === '/') { throw vscode.FileSystemError.FileIsADirectory(uri); } + const tx = this.db.transaction(STORE, 'readwrite'); + const store = tx.objectStore(STORE); + const existing = await request(store.get(path)); + if (existing && existing.type === DIR) { tx.abort(); throw vscode.FileSystemError.FileIsADirectory(uri); } + if (!existing && !options.create) { tx.abort(); throw vscode.FileSystemError.FileNotFound(uri); } + if (existing && options.create && !options.overwrite) { tx.abort(); throw vscode.FileSystemError.FileExists(uri); } + const changes = []; + await this._mkdirs(store, path, changes); + const t = Date.now(); + store.put({ path, type: FILE, ctime: existing ? existing.ctime : t, mtime: t, size: content.byteLength, data: content }); + await done(tx); + changes.push({ type: existing ? vscode.FileChangeType.Changed : vscode.FileChangeType.Created, path }); + this._changed(changes); + } + + async createDirectory(uri) { + const path = norm(uri.path); + if (path === '/') { return; } + const tx = this.db.transaction(STORE, 'readwrite'); + const store = tx.objectStore(STORE); + const existing = await request(store.get(path)); + if (existing) { tx.abort(); throw vscode.FileSystemError.FileExists(uri); } + const changes = []; + await this._mkdirs(store, path, changes); + const t = Date.now(); + store.put({ path, type: DIR, ctime: t, mtime: t, size: 0 }); + await done(tx); + changes.push({ type: vscode.FileChangeType.Created, path }); + this._changed(changes); + } + + async delete(uri, options) { + const path = norm(uri.path); + const tx = this.db.transaction(STORE, 'readwrite'); + const store = tx.objectStore(STORE); + const n = await request(store.get(path)); + if (!n && path !== '/') { tx.abort(); throw vscode.FileSystemError.FileNotFound(uri); } + const changes = []; + if (n && n.type === DIR) { + const kids = await request(store.getAllKeys(IDBKeyRange.bound(under(path), under(path) + '￿'))); + if (kids.length && !(options && options.recursive)) { tx.abort(); throw vscode.FileSystemError.NoPermissions(uri); } + for (const k of kids) { store.delete(k); changes.push({ type: vscode.FileChangeType.Deleted, path: k }); } + } + if (n) { store.delete(path); changes.push({ type: vscode.FileChangeType.Deleted, path }); } + await done(tx); + this._changed(changes); + } + + async rename(from, to, options) { return this._transfer(from, to, options, true); } + async copy(from, to, options) { return this._transfer(from, to, options, false); } + + async _transfer(from, to, options, move) { + const a = norm(from.path); + const b = norm(to.path); + if (a === b) { return; } + if (b.startsWith(under(a))) { throw vscode.FileSystemError.NoPermissions(to); } + const tx = this.db.transaction(STORE, 'readwrite'); + const store = tx.objectStore(STORE); + const src = await request(store.get(a)); + if (!src) { tx.abort(); throw vscode.FileSystemError.FileNotFound(from); } + const dst = await request(store.get(b)); + if (dst && !(options && options.overwrite)) { tx.abort(); throw vscode.FileSystemError.FileExists(to); } + const changes = []; + await this._mkdirs(store, b, changes); + if (dst) { store.delete(b); } + const t = Date.now(); + const nodes = [src]; + if (src.type === DIR) { nodes.push(...await request(store.getAll(IDBKeyRange.bound(under(a), under(a) + '￿')))); } + for (const n of nodes) { + const np = b + n.path.slice(a.length); + store.put(Object.assign({}, n, { path: np, mtime: move ? n.mtime : t, ctime: move ? n.ctime : t })); + changes.push({ type: vscode.FileChangeType.Created, path: np }); + if (move) { store.delete(n.path); changes.push({ type: vscode.FileChangeType.Deleted, path: n.path }); } + } + await done(tx); + this._changed(changes); + } +} + +/* ----- search ----------------------------------------------------------------------------- */ + +// Quick Open, the Search view and workspace.findFiles (which the agent's list_files uses) ask the +// provider registered for the folder's scheme. Without one they wait forever, so the scratch +// workspace answers its own questions. Everything is in IndexedDB; a scan is cheap at this size. + +/** A VS Code glob as a RegExp over a '/'-relative path. A pattern with no '/' matches at any depth. */ +function globToRegExp(glob) { + const src = String(glob); + const anyDepth = !src.includes('/'); + const conv = (g) => { + let re = ''; + for (let i = 0; i < g.length;) { + const c = g[i]; + if (c === '*') { + if (g[i + 1] === '*') { + i += 2; + if (g[i] === '/') { i++; re += '(?:.*/)?'; } else { re += '.*'; } + } else { re += '[^/]*'; i++; } + } else if (c === '?') { re += '[^/]'; i++; } + else if (c === '{') { + const j = g.indexOf('}', i); + if (j < 0) { re += '\\{'; i++; } + else { re += '(?:' + g.slice(i + 1, j).split(',').map(conv).join('|') + ')'; i = j + 1; } + } else if (c === '[') { + const j = g.indexOf(']', i + 1); + if (j < 0) { re += '\\['; i++; } + else { re += '[' + g.slice(i + 1, j).replace(/^!/, '^').replace(/\\/g, '\\\\') + ']'; i = j + 1; } + } else { re += c.replace(/[.+^${}()|\\\/]/g, '\\$&'); i++; } + } + return re; + }; + return new RegExp('^' + (anyDepth ? '(?:.*/)?' : '') + conv(src.replace(/^\.\//, '').replace(/^\//, '')) + '$'); +} + +/** True when `rel` or any directory above it matches one of the globs. */ +function matchesAny(rel, res) { + if (!res.length) { return false; } + const parts = rel.split('/'); + let cur = ''; + for (const part of parts) { + cur = cur ? cur + '/' + part : part; + if (res.some((r) => r.test(cur))) { return true; } + } + return false; +} + +function subsequence(hay, needle) { + let j = 0; + for (let i = 0; i < hay.length && j < needle.length; i++) { if (hay[i] === needle[j]) { j++; } } + return j === needle.length; +} + +class ScratchSearch { + constructor(fsp) { this.fsp = fsp; } + + async _files(folder, includes, excludes) { + const root = norm(folder.path); + const prefix = under(root); + const inc = (includes || []).map(globToRegExp); + const exc = (excludes || []).map(globToRegExp); + const nodes = await this.fsp._range(root); + const out = []; + for (const n of nodes) { + if (n.type !== FILE) { continue; } + const rel = n.path.slice(prefix.length); + if (matchesAny(rel, exc)) { continue; } + if (inc.length && !inc.some((r) => r.test(rel))) { continue; } + out.push({ rel, node: n }); + } + return out; + } + + async provideFileSearchResults(query, options, token) { + const want = String(query.pattern || '').toLowerCase().replace(/\\/g, '/'); + const files = await this._files(options.folder, options.includes, options.excludes); + const hits = []; + for (const f of files) { + if (token.isCancellationRequested) { break; } + const hay = f.rel.toLowerCase(); + if (!want || hay.includes(want) || subsequence(hay, want)) { hits.push(f); } + } + const max = options.maxResults || 10000; + return hits.slice(0, max).map((f) => vscode.Uri.from({ scheme: SCHEME, path: f.node.path })); + } + + async provideTextSearchResults(query, options, progress, token) { + const files = await this._files(options.folder, options.includes, options.excludes); + let re; + try { + const src = query.isRegExp ? query.pattern : query.pattern.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); + re = new RegExp(query.isWordMatch ? '\\b(?:' + src + ')\\b' : src, query.isCaseSensitive ? 'g' : 'gi'); + } catch (e) { return { limitHit: false }; } + const max = options.maxResults || 10000; + const maxSize = options.maxFileSize || 1024 * 1024; + let count = 0; + const decoder = new TextDecoder('utf-8'); + for (const f of files) { + if (token.isCancellationRequested) { break; } + const data = f.node.data; + if (!data || data.byteLength > maxSize) { continue; } + if (data.subarray(0, 8000).includes(0)) { continue; } // binary + const lines = decoder.decode(data).split('\n'); + const uri = vscode.Uri.from({ scheme: SCHEME, path: f.node.path }); + for (let ln = 0; ln < lines.length; ln++) { + const line = lines[ln].replace(/\r$/, ''); + re.lastIndex = 0; + const ranges = []; + const previews = []; + let m; + while ((m = re.exec(line)) !== null) { + if (m[0].length === 0) { re.lastIndex++; continue; } + ranges.push(new vscode.Range(ln, m.index, ln, m.index + m[0].length)); + previews.push(new vscode.Range(0, m.index, 0, m.index + m[0].length)); + } + if (!ranges.length) { continue; } + progress.report({ uri, ranges, preview: { text: line, matches: previews } }); + count += ranges.length; + if (count >= max) { return { limitHit: true }; } + } + } + return { limitHit: false }; + } +} + +/* ----- activation ------------------------------------------------------------------------- */ + +const HOME = '/scratch'; + +async function seedWelcome(fsp) { + // The workspace folder is /scratch. A brand-new one is created, with one file so the Explorer is not a + // blank panel. An existing one is left exactly as it is. + try { + const dir = vscode.Uri.from({ scheme: SCHEME, path: HOME }); + let exists = true; + try { await fsp.stat(dir); } catch { exists = false; } + if (!exists) { + await fsp.createDirectory(dir); + await fsp.writeFile(vscode.Uri.from({ scheme: SCHEME, path: HOME + '/README.md' }), new TextEncoder().encode(WELCOME), { create: true, overwrite: false }); + return true; + } + } catch (e) { /* seeding is a courtesy */ } + return false; +} + +async function activate(context) { + let db; + try { db = await openDb(); } + catch (e) { + vscode.window.showWarningMessage('LevelCode could not open browser storage, so the scratch workspace is unavailable. Private windows in some browsers block it. Open a folder from your computer instead.'); + return; + } + const fsp = new ScratchFileSystem(db); + context.subscriptions.push(vscode.workspace.registerFileSystemProvider(SCHEME, fsp, { isCaseSensitive: true })); + + await seedWelcome(fsp); + + // Search is a proposed API; this extension is built in and declares it (package.json). + if (typeof vscode.workspace.registerFileSearchProvider === 'function') { + const search = new ScratchSearch(fsp); + context.subscriptions.push( + vscode.workspace.registerFileSearchProvider(SCHEME, search), + vscode.workspace.registerTextSearchProvider(SCHEME, search), + ); + } + + const item = vscode.window.createStatusBarItem(vscode.StatusBarAlignment.Left, 50); + item.text = '$(archive) Scratch'; + item.tooltip = 'Scratch workspace: files are saved in this browser only.'; + item.command = 'levelcode.web.aboutScratch'; + const inScratch = () => (vscode.workspace.workspaceFolders || []).some((f) => f.uri.scheme === SCHEME); + const refresh = () => { if (inScratch()) { item.show(); } else { item.hide(); } }; + refresh(); + context.subscriptions.push(item, vscode.workspace.onDidChangeWorkspaceFolders(refresh)); + + context.subscriptions.push( + vscode.commands.registerCommand('levelcode.web.openLocalFolder', () => openLocalFolder(vscode)), + vscode.commands.registerCommand('levelcode.web.openScratch', () => vscode.commands.executeCommand( + 'vscode.openFolder', vscode.Uri.from({ scheme: SCHEME, path: HOME }), { forceReuseWindow: true })), + vscode.commands.registerCommand('levelcode.web.aboutScratch', async () => { + const open = 'Open folder from your computer'; + const pick = await vscode.window.showInformationMessage( + 'This is your scratch workspace. Its files are saved in this browser on this computer, and nowhere else. To keep them, download them from the Explorer.', + open); + if (pick === open) { await vscode.commands.executeCommand('levelcode.web.openLocalFolder'); } + }), + ); +} + +/** + * Open a folder from the user's computer: the browser's own directory picker, then that folder as the workspace. + * + * The workbench chooses between its own file browser and the browser's picker by the SCHEME of the default + * location, and from inside the scratch workspace that scheme is this extension's: `File: Open Folder` would show + * the scratch workspace's own folders and never reach the picker. A dialog that starts from the `file` scheme — + * which the web workbench serves with the File System Access handles the user picks — does. A browser without + * File System Access (Safari, Firefox) is told so by the workbench itself, and nothing here runs after that. + * + * @param {typeof import('vscode')} api + * @returns {Promise} true when a folder was opened; false when the user cancelled + */ +async function openLocalFolder(api) { + const picked = await api.window.showOpenDialog({ + canSelectFolders: true, + canSelectFiles: false, + canSelectMany: false, + defaultUri: api.Uri.from({ scheme: 'file', path: '/' }), + openLabel: 'Open Folder', + title: 'Open a folder from your computer', + }); + if (!picked || picked.length === 0) { return false; } + await api.commands.executeCommand('vscode.openFolder', picked[0], { forceReuseWindow: true }); + return true; +} + +function deactivate() { } + +module.exports = { activate, deactivate, openLocalFolder }; diff --git a/web/workspace/package.json b/web/workspace/package.json new file mode 100644 index 0000000..601a191 --- /dev/null +++ b/web/workspace/package.json @@ -0,0 +1,24 @@ +{ + "name": "levelcode-web", + "displayName": "LevelCode Web", + "description": "The scratch workspace and the browser-only touches of LevelCode in your browser.", + "publisher": "levelcode", + "version": "0.1.0", + "license": "MIT", + "engines": { "vscode": "^1.126.0" }, + "categories": ["Other"], + "browser": "./extension.js", + "enabledApiProposals": ["fileSearchProvider", "textSearchProvider"], + "activationEvents": ["onFileSystem:levelcode-scratch", "onStartupFinished"], + "capabilities": { + "virtualWorkspaces": true, + "untrustedWorkspaces": { "supported": true } + }, + "contributes": { + "commands": [ + { "command": "levelcode.web.openLocalFolder", "title": "Open Folder from Your Computer…", "category": "LevelCode" }, + { "command": "levelcode.web.openScratch", "title": "Open the Scratch Workspace", "category": "LevelCode" }, + { "command": "levelcode.web.aboutScratch", "title": "About the Scratch Workspace", "category": "LevelCode" } + ] + } +}