Skip to content

Fix #79: Fall back from SQLite WAL mode when the filesystem can't support it - #324

Open
zwanski-start wants to merge 1 commit into
AutoForgeAI:masterfrom
zwanski-start:swarm/issue-79-guido
Open

zwanski-start wants to merge 1 commit into
AutoForgeAI:masterfrom
zwanski-start:swarm/issue-79-guido

Conversation

@zwanski-start

Copy link
Copy Markdown

Closes #79

Root cause

The traceback has nothing to do with GLM. create_database() runs PRAGMA journal_mode=WAL on features.db, and SQLite accepts it. On filesystems that can't memory-map the WAL -shm index, the next statement (here PRAGMA table_info("features") inside create_all) then fails with sqlite3.OperationalError: disk I/O error. Examples are WSL /mnt/c (9p/drvfs), Docker or VM shared folders (virtiofs, vboxsf, grpcfuse), FUSE mounts and eCryptfs-encrypted homes. The orchestrator and the feature MCP server both die at startup, so features never start.

The old check (_is_network_path) only knew nfs, cifs, smbfs and sshfs. It also matched mount points by plain string prefix, so /mnt/nfs also matched /mnt/nfs-local.

Changes (api/database.py)

  • WAL probe: before using WAL, _probe_wal_support() tries it on a throwaway .wal_probe_<pid>.db in the DB directory: write on one connection, read on a second one so the shm index is used, then clean up. If that fails, the code uses DELETE mode. The result is cached per directory.
  • Better FS detection: finds the most specific mount containing the path, decodes /proc/mounts escapes, and adds 9p, drvfs, virtiofs, vboxsf, vmhgfs, prl_fs, ecryptfs, smb3, ceph, glusterfs, lustre, afs and all fuse.* types. fuseblk (local ntfs-3g) is left out on purpose.
  • Recovery for databases already stuck in WAL: a database left in WAL mode (like the reporter's, after the failed run) can't be opened normally. When a rollback mode is chosen and the file header says WAL, it is opened with locking_mode=EXCLUSIVE, which keeps the WAL index in heap memory, then checkpointed and switched to the chosen mode. Data is kept.
  • Escape hatch: the AUTOFORGE_SQLITE_JOURNAL_MODE env var (WAL/DELETE/TRUNCATE/PERSIST) forces a mode. Invalid values are ignored with a warning, since the value goes into a PRAGMA. It is documented in .env.example.

Verification

  • New test_database_journal_mode.py (16 tests) covers mount matching (longest mount wins, whole path component, escaped spaces), unsafe-FS detection, the probe passing on local disk and failing on a simulated disk I/O error with cleanup and caching, WAL on local disk, fallback to DELETE, the env override and invalid values, and converting an existing WAL database to DELETE with its rows kept.
  • ruff check . and mypy on the changed files: clean. python test_security.py: all pass. pytest test_client.py test_dependency_resolver.py test_rate_limit_utils.py: pass.
  • The real failure couldn't be reproduced here (no 9p/virtiofs mount), so the probe failure is simulated by making sqlite3.connect raise disk I/O error.

Assumptions

  • The reporter's environment is unknown (their path /home/william/... could be WSL, a container bind mount, or an encrypted home). That's why the main fix is a runtime probe rather than relying only on an FS-type list.
  • Every process (server, orchestrator, MCP servers) runs the probe itself, so they all pick the same mode on the same filesystem. The probe costs a few ms once per process per project.
  • No UI changes, so no browser check was needed.

🤖 Generated with Claude Code

…ort it

SQLite accepted PRAGMA journal_mode=WAL on filesystems where the -shm
shared-memory index can't be mmapped (WSL /mnt/c, Docker/VM shared
folders, FUSE, eCryptfs), then failed the next query with
'disk I/O error', killing the orchestrator and MCP server.

- Probe WAL on a throwaway database in the DB directory before using it
- Detect more unsafe filesystems (9p, drvfs, virtiofs, fuse.*, ...) and
  match the most specific mount point instead of any string prefix
- Convert a database left in WAL mode using EXCLUSIVE locking, which
  keeps the WAL index in heap memory, so it can be opened again
- Add AUTOFORGE_SQLITE_JOURNAL_MODE override

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zwanski-start

Copy link
Copy Markdown
Author

🔍 QA report: ✅ Passed

Tester: Dennis (cubefarm agent, in a fresh session) · Round: 1 · Author: Guido

Reproduced issue #79 for real: on a FUSE direct_io mount (no shared mmap, like WSL /mnt/c or virtiofs), master's create_database() and the feature MCP server die with the exact sqlite3.OperationalError: disk I/O error. With this PR the WAL probe detects the failure and falls back to DELETE mode: the server REST API, the feature MCP server and the UI (dashboard, Kanban, graph) all work on that mount, and a database left in WAL mode with uncheckpointed writes is converted with all rows intact. FS-type detection (fuse.*), the env override and its PRAGMA-injection guard behave as described; the new tests, security tests, ruff and the UI build pass, and the full-suite pytest and mypy failures are the same on master.

Check Details
✅ Issue repro on master (real FS, not mocked) Mounted a fusepy passthrough FS with direct_io=True (type 'fuse'). Plain sqlite3: PRAGMA journal_mode=WAL returns 'wal', then the next statement fails with 'disk I/O error', the issue's symptom. Master create_database() on a project there: OperationalError disk I/O error. Master feature MCP server on the mount: crashes (McpError: Connection closed).
✅ Fix: new project on broken-WAL FS PR create_database() logs '[database] WAL mode is not supported ... using DELETE journal mode instead'. Feature insert, count and reopen all work; journal_mode=delete; no .wal_probe_* files left behind.
✅ Recovery of a database stuck in WAL mode Built a WAL features.db on local disk and exited with os._exit, leaving a 28KB -wal holding uncheckpointed rows. Master opening it through the mount: disk I/O error. PR: converts it to DELETE; all previous rows, including the uncheckpointed ones, are present and a new row can be added (3 features). A stale harmless -shm file remains.
✅ FS-type detection (fuse.* subtype) Mounted with subtype=sshfs (fuse.sshfs): DELETE mode chosen via _is_network_path, without needing the probe.
✅ Local disk keeps WAL A project on local btrfs got journal_mode=wal.
✅ AUTOFORGE_SQLITE_JOURNAL_MODE override and validation TRUNCATE is applied (and converts an existing WAL DB). Invalid value 'wal; drop table features' is ignored with a warning and WAL is used, so nothing unchecked reaches the PRAGMA. Forcing WAL on the broken mount fails, as expected for a user override.
✅ FastAPI server on the broken-WAL FS uvicorn on 127.0.0.1:5247 with isolated HOME. POST /api/projects 200, POST features/bulk 200, GET features 200, GET features/graph 200, GET features/9999 404. Fallback warning in the server log, no tracebacks.
✅ Feature MCP server (component that died in the issue) PR: stdio MCP client against mcp_server.feature_mcp with PROJECT_DIR on the mount; feature_get_stats returns total=2 while the API server also has the DB open. Master: crashes at startup.
✅ UI in browser (Playwright) Project picker shows walfuse 0/2; dashboard (WebSocket shows Live), Kanban and dependency graph render with 0 console errors and 0 warnings. At 390x844 the layout is cramped because the sidebar stays expanded; same without this PR, which changes no UI.
✅ New unit tests test_database_journal_mode.py: 16 passed.
✅ Full pytest suite 21 failed / 160 passed on the PR vs 21 failed / 144 passed on master: the same 21 failures (all in test_devserver_security.py) exist on master; the 16 new tests are the difference.
✅ Security not weakened test_security.py: ALL TESTS PASSED. security.py untouched. The env value is allowlisted before reaching the PRAGMA.
✅ Lint / type check ruff check .: all checks passed. mypy .: 24 errors, the same count as on master, none in api/database.py or test_database_journal_mode.py.
✅ Cleanup / orphan processes Server and FUSE daemons stopped, mounts unmounted, temporary master worktree removed; git status clean.
🧪 Commands run
Command Result
venv/bin/ruff check . All checks passed
venv/bin/mypy . 24 errors in 9 files, same count as on master; none in changed files
venv/bin/python -m pytest test_database_journal_mode.py -q 16 passed
venv/bin/python test_security.py ALL TESTS PASSED
venv/bin/python -m pytest -q (PR vs master worktree) PR 21 failed/160 passed; master 21 failed/144 passed (same failures, all test_devserver_security.py)
fusepy direct_io passthrough mount + create_database() on master vs PR (scratch script) master: disk I/O error; PR: DELETE fallback, data OK
stuck-WAL recovery scenario (os._exit with uncheckpointed WAL, then open through the mount) master: disk I/O error; PR: converted to DELETE, all rows kept
AUTOFORGE_SQLITE_JOURNAL_MODE=TRUNCATE / 'wal; drop table features' / WAL truncate applied / ignored with warning / forced WAL fails as expected
HOME=<scratch> venv/bin/uvicorn server.main:app --host 127.0.0.1 --port 5247 + curl REST calls 200/200/200/200/404 as expected
MCP stdio client -> python -m mcp_server.feature_mcp (PROJECT_DIR on the mount) PR: feature_get_stats total=2; master: server crashes
npm --prefix ui run build built successfully
npm --prefix ui run lint 0 errors, 6 warnings

No browser screenshots were taken in this round.

Posted by cubefarm · merges automatically once GitHub's checks pass

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MCP Tools not working with GLM 4.7

1 participant