Skip to content

fix(liquid): variables declared with {% capture %} are indexed, as assigned ones are - #2814

Merged
colbymchenry merged 1 commit into
mainfrom
claude/issue-2498-liquid-capture
Oct 22, 2026
Merged

colbymchenry merged 1 commit into
mainfrom
claude/issue-2498-liquid-capture

Conversation

@colbymchenry

@colbymchenry colbymchenry commented Oct 22, 2026 •

Copy link
Copy Markdown
Owner

Summary

codegraph indexed a Liquid variable declared with {% assign %} but not one declared with {% capture %}, Liquid's other way to declare a variable. A captured variable could not be found, and codegraph_explore reported its name as not in the index. This PR indexes capture declarations exactly as assign ones are indexed.

This PR is code only. The README and docs-site wording that the issue also asks about is in PR #2815, which is held for Colby's review, so this PR does not close the issue.

Refs #2498

Verdict: a bug, not a feature

The reporter calls #2498 "a feature/documentation request", so this is a judgement call, and the reasoning is here so it can be overridden:

  • The extractor already indexes Liquid variable declarations: assign, in both spellings and comment-aware. Both language tables say "Full support" for Liquid.
  • capture declares a variable in the same namespace. Missing it is an omission in an existing capability, not a new kind of behaviour: no new node kind, edge or tool.
  • On main, codegraph explore captured bare_capture tells the agent that bare_capture is "Not in the index … If it is defined in this project, the file defining it is not indexed", which points it the wrong way.
  • The issue's own "separate follow-ups" are features and stay out: Jekyll include rules, embedded JS/CSS, assets/blocks, settings and translations.

Root cause

extractAssignments ran findTagOccurrences('assign', …) and nothing read capture. Reproduced on efb091e with the issue's example, plus whitespace control, a quoted name, a {% liquid %} block and a commented-out control: both assigns became variables, and none of the four captures did.

Fix

src/extraction/liquid-extractor.ts:

  • extractAssignments also reads capture through the same findTagOccurrences scanner, so both spellings (braced, and as a line of a {% liquid %} tag) and the comment/raw rule apply unchanged.
  • The name rule is word characters, as for assign. The name may be in single or double quotes: Shopify's runtime reads {% capture 'title' %} as title, as the lax parser in Shopify/theme-tools documents. It must end at whitespace or the tag's end, -%} included.
  • A name that goes on with - or ., which the assign rule doesn't read either, gets no node rather than a cut-short one.
  • Captures are read after assigns. Node ids are keyed by file, kind, name and line, so an assign keeps its id when a capture of the same name shares its line.
  • One addVariable helper builds the node and its contains edge for both. It computes the declaration's column once, where the old assign code computed it three times, and each computation split the whole file.

Validation

Tests: 13 new tests in __tests__/extraction.test.ts.

  • 11 fail with main's extractor and pass with the fix:
    • 8 declaration forms: braced, whitespace-trimmed, tight -%}, single- and double-quoted, {% liquid %} lines with LF and CRLF, and a quoted {% liquid %} line;
    • an assign and two captures of one name on one line (distinct ids, and the assign keeps its own);
    • tags inside a capture body, which are still read;
    • an index-level test that searchNodes finds captured variables in a non-theme _includes/ folder.
  • 2 guards pass on both:
    • no capture is read from comment or raw regions, inline # comments, prose, a nameless tag, or a tag inside a string;
    • a hyphenated or dotted name is never cut short.
  • Red/green on the committed tree: 42 passed and 11 failed with efb091e's liquid-extractor.ts put back; 53 passed with the fix.

Full suite on f39fd69 (CODEGRAPH_KERNEL_EXPECT=1 npx vitest run --maxWorkers=4 --minWorkers=1): 727 files passed and 6 skipped; 8,700 tests passed, 82 skipped and 0 failed, in 660 s.

  • f39fd69 was this PR's previous head. Its tree differs from the current head only in README.md and site/src/content/docs/reference/languages.md, which are now back to main's exact text, and one sentence about those tables, now dropped from the CHANGELOG bullet.
  • On the current head: npm run build passes, the Liquid tests pass (53), and the 5 test files that mention the README pass (explore-long-query, frontload-hook, security, ui-search-model, worktree-detection: 161 passed, 2 POSIX-only skips).

A/B on efb091e, with each arm built from its own checkout. The repos and controls are the same as for #2812.

Repo (sha) Files Captures found by a separate scan New variable nodes Checked against source
Shopify/skeleton-theme (a7a655e7) 26 0 0 byte-identical
Shopify/dawn (05222537) 143 36 36 36 right
Shopify/horizon (dfad90d8) 381 192 192 192 right
ChumsInc/chums-impulse-dev (4fd6f08e) 219 90 90 90 right
agencecinq/cinquante-et-un (ecb0035a) 182 25 25 25 right
noclocks/shopify-devstore-theme (e1747695) 178 27 27 27 right
panoply/syncify (98d75fd0) 467 25 25 25 right
Shopify/theme-tools (20f8edfc) 1206 49 49 49 right
expressjs/express (0dee2c52), control 148 0 byte-identical
psf/requests (611c6162), control 49 0 byte-identical
  • The only change is one variable node per capture, each with one contains edge.
    • 444 in all: 383 braced and 61 inside {% liquid %} tags.
    • Nothing is removed or moved, no unresolved ref changes, and no node present on both arms changed its id.
  • A script checked every new node against the source text: it is a Liquid variable at the start of a capture tag naming it (a tag split over lines included), contained by its file node, with no other edge. 444 right, 0 wrong.
  • In these themes, every capture and assign name is a plain word. None are quoted, hyphenated or dotted, so the quoted and cut-short rules are covered by the tests.
  • The main arm's theme dumps are byte-identical to ones indexed from the same tree in the other checkout (the post-rebase run for fix(liquid): editor-saved Shopify JSON and {% sections %} groups link their sections #2812).

Cost:

  • The extractor is faster with the fix. A microbenchmark over 1,386 Liquid files (8 MB), 15 runs in each of 3 interleaved rounds, gave per-round medians of 443–465 ms on main and 316–387 ms with the fix.
  • Counting calls shows why: whole-file getLineStart scans fall from 29,807 to 20,671. That is 4,790 assigns going from three scans to one, plus one scan for each of the 444 captures.
  • Index wall time, median of 3 interleaved runs:
Repo main (ms) fix (ms)
horizon 2975 2935
chums-impulse-dev 2404 2311
syncify 4329 3867
theme-tools 8405 8108
express (control, byte-identical) 2427 2417
requests (control, byte-identical) 2006 1942

Other agents' jobs were running on the machine, so single runs spread widely, and the medians are within noise. The controls index identical graphs on both arms.

Kernel parity: not applicable, because the kernel has no Liquid code.

Rebase:

Contributor PR

PR #2504 by @Puma7 was read for guidance; this branch does not build on it.

Left out

  • README and docs-site wording: this is PR docs(liquid): say what Liquid support covers #2815, held for Colby's review.
  • Hyphenated and dotted names ({% capture my-var %}, {% capture a.b %}): Liquid allows them, but the assign rule doesn't read them either, and none of the 8 theme repos uses one. Widening both rules together is a separate change.
  • Variable-use edges: a captured variable, like an assigned one, is a searchable declaration. Its uses are not linked.
  • increment / decrement: these declare counters in a separate namespace. They are not part of this report.
  • The issue's follow-ups: Jekyll include and file-extension rules, Liquid in .html/.md, embedded JS/CSS, assets and theme blocks, settings and translation consumers. They need host-aware work and are features.

🤖 Generated with Claude Code

…signed ones are

The Liquid extractor indexed a variable declared with {% assign %} as a
variable node, in both spellings (braced, and as a line of a
{% liquid %} tag) and skipping comment and raw regions, but it never
read {% capture %}, Liquid's other way to declare a variable. A
captured variable could not be found: search missed it, and
codegraph_explore reported its name as not in the index.

Fix:
- extractAssignments reads capture tags through the same
  findTagOccurrences scanner as assign, so both spellings and the
  comment/raw rule apply. The name is word characters, as for assign,
  optionally in single or double quotes (Liquid reads 'title' as
  title), and must end at whitespace or the tag's end (-%} included). A
  name that goes on with "-" or ".", which the assign rule doesn't read
  either, gets no node rather than a cut-short one.
- Captures are read after assigns. Node ids are keyed by file, kind,
  name and line, so an assign keeps its id when a capture of the same
  name shares its line.
- One helper now builds the variable node and its contains edge for
  both. It computes the declaration's column once, where the assign
  code computed it three times, each a split of the whole file.

Validation:
- 13 new tests in __tests__/extraction.test.ts. 11 fail with main's
  extractor and pass with the fix: 8 declaration forms (braced,
  whitespace-trimmed, tight -%}, single- and double-quoted, {% liquid %}
  lines with LF and CRLF, a quoted {% liquid %} line), an assign and two
  captures of one name on one line (distinct ids, the assign keeps
  its), tags inside a capture body, and an index-level test that search
  finds captured variables in a non-theme folder. 2 guards pass on
  both: no capture from comment, raw, inline comments, prose, a
  nameless tag or a tag inside a string, and no cut-short hyphenated
  or dotted name. Red/green on the committed tree: 42 passed and 11
  failed with main's extractor, 53 passed with the fix.
- A/B on efb091e, each arm built from its own checkout: 8 Shopify
  theme repos (Shopify/dawn, Shopify/horizon, Shopify/skeleton-theme,
  ChumsInc/chums-impulse-dev, agencecinq/cinquante-et-un,
  noclocks/shopify-devstore-theme, panoply/syncify, Shopify/theme-tools)
  and 2 controls (express, requests). The controls and skeleton-theme,
  which has no capture, are byte-identical. Every other theme gains
  only variable nodes, each with one contains edge: dawn 36, horizon
  192, chums-impulse-dev 90, cinquante-et-un 25, shopify-devstore-theme
  27, syncify 25, theme-tools 49, 444 in all (383 braced, 61 in
  {% liquid %} tags). That is exactly the count of capture tags a
  separate scan finds outside comment and raw regions, and a script
  checked each new node against the source text: it sits on a capture
  tag naming it, with no edge but its contains. 444 right, 0 wrong.
  Nothing is removed or moved, no unresolved ref changes, and no node
  present on both arms changed its id. Every capture and assign name in
  these themes is a plain word.
- Cost: the extractor is faster with the fix. A microbenchmark over
  1,386 Liquid files (8 MB), 15 runs in each of 3 interleaved rounds,
  gave per-round medians of 443-465 ms on main and 316-387 ms with the
  fix, because whole-file line-start scans fall from 29,807 to 20,671.
  Index wall time, median of 3 interleaved runs, main vs fix: horizon
  2975 vs 2935 ms, chums-impulse-dev 2404 vs 2311, syncify 4329 vs
  3867, theme-tools 8405 vs 8108, and the byte-identical controls
  express 2427 vs 2417 and requests 2006 vs 1942, which is within the
  noise of a shared machine.
- Rebased onto 68fcf2f, which adds only PR #2813 (JS/TS static and
  instance overrides, with a kernel change, so the kernel was rebuilt).
  There the Liquid tests pass, and re-indexing the 8 theme repos
  changes no Liquid or JSON line: 4 are byte-identical to the
  pre-rebase fix arm, and the others differ only in PR #2813's
  is_static flag on JavaScript methods (dawn 30, horizon 26,
  shopify-devstore-theme 14) and one synthesized TypeScript call in
  syncify.
- Kernel parity: not applicable, the kernel does not extract Liquid.

The issue also asks for README and docs-site wording about what Liquid
support covers. That is PR #2815, a docs-only change held for the
maintainer's review, so this change leaves the issue open.

Refs #2498

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@colbymchenry
colbymchenry force-pushed the claude/issue-2498-liquid-capture branch from f39fd69 to 2cfd2b4 Compare October 22, 2026 21:48
@colbymchenry
colbymchenry merged commit 53f72fe into main Oct 22, 2026
@colbymchenry
colbymchenry deleted the claude/issue-2498-liquid-capture branch October 22, 2026 21:49
UntaDotMy added a commit to UntaDotMy/codegraphV that referenced this pull request Oct 25, 2026
…s and {% capture %} (DEV-35)

Port of upstream efb091e (colbymchenry#2812) and 53f72fe (colbymchenry#2814): theme JSON is read
as JSONC (comments, trailing commas) with each link at its type value; a
{% sections %} tag links its group's JSON; comment/raw regions are shared
with the schema scan; captured variables are indexed after assigns.
templates.mjs gains JSONC and tag cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AT1QicrEna4ne7aDRkMzV6
colbymchenry added a commit that referenced this pull request Oct 30, 2026
…e name (#3080)

Liquid allows hyphens in variable names (`{% assign form-field-label = … %}`,
`{% capture icon-cart %}`), but the extractor read assign and capture
names as \w+. PR #2814, which added capture, made such a name get no
node rather than a cut-short one, so hyphenated variables were missing
from search and codegraph_explore.

Both declarations now read a name as a word character followed by word
characters and hyphens. A capture's bare name is matched lazily up to
whitespace or the tag's end, so in `{% capture title-%}` the hyphen
still belongs to the -%} whitespace control and the name is `title`,
while `{% capture title-text-%}` reads `title-text`. A name that goes on
with a dot (`ghost.name`, `ghost-name.x`) still gets no node, in both
tags. Liquid isn't extracted by the native kernel, so there is no
kernel side.

Validation: extraction.test.ts gains 5 capture forms (bare, tight -%},
trimmed, quoted, a {% liquid %} line) and an assign test (braced,
trimmed with no spaces, a {% liquid %} line); they fail 6 of 6 with
main's extractor and pass with the fix, and the cut-short guard now
covers dotted names in both tags. A/B, each arm built from its own
checkout, on PR #2814's theme set (Shopify/dawn 05222537,
Shopify/horizon dfad90d8, Shopify/skeleton-theme a7a655e7,
ChumsInc/chums-impulse-dev 4fd6f08e, agencecinq/cinquante-et-un
ecb0035a, noclocks/shopify-devstore-theme e1747695, panoply/syncify
98d75fd0, Shopify/theme-tools bb33f49c) and controls expressjs/express
0dee2c52 and psf/requests 3c7baaa0: all ten byte-identical, as none of
them uses a hyphenated name. On three themes that do (Zoront22/LOS
58dc2dfc, lucidnz/bootstrapify-1 b36ab7ff, ibarake/bittergracedc
e36fa5db) the fix adds 4, 25 and 39 variable nodes, each with its one
contains edge: exactly the 68 hyphenated declarations a separate scan
finds, each checked to sit on a capture or assign tag naming it.
Nothing is removed or retargeted, no unresolved ref changes, and file
rows differ only in their node counts. Index time, median of six
alternating runs: horizon 3.60 vs 3.64 s, bittergracedc 28.39 vs
28.24 s.

Reimplements the part of PR #2504 by @Puma7 that PR #2814 left out.

Refs #2498

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bompus added a commit to bompus/codegraph that referenced this pull request Oct 30, 2026
#596)

* fix(node): every language and folder of an overloaded name shows in codegraph_node (colbymchenry#3030)

codegraph_node's multi-definition answer rendered a name's definitions in
index order (getNodesByName: file_path, start_line) until its caps ran
out: 16 bodies within 12,000 characters, then 20 list lines, then a bare
"+N more". Forty definitions under an early-sorting folder hid the only
one in another language or a late folder: on the triage fixture (40
TypeScript `num` in aaa_client/, one Python `num` in zzz_server/),
zzz_server/format.py appeared nowhere. Without includeCode, a long list
was cut by the output cap at "... (output truncated)".

The bodies still fill in index order, so every overload that rendered
before still renders. On top of them, each language with no body gets its
first hand-written definition rendered, while the whole answer stays
under the output cap. The list of the rest fills its 20 slots in a new
definitionCapOrder (the first definition of each language, then of each
top-level folder within a language, then the rest, each tier in index
order) and is shown in index order; the overflow line names what it
leaves out by language and folder, like "+365 more: 365 typescript in
aaa_client/". The list-only answer is unchanged unless it is longer than
the output cap; then it fills in the same order and names the rest the
same way. Generated stubs stay behind hand-written definitions.

Promoting each folder's first definition into the bodies, as first
tried, cost Alamofire's `validate` three Validation.swift overloads to a
Tests/ helper, and `adapt`, `urlSession`, `encode` and `cancel` one or
two Source/ bodies each, so folders get early list slots only.

Validation: new node-overload-spread tests (6): 5 fail on main's source
(the fits-every-cap guard passes on both) and all pass with the fix. A/B
of every multi-definition name, with and without includeCode, main vs
fix: gin 171 names, 170 identical (`_` only names its overflow);
Alamofire 706 names, 702 identical. No body that rendered before is lost:
`cancel` adds the jQuery `cancel` from docs/ as a 13th body, and
`request`, `urlSession` and `encode` swap one Source/ list line for the
first Tests/ one and name their overflow. Alamofire's `validate`, `task`
and `didCompleteTask` and gin's `reset`, `Bind` and `Render` are
byte-identical.

Reimplements PR colbymchenry#1657 by @ferrine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(go): a package-level var or const named in uppercase is exported (colbymchenry#3043)

Both extractors created a Go var/const node from its var_spec or
const_spec without exportedness: the TS Go branch passed no isExported
(the declaration-level value is read off the var_declaration, which has
no name), and the kernel's extract_variable set none. Every such node was
unexported, so findGoPackageMember, which looks up pkg.Name among the
package's exported declarations, skipped it: a call through
`var Hook = func...` from another package linked to nothing, though the
same call inside the package linked to the variable.

Both engines now mark the node exported by its own name's case, read
from the spec through goExtractor.isExported / is_exported, as
functions, methods and types already were.

Validation:
- __tests__/go-package-value-exported.test.ts, 16 tests (extraction on
  the kernel and wasm, LF and CRLF; cross-package calls on both paths):
  12 fail on the base main, 6 with only the TS half (the kernel
  checks), all pass with the fix.
- A/B, main vs fix, kernel path: gin 108 var/const flags, 0 edges;
  prometheus 625 flags, 0 edges; etcd 806 flags and 12 new calls,
  all right (e2e.GetVersionFromBinary x11 and
  model.NonDeterministicModel, both `var X = func(...)`); flask and
  express byte-identical. The fix's kernel and wasm graphs are
  byte-identical on all three Go repos, and the wasm A/B matches.
- kernel-parity --lang go: gin 99/99, prometheus 755/755, etcd
  1065/1105 with 40 deferred on main too; 0 diffs.
- etcd index time, median of 3 interleaved: 13.4 s main, 12.6 s fix.

Reimplements PR colbymchenry#1521 by ericquan8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(deps): take the security fixes for devalue and postcss, drop an unused types package (colbymchenry#3044)

devalue 5.9.1 (root, through svelte) and 5.8.1 (site/, through astro)
fall in seven advisories published 2026-10-01 and fixed in 5.9.3, among
them GHSA-j22f-vq7h-c4qm (CVE-2026-92708, high). postcss 8.5.6 (root,
through vite) and 8.5.15 (site/) fall in GHSA-r28c-9q8g-f849
(CVE-2026-73646, high, fixed in 8.5.18) and GHSA-fxqj-rqcc-2cmp (fixed
in 8.5.23); the root copy also in GHSA-6g55-p6wh-862q and
GHSA-qx2v-qp2m-jg93. All of these are build-time dependencies.

Lockfile-only updates inside the existing ranges: npm update devalue and
npm update postcss, in the root and in site/. devalue goes to 5.9.4 and
postcss to 8.5.32, whose own ranges move nanoid to 3.3.20 and
source-map-js to 1.2.2, which clears their advisories as well
(GHSA-28wg-ghj8-5hjv, GHSA-2v37-7h3g-55p8, GHSA-68fv-2mgg-jv7q). The
@types/better-sqlite3 devDependency goes: nothing has imported
better-sqlite3 since the switch to node:sqlite. In site/, npm 10 also
drops five "peer": true markers that the npm which wrote that lockfile
had added; they are metadata only, and npm ci installs the same tree.

Validation:
- npm audit: root 14 advisories before, 10 after; site/ 13 before, 9
  after. devalue, postcss, nanoid and source-map-js are gone from both.
- npm run build on main's lockfile and on this one: all 1,264 files of
  dist/, the viewer included, are byte-identical. The site builds its
  21 pages, and its 101-file dist/ is byte-identical too.
- The release bundle's install (npm ci --omit=dev --ignore-scripts on
  the staged lockfile) installs the same 12 packages, file for file.
- All 8 changed resolved/integrity entries match the npm registry's own
  metadata and sit on registry.npmjs.org. No git, URL or tarball
  sources, and no new install scripts. devalue, postcss and nanoid carry
  provenance from their own repositories' release workflows.

Raised by PR 2885 (@johnhldiver-beep) and PR 1477 (@anupamme).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>


[skip actions]

* fix(deps): read .gitignore rules the way git does, with ignore 7.0.14 (colbymchenry#3050)

codegraph applies .gitignore rules itself, through the ignore package:
on the filesystem walk for every .gitignore, and in git repositories on
top of git ls-files for its built-in defaults merged with the root
.gitignore and git's exclude files. codegraph.json's include, exclude
and includeIgnored patterns go through it too. ignore 7.0.5 read several
forms differently from git: a backslash escape (\d was a regex digit),
POSIX classes and a ] or ! first in a bracket, and a run of three
asterisks. A lone ! line matched every path, so it re-included
everything the rules and the built-in defaults leave out, node_modules
included. 2,031 of 200,000 random patterns also threw at match time,
which readGitignorePatterns has to guard against (issue 682).

ignore 7.0.14, with the range raised to ^7.0.14, reads all of these as
git does, and none of the 200,000 patterns throws. Its code diff from
7.0.5 was read in full: pattern translation only, with no I/O, no
dependencies and no install scripts. The lockfile entry matches the npm
registry's own metadata on registry.npmjs.org. The package has no
provenance, like every release its maintainer has published.

Validation:
- New __tests__/gitignore-pattern-parity.test.ts checks scanDirectory
  against git's own answer for these patterns, on the filesystem walk
  and in a git repository: its 4 tests fail with 7.0.5 and pass with
  7.0.14.
- scanDirectory on react, astro and django, in git mode and in walk
  mode on a copy without .git, returns the same 4,631, 4,946 and 3,027
  files with either version.
- On a fixture of the changed forms, 7.0.14 leaves out exactly what git
  leaves out in both modes, node_modules aside, which codegraph's
  defaults always exclude. With a lone ! line, 7.0.5 kept 16 of the 18
  files on the walk, node_modules among them, where git keeps 6. Without
  that line it differed from git on 8 files on the walk and on 2 in git
  mode.
- The gitignore and scope test files (extraction,
  git-ignored-dirs-scope, the include, exclude and includeIgnored
  configs, src-build-folders and others): 853 passed.

Raised by PR 2885 (@johnhldiver-beep).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): only one process takes over a stale write lock (colbymchenry#3056)

Issue colbymchenry#2979: FileLock.acquire() read the lock file, saw a dead PID,
unlinked the file and created its own with an exclusive create. Two
processes that read the same dead PID could both succeed: the first one
replaced the stale lock, then the second one, acting on what it had read
a moment earlier, unlinked the first one's live lock and created its own.
Both returned from acquire() and wrote to the index as its only writer.
The same steal happened in two narrower ways: a lock file read in the
instant between its exclusive create and the write of its PID was empty,
so it counted as stale, and a PID the process may not signal (EPERM:
another user's process, or an elevated one on Windows) counted as dead.

The fix keeps the lock file and its format (a bare PID) and changes how
it is taken:
- A free lock costs one create, as before. The lock file is created with
  its PID already in it: a private file hard-linked into place, which
  fails if the lock exists. Where hard links are missing it falls back
  to the exclusive create, and an empty lock younger than 10 s counts as
  held, since its creator is still writing its PID.
- A stale lock (dead PID, malformed, or empty and old) is replaced only
  by the process holding <lock>.break, itself a FileLock, after reading
  the lock again there, the way Mercurial breaks its locks. A break lock
  a crashed process left is broken the same way, one level up, at most
  three levels.
- EPERM from a signal-0 probe means the holder is alive.

Validation: __tests__/filelock-stale-takeover.test.ts runs two real
processes with a barrier: the slow one stops right after reading the
dead PID and resumes only once the other holds the lock. On main both
held it, 5 runs of 5; with the fix the slow one is refused and the lock
stays the holder's. The other cases cover the slow one getting in once
the lock is free, a live and a dead break lock, the empty-lock window,
EPERM (mocked, and for real on Windows with the System process), and no
leftover files.

Fixes colbymchenry#2979

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(db): processes opening an index that needs a migration all open it (colbymchenry#3058)

Issue colbymchenry#2980: opening an index whose schema is behind read its version,
then ran each pending migration in a deferred transaction. When several
processes opened it at the same moment, those that read the version
just before another one migrated applied the same migration again, and
their open failed: on this Windows machine with "UNIQUE constraint
failed: schema_versions.version", since the version was already
recorded, and in the report with SQLITE_BUSY, which a deferred
transaction that has already read gets at once when it comes to write.

Each migration now runs in a transaction that takes the write lock up
front (BEGIN IMMEDIATE, a new option of the adapter's transaction()),
so a process that finds another one migrating waits it out on
busy_timeout, and reads the version again inside it, skipping a
migration already applied. The version the caller read stays a floor,
as before: a database built from schema.sql records only v1 and the
current version, and tests replay migrations above a version they
rewound to. An open of a current index takes no lock and waits for
nothing, as before.

Validation: __tests__/migration-concurrent-opens.test.ts. A second
process holds the write lock with version 16 recorded but uncommitted,
so the open reads 15: main fails with the UNIQUE error after waiting
1.1 s; the fix waits, opens and leaves the migration to the other
process. Three processes released at once from a stdin barrier on an
index with 150k unresolved references: main fails in round 0, the fix
opens all three in both rounds. On excalidraw's real index, 8 rounds of
three openers: main 7 rounds with failures, the fix 0. The single
process and the current-index cases pass on both.

Fixes colbymchenry#2980

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(mcp): a session whose host stops reading its output shuts down cleanly (colbymchenry#3060)

Issue colbymchenry#2991: a host can stop reading an MCP stdio server's output while
it keeps the server's stdin open and stays alive. The next response the
server wrote then failed (EPIPE), and with no listener on stdout Node
handed the error to the last-resort fatal handler, which exits 1 at
once. That skipped the shutdown that stops the engine, releases the
writer lock and waits for query workers still starting, which on
Windows can crash a process that exits under them. Stdin, the host's
PID and the liveness watchdog all still look healthy, so nothing else
noticed.

A new treatStdoutFailureAsShutdown, the twin of the stdin teardown from
issue colbymchenry#799, routes a stdout 'error' (and the 'close' after it) into the
owner's own shutdown, at most once, destroys stdout so later writes are
dropped, and keeps its listener so a write queued before the failure
cannot escape during the shutdown. Backpressure is left alone. It is
wired into the direct-mode transport and into the local-handshake proxy,
whose shutdown closes only this session's daemon connection and stops
the engines it owns. The legacy socket proxy (runProxy) has had no
caller since PR #580 and is left as it is.

Validation: __tests__/mcp-stdout-failure.test.ts runs a real
direct-mode serve --mcp, and the local proxy with an in-process engine
and with a stand-in shared daemon. After the test destroys its end of
stdout and sends one more request, main exits 1 in all three, the
server with "Uncaught exception: Error: EPIPE"; with the fix all three
exit 0, the server's writer.pid is released, the proxy's engine stop
runs, and the daemon closes only the proxy's connection and still
answers its other client. __tests__/stdout-teardown.test.ts covers the
guard itself. proxy-shutdown, stdin-teardown and mcp-initialize pass.

Fixes colbymchenry#2991

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(astro,svelte): a self-closing <script /> no longer swallows the next script

A self-closing `<script ... />`, such as Astro's
`<script is:inline set:html={code} />` or a third-party
`<script src="/analytics.js" />` in Svelte's `<svelte:head>`, matched the
extractors' script regex as an opening tag, so the match ran on to the next
`</script>`. The real script that followed was parsed with the markup in
between, failed with "parse produced no symbols", and contributed nothing.
The covered-range regexes had the same shape, so the template scans also
skipped the markup between the two tags, and a self-closing `<style />` hid
the markup after it.

The opening-tag pattern now matches a `/>` ending on its own as an empty
tag: in the Astro extractor's script blocks and covered ranges, the Svelte
extractor's script blocks and both of its covered-range scans, and the
viewer's highlight regions.

With a fix of ours: a self-closing tag has no body, so the covered-range
scans skip it, and a call in its attributes, like
`set:html={serializeJsonLd(schema)}`, is read as on main. Matched as a
range, such a tag hid those calls even where no script followed it.

Validation on main bc9291a (Windows 11):
- Red/green, 6 tests: 4 from the PRs, plus 2 of ours for an attribute call
  on a self-closing script in Astro and in Svelte. With main's
  astro-extractor.ts, svelte-extractor.ts and syntax-tokens.ts all 6 fail;
  with the PRs' code as written the 2 of ours fail; with this commit all 6
  pass.
- Svelte 5.56.10's own compiler reads the Svelte fixture as an empty head
  script plus an instance script, as the extractor now does.
- A/B against main on 7 Astro repos, with flask and express as controls.
  starlight, diploi.com and both controls are byte-identical, and
  devfest-website keeps every edge. Gains: the genkit docsite 2 import
  links, nodejsdesignpatterns.com 2 nodes and 2 import links, a portfolio
  25 component references, and wm-project-astro-components 3
  `serializeJsonLd` calls from JSON-LD scripts plus 1 call in a script main
  swallowed. That repo also drops 31 false calls, CSS like
  `border-radius: var(...)` that main read as calls after parsing a
  swallowed `<style>` block as TypeScript. No other edge changes.
- Index time unchanged: median of 3 interleaved runs on
  wm-project-astro-components, 4.67 s on main and 4.52 s with the fix.

Vue's extractor has the same pattern and the same bug; a follow-up shares
one pattern across the three extractors.

Original PRs colbymchenry#2995 and colbymchenry#2997 by @maxmilian.
Refs colbymchenry#2989

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(search): a pasted call like leaf(entry) finds both names instead of nothing (colbymchenry#3069)

Full-text search dropped the FTS5 syntax characters ( ) ^ " ' * : from a
query before quoting each word, so the words on either side of one were
glued together: leaf(entry) became the single prefix term "leafentry"*,
which no symbol has, and the LIKE and fuzzy fallbacks don't recover it.
codegraph query, the codegraph_search tool and searchNodes() all came
back empty for a pasted call like render(props).

The index's unicode61 tokenizer splits stored text at anything but
letters and digits (a signature's leaf(entry) is stored as leaf and
entry), so the query now splits there too: each of those characters
becomes a space, the treatment :: already gets (issue #173).

Validation: new search-special-characters.test.ts (8 tests) fails 8 of 8
on main and passes with the fix; fts5-fallback, search-field-filters,
search-query-parser, cli-query-command and symbol-lookup pass. A query
A/B on expressjs/express 0dee2c52, both arms over one index: 12 control
queries without such a character between words are byte-identical; 10
pasted calls changed, 7 of them from no results to results led by the
function they name (res.send(body), res.json(obj),
View.prototype.lookup(name)).

Reimplements PR colbymchenry#2987 by @kjgggggg.

Fixes colbymchenry#2971

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(mcp): a tools/call whose arguments aren't an object gets invalid params, not a crash (colbymchenry#3071)

A tools/call request passes its arguments as an object. MCPSession took
params.arguments || {} without checking that, so a string or number went
straight to the tool handler, where the session-view check
('_cgExploreSession' in args) threw. The call then came back as a tool
result flagged isError: "This is an internal codegraph error, retry the
call once; if it persists, continue without codegraph". That reports a
client's malformed request as codegraph's own malfunction, with advice to
stop using it. An array went through as if it were the arguments. The
stdio proxy, serving in process while the daemon is unavailable, had the
same unchecked read.

Both now read the arguments through one helper, toolCallArguments(), and
answer anything but an object with a JSON-RPC invalid-params error
(-32602, "Tool arguments must be an object"), as a call with no tool
name already is; no tool runs, and the proxy starts no engine for it.
Missing, null or another empty value still means no arguments.

Validation: new mcp-tool-call-arguments.test.ts covers the session (the
direct stdio server and every daemon client) and the real proxy entry
point in a child process with the daemon unavailable: red on main (2 of
3 failed; the well-formed control passed), green with the fix.
explore-session-state and proxy-shutdown pass. Driving serve --mcp over
stdio: on main, codegraph_explore, codegraph_node and codegraph_search
with a string argument returned isError "internal codegraph error" and
codegraph_status accepted [1], 42 and true; with the fix those six get
-32602, while "", null, {} and absent arguments run as before.

Reimplements PR colbymchenry#2986 by @kjgggggg.

Fixes colbymchenry#2981

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(liquid): a variable named with a hyphen is indexed under its whole name (colbymchenry#3080)

Liquid allows hyphens in variable names (`{% assign form-field-label = … %}`,
`{% capture icon-cart %}`), but the extractor read assign and capture
names as \w+. PR colbymchenry#2814, which added capture, made such a name get no
node rather than a cut-short one, so hyphenated variables were missing
from search and codegraph_explore.

Both declarations now read a name as a word character followed by word
characters and hyphens. A capture's bare name is matched lazily up to
whitespace or the tag's end, so in `{% capture title-%}` the hyphen
still belongs to the -%} whitespace control and the name is `title`,
while `{% capture title-text-%}` reads `title-text`. A name that goes on
with a dot (`ghost.name`, `ghost-name.x`) still gets no node, in both
tags. Liquid isn't extracted by the native kernel, so there is no
kernel side.

Validation: extraction.test.ts gains 5 capture forms (bare, tight -%},
trimmed, quoted, a {% liquid %} line) and an assign test (braced,
trimmed with no spaces, a {% liquid %} line); they fail 6 of 6 with
main's extractor and pass with the fix, and the cut-short guard now
covers dotted names in both tags. A/B, each arm built from its own
checkout, on PR colbymchenry#2814's theme set (Shopify/dawn 05222537,
Shopify/horizon dfad90d8, Shopify/skeleton-theme a7a655e7,
ChumsInc/chums-impulse-dev 4fd6f08e, agencecinq/cinquante-et-un
ecb0035a, noclocks/shopify-devstore-theme e1747695, panoply/syncify
98d75fd0, Shopify/theme-tools bb33f49c) and controls expressjs/express
0dee2c52 and psf/requests 3c7baaa0: all ten byte-identical, as none of
them uses a hyphenated name. On three themes that do (Zoront22/LOS
58dc2dfc, lucidnz/bootstrapify-1 b36ab7ff, ibarake/bittergracedc
e36fa5db) the fix adds 4, 25 and 39 variable nodes, each with its one
contains edge: exactly the 68 hyphenated declarations a separate scan
finds, each checked to sit on a capture or assign tag naming it.
Nothing is removed or retargeted, no unresolved ref changes, and file
rows differ only in their node counts. Index time, median of six
alternating runs: horizon 3.60 vs 3.64 s, bittergracedc 28.39 vs
28.24 s.

Reimplements the part of PR colbymchenry#2504 by @Puma7 that PR colbymchenry#2814 left out.

Refs colbymchenry#2498

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: add a contributing guide and a documentation index (colbymchenry#3081)

A new contributor had nothing to start from: the README covers using
CodeGraph, and the build and test rules live in AGENTS.md, written for
agents. CONTRIBUTING.md now covers building from source, testing a change,
preparing a pull request and what happens to it. docs/README.md indexes the
guides and design notes, and the README links to both.

Every relative link and heading anchor was checked against main.

Reimplements PR 1931 by @mixxer. Also folds in the fork-and-upstream setup,
`npm run cli`, `npm run test:watch` and the version rule from PR 306 by
@marcelovani, and the commit format, the single-test command, the bug-report
list and the `/add-lang` pointer from PR 665 by @ymstar.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: adapt upstream's migration-concurrency and stdout-failure tests to the fork's migration numbers and kernel-only parser

* fix: create the lock's private file exclusively; document the kernel as required

Review of the upstream lock change: a link planted at the private file path was
written through. Clear the path, then create with 'wx'. CONTRIBUTING no longer
calls the kernel optional.

* docs: state the Node and Bun minimums the CLI enforces

---------

Co-authored-by: Colby Mchenry <me@colbymchenry.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Max Hsu <maxmilian@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant