Skip to content

feat(index): add worker resource watchdogs - #1724

Merged
DeusData merged 10 commits into
DeusData:mainfrom
liuchong:feat/index-resource-worker
Nov 3, 2026
Merged

DeusData merged 10 commits into
DeusData:mainfrom
liuchong:feat/index-resource-worker

Conversation

@liuchong

@liuchong liuchong commented Aug 19, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #1723. A fork PR must target a branch in this repository, so the diff below also contains #1723's commit. The new work here is the last commit, ece90423. Once #1723 merges, this diff shrinks to that one commit on its own.

Related to #1347.

Problem

Discovery limits bound what indexing accepts, not what it then costs. A repository well inside those bounds can still exhaust the host through parser memory, or simply never finish. A supervised worker that hangs leaves the parent waiting with nothing to report and no way to attribute the stall.

What this changes

Two opt-in limits, enforced by the parent against the worker process tree:

Key Meaning
index_max_rss_mb Maximum resident memory of the worker and its descendants
index_max_duration_seconds Maximum wall-clock duration of the attempt

Both default to off.

Measuring the tree rather than the worker process alone matters: indexing spawns children, and a runaway child hiding behind a small parent is exactly the case that goes unattributed today. Resident memory is sampled through the platform interface on macOS, Linux and Windows.

Crossing a limit terminates the process tree and produces one trusted, structured terminal result attributing the failure to the resource that caused it, so the caller learns which limit ended the attempt rather than seeing a generic worker failure.

A measurement that cannot be taken fails the attempt instead of passing it. A watchdog that quietly stops watching is worse than no watchdog, so the probe is fail-closed and says so in the result.

Testing

make -f Makefile.cbm test and make -f Makefile.cbm lint-ci on macOS. New coverage: process-tree RSS measurement, watchdog termination on each limit, probe-failure fail-closed behaviour, terminal-result attribution, and policy validation for the two new keys.

Stack

  1. feat(index): add opt-in discovery resource limits #1723 — discovery limits
  2. this PR — worker watchdogs
  3. feat(index): enforce opt-in storage resource limits #1725 — storage limits
  4. feat(index): add opt-in index resource profiles #1726 — resource profiles
  5. feat(index): record and report the latest index attempt #1727 — attempt record and freshness
  6. feat(mcp): warn on answers served from a failed rebuild #1728 — stale-index warning on answers

@github-actions

Copy link
Copy Markdown

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

  • It will not be closed for inactivity. No stale bot touches pull requests here.
  • It may still sit a while before a human reads it. That is on us, not on you.
  • Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

  • Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
  • Get CI green, or say which failures you believe are pre-existing.
  • Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
  • Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

@DeusData

DeusData commented Sep 1, 2026

Copy link
Copy Markdown
Owner

Reviewed slice 2. The direction is already accepted (see #1723); this is the merit read, and the design holds up well.

The three-valued status is the right call and it is the thing most people get wrong here.

CBM_PROC_TREE_RSS_ERROR = -1,
CBM_PROC_TREE_RSS_EMPTY = 0,
CBM_PROC_TREE_RSS_OK    = 1,

"No trustworthy measurement" and "the tree has no observable members" are different facts, and collapsing them into 0 is exactly how a watchdog silently stops watching. Keeping them apart is what makes the fail-closed behaviour possible rather than aspirational — and I checked it is real: resource_probe_failed gates the success predicate and is logged through index.supervisor.resource_probe_failed, so a probe that cannot measure fails the attempt instead of waving it through.

"A watchdog that quietly stops watching is worse than no watchdog" is the right principle and you built to it.

Using a Windows Job Object is the detail I want to single out. QueryInformationJobObject with JobObjectBasicProcessIdList gives a kernel-maintained, authoritative membership list for the contained tree. The obvious alternative — enumerating descendants by walking parent PIDs — is racy by construction, and we have a live defect in this repo where exactly that style of Windows descendant probe fails closed under a tight timeout. You avoided that entire class by picking the right primitive rather than by handling its symptoms.

The error paths around it are thorough too: allocation failure, size overflow past UINT32_MAX, an unexpected GetLastError(), and a capacity guard so the ERROR_MORE_DATA retry cannot grow without bound. Each returns ERROR rather than a plausible number.

Measuring the tree rather than the root process is the correct scope, and your reason for it — "a runaway child hiding behind a small parent is exactly the case that goes unattributed today" — is the sentence that justifies the extra complexity. Ignoring processes that exit mid-enumeration is the right way to handle the inherent race.

Exposing cbm_subprocess_rss_sum_for_testing so the overflow-safe summation can be tested directly, rather than only through a live process tree, is a good seam.

Process notes

Clearance: this shows REVIEW(3) — Makefile.cbm and scripts/test-runtime.sh (both inherited from #1723's stacked commit) plus a test that spawns processes, which is unavoidable for a watchdog test. Nothing here is an exemption; it is the same shape already cleared elsewhere. I will get the marker written before merge.

Rebase when #1723 lands. This diff currently carries #1723's commit; once that merges it collapses to ece90423 alone, which will make the final read much easier. main also moved three times today, so rebase onto current main rather than anything older.

Slice 3 onward is where the configuration surface really grows, so expect those to get a closer look than this one — but this slice is in good shape.

@DeusData DeusData added enhancement New feature or request stability/performance Server crashes, OOM, hangs, high CPU/memory priority/high Needs near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker. labels Sep 3, 2026
@liuchong
liuchong force-pushed the feat/index-resource-worker branch 2 times, most recently from 70b411d to 87da442 Compare September 9, 2026 08:07
@liuchong
liuchong force-pushed the feat/index-resource-worker branch from 87da442 to 42a3d71 Compare September 21, 2026 04:32
@liuchong

Copy link
Copy Markdown
Contributor Author

@DeusData rebased #1724 onto current main (92abefa3, which includes the #1723 merge) and stopped there. #1725–#1728 are untouched.

One conflict worth naming: main now has cbm_index_worker_job_memory_limit (the Windows job-object commit cap). This slice's RSS/duration watchdog is a separate check, so both stayed — the OS cap is still applied at spawn, and the parent still samples the process tree against index_max_rss_mb / index_max_duration_seconds. Local index_supervisor suite: 15 passed, including the job-memory floor test from main and the watchdog cases from this slice.

CI is restarting. I will wait for you to say when the next slice should move.

@liuchong
liuchong force-pushed the feat/index-resource-worker branch from 42a3d71 to 81ff2ca Compare September 21, 2026 07:51
@liuchong

Copy link
Copy Markdown
Contributor Author

@DeusData rebased #1724 onto current main again (def38f3e, including #2257) and stopped there. #1725–#1728 are still untouched.

The previous matrix's reds were:

  • lint / lint — ours. The memory-core ratchet caught new raw malloc/realloc/free in the process-tree RSS probe and four extra free sites in the worker resource response. The probe now allocates through mem_core; the worker JSON responses share one release helper so the file does not grow past baseline.
  • ci-ok / test — follow-on. test is needs: lint, so it skipped; ci-ok then failed because lint (and therefore test) did not succeed.
  • memwaste event-windows / event-macos / access-linux — expected skips. Those jobs only run in mode: full; PR validation uses mode: pr (Linux event + scaling only). They are report-only and not in ci-ok.

Local index_supervisor + subprocess: 49 passed. CI is restarting.

@liuchong
liuchong force-pushed the feat/index-resource-worker branch 3 times, most recently from d48afe8 to 3743fe9 Compare September 21, 2026 18:23

@DeusData DeusData left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you — for the two rebases, for stopping exactly where you said you would, and for two fixes I was about to ask for and found already done.

I read head 42a3d710 in full this morning and wrote three blocking findings. By the time I could post, you had fixed two of them yourself:

  • The memory-core ratchet — the process-tree probe now allocates through mem_core and the worker resource responses share one release helper. lint / lint is green on 3743fe95.
  • The Step 5f collision — this one was ours, not yours, and you fixed it anyway. tests/test_worker_session_scope.sh on main hard-coded a two-key policy literal while the worker demands the exact key set, so any slice that adds a policy key turns main red after a textually clean merge. That is precisely what #1723 did to us at the weekend, and this slice (2 → 4 keys) would have done it again. Your last commit routes the fixture through cbm_test_index_worker_policy_json, which is the durable fix: every later slice only has to touch the helper. For the record, I think the worker's exact-key-count check should stay as strict as it is — a supervisor that silently drops a key would silently disable a limit.

That leaves one blocker, and it is a one-word change with a long story behind it.

Blocking: macOS samples ri_resident_size; it must be ri_phys_footprint

src/foundation/subprocess.c:577 sums usage.ri_resident_size over the process group. On macOS, resident size keeps counting every page mimalloc has already handed back (MADV_FREE_REUSABLE) until the kernel gets round to reclaiming it. We measured this the hard way — it is written down in src/foundation/mem.h on main: on the Linux-kernel corpus the worker sat at 17.4 GB resident with 5.5 GB actually charged. With this slice as it stands, a user who sets index_max_rss_mb to 8 GB on a Mac can get a healthy run killed while the OS is charging it a fraction of that, with an error message naming a number the OS itself does not hold against them.

rusage_info_v2 already carries ri_phys_footprint, so the probe needs no new call — just the other field. That is also the number cbm_mem_charged() enforces the cooperative budget against, so the two limits would finally agree on what "memory" means. Linux and Windows are fine as they are. If you can, make the fake-RSS seam's test name say which quantity it represents, so nobody "fixes" it back.

Not blocking — but please read

  1. The daemon's default start path has no test. application_worker_start_default (src/daemon/application.c, the policy parse plus cbm_index_worker_start_with_policy) is only reachable through the real .start; every case in tests/test_daemon_application.c installs app_fake_worker_start. So "a malformed internal policy makes the daemon refuse the job" is currently asserted by nothing. One case that feeds a bad _cbm_index_policy through the default start and expects -1 plus daemon.index.policy would close it.
  2. index_max_duration_seconds is per attempt, not per request. started_ms is set when a worker handle starts, and recovery starts a new handle per attempt — so a request that crashes and recovers can run for several times the configured duration. Either is defensible; what is not is the ambiguity. Please pick one, name it in the option's help text, and pin it with a fake-clock test across a recovery. My preference is per request: a user who writes "30 minutes" means the thing they are waiting for.
  3. A duration limit under 15 minutes pre-empts hang quarantine. DEFAULT_QUIET_TIMEOUT_MS is 900 000; a shorter duration limit kills a hung worker first, and because a limit kill (correctly) never enters the quarantine loop, as far as I can see the file that hangs is never identified, so the next run would hang on it again. Worth one sentence in the docs at minimum; better, log the in-flight file marker on a duration kill.
  4. Windows probe, small. OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_VM_READ, …) matches the documented requirement for GetProcessMemoryInfo, so this is only a suggestion: PROCESS_QUERY_LIMITED_INFORMATION is the documented lesser alternative to PROCESS_QUERY_INFORMATION and succeeds on more kinds of process, which matters because a member you cannot open counts as a probe failure when it is the root. The Job Object membership list already closes most of the pid-reuse window between listing and opening; a comment saying so would help the next reader.

What I checked and found right

The limit kill reuses the existing graceful-then-forced tree shutdown rather than inventing one. A limit kill never enters quarantine. The daemon path and the direct path behave the same, and the client gets a structured error naming the limit, the observed value and the configured value. Sampling piggy-backs on the existing poll loops — no new sleep anywhere — and graph content is never affected by a limit, which is the line we care most about. The tests use clock and RSS seams instead of real time. The coexistence with cbm_index_worker_job_memory_limit is exactly as you describe it: OS cap at spawn, parent-side sampling on top.

One thing I have not done yet: build and run this head locally. External changes that spawn processes or edit executed scripts go through a per-head clearance step on our side first, and this head has not been through it. That is our queue, not a request to you.

When should the next slice move?

After this one merges — not before. Concretely: the ri_phys_footprint change, then a decision on point 2 (it changes an option's meaning, so it is cheaper to settle here than in #1725). Points 1, 3 and 4 can ride in this PR or a follow-up, your choice. Please keep #1725–#1728 untouched until then; each of them will need exactly one helper line for its new policy key now, instead of a main breakage — which is thanks to you.

A design question that is ours to decide, not yours to solve: whether index_max_rss_mb should also clamp the worker's cooperative memory budget, so a run stays under the cap instead of being killed at it. I will settle it and report back here.

Thank you for the care in this series. It shows in every error path.

@liuchong

Copy link
Copy Markdown
Contributor Author

@DeusData addressed on 5033f97d (still contains current main d615088a; #1725–#1728 untouched).

Blocker — macOS ri_phys_footprint. The process-tree probe now sums usage.ri_phys_footprint instead of ri_resident_size, matching cbm_mem_charged(). The fake seam and the equality test are named charged_rss so that quantity is hard to “fix” back to resident size.

Point 2 — duration is per request. index_max_duration_seconds covers the whole index request: the clock is stamped once at the first spawn and recovery respawns reuse that origin. Help text, CONFIGURATION.md, and INDEX_RESOURCE_LIMITS.md say so. Pinned by index_supervisor_duration_spans_recovery_attempts (fake clock, origin 100, second attempt at 1101 observes 1001 against a 1000 ms limit).

Point 3. Docs now state that a duration shorter than the 15-minute quiet timeout kills a hung worker before hang quarantine can name the in-flight file.

Point 4. Comment on the Windows OpenProcess flags and that the Job Object membership list already closes most of the pid-reuse window. Flags unchanged.

Point 1 (daemon default-start malformed-policy test) left for a follow-up.

Local index_supervisor + subprocess: 50 passed, 2 Windows-only skips. CI is restarting on this head.

@liuchong

Copy link
Copy Markdown
Contributor Author

@DeusData dco and ci-ok are green on 08d5e206, which is on current main (e783f73d). The review content you already verified on 5033f97d is unchanged (phys_footprint, per-request duration, docs). The only extra commit is the MSan-only startup budget for daemon_bootstrap_fails_fast_when_daemon_dies_at_publication, which is what made test-msan fail last time.

If you have a moment for the local clearance build, that would keep this slice from picking up more main drift. I will not rebase unless you ask. #1725–#1728 stay parked.

@DeusData

Copy link
Copy Markdown
Owner

Please drop the 180 s client-startup budget from 08d5e206 — that failure is ours, not yours, and there is already a fix in flight for it that removes the deadline instead of enlarging it.

daemon_bootstrap_fails_fast_when_daemon_dies_at_publication is the only test that installs bootstrap_enospc_host_spawn, and that seam returned while the spawned host was still running. Before the host listens it SHA-256s its own image, which for the ~450 MB test runner is about 3 s natively and about 26 s under MSan (48 CI samples: 20–32 s). So the client's 30 s startup deadline was racing the host's start-failure record, and on a slow run the deadline won and the reaper killed a host that had not failed yet. That is why ASSERT(daemon_named_cause) has gone red on PRs that touch no C at all, including #2245, #2158 and #2140 — and now yours.

#2272 makes that spawn synchronous: the host writes its start-failure record and releases its lifetime reservation before exiting, so once it has been reaped, "record on disk, reservation released" is a stable state pinned by construction, and the client's first look finds it however long the host took. The verdict stops depending on any deadline. Two reasons to prefer that here: a 180 s budget still lets a timeout decide the test, which is the one thing our test doctrine forbids outright; and your hunk and #2272 edit the same function in the same file, so whichever lands second gets a conflict for no gain.

Concretely: drop that hunk, keep fix(index): charge macOS phys_footprint and pin duration across recovery (that one is yours and I have not reviewed it yet), and this red clears itself when #2272 lands — no push needed from you for it. If MSan still fails after that with a different assertion, send it my way and I will take it.

On the local clearance build you asked about: it is granted, for your head 08d5e206, so the local build is next on my side, and I am sorry it took this long. A clearance covers exactly one commit, so when you push the change without the budget hunk I will re-scan the new head before building it; that is a formality, not another wait on you.

One practical note, offered rather than asked: each push restarts your whole matrix from the back of a busy queue, so batching the next changes into one push will get you a verdict sooner. And thank you for the precise diagnosis — naming the assertion and confirming index_supervisor was green is what made this five minutes of work instead of an hour.

@liuchong
liuchong force-pushed the feat/index-resource-worker branch from 48f9de8 to a8da9a3 Compare September 24, 2026 02:50
@liuchong

Copy link
Copy Markdown
Contributor Author

@DeusData the 180s startup budget is gone. Head is a8da9a36 on current main (5958f546). tests/test_daemon_bootstrap.c is back to the original 30s startup_timeout_ms; the phys_footprint / per-request duration change is otherwise the same. #1725–#1728 stay parked.

@DeusData

DeusData commented Oct 1, 2026

Copy link
Copy Markdown
Owner

Thank you, @liuchong, for removing the unrelated startup-budget change and confirming that the later slices remain parked. We need more time to review and validate the current head. The next step is on our side, and I am sorry for the repeated waits and inconsistent messages earlier in this thread. Please do not rebase again just to keep the branch fresh, and keep #1725 through #1728 where they are until we give you a concrete handoff. Thank you again.

@liuchong

liuchong commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

@DeusData I will not rebase for now.

There is already a conflict, and it was there when you wrote that note. src/daemon/application.c does not merge with #2144 (28b41c35): this branch adds request_started_ms on the job, and #2144 adds async_owned and worker_started in the same place. The other files in this slice merge cleanly. Head is still a8da9a36.

When would it be convenient to rebase, retest, and merge? Does this conflict need to be resolved on this branch first, or is it fine to leave until your side is ready?

@liuchong
liuchong force-pushed the feat/index-resource-worker branch from a8da9a3 to a80da0b Compare October 12, 2026 11:12
Discovery limits bound what indexing accepts, not what it then costs. A
repository well inside those bounds can still exhaust the host through parser
memory, or simply never finish, and a supervised worker that hangs leaves the
parent waiting with nothing to report.

Add index_max_rss_mb and index_max_duration_seconds, enforced by the parent
against the worker process tree rather than the worker process alone, so a
runaway child cannot hide behind a small parent. Resident memory is sampled
through the platform interface on macOS, Linux and Windows. Crossing a limit
terminates the tree and yields one trusted, structured terminal result that
attributes the failure to the resource that caused it.

Both limits default to off. A measurement that cannot be taken fails the
attempt instead of passing it: a watchdog that quietly stops watching is
worse than no watchdog at all.

The shell fixture that stands in for the supervisor names the two new keys.
The worker accepts only a policy that spells out every key it knows, which is
what keeps a stale supervisor from starting a worker it cannot bound.

Signed-off-by: 刘冲 <mail@liuchong.dev>
…ture

PR 2257 still named only the two discovery keys in the worker-scope script.
The worker now fail-closes unless every known policy key is present, so the
fixture must stand in for the supervisor with cbm_test_index_worker_policy_json.

Signed-off-by: 刘冲 <mail@liuchong.dev>
Watchdogs now sample the OS-charged footprint instead of resident size on
Darwin, and index_max_duration_seconds covers the whole request so a
crash/hang respawn cannot reset the clock.

Signed-off-by: 刘冲 <mail@liuchong.dev>
@DeusData

Copy link
Copy Markdown
Owner

Hi @liuchong, thank you, and sorry that your October 3 question went unanswered for so long. That is on us.

To answer it: the conflict with #2144 in src/daemon/application.c is resolved on your current head 1e4751d, the branch merges cleanly into today's main, and dco and ci-ok are both green. We also read the rebase against the head we had already verified: the fixture commit is unchanged, and the only new code is the adapter that hands the policy, the log callback and the request duration origin to the daemon's newer worker start path. That is the right way to fit it in.

Nothing further is needed from you on this slice. The next steps are ours: the per-head clearance step for 1e4751d (it is new code that spawns processes in a test and edits a test script, so it gets a fresh look), then our local build of the merge result, then the merge decision. Please do not rebase #1724 again unless we ask; we rebuild the merge result locally against the real main in any case.

For when the next slices move (please keep them where they are until #1724 lands):

  1. feat(index): enforce opt-in storage resource limits #1725: lint fails only on a stray re-indent of CLI_MB_FACTOR in src/cli/cli.c; dropping that hunk clears it.
  2. feat(index): add opt-in index resource profiles #1726: after the October 19 rebase it no longer compiles. application.c reads application->worker_memory_budget_bytes, which main has since replaced. Please pick the per-worker budget main uses today, since that is what the strict profile is defined against. The September 1 notes on feat(index): add opt-in index resource profiles #1726 to feat(mcp): warn on answers served from a failed rebuild #1728 still stand.
  3. The memory-budget clamp we discussed on September 21 is still yours if you would like it, as its own small slice after this merges; if not, we will do it and credit the idea to this thread.

Thank you for the patience and the care across the whole series.

@liuchong

Copy link
Copy Markdown
Contributor Author

@DeusData #1724 stays at 1e4751d3. I have not rebased it.

The later slices now match the notes on this PR:

The memory-budget clamp stays out of this stack. I will send it as its own slice after #1724 lands. #1725–#1728 stay stacked here until then.

@DeusData

Copy link
Copy Markdown
Owner

Thanks for the October 21 update and for keeping this head steady while preparing the later slices. I have noted the completed stack fixes and the separate budget-clamp follow-up. The clearance, integration validation and handoff are still ours; sorry for the delay—I’ll follow up soon. No further rebase is requested.

index_supervisor_quiet_timeout_remains_hang_with_duration_enabled ran the
duration watchdog on the real clock with a 60 s limit next to a 1 s quiet
timeout. On a starved host the duration limit can be crossed before the
quiet timeout fires, and the verdict then turns from HANG into a resource
kill, so the test depended on how fast the machine was.

The resource clock is now pinned at the request origin through the existing
clock seam. The quiet timeout runs on the subprocess clock, so it is the only
way the hung worker can end, however slowly the host runs. The limit is set
equal to the quiet timeout on purpose: more real time than the limit always
passes and it still cannot fire. A new assertion checks that the duration
check read the pinned clock, so HANG wins against an armed watchdog.

RED (real clock, limit crossed first): FAIL ASSERT(remained_hang) with
index.supervisor.resource_limit resource=duration_ms. GREEN with the pin.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
subprocess_tree_rss_measures_contained_descendants asserted only that the
tree total was above zero, which the root alone satisfies, and it was
skipped on Windows although its skip text said it ran there.

The test runner gains two roles: __cbm_rss_tree_root starts one
__cbm_rss_tree_hold descendant (fork on POSIX, CreateProcessW into the same
Job Object on Windows) that touches a controlled allocation and publishes its
pid by an atomic rename. The test measures two otherwise identical trees, one
whose descendant holds nothing and one whose descendant holds 128 MiB, and
requires the second total to be at least 96 MiB higher. Only a probe that
measures past the root can pass. The test now runs natively on Windows.

The waits have no deadline: they end when the descendant is ready or the
tree stops running, and the root exits when its descendant dies.

Binding check: with the macOS probe reduced to the root only, the test fails
(the loaded tree is not 96 MiB above the idle one). GREEN on the real probe
(idle 43 MiB, loaded 187 MiB under ASan).

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_subprocess_tree_rss_bytes skipped any non-root process it could not
measure and still returned OK with a smaller total, so a descendant the
platform refused to measure silently dropped out of the RSS watchdog.

Now a member is skipped only when the platform says it no longer exists:
ESRCH from proc_pid_rusage on macOS, ENOENT or ESRCH for /proc/<pid>/stat on
Linux, ERROR_INVALID_PARAMETER from OpenProcess (or an exit code from a
handle whose memory query failed) on Windows. Any other failure fails the
probe, which the supervisor already turns into resource_probe_failed after
three consecutive errors. The root stays strict as before.

On macOS the members now come from proc_listpids(PROC_PGRP_ONLY): the old
all-process scan read every process with proc_pidinfo, which answers EPERM
for every process of another user (178 of 6438 on this machine), so it could
not tell a foreign process from an unreadable member and fail-closed
semantics would have failed every probe there. Linux keeps its /proc scan;
stat files are world-readable on a default mount, and a hidepid=1 mount now
fails closed, which the doc states.

A test seam makes every non-root measurement fail with one platform error.
RED (old skip-any-failure decision): the live-member test returns OK where
ERROR is required. GREEN with the fix; the exited-member test passes in both,
so verified exit races are still ignored.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A worker resource limit is enforced by the supervisor, but only the worker
replaces the serving database, and it does so before the persistence
artifact export. A limit observed in that window stopped a worker that had
already published, and the response still said serving_index_preserved=true
while the previous index had been replaced.

Publication and resource termination now coordinate through a per-attempt
publication fence: a never-created path next to a reserved unique file in
the private worker directory, which both sides claim by creating it
exclusively (cbm_fopen "wbx": O_EXCL on POSIX, CREATE_NEW on Windows), so
exactly one of them can win.

- The worker claims it through a new pipeline publication gate that
  cbm_pipeline_run calls once, after the last pre-publication check and
  before the serving database is touched. Losing means the supervisor
  already stopped the attempt for a limit: the run aborts before
  publication and the previous generation stays in place.
- The supervisor claims it before it terminates the worker for a limit or
  a failed RSS probe. Losing means the worker already committed to
  publishing: the limit is no longer enforced for that attempt, which runs
  to completion, and index.supervisor.resource_limit_after_publication is
  logged instead. A claim that can be neither won nor lost is retried on
  the next poll rather than risk stopping a committed worker.
- At terminal the supervisor settles the fence into
  result.publication_committed. An unquiesced tree keeps the fence, so a
  surviving member can never publish after the report.

serving_index_preserved now describes the request, not only its last
attempt: a servable index existed when the request began, no attempt of
the request committed to publishing (an earlier attempt can publish and
then crash into recovery), and that index is still servable. The direct
path and the daemon job both fold every attempt into these request facts;
cbm_mcp_index_worker_resource_response takes them as a third argument, and
NULL claims nothing.

The fence path travels as an optional last pair of the exact worker grammar
(--index-worker-publish-fence), accepted by the worker argv parser and by
the daemon bootstrap role check, stripped by the CLI and installed by both
worker entries.

Limits stay opt-in: without index_max_rss_mb or index_max_duration_seconds
there is no fence, no extra argv, no claim and no extra database read.
Graph content is unchanged either way.

RED (c78c077 plus only the new end-to-end test): a FIFO at the artifact
temp path holds the real worker after the rename, and the duration clock
crosses its limit only once the serving database has been replaced. The
supervisor stops the worker there and answers resource_limit_exceeded with
serving_index_preserved=true over a replaced database. GREEN: the worker
claimed first, the supervisor logs the limit as not enforced, and the
released worker finishes and reports the published index.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Brings the branch up to main 9c19b5d.

Two same-spot insertion conflicts, each resolved as the main version
of the file plus the own diff of this branch:
- src/daemon/application.c: the job struct gained watch_owners on main
  and request_started_ms plus the request facts here; both kept.
- tests/test_daemon_application.c: main added the DeusData#1818
  autoindex-skipped helpers where this branch added the worker resource
  failure test; both kept, the test placed after the doc_candidates
  test.

Everything else merged cleanly. src/main.c changed on both sides: main
(DeusData#2838) lowers the worker CPU priority inside the worker role block,
and this branch strips the publication fence flag before that block
and installs the fence after the worker role options; the hunks are
disjoint and both are kept. tests/test_pipeline.c gained the Next.js
route tests (DeusData#2747) on main and the publication gate test here, also
disjoint. No call site needed adapting: no new main code calls a
function this branch re-signed, and main re-signed none that this
branch calls.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData

DeusData commented Nov 2, 2026

Copy link
Copy Markdown
Owner

Hi @liuchong, thank you for this slice, and for keeping #1724 steady at 1e4751d for so long while we caught up on our side. It holds up very well: the three-valued probe status, the Job Object membership list, the per-request duration origin threaded through both the direct and the daemon path, and graph content that no limit can ever change. Those are exactly the decisions we care about most, and you built every one of them with care.

The per-head clearance and our local build of the merge result are done now. On that build we went through the last places where a watchdog could report something that is not quite true, and since maintainer edits are enabled, we pushed the fixes onto your branch ourselves instead of sending you another round. Here is every commit and why:

  1. test(index): pin the duration clock in the quiet-timeout watchdog test. The quiet-timeout test ran the duration watchdog on the real clock, so on a slow or starved runner the duration limit could fire first and turn the expected HANG into a resource kill. The duration clock is now pinned through your own clock seam, so the quiet timeout is the only possible ending, and the test also checks that the duration check really ran.

  2. test(subprocess): prove descendant memory is measured on every platform. The descendant test only asserted a total above zero, which the root alone already gives, and it was skipped on Windows. The test runner now has a small root/hold process pair where only the descendant touches 128 MiB, and the test requires the tree total to rise by at least 96 MiB. It is no longer skipped on Windows: there the descendant is started with CreateProcessW inside the same Job Object, so the Windows CI legs exercise the native probe.

  3. fix(subprocess): fail the tree probe on a live member it cannot measure. A process that could not be measured was skipped and the probe still returned OK with a smaller total. Now only a process the platform reports as gone is skipped (ESRCH on macOS, ENOENT/ESRCH on Linux, ERROR_INVALID_PARAMETER or an exit code on Windows); anything else fails the probe, which your supervisor already turns into resource_probe_failed. On macOS the members now come from proc_listpids(PROC_PGRP_ONLY), because proc_pidinfo answers EPERM for every process of another user, so the old full scan could not tell a foreign process from an unreadable member of our tree. Your "a watchdog that quietly stops watching is worse than no watchdog" principle is exactly why.

  4. fix(index): coordinate publication between the worker and its supervisor. This is the one larger change, and it closes a gap that only shows up because your watchdogs are strict. The supervisor enforces the limits, but only the worker replaces the serving database, and it does that before the persistence artifact export. So a limit could stop a worker that had already published, and the response would still say serving_index_preserved=true while the previous index had in fact been replaced. Rather than only reporting that case, the worker and the supervisor now coordinate so it cannot happen:

    • When index_max_rss_mb or index_max_duration_seconds is enabled, every worker attempt gets a publication fence: a private path that both sides claim by creating it exclusively (O_EXCL on POSIX, CREATE_NEW on Windows), so exactly one of them wins. The path reaches the worker as one more optional pair at the end of your exact worker grammar (--index-worker-publish-fence), accepted by both the worker argv parser and the daemon bootstrap role check.
    • The worker claims it through a small publication gate in cbm_pipeline_run, right after the last pre-publication check and before the serving database is touched. If the supervisor got there first, the run stops before publication and the previous index stays exactly as it was.
    • The supervisor claims it before it stops the worker for a limit or a failed RSS probe. If the worker got there first, the attempt has committed to publishing, so the limit is no longer enforced for it; the supervisor logs index.supervisor.resource_limit_after_publication and lets the attempt finish.
    • serving_index_preserved now describes the whole request: a servable index existed when the request began, no attempt of the request committed to publishing (an earlier attempt can publish and then crash into recovery), and that index still serves. The direct path and the daemon job both carry these request facts across attempts.
    • With both worker limits off there is no fence, no extra argument and no extra database read, so the default path is unchanged, and graph content is identical with limits on or off (checked on prometheus/prometheus).
      A new end-to-end test pins this without any timing: a FIFO holds the real worker just after it published, and the duration clock crosses its limit only once the serving database was replaced. It fails on the previous head with exactly the untruthful response and passes now. This replaces the earlier idea of only reporting a replaced index in a new response field, so there is no new public field.
  5. A merge of current main. The only two conflicts were same-spot insertions in src/daemon/application.c (main added watch_owners to the job struct right where you added request_started_ms) and tests/test_daemon_application.c (main added helpers right where you added the worker resource test). Both sides are kept, and we checked that the result is main plus exactly your change.

Please pull before pushing again, so these commits are not lost. For the later slices: cbm_mcp_index_worker_resource_response now takes a third argument, the request facts (NULL is fine in a test that does not check serving_index_preserved, which then reports false). #1725 adds one test call to it, and the stacked slices will also see the new optional worker argument. We will say the same on #1725 when it moves.

Thank you again for the care across this whole series. It shows in every error path.

The memory-core ratchet (lint-memory-core) failed on the publication
fence commit: it added six raw free() calls, each releasing a block that
existing code allocates with raw calloc, malloc or strdup. cbm_free cannot
release such a block without the core class counters drifting, so every
new exit now reuses an existing release point instead:

- index_supervisor.c: worker_artifacts_create() creates the response, log
  and publish fence files and names the one that failed, so all three
  failures share the single artifact_create_failed exit.
- pipeline.c: a refused publication gate takes the existing source-changed
  abort exit (same CBM_PIPELINE_ABORT_PRESERVE_DB result and cleanup).
- mcp.c: index_request_names_free() is the one release point for the
  request name strings, used by index_worker_json_result() and
  index_args_serving_index_servable().

Raw-site counts return to the origin/main values (index_supervisor.c 8,
pipeline.c 129, mcp.c 765) with no baseline change. No behaviour change.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData

DeusData commented Nov 3, 2026

Copy link
Copy Markdown
Owner

Thank you again, @liuchong, for the careful work on the resource worker and for your patience while we iterated on the publication handshake. We pushed one more small commit on top (7a1bee2) that only reroutes the new error exits from our publication-fence change through existing cleanup helpers, so the memory-core lint check that keeps raw allocator calls from growing is green again, with no change in behaviour. Please pull the branch before pushing again so your next push builds on it.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData
DeusData merged commit 008b6c3 into DeusData:main Nov 3, 2026
41 checks passed
@DeusData

DeusData commented Nov 3, 2026

Copy link
Copy Markdown
Owner

Merged - thank you very much, @liuchong! The opt-in index resource limits are on main now: a supervised worker with RSS and duration watchdogs, a tree probe that fails closed instead of quietly measuring less, structured and non-retried limit responses, and graph content that no limit can ever change. Together with the publication fence on top, a limit can no longer report a preserved index that was in fact replaced. Thank you for keeping this slice steady for so long while we worked through our side.

For the next slices: #1725 adds one test call to cbm_mcp_index_worker_resource_response, which now takes a third argument (the request facts; NULL is fine in a test that does not check serving_index_preserved), and the stacked slices will also see the new optional --index-worker-publish-fence worker argument. We will pick #1725 up next.

DeusData added a commit to Andy11-cpu/codebase-memory-mcp that referenced this pull request Nov 3, 2026
Bring the branch up to main e0c7a1d
(DeusData#2395), which includes DeusData#1724 (index resource worker and publication
fence, 008b6c3).

Conflict resolved (1 file):
- docs/CONFIGURATION.md: both sides appended rows to the "Current keys"
  table at the same spot. Resolved as main rows index_max_rss_mb and
  index_max_duration_seconds first, then the branch row install_hooks
  and its paragraph that closes the table. No conflict side was
  concatenated as-is; the branch paragraph must follow the last table
  row.

Auto-merged without conflict: README.md, src/cli/cli.c (CONFIG_KEYS
now holds the new index_max_rss_mb / index_max_duration_seconds keys
and install_hooks), src/main.c (worker publish-fence argv next to the
--hooks/--no-hooks help line).

Check: for all seven branch files, the zero-context patch-id of the
merged tree against main equals the branch own change. Semantic check:
the branch re-signs no function; main re-signed the daemon worker start
callback and the supervisor result struct, which the branch does not
use. No call site adapted.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request priority/high Needs near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker. stability/performance Server crashes, OOM, hangs, high CPU/memory

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants