Repository navigation
feat(index): add worker resource watchdogs - #1724
Conversation
|
Thanks for opening this — it has been seen, and it is queued. This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence. Current review status: working through a backlog. What that means for this PR, concretely:
Things that will genuinely speed it up whenever review does happen:
If this fixes a bug, a reproduction we can run is worth more than a description of the symptom. Thanks for contributing, and sorry in advance for the wait. |
ece9042 to
eaab28c
Compare
a1339ec to
38ec2e4
Compare
|
Reviewed slice 2. The direction is already accepted (see #1723); this is the merit read, and the design holds up well. The three-valued status is the right call and it is the thing most people get wrong here. CBM_PROC_TREE_RSS_ERROR = -1,
CBM_PROC_TREE_RSS_EMPTY = 0,
CBM_PROC_TREE_RSS_OK = 1,"No trustworthy measurement" and "the tree has no observable members" are different facts, and collapsing them into "A watchdog that quietly stops watching is worse than no watchdog" is the right principle and you built to it. Using a Windows Job Object is the detail I want to single out. The error paths around it are thorough too: allocation failure, size overflow past Measuring the tree rather than the root process is the correct scope, and your reason for it — "a runaway child hiding behind a small parent is exactly the case that goes unattributed today" — is the sentence that justifies the extra complexity. Ignoring processes that exit mid-enumeration is the right way to handle the inherent race. Exposing Process notesClearance: this shows Rebase when #1723 lands. This diff currently carries #1723's commit; once that merges it collapses to Slice 3 onward is where the configuration surface really grows, so expect those to get a closer look than this one — but this slice is in good shape. |
70b411d to
87da442
Compare
87da442 to
42a3d71
Compare
|
@DeusData rebased One conflict worth naming: CI is restarting. I will wait for you to say when the next slice should move. |
42a3d71 to
81ff2ca
Compare
|
@DeusData rebased The previous matrix's reds were:
Local |
d48afe8 to
3743fe9
Compare
DeusData
left a comment
There was a problem hiding this comment.
Thank you — for the two rebases, for stopping exactly where you said you would, and for two fixes I was about to ask for and found already done.
I read head 42a3d710 in full this morning and wrote three blocking findings. By the time I could post, you had fixed two of them yourself:
- The memory-core ratchet — the process-tree probe now allocates through
mem_coreand the worker resource responses share one release helper.lint / lintis green on3743fe95. - The
Step 5fcollision — this one was ours, not yours, and you fixed it anyway.tests/test_worker_session_scope.shonmainhard-coded a two-key policy literal while the worker demands the exact key set, so any slice that adds a policy key turnsmainred after a textually clean merge. That is precisely what #1723 did to us at the weekend, and this slice (2 → 4 keys) would have done it again. Your last commit routes the fixture throughcbm_test_index_worker_policy_json, which is the durable fix: every later slice only has to touch the helper. For the record, I think the worker's exact-key-count check should stay as strict as it is — a supervisor that silently drops a key would silently disable a limit.
That leaves one blocker, and it is a one-word change with a long story behind it.
Blocking: macOS samples ri_resident_size; it must be ri_phys_footprint
src/foundation/subprocess.c:577 sums usage.ri_resident_size over the process group. On macOS, resident size keeps counting every page mimalloc has already handed back (MADV_FREE_REUSABLE) until the kernel gets round to reclaiming it. We measured this the hard way — it is written down in src/foundation/mem.h on main: on the Linux-kernel corpus the worker sat at 17.4 GB resident with 5.5 GB actually charged. With this slice as it stands, a user who sets index_max_rss_mb to 8 GB on a Mac can get a healthy run killed while the OS is charging it a fraction of that, with an error message naming a number the OS itself does not hold against them.
rusage_info_v2 already carries ri_phys_footprint, so the probe needs no new call — just the other field. That is also the number cbm_mem_charged() enforces the cooperative budget against, so the two limits would finally agree on what "memory" means. Linux and Windows are fine as they are. If you can, make the fake-RSS seam's test name say which quantity it represents, so nobody "fixes" it back.
Not blocking — but please read
- The daemon's default start path has no test.
application_worker_start_default(src/daemon/application.c, the policy parse pluscbm_index_worker_start_with_policy) is only reachable through the real.start; every case intests/test_daemon_application.cinstallsapp_fake_worker_start. So "a malformed internal policy makes the daemon refuse the job" is currently asserted by nothing. One case that feeds a bad_cbm_index_policythrough the default start and expects-1plusdaemon.index.policywould close it. index_max_duration_secondsis per attempt, not per request.started_msis set when a worker handle starts, and recovery starts a new handle per attempt — so a request that crashes and recovers can run for several times the configured duration. Either is defensible; what is not is the ambiguity. Please pick one, name it in the option's help text, and pin it with a fake-clock test across a recovery. My preference is per request: a user who writes "30 minutes" means the thing they are waiting for.- A duration limit under 15 minutes pre-empts hang quarantine.
DEFAULT_QUIET_TIMEOUT_MSis 900 000; a shorter duration limit kills a hung worker first, and because a limit kill (correctly) never enters the quarantine loop, as far as I can see the file that hangs is never identified, so the next run would hang on it again. Worth one sentence in the docs at minimum; better, log the in-flight file marker on a duration kill. - Windows probe, small.
OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_VM_READ, …)matches the documented requirement forGetProcessMemoryInfo, so this is only a suggestion:PROCESS_QUERY_LIMITED_INFORMATIONis the documented lesser alternative toPROCESS_QUERY_INFORMATIONand succeeds on more kinds of process, which matters because a member you cannot open counts as a probe failure when it is the root. The Job Object membership list already closes most of the pid-reuse window between listing and opening; a comment saying so would help the next reader.
What I checked and found right
The limit kill reuses the existing graceful-then-forced tree shutdown rather than inventing one. A limit kill never enters quarantine. The daemon path and the direct path behave the same, and the client gets a structured error naming the limit, the observed value and the configured value. Sampling piggy-backs on the existing poll loops — no new sleep anywhere — and graph content is never affected by a limit, which is the line we care most about. The tests use clock and RSS seams instead of real time. The coexistence with cbm_index_worker_job_memory_limit is exactly as you describe it: OS cap at spawn, parent-side sampling on top.
One thing I have not done yet: build and run this head locally. External changes that spawn processes or edit executed scripts go through a per-head clearance step on our side first, and this head has not been through it. That is our queue, not a request to you.
When should the next slice move?
After this one merges — not before. Concretely: the ri_phys_footprint change, then a decision on point 2 (it changes an option's meaning, so it is cheaper to settle here than in #1725). Points 1, 3 and 4 can ride in this PR or a follow-up, your choice. Please keep #1725–#1728 untouched until then; each of them will need exactly one helper line for its new policy key now, instead of a main breakage — which is thanks to you.
A design question that is ours to decide, not yours to solve: whether index_max_rss_mb should also clamp the worker's cooperative memory budget, so a run stays under the cap instead of being killed at it. I will settle it and report back here.
Thank you for the care in this series. It shows in every error path.
|
@DeusData addressed on Blocker — macOS Point 2 — duration is per request. Point 3. Docs now state that a duration shorter than the 15-minute quiet timeout kills a hung worker before hang quarantine can name the in-flight file. Point 4. Comment on the Windows Point 1 (daemon default-start malformed-policy test) left for a follow-up. Local |
|
@DeusData If you have a moment for the local clearance build, that would keep this slice from picking up more |
|
Please drop the 180 s client-startup budget from
#2272 makes that spawn synchronous: the host writes its start-failure record and releases its lifetime reservation before exiting, so once it has been reaped, "record on disk, reservation released" is a stable state pinned by construction, and the client's first look finds it however long the host took. The verdict stops depending on any deadline. Two reasons to prefer that here: a 180 s budget still lets a timeout decide the test, which is the one thing our test doctrine forbids outright; and your hunk and #2272 edit the same function in the same file, so whichever lands second gets a conflict for no gain. Concretely: drop that hunk, keep On the local clearance build you asked about: it is granted, for your head One practical note, offered rather than asked: each push restarts your whole matrix from the back of a busy queue, so batching the next changes into one push will get you a verdict sooner. And thank you for the precise diagnosis — naming the assertion and confirming |
48f9de8 to
a8da9a3
Compare
|
@DeusData the 180s startup budget is gone. Head is |
|
Thank you, @liuchong, for removing the unrelated startup-budget change and confirming that the later slices remain parked. We need more time to review and validate the current head. The next step is on our side, and I am sorry for the repeated waits and inconsistent messages earlier in this thread. Please do not rebase again just to keep the branch fresh, and keep #1725 through #1728 where they are until we give you a concrete handoff. Thank you again. |
|
@DeusData I will not rebase for now. There is already a conflict, and it was there when you wrote that note. When would it be convenient to rebase, retest, and merge? Does this conflict need to be resolved on this branch first, or is it fine to leave until your side is ready? |
a8da9a3 to
a80da0b
Compare
Discovery limits bound what indexing accepts, not what it then costs. A repository well inside those bounds can still exhaust the host through parser memory, or simply never finish, and a supervised worker that hangs leaves the parent waiting with nothing to report. Add index_max_rss_mb and index_max_duration_seconds, enforced by the parent against the worker process tree rather than the worker process alone, so a runaway child cannot hide behind a small parent. Resident memory is sampled through the platform interface on macOS, Linux and Windows. Crossing a limit terminates the tree and yields one trusted, structured terminal result that attributes the failure to the resource that caused it. Both limits default to off. A measurement that cannot be taken fails the attempt instead of passing it: a watchdog that quietly stops watching is worse than no watchdog at all. The shell fixture that stands in for the supervisor names the two new keys. The worker accepts only a policy that spells out every key it knows, which is what keeps a stale supervisor from starting a worker it cannot bound. Signed-off-by: 刘冲 <mail@liuchong.dev>
…ture PR 2257 still named only the two discovery keys in the worker-scope script. The worker now fail-closes unless every known policy key is present, so the fixture must stand in for the supervisor with cbm_test_index_worker_policy_json. Signed-off-by: 刘冲 <mail@liuchong.dev>
Watchdogs now sample the OS-charged footprint instead of resident size on Darwin, and index_max_duration_seconds covers the whole request so a crash/hang respawn cannot reset the clock. Signed-off-by: 刘冲 <mail@liuchong.dev>
691a084 to
1e4751d
Compare
|
Hi @liuchong, thank you, and sorry that your October 3 question went unanswered for so long. That is on us. To answer it: the conflict with #2144 in src/daemon/application.c is resolved on your current head 1e4751d, the branch merges cleanly into today's main, and dco and ci-ok are both green. We also read the rebase against the head we had already verified: the fixture commit is unchanged, and the only new code is the adapter that hands the policy, the log callback and the request duration origin to the daemon's newer worker start path. That is the right way to fit it in. Nothing further is needed from you on this slice. The next steps are ours: the per-head clearance step for 1e4751d (it is new code that spawns processes in a test and edits a test script, so it gets a fresh look), then our local build of the merge result, then the merge decision. Please do not rebase #1724 again unless we ask; we rebuild the merge result locally against the real main in any case. For when the next slices move (please keep them where they are until #1724 lands):
Thank you for the patience and the care across the whole series. |
|
@DeusData #1724 stays at The later slices now match the notes on this PR:
The memory-budget clamp stays out of this stack. I will send it as its own slice after #1724 lands. #1725–#1728 stay stacked here until then. |
|
Thanks for the October 21 update and for keeping this head steady while preparing the later slices. I have noted the completed stack fixes and the separate budget-clamp follow-up. The clearance, integration validation and handoff are still ours; sorry for the delay—I’ll follow up soon. No further rebase is requested. |
index_supervisor_quiet_timeout_remains_hang_with_duration_enabled ran the duration watchdog on the real clock with a 60 s limit next to a 1 s quiet timeout. On a starved host the duration limit can be crossed before the quiet timeout fires, and the verdict then turns from HANG into a resource kill, so the test depended on how fast the machine was. The resource clock is now pinned at the request origin through the existing clock seam. The quiet timeout runs on the subprocess clock, so it is the only way the hung worker can end, however slowly the host runs. The limit is set equal to the quiet timeout on purpose: more real time than the limit always passes and it still cannot fire. A new assertion checks that the duration check read the pinned clock, so HANG wins against an armed watchdog. RED (real clock, limit crossed first): FAIL ASSERT(remained_hang) with index.supervisor.resource_limit resource=duration_ms. GREEN with the pin. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
subprocess_tree_rss_measures_contained_descendants asserted only that the tree total was above zero, which the root alone satisfies, and it was skipped on Windows although its skip text said it ran there. The test runner gains two roles: __cbm_rss_tree_root starts one __cbm_rss_tree_hold descendant (fork on POSIX, CreateProcessW into the same Job Object on Windows) that touches a controlled allocation and publishes its pid by an atomic rename. The test measures two otherwise identical trees, one whose descendant holds nothing and one whose descendant holds 128 MiB, and requires the second total to be at least 96 MiB higher. Only a probe that measures past the root can pass. The test now runs natively on Windows. The waits have no deadline: they end when the descendant is ready or the tree stops running, and the root exits when its descendant dies. Binding check: with the macOS probe reduced to the root only, the test fails (the loaded tree is not 96 MiB above the idle one). GREEN on the real probe (idle 43 MiB, loaded 187 MiB under ASan). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_subprocess_tree_rss_bytes skipped any non-root process it could not measure and still returned OK with a smaller total, so a descendant the platform refused to measure silently dropped out of the RSS watchdog. Now a member is skipped only when the platform says it no longer exists: ESRCH from proc_pid_rusage on macOS, ENOENT or ESRCH for /proc/<pid>/stat on Linux, ERROR_INVALID_PARAMETER from OpenProcess (or an exit code from a handle whose memory query failed) on Windows. Any other failure fails the probe, which the supervisor already turns into resource_probe_failed after three consecutive errors. The root stays strict as before. On macOS the members now come from proc_listpids(PROC_PGRP_ONLY): the old all-process scan read every process with proc_pidinfo, which answers EPERM for every process of another user (178 of 6438 on this machine), so it could not tell a foreign process from an unreadable member and fail-closed semantics would have failed every probe there. Linux keeps its /proc scan; stat files are world-readable on a default mount, and a hidepid=1 mount now fails closed, which the doc states. A test seam makes every non-root measurement fail with one platform error. RED (old skip-any-failure decision): the live-member test returns OK where ERROR is required. GREEN with the fix; the exited-member test passes in both, so verified exit races are still ignored. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A worker resource limit is enforced by the supervisor, but only the worker replaces the serving database, and it does so before the persistence artifact export. A limit observed in that window stopped a worker that had already published, and the response still said serving_index_preserved=true while the previous index had been replaced. Publication and resource termination now coordinate through a per-attempt publication fence: a never-created path next to a reserved unique file in the private worker directory, which both sides claim by creating it exclusively (cbm_fopen "wbx": O_EXCL on POSIX, CREATE_NEW on Windows), so exactly one of them can win. - The worker claims it through a new pipeline publication gate that cbm_pipeline_run calls once, after the last pre-publication check and before the serving database is touched. Losing means the supervisor already stopped the attempt for a limit: the run aborts before publication and the previous generation stays in place. - The supervisor claims it before it terminates the worker for a limit or a failed RSS probe. Losing means the worker already committed to publishing: the limit is no longer enforced for that attempt, which runs to completion, and index.supervisor.resource_limit_after_publication is logged instead. A claim that can be neither won nor lost is retried on the next poll rather than risk stopping a committed worker. - At terminal the supervisor settles the fence into result.publication_committed. An unquiesced tree keeps the fence, so a surviving member can never publish after the report. serving_index_preserved now describes the request, not only its last attempt: a servable index existed when the request began, no attempt of the request committed to publishing (an earlier attempt can publish and then crash into recovery), and that index is still servable. The direct path and the daemon job both fold every attempt into these request facts; cbm_mcp_index_worker_resource_response takes them as a third argument, and NULL claims nothing. The fence path travels as an optional last pair of the exact worker grammar (--index-worker-publish-fence), accepted by the worker argv parser and by the daemon bootstrap role check, stripped by the CLI and installed by both worker entries. Limits stay opt-in: without index_max_rss_mb or index_max_duration_seconds there is no fence, no extra argv, no claim and no extra database read. Graph content is unchanged either way. RED (c78c077 plus only the new end-to-end test): a FIFO at the artifact temp path holds the real worker after the rename, and the duration clock crosses its limit only once the serving database has been replaced. The supervisor stops the worker there and answers resource_limit_exceeded with serving_index_preserved=true over a replaced database. GREEN: the worker claimed first, the supervisor logs the limit as not enforced, and the released worker finishes and reports the published index. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Brings the branch up to main 9c19b5d. Two same-spot insertion conflicts, each resolved as the main version of the file plus the own diff of this branch: - src/daemon/application.c: the job struct gained watch_owners on main and request_started_ms plus the request facts here; both kept. - tests/test_daemon_application.c: main added the DeusData#1818 autoindex-skipped helpers where this branch added the worker resource failure test; both kept, the test placed after the doc_candidates test. Everything else merged cleanly. src/main.c changed on both sides: main (DeusData#2838) lowers the worker CPU priority inside the worker role block, and this branch strips the publication fence flag before that block and installs the fence after the worker role options; the hunks are disjoint and both are kept. tests/test_pipeline.c gained the Next.js route tests (DeusData#2747) on main and the publication gate test here, also disjoint. No call site needed adapting: no new main code calls a function this branch re-signed, and main re-signed none that this branch calls. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
Hi @liuchong, thank you for this slice, and for keeping #1724 steady at 1e4751d for so long while we caught up on our side. It holds up very well: the three-valued probe status, the Job Object membership list, the per-request duration origin threaded through both the direct and the daemon path, and graph content that no limit can ever change. Those are exactly the decisions we care about most, and you built every one of them with care. The per-head clearance and our local build of the merge result are done now. On that build we went through the last places where a watchdog could report something that is not quite true, and since maintainer edits are enabled, we pushed the fixes onto your branch ourselves instead of sending you another round. Here is every commit and why:
Please pull before pushing again, so these commits are not lost. For the later slices: cbm_mcp_index_worker_resource_response now takes a third argument, the request facts (NULL is fine in a test that does not check serving_index_preserved, which then reports false). #1725 adds one test call to it, and the stacked slices will also see the new optional worker argument. We will say the same on #1725 when it moves. Thank you again for the care across this whole series. It shows in every error path. |
The memory-core ratchet (lint-memory-core) failed on the publication fence commit: it added six raw free() calls, each releasing a block that existing code allocates with raw calloc, malloc or strdup. cbm_free cannot release such a block without the core class counters drifting, so every new exit now reuses an existing release point instead: - index_supervisor.c: worker_artifacts_create() creates the response, log and publish fence files and names the one that failed, so all three failures share the single artifact_create_failed exit. - pipeline.c: a refused publication gate takes the existing source-changed abort exit (same CBM_PIPELINE_ABORT_PRESERVE_DB result and cleanup). - mcp.c: index_request_names_free() is the one release point for the request name strings, used by index_worker_json_result() and index_args_serving_index_servable(). Raw-site counts return to the origin/main values (index_supervisor.c 8, pipeline.c 129, mcp.c 765) with no baseline change. No behaviour change. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
Thank you again, @liuchong, for the careful work on the resource worker and for your patience while we iterated on the publication handshake. We pushed one more small commit on top (7a1bee2) that only reroutes the new error exits from our publication-fence change through existing cleanup helpers, so the memory-core lint check that keeps raw allocator calls from growing is green again, with no change in behaviour. Please pull the branch before pushing again so your next push builds on it. |
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
Merged - thank you very much, @liuchong! The opt-in index resource limits are on main now: a supervised worker with RSS and duration watchdogs, a tree probe that fails closed instead of quietly measuring less, structured and non-retried limit responses, and graph content that no limit can ever change. Together with the publication fence on top, a limit can no longer report a preserved index that was in fact replaced. Thank you for keeping this slice steady for so long while we worked through our side. For the next slices: #1725 adds one test call to |
Bring the branch up to main e0c7a1d (DeusData#2395), which includes DeusData#1724 (index resource worker and publication fence, 008b6c3). Conflict resolved (1 file): - docs/CONFIGURATION.md: both sides appended rows to the "Current keys" table at the same spot. Resolved as main rows index_max_rss_mb and index_max_duration_seconds first, then the branch row install_hooks and its paragraph that closes the table. No conflict side was concatenated as-is; the branch paragraph must follow the last table row. Auto-merged without conflict: README.md, src/cli/cli.c (CONFIG_KEYS now holds the new index_max_rss_mb / index_max_duration_seconds keys and install_hooks), src/main.c (worker publish-fence argv next to the --hooks/--no-hooks help line). Check: for all seven branch files, the zero-context patch-id of the merged tree against main equals the branch own change. Semantic check: the branch re-signs no function; main re-signed the daemon worker start callback and the supervisor result struct, which the branch does not use. No call site adapted. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Related to #1347.
Problem
Discovery limits bound what indexing accepts, not what it then costs. A repository well inside those bounds can still exhaust the host through parser memory, or simply never finish. A supervised worker that hangs leaves the parent waiting with nothing to report and no way to attribute the stall.
What this changes
Two opt-in limits, enforced by the parent against the worker process tree:
index_max_rss_mbindex_max_duration_secondsBoth default to
off.Measuring the tree rather than the worker process alone matters: indexing spawns children, and a runaway child hiding behind a small parent is exactly the case that goes unattributed today. Resident memory is sampled through the platform interface on macOS, Linux and Windows.
Crossing a limit terminates the process tree and produces one trusted, structured terminal result attributing the failure to the resource that caused it, so the caller learns which limit ended the attempt rather than seeing a generic worker failure.
A measurement that cannot be taken fails the attempt instead of passing it. A watchdog that quietly stops watching is worse than no watchdog, so the probe is fail-closed and says so in the result.
Testing
make -f Makefile.cbm testandmake -f Makefile.cbm lint-cion macOS. New coverage: process-tree RSS measurement, watchdog termination on each limit, probe-failure fail-closed behaviour, terminal-result attribution, and policy validation for the two new keys.Stack