Skip to content

tests, task: fix load-dependent test failures - #4674

Open
grandixximo wants to merge 8 commits into
LinuxCNC:masterfrom
grandixximo:flaky-tests-4669
Open

grandixximo wants to merge 8 commits into
LinuxCNC:masterfrom
grandixximo:flaky-tests-4669

Conversation

@grandixximo

@grandixximo grandixximo commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Part of #4669. @hdiethelm, this covers the failures you listed. I reproduced them by pinning the test and a few busy loops to one CPU (taskset -c 7).

  • touchy, touchy-quit: the sim loaded the gladevcp test panel without -W, so touchy_postgui.hal could run before touchy_test existed and touchy exited. Now loadusr -Wn touchy_test.
  • touchy-fit: xwininfo -tree also lists unmapped windows, so window-fit measured touchy before it was mapped. It now polls for a visible window of stable size; an oversized window still fails.
  • motion-logger tests: the canon commands queued at startup reached motion only when the UI sent its first command late. On your question about the wait: INTERP_IDLE is already true while interp_list still holds them, so the race stays. The test UIs now wait for s.state == linuxcnc.RCS_DONE, and the expected logs include those commands. startup-gcode-abort waits for EXEC_WAITING_FOR_DELAY instead, since its startup code sits in a dwell.
  • hal-show: halcmd's hal_init() right after fork races the loaded module for the next uspace component id. A 200 ms delay there reproduces your 13 => 11 diff. The owner column is masked together with its padding (thanks for catching the 9 vs 10 width).
  • gmoccapy: when homing took over 15 s, the retry asked for STATE_ESTOP_RESET with the machine already on, and task never reports that state while traj is enabled. Forcing the first homing wait to 50 ms reproduces your log exactly.
  • multiclick, mux: a stream of depth N held only N-1 samples, so the 100 line input never fit before the thread started, and it was preloaded behind a fixed 0.5 s sleep. A late halstreamer left the streamer empty and shifted the end of the output. The tests now use depth 128 and wait for halstreamer with loadusr -w; the stream capacity itself is fixed in hal: a stream of depth N holds N samples #4678.
  • linuxcncrsh: this one is in task, not the test. Motion reports a G38.2/G38.4 move that ended without contact one servo cycle after the planner goes idle, but task already counted motion DONE in that cycle. The MDI then finished without the error, and the error was returned for the next command. Task now waits for emcmotStatus.probing to clear before reporting motion DONE, which costs at most one servo cycle. With a program, the same window could let task issue the next line before the probe error. A 50 cycle delay before the verdict reproduces your diff, and it passes with the fix. @BsAtHome, since this is what your linuxcncrsh probe test was hitting, I'd like your view on fixing it in task.

Under load the affected tests went from failing to passing in every run (motion-logger 16/16, touchy tests 3/3 each). The full suite passes apart from blendmath and tool-frame, which don't build with GCC 14 because of EXPORT_SYMBOL (#4673).

touchy_postgui.hal links touchy_test pins, but the panel was loaded
without -W. Under load the pins did not exist yet and touchy exited.

Part of LinuxCNC#4669.
xwininfo -tree also lists unmapped windows, so under load window-fit
measured touchy at its creation size. Poll for a viewable window of
stable size.

Part of LinuxCNC#4669.
Whether the canon commands queued by interpreter init and the startup
code reach motion depended on how fast the UI sent its first command.
Wait for RCS_DONE (the dwell in startup-gcode-abort) and expect them.

Part of LinuxCNC#4669.
halcmd's hal_init() after fork races the loaded module for the next
uspace id, so the owner column changed under load. Mask it as the
mb2hal tests do.

Part of LinuxCNC#4669.
…ine on

ESTOP_RESET leaves traj enabled, so task keeps reporting ON and the
retry after a slow first homing attempt could never succeed.

Part of LinuxCNC#4669.
@hdiethelm

Copy link
Copy Markdown
Contributor

Thanks for taking this on! And a good idea to use taskset, so I don't have to reboot my VM between compile and testruns.

Would it be worth creating a scripts/runtests_flaky that basically runs taskset -c 0 scripts/runtests -p -j 32 for a few 100 rounds to check for check flaky tests? Probably not for CI but just to run once in a while and might be before a release.

I was not shure if INTERP_IDLE is the right state... I don't know these parts of linuxcnc well (yet).

I pulled your commits in and will run paralell tests for a few hours. Until now, it looks good! Exept one test that went under in the noise, see: #4669 (comment)

@hdiethelm

Copy link
Copy Markdown
Contributor

Caught a new one:

--- expected	2026-10-12 19:59:49.609852828 +0200
+++ result	2026-10-12 21:17:22.596376249 +0200
@@ -1,6 +1,6 @@
 Component Pins:
 Owner   Type  Dir                 Value  Name
-    (ignored comp id)  bool  IN                  FALSE  conv-bool-uint.0.in <== net-conv-bool-uint.0.in
+     (ignored comp id)  bool  IN                  FALSE  conv-bool-uint.0.in <== net-conv-bool-uint.0.in
     (ignored comp id)  real  IN                      0  conv-real-sint.0.in <== net-conv-real-sint.0.in
     (ignored comp id)  sint  IN                      0  conv-sint-real.0.in <== net-conv-sint-real.0.in
     (ignored comp id)  sint  IN                      0  conv-sint-uint.0.in <== net-conv-sint-uint.0.in
@@ -36,7 +36,7 @@
 
 Component Pins:
 Owner   Type  Dir                 Value  Name
-    (ignored comp id)  bool  IN                   TRUE  conv-bool-uint.0.in <== net-conv-bool-uint.0.in
+     (ignored comp id)  bool  IN                   TRUE  conv-bool-uint.0.in <== net-conv-bool-uint.0.in
     (ignored comp id)  real  IN           2.147484e+09  conv-real-sint.0.in <== net-conv-real-sint.0.in
     (ignored comp id)  sint  IN            -2147483648  conv-sint-real.0.in <== net-conv-sint-real.0.in
     (ignored comp id)  sint  IN   -9223372036854775808  conv-sint-uint.0.in <== net-conv-sint-uint.0.in
*** /home/hannes/linuxcnc-src/tests/hal-show: FAIL: checkresult exited with 1

Probably an when the ID changes from 10 to 9.
echo " 10 bool IN FALSE conv-bool-uint.0.in <== net-conv-bool-uint.0.in" | sed --regexp-extended --expression='s/^( *)[0-9]+( +[a-z0-9]+ +(IN|OUT|I\/O) .*)$/\1(ignored comp id)\2/'
vs
echo " 9 bool IN FALSE conv-bool-uint.0.in <== net-conv-bool-uint.0.in" | sed --regexp-extended --expression='s/^( *)[0-9]+( +[a-z0-9]+ +(IN|OUT|I\/O) .*)$/\1(ignored comp id)\2/'

sed --regexp-extended --expression='s/^( *)[0-9]+( +[a-z0-9]+ +(IN|OUT|I\/O) .*)$/(ignored comp id)\2/ not adding the spaces before the (ignored comp id) and updating the expected should fix it. I have to admit, chatgpt was of help, regexp are not compatible with my brain.

The owner column is right-aligned, so an id below 10 gets one more
leading space and still failed the diff. Reported by Hannes Diethelm.

Part of LinuxCNC#4669.
Motion flags a G38.2/G38.4 move that ended without contact one servo
cycle after the planner goes idle. Task could read that cycle as DONE,
finish the MDI command and miss the error, which linuxcncrsh then
returned for the next command. Wait for emcmotStatus.probing to clear.

Part of LinuxCNC#4669.
A 100 deep fifo holds 99 samples, and the 100 line input was preloaded
with a fixed 0.5s sleep, so a late halstreamer left the streamer empty
and shifted the end of the output. Use deeper fifos and loadusr -w.

Part of LinuxCNC#4669.
@grandixximo grandixximo changed the title tests: fix five load-dependent test failures tests, task: fix load-dependent test failures Oct 13, 2026
@grandixximo

Copy link
Copy Markdown
Contributor Author

Pushed three more commits: the hal-show padding fix, multiclick/mux, and a task fix for linuxcncrsh (see description).

Comment thread tests/multiclick/test.hal
Comment on lines +5 to +6
loadrt sampler depth=128 cfg=bbbbbbbbbsbbbbbbbbbs
loadrt streamer depth=128 cfg=bb

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is essentially a bug in the streamer/sampler interface in HAL.

When a buffer is requested of size N, then you should be able to put N elements in it. The bug causes only N-1 elements to be available.
The hal_stream* code should be fixed to allocate a buffer of sufficient size to encapsulate the requested size. But the reported max size should also just be N, regardless whether the allocation is one larger. Therefore, this is not the most trivial of fixes.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. I fixed it in hal_lib: a stream of depth N now allocates N+1 slots, holds N samples and still reports N as maxdepth. histobinstream and latencybinstream had a +1 to work around it, which is gone now. The multiclick and mux streamers are back to depth 100, and with loadusr -w halstreamer the test would hang if a 100 deep stream still held only 99. tests/halmodule/stream asserted the old capacity, so it now fills 10 of 10.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seen the fix. That should do it.

This is such a specific fix that I think it needs to be in its own PR. It isn't a "test-under-load fix", but a bug in HAL stream code.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Split out into #4678. This PR is back to streamer depth 128 for multiclick and mux, so it doesn't depend on that fix.

@BsAtHome

Copy link
Copy Markdown
Contributor

The linuxcncrsh issue, well, it was a probe error that I never was able to find and bothered only when I hit it ;-)

Now that, under load, it has surfaced to be in task, that would be the right place to fix it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants