Skip to content

fix(test): keep a call's own on error clause; the generated startup flow applies entity access - #1748

Merged
ako merged 2 commits into
mainfrom
fix/1448-test-runner-generated-flows
Oct 27, 2026
Merged

ako merged 2 commits into
mainfrom
fix/1448-test-runner-generated-flows

Conversation

@ako

@ako ako commented Oct 27, 2026

Copy link
Copy Markdown
Owner

Closes #1448 (mirror of mendixlabs#1550). Closes #1654 (mirror of mendixlabs#1645).

mxcli test generated a flow that could not be built, so no test ran. One commit per issue. Both verified with mxcli test --local --db-type hsqldb on a blank 11.13.0 app.

#1448 — a call with its own on error clause got a second one

Cause: attachOnError (generator_endpoint.go) appended ON ERROR BEGIN … END ERROR to every line a regex matched, with no check for an existing clause; the legacy runner had two more copies of the same join, which also broke on a custom handler's own ; and on a trailing -- comment.

Fix:

  • One parser-driven attachOnError for both runners, with and without @throws. New visitor.FlowStatements parses the body inside the generator's own wrapper and language version and returns each statement's span and its own ON ERROR clause; a body that does not parse is left unchanged. The regex is replaced by errorRaisingCallRules, a set of grammar rule names pinned by a test.
  • on error continue and custom handlers are kept as written. on error rollback is replaced by the runner's handler: it is Mendix's default spelled out (the same ErrorHandlingType as no clause), and keeping it gave ERROR instead of FAIL, failed a matching @throws, and under the legacy runner aborted the suite.
  • A false green found at runtime: a custom handler that ends normally rejoins the flow past every statement reading the failed call's output — past the @expect checks — and reported PASS with nothing evaluated. Each assertion now counts itself in both branches, and a guard fails the test with "not every assertion was evaluated" when the count is short (both runners).
  • Legacy log parser: a RUN line with no verdict read as PASS (the zero status); it is now an ERROR saying the test started and gave no verdict.

Verdict table (Java action that throws, endpoint runner):

Clause @expect @throws 'boom'
none (control) FAIL exception during execution: boom: a PASS
on error rollback FAIL, same as none PASS
on error continue FAIL expected $r = 'X', actual: FAIL expected an exception but none was thrown
custom handler, ends normally FAIL not every assertion was evaluated (was PASS) FAIL expected … none was thrown
custom handler, re-raises ERROR boom: a ERROR boom: a

The issue's statements with succeeding calls PASS; an @expect on a flag the handler sets still PASSes; --legacy-runner gives the same verdicts.

#1654 — CE0114 on the generated startup flow

Measured with mx check 11.13.0, security Off and Production: a non-applying microflow calling an applying one is CE0114; an applying one calling either kind is 0 errors (MDL-SEC22 records the same for 11.14.0).
Cause: GenerateEndpointMDL wrote MxTest.RegisterEndpoint without the setting, and it calls the project's after-startup microflow. Separately, buildFailureHint called every error outside a Test_* flow "in the project".
Fix: RegisterEndpoint always carries @applyentityaccess — the measured direction makes a constant correct for every project, with nothing to read from the user's microflow. Errors located in mxcli's generated MxTest flows are reported as mxcli's own, with --skip-app-startup as the escape hatch.

Verified

Noticed, not fixed

  1. Legacy runner false green (Docker default and --legacy-runner): a later assertion's PASS line overwrites an earlier FAIL (results.go case "PASS"), so a test whose first @expect fails and second passes reports PASS. Confirmed at unit level.
  2. Legacy runner: assignPattern matches only upper-case CALL|CREATE, so mdl 1's $r = call … is not renamed per test; two tests using $r fail injection (MDL063 / CE0111, measured).
  3. Endpoint @throws false red: an error that leaves the test microflow (a re-raising handler, or a non-call activity) is reported ERROR instead of being checked against @throws.
  4. run --local --test-endpoint does not use buildFailureHint, so it lacks the new attribution.

🤖 Generated with Claude Code

ako and others added 2 commits October 27, 2026 15:54
…or clause (#1448)

A call in a test body that already had an error-handling clause (`on error
rollback`, `on error continue`, a custom handler) got the runner's
`ON ERROR BEGIN … END ERROR` appended as a second one. The generated MDL did
not parse ("mismatched input 'ON' expecting ';'", plus a misleading hint about
`Error` being a keyword) and the run stopped at injection, before the app
started. Upstream: mendixlabs#1550.

Cause: attachOnError (endpoint runner) and its two copies in the legacy
after-startup runner wrapped every line startsErrorRaisingCall matched,
unconditionally. Their line-based join also broke on a custom handler (the
handler body's first `;` ended the "statement") and on a trailing comment.

Fix:
- One attachOnError for both runners, with and without @throws. It parses the
  body inside the generator's own wrapper and language version
  (visitor.FlowStatements, new) and reads each call's own onErrorClause from
  the tree — not text, so a clause on a later line counts and `on error` in a
  string, a comment or a nested handler does not. The set of wrapped
  statements is now a set of grammar rule names, pinned to the grammar.
- A call with `on error continue` or a custom handler keeps it verbatim and
  gets no second handler. `on error rollback` is Mendix's default spelled out
  (the same ErrorHandlingType as no clause), so it is replaced by the runner's
  handler, which also rolls back: same model, same verdict.
- Keeping custom handlers opened a false green that only the runtime showed:
  a handler that ends normally rejoins the flow past every statement reading
  the failed call's output — the @expect checks — and the test returned PASS
  with nothing evaluated. Each assertion now counts itself in both branches of
  its decision, and a guard after the last fails a short count ("not every
  assertion was evaluated"). Both runners.
- Legacy log parser: a test with a RUN line and no verdict read as PASS
  (StatusPass is the zero value). A re-raising handler ends the legacy runner
  exactly there; it is now an ERROR that says so.

Verification (Mendix 11.13.0, --local --db-type hsqldb, the issue's setup):
the baseline binary stops with the reported parse error; the fixed one runs
mdl-examples/bug-tests/test-runner-1448-own-error-handler.test.mdl:
plain/rollback throwing FAIL "exception during execution: boom: a"; continue
FAIL on its assertion; custom-handler-ends-normally FAIL "not every assertion
was evaluated" (PASS before the guard); re-raise ERROR "boom: a"; @throws
plain/rollback PASS, continue/custom FAIL "expected an exception but none was
thrown", re-raise ERROR; the report's two statements PASS; an assertion on
what the handler set PASS. --legacy-runner gives the same verdicts; its
re-raise case was PASS and is now ERROR. Revert proofs (stubbing each guard):
the always-attach stub reproduces "mismatched input 'ON'" in the unit tests;
the rollback, counter and RUN-without-verdict stubs each fail their test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…roflow applies entity access (#1654)

`mxcli test --local` generates MxTest.RegisterEndpoint, the startup microflow
that registers the test endpoint and then calls the project's own after-startup
microflow. It was written without "apply entity access", and MxBuild refuses a
microflow that does not apply entity access calling one that does (CE0114). In
any project whose after-startup microflow applies entity access the build
failed, no test ran, and the error said "These errors are in the project, not
in this run's tests" about a microflow mxcli generated. Upstream:
mendixlabs#1645.

Measured direction (mx check, Mendix 11.13.0, security Off and Production):
non-applying -> applying is CE0114; applying -> non-applying and
applying -> applying (with a Java action call beside it) are 0 errors, and an
applying microflow is accepted as the after-startup microflow. MDL-SEC22
records the same table for 11.14.0.

Fix:
- GenerateEndpointMDL writes `@applyentityaccess` on RegisterEndpoint. Since
  an applying caller may call either kind, a constant serves every project
  with nothing read from the user's microflow (the suggested mirror would have
  added a second scrape of describe output). It governs only the flow's own
  activities: the Java action ignores it, the chained microflow runs with its
  own setting. Also covers `run --local --test-endpoint`, which shares the
  generator.
- buildFailureHint attributes an error located in MxTest.RegisterEndpoint or
  MxTest.RegisterTestEndpoint to mxcli — "not part of your project: mxcli
  generated it for this run … a defect in mxcli … please report it" — with
  --skip-app-startup as the escape hatch, and says "in the project" only for
  the errors that are.

Verification: on a blank 11.13.0 app with
mdl-examples/bug-tests/test-runner-1654-setup.mdl (startup microflow applies
entity access and leaves a marker), the baseline binary fails with the
reported CE0114; the fixed one passes both tests, the second proving the
startup microflow ran at boot. Controls: the same run with --skip-app-startup
fails that test ("expected $Count > 0, actual: 0") — the test consumes the
marker so an earlier boot's cannot pass it; a non-applying startup microflow
still passes both. A binary with only the attribution half shows the new
message on the live CE0114. Revert proof (patch method): with endpoint.go and
build_attribution.go checked out, TestRegisterEndpointAppliesEntityAccess and
TestBuildErrorInGeneratedStartupFlowIsMxclis fail; the project-only control
passes either way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ako
ako merged commit 1b8bbc7 into main Oct 27, 2026
33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant