Problem
The web integration suite reports All tests passed regardless of what the tests do. A deliberate expect(1, 2) placed inside Timebased playback audio starts without a reader view DiViNa comic produced a clean pass and exit code 0.
The driver's verdict comes from IntegrationTestWidgetsFlutterBinding:
List<Failure> get failureMethodsDetails => results.values.whereType<Failure>().toList();
...
_allTestsPassed.complete(failureMethodsDetails.isEmpty);
results is only ever written from two places, and both are reached only by testWidgets:
results[description] ??= _success; in runTestresults[testDescription] = Failure(...) via reportTestException
integration_test/groups/ is written almost entirely with plain test(). Those never touch results, so on web nothing is recorded — and an empty result map is indistinguishable from a perfect run. Native is unaffected: flutter test uses the normal runner and never consults that map.
Consequence: web integration coverage is currently decorative. A test that stops running, or fails, looks exactly like a test that passes.
Fix
Convert the plain test() calls under flutter_readium/example/integration_test/groups/ to testWidgets() so results are reported. Roughly 47 tests across 8 files; mechanical, no logic changes.
Worth adding alongside it: a self-check that fails the run when the reported result count is zero, so this class of hollow pass cannot come back silently.
Related
.github/workflows/integration-test-web.yml is on: workflow_call with no caller anywhere in the repo, and is marked continue-on-error: true # allowed to fail for now. The suite has never run in CI, which is why this went unnoticed. Wiring it up is only worthwhile once results are real.- Separately fixed: the runner used
-d chrome, which hangs indefinitely when the developer already has Chrome open. bin/integration_test_web now uses -d web-server.
Problem
The web integration suite reports
All tests passedregardless of what the tests do. A deliberateexpect(1, 2)placed insideTimebased playback audio starts without a reader view DiViNa comicproduced a clean pass and exit code 0.The driver's verdict comes from
IntegrationTestWidgetsFlutterBinding:resultsis only ever written from two places, and both are reached only bytestWidgets:results[description] ??= _success;inrunTestresults[testDescription] = Failure(...)viareportTestExceptionintegration_test/groups/is written almost entirely with plaintest(). Those never touchresults, so on web nothing is recorded — and an empty result map is indistinguishable from a perfect run. Native is unaffected:flutter testuses the normal runner and never consults that map.Consequence: web integration coverage is currently decorative. A test that stops running, or fails, looks exactly like a test that passes.
Fix
Convert the plain
test()calls underflutter_readium/example/integration_test/groups/totestWidgets()so results are reported. Roughly 47 tests across 8 files; mechanical, no logic changes.Worth adding alongside it: a self-check that fails the run when the reported result count is zero, so this class of hollow pass cannot come back silently.
Related
.github/workflows/integration-test-web.ymlison: workflow_callwith no caller anywhere in the repo, and is markedcontinue-on-error: true # allowed to fail for now. The suite has never run in CI, which is why this went unnoticed. Wiring it up is only worthwhile once results are real.-d chrome, which hangs indefinitely when the developer already has Chrome open.bin/integration_test_webnow uses-d web-server.