CI is flaky after test modernization #747

Activity

1. amotl

   changed the title [-]Test flakyness after modernization[/-] [+]CI is flaky after test modernization[/+] on Dec 9, 2025
2. surister commented on Dec 10, 2025

   

   surister

   on Dec 10, 2025

   Contributor

   More actions

   I don't think it's related, I have seen in the past these flaky errors where the test layer errors because the array result might have a different order.

   See flaky errors before test modernization in nightly - main:

   • https://github.com/crate/crate-python/actions/runs/18862248255/job/53822730843
   • https://github.com/crate/crate-python/actions/runs/18394732908/job/52411734583
   • https://github.com/crate/crate-python/actions/runs/18394732908/job/52411734583
   • https://github.com/crate/crate-python/actions/runs/18026176366/job/51293519885

   Additionally, the test_layer is was not modified in the PR, the only difference is that tests are now being run with pytest instead of zope test runner, my hypothesis is that either:

   • The tests are not correctly written to take into account the driver's multi-threading behavior.
   • There is a subtle bug in the multi-threading part of the code.

   Having the right result but different order in multi-threading python code is common imo.
3. amotl commented on Dec 10, 2025

   

   amotl

   on Dec 10, 2025

   Last edited by amotl

   ContributorAuthor

   More actions

   Yes I agree. I hadn't seen the test_additional_settings fluke yet, but the invalid_host vs. even_more_invalid_host error is a known fluke:

   • Tests: Flakyness with "active servers" logic within test suite #404

   In this case, the test case modernizations made the error more apparent, probably by introducing different timings. Do you see a chance to fix it positively?
4. amotl commented on Dec 10, 2025

   

   amotl

   on Dec 10, 2025

   Last edited by amotl

   ContributorAuthor

   More actions

   I've re-opened GH-404, and transferred the gist of your evaluation, thanks! In this spirit, I think it's fine to close this issue again.
5. amotl

   closed this as not plannedon Dec 10, 2025