Skip to content

[BUG] Diagnose and fix current test failures in CI and local #1544

Open
Open
[BUG] Diagnose and fix current test failures in CI and local#1544
Assignees
rohansen856
Labels
bugmaintenance & CIContinuous integration & package distribution

Description

@jgyasu
jgyasu
opened on Dec 23, 2025

Update - 29 Dec, 2025: Now the tests are failing in local as well as on CI as pointed by @fkiraly

While following the testing-your-installation guide, running pytest locally results in multiple test failures.

We should investigate the failing tests and debug them.

Todos after resolving tests:

  • remove the pytest.xfails
  • set sklearn-only: "false" in the include jobs in the test.yml test job

Note: originally it was thought that only local tests fail while main CI passes. This was a confusion that was caused by a state of the repo in which main was no longer used and develop used as the main branch, and for releases. This has since been rectified (main is now the main branch), so remote CI and local failures agree.

Activity

  1. rohansen856 commented on Dec 25, 2025

    @rohansen856
    rohansen856
    on Dec 25, 2025
    Contributor

    I would like to work on this issue. Looking into it now.

  2. PGijsbers commented on Dec 25, 2025

    @PGijsbers
    PGijsbers
    on Dec 25, 2025
    Collaborator

    Quite a few tests rely on connections to the servers (both production and test). In some cases they even rely on server state. This means that it's possible for tests to fail "randomly". I'm on mobile so I can't easily search but I do believe there is already a different issue about making tests rely on connections less by a combination of reducing the surface each individual test tests, and using mocked responses.

  3. rohansen856 commented on Dec 26, 2025

    @rohansen856
    rohansen856
    on Dec 26, 2025
    Contributor

    Did some digging and tried to backtrack the test calls and found this:

    Call Stack:

    openml.tasks.get_task(115)
  → dataset.retrieve_class_labels(task.target_name)
    → self.features.values()
      → self._load_features()
        → _get_dataset_features_file()
          → _get_features_xml(dataset_id=20)
            → OpenMLServerException (code 274)

    The 20th dataset throws error when called from local: https://test.openml.org/api/v1/xml/data/features/20

    Got this when i checked the url myself:

    Image Image

    Although i am still looking as to the root cause of the failures i have these doubts about why it locally fails but CI Passes:**

    • CI configuration in /.github/workflows/test.yml runs tests in two separate steps:
      1. Test server tests (line 106-113): Excludes production marker ->marks='sklearn and not production'
      2. Production server tests (line 114-121): Only runs production marker ->marks='sklearn and production'
    • These failing tests have @pytest.mark.sklearn() but NOT @pytest.mark.production()

    Will update on the matter as i dig more into it!

  4. jgyasu
    changed the title [-][MNT] Local pytest failures despite CI passing on main[/-] [+][MNT] Test failures in CI and local[/+] on Dec 29, 2025
  5. fkiraly
    assigned
    rohansen856
    on Dec 29, 2025

  6. fkiraly commented on Dec 29, 2025

    @fkiraly
    fkiraly
    on Dec 29, 2025
    Contributor

    @EmanAbdelhaleem agreed to add temporary skips in a PR so other PR do not get blocked (or delayed) by the recurring failures. @rohansen856 will investigate and try to debug.

  7. EmanAbdelhaleem commented on Dec 29, 2025

    @EmanAbdelhaleem
    EmanAbdelhaleem
    on Dec 29, 2025
    Contributor

    I am working on marking these failures as xfail temporarily

  8. EmanAbdelhaleem
    added a commit that references this issue on Dec 29, 2025

    mark.xfail for failures in issue openml#1544

    3b995d9
  9. fkiraly
    added a commit that references this issue on Dec 29, 2025

    [BUG] Temporarily fix issue #1544 by marking the failed tests as expe…

    0dd7887

  10. fkiraly commented on Dec 29, 2025

    @fkiraly
    fkiraly
    on Dec 29, 2025
    Contributor

    @jgyasu, we need to investigate exactly which tests are failing when:

    • remote or local, how many and which tests are failing?
    • sklearn-only False vs True flag in CI
  11. fkiraly
    added
    maintenance & CIContinuous integration & package distribution
    on Dec 29, 2025

  12. fkiraly commented on Dec 29, 2025

    @fkiraly
    fkiraly
    on Dec 29, 2025
    Contributor

    Quite a few tests rely on connections to the servers (both production and test). In some cases they even rely on server state. This means that it's possible for tests to fail "randomly". I'm on mobile so I can't easily search but I do believe there is already a different issue about making tests rely on connections less by a combination of reducing the surface each individual test tests, and using mocked responses.

    @PGijsbers, the test failures are not random but consistent, always the same on all the CI steps.

  13. fkiraly commented on Dec 29, 2025

    @fkiraly
    fkiraly
    on Dec 29, 2025
    Contributor

    Failures appear under the following conditions on CI:

    • always: 41 failures
    • sklearn-only is False - 42 additional failures
    • windows operating system, windows-latest - 41 failures

    These should be investigated.

  14. rohansen856 commented on Dec 30, 2025

    @rohansen856
    rohansen856
    on Dec 30, 2025
    Contributor

    So i have been trying to run the tests with all different configurations possible. Attaching all the results below:

    With sklearn and production

    • No test fails, 1 pass
    C:\Users\ASUS\Documents\work\opensource\openml-python>pytest -m "sklearn and production"
======================================= test session starts =======================================
platform win32 -- Python 3.14.0, pytest-9.0.2, pluggy-1.6.0
rootdir: C:\Users\ASUS\Documents\work\opensource\openml-python
configfile: pyproject.toml
testpaths: tests
plugins: anyio-4.12.0, flaky-3.8.1, asyncio-1.3.0, cov-7.0.0, mock-3.15.1, rerunfailures-16.1, timeout-2.4.0, xdist-3.8.0, requests-mock-1.12.1
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 358 items / 354 deselected / 4 selected                                                  

tests\test_flows\test_flow_functions.py sss.                                                 [100%]

========================== 1 passed, 3 skipped, 354 deselected in 2.88s ===========================

    With sklearn and not production

    • 41 fails 12 passes
    ============================================== short test summary info ============================================== 
FAILED tests/test_runs/test_run.py::TestRun::test_offline_and_online_run_identical - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run.py::TestRun::test_publish_with_local_loaded_flow - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run.py::TestRun::test_to_from_filesystem_no_model - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run.py::TestRun::test_to_from_filesystem_search - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run.py::TestRun::test_to_from_filesystem_vanilla - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test__run_exists - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test__run_task_get_arffcontent - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/2 returned code 274: No...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_check_erronous_sklearn_flow_fails - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_delete_run - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/6 returned code 274: No...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_initialize_cv_from_run - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/2 returned code 274: No...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_initialize_model_from_run - Exception: OpenMLServerException('Input value does not match allowed values in foreign column. - problematic inpu...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_learning_curve_task_1 - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_learning_curve_task_2 - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_local_run_metric_score - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/2 returned code 274: No...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_local_run_swapped_parameter_order_flow - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/2 returned code 274: No...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_local_run_swapped_parameter_order_model - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/100 returned code 274: ...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_and_upload_column_transformer_pipeline - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_and_upload_gridsearch - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_and_upload_linear_regression - KeyError: 'oml:error'  
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_and_upload_logistic_regression - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_and_upload_maskedarrays - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_and_upload_pipeline_dummy_pipeline - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_and_upload_randomsearch - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_flow_on_task_downloaded_flow - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_on_dataset_with_missing_labels_array - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/1 returned code 274: No...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_on_dataset_with_missing_labels_dataframe - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/1 returned code 274: No...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_regression_on_classif_task - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/44 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_with_illegal_flow_id - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_with_illegal_flow_id_1 - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_with_illegal_flow_id_1_after_load - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_with_illegal_flow_id_after_load - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::test__run_task_get_arffcontent_2 - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/2 returned code 274: No...
FAILED tests/test_runs/test_run_functions.py::test_joblib_backends[2-None-0] - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/2 returned code 274: No...
FAILED tests/test_runs/test_run_functions.py::test_joblib_backends[-1-None-0] - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/2 returned code 274: No...
FAILED tests/test_runs/test_run_functions.py::test_joblib_backends[1-None-10] - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/2 returned code 274: No...
FAILED tests/test_runs/test_run_functions.py::test_joblib_backends[1-sequential-10] - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/2 returned code 274: No...
FAILED tests/test_runs/test_run_functions.py::test_joblib_backends[1-threading-10] - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/2 returned code 274: No...
FAILED tests/test_runs/test_run_functions.py::test_joblib_backends[-1-threading-10] - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/2 returned code 274: No...
FAILED tests/test_setups/test_setup_functions.py::TestSetupFunctions::test_exisiting_setup_exists_2 - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_setups/test_setup_functions.py::TestSetupFunctions::test_existing_setup_exists_1 - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_setups/test_setup_functions.py::TestSetupFunctions::test_existing_setup_exists_3 - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
============= 41 failed, 12 passed, 3 skipped, 302 deselected, 4 warnings, 1 rerun in 446.43s (0:07:26) =============

    Running all of them together

    ============================================== short test summary info ============================================== 
FAILED tests/test_datasets/test_dataset.py::test_add_remove_ontology_to_dataset - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/feature/ontology/add returned co...
FAILED tests/test_datasets/test_dataset.py::test_add_same_ontology_multiple_features - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/feature/ontology/add returned co...
FAILED tests/test_datasets/test_dataset_functions.py::TestOpenMLDataset::test__get_dataset_features - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/2 returned code 274: No...
FAILED tests/test_datasets/test_dataset_functions.py::TestOpenMLDataset::test__get_dataset_qualities - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/qualities/2 returned code 364: D...
FAILED tests/test_datasets/test_dataset_functions.py::TestOpenMLDataset::test__getarff_path_dataset_arff - KeyError: 'oml:error'
FAILED tests/test_datasets/test_dataset_functions.py::TestOpenMLDataset::test_data_edit_cannot_edit_critical_field_if_dataset_has_task - ValueError: TIMEOUT: Failed to fetch uploaded dataset - 1094
FAILED tests/test_datasets/test_dataset_functions.py::TestOpenMLDataset::test_data_edit_critical_field - ValueError: TIMEOUT: Failed to fetch uploaded dataset - 1095
FAILED tests/test_datasets/test_dataset_functions.py::TestOpenMLDataset::test_get_dataset_by_name - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/1 returned code 274: No...
FAILED tests/test_datasets/test_dataset_functions.py::TestOpenMLDataset::test_get_dataset_cache_format_feather - KeyError: 'oml:error'
FAILED tests/test_datasets/test_dataset_functions.py::TestOpenMLDataset::test_get_dataset_cache_format_pickle - KeyError: 'oml:error'
FAILED tests/test_datasets/test_dataset_functions.py::TestOpenMLDataset::test_get_dataset_lazy_all_functions - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/1 returned code 274: No...
FAILED tests/test_datasets/test_dataset_functions.py::TestOpenMLDataset::test_get_dataset_sparse - KeyError: 'oml:error'
FAILED tests/test_datasets/test_dataset_functions.py::TestOpenMLDataset::test_get_dataset_uint8_dtype - KeyError: 'oml:error'
FAILED tests/test_datasets/test_dataset_functions.py::TestOpenMLDataset::test_get_online_dataset_arff - KeyError: 'oml:error'
FAILED tests/test_datasets/test_dataset_functions.py::TestOpenMLDataset::test_publish_dataset - KeyError: 'oml:error' 
FAILED tests/test_datasets/test_dataset_functions.py::test_list_datasets_by_number_features - assert 8 <= 0
FAILED tests/test_datasets/test_dataset_functions.py::test_list_datasets_by_number_missing_values - assert 5 <= 1     
FAILED tests/test_datasets/test_dataset_functions.py::test_list_datasets_combined_filters - assert 1 <= 0
FAILED tests/test_datasets/test_dataset_functions.py::test_get_dataset_lazy_behavior[True-True-True] - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/1 returned code 274: No...
FAILED tests/test_datasets/test_dataset_functions.py::test_get_dataset_lazy_behavior[True-True-False] - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/qualities/1 returned code 364: D...
FAILED tests/test_datasets/test_dataset_functions.py::test_get_dataset_lazy_behavior[True-False-True] - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/1 returned code 274: No...
FAILED tests/test_datasets/test_dataset_functions.py::test_get_dataset_lazy_behavior[True-False-False] - KeyError: 'oml:error'
FAILED tests/test_datasets/test_dataset_functions.py::test_get_dataset_lazy_behavior[False-True-True] - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/1 returned code 274: No...
FAILED tests/test_datasets/test_dataset_functions.py::test_get_dataset_lazy_behavior[False-True-False] - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/qualities/1 returned code 364: D...
FAILED tests/test_datasets/test_dataset_functions.py::test_get_dataset_lazy_behavior[False-False-True] - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/1 returned code 274: No...
FAILED tests/test_datasets/test_dataset_functions.py::test_get_dataset_lazy_behavior[False-False-False] - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/1 returned code 274: No...
FAILED tests/test_runs/test_run.py::TestRun::test_offline_and_online_run_identical - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run.py::TestRun::test_publish_with_local_loaded_flow - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run.py::TestRun::test_tagging - AssertionError: Test server state is incorrect
FAILED tests/test_runs/test_run.py::TestRun::test_to_from_filesystem_no_model - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run.py::TestRun::test_to_from_filesystem_search - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run.py::TestRun::test_to_from_filesystem_vanilla - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test__run_exists - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test__run_task_get_arffcontent - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/2 returned code 274: No...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_check_erronous_sklearn_flow_fails - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_delete_run - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/6 returned code 274: No...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_format_prediction_classification_incomplete_probabilities - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_format_prediction_classification_no_probabilities - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...        
FAILED tests/test_runs/test_run_functions.py::TestRun::test_format_prediction_task_learning_curve_sample_not_set - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...     
FAILED tests/test_runs/test_run_functions.py::TestRun::test_format_prediction_task_without_classlabels_set - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_initialize_cv_from_run - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/2 returned code 274: No...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_initialize_model_from_run - Exception: OpenMLServerException('Input value does not match allowed values in foreign column. - problematic inpu...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_learning_curve_task_1 - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_learning_curve_task_2 - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_local_run_metric_score - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/2 returned code 274: No...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_local_run_swapped_parameter_order_flow - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/2 returned code 274: No...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_local_run_swapped_parameter_order_model - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/100 returned code 274: ...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_and_upload_column_transformer_pipeline - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_and_upload_gridsearch - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_and_upload_linear_regression - KeyError: 'oml:error'  
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_and_upload_logistic_regression - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_and_upload_maskedarrays - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_and_upload_pipeline_dummy_pipeline - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_and_upload_randomsearch - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_flow_on_task_downloaded_flow - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_on_dataset_with_missing_labels_array - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/1 returned code 274: No...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_on_dataset_with_missing_labels_dataframe - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/1 returned code 274: No...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_regression_on_classif_task - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/44 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_with_illegal_flow_id - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_with_illegal_flow_id_1 - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_with_illegal_flow_id_1_after_load - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::TestRun::test_run_with_illegal_flow_id_after_load - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_runs/test_run_functions.py::test__run_task_get_arffcontent_2 - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/2 returned code 274: No...
FAILED tests/test_runs/test_run_functions.py::test_joblib_backends[2-None-0] - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/2 returned code 274: No...
FAILED tests/test_runs/test_run_functions.py::test_joblib_backends[-1-None-0] - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/2 returned code 274: No...
FAILED tests/test_runs/test_run_functions.py::test_joblib_backends[1-None-10] - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/2 returned code 274: No...
FAILED tests/test_runs/test_run_functions.py::test_joblib_backends[1-sequential-10] - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/2 returned code 274: No...
FAILED tests/test_runs/test_run_functions.py::test_joblib_backends[1-threading-10] - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/2 returned code 274: No...
FAILED tests/test_runs/test_run_functions.py::test_joblib_backends[-1-threading-10] - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/2 returned code 274: No...
FAILED tests/test_setups/test_setup_functions.py::TestSetupFunctions::test_exisiting_setup_exists_2 - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_setups/test_setup_functions.py::TestSetupFunctions::test_existing_setup_exists_1 - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_setups/test_setup_functions.py::TestSetupFunctions::test_existing_setup_exists_3 - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_setups/test_setup_functions.py::TestSetupFunctions::test_setuplist_offset - assert 0 == 10
FAILED tests/test_study/test_study_functions.py::TestStudyFunctions::test_publish_study - assert 0 == 10
FAILED tests/test_study/test_study_functions.py::TestStudyFunctions::test_study_attach_illegal - assert 0 == 10       
FAILED tests/test_tasks/test_classification_task.py::OpenMLClassificationTaskTest::test_class_labels - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_tasks/test_classification_task.py::OpenMLClassificationTaskTest::test_download_task - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_tasks/test_classification_task.py::OpenMLClassificationTaskTest::test_get_X_and_Y - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_tasks/test_classification_task.py::test_get_X_and_Y - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_tasks/test_learning_curve_task.py::OpenMLLearningCurveTaskTest::test_class_labels - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_tasks/test_learning_curve_task.py::OpenMLLearningCurveTaskTest::test_download_task - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_tasks/test_learning_curve_task.py::OpenMLLearningCurveTaskTest::test_get_X_and_Y - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/20 returned code 274: N...
FAILED tests/test_tasks/test_regression_task.py::OpenMLRegressionTaskTest::test_get_X_and_Y - KeyError: 'oml:error'   
FAILED tests/test_tasks/test_task_functions.py::TestTask::test_download_split - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/1 returned code 274: No...
FAILED tests/test_tasks/test_task_functions.py::TestTask::test_get_task - KeyError: 'oml:error'
FAILED tests/test_tasks/test_task_functions.py::TestTask::test_get_task_lazy - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/1 returned code 274: No...
FAILED tests/test_tasks/test_task_functions.py::TestTask::test_get_task_with_cache - openml.exceptions.OpenMLServerException: https://test.openml.org/api/v1/xml/data/features/1 returned code 274: No...
ERROR tests/test_tasks/test_task_functions.py::TestTask::test_get_task_with_cache - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
ERROR tests/test_tasks/test_task_functions.py::TestTask::test_list_clustering_task - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
ERROR tests/test_tasks/test_task_functions.py::TestTask::test_list_tasks - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
ERROR tests/test_tasks/test_task_functions.py::TestTask::test_list_tasks_by_tag - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
ERROR tests/test_tasks/test_task_functions.py::TestTask::test_list_tasks_by_type - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
ERROR tests/test_tasks/test_task_functions.py::TestTask::test_list_tasks_empty - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
ERROR tests/test_tasks/test_task_functions.py::TestTask::test_list_tasks_length - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
ERROR tests/test_tasks/test_task_functions.py::TestTask::test_list_tasks_paginate - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
ERROR tests/test_tasks/test_task_functions.py::TestTask::test_list_tasks_per_type_paginate - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
ERROR tests/test_tasks/test_task_functions.py::TestTask::test_removal_upon_download_failure - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
ERROR tests/test_tasks/test_task_functions.py::test_delete_task_not_owned - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
ERROR tests/test_tasks/test_task_functions.py::test_delete_task_with_run - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
ERROR tests/test_tasks/test_task_functions.py::test_delete_success - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
ERROR tests/test_tasks/test_task_functions.py::test_delete_unknown_task - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
ERROR tests/test_tasks/test_task_methods.py::OpenMLTaskMethodsTest::test_get_train_and_test_split_indices - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
ERROR tests/test_tasks/test_task_methods.py::OpenMLTaskMethodsTest::test_tagging - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
ERROR tests/test_utils/test_utils.py::test_list_all - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
ERROR tests/test_utils/test_utils.py::test_list_all_for_tasks - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
ERROR tests/test_utils/test_utils.py::test_list_all_with_multiple_batches - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
ERROR tests/test_utils/test_utils.py::test_list_all_for_datasets - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
ERROR tests/test_utils/test_utils.py::test_list_all_for_flows - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
ERROR tests/test_utils/test_utils.py::test_list_all_for_setups - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
ERROR tests/test_utils/test_utils.py::test_list_all_for_runs - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
ERROR tests/test_utils/test_utils.py::test_list_all_for_evaluations - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
ERROR tests/test_utils/test_utils.py::test_list_all_few_results_available - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
ERROR tests/test_utils/test_utils.py::test__create_cache_directory - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
ERROR tests/test_utils/test_utils.py::test_correct_test_server_download_state - AssertionError: Expected path C:\Users\ASUS\Documents\work\opensource\openml-python\tests\files\org\openml\test\t...
============= 87 failed, 229 passed, 16 skipped, 23 warnings, 27 errors, 5 rerun in 3462.30s (0:57:42) ==============

  15. rohansen856 commented on Dec 30, 2025

    @rohansen856
    rohansen856
    on Dec 30, 2025
    Contributor

    now, upon further investigation, the sklear and production and sklearn and not production is run separately on the CI test.yaml:

    # Ubuntu Test Server (line 106-113)
- name: Run tests on Ubuntu Test
  if: matrix.os == 'ubuntu-latest'
  run: |
    if [ ${{ matrix.sklearn-only }} = 'true' ]; then 
      marks='sklearn and not production'
    else 
      marks='not production'
    fi
    pytest -n 4 --durations=20 --dist load -sv -m "$marks"

# Ubuntu Production Server (line 114-121)
- name: Run tests on Ubuntu Production
  if: matrix.os == 'ubuntu-latest'
  run: |
    if [ ${{ matrix.sklearn-only }} = 'true' ]; then 
      marks='sklearn and production'
    else 
      marks='production'
    fi
    pytest -n 4 --durations=20 --dist load -sv -m "$marks"

    now, what i am trying to do next is running the tests locally inside a CI like environment using a open source tool act (https://github.com/nektos/act/releases)

  16. PGijsbers commented on Dec 31, 2025

    @PGijsbers
    PGijsbers
    on Dec 31, 2025
    Collaborator

    The thing that strikes me as odd is that it worked 3 weeks ago (#1502 - the PR merges to "develop" but that's the same branch as the current "main" after the rename). That's why I had assumed they were sporadic failures. Thanks for the investigation.

    A quick scan of the failing test tells me this is related to test server state (at least a large swath of them). I don't know what happened, I am asking around with the other core contributors that would have had access. I'll try to see if I can fix this on the test server side. It's possible a few additional tests may fail in the meantime.

  17. fkiraly
    added 2 commits that reference this issue on Dec 31, 2025

    [MNT] Update xfail for new test server state (#1585)

    6043686

    [MNT] extend CI to newer python versions, deprecate python versions 3…

    bd8ae77
  18. fkiraly
    added
    bug
    on Jan 5, 2026
  19. fkiraly
    changed the title [-][MNT] Test failures in CI and local[/-] [+][BUG] Diagnose and fix current test failures in CI and local[/+] on Jan 5, 2026

  20. rohansen856 commented on Jan 6, 2026

    @rohansen856
    rohansen856
    on Jan 6, 2026
    Contributor

    So, currently failing tests are marked with xfail to make them pass temporarily by @EmanAbdelhaleem in Pr #1572
    My approach is first discarding the commit bt Eman through a rebase then picking upp the tests and investigating why they are failing (git rebase -i 3b995d9^). I have setup both the php and fastapi backend locally and using those i think most of the complexities regarding the errors returned from test.openml.org would be easier to resolve.
    @fkiraly please do inform me if you think this approach has any problem...

  21. fkiraly commented on Jan 9, 2026

    @fkiraly
    fkiraly
    on Jan 9, 2026
    Contributor

    @PGijsbers, I think it is due to race conditions when you run multiple test runs in parallel, or aborted test runs. See #1586

  22. EmanAbdelhaleem commented on Jan 12, 2026

    @EmanAbdelhaleem
    EmanAbdelhaleem
    on Jan 12, 2026
    Contributor

    This is a list of the most repeated failed tests in the failed jobs:

    tests/test_flows/test_flow.py::TestFlow::test_publish_flow_with_similar_components
tests/test_setups/test_setup_functions.py::TestSetupFunctions::test_exisiting_setup_exists_2
tests/test_flows/test_flow.py::TestFlow::test_existing_flow_exists

tests/test_runs/test_run_functions.py::TestRun::test__run_exists
tests/test_runs/test_run.py::TestRun::test_publish_with_local_loaded_flow
tests/test_runs/test_run_functions.py::TestRun::test_learning_curve_task_1
tests/test_setups/test_setup_functions.py::TestSetupFunctions::test_existing_setup_exists_3

tests/test_flows/test_flow_functions.py::TestFlowFunctions::test_get_flow_id
tests/test_runs/test_run_functions.py::TestRun::test_run_flow_on_task_downloaded_flow
tests/test_runs/test_run_functions.py::TestRun::test_delete_run

tests/test_setups/test_setup_functions.py::TestSetupFunctions::test_nonexisting_setup_exists
tests/test_setups/test_setup_functions.py::TestSetupFunctions::test_existing_setup_exists_1
tests/test_runs/test_run_functions.py::TestRun::test_initialize_model_from_run

tests/test_runs/test_run.py::TestRun::test_to_from_filesystem_vanilla
tests/test_flows/test_flow.py::TestFlow::test_semi_legal_flow
tests/test_runs/test_run_functions.py::TestRun::test_initialize_cv_from_run
tests/test_runs/test_run_functions.py::TestRun::test_learning_curve_task_2
tests/test_datasets/test_dataset_functions.py::TestOpenMLDataset::test_publish_fetch_ignore_attribute```

  23. fkiraly commented on Jan 12, 2026

    @fkiraly
    fkiraly
    on Jan 12, 2026
    Contributor

    @EmanAbdelhaleem, can you please always state the condition under which tests are failing? local, remote, what version/OS if that is relevant

  24. EmanAbdelhaleem commented on Jan 12, 2026

    @EmanAbdelhaleem
    EmanAbdelhaleem
    on Jan 12, 2026
    Contributor

    @EmanAbdelhaleem, can you please always state the condition under which tests are failing? local, remote, what version/OS if that is relevant

    That was a list of the most frequent failed tests in the failed remote jobs. I skipped all of them and more (less frequent failed tests) back then in my previous merged PR #1587 . I don't have more details about these tests, they were from many different jobs.

  25. geetu040 commented on Jan 12, 2026

    @geetu040
    geetu040
    on Jan 12, 2026
    Contributor

    @EmanAbdelhaleem, can you please always state the condition under which tests are failing? local, remote, what version/OS if that is relevant

    From the above shared list, all these stated tests are marked with uses_test_server so I believe they must have been run and failed locally and skipped on remote.

  26. EmanAbdelhaleem commented on Jan 13, 2026

    @EmanAbdelhaleem
    EmanAbdelhaleem
    on Jan 13, 2026
    Contributor

    @EmanAbdelhaleem, can you please always state the condition under which tests are failing? local, remote, what version/OS if that is relevant

    From the above shared list, all these stated tests are marked with uses_test_server so I believe they must have been run and failed locally and skipped on remote.

    This is an old list of test failures, the one I sent you on discord before the uses_test_server mark. Dr. Franz was asking if we had a similar list this morning. That's why I mentioned it.

  27. satvshr commented on Jan 31, 2026

    @satvshr
    satvshr
    on Jan 31, 2026
    Contributor

    Got a sporadic error here because the error no longer occurred after I made a dummy (empty) commit. Commenting here under the assumption that this is the issue tracking sporadic errors.

  28. PGijsbers
    added a commit that references this issue on Oct 20, 2026

    [MAINT] Remove skip markers because tests work again (#1781)

    c8e2aaf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugmaintenance & CIContinuous integration & package distribution

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions