Set up end-to-end pytest with CI and requirement traceability

[AlexanderLanin]

Why

Make end-to-end pytest results available in CI and the requirement verification report, providing the foundation for migrating the existing shell assertions in a separate PR.

Changes

• Add a local Bazel pytest target and helpers that run against the checked-in consumer workspace. Reuse its existing output base and the host download cache so toolchains remain unpacked between runs.
• Add one CLI case verifying the missing-report diagnostic and exit code 2, with requirement metadata. Delete any existing generated report before running the check.
• Run pytest alongside the existing shell pipeline, retain the coverage artifact, and combine end-to-end and unit-test JUnit results for the documentation build.

[github-actions]

License Check Results

🚀 The license check job ran with the Bazel command:

bazel run //:license-check

Status: ⚠️ Needs Review

Click to expand output

[License Check Output]
2026/10/09 15:56:36 Downloading https://releases.bazel.build/8.6.0/release/bazel-8.6.0-linux-x86_64...
Extracting Bazel installation...
Starting local Bazel server (8.6.0) and connecting to it...
INFO: Invocation ID: 4c1187cb-da31-4eb2-b73f-2c8655506abc
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Loading: 
Loading: 0 packages loaded
Loading: 0 packages loaded
Analyzing: target //:license-check (1 packages loaded, 0 targets configured)
Analyzing: target //:license-check (1 packages loaded, 0 targets configured)

Analyzing: target //:license-check (5 packages loaded, 6 targets configured)

Analyzing: target //:license-check (5 packages loaded, 6 targets configured)

Analyzing: target //:license-check (5 packages loaded, 6 targets configured)

Analyzing: target //:license-check (5 packages loaded, 6 targets configured)

Analyzing: target //:license-check (5 packages loaded, 6 targets configured)

Analyzing: target //:license-check (5 packages loaded, 6 targets configured)

Analyzing: target //:license-check (64 packages loaded, 10 targets configured)

Analyzing: target //:license-check (80 packages loaded, 10 targets configured)

Analyzing: target //:license-check (140 packages loaded, 329 targets configured)

Analyzing: target //:license-check (151 packages loaded, 3277 targets configured)

Analyzing: target //:license-check (151 packages loaded, 3277 targets configured)

Analyzing: target //:license-check (151 packages loaded, 3277 targets configured)

Analyzing: target //:license-check (162 packages loaded, 9758 targets configured)

Analyzing: target //:license-check (166 packages loaded, 9773 targets configured)

Analyzing: target //:license-check (169 packages loaded, 9794 targets configured)

Analyzing: target //:license-check (176 packages loaded, 11712 targets configured)

Analyzing: target //:license-check (176 packages loaded, 11712 targets configured)

Analyzing: target //:license-check (177 packages loaded, 11828 targets configured)

Analyzing: target //:license-check (177 packages loaded, 11828 targets configured)

Analyzing: target //:license-check (178 packages loaded, 11949 targets configured)

INFO: Analyzed target //:license-check (179 packages loaded, 12076 targets configured).
[12 / 16] JavaToolchainCompileClasses external/rules_java+/toolchains/platformclasspath_classes; 0s disk-cache, processwrapper-sandbox ... (2 actions running)
[15 / 16] Building tools/license.check.license_check.jar (); 0s disk-cache, multiplex-worker
INFO: Found 1 target...
Target //tools:license.check.license_check up-to-date:
  bazel-bin/tools/license.check.license_check
  bazel-bin/tools/license.check.license_check.jar
INFO: Elapsed time: 33.824s, Critical Path: 1.62s
INFO: 16 processes: 12 internal, 3 processwrapper-sandbox, 1 worker.
INFO: Build completed successfully, 16 total actions
INFO: Running command line: bazel-bin/tools/license.check.license_check tools/formatted.txt <args omitted>
usage: org.eclipse.dash.licenses.cli.Main [-batch <int>] [-cd <url>]
       [-confidence <int>] [-ef <url>] [-excludeSources <sources>] [-help] [-lic
       <url>] [-project <shortname>] [-repo <url>] [-review] [-summary <file>]
       [-timeout <seconds>] [-token <token>]

[github-actions]

Documentation preview for this pull request is available at:
pr-33: https://eclipse-score.github.io/coverage_tool/pr-33/

[Copilot]

🟢 Approval recommended

The infrastructure, test behavior, CI report aggregation, and documentation integration are consistent and complete.

0 open findings

What changed in this PR

Adds traceable end-to-end pytest infrastructure for validating public CLI behavior and incorporating results into documentation.

Changes:

• Adds an isolated Bazel-backed pytest fixture and CLI scenario.
• Collects and combines unit and end-to-end JUnit reports in CI.
• Documents the new test target and traceability metadata.

File	Description
tests/​end_to_end/​testcases/​test_cli.py	Tests missing-report CLI behavior.
tests/​end_to_end/​testcases/​conftest.py	Creates an isolated consumer workspace.
tests/​end_to_end/​testcases/​BUILD	Defines the Bazel pytest target.
tests/​end_to_end/​testcases/​_blackbox_support.py	Provides Bazel and traceability helpers.
README.md	Documents the new test command.
docs/​verification/​verification_report.rst	Adds the test to verification documentation.
docs/​BUILD	Includes end-to-end JUnit results.
.github/​workflows/​tests.yml	Runs the test and combines reports.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟢 Approval recommended

The implementation is coherent, with only a minor documentation terminology correction identified.

0 open findings

Previously missed (1)

In code that hasn't changed since last review

Name the @​verifies decorator instead of the score_pytest rule

docs/​verification/​verification_report.rst:103

This names the wrong mechanism: score_pytest is the Bazel rule, while the function metadata is attached by the custom @verifies decorator in tests/end_to_end/testcases/_blackbox_support.py. Naming the actual decorator keeps the traceability documentation actionable and consistent with the code.

🧠 Review effort: Balanced

[Copilot]

🟢 Approval recommended

The implementation and CI integration are sound, with only a minor documentation terminology correction remaining.

1 open finding

• Correct test decorator reference from score_pytest to verifies · New

🧠 Review effort: Balanced

[Copilot]

🟢 Approval recommended

The implementation is coherent and tested, with only a minor documentation terminology correction outstanding.

1 open finding

• Correct test decorator reference from score_pytest to verifies

🧠 Review effort: Balanced

[MaximilianSoerenPollak]

Talked about in a call. Looks good (after changes)