Set up traceable pytest black-box checks

[AlexanderLanin]

Introduce requirement-linked pytest results for consumer-facing coverage checks so the shell scenarios can move to named, traceable test cases incrementally.

This first step adds a Bazel pytest target, an isolated consumer-workspace fixture, CI JUnit collection and merging, and documentation ingestion. One small scenario runs the public report command with a non-numeric COVERAGE_THRESHOLD and requires exit 2 plus the explicit configuration error. The target always runs because its consumer checkout and host Bazel installation are runtime inputs.

The existing shell suite continues to run alongside the new pytest case.

[github-actions]

License Check Results

🚀 The license check job ran with the Bazel command:

bazel run //:license-check

Status: ⚠️ Needs Review

Click to expand output

[License Check Output]
Extracting Bazel installation...
Starting local Bazel server (8.6.0) and connecting to it...
INFO: Invocation ID: ae91d694-2989-4f03-9d1c-3abd4c5fbd0d
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Loading: 
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Analyzing: target //:license-check (1 packages loaded, 0 targets configured)
Analyzing: target //:license-check (1 packages loaded, 0 targets configured)

Analyzing: target //:license-check (5 packages loaded, 6 targets configured)

Analyzing: target //:license-check (5 packages loaded, 6 targets configured)

Analyzing: target //:license-check (5 packages loaded, 6 targets configured)

Analyzing: target //:license-check (5 packages loaded, 6 targets configured)

Analyzing: target //:license-check (5 packages loaded, 6 targets configured)

Analyzing: target //:license-check (5 packages loaded, 6 targets configured)

Analyzing: target //:license-check (5 packages loaded, 6 targets configured)

Analyzing: target //:license-check (5 packages loaded, 6 targets configured)

Analyzing: target //:license-check (5 packages loaded, 6 targets configured)

Analyzing: target //:license-check (25 packages loaded, 10 targets configured)

Analyzing: target //:license-check (89 packages loaded, 10 targets configured)

Analyzing: target //:license-check (132 packages loaded, 484 targets configured)

Analyzing: target //:license-check (151 packages loaded, 3291 targets configured)

Analyzing: target //:license-check (151 packages loaded, 3291 targets configured)

Analyzing: target //:license-check (151 packages loaded, 3291 targets configured)

Analyzing: target //:license-check (161 packages loaded, 6952 targets configured)

Analyzing: target //:license-check (163 packages loaded, 9808 targets configured)

Analyzing: target //:license-check (164 packages loaded, 9821 targets configured)

Analyzing: target //:license-check (164 packages loaded, 9821 targets configured)

Analyzing: target //:license-check (171 packages loaded, 9963 targets configured)

Analyzing: target //:license-check (178 packages loaded, 11994 targets configured)

INFO: Analyzed target //:license-check (179 packages loaded, 12118 targets configured).
[13 / 16] JavaToolchainCompileClasses external/rules_java+/toolchains/platformclasspath_classes; 0s disk-cache, processwrapper-sandbox
[14 / 16] JavaToolchainCompileBootClasspath external/rules_java+/toolchains/platformclasspath.jar; 0s disk-cache, processwrapper-sandbox
INFO: Found 1 target...
Target //tools:license.check.license_check up-to-date:
  bazel-bin/tools/license.check.license_check
  bazel-bin/tools/license.check.license_check.jar
INFO: Elapsed time: 36.324s, Critical Path: 2.70s
INFO: 16 processes: 12 internal, 3 processwrapper-sandbox, 1 worker.
INFO: Build completed successfully, 16 total actions
INFO: Running command line: bazel-bin/tools/license.check.license_check tools/formatted.txt <args omitted>
usage: org.eclipse.dash.licenses.cli.Main [-batch <int>] [-cd <url>]
       [-confidence <int>] [-ef <url>] [-excludeSources <sources>] [-help] [-lic
       <url>] [-project <shortname>] [-repo <url>] [-review] [-summary <file>]
       [-timeout <seconds>] [-token <token>]

[Copilot]

🟡 Changes recommended

The Ruff exclusion no longer excludes consumer fixtures, and the new scenario links to the wrong requirement.

2 open findings

• Use project-relative Ruff exclusion for consumer fixtures · New
• Correct coverage gate requirement metadata mapping · New

What changed in this PR

Introduces requirement-linked pytest consumer checks alongside the existing shell integration suite.

Changes:

• Adds an isolated consumer fixture and an invalid-threshold scenario.
• Collects and merges JUnit results for documentation.
• Updates development commands, test inventory, and Ruff exclusions.

File	Description
tools/​integration_tests/​test_error_handling.py	Adds the invalid-threshold CLI check.
tools/​integration_tests/​conftest.py	Copies an isolated consumer workspace.
tools/​integration_tests/​BUILD	Defines an always-run pytest target.
README.md	Documents separate integration test commands.
pyproject.toml	Adjusts Ruff exclusions.
docs/​verification/​verification_report.rst	Documents pytest coverage and traceability.
docs/​BUILD	Includes integration JUnit results.
.github/​workflows/​tests.yml	Runs pytest and merges test artifacts.

🧠 Review effort: Balanced

---

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

[github-actions]

Documentation preview for this pull request is available at:
pr-26: https://eclipse-score.github.io/coverage_tool/pr-26/