Skip to content

Add a sample report matrix for developers - #108

Merged
bencehezso merged 1 commit into
release/2.0from
task/report_sample_matrix
Oct 25, 2026
Merged

bencehezso merged 1 commit into
release/2.0from
task/report_sample_matrix

Conversation

@bencehezso

Copy link
Copy Markdown
Contributor

Summary

A developer tool renders a fixed matrix of fictional sample assessments (AWS, Azure and Terraform state; with and without egress; offline and uploaded; and edge cases) through the real report pipeline, offline and deterministically. The same scenario files run in CI against a test catalogue, so the existing guards cover every case.

Changes

Scenarios

  • Twelve scenario files in tests/fixtures/samples/ and two fictional tfstate fixtures; no real names, accounts, tenants or subscriptions.
  • A loader that names the file and field of any error.

Tool (python -m tests.samples)

  • Runs the real stages with the collectors replaced by scenario data, under a frozen clock: report.pdf, index.html, egress.pdf, egress.html, a text dump of each, an overview page, and optional contact sheets (PyMuPDF, optional).
  • --case, --out, --catalogue, --sheets; never downloads a catalogue; exits non-zero when a case fails and still renders the rest.
  • The list of files a case holds is in one place (EXPECTED_FILES).

Tests

  • 46 tests: loader, a missing or inactive type fails only its case, the command, and per case the expected files, credential guard, term/wording/retired-term guards, fonts, notices, upload states, case-specific checks and determinism.
  • html_text moves to tests/support/html.py, with a per-heading dump used for the HTML text files.

Repository

  • .gitignore: /samples/ and the sample tfstate fixtures.

Compatibility

Config, exit codes, report output, the dataset and the API v2 definition copy are unchanged. No product code changes; the only files outside tests/ are .gitignore.

@bencehezso bencehezso self-assigned this Oct 25, 2026
@bencehezso bencehezso added the enhancement New feature or request label Oct 25, 2026
@bencehezso
bencehezso merged commit fc7b284 into release/2.0 Oct 25, 2026
11 checks passed
@bencehezso
bencehezso deleted the task/report_sample_matrix branch October 25, 2026 17:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant