Repository navigation
[v0.5.0] benchmark reporting — generate raw report and README-safe summary #74
Description
Context
Parent: #63. Methodology: docs/benchmarks/PUBLIC-BENCHMARK-METHODOLOGY.md.
The benchmark must produce public-facing output that is credible, boringly auditable, and impossible to confuse with hand-picked marketing numbers.
Goal
Add benchmark reporting that converts raw run artifacts into a full report plus a README-safe summary block gated on reproducible data.
Acceptance criteria
- README-safe summaries present strict tool + model pairings, for example
python-docs-mcp-server + gpt-4o, instead of tool-only rows. - Generated tables include an error/timeout rate column so unstable tools cannot hide behind surviving-query correctness.
- A report generator reads raw benchmark artifacts and writes
docs/benchmarks/results/<run-id>/REPORT.md. - The report includes methodology link, corpus hash, repo commit, model/client matrix, competitor manifest, correctness by category, token counts after client rewrap, latency median/p95, failures/exclusions, and environment metadata.
- A README summary template is generated separately and includes only compact tables plus links to the methodology and raw result bundle.
- The generator refuses to produce a README-ready block if required metadata, corpus hash, or raw result files are missing.
- Tests cover report generation, missing metadata failure, failed competitor disclosure, and aggregate/per-model separation.
- README itself is not updated with benchmark claims until real data exists.
Scope boundaries
In scope:
- Report generator.
- README-safe summary template generation.
- Tests for gating and disclosure rules.
Out of scope:
- Running providers.
- Scoring answer correctness manually.
- Editing README with final benchmark results.
Forbidden-territory reminder
Do not modify MCP tool names, parameters, return shapes, schema.sql, .github/workflows/, pyproject.toml project metadata, .planning/POSITIONING.md, the README hero section, LICENSE, SECURITY.md, or existing tests by weakening/deleting assertions.
README edits, if any, must be below the install/tooling sections and must not touch the hero.
Validation commands
uv run ruff check src/ tests/
uv run pyright src/
uv run pytest --tb=short -q
uv run python-docs-mcp-server doctorRun any new reporting tests directly as well.
PR template
Use Refs #63, not Closes #63.
The PR must include:
- Example generated report from fixture data.
- Test output.
- Confirmation that README claim publication is still gated on real benchmark data.
Recovery
If report generation needs result fields not specified by the runner, stop and comment with the missing fields so the runner issue can be corrected.
Effort estimate
4-6 hours.
Activity
ayhammouda commentedon Jul 8, 2026
Agent context — #74 [v0.5.0] benchmark reporting — generate raw report and README-safe summary
Status: FINAL (verified against merged
main@1f202b6; PR #75 mergedad18406, PR #85 merged1f202b6, both 2026-07-08).Auditable copy: this content is mirrored as a comment on issue #74 (
.planning/is gitignored — the issue comment is the copy pre-flight verifies and dispatch links).1. Roadmap excerpt (why this exists)
STRATEGIC-ROADMAP-2026-05-29.md §4 (v0.5.0): public benchmark harness with mandatory methodology disclosure. Decision 5.17 (evidence ladder): no comparative/benchmark claim enters README/PyPI/launch copy until reproducible data exists — your generator is the enforcement point: it produces a README-safe template, never a README edit, and refuses to render when required metadata/corpus-hash/raw files are missing.
2. Code touch-points (verify against merged main before starting)
run-summary.json,environment.json,planned-cells.json,snapshots/(byte-copies of corpus + manifest after the [v0.5.0] benchmark runner — add reproducible CLI and artifact layout #75 merge-readiness fixes — corpus hash = hash of the snapshot bytes, which must equal the frozen input file's), and per-celltranscripts|tokens|latency|scoring|failures/<competitor>/<qid>.json.included_in_correctness_denominator,error_category,score(0.0 for failures,None+requires_manual_scoringfor placeholders); latency records carrylatency_ms,status,error_category; token records carryclient_wrapped_tokens/raw_payload_tokens(may beNoneplaceholders — report must surface that honestly, not as zero);tool_model_key= strict tool+model pairing (id:provider/model) — README-safe tables use these pairings, never tool-only rows.docs/benchmarks/model-matrix.yml(from [v0.5.0] benchmark adapters — define OpenAI/Google model matrix #73/PR [v0.5.0] benchmark adapters — define OpenAI/Google model matrix #85, on main): entries record provider, model_id, client, token_count_method, latency_scope, headline_eligible. Loader:benchmarks/model_matrix.py(validates schema, exposestool_model_cells()for tool×model cross-multiplication andMETHODOLOGY_TOKEN_LABEL = "Claude Tokens (Normalized Payload)"). The report must include the model/client matrix, keep per-model results visible, mark any aggregate as aggregate, and never average across model families — this is [v0.5.0] benchmark adapters — define OpenAI/Google model matrix #73's one deferred acceptance criterion, now YOURS to satisfy.benchmarks/runner.py's_execute_celldispatch (deliberate [v0.5.0] benchmark adapters — define OpenAI/Google model matrix #73 deferral, tracked for the WP3 issue). Your generator consumes run artifacts + the matrix file as they exist. If the report needs a per-model artifact field the runner doesn't emit, that is exactly issue [v0.5.0] benchmark reporting — generate raw report and README-safe summary #74's recovery clause: STOP and comment on [v0.5.0] benchmark reporting — generate raw report and README-safe summary #74 naming the missing fields — do not invent fields and do not wire the runner yourself.docs/benchmarks/results/<run-id>/REPORT.mdgenerator + separately generated README-safe summary block (compact tables + links to methodology and raw bundle) + error/timeout-rate column so unstable tools can't hide behind surviving-query correctness.benchmarks/report.pybehind a thin subcommand — keepbenchmarks/__main__.pychurn minimal (shared surface with [v0.5.0] benchmark adapters — define OpenAI/Google model matrix #73's changes).3. Test patterns to follow
tests/benchmarks/test_runner.py:_write_yaml-style helpers build fixtures undertmp_path; assert file paths and parsed JSON content. For reporting: generate a fake run-artifact tree undertmp_path, run the generator, assert REPORT.md content, the refusal path (missing metadata/corpus-hash/raw files → clean validation error), failed-competitor disclosure, and aggregate/per-model separation. New tests MUST live undertests/benchmarks/(souv run pytest tests/benchmarks -qexercises them).4. Known pitfalls
tmp_pathortests/benchmarks/fixtures/— NEVER commit anything underdocs/benchmarks/results/unless unmistakably labeled SYNTHETIC/FIXTURE DATA in the title and every table. The PR-template's "example generated report from fixture data" belongs in the PR body, not committed to the public docs tree.uv run ruff check benchmarks/anduv run pyright benchmarks/(see [v0.5.0] ci — lint and type-check the benchmarks/ package (maintainer-only) #84).Refs #63(neverCloses #63),Closes #74only if all criteria met. Diff >500 lines →supervisor-reviewlabel + explanation section.5. Decision log (worker fills in)
agent: refuse to run into a non-empty output directory
agent: write byte-identical corpus/manifest snapshots
[v0.5.0] benchmark runner — add reproducible CLI and artifact layout (#…
agent: restructure implicit string concatenation flagged by CodeQL
[v0.5.0] benchmark reporting — generate raw report and README-safe su…
agent: tests — fix CodeQL unnecessary-lambda + CodeRabbit review find…
[v0.5.0] benchmark adapters — competitor MCP tool adapters (Context7,…