Skip to content

ci: add advisory AI docs PR reviewer with shadow-mode risk label - #2152

Merged
soheimam merged 3 commits into
masterfrom
soheimacanton/devrel-828-docs-pr-reviewer
Oct 19, 2026
Merged

soheimam merged 3 commits into
masterfrom
soheimacanton/devrel-828-docs-pr-reviewer

Conversation

@soheimam

@soheimam soheimam commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

What changed? Why?

Adds an advisory AI reviewer for docs PRs from base-devrel and base-eng members.

  • It posts one review comment covering fit (is the page live, not redirected, next to the right neighbor pages, do links resolve) and writing quality against the Base guidelines.
  • Optional one-click suggestions are posted only when they add no new links, addresses, numbers, or code.
  • It sets one risk: low|medium|high|critical label using fixed rules from the review ladder proposal. No AI is involved in the label. Shadow mode: nothing is enforced.
  • It never pushes, approves, requests changes, or merges, and it is not a required check.

Two workflows:

  • docs-pr-review-trigger.yml runs on pull_request on ubuntu-latest with no secrets and no permissions. It only starts stage 2.
  • docs-pr-review.yml runs on workflow_run from the default branch on BaseRunnerGroup. It rejects forks, re-checks the PR through the API, checks team membership, and then makes one LLM Gateway call through the existing scripts/sync-from-base-std client.

Notes to reviewers

  • Security review: SPA-35356, approved (Self-served). Findings SECREQ-7592 and SECREQ-7593 are fixed in this code.
  • Runner and Gateway access reuse the existing approvals (SPA-26751, infra/entry-gateway#6123), the same as base-std-docs-sync.yml.
  • Changes .github/workflows/, so the CI Configuration gate needs 3 writer approvals.
  • Merging adds no running job until the workflows are on master, because workflow_run only runs from the default branch.

Before merge:

  • Fork PR workflow approval set to "Require approval for all external contributors"
  • Add the DOCS_PR_REVIEW_MEMBERS_TOKEN secret (org Members: read only)
  • Create the skip-ai-review label and the four risk: low, risk: medium, risk: high, risk: critical labels

Turn off for everyone with the repository variable DISABLE_DOCS_PR_REVIEW=true, or for one PR with the skip-ai-review label.

How has it been tested?

Source and security doc: cb-pages/devrel-ai-cc PR #29.

Screenshots

N/A (no user-facing changes)

Generated with Toshi

@cb-heimdall

cb-heimdall commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Heimdall Review Status

Requirement Status More Info
Reviews ✅ 1/1
Denominator calculation
Show calculation
1 if user is bot 0
1 if user is external 0
2 if repo is sensitive 0
From .codeflow.yml 1
Additional review requirements
Show calculation
Max 0
0
From CODEOWNERS 0
Global minimum 0
Max 1
1
1 if commit is unverified 0
Sum 1

@linear

linear Bot commented Oct 17, 2026

Copy link
Copy Markdown

DEVREL-828

Comment thread scripts/docs-pr-review/render.mjs Fixed
Comment thread scripts/docs-pr-review/render.mjs Fixed
Comment thread scripts/docs-pr-review/render.mjs Fixed
Comment thread scripts/docs-pr-review/render.mjs Fixed
@soheimam
soheimam marked this pull request as ready for review October 19, 2026 17:49
Two-stage workflow (pull_request trigger on ubuntu-latest, workflow_run review on BaseRunnerGroup) that posts an advisory review comment and one risk: label on docs PRs from base-devrel and base-eng members. Security review: SPA-35356 (approved).
Addresses CodeQL js/incomplete-multi-character-sanitization and js/bad-tag-filter: fragments can no longer recombine into <!--, and the --!> end form is removed.
A page added together with its navigation entry was reported as not in navigation, because facts came from the default-branch docs.json. Facts now use the PR copy when docs.json changes (falling back to the default branch if it does not parse), links to pages added in the same PR resolve, and links to pages the PR removes are reported.
@soheimam
soheimam force-pushed the soheimacanton/devrel-828-docs-pr-reviewer branch from 9ef55d9 to 1ac2495 Compare October 19, 2026 17:53
@soheimam
soheimam merged commit a848056 into master Oct 19, 2026
14 checks passed
@soheimam
soheimam deleted the soheimacanton/devrel-828-docs-pr-reviewer branch October 19, 2026 18:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants