Skip to content

Add the Test Companion action - #91

Open
naimish-browse wants to merge 18 commits into
browserstack:masterfrom
naimish-browse:add-test-companion-action
Open

naimish-browse wants to merge 18 commits into
browserstack:masterfrom
naimish-browse:add-test-companion-action

Conversation

@naimish-browse

Copy link
Copy Markdown

Add the Test Companion action

Adds test-companion/, a composite action that lets a team ask BrowserStack
Test Companion for test work from a pull request comment.

Someone comments @test-companion test this PR. It reads the change, posts the
tests it would write, and writes them once a person replies go. The result
comes back as one comment and one check.

Jira: STM-3663 / STM-5434.

Why this repository

It is public and MIT, which is what customers need in order to reference the
action at all. Today it lives in a private repository, so nobody outside
BrowserStack can use it.

What is in here

  • test-companion/action.yml — the action
  • test-companion/src/ — 15 files, ~3,300 lines
  • test-companion/test/ — 374 tests, no network and no credentials needed
  • test-companion/README.md — customer-facing setup
  • test-companion/starter-workflow.yml — the file customers copy

It differs from the actions already here, in one good way

The four existing sub-actions are JavaScript actions with dist/ bundles
committed by hand at 8–15MB. This one is a composite action with zero
dependencies
and no bundle. What you see in src/ is what runs, and there is
nothing to rebuild on release.

Two things this PR needs from maintainers

1. A floating v1 tag. This repository has v1.0.0–v1.0.6 and no v1.
The README tells customers @v1, which will not resolve until that tag exists.
The current README elsewhere tells people @master, which is unpinned — the
same hazard that bit us on 2026-10-29, when a point release changed behaviour
mid-pipeline. A floating v1 that moves on release is the fix.

2. Agreement on review ownership. CODEOWNERS is
@browserstack/automate-public-repos. This action is owned and released by
@browserstack/test-companion. Worth agreeing upfront whether that means a
CODEOWNERS entry for this directory or a standing review arrangement, rather
than discovering it at the first urgent release.

What this does not do

It does not get a Marketplace listing. Marketplace needs a single
action.yml at the repository root, and that is setup-env. Every sub-action
here is currently unlisted (checked: all three 404). If a Marketplace entry is
a requirement, it needs a separate single-action repository.

Nothing proprietary is published here. The agent itself is in the tc CLI,
which the action downloads at runtime and which stays private. This is the
adapter between GitHub and that CLI.

Testing

cd test-companion && node --test

374 passing. Beyond that, the action has been exercised on real GitHub runs
against a live repository: the comment flow, the approval gate, stopping a run,
closing a pull request mid-run, the oversized-change warning, writing tests and
committing them to the branch, and refusing a settings file planted on a pull
request branch.

Lets a team ask BrowserStack Test Companion for test work from a pull
request comment. Someone comments "@test-companion test this PR"; it
reads the change, posts the tests it would write, and writes them once a
person replies "go". The result comes back as one comment and one check.

Jira: STM-3663, STM-5434.

A composite action with ZERO dependencies and no dist/ bundle — unlike
the JavaScript actions already here, which commit 8-15MB bundles by
hand. What is in src/ is what runs, and there is nothing to rebuild on
release.

422 tests, no network and no credentials needed: cd test-companion &&
node --test
Picked up from the source repo's security review: these git commands
run after the agent, which has --yolo write access to the tree, so a
planted .git/hooks/pre-commit would have been executed with the App's
private key in its environment.
@naimish-browse
naimish-browse requested a review from a team as a code owner October 30, 2026 11:09
Comment thread test-companion/action.yml Fixed
Comment thread test-companion/action.yml Fixed
Comment thread test-companion/src/intent.mjs Fixed
Comment thread test-companion/src/report.mjs Fixed
with:
fetch-depth: 0 # jobs that look at what changed need history

- uses: browserstack/github-actions/test-companion@v1
naimish-browse and others added 7 commits October 30, 2026 16:45
The push carried the token as a global http.extraheader and pushed to the
remote named `origin`. Both are decided by .git/config, which the agent
has had write access to — `url.<base>.insteadOf` redirects `origin` to
another host and a global header follows it there.

The URL is now built from GITHUB_REPOSITORY, the push targets that URL,
and the header is scoped to it. Also closed the config keys that make git
run a program (core.sshCommand, core.pager, diff.external, gpg.program,
commit.gpgsign, core.attributesFile) and disabled system/global config.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…r execute

Pushing to an explicit URL does not escape `url.<base>.insteadOf`, and
`credential.helper` is a shell command git runs on a 401. Before touching
the tree, read the repository's own config and refuse on any key that runs
a program or rewrites a URL. The credential now travels in the environment
rather than argv, which is world-readable via `ps`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Config is where git keeps every "run this program for me" hook, and it
gains more each release. Listing the keys an ordinary actions/checkout
writes and refusing everything else fails closed on keys that do not
exist yet.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PATH directories on a runner are writable by the job's own user, which is
the user the agent ran as. git is now resolved to an absolute path that
this process cannot write to, and PATH is pinned rather than inherited.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"go" names no job, so the job comes from the proposal being approved.
That assignment sat below the lookup that needs it, so asking worked and
approving did not. Moved to where the workflow is final for both paths.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`git config --list` includes command-line config, so the check reported
every GIT_SAFE_ARGS key as unexpected and refused to save. `--local`
fixes it; the arguments are now shared between the code and a real-git
test that fails without them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Central YAML (base), Organization UI (inherited), Workspace UI (inherited); Enforced overrides: Organization
  • Review profile: ASSERTIVE
  • Plan: Enterprise
  • Run ID: b193ec81-887e-463b-bf44-8481ea981b61

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

naimish-browse and others added 3 commits October 30, 2026 17:21
…equest

The agent's explanation was written to the result file only on a planning
pass, so an rca answered into the void — the comment showed a table and
no answer. And a read-only question had to be approved first, through a
proposal that was empty because only test-pr writes a plan.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both from GitHub Advanced Security on this PR. The pinned tag is v4.6.2.
The dynamic RegExp was already escaped by both callers; the escaping now
lives in the helper, because an unescaped sha quietly matches OTHER
commits rather than failing loudly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@naimish-browse

Copy link
Copy Markdown
Author

Thanks — six findings from code scanning, all addressed in bc2c3ea. Two were real, two were false positives worth acting on anyway, and two I've deliberately left. Reasoning for each:

Fixed: actions/upload-artifact@v4 in action.yml (findings 6, 7)

Pinned to ea165f8d65b6e75b540449e92b4886f43607fa02, which is v4.6.2. This one matters more than the usual version-pinning argument: the step runs inside the customer's job with the customer's token, so a repointed tag is their supply chain, not ours to be relaxed about.

Fixed: dynamic RegExp in report.mjs (finding 9)

The ReDoS reading is a false positive — both callers already escaped their input, and the pattern is linear. But the scanner pointed at something real next to it. The escaping lived in the callers, and "every caller remembers" is not a property anyone can verify. An unescaped sha also doesn't fail loudly; it quietly matches other commits, because a.c*d+ matches abcccd too — so a proposal for one commit could be approved against another.

The escaping now lives in markerPattern itself, so it can't be skipped from a call site. Two tests pin it: a sha full of metacharacters matches itself and not a lookalike, and a 50,000-character run returns immediately.

Not changed: dynamic RegExp in intent.mjs (finding 8)

Already escaped one line up, via the same escapeRe helper, and matched against a comment body of bounded length. The pattern is \\b<literal>\\b — no nested quantifier, so there's nothing to backtrack on.

Not changed: tags in starter-workflow.yml (findings 10, 11)

This file is a template customers copy into their own repository and then own, which makes it a different case from action.yml:

  • browserstack/github-actions/test-companion@v1 — pinning our own action to a commit is precisely what a floating major tag exists to avoid. Customers follow @v1 to get fixes; a commit would freeze every one of them on whatever we shipped that day.
  • actions/checkout@v4 — a pinned SHA in a starter goes stale in the customer's repo, where nothing will ever update it. A major tag from GitHub's own first-party action is the usual practice for templates.

Happy to pin actions/checkout if the team would rather be consistent — it's a one-line change. The @v1 on our own action I'd argue to keep either way.

Comment thread test-companion/src/report.mjs Fixed
Both dynamic regexes are gone. The proposal marker is one literal
pattern with the sha captured and compared with ===, which also removes
the bug the escaping existed to prevent: an unescaped sha quietly
matched other commits. Naming a job compares words instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@naimish-browse

Copy link
Copy Markdown
Author

Finding 12 is the same line again — escaping the input doesn't satisfy the rule, only not interpolating does. Fair enough, so I've removed both dynamic patterns rather than argue it twice. Pushed in 0bbda92.

The proposal marker is now one literal pattern that captures the sha, compared with ===. This is better than the escaped version, not just quieter: building the pattern per search meant the sha had to be escaped or it would silently match other commits — a.c*d+ also finds abcccd, so a plan written for one commit could be approved against another. String equality removes the rule instead of restating it.

Naming a job (finding 8) now compares words instead of building \b<name>\b per name. One deliberate behaviour change, documented at the call site: punctuation inside a name is now a word break, so a job called a.c is also found in the words "a c". Job names come from the repository's own pipeline file and are ordinary words in practice, and being slightly more willing to recognise one is the safe direction — the alternative is telling someone their own job name isn't a job.

Both escape helpers are deleted, because nothing needs escaping now.

There is one new RegExp left, in plan-run.mjs: it builds from a module constant (@test-companion), not from input. I kept it so the mention has a single spelling rather than a string and a pattern that can drift apart. Happy to split it if you'd rather see zero.

460 tests pass, including six added before the change — four pinning that a name is matched on whole words (regenerated is not a request to run generate), and two pinning that a sha full of metacharacters matches itself and not a lookalike.

naimish-browse and others added 5 commits October 30, 2026 18:13
semgrep/ci is a required check and was failing on both. checkout is
third party and now pinned to v4.4.0's commit; our own action stays on
@v1 with a nosemgrep and the reason, because a floating major tag is how
customers receive fixes without editing their copy.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The rule id repeats its last segment; written once, it matched nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ting#3101

B1 a skip or a crash left the "in progress" check pending for ever, and
a crash posted its check on the default branch instead of the PR head.
B2 `stop` shared a concurrency group with the run it was sent to stop.
B3 the App private key stayed in the environment after it had been used.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Identity on comments (another bot could get a proposal approved into a
live run), the push no longer carries agent commits or whole untracked
directories, persist-credentials off, one checkout path without --depth,
approvals leave a trace, stop drops what it says it drops, and comments
that are not for us no longer boot a runner.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…oved

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants