Repository navigation
ci: Run optional E2E tests in a separate workflow #25548
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+418
−102
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
be11fd0
ci: Run optional E2E tests in a separate workflow
andreiborza 702afc8
Move optional E2E dispatch into an in-repo action
andreiborza 1983247
Merge remote-tracking branch 'origin/develop' into ab/optional-e2e-wo…
andreiborza 6122e6d
Wait for optional E2E tests on a GitHub runner
andreiborza fd0819f
Do not fail the job when the optional E2E report fails
andreiborza 9350429
Merge branch 'develop' into ab/optional-e2e-workflow
andreiborza File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,167 @@ | ||
| name: 'CI: Optional E2E Tests' | ||
| run-name: Optional E2E tests for ${{ inputs.commit }} (${{ inputs.correlation_id }}) | ||
|
|
||
| # Dispatched by `job_optional_e2e_tests` in build.yml. This is a separate workflow because GitHub | ||
| # does not count failed checks of a dispatched run in the PR status, so these failures do not mark | ||
| # the PR as failed. | ||
| on: | ||
| # Only here so that GitHub registers this workflow before it is on the default branch, which a | ||
| # dispatch needs. The job is skipped for this event. Remove this trigger after the merge. | ||
| pull_request: | ||
| workflow_dispatch: | ||
| inputs: | ||
| commit: | ||
| description: The commit that the tarballs were built from | ||
| required: true | ||
| build_run_id: | ||
| description: The build.yml run that uploaded the tarball artifacts | ||
| required: true | ||
| matrix: | ||
| description: The optional E2E test matrix as JSON | ||
| required: true | ||
| pr_number: | ||
| description: The PR that the run is for, if there is one | ||
| required: false | ||
| correlation_id: | ||
| description: Lets the dispatching job find this run | ||
| required: true | ||
|
|
||
| # Like build.yml: a new push to a PR cancels the old run, other runs are never cancelled. | ||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ inputs.pr_number || github.run_id }} | ||
| cancel-in-progress: true | ||
|
|
||
| env: | ||
| HEAD_COMMIT: ${{ inputs.commit }} | ||
|
|
||
| # See build.yml | ||
| TARBALL_ARTIFACT_DOWNLOAD_PATH: ${{ github.workspace }}/packages | ||
| DISABLE_V8_COMPILE_CACHE: '1' | ||
|
|
||
| jobs: | ||
| job_optional_e2e_tests: | ||
| name: E2E ${{ matrix.label || matrix.test-application }} Test (optional) | ||
| if: github.event_name == 'workflow_dispatch' | ||
| # Same cache volume as the E2E jobs in build.yml. The event is always `workflow_dispatch`, so the PR number | ||
| # decides if the run gets the PR volume. | ||
| runs-on: | ||
| namespace-profile-sentry-javascript;overrides.cache-tag=sentry-javascript-e2e${{ inputs.pr_number != '' && '-pr' | ||
| || '' }} | ||
| timeout-minutes: 15 | ||
| permissions: | ||
| contents: read | ||
| # To download the artifacts of the build.yml run | ||
| actions: read | ||
| env: | ||
| E2E_TEST_AUTH_TOKEN: ${{ secrets.E2E_TEST_AUTH_TOKEN }} | ||
| E2E_TEST_DSN: ${{ secrets.E2E_TEST_DSN }} | ||
| # Needed because some apps expect a certain prefix | ||
| NEXT_PUBLIC_E2E_TEST_DSN: ${{ secrets.E2E_TEST_DSN }} | ||
| PUBLIC_E2E_TEST_DSN: ${{ secrets.E2E_TEST_DSN }} | ||
| REACT_APP_E2E_TEST_DSN: ${{ secrets.E2E_TEST_DSN }} | ||
| E2E_TEST_SENTRY_ORG_SLUG: 'sentry-javascript-sdks' | ||
| E2E_TEST_SENTRY_PROJECT: 'sentry-javascript-e2e-tests' | ||
| E2E_OPENROUTER_API_KEY: ${{ secrets.E2E_OPENROUTER_API_KEY }} | ||
| # Only this app talks to OpenAI directly (OpenRouter has no Decisions route), so only it gets the key. | ||
| E2E_OPENAI_DECISIONS_API_KEY: | ||
| ${{ matrix.test-application == 'node-openai-decisions-send-to-sentry' && secrets.E2E_OPENAI_DECISIONS_API_KEY || | ||
| '' }} | ||
| # Used by test apps that deploy a real Cloudflare Worker, e.g. cloudflare-workers-send-to-sentry | ||
| CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }} | ||
| CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }} | ||
| # Keeps the Worker name at `<prefix>-pr-<number>`, which the cleanup-e2e-workers workflow deletes | ||
| E2E_PR_NUMBER: ${{ inputs.pr_number }} | ||
| # The `runtime` of a runtime variant, see dev-packages/e2e-tests/README.md | ||
| RUNTIME: ${{ matrix.runtime }} | ||
| # Without the root node_modules, `pnpm run` would first install the whole workspace. | ||
| pnpm_config_verify_deps_before_run: 'false' | ||
| strategy: | ||
| fail-fast: false | ||
| # The fallback only keeps the expression valid for the skipped `pull_request` run. | ||
| matrix: ${{ fromJson(inputs.matrix || '{"include":[{}]}') }} | ||
|
|
||
| steps: | ||
| - name: Check out commit ${{ env.HEAD_COMMIT }} | ||
| uses: actions/checkout@v7 | ||
| with: | ||
| ref: ${{ env.HEAD_COMMIT }} | ||
| - name: Set up Node | ||
| uses: actions/setup-node@v7 | ||
| with: | ||
| node-version-file: 'dev-packages/e2e-tests/test-applications/${{ matrix.test-application }}/package.json' | ||
| - name: Set up Bun | ||
| if: | ||
| matrix.test-application == 'node-exports-test-app' || contains(matrix.test-application, 'bun') || | ||
| matrix.runtime == 'bun' | ||
| uses: oven-sh/setup-bun@v2 | ||
| with: | ||
| bun-version: '1.3.14' | ||
| - name: Set up Deno | ||
| if: contains(matrix.test-application, 'deno') || matrix.runtime == 'deno' | ||
| uses: denoland/setup-deno@v2.0.5 | ||
| with: | ||
| deno-version: ${{ matrix.deno-version || 'v2.8.3' }} | ||
| # Keeps the pnpm store on the cache volume for the `pnpm install` of the test application | ||
| - name: Mount pnpm store | ||
| uses: ./.github/actions/mount-pnpm-store | ||
| - name: Install pnpm | ||
| uses: ./.github/actions/install-pnpm | ||
|
|
||
| - name: Restore tarball artifacts | ||
| uses: actions/download-artifact@v8 | ||
| with: | ||
| name: build-tarball-output | ||
| path: ${{ env.TARBALL_ARTIFACT_DOWNLOAD_PATH }} | ||
| run-id: ${{ inputs.build_run_id }} | ||
| github-token: ${{ github.token }} | ||
|
|
||
| - name: Restore test-utils tarball artifact | ||
| uses: actions/download-artifact@v8 | ||
| with: | ||
| name: build-test-utils-tarball-output | ||
| path: dev-packages/test-utils | ||
| run-id: ${{ inputs.build_run_id }} | ||
| github-token: ${{ github.token }} | ||
|
|
||
| - name: Prepare E2E tests | ||
| run: pnpm run test:prepare | ||
| working-directory: dev-packages/e2e-tests | ||
|
|
||
| - name: Validate test setup | ||
| run: pnpm run test:validate | ||
| working-directory: dev-packages/e2e-tests | ||
|
|
||
| - name: Copy to temp | ||
| run: | ||
| pnpm run ci:copy-to-temp ./test-applications/${{ matrix.test-application }} ${{ runner.temp | ||
| }}/test-application | ||
| working-directory: dev-packages/e2e-tests | ||
|
|
||
| - name: Add pnpm overrides | ||
| run: | ||
| pnpm run ci:pnpm-overrides ${{ runner.temp }}/test-application ${{ github.workspace | ||
| }}/dev-packages/e2e-tests/packed | ||
| working-directory: dev-packages/e2e-tests | ||
|
|
||
| # Installs Playwright while the app builds. Keep it right before the build: the runner sets up a background | ||
| # composite action's steps from the job's shared `github` context, which fails with "Collection was modified" if | ||
| # a foreground step starts at that moment. Starting next to `download-artifact` also made it fail to read the event file. | ||
| - name: Install Playwright | ||
| id: install-playwright | ||
| background: true | ||
| uses: ./.github/actions/install-playwright | ||
| with: | ||
| browsers: chromium | ||
| cwd: dev-packages/e2e-tests/test-applications/${{ matrix.test-application }} | ||
|
|
||
| - name: Build E2E app | ||
| working-directory: ${{ runner.temp }}/test-application | ||
| timeout-minutes: 7 | ||
| run: ${{ matrix.build-command || 'pnpm test:build' }} | ||
|
|
||
| - wait: [install-playwright] | ||
|
|
||
| - name: Run E2E test | ||
| working-directory: ${{ runner.temp }}/test-application | ||
| timeout-minutes: 10 | ||
| run: ${{ matrix.assert-command || 'pnpm test:assert' }} | ||
|
andreiborza marked this conversation as resolved.
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| { | ||
| "$schema": "../../node_modules/oxlint/configuration_schema.json", | ||
| "extends": ["../.oxlintrc.json"] | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| name: 'optional-e2e-gh-action' | ||
| description: 'Run the optional E2E tests in their own workflow, wait for them and report failures in a PR comment' | ||
| inputs: | ||
| github_token: | ||
| required: true | ||
| description: 'a github access token' | ||
| commit: | ||
| required: true | ||
| description: 'The commit that the tarballs were built from' | ||
| matrix: | ||
| required: true | ||
| description: 'The optional E2E test matrix as JSON' | ||
| runs: | ||
| using: 'node24' | ||
| main: 'index.mjs' | ||
|
sentry-warden[bot] marked this conversation as resolved.
|
||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 good call!