Repository navigation
feat(runners): boot modes for warm pool instances - #5495
Open
Brend-Smits wants to merge 3 commits into
Open
Brend-Smits wants to merge 3 commits into
Brend-Smits wants to merge 3 commits into
Conversation
This was referenced Sep 30, 2026
Contributor
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
This was referenced Sep 30, 2026
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Hook re-extraction, macOS timestamp parsing, and stale tracing context can cause incorrect warm-runner behavior.
Review effort: Balanced
Findings: 1
Open (3)
What changed in this PR
Adds warm-pool boot modes so Linux runners can prime, stop, and resume for assigned jobs.
Changes:
- Adds RUN, PRIME, and WAIT boot handling with a systemd hook.
- Adds shell-based boot-mode tests.
- Documents custom user-data requirements.
| File | Description |
|---|---|
modules/runners/templates/start-runner.sh |
Implements boot modes and hook installation. |
modules/runners/templates/tests/start-runner.test.sh |
Tests boot-mode behavior. |
docs/configuration.md |
Documents warm-pool boot hooks. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This was referenced Sep 30, 2026
Brend-Smits
force-pushed
the
feat/warm-pool-05-boot-hook
branch
from
October 1, 2026 12:27
6e5ca5c to
7bda80c
Compare
Brend-Smits
force-pushed
the
feat/warm-pool-04-scale-down-sweep
branch
from
October 1, 2026 12:27
71db0c7 to
26def47
Compare
Brend-Smits
marked this pull request as ready for review
October 1, 2026 13:58
This was referenced Oct 1, 2026
Brend-Smits
force-pushed
the
feat/warm-pool-05-boot-hook
branch
from
October 6, 2026 11:51
7bda80c to
73853b7
Compare
Cloud-init runs user data only on the first boot, but a warm instance boots twice: once to prime and stop, and again when a job is assigned. start-runner.sh now installs a systemd unit that reruns the start logic on later boots and picks a boot mode: - RUN: a runner configuration exists in SSM for this instance, so configure and start the runner as before - PRIME: the instance is tagged as warm standby and not yet activated, so stop the machine and wait in the pool - WAIT: otherwise, poll for the configuration as before Cleanup no longer terminates a warm instance while it primes. An activated instance logs the time from activation to runner start, and later reboots do not rerun the start logic. The start logic is wrapped in begin and end markers, which the boot hook uses to copy it out of the user data into the systemd script. Adds a shell test harness for the boot modes and documents the requirement for custom user data templates. Signed-off-by: Brend Smits <brend.smits@philips.com>
- install_boot_hook extracts the start logic to a temporary file and moves it into place. When a parked instance primed again, $0 was the hook itself and the redirect truncated it before it was read. - The BSD date fallback for the activation latency strips both the fractional seconds and the trailing Z. - Warm instances are launched without the pool's ghr:trace_id. Scale-up tags its own trace on activation and removes it again on rollback, so the runner boot is traced under the scale-up that assigned it. Signed-off-by: Brend Smits <brend.smits@philips.com>
The EC2 compute provider used by the multi-runner v2 stack keeps its own copy of start-runner.sh, which did not get the warm pool boot modes. Warm instances launched through that stack would never prime and stop, so the pool would evict them as stuck. Sync the copy with modules/runners (which also brings over the existing SEGMENT fix in the cleanup trap) and let the boot-mode harness take a template path and check that both copies match. Signed-off-by: Brend Smits <brend.smits@philips.com>
Brend-Smits
force-pushed
the
feat/warm-pool-04-scale-down-sweep
branch
from
October 16, 2026 18:11
6599adf to
2b57f0a
Compare
Brend-Smits
force-pushed
the
feat/warm-pool-05-boot-hook
branch
from
October 16, 2026 18:11
73853b7 to
769d4dd
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Warm pool stack (review in order; each PR is based on the previous one):
Description
Cloud-init runs user data only on the first boot, but a warm instance boots twice: once to prime and stop, and again when a job is assigned.
start-runner.shnow installs a systemd unit that reruns the start logic on later boots and picks a boot mode:Cleanup no longer terminates a warm instance while it primes. An activated instance logs the time from activation to runner start, and later reboots do not rerun the start logic. The start logic is wrapped in begin/end markers, which the boot hook uses to copy it out of the user data into the systemd script. Behaviour for non-warm runners is unchanged.
EC2 compute provider template
The EC2 compute provider used by the
multi-runner-v2stack keeps its own copy ofstart-runner.sh. That copy is synced withmodules/runners(which also brings over the existingSEGMENTfix in the cleanup trap), so warm instances launched through the v2 stack prime and stop as well. The boot-mode harness takes a template path and fails when the two copies differ.Test Plan
modules/runners/templates/tests/start-runner.test.shwith mocked IMDS, SSM and systemd (56 checks), run against both template copies, including the check that they match.terraform testformodules/compute-providers/aws/ec2; the only failure (separates_control_plane_contract_from_ec2_resources) also fails onmain.Related Issues
Supersedes #5204.