Implemented in crates/sentinel-worker (Linux only; empty elsewhere); the report and spec messages are in the worker link, the controller side is dispatch::report/spec_bytes. sentinel worker uses it whenever rootless Podman answers; otherwise it declines every offer and says why.
Every attempt gets <data_dir>/workspaces/<attempt> (workspaces/ is 0700; the attempt directory keeps the umask's mode so a non-root image user can enter its bind-mounted /workspace), created empty exactly once (Workspace::create refuses an existing directory — a leftover belongs to reconciliation, not to a new attempt) and destroyed after finalization. A rerun is a new attempt with a new directory. Workspace::leftovers lists what an earlier process left behind.
checkout::checkout asks Git for the pinned SHA itself: git init, git fetch --no-tags --depth 1 -- <repo> <sha>, git checkout --detach FETCH_HEAD, then git rev-parse HEAD must equal the SHA or the checkout is a Preparation failure. The branch name on the run is provenance only. Every Git invocation runs with a cleared environment, GIT_TERMINAL_PROMPT=0, GIT_CONFIG_NOSYSTEM=1, GIT_CONFIG_GLOBAL=/dev/null and a HOME/XDG_CONFIG_HOME that name no directory (never the workspace, so a committed .gitconfig cannot configure a later Git command, and Git before 2.32, which ignores GIT_CONFIG_GLOBAL, finds no global config either), in its own process group under one shared deadline (CHECKOUT_TIMEOUT, 10 min); past it the whole group is killed. A repository argument that looks like an option is refused before Git sees it.
By default a bound repository's checkout runs through the worker's per-repository object mirrors (a manual run, which names its own remote, always checks out direct): a serialized incremental fetch into <data_dir>/mirrors/<rep_id>/, the pinned SHA verified as a commit, then a private object store materialized into the workspace — reflink-copied where the filesystem allows, byte-copied when the store is small, and otherwise fetched at depth 1 out of the mirror so only the pinned tree is written — while a reader lease keeps GC from pruning mid-read. A mirror that cannot serve (lock wait, IO, a damaged store) falls back to the direct fetch above with the reason recorded; the remote's own answer never retries, and git_mirrors = false disables mirrors entirely.
Credentials reach Git through GIT_ASKPASS — an owner-only helper script that answers from its own environment — or, for SSH deploy keys, through a GIT_SSH wrapper that pins ssh -F /dev/null with BatchMode, IdentitiesOnly, StrictHostKeyChecking and the bound known_hosts. Bound source access arrives with the spec (sources) and is verified before Git starts: exact remote, allowed ref, unexpired. Everything the checkout installs lives in a sibling *-askpass directory created 0700 and deleted on every path, including failure. The credential is never in a URL, never in .git/config, never in the job's environment, never in a log line; a failure excerpt is one bounded line with the credential replaced. HTTP redirects are off, the credential helper list is empty, and exactly one Git protocol is allowed. Unbound repositories keep the explicit manual mode: a remote named by the run, no credential.
Submodules and LFS are not fetched, and only the one repository is checked out.
podman::probe refuses to start without rootless Podman on cgroup v2: an executor that cannot isolate is not one. Every container is created with the baseline the probes proved enforceable (feasibility probes):
| Flag | Why |
|---|---|
--cpus <job> |
CPU quota from the compiled resources (cpu_millis) |
--memory <job> --memory-swap <job> |
memory limit with no swap escape; the kernel OOM-kills inside the cgroup |
--pids-limit 4096 |
a fork bomb ends inside its cgroup |
--network none |
loopback only; no egress from job steps (the default; see job network) |
--read-only --tmpfs /tmp |
the image is immutable; /tmp (1 GiB tmpfs) and the workspace are the writable places |
--cap-drop ALL --security-opt no-new-privileges |
CapEff all zero, and nothing can regain it |
--ulimit core=0:0 |
no core dump of a step, whatever it held |
--ulimit fsize=<disk>:<disk> |
no single file a step writes may exceed the job's declared resources.disk (a step that tries gets SIGXFSZ); a cap on one file, not on the total — see Disk |
--entrypoint /bin/sh |
the keepalive is the container's main process whatever the image's ENTRYPOINT says; left alone, an entrypoint would take the keepalive for its arguments |
| rootless user namespace | uid 0 inside is the worker account outside; files the job creates in the workspace are the worker's |
--volume <workspace>:/workspace --workdir /workspace |
the workspace; the only other writable binds are the attempt's private cache views for absolute cache paths |
--pull never, image name@sha256:… |
the bytes the run pinned; the pull is a separate, bounded preparation step (IMAGE_PULL_TIMEOUT, 15 min), refused for an unpinned reference |
--name sentinel-<attempt>, labels io.sentinel.worker/io.sentinel.attempt |
ownership recorded in the runtime itself: podman::owned(worker) lists what this worker created, whatever its state, for reconciliation |
The container's main process is a keepalive: a /bin/sh countdown that exits on TERM and, in any case, after the job's timeout plus a fixed 5 minutes (podman::KEEPALIVE_SLACK_SECS: the secret-clearing passes, artifact capture and the graceful stop that precede teardown) — 24 h plus that when the caller names no bound. When the keepalive ends, the container's PID namespace goes with it, so a container that a dead worker never tore down ends by itself instead of running on while the controller reallocates its capacity; startup reconciliation (reconciliation) removes what is left. The bound is a shell countdown rather than podman create --timeout, which not every Podman supports; steps run in it with podman exec — the image must provide /bin/sh, which the run spec's shells already require. There is no Docker socket, no privileged flag, and no pipeline key that loosens any of this.
Container::exec runs one StepCommand: its argv, its environment with the worker's context appended after the pipeline's (SENTINEL_RUN, SENTINEL_JOB, SENTINEL_ATTEMPT, SENTINEL_SHA, SENTINEL_WORKSPACE, CI), so a pipeline cannot spoof them; its working directory under /workspace; its timeout. Exit status and signal (Podman's 128 + n) are reported as such; a step past its timeout stops the whole container — steps are sequential and the attempt is over. Output tails of 64 KiB per stream are kept for diagnostics; the same bytes stream through the log pipeline as they are produced.
Shutdown. On SIGTERM/SIGINT the worker calls Executor::shutdown before it stops the link: no further offer is taken, offers still waiting for their spec are handed back (Decline, so the controller requeues them), every running attempt's container is removed in parallel (podman rm -f), the attempts' own threads tear down their workspaces and secret directories, and nothing is reported for them — the controller settles them by expiry exactly as for a lost lease, instead of recording a cancel nobody asked for. The wait for the teardowns is bounded (20 s); attempts_shut_down logs how many ended and how many were still going. A crash (no shutdown) leaves the containers to their keepalive bound and to startup reconciliation.
Container::destroy is podman stop -t 2 then podman rm -f — the removal runs even when the stop failed or timed out (a container whose graceful stop hung must still go before its capacity is released), and the first failure is returned. Finalization runs both teardowns (container and workspace) even when one fails.
Every helper (git, podman) is also asked of the kernel to die with the process that spawned it (PR_SET_PDEATHSIG with SIGKILL, checked against a parent that died before the call; sentinel_git::die_with_parent), so a fetch holding a credential in its environment cannot outlive a crashed worker or controller until its deadline. The signal follows the thread that forked the helper — every call starts and waits for its helper on one thread — and reaches the helper itself, not what the helper starts; a container's own processes are the keepalive bound's business above. A helper wait that fails with an I/O error kills the helper's process group and settles its readers instead of returning with the child still running. Spawning with a pre_exec hook means fork instead of posix_spawn. Every helper runs in its own process group under a deadline; the wait for it is a pidfd poll, so its exit is noticed at once rather than on a 20 ms sleep tick, and the preparation helpers are also killed, group and all, when the attempt is cancelled. Diagnostic output tails are kept in a fixed ring (no per-chunk memmove).
A job container has no network by default: --network none leaves it loopback only, so a step cannot fetch anything and dependencies come from caches or the source. An operator can grant outbound network on one worker with job_network = "egress" in its configuration. Its job containers then run with --network pasta:--no-map-gw: rootless pasta forwards outbound connections through the worker account's network stack, and with the gateway mapping off the container's gateway address is not the host, so a listener on the host's loopback (the worker's metrics port, a local database) is unreachable from a step. Every other address the worker account can reach is reachable, including other hosts on its networks and host services bound to a non-loopback address; firewall the worker account where that matters. The setting is per worker, never per job: a tenant cannot ask for it, the worker advertises the scheduling label egress, and a pipeline that needs the network selects such a worker with runs_on: {labels: [egress]}. The worker logs job_egress_enabled at start. Grant it to pools that run trusted code only: like every rootless container, a job shares the kernel (plan). crates/sentinel-worker/tests/podman.rs (a_granted_egress_container_has_a_route_out_but_cannot_reach_the_hosts_loopback) checks that an egress container has an interface and a default route but reaches neither a host loopback listener directly nor through its gateway.
A job's scratch disk is accounted by placement: the controller fits resources.disk against the disk the worker advertises (its free space less an eighth, 512 MiB to 2 GiB). A worker that could not measure its disk advertises zero, which is no disk at all: no job declaring scratch disk (every compiled job does) is placed on a protocol-7 or later worker that advertises it. The offer itself carries no disk figure, so the worker's own check is a watermark on the data directory's file system: while its free space is below executor::admission_floor — one eighth of the file system, kept between 512 MiB and 2 GiB, the same reserve the advertised disk leaves; Executor::set_admission_floor overrides it — every offer is refused and the controller places it elsewhere or waits, with offer_refused_low_disk logged at most every 30 s. A file system that cannot be measured is admitted: an unmeasured disk is not a full one. The check is one statvfs per offer, outside the state lock. Per job, Podman can bound only what it can meter: --ulimit fsize caps any single file at the declared disk, and /tmp is a 1 GiB tmpfs. Rootless Podman has no storage quota on a bind mount (no io delegation, see feasibility probes), so the total a job writes into its workspace is bounded by admission and by the declared figure the controller placed it with, not enforced per byte; the spool and the cache keep their own quotas.
podman::pull is the only way an image arrives — the worker checks podman image exists, then runs an explicit podman pull --authfile … for a pinned name@sha256:… reference under IMAGE_PULL_TIMEOUT. The result of the first check is the image_present signal. A credentialed pull, or any pull into a tenant's private store, contacts the registry even for a resident digest, so a warm digest never skips that tenant's authorization. An anonymous pull into the shared store uses a resident digest without asking the registry (podman::registry_check_needed): it carries no authority to check, the shared store only holds what was anonymously pullable, and a digest names immutable content. Asking anyway would cost a registry round trip per job (about 1 s) and exhaust Docker Hub's anonymous limit of 100 manifest requests an hour. images::Images keeps one in-flight pull per exact reference, tenant and credential fingerprint. Matching attempts in the same tenant share a pull; another tenant or rotated credential must authorize its own pull. Followers wait on their leader's result, bounded by their own deadline and stopped early by their cancel flag. A stale slot whose leader vanished is dropped by the first follower whose deadline passes, so nothing accumulates and a failure retries fresh on the next attempt. A tag never reaches this path.
Preparation overlaps the pull with the checkout — they touch different state (the content store versus the fresh workspace): the checkout runs on its own thread while the attempt thread pulls, joined before Container::start. A checkout that has already failed still skips the pull, a cancel still aborts preparation (the checkout is joined first — it owns the workspace until it ends), and when both fail the checkout's error is reported, exactly as when the two ran in sequence. checkout_ns and image_pull_ns each measure their own phase's wall time, so their sum can exceed the wall-clock time preparation took.
Pulls run as the worker's rootless user with an explicit auth file from the acknowledged attempt's secret bundle. The controller resolves the job's declared registry_auth secret inside the same writer transaction that checks the live worker, attempt and fence; its version and attempt are audited. The secret is host-only and is never added to a step environment. Pulls remove REGISTRY_AUTH_FILE and DOCKER_CONFIG, pass the scoped --authfile, clear PATH after resolving the Podman executable, and accept only an OCI auth JSON object containing auths. This prevents ambient auth files and named external credential helpers from authorizing one tenant with another tenant's worker credentials. A credentialed pull always contacts the registry for the manifest and config, including when the digest is already in the local store; resident layers are reused without a fetch, so authorization is per manifest, not per layer. A credentialed pull therefore goes to the tenant's private store (podman::Store::Private, --root/--runroot on every command for that image and its container) and never to the shared one, which only anonymous pulls fill (secrets). A public prefetch uses an empty scoped auth file and lands in the shared store; only shared-store pulls join the held set placement reads. An unavailable external helper fails closed, so workers should move private-registry access to the per-tenant registry_auth binding.
Secret environment values are exported by a /bin/sh wrapper from a file mounted read-only at /run/sentinel-env, never passed to podman exec (which would write them into its on-disk exec spec); file secrets appear read-only at /run/sentinel-secrets/<path>, readable by a non-root image user. Both are staged under the worker's runtime directory (tmpfs when XDG_RUNTIME_DIR is usable), outside the workspace, cache mounts and artifact roots. Only values needed by the current step are materialized, and the container is cleared of every process but its keepalive before such a step and after it (Container::clear_strays: freeze the cgroup, SIGKILL, thaw; about 2.3 ms), so no other step's process sees them. Containers are created with --ulimit core=0:0. The log redactor registers every delivered value before a log pipe opens and retains enough bytes to redact matches split across read frames. Files are zeroed and removed after their step; the attempt's private secret directory and registry auth file are removed on success, error, cancellation and panic. Startup recovery reaps leftovers from a worker crash before another attempt can use them. The current active secret version is selected for every new attempt, including a rerun; historical versions remain auditable but are not implicitly replayed.
Bounded prefetch. An offer names only the image digest, so a pull cannot start from an offer; the controller instead hints references ahead of placement. After every dispatch pass (at most every 500 ms), dispatch::prefetch_hints reads the longest-waiting ready jobs (at most 256, off the jobs_queued_since index) whose image name and digest are on the job row (jobs.image_name, migration 36) and picks, per idle or underused protocol-9 worker, up to 4 name@sha256:… references of jobs it could be placed — its tenant may use the worker's pool, architecture, labels, CPU, memory and disk fit, the worker is not draining and does not already report the image warm — hinting each image to at most 2 of the most idle such workers. The session sends Prefetch { images } (the whole wanted set) only when that set changed, empty included. On the worker, prefetch::Prefetcher rebuilds each reference from its parts (digest-pinned, reference charset, no tag), replaces its wanted set, and pulls in the background through the same Images single-flight lane. Prefetch uses no tenant secret; each actual attempt performs its own authorized pull before using a resident digest. The pull lands in the held set, the next profile refresh reports it warm, and a profile refresh wakes placement. Bounds: one prefetch pull at a time and none while an attempt leads a pull of its own; a byte budget of 4 GiB per 10 minutes charged with each pulled image's real size (podman image inspect), since podman pull has no rate limit; a pull starts only while podman's graph root keeps a tenth of its filesystem (at least 2 GiB) free, and never when that cannot be measured; IMAGE_PULL_TIMEOUT per pull. A reference that leaves the wanted set is stale: not started, it is dropped; under way, its podman pull is killed — unless an attempt already joined it. A failed or skipped reference is retried only when a later hint names it again. Measured: with a fake 300 ms pull over the real link, a job a locality hold kept from an idle cold worker was offered to it after 5.0–5.2 s instead of 30.0 s (the hold's bound; the refresh waits for the next heartbeat); with rootless Podman on WSL2, an attempt's pull of the 123 MB python:3.12-slim took 2.25–2.39 s cold and 29–52 ms after a prefetch (tests/prefetch.rs in sentinel-link and sentinel-worker; bench/b04-transfer-copies.jsonl).
Every confirmed pull records the digest in a bounded held set (images::MAX_HELD, oldest-first eviction) exposed by Executor::images().held() — the locality record every session's profile advertises for placement (the newest 64 as 8-byte keys, refreshed from protocol 8 when it changes; worker link). It is learned from pull outcomes alone, never scanned at startup: a restarted worker's set is empty and relearns on the next pull, whose image exists check is the same authority either way.
attempt::run is the lifecycle the controller's state machine expects, reported under the attempt's fence: PreparationStarted → workspace, checkout, image pull, container start → StepsStarted → each step in order → FinalizationStarted → artifact capture while the workspace still exists → teardown → durable log completion (the spool is kept until the controller's LogEndAck on protocol 5, or the LogEnd send on older links — logs) → Passed or Failed(class):
| What happened | Class |
|---|---|
| fetch, pull or container start failed | Preparation |
| a step exited non-zero | CommandFailed |
| a step died from a signal | CommandSignaled |
| a step passed its timeout | ExecutionTimeout |
| cancel seen between steps | Canceled |
a required artifact absent or unpublished |
Publication (infra_failed) |
executor::Executor is the link's Executor: it takes an offer when the runtime is usable and fewer than MAX_LIST_ITEMS attempts are held — a repeated offer of an attempt it already holds or awaits is re-acknowledged, never started twice — asks for the run spec over the link (NeedSpec → Spec chunks of 48 KiB, at most 1 MiB), decodes it and starts the attempt on its own thread. A spec request that goes unanswered is asked again every SPEC_RETRY (10 s); an attempt still without a spec after SPEC_DEADLINE (60 s) is handed back with Decline — it never started, so the controller requeues it (Notice::HandedBack); a NoSpec is definitive and the attempt is reported Failed(Preparation) at once. Reports go out on the live session in order; without a session they queue in memory and are replayed at the next attach (a durable spool is not part of the executor). A stop from the controller flips the attempt's cancel flag and removes its container off the heartbeat thread; it is not reported, because the controller already counts the attempt as gone. A panic in an attempt's thread is contained: its container and workspace are removed, what it printed is delivered and its log closed (the spool goes with it), and it is reported Failed(Runtime) instead of staying held until the controller's backstop. Every attempt's spool draws on one SpoolSpace: a quota on the bytes all spools hold together and a free-space reserve on the data directory (set_spool_limits, from the worker's spool_quota_bytes and spool_reserve_bytes; defaults 4 GiB and 1 GiB). Output past either is declared as gaps in the attempt's log, never lost in silence, and Notice::SpoolRefused reports the refused frames by cause when the attempt finishes (logs).
Secret values are redacted per attempt: Executor::register_secret(attempt, value) registers into that attempt's log pipe — immediately, also while it runs, or before it starts when it is still waiting for its spec — and returns whether the value was accepted (at least MIN_SECRET_BYTES). Nothing registered for one attempt touches another's output, and it goes with the attempt. The verdict detail and the stored summary pass through the same redactor, and a failed step's stderr excerpt is redacted from the raw tail before it is decoded, reduced to its last line and cut to 200 characters — multi-line values line by line, a wrapped ring without its first line — so neither a whole value nor a line of a multi-line one, a value with a control character, a binary value, or one straddling the cut reaches attempts.summary or the worker's own log. A value shorter than MIN_SECRET_BYTES (4) is refused by the redactor, and a bound secret that short fails its attempt as Preparation instead of being delivered unredacted; a value of up to 16 KiB is also redacted as base64 and percent-encoded (logs); a value the step itself printed only in part, or transformed in any other way, cannot be recognized by any exact-match redactor.
On the controller, dispatch::report checks that the attempt is held by that worker under that fence and then applies the event through the state machine; a stale or foreign report changes nothing and is counted. A terminal report is also the finalization boundary: the same transaction stamps log_state from the end marker's durability, checks every due artifact declaration against its row — a missing one is recorded failed, and a due required one that never reached captured turns a reported Passed into Failed(Publication) — releases the reservation, decides dependents and wakes the dispatcher. expire, abandon and startup reconciliation stamp the same way, so a lost worker's attempt reports its real data status rather than a defaulted one.
Artifact capture. Once the steps settle, artifacts::capture walks each declared artifact whose when matches the step verdict — the same descriptor-rooted resolver as hash_files (openat2 BENEATH | NO_SYMLINKS | NO_XDEV under the pinned workspace root, shared traversal budgets), so a pattern can never escape into the host or follow a link planted by the job. Each file streams to the controller in ArtifactData chunks — nothing buffers a whole artifact — and the controller stages, digests and commits it into the object store, ending in the {job}/{name} manifest and the durable artifacts row (storage). Capture runs before workspace teardown and before the terminal report, so a controller-side verdict is always known before the attempt goes terminal; a required artifact that ends absent/failed/unpublished converts to Publication, an optional one is recorded and the job keeps its verdict. On a protocol-3 link the sink reports itself incapable and every declaration resolves absent without touching the filesystem.
Steps run in order in the attempt's container, each as the run spec compiled it: /bin/sh -e -c (fail at the first failing command, with its status) or bash -eo pipefail -c (the image must provide Bash); the job's environment, then the step's overriding by name, then the worker's context (SENTINEL_RUN|JOB|ATTEMPT|SHA|WORKSPACE, CI) last so a pipeline cannot spoof it; the working directory joined under /workspace; the step's timeout, or the job's, bounded by what is left of the job's budget so a job cannot outlive its timeout through many steps each within theirs. A step after a failed one never starts and is recorded as NotRun.
A step's if: is evaluated on the worker (Phase::Worker, the expression evaluator of the pipeline schema) against the job context the controller sends with the spec — job.id|name, repo.id|name, run.id, event.sha (the pinned commit), event.ref (the run's ref name), needs.<job>.result and success()/failure()/always()/cancelled() from the dependency outcomes and cancel state — and hash_files over the private checkout. false skips the step (Skipped); a non-boolean, a type error or a value the run does not hold is a Preparation failure naming the step and the reason, never a default. Event fields intake does not record (event.name|key|base_ref|pr_number) are exactly that case.
The verdict keeps these distinctions, and the summary records them per step:
| What happened | Failure class | Step outcome |
|---|---|---|
| exit 0 | — | Passed |
if false |
— | Skipped |
| non-zero exit | CommandFailed |
Failed { code } |
| killed by a signal, OOM counter unchanged | CommandSignaled |
Signaled { signal } |
killed and the cgroup's oom_kill counter moved |
OutOfMemory |
OutOfMemory |
| past its (bounded) timeout; the container is stopped | ExecutionTimeout |
TimedOut |
Podman could not run it (exit 125–127 with its Error: line; a shell's own 126/127 stays a command failure) |
Runtime (infra_failed) |
Runtime |
fetch, pull, container start, if unresolved |
Preparation |
— |
| cancel seen between steps | Canceled |
— |
OOM is read from the host's view of the container's cgroup (/sys/fs/cgroup<CgroupPath>/memory.events, path from podman inspect), before and after a failed step: after an OOM the pages that caused it stay charged, so a process exec'd inside to read the counter could be the next victim and the OOM would be misreported as a signal.
Timings. Every phase is measured with a monotonic clock in nanoseconds: checkout, image pull, container start, all steps, finalization, and each step. Absent means not measured. They travel in the AttemptSummary (sentinel-protocol::summary, format byte 3, at most 32 KiB) with the terminal Report and are stored once on the attempt row (migration 15; the trigger refuses a replacement). A summary over the cap is not an error: the controller drops it, keeps the verdict, and the session stays open. checkout_ns stays the whole checkout; checkout_fetch_ns and checkout_materialize_ns split the mirror update from the worktree materialization, and checkout_route says which path produced it (mirrors). image_present records whether the pinned image was already held locally before the required scoped registry authorization pull, and caches carries one bounded record per declared cache: entry: restore outcome and phase timings (lookup/lock_wait/clone/first_touch), file/byte/copied counts, the commit's commit_ns and staged/reused/dirty split with its publish answer, and the costly_hit flag (cache). dispatch::attempt_summary reads it back.
Worker availability. The same bounded sweep that reclaims the cache root — at worker start and after every attempt's finalization, never on a timer — ends by emitting Notice::Availability, a compact snapshot of what this worker could serve: the sorted digests its image store is known to hold (images::held, bounded by MAX_HELD), pulls in flight, and cache-root occupancy (entries, generations, payload bytes held, truncated when the pass hit its work bound). The pass computed those counts already, so the snapshot costs no second walk; the digest list never reaches a log line — only counts do. A nominal hit whose restore paid rebuild-scale cost is flagged separately as Notice::CostlyCacheHit under the documented rule in cache — a warning, never a verdict.
Cancellation, graceful/forced termination, timeouts and lease expiry are in cancellation. Crash reconciliation of owned containers, leftover workspaces and spools is in reconciliation. Cache and secret execution contracts are documented in cache and secrets. Disk quotas on the workspace are not enforced per byte: admission and a per-file cap bound it, see Disk.
Tests of this area live in crates/sentinel-worker:
tests/checkout.rs(any Linux, needsgit): the pinned commit is checked out even when it is not the branch head and only it is fetched; a workspace is never reused; an unknown revision is aPreparationfailure; a transport Git refuses and an option-shaped repository are refused in bounded time; a credential delivered through askpass leaves neither the helper nor the secret behind.tests/images.rs(any Linux, needsgit, no registry): the pull begins before the checkout finishes; a tag-only reference is refused on the shared path. The unit tests ofsrc/images.rscover single-flight pulls, followers' errors and retries, early cancel and the bounded held record.tests/runtime_failures.rs(any Linux,podmanshimmed onPATH): a container whose stop fails is still removed; recovery refuses to proceed when the runtime cannot list or remove what the worker owns.tests/executor_faults.rsand the executor's own unit tests run the real executor against a live controller (tests/support/live.rs): a worker clock a minute ahead never ends live work; a cancel that finds the step still starting is tried again; a stop order never holds the heartbeat thread; a panicking attempt is torn down and reportedinfra_failed/runtime; a shutdown ends the running attempt without reporting it.tests/podman.rsandtests/end_to_end.rsrun only withSENTINEL_PODMAN_TESTS=1as an account with rootless Podman (otherwise they report "skipped"). They check the enforced limits inside a real container, a resident image still going throughpodman pullwith its auth file, exit codes and signals, OOM, timeouts, graceful and forced cancel,if:gating, secret redaction and that no container, workspace or spool is left behind.src/redact.rs,src/logpipe.rs,src/process.rsandsrc/workspace.rsunit tests cover redaction (floor, base64, percent-encoding, size cap, per-attempt registration), helper cancellation and output tails, and workspace removal of read-only trees and planted symlinks.
Not covered by a test of their own: PR_SET_PDEATHSIG (needs a killed parent process), the cgroup freeze around terminate_named and --ulimit fsize (both need rootless Podman).
See worker link for the session, core contracts for the state machine, and development for preparing an account.