Skip to content

cache.md

Latest commit

 

History

History
208 lines (129 loc) · 47.8 KB

cache.md

File metadata and controls

208 lines (129 loc) · 47.8 KB
Raw

Worker-local cache: classes, scopes, manifests and restore

Sentinel keeps its caches on the worker that runs the job: no network fetch on a warm hit, no shared mutable volume between tenants. This document is the metadata contract — which cache classes exist, what boundary a generation is sealed under, how a job receives its private writable view, and the vocabulary a lookup answers with. Publication, garbage collection, compiler persistence and remote hydration each have their own section below; everything here is what they share: paths, formats and reasons.

Why three classes

A cache: entry declares a class (sentinel_protocol::cache::Class) because the classes have different validity rules, and a permissive rule must never loosen an exact one:

  • downloads — tool-addressed blobs: package registry responses, fetched archives. The installing tool validates content by its own digest or protocol, so an entry may serve a different key under the same stem (cargo-<h1> serving cargo-<h2>): lockfile edits that leave the stem intact can still hit. Compat::Downloads { tool } pins the installing tool's identity (cargo, npm, …).
  • dependencies — an exact materialization of a dependency set. It may serve only a request whose recorded inputs match completely: the rendered key, the lockfile digest, the installer identity, its flag string and the ABI tag (Compat::Dependencies). This is the default for a class-less document — the only rule that is never wrong to assume.
  • compiler — compiler intermediates in a stable toolchain namespace. The compiler performs input-level invalidation inside the namespace, so entries persist across source and lockfile edits; the store pins the key stem and the flags namespace (Compat::Compiler).

The boundary

Every generation is sealed under a scope — and the manifest records every dimension of it, so a directory moved anywhere else can never serve:

cache/<repo>/<class>/<trust>/<os>-<arch>/<toolchain16>/<name>/<entry>/gen-<unix_ms>-<rand>/
cache/<repo>/<class>/<trust>/<os>-<arch>/<toolchain16>/<name>/<entry>/current
cache/<repo>/<class>/<trust>/<os>-<arch>/<toolchain16>/<name>/<entry>/writing/
cache/<repo>/<class>/<trust>/<os>-<arch>/<toolchain16>/<name>/<entry>/lease/

The cache directory name is CACHE_DIR ("cache"); attach::ROOT_DIR is an alias of the same constant.

  • <repo> leads: repository IDs are globally unique, so two tenants can never produce the same scope directory. The tenant is still recorded in every manifest — a moved directory fails WrongTenant, not the path check.
  • <class> and <trust> are the enum names: downloads/dependencies/compiler and protected/pull_request/unprotected.
  • <os>-<arch> is one component (linux-x86_64); a second OS can never silently reuse a scope directory.
  • <toolchain16> is the first 8 bytes of the BLAKE3 digest of the toolchain descriptor — image digest, tool versions, whatever pins "the toolchain" — as 16 lowercase hex characters.
  • <name> is the pipeline's cache name, already valid_id-shaped; Scope::new re-checks rather than trusts.
  • <entry> is a readable slug of the rendered key plus 8 bytes of its digest: different keys can never share a directory, and a slug collision only produces a WrongKey miss — never wrong content.
  • gen-<unix_ms>-<rand> names a generation: creation order in the name, an 8-hex random suffix so two generations sealed in the same millisecond never collide. Reclamation sorts evictable generations by name and relies on that being creation order, which holds only while the millisecond timestamp keeps its digit count, so any change to the generation name format must preserve a lexicographically sortable time prefix. current names the live generation; writing/ holds generations being staged plus the writer lock; lease/ holds the active-use pins that keep a generation away from reclamation.

Trust derivation

Trust is keyed to who could have produced a run's content, not merely to its event name. Trust has three values with one rule (Trust::derive, applied once by provenance::cache_trust for dispatch::job_context and the remote-cache gate):

  • protected — a push or tag that arrived through verified intake (a recorded webhook or ref-poll delivery, never the API) for a ref the repository's live source binding names exactly in allowed_refs (no wildcard). Name the default branch, and any release refs whose builds may seed protected state, explicitly — e.g. allowed_refs = ["refs/heads/main", "refs/heads/*", "refs/tags/*"] builds every branch but gives only main protected cache state.
  • pull_request — pull-request runs: content a fork can influence.
  • unprotected — everything else: manual API runs (the caller chose the checkout URL, SHA and possibly the whole pipeline — the repository RUN bit is enough), pushes and tags admitted only by a wildcard pattern (anyone with push rights to any branch), runs of an unbound or revoked repository, and runs with no recorded provenance.

Each class reads and writes only its own scope directory and manifests record it, so neither untrusted class can plant node_modules, a .venv or compiler output a protected build restores, and neither reads protected state. Workers that negotiated protocol 8 receive the class as is in Context2; a protocol 6–7 worker cannot carry unprotected and is sent pull_request for such a job (the controller's remote-cache gate accepts that downgrade for it); protocol ≤5 workers get Context and scope to pull_request. Every downgrade is toward the scope that can never touch protected state. A trust byte that does not decode is a protocol error, never a guess.

The manifest

Each generation directory carries manifest: sentinel.cache magic, a format byte (1), a reserved zero byte, then a postcard body — 16 bytes of head, at most MAX_MANIFEST_BYTES (64 KiB) ever read. The manifest records the full boundary (tenant, repo, class, trust, platform, the complete toolchain digest, the cache name), the rendered key, the class's Compat inputs, the seal timestamp, and the files blob's entry count, total bytes and BLAKE3 digest. sealed_ms == 0 marks an interrupted write; an unsealed generation is always a miss.

The sibling files blob (sentinel.files magic, format byte 1, at most MAX_FILES_BLOB_BYTES, MAX_FILE_ENTRIES entries) lists each payload file: path, size, digest, mode. It is separate because lookups read manifests and materialization reads listings — a 200k-file listing must never inflate the lookup path. Every entry is re-validated as a normalised relative path, so a listing can never name a file outside its generation.

Explainable outcomes

A lookup answers an Outcome, never an error: Hit (the verified manifest plus its recorded payload bytes) or a Miss naming the reason. The check order is the explanation order — the outermost broken boundary wins:

Miss Meaning
absent no directory, current or manifest where expected
corrupt short head, bad magic, undecodable body, unreadable or oversize file
unsupported_version a manifest format this build does not know
unsealed the write never completed
wrong_class sealed under a different cache class
wrong_tenant sealed under another tenant
wrong_repo sealed under another repository
wrong_trust sealed under another trust class
wrong_platform sealed on another os-arch
wrong_toolchain sealed under another toolchain digest
wrong_name names a different cache entry
wrong_key the rendered key differs (or, for downloads/compiler, the stem does)
incompatible the class's own inputs differ: lockfile digest, installer, flags, ABI tag or tool
invalid decoded but semantically wrong: bad IDs, invalid name, a compat payload from another class
unavailable a transient filesystem or lease failure on the restore path — the bytes may be fine and the miss is worth retrying

A cache must degrade a build to a rebuild, never break it: every failure mode above is a reasoned miss on the lookup path, and nothing in the read path panics or guesses.

Restore: private views of immutable generations

A sealed generation is immutable forever — a job never writes into the store. Preparation (attempt.rs, after checkout and image pull, before container start) gives every declared cache: entry a private writable view:

  1. The key renders against the fresh checkout — hash_files reads real bytes — with a 1024-byte cap (attach::MAX_KEY_BYTES). A key that will not render is an explainable miss, not a failure.
  2. entry/current names the live generation; it is read bounded (128 bytes) and must name a gen-<unix_ms>-<rand>-shaped directory.
  3. manifest::lookup reads the manifest and applies the request's boundary — the Miss vocabulary above is the whole answer set.
  4. An entry that has a current is pinned with a lease (Lease::acquire, the attempt id as owner, DEFAULT_TTL) before the pointer is trusted: current is read again under the pin, so a sweep can never take the generation between the lookup and the clone (an entry without current is the common miss and takes no pin). The files blob's BLAKE3 digest is verified against the manifest — that one digest is the trust anchor for the whole payload, so materialization skips per-file hashing (re-reading every byte would defeat reflinking).
  5. Every payload/<i> tree is checked against its listing first; only then is each materialized into the declared path's target, and the Attached carrier — scope, key, compat, generation, outcome, targets, lease, stats — lands on job.caches in declaration order for publish. A clone that fails part-way (a full disk, an I/O error) empties every target again, so a miss never leaves a partial materialization behind; a generation that vanished although pinned is unavailable, not corrupt.

The clone (clone::tree) is capability-detected once per cache root: FICLONE reflink per file where the filesystem answers (cost per file, not per byte), and a bounded 256 KiB byte copy everywhere else. A capable filesystem can still refuse an individual file — that file falls back to copying while the rest reflink. Hardlinks are never used for a job's view: a shared inode would let one job's write corrupt the generation for every other. A sealed payload holds regular files only: publication never encodes a symlink (the files listing has no link kind) and restore refuses a payload tree that has one. A cache that relied on symlinks is not an exact materialization — a Python venv's bin/python, node_modules/.bin shims and pnpm's isolated node_modules layout lose their links; the recipes work around it (recipes). Entries that are neither file nor directory are skipped and counted. Whatever the backend, a job's writes can never reach the sealed generation.

The target mapping is the contract a job sees:

  • A relative declared path materializes inside the workspace and reaches the container through the existing /workspace mount.
  • An absolute declared path gets a private host directory under .sentinel-cache/<name>/<index> inside the workspace plus a bind mount to the declared container path — the job's writes stay private without the workspace mount ever serving outside its tree.
  • On any miss the declared paths still exist as empty writable directories: a job always sees writable cache paths.
  • Path components are re-validated against the schema's rule (.., absolute-inside-workspace and the .sentinel-cache name as a relative declaration are refused invalid), existing components are never resolved through symlinks, and a symlink sitting exactly at a target is replaced with the real directory it hid. An absolute path containing : or , cannot be expressed as a bind-mount spec and is refused invalid too; a refused or uncreatable target is never offered as a mount, so a cache-path error can never fail the container start.
  • Restore runs after the checkout and before the container starts, so nothing but the worker touches the workspace while targets are resolved and filled; every slot a generation claims is cleared without following a link (clone::tree).

Restore is total from the attempt's perspective: nothing on the cache path is fatal. The stats the carrier records — lookup_ns, lock_wait_ns, clone_ns, files, bytes, copied bytes, the backend flag — feed the availability summaries below; a phase that never ran stays absent, never zero.

First touch. After a hit's clone materializes, restore measures one bounded first-read sample per target (attach::Stats::first_touch_ns): open the largest listed file and read its first 4 KiB. The clone's wall time is per-file work — FICLONE returns without touching extents — while the first real read is what faults a cold extent, so that cost is measured separately rather than folded into clone_ns. At most one small read per target; a file that will not open is simply not measured, and a miss or an empty listing records None.

The worker advertises Capabilities::REFLINK in its Hello exactly when clone::detect(<data_dir>/cache) reports Backend::Reflink — the same probe restore uses, so the advertised bit and the backend in use can never disagree.

Publication

Publication is finalization work. The verdict's outcome never depends on it, but its report waits for it (bounded by CACHE_PUBLISH_TIMEOUT): after a job's steps finish, the worker stops and removes the container, then — before its workspace is destroyed — commits each attached cache whose verdict left real state — a passed job, or one that failed its commands (command_failed, command_signaled, out_of_memory). A canceled, timed-out, un-started or infrastructure-failed attempt publishes nothing, and a cache failure never changes the verdict: each entry's outcome is a diagnostic note (cache_published), never a build failure.

Authorization. publish::commit refuses an attachment whose scope trust is not the job's (TrustMismatch), before a single directory is made. There is no cross-trust promotion in v1: a pull-request publication can only ever produce pull_request entries. The entry directory is derived exactly as restore derives it (attach::entry_key — the key stem for downloads/compiler, the full key for dependencies), so the two paths always meet.

One writer per entry. Staging ownership is writing/.lock, taken with create_new — never waited on. A live marker is a busy skip; a marker older than lease::MAX_TTL (by declared expiry and by mtime, whichever reads live shorter) is a dead writer's and is reaped in place. Releasing removes the marker and the writing/ directory while empty, so a release can never pull staging from under the next writer. Reaping renames the dead marker aside and judges the moved marker again: a reaper that raced another and moved its fresh marker puts it back and answers busy instead of deleting it. That judgment of the moved marker, rather than the one first inspected, is why a plain remove_file is not a substitute: it could let two writers stage into one entry. Directory creation for the marker and the lease retries on NotFound and AlreadyExists, because collection renames an unpinned entry aside and a directory can vanish between create_dir_all and the marker write.

Confined reads. Everything below the workspace is content a job could rewrite, so publication trusts only the workspace directory itself (Target::root). Each declared target is resolved beneath it component by component with openat(O_NOFOLLOW); directories are enumerated and classified through their descriptors (fstatat(AT_SYMLINK_NOFOLLOW)); every file is reopened by its path beneath the target's descriptor with openat2(RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS | RESOLVE_NO_XDEV) (per-component O_NOFOLLOW where the kernel lacks openat2) and O_NONBLOCK, and its size and mode are taken from the descriptor actually read. A symlink at any component — planted by a step, raced in mid-publish, or checked in — is never followed: the entry is skipped and counted. The container is stopped before publication starts, so no job process runs while the walk reads; the confinement holds even when a container would not stop. An entry restore refused (invalid) or whose key never rendered publishes nothing (refused).

Staging. The generation is built at writing/gen-<unix_ms>-<rand> — never inside a live generation. The walk skips non-regular and unencodable entries and is bounded in depth (64, the hash_files bound, and the same MAX_WALK_DEPTH restore refuses at, so the two limits cannot change independently: a sealed generation never holds a deeper tree), entry count (MAX_FILE_ENTRIES), blob size (MAX_FILES_BLOB_BYTES), logical payload (one quarter of the worker cache budget captured for the attempt, checked against the planned sizes before a byte is staged; MAX_GENERATION_BYTES, 12.5 GiB, is the default derived from DEFAULT_BUDGET_BYTES) and time: the whole publish batch runs under CACHE_PUBLISH_TIMEOUT (3 minutes) with cancel polled every 256 files. Before staging, the store's filesystem must have room for the bytes that will be copied plus a reserve (5 % of the filesystem, at most 1 GiB), or the commit fails NoSpace. A sparse file (allocated blocks under half its size) is copied hole-for-hole: an all-zero block becomes a seek, so a mostly-empty file never amplifies into dense writes; its digest is still of the full logical content. One 1 MiB buffer serves every hash and copy of a commit. Durability is settled once per commit, before the listing and manifest are written: up to 64 copied files are fdatasynced individually, beyond that one syncfs covers them all (a 20k-file publish measured 0.35 s median under WSL2, against 108 s with per-file syncs).

Reuse. A file whose path, size and mode match the source generation's listing is hashed once; a digest match is staged by hardlinking out of that generation — one inode, no copy — and the staged name is re-hashed before seal, so a drifted generation can never poison its successor: a failed check falls back to copying the job's own bytes. Files that do not match are copied with hashing folded into the copy, so the recorded digest is always of the bytes actually staged.

Sealing and promotion. The files blob is sorted by path, then manifest records bytes, files and files_digest and is sealed. Promotion is two renames: the sealed generation lands inside the entry (durable via directory sync on unix), then current.tmp is written and synced and renamed over current. A reader resolving current sees the old generation or the new one — never a torn name, never a directory still being written. A canceled or failed publish removes its staging and leaves current untouched.

Skip vocabulary. A commit that published nothing answers a stable reason: busy (a live writer holds the entry), canceled (cancel or deadline), empty (zero files across every target), unchanged (the staged listing is identical to the source generation's — the bytes are already published), refused (restore answered invalid or the key never rendered — nothing that could serve).

Measured availability and the costly hit

Every declared entry's carrier (attach::Attached) accumulates what the attempt measured, and finalization stamps the commit's half back onto it: commit_ns plus a Committed answer — Sealed { staged_bytes, reused_bytes } (the dirty side is staged − reused: what the job's view rewrote versus the source generation), Skipped(reason), or Failed. The terminal AttemptSummary (format 3) carries one bounded CacheRecord per entry in declaration order — at most MAX_CACHE_RECORDS (16; the schema caps declarations at 8) — with the class's wire code, the outcome as "hit" or the Miss reason, every restore/commit duration as Option (a phase that never ran is absent, never zero), file/byte/copied counts, the backend flag, the commit's staged/reused/dirty bytes and publish answer ("sealed", a skip reason or "failed", absent when the verdict left nothing to publish), and the costly_hit flag below. image_present records whether the pinned image was already in the worker's store before the attempt's scoped registry authorization pull. The pull still runs for a resident digest; this flag distinguishes a warm local image from one absent before the pull.

The costly-hit rule (attach::Attached::costly_hit) flags a nominal hit whose restore plausibly cost more than rebuilding the dependency would have — because that is exactly when a cache is hurting rather than helping. A hit is flagged when either clause holds:

  • copied_all — the clone backend is Backend::Reflink yet copied_bytes == bytes with a non-empty payload: the filesystem refused every file, so the "hit" paid a full copy anyway.
  • slow — lock_wait_ns + clone_ns exceeds COSTLY_HIT_NS (5 s). A healthy restore — a per-file FICLONE pass, or a warm copy of a typical dependency payload at even 100 MiB/s — is nowhere near that; five seconds is deliberately generous so only implausible restores are flagged.

The flag is a diagnostic, never a verdict: it lands as costly_hit on the summary record and as a cache_costly_hit notice (executor::Notice::CostlyCacheHit) carrying the measured stats as evidence. It never fails an attempt.

The same sweep that reclaims (gc::resume, at worker start and after every attempt's finalization) also produces the worker's availability snapshot — the pass already counts entries, generations and payload bytes, so the snapshot costs no second walk: Notice::Availability carries the sorted held-image digests (bounded by images::MAX_HELD), the in-flight pull count, and cache-root occupancy (entries, generations, payload bytes held — the store-wide estimate even when the pass truncated). The notice is diagnostics; placement is fed through the link instead: every session opens with a Profile whose availability is the executor's record — the newest held image keys (first 8 digest bytes, at most 64) and the estimated cache bytes — and a protocol-8 session refreshes the profile whenever that record changes (worker link). How placement weighs it is the fleet scheduler's concern.

Leases and reclamation

A lease is a pin on an entry (lease/<id>, body <expires> <owner>): while any marker is live, nothing inside the entry is collected — a reader mid-clone or a writer mid-publish is never undercut. A lease reads live while min(declared expiry, mtime + MAX_TTL) is in the future, so a torn or corrupt marker still pins until it is provably old. A restore holds one for the attempt's run, however long: every lease the process holds is registered with a keeper thread that renews it to its own TTL every RENEW_EVERY (2.5 minutes), so a 24-hour job keeps its pin; release is delete-on-drop (deregistered first — the keeper never recreates a released marker), and a crashed holder stops renewing, so its marker simply expires.

gc::resume runs one bounded pass (gc::sweep is the same pass from the beginning, without a cursor) — at worker start (off the start path, after recovery) and after every attempt's finalization, one at a time (WouldBlock skips; the running pass covers the work). The pass is bounded by a work counter (DEFAULT_PASS_WORK, 100k directory entries and removals): hitting the bound stops the pass cleanly, marked truncated. The worker keeps a gc::Cursor between passes: directories are walked in name order and each pass resumes after the last entry the previous one finished, wrapping at the end, so successive bounded passes cover the whole tree however large it is — leases, stale staging and retention reach every entry.

Per entry, in order: sweep expired lease markers; remove stale writing/ trees (no live lock and the directory older than the lease bound); then retention — current's generation stays, the newest other generation stays as the spare, older non-current generations are removed oldest-first. current is the pointer GC trusts: a generation it names is never deleted, and a pointer naming nothing — or carrying bytes that are not a name — is counted stale_current and left for the next publication to overwrite.

Last use and idleness. An entry's last use is current's mtime: publication writes it and every restore hit touches it. An entry idle longer than IDLE_TTL (14 days) is removed whole — current included — whatever the budget; an entry with neither generations nor pointer is an empty shell and goes too. Scope directories a removal leaves empty are removed with it.

Budget. When total payload bytes (per the sealed manifests) exceed the worker's cache budget (cache_budget_bytes; default a fifth of the data file system, clamped to 1–50 GiB by StorageBudgets::sized; DEFAULT_BUDGET_BYTES, 50 GiB, is only the value before the worker applies it), the pass evicts non-current spares oldest-first across entries, then — if that is not enough — whole entries, current included, least recently used first, until under budget or out of candidates. Nothing is exempt but what is in use: each entry's leases (and its staging writer) are re-checked at eviction time, so a reader that pinned mid-pass keeps its generation. A whole entry is first renamed aside and judged again there: a pin taken before the rename is inside the moved directory and puts it back, one taken after it lands in a fresh entry the removal never touches — and since readers pin before trusting current, and every commit pins its entry for its duration, no restore or publication ever works in a directory being removed. The budget is enforced on every pass, truncated or not: the total it is compared against is the larger of what the current cycle has seen and what the last complete cycle measured (less what was freed since), and eviction draws on the candidates the pass gathered, under its own work allowance. The availability snapshot's cache_bytes is that estimate.

Every stat is counted — entries seen, active and expired leases, removed staging, generations, whole entries, freed bytes, stale pointers, failures, truncation — and reported once as cache_swept when the pass did work. A quiet pass stays quiet.

Remote hydration

A worker's warm path stays local: a local hit never traverses the controller or the WAN. Remote hydration is a miss path only — restore::restore_remote consults the transport exclusively in the branch where the local lookup answered a Miss, and never for invalid (a declared path that cannot serve wants no bytes at all). What a hydration installs becomes a normal local generation, so the next compatible attempt hits it locally.

The bundle. One sealed generation travels as a canonical byte stream: a 40-byte head (12-byte magic snc.cache.b1, format 1, three reserved zero bytes, then manifest_len, files_len and payload_len as big-endian u64), the exact manifest bytes, the exact files bytes, then every payload file concatenated in listing order (payload/… sorted by path). The stream's BLAKE3 digest is the bundle id: content addressing, store dedup and resume validation in one value. Every chunk carries the running BLAKE3 of the prefix it extends, so corruption is caught at the chunk that caused it; the whole-stream digest is checked again before anything is installed.

Messages. Protocol 7, preferring the bulk connection; a worker whose bulk connection is down sends them on the control connection instead (the same fallback every bulk-class message has — latency, never correctness). A fetch is CacheNeed { attempt, tenant, repo, class, trust, os, arch, toolchain, name, key, offset, have } (key is the entry key; offset/have are the resume request, the empty digest at offset 0) answered by CacheGrant { total, offset, prefix, digest }, then ordered CacheChunk { offset, bytes, prefix } frames, then the terminal CacheEnd { digest }. An offer is CacheOffer { …, total, digest } answered by a grant (or CacheEnd when the store already holds that digest), then CachePush { offset, bytes } frames, then CachePushEnd { digest }. Refusals are CacheRefused { attempt, code } with a stable code: no_bundle (1), denied (2, no store configured, not owned or tenant/repo/trust mismatch), wrong_scope (3), busy (4), too_large (5, over 64 GiB or no room on the store), store (6), aborted (7, the terminal answer to a cancel). Protocol 8 adds CacheCancel { attempt }: a worker that abandons a transfer — a fetch past its budget or a failed offer — tells the controller, which stops the serve at its next chunk (answering aborted) or drops the upload's staging. Routing is by attempt: a worker keeps an abandoned transfer's attempt draining until its terminal answer arrives, and a new transfer of that attempt is busy meanwhile, so the tail of an abandoned stream can never be routed into the next one; answers queue at most 64 deep, and a caller that falls further behind abandons its transfer rather than buffering without bound. A serve hands the link one chunk per frame — 48 KiB, because a chunk's running digest cannot be re-derived from split pieces — through one buffer per transfer, and a receiver accepts up to 1 MiB, so a wider framing needs no store change. Bundles are capped at 64 GiB.

Authorization. A request or offer is fenced by the attempt it names: the controller resolves it through dispatch::cache_scope (one indexed row plus the trust probe) and requires the attempt to be owned by that worker and the message's tenant and repo to equal the job's and its trust to be one the job's class admits (its own, or the pull_request a pre-8 worker was told to use for an unprotected job) — the same trust boundary the manifest enforces again on arrival. A fetch requires the attempt still held; an offer — sent only after the terminal report released the attempt — accepts one released by the same worker within OFFER_WINDOW_MS (10 minutes). A refusal is denied and counted (Stats::cache_denied); nothing about the store's contents crosses it.

Resume. A hydration streams into a staging generation, <entry>/writing/remote.stage/, under the entry's single-writer marker (writing/.lock) and a lease — a live publisher is a skip, and GC can never take the entry mid-transfer. The stream is written there exactly once, as it arrives: the head, manifest and listing are held in memory (and appended to prefix.part) until whole, then checked against this request and written as the generation's own manifest and files; every payload byte then goes straight into its file under payload/, and each file is checked against its listing entry and given its mode the moment its last byte lands. To resume, the worker re-derives the prefix it holds from that staging tree — prefix.part, or the head rebuilt from manifest and files followed by the staged payload files in listing order — hashes it (within the job's hydration budget) and asks to continue from it; the controller serves from that offset only when its bundle really extends the prefix — and only proves prefixes up to MAX_RESUME_HASH (1 GiB), so a claimed have can never make it hash gigabytes per request — otherwise it answers offset 0 and the worker restarts the staging empty. remote_from records the offset the controller granted. Offers never resume: the controller's upload staging (Receiving::begin) truncates its .part file on every offer, so a retried offer restarts from byte zero, because splicing onto a stream the sender did not reproduce would only fail the end-of-stream digest with no clear cause. Bytes already on disk are never re-sent and never spliced: a prefix that does not match is dropped (corrupt — the staging can never verify), no_bundle drops it, while a transfer cut short by the network, the budget or a failed staging write, or refused for now (busy, denied), keeps what it staged for the next attempt. A legacy single-file remote.part is removed and its transfer restarts cold. Staging that never completed is reaped with the rest of writing/.

The restore-cost bound. A job's hydrations together may spend at most min(5 s, job timeout / 4) (remote::BUDGET_MAX, the job's own allowance read from its spec): one deadline is fixed before the job's first restore and every cache shares it, so N declared caches never cost N budgets. The deadline covers everything a hydration adds — hashing a partial to resume it, the transfer, and the install: once at least 250 ms has been measured the worker extrapolates the measured rate (compared in 128-bit arithmetic, so a fast link never truncates to "free") over the remaining transfer plus, when the job's view is byte-copied (no reflink), the install's copy of the payload into it, and stops when that no longer fits what is left. A stream that completed too late to install stays staged, so the next attempt installs it without a byte on the wire. An aborted transfer is Miss::Unavailable on the entry — a rebuild, never a wait — and keeps what it staged for a later attempt. A stalling link cannot outlive the deadline.

Install and promotion. The head is checked as soon as it is whole, before a payload byte is written: the manifest must decode, be sealed and pass the same Manifest::compatible boundary check the local lookup runs (wrong tenant, repo, class, trust, platform, toolchain, name, key or compat is the same typed miss a local generation would answer, and the transfer stops there instead of after the whole payload crossed the link); the files blob's digest must match the manifest's pin; the listing may only name payload/<i>/… for a declared path, and its sizes must add up to the stream. Every payload file is hashed against the listing entry that names it as it lands. Then the targets are materialized from the staging generation through the same restore::materialize a local hit uses — reflink or bounded copy — and the generation is promoted with the same two renames a publication uses: the directory lands inside the entry, then current is swapped atomically. Until that swap, nothing of the transfer is visible to a reader; a bundle that fails any check never serves, and no staging of it survives.

Measured with tests/remote_cost.rs (release, WSL2, 256 MiB payload of 64 files, in-memory controller, 7 runs; bench/b04-transfer-copies.jsonl): on ext4 with the byte-copy backend a hydration reads the payload once and writes it twice (staging, the job's view): 367 and 430 ms medians over two series (339–545 ms), and 792 and 1,229 ms (548–1,903 ms) including the syncfs write-back. Staging the stream through a separate partial first, which wrote the payload three times and read it twice, measured 495.6 ms (376–599 ms) and 1,778 ms (725–4,847 ms). On a reflink XFS a hydration writes the payload once and reads nothing back: 741 ms median, against 802 ms with the separate partial.

Offers. After a verdict's terminal report has left — never before, and never on its critical path — the worker may offer each generation it sealed to the controller: offer the metadata, stream the canonical bytes. A protocol-9 controller takes the stream's digest in the end marker: the offer names DIGEST_AT_END (all zero) in CacheOffer, the worker hashes the bytes as it pushes them and states the digest in CachePushEnd, and the controller verifies it against its own running hash before storing under it — the generation is read once (156.7 ms median for 256 MiB in the harness above, against 216.6 ms with a separate hashing pass). An older controller needs the digest up front, so the worker hashes the generation in a pass of its own first. Such an offer cannot be answered "already stored" before its bytes flow; that dedup only ever fired for a re-offer of the very same generation, because the sealed time in the manifest is part of the stream. Pushes are encoded straight from the read buffer (PushRef, byte-for-byte the CachePush message), not copied into an owned chunk first. The controller accepts an offer for an attempt the same worker released within OFFER_WINDOW_MS (10 minutes): the offer always follows the report that released it. The whole offer is bounded by OFFER_BUDGET (30 s) and the attempt's cancel flag; a refusal, a lost session or a passed deadline simply drops it. The controller's store mirrors the worker layout under <data_dir>/remote-cache: <repo>/<class>/<trust>/<platform>/<toolchain16>/<name>/<entry>/<bundle-id>.bundle, with current naming the bundle id a fetch is served. An incoming offer is admitted only when the store's filesystem has room for it plus a reserve (5 %, at most 1 GiB), stages under the same writing/ marker, is hashed as its bytes land (no re-read at the end on the session's thread), and promotes by rename plus the current swap, after which the bundle it superseded is removed — an entry keeps one bundle; an offer of a digest already stored is answered as stored and changes nothing. A connection holds at most four uploads; one idle for 60 s, cancelled, or abandoned with its connection is dropped together with its staging file and entry lock.

Controller store reclamation. Setting the remote-cache root starts a reclamation pass every 10 minutes (remote::sweep_store): superseded bundles and upload staging older than an hour go, entries with nothing to serve go, and while the kept bundles exceed STORE_BUDGET_BYTES (50 GiB) whole entries are evicted least recently served first (every serve touches current). An entry with a live upload is never touched. Freed and kept bytes are exposed on the controller's stats.

What is never served. Corrupt content (a chunk that fails its running digest, a stream that fails its digest, a listing that disagrees with the manifest) and wrong-scope content (any boundary or compat mismatch) are typed misses, and a hydration neither materializes nor promotes them.

Measured. A hydrated restore records three facts on its Stats: remote_ns, the transfer's wall time (a duration, absent when no transfer ran — a local hit leaves it None); remote_bytes, the bytes actually delivered (a count, 0 when none, never counting a resumed prefix); and remote_from, the offset a resume began at (absent for a cold transfer and for no transfer at all).

Compiler state

A class: compiler entry is a stable toolchain namespace whose tool owns per-input invalidation: Sentinel pins where compiler state lives and guarantees a writable view on every attempt, while the compiler (ccache, sccache, rustc's incremental store, go's build cache) decides per input what is still fresh. That split is what lets compiler state persist across small source and lockfile edits — a complete source hash in the outer key would destroy reuse that the tool's own content keys already express inside the namespace.

Stem serving. attach::entry_key maps a compiler request to its key stem — everything before the last - of the rendered key — so the entry directory and its current pointer span every volatile tail under one stem, and Manifest::compatible re-verifies the stem against the recorded key. A generation sealed under cc-normal-<h1> therefore serves a request for cc-normal-<h2>: a one-line source or lockfile edit moves the hash tail, not the stem, and the restored view carries whatever the last publisher left — stale inputs and all, for the tool to reconcile.

Mode namespacing. The convention is cc-<mode>-<hash>: cc-normal-…, cc-race-…, cc-coverage-…, cc-experiment-… are four stems, hence four entry directories with four current pointers. Instrumented and uninstrumented outputs can never serve each other, because the store-level boundary is the stem itself rather than a flag a step might mis-set. Keep exactly one volatile component last: everything before the final - is the namespace, so a second hash_files inside it (say, of the lockfile) makes that input part of the namespace — a deliberate choice, not an accident.

Architecture and toolchain isolation. linux-x86_64 versus linux-aarch64 are different scope-path components and different manifest platforms: a cross-arch lookup answers absent, a manifest moved across answers wrong_platform — never a cross-arch serve. The <toolchain16> component pins the image digest or tool versions the same way, so a toolchain upgrade is a new namespace too.

flags is the recipe channel. Compat::Compiler { flags } is a pin inside a stem for recipes that want a check tighter than the stem provides. A bare declaration leaves it "": the stem is the whole namespace — it already picks the entry directory and is re-verified in the manifest — so recording key_stem(key) there would only ever duplicate the check just run, and a stem longer than the 256-byte field bound would make the manifest unreadable for no benefit. A recipe that needs a flag-regime split under one stem pins a rendered token instead. Note that current is one pointer per entry directory: regimes that must not interleave belong in the stem, since two flag namespaces sharing a stem would contend on the pointer.

A hit is never a cached verdict. A compiler-cache hit restores bytes — the writable view the job's steps read — and nothing else. execute runs every declared step unconditionally; job.caches is consulted only by restore during preparation and publish during finalization, and the Outcome::Hit type itself carries only a manifest and a byte count. Go's documented build-cache/test-result distinction applies: forcing fresh test execution (-count=1) does not require discarding compiler reuse. A cache hit can never suppress a test, skip a step, or inherit a previous run's verdict. The Linux fixture fixtures/compiler/fakecc.sh — a content-keyed compiler stand-in — exercises this through real container mounts and restore/publish round-trips in sentinel-worker/tests/compiler_cache.rs and sentinel-cache/tests/compiler.rs.

Recipe library

recipes publishes nine tested pipelines — a custom compiler tool, Go, Rust, npm, pnpm, Bun, Python, Maven and Gradle — showing how each toolchain's stores map onto these classes, which environment variables wire the mounts, how lockfile and toolchain changes move through the key and scope, and the measured cold/warm/small-edit/changed-dependency/changed-toolchain behavior.

Verification

The adversarial tests live in crates/sentinel-cache/tests/k09.rs (store level, threaded, both OSes) and crates/sentinel-worker/tests/k09.rs (executor level, gated on SENTINEL_PODMAN_TESTS like the rest of the suite). What each case proves:

  • Concurrent clone/write isolation: concurrent_clones_during_republishing_are_never_torn. Eight readers loop restore on one entry while a writer republishes it and gc::sweep runs under a deliberately tiny budget. Every cloned view reads as exactly one generation's bytes, the only answers are hit or a typed miss (absent, corrupt, unavailable), and every surviving generation re-verifies afterward (manifest, listing digest, every listed file's digest).
  • Canceled writers: a_writer_canceled_mid_stage_promotes_nothing_and_the_store_recovers. The cancel trips inside materialization, not the walk. Nothing promotes, current does not move, no staged tree survives; a dead writer's remains are reaped once provably old (a still-warm one never), and the next publish seals cleanly.
  • Eviction during active use: eviction_during_active_use_never_undercuts_a_lease. Under a forcing budget the leased entry keeps both generations while the unleased sibling loses its spare; once the pin drops, the next sweep takes it. The mid-clone pin is tests/gc.rs::a_reader_mid_clone_keeps_its_generation_through_budget_eviction.
  • Tampered state: tampered_state_is_a_typed_miss_and_republish_heals. A rewritten files blob, a flipped manifest byte, a garbage current, a payload file the listing never recorded, a listed file deleted, and a listed file swapped for a symlink are each a typed miss, never a panic or a wrong serve, and a republish heals the entry. This holds because restore::materialize verifies each payload/<i> tree is exactly the sealed listing (check_payload_tree) before cloning: the listing is the payload's whole authority.
  • Trust change: protected_state_never_serves_or_writes_pull_request and the executor-level a_pull_request_attempt_never_touches_protected_state. Protected bytes answer absent through the scope path and wrong_trust through a moved directory; a pull-request attempt seals only pull_request state and leaves the protected entry byte-identical; a wrong-trust commit is refused before a directory exists (also tests/publish.rs).
  • Disk pressure: a_full_filesystem_fails_the_publish_cleanly (Linux only, gated on SENTINEL_MOUNT_TESTS=1 as a mount-capable account). A real ENOSPC mid-publish on an 8 MiB tmpfs promotes nothing, leaves current untouched, removes staging, and seals cleanly once space returns. Its executor half is sentinel-worker/tests/k09.rs::a_failing_publish_is_a_note_never_the_verdict: a publication that cannot take the write lock still passes the job, and the failure lands as a cache note and publish: "failed" on the summary record.
  • Concurrent writers on one entry: parallel_writers_are_serialized_and_never_double_promote. A held writing/.lock makes every commit busy; raced behind a barrier with a deliberately slow stage, exactly one writer stages at a time, the loser skips, and current never double-promotes.
  • Remote hydration: tests/remote.rs drives the real miss path against an in-memory controller implementing the link's Remote seam. A cold transfer serves the job and installs a generation the store then serves locally, so a later attempt never calls the transport; an interrupted transfer resumes exactly at the offset it stopped; a corrupted chunk and a bundle sealed for another trust are typed misses (corrupt, wrong_trust) that materialize nothing and leave no current; a controller without the bundle leaves the local miss, partial and pointer untouched. The same file drives the controller store itself (one bundle per entry, the byte budget, abandoned uploads, verify-as-it-lands, proven resume prefixes, a kept partial on busy, one job deadline). sentinel-link/tests/remote_cache.rs runs the whole path over TLS against a real Controller: an offer after the terminal report is stored and hydrates the next attempt, an abandoned fetch is cancelled and the next fetch of that attempt is clean, a foreign worker's offer is denied, and the profile carries and refreshes measured availability.
  • Confinement and bounds: tests/publish.rs proves a symlink at any component (planted, raced between plan and copy, or checked in) is never followed, a refused entry publishes nothing, the byte cap refuses before staging, and a sparse file stays sparse. sentinel-worker/tests/k09.rs::publish_runs_with_the_container_stopped proves the container is gone when publication reads the views. tests/gc.rs proves currents are evicted least recently used under the budget, idle entries age out, and bounded passes resume to cover the whole tree while still enforcing the budget.

Lease staleness and writing/ reap boundaries (tests/gc.rs), walk-phase cancellation and trust-mismatch refusal (tests/publish.rs) and malformed compiler manifests (tests/compiler.rs) are covered by their own tests. The mount-gated case reports its skip rather than passing silently; on hosts without mount(2) it is left to a staging host's run.

Versioned surface

The manifest and files blob formats are versioned contracts under compatibility; the miss-reason strings are the report vocabulary and change only there too. The remote bundle stream (magic snc.cache.b1, format 1) is a versioned transfer contract in the same sense. The pipeline-facing half — cache.class, its default, and the compiled digest — is schema 1 behavior documented in pipeline schema.