fix(git)!: keep the branch worker's decided writes pending, with one materializer

.coverage-baseline

@@ -1 +1 @@
-79.5
+79.9

README.md

@@ -80,6 +80,10 @@ several source clusters, repositories, branches, and folders.
 - **Mirror only the objects you label.** A rule's `objectSelector` lets the API server choose which
   objects of a type are mirrored; an object that loses its label leaves the mirror like a deleted
   one. See [selecting objects by label](docs/configuration.md#selecting-objects-by-label-rulesobjectselector).
+- **Each object reaches a target once.** A target watches each object through one collection, so an
+  edit is never delivered twice and a status-only update never splits a commit. A rule that would
+  overlap another on the same target is refused, naming both. See
+  [overlapping collections](docs/configuration.md#overlapping-collections).
 - **In-place edits keep the shape of your file.** Updates preserve key order, comments, and
   untouched documents in multi-document files.
 - **Inspect a repository before you point at it.** `manifest-analyzer --mode scan-repo` classifies

charts/gitops-reverser/README.md

@@ -176,8 +176,8 @@ nodeSelector:
 | `servers.audit.tls.secretNameOverride` | Override Secret name for audit TLS cert/key | `<release>-audit-server-cert` |
 | `controllerManager.additionalSensitiveResources` | Extra Secret-shaped resource types encrypted as `resource` or `group/resource` | `[]` |
 | `controllerManager.insecureAllowGitHTTP` | Permit credentials with `http://` GitProvider URLs for trusted in-cluster development Git servers | `false` |
-| `controllerManager.branchWorkerQueueDepth` | Event queue depth per branch worker. A full queue drops the write (`git_queue_drops_total`) instead of stalling the watch path, so the live attributed commit is lost even though convergence heals the mirror. Costs queue depth × payload of pod memory, which `branchBufferMaxSize` does **not** cover | `1000` |
-| `controllerManager.branchBufferMaxSize` | Maximum in-memory event buffer per branch worker. Bounds the open commit window plus the writes retained for replay until a push succeeds | `8Mi` |
+| `controllerManager.branchWorkerQueueDepth` | Event queue depth per branch worker. A full queue refuses the write (`git_queue_refusals_total`) instead of stalling the watch path; the watch delivers it again from its cursor, and only a cursor that expires first loses that write's own commit (and, under `prune.mode: OnEvent`, a delete's removal). Costs queue depth × payload of pod memory, which `branchBufferMaxSize` does **not** cover | `1000` |
+| `controllerManager.branchBufferMaxSize` | Maximum in-memory event buffer per branch worker. Bounds the open commit window plus the writes retained for replay until a push succeeds. While the remote cannot be reached, retained writes that reach it close the worker to new writes, saves and resyncs until they are published; the watch delivers a refused event again | `8Mi` |
 | `controllerManager.commitRequestTTL` | How long a `CommitRequest` is kept once it finishes. Written onto the request as the `configbutler.ai/delete-after` annotation; remove or edit that to keep one. `"0"` writes none | `48h` |
 | `git.refreshInterval` | How often an idle `GitTarget` re-proves where its branch is on the Git remote. `"0"` disables it | `10m` |
 | `auditService.type` | Service type for the dedicated audit Service | `NodePort` |

charts/gitops-reverser/values.schema.json

@@ -107,7 +107,7 @@
         "branchWorkerQueueDepth": {
           "type": "integer",
           "minimum": 1,
-          "description": "Event queue depth per branch worker. A full queue drops the write (git_queue_drops_total) rather than stalling the watch path, losing that live attributed commit. Costs queue depth x payload of pod memory, which branchBufferMaxSize does not cover."
+          "description": "Event queue depth per branch worker. A full queue refuses the write (git_queue_refusals_total) rather than stalling the watch path; the watch delivers it again from its cursor. Costs queue depth x payload of pod memory, which branchBufferMaxSize does not cover."
         },
         "branchBufferMaxSize": {
           "type": "string",

charts/gitops-reverser/values.yaml

@@ -68,23 +68,27 @@ controllerManager:
   # writing to one (GitProvider, branch), so pointing another GitTarget at a branch
   # divides the headroom of a worker already in use.
   #
-  # A full queue DROPS the write and counts git_queue_drops_total, rather than stalling
-  # the watch path behind a slow remote. Convergence heals the mirror afterwards, but the
-  # live attributed commit for a dropped write never happens — so raise this if
-  # git_queue_drops_total is ever non-zero. Size it so a bounded burst cannot overrun it:
+  # A full queue REFUSES the write and counts git_queue_refusals_total, rather than stalling
+  # the watch path behind a slow remote. The watch keeps its cursor and delivers the write
+  # again, but its stream stalls meanwhile, and a cursor that expires first loses that
+  # write's own commit (and, under prune.mode OnEvent, a delete's removal) — so raise this if git_queue_refusals_total is non-zero on a healthy
+  # branch. Size it so a bounded burst cannot overrun it:
   # roughly (concurrent writers) x (GitTargets sharing the branch worker). Note that an
   # administrative burst (an unpaced `kubectl delete` of many objects) is far spikier than
   # the steady-state traffic the depth gauge shows you.
   #
-  # Costs pod memory ON TOP of branchBufferMaxSize, which does NOT cover this queue: that
-  # cap is accounted only after an item leaves it. Budget queue depth x serialized size
+  # Costs pod memory ON TOP of branchBufferMaxSize on a healthy branch, where that cap does
+  # NOT cover this queue (during a Git outage it does). Budget queue depth x serialized size
   # x ~6 (a queued object is held as an unstructured map), per SATURATED worker — about
   # 5Mi at this default for ~800-byte resources, but the multiplier is on the payload, so
   # size it against your own if you mirror megabyte-sized ConfigMaps or Secrets.
   branchWorkerQueueDepth: 1000
   # Maximum in-memory event buffer per branch worker, as a Kubernetes resource quantity.
   # Bounds the open commit window plus the writes retained for replay until a push
-  # succeeds; tripping it finalizes the window early, ignoring the commit cadence.
+  # succeeds; tripping it finalizes the window early, ignoring the commit cadence. While the
+  # remote cannot be reached, it also bounds everything the worker accepts, queued work
+  # included: work that would exceed it pauses the worker's intake of new writes, saves and
+  # resyncs until a push lands, and the watch delivers what was refused once intake reopens.
   branchBufferMaxSize: 8Mi
   # How long a CommitRequest is kept once it finishes, as a duration. Every save leaves one object
   # behind, so without this they accumulate for as long as the namespace lives. The controller

cmd/main.go

@@ -633,9 +633,10 @@ func parseFlagsWithArgs(fs *flag.FlagSet, args []string) (appConfig, error) {
 		git.DefaultBranchWorkerQueueDepth,
 		fmt.Sprintf("Event queue depth per branch worker (default %d). ", git.DefaultBranchWorkerQueueDepth)+
 			"A branch worker is shared by every "+
-			"GitTarget writing to one (GitProvider, branch), and a full queue DROPS the write "+
-			"(git_queue_drops_total) rather than stalling the watch path behind a slow remote, "+
-			"losing that live attributed commit even though convergence later heals the mirror. "+
+			"GitTarget writing to one (GitProvider, branch), and a full queue REFUSES the write "+
+			"(git_queue_refusals_total) rather than stalling the watch path behind a slow remote; "+
+			"the watch delivers it again from its cursor, and a cursor that expires first loses "+
+			"that write's own commit (and, under prune.mode OnEvent, a delete's removal). "+
 			"Size it so a bounded burst cannot overrun it: roughly (concurrent writers) x "+
 			"(GitTargets sharing the branch worker). Costs queue depth x payload of pod memory "+
 			"ON TOP of --branch-buffer-max-size, which does not cover this queue.")

docs/INDEX.md

@@ -128,8 +128,10 @@ so you never have to read a proposal to find out it already shipped.
 
 | Doc | Open question |
 |---|---|
-| [`branch-worker-event-model.md`](design/branch-worker-event-model.md) | **proposal; its first fix is built in #412.** The quiet-branch `WaitingForPush` failure (a failed push now retries on its own backoff) and the Git operation deadlines still to do. Worker ownership, FIFO inputs, save and deadline semantics, and the transition boundary for later durability. Journal storage and rollout belong to the HA plan. |
-| [`gittarget-state-of-affairs.md`](design/gittarget-state-of-affairs.md) | **snapshot, 2026-10-02.** What is built for `GitTarget` and the branch worker, which page owns which decision, the known gaps ranked by user impact (verified in code), and the recommended order. Read it first to find the right page on this topic. |
+| [`branch-worker-event-model.md`](design/branch-worker-event-model.md) | **proposal; its in-process fixes are built in #412 and #413.** Publication retry, Git operation deadlines, intake pause and resume, and publication status are done. Worker ownership, FIFO inputs, save and deadline semantics, and the transition boundary for later durability, which is next. Journal storage and rollout belong to the HA plan. |
+| [`gittarget-state-of-affairs.md`](design/gittarget-state-of-affairs.md) | **snapshot, 2026-10-04.** What is built for `GitTarget` and the branch worker, which page owns which decision, the review findings and how each was closed, and what is left elsewhere. Read it first to find the right page on this topic. |
+| [`gittarget-branch-worker-pending-writes.md`](design/gittarget-branch-worker-pending-writes.md) | **built in #413.** Ordered pending writes, one materializer and one retry schedule; a safe admission boundary for the watch; Git call deadlines; intake pause and resume; publication status. Holds the recovery contract for an outage, replay limits, and the acceptance scenarios. |
+| [`event-pipeline-overview.md`](design/event-pipeline-overview.md) | **snapshot, 2026-10-04.** Diagrams of the pipeline as built: watch streams and their replay/resume rules, the three meanings of "replay", the branch worker's inputs, the pending writes' lifecycle, retry and intake, a save end to end, what is an event today, and an honest list of what is still missing. |
 | [`gittarget-parent-branch.md`](design/gittarget-parent-branch.md) | **merged in #407.** `spec.parentBranch`: a target stays on standby until a write needs a branch, then creates its write branch from the parent's tip as checked at publication. The contract as built, the defect it started from, prior art and naming. Open: parent per `GitProvider.status.branches` entry. |
 | [`gittarget-parent-hardening.md`](design/gittarget-parent-hardening.md) | **done in #407.** Parent changes on a live worker, an empty repository gaining a ref, an unresolved default branch, recovery that cannot strand work, and the refused upload, without adding a pull to the fast path; plus the two recovery fixes from the review at `93120caf`. Records the merge boundary and the next separate PRs. |
 | [`gittarget-parent-empty-repository.md`](design/gittarget-parent-empty-repository.md) | **deferred.** An empty repository's default branch: the server reports it over protocol v2 but go-git discards it (upstream fix first); then a separate policy decision on which branch the first commit may create. Assuming `main` was considered and rejected. |
@@ -142,7 +144,7 @@ so you never have to read a proposal to find out it already shipped.
 | [`target-watch-plan.md`](design/target-watch-plan.md) | **partly built.** The diff is built and applied; removal semantics are not. The companion to [`watch-manager-ownership.md`](design/watch-manager-ownership.md): the ownership page says WHO applies a plan, this one says WHAT a plan is and what changing it may touch. A resource collection — group, resource, namespace, deliberately no served version — is the one identity the watch stream, the render-fidelity scope and the mark-and-sweep boundary all agree on, because a key that does not round-trip to the scope it sweeps under is the class of error that deletes user data. The plan is diffed into `keep`/`start`/`restart`/`stop` and applied per collection, so adding one WatchRule stops re-initializing every unrelated collection into a queue shared with other tenants; a `restart` is a served-version change, which is why the version is spec DATA rather than identity. Readiness and the fidelity revision are per scope, so a KEPT collection holds the result its own initialization produced rather than being asked to prove itself again over an unrelated edit. `stop` never touches files — removal is a Git-side sweep under the target's existing `spec.prune.mode`, not a watch-layer delete. "Cut at the producer" is the accepted consequence: nothing fences the queue, so a deselected collection may leave a short tail of writes, bounded by the queue and converged afterwards. Still open: the `stop` classification wants a settled `TypeRemoved` from `typeset` (see TODO), and removal on INTENT is undecided. |
 | [`collection-terminology-rename.md`](design/collection-terminology-rename.md) | **step 1, shipped.** Replaced cells with resource collections and removed per-rule `operations` from both rule kinds. Rules observe every object event; GitTarget prune mode controls removal. Records the rename census (760 Go identifiers, 265 documentation occurrences), the operation-filter inventory, API/Helm migration, compatibility boundaries, and test cases. |
 | [`watches-labels-simplification.md`](design/watches-labels-simplification.md) | **step 2, implemented.** After the collection rename and operation-filter removal, native per-rule `objectSelector` (#146). Kubernetes is authoritative: `Always` compares selected identities with all managed Git identities in the type/namespace scope, without comparing Git labels. Full support includes supported Kustomize layouts from the first release, plus filtered-removal attribution and ordered producer replacement. |
-| [`label-selection-follow-ups.md`](design/label-selection-follow-ups.md) | **design, open.** What the label-selection review rounds left open, with the evidence for each: real deletions in a selected collection wait the full 3s attribution grace (the watch cannot tell them from a label exit, measured), the resync fence that stops overlapping collections from coalescing (kept on purpose), same-second selector-conflict ties, and one store rebuild per object coming back. Also lists what was fixed on the branch |
+| [`label-selection-follow-ups.md`](design/label-selection-follow-ups.md) | **design, open.** What the label-selection review rounds left open, with the evidence for each: real deletions in a selected collection wait the full 3s attribution grace (the watch cannot tell them from a label exit, measured), the resync fence that stops overlapping collections from coalescing (kept on purpose), same-second overlap ties, and one store rebuild per object coming back. Also lists what was fixed on the branch |
 | [`attribution-removal-wait-options.md`](design/attribution-removal-wait-options.md) | a removal now waits for evidence about the DELETION rather than accepting the object's last write, which stopped it naming whoever last edited the object as the author of a deletion they did not perform. Enumerates the eight situations a resolution can be in and shows the cost is concentrated in exactly one: a removal for which no delete fact will ever arrive (a graceful pod delete, a status-only removal, a type the audit policy skips) spends the whole grace to return the answer it had at t=0, measured at ~3.1s against ~70ms when evidence is present. Prices five options against that, and recommends a per-route watermark — stop waiting once the fact stream has demonstrably moved past this event — over a second timeout flag whose right value lives in the API server's config rather than ours. Open: the decision, and how common the case is outside the e2e suite |
 | [`commitrequest-save-wait-options.md`](design/commitrequest-save-wait-options.md) | **design, decided** — a five-phase plan, reported from a live cluster: a `CommitRequest` can expire before the write it exists to publish arrives, because the close delay defaulted to `0` while the write is still held head-of-line waiting for an audit fact, so the save resolves `Ready=True`/`NoWindowInGrace` and the edit commits seconds later under the target's `liveTemplate`. The decision is to repair the default (`spec.closeDelay`, now a duration string defaulting to `"2s"`, **shipped as phase 0**) and then let a request **name the write it is waiting for**, reusing the `(uid, resourceVersion)` identity the attribution join already keys on, including its settled `ExactCapable` rule. Records why the fact index cannot BE the wait (it sits upstream of the watch event, so a fact-keyed wait fires before the window opens) and what to take from it anyway — the register-then-check waiter discipline, and a best-effort retrospective oracle bounded by a TUNABLE TTL. Twelve situations, and the page is deliberate about the two it cannot fix cheaply: row 11 (a saturated queue is rejected in the WATCH path and never reaches the worker, so no worker-side wait can diagnose it) is an accepted, named gap rather than a promise, and row 10 is fixed only by a hook in the fidelity-drop branch that marks a wait terminally UNSATISFIABLE — satisfying before that gate would arm a finalize for a write that never commits. Carries a **preconditions chapter** for saves on today's build (audit policy first, then delay sizing, fidelity warm-up, save-button debounce, and assert on the commit rather than on `Ready`). `CommitterActivity` is deferred, not recommended: by its own arrival-not-existence rule it cannot satisfy the already-arrived case. Open: situation 12 (one actor, two saves, one window — blocks phase 2), whether the fact-index lookup earns its coupling, and how an unsatisfied wait is graded |
 | [`commit-timing-surface.md`](design/commit-timing-surface.md) | **built.** One configuration surface for commit windows and saves, an in-place `v1alpha3` break with one tested upgrade procedure: only writes open a window; a `window` block (`idleTimeout`, `maxDuration`, target default `1m`) on both kinds, which a `CommitRequest` replaces whole for the window it attaches to; `attach: CurrentOrNext \| Next` with `attachTimeout`; opt-in `whenNothingToCommit: CommitEmpty` that keeps the cause in the reason. Precise clock and zero rules. Supersedes the save-wait timer model. Open: the target's `maxDuration` value |

docs/TODO.md

@@ -13,10 +13,9 @@ a request on a target that may not be written fails, whatever path ends it
 (`internal/git/write_gate.go`).
 
 Retry progress on quiet branches is fixed: a failed publication schedules its own bounded retry
-(`internal/git/publication_retry.go`, [`push-cooldown.md`](design/push-cooldown.md) §7 option C).
-Still open: when a rebuild's fetch fails, the open window is dropped, and unless the cause was a
-missing parent nothing records the dropped scope or asks for a snapshot, so those writes wait for
-the next resync.
+(`internal/git/retry.go`, [`push-cooldown.md`](design/push-cooldown.md) §7 option C). A window whose
+rebuild fails is no longer dropped: it stays pending for that retry
+([`gittarget-branch-worker-pending-writes.md`](design/gittarget-branch-worker-pending-writes.md)).
 
 **Later, non-blocking:**