From fbbfcec75309577e33bf36243361428dea81002b Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Thu, 28 May 2026 15:14:26 +0000 Subject: [PATCH 01/40] docs: initial docs (also some findings from preps for Cozysummit) --- docs/cozystack-bugreport.md | 41 +++ docs/design/e2e-speedup-plan.md | 410 +++++++++++++++++++++ docs/design/e2e-speedup-skipped.md | 122 ++++++ docs/design/initial-seed-via-watch-list.md | 241 ++++++++++++ docs/upgrade-finding.md | 157 ++++++++ 5 files changed, 971 insertions(+) create mode 100644 docs/cozystack-bugreport.md create mode 100644 docs/design/e2e-speedup-plan.md create mode 100644 docs/design/e2e-speedup-skipped.md create mode 100644 docs/design/initial-seed-via-watch-list.md create mode 100644 docs/upgrade-finding.md diff --git a/docs/cozystack-bugreport.md b/docs/cozystack-bugreport.md new file mode 100644 index 000000000..8d05c6b27 --- /dev/null +++ b/docs/cozystack-bugreport.md @@ -0,0 +1,41 @@ +Title: core.cozystack.io REST storage does not emit k8s.io/initial-events-end bookmark for SendInitialEvents=true watches + +Summary + +The aggregated apiserver's resources in group core.cozystack.io (tenantsecrets, tenantmodules, tenantnamespaces) do not honor the SendInitialEvents field of ListOptions. Watch clients that opt into the WatchList / streaming-list protocol (kube-apiserver ≥ 1.27, client-go WatchListClient feature gate, on by default since client-go v1.35) never receive the required k8s.io/initial-events-end bookmark, so reflectors stay stuck in their initial-stream phase and log hasn't received required bookmark event marking the end of initial events stream every ~10s. The apps.cozystack.io/Application storage already implements this contract correctly — the three core resources do not. + +Reproduction + +Any client running client-go ≥ v0.30 with the default WatchListClient=true informing over tenantsecrets, tenantmodules, or tenantnamespaces: + + +Warning: event bookmark expired err="reflector.go:343: hasn't received required bookmark event marking the end of initial events stream, received last event 59.998946102s ago" +The warning fires on every reflector restart and continuously while the cache is unsynced from the WatchList perspective. + +Evidence (commit 1810263) + +Reference implementation that does it correctly: +pkg/registry/apps/application/rest.go:684-807 — reads options.SendInitialEvents, tracks lastResourceVersion, emits a watch.Bookmark with metadata.annotations["k8s.io/initial-events-end"] = "true" once initial ADDED events have been delivered, and even handles the "underlying watcher closes during initial snapshot" edge case. + +Missing in: + +pkg/registry/core/tenantsecret/rest.go:454 — Watch() ignores opts.SendInitialEvents and only forwards bookmark events from the backing corev1.Secret watcher; those carry no k8s.io/initial-events-end annotation. +pkg/registry/core/tenantmodule/rest.go:311 — same pattern, forwards bookmarks from the backing helmv2.HelmRelease watcher. +pkg/registry/core/tenantnamespace/rest.go:158 — same pattern, forwards bookmarks from the backing corev1.Namespace watcher. +A grep for SendInitialEvents or initial-events-end across pkg/registry/core/ returns zero hits. + +Proposed fix + +Lift the sendInitialEventsEndBookmark helper and its caller sites from application/rest.go into the three core resources, adapting the bookmark object to the resource's own TypeMeta. The mechanical shape per file: + +Near the top of Watch(): sendInitialEvents := opts.SendInitialEvents != nil && *opts.SendInitialEvents. +Track lastResourceVersion on every translated event from the backing watcher. +After delivering the initial Added events, emit a watch.Bookmark whose object is an empty TenantSecret / TenantModule / TenantNamespace carrying the right TypeMeta, ResourceVersion=lastResourceVersion, and annotations["k8s.io/initial-events-end"]="true". +Handle "underlying watcher closes before initial snapshot is complete" — emit the bookmark on the way out, same as application/rest.go:816-817. +Acceptance criteria + +A test alongside rest_watch_test.go for each of the three resources asserting: +When SendInitialEvents=true, all existing objects are emitted as ADDED. +A Bookmark event follows with annotations["k8s.io/initial-events-end"] == "true" and a non-empty ResourceVersion. +Subsequent live events are delivered after the bookmark. +An informer using client-go's WatchListClient feature reaches HasSynced=true against these resources without the missing-bookmark warning. diff --git a/docs/design/e2e-speedup-plan.md b/docs/design/e2e-speedup-plan.md new file mode 100644 index 000000000..3ec497a86 --- /dev/null +++ b/docs/design/e2e-speedup-plan.md @@ -0,0 +1,410 @@ +# E2E Speedup Plan + +## Goal + +Make `task test-e2e*` faster for the three audiences that pay the cost: +local devs, AI agents, and CI. + +Phased plan with explicit exit criteria. **Phase 0** ships the visibility +tooling that lets every later phase be measured against the same baseline. + +Items considered and excluded (BeforeSuite short-circuit, CI matrix +sharding, cluster reuse, Tekton/Shipwright, dropping `Ordered` inside the +new Manager files) are in [e2e-speedup-skipped.md](e2e-speedup-skipped.md) +with rationale. + +## Non-goals + +- Rewriting the e2e harness from scratch. +- Reducing the number of e2e tests (this plan does not merge or delete + specs; Phase 2 is a mechanical split that preserves spec count). + +## Baseline (2026-05-28, warm devcontainer, k3d reused) + +| | | +|---|---| +| Warm `task prepare-e2e` | **1.4 s** (everything stamp-cached) | +| Smoke suite (`task test-e2e`, 20 of 47 specs) | **418.7 s** (~7 min) | +| BeforeSuite (Go re-invoking warm prepare) | 1.4 s | + +### Per-spec ranking (smoke filter, warm) + +``` + Duration % Spec + 128.4 s 30.8 Restart Snapshot Safety keeps the git mirror intact when the controller restarts + 40.9 s 9.8 Commit Window Batching collapses a burst of events into one grouped commit and one push + 34.1 s 8.2 Aggregated API server should install and serve flunders through the aggregation layer + 28.9 s 6.9 Manager should create Git commit when IceCreamOrder CRD is installed via ClusterWatchRule + 25.7 s 6.2 Commit Signing should produce per-event commits verifiable locally and by Gitea + 25.2 s 6.0 Audit Redis Queue should enqueue incoming audit webhook events into a Redis stream + 24.4 s 5.9 Audit Redis Consumer should attribute the commit to the OIDC display name and email + 22.3 s 5.3 Commit Request finalizes a CommitRequest created with metadata.generateName + 21.9 s 5.2 Commit Request finalizes the open commit window on demand and reports the resulting SHA + 16.4 s 3.9 Audit Redis Consumer should produce a Git commit from an audit stream event + 15.5 s 3.7 Manager should delete Git file when ConfigMap is deleted via WatchRule + 9.9 s 2.4 Manager should create Git commit when ConfigMap is added via WatchRule + 9.1 s 2.2 Manager should commit encrypted Secret manifests when WatchRule includes secrets + 4.4 s 1.1 Manager should backfill pre-existing ConfigMap when WatchRule is added afterwards + 4.1 s 1.0 Manager should receive audit webhook events from kube-apiserver + 2.1 s 0.5 Manager should handle a normal and healthy GitProvider + 1.9 s 0.5 Manager should run successfully + 1.7 s 0.4 Manager should validate GitProvider with real Gitea repository + 0.1 s 0.0 Manager should ensure the metrics endpoint is serving metrics + 0.1 s 0.0 Manager should expose the controller service + 417.3 s total +``` + +### Findings from the data + +1. **Top 5 specs = 62 %** of smoke wallclock. +2. **`Restart Snapshot Safety` is 31 % alone.** 75 s of its 128 s is a + passive negative-assertion wait. +3. **Namespace teardown is consistently 20–25 s** per spec (finalizer + waits). Dominant cleanup cost — relevant to Phase 2's risk profile. +4. **Per-spec setup ~5–10 s.** Specs already use per-spec namespaces + + repos; the structural blockers to parallelism are the shared + `Manager` Describe and cluster-scoped resources (CRDs). +5. **`prepare-e2e` is a script, not a DAG.** Eight sub-tasks run + sequentially via `cmds: - task: X` + ([test/e2e/Taskfile.yml:106-119](../../test/e2e/Taskfile.yml#L106-L119)) + even though several stamps have no real interdependency. + +## Phases + +### Phase 0 — Visibility via Ginkgo's built-in reporting + +**Why first:** every later phase claims a speedup. We need a structured, +machine-readable record before and after. + +**Primary mechanism:** add `-ginkgo.json-report=...` to the e2e `go test` +invocations. Ginkgo writes a per-spec JSON report with names, durations, +labels, and pass/fail without any log scraping. Parse it with a small Go +tool (`test/e2e/tools/spec-timings/`) that takes the JSON path and prints +a duration-sorted table plus totals — same shape as the baseline above, +but driven by structured data, not regex-on-logs. + +**Secondary mechanism (optional):** a tiny line-buffered timestamper at +`test/e2e/tools/ts/` (~20 lines of Go, `bufio.Scanner` over stdin) so +free-form logs can be decorated with wallclock when debugging cold runs +where structured JSON isn't enough. Not on the critical path. + +**Why not a `ReportAfterEach` reporter:** Ginkgo's built-in +`--json-report` already covers this. No new reporter code needed. + +**Exit criterion:** `go run ./test/e2e/tools/spec-timings ` +reproduces the per-spec table in this doc from a fresh `task test-e2e` +run. + +### Phase 1 — Express `prepare-e2e` as a real DAG + +**Change:** replace flat `cmds: - task: X` chains in `prepare-e2e` and +the upstream `install` flow with proper `deps:` so `task` schedules the +graph. + +**Concrete parallel branches** (verified against current stamp wiring): + +- `_age-key` (instant) is independent of `_cluster-ready` and + `_project-image-ready`. +- `_cluster-ready` (~30–60 s cold) and `_project-image-ready` (~10–40 s + on Go change) can run concurrently. +- `_image-loaded` joins both. +- `_flux-installed` → `_flux-setup-ready` → `_services-ready` chain + depends only on `_cluster-ready`. +- `_sops-secret-yaml` only needs `_age-key`. +- `_aggregated-api-ready` joins `_services-ready` ∧ `_webhook-tls-ready`. + +**Risk:** very low. Stamps make tasks idempotent; concurrent execution +can't corrupt. Side effect: interleaved logs. If that becomes a debugging +pain, keep a `prepare-e2e-verbose` alias that forces sequential +execution. + +**Exit criterion:** + +- Cold `task prepare-e2e` ≥ 25 % faster than the sequential baseline. +- Warm `task prepare-e2e` still ≈ 1.4 s. + +### Phase 2 — Mechanical split of the `Manager` Describe + +The `Manager` Describe in +[test/e2e/e2e_test.go](../../test/e2e/e2e_test.go) is a single `Ordered` +block of 21 specs sharing a package-global `managerRepo` and a +`gitprovider-normal` GitProvider implicitly created by spec #10. Two +specs hold `time.Sleep(5*time.Second)` admitting cross-spec state leak. + +**Scope.** This phase is a *mechanical* refactor: + +- Keep all 21 specs as they are; preserve spec count and observable + behavior. +- Move them into four files (see below), each its own top-level Describe + with its own per-file fixtures. +- Replace the package-global `var managerRepo *RepoArtifacts` with a + per-Describe local; each file's BeforeAll calls `SetupRepo` for itself. +- Move the cluster-scoped CRD ownership into the file that owns it + (see Phase 2.5 / CRD isolation). + +**Out of scope for this phase:** + +- No spec merging (e.g. old #18+#19+#20 stay three separate specs; + collapsing them is a follow-up). +- No `Ordered` removed *within* the new files. +- No Ginkgo parallelism enabled yet (Phase 2.5). +- No fix for the two `time.Sleep(5s)` hacks (separate root cause). + +**Spec inventory** (read 2026-05-28): + +| # | Spec | What it needs | Cleanup | +|---|------|---------------|---------| +| 1 | should run successfully | controller pod up | sets `controllerPodName` | +| 2 | should expose the controller service | `controllerPodName` from #1 | — | +| 3 | should expose the audit service separately | `controllerPodName` from #1 | — | +| 4 | should ensure the metrics endpoint is serving metrics | `controllerPodName` from #1 + log scrape | — | +| 5 | should receive audit webhook events from kube-apiserver | global Prometheus audit counter (delta-based) | deletes its CM | +| 6 | should validate GitProvider with real Gitea repository | `managerRepo` HTTP creds | deletes its GitProvider | +| 7 | should handle GitProvider with invalid credentials | `managerRepo` invalid creds | deletes its GitProvider | +| 8 | should handle GitTarget with nonexistent branch pattern | `managerRepo` HTTP creds | deletes its GitProvider + GitTarget | +| 9 | should validate GitProvider with SSH authentication | `managerRepo` SSH creds | deletes its GitProvider | +| 10 | should handle a normal and healthy GitProvider | `managerRepo` HTTP creds | **leaves `gitprovider-normal`** | +| 11 | should reconcile a WatchRule CR | `gitprovider-normal` | deletes own WR + GT | +| 12 | should commit encrypted Secret manifests when WatchRule includes secrets | `gitprovider-normal` | deletes own resources | +| 13 | should generate missing SOPS age secret when … generateWhenMissing | `gitprovider-normal` | deletes own resources | +| 14 | should create Git commit when ConfigMap is added via WatchRule | `gitprovider-normal` + **5 s sleep** | deletes own resources | +| 15 | should backfill pre-existing ConfigMap when WatchRule is added afterwards | `gitprovider-normal` | deletes own resources | +| 16 | should delete Git file when ConfigMap is deleted via WatchRule | `gitprovider-normal` + **5 s sleep** | deletes own resources | +| 17 | should create Git commit when IceCreamOrder CRD is installed via ClusterWatchRule | `gitprovider-normal` | **leaves CRD installed** | +| 18 | should create Git commit when IceCreamOrder is added via WatchRule | CRD from #17 (also re-applies) | **leaves WR + GT + CRD** | +| 19 | should update Git file when IceCreamOrder is modified via WatchRule | WR + GT from #18 | leaves them | +| 20 | should delete Git file when IceCreamOrder is deleted via WatchRule | WR + GT from #18 | leaves them | +| 21 | should delete Git file when IceCreamOrder CRD is deleted via ClusterWatchRule | CRD still installed | deletes CRD globally | + +The shared `AfterAll` (L1712) then cleans up `watchrule-icecream-orders`, +its GitTarget, `gitprovider-normal`, and the CRD. + +**Proposed file layout** (label `manager` kept on all four so +`task test-e2e-manager` keeps working unchanged): + +### Characteristics of the new files + +| File | Specs | Needs Gitea repo | Controller intervention | Ordered by design | Touches cluster-scoped objects | Touches global metrics | Leaves resources behind across specs | Parallel-safe with other groups | Smoke time today | +|---|---|---|---|---|---|---|---|---|---| +| `controller_basics_e2e_test.go` (Group A: specs 1–5) | 5 | **no** | no | partial — specs 2–4 reuse `controllerPodName` from #1 | no | yes — Prometheus audit counters (delta read) | no | yes | ~6 s | +| `gitprovider_validation_e2e_test.go` (Group B: specs 6–9) | 4 | yes — `managerRepo` HTTP/SSH/invalid creds | no | no — each spec creates+deletes its own GitProvider | no | no | no | yes | ~2 s (only #6 in smoke) | +| `watchrule_configmap_secret_e2e_test.go` (Group C: specs 10–16) | 7 | yes — `managerRepo` + shared `gitprovider-normal` in BeforeAll | no | no — once `gitprovider-normal` is in BeforeAll, specs 11–16 are independent (the two `time.Sleep(5s)` hacks aside) | no | no | yes — own `managerRepo` accumulates commits across specs in distinct subpaths | yes (own namespace) | ~41 s | +| `crd_lifecycle_e2e_test.go` (Group D: specs 17–21) | 5 | yes — `managerRepo` + `gitprovider-normal` in BeforeAll | no | **yes** — #18 hands WR+GT to #19+#20; #21 destroys the CRD others rely on | **yes — owns its own per-file CRD** (see CRD isolation, Phase 2.5) | no | yes within the file (intentional) | yes (own CRD name → no collision with other groups) | ~29 s (only #17 in smoke) | + +**Column glossary:** + +- **Needs Gitea repo** — does the file need `SetupRepo()` in BeforeAll? + A "no" is a meaningful setup saving. +- **Controller intervention** — does any spec restart, scale, or + perturb the controller deployment? None of these do; kept as a column + because it's the property that flags Phase 2.5's Serial registry. +- **Ordered by design** — would removing `Ordered` change the test's + observable contract? "Partial" / "no" means we keep it for now but + it could be dropped without semantic change. +- **Touches cluster-scoped objects** — flags CRD, ClusterWatchRule, + cluster RBAC. Main parallel-safety risk. +- **Touches global metrics** — Prometheus counters shared across + specs. Delta-based reads make this safe within one run. +- **Leaves resources behind across specs** — depends on a resource + created by a previous spec in the same file (Ordered intent), or + leaks past `AfterAll`? +- **Parallel-safe with other groups** — could this file run on a + Ginkgo worker in parallel with the others? After Phase 2.5's CRD + isolation, yes for all four. +- **Smoke time today** — sum of per-spec durations from the + 2026-05-28 baseline for specs labelled `smoke`. Lower bound, not a + total. + +**Per-file design notes:** + +1. **`controller_basics_e2e_test.go`** — no `managerRepo` setup at all. + BeforeAll only verifies the controller pod exists. Specs 1–5. Saves + the repo/secrets/SOPS setup this group never used. +2. **`gitprovider_validation_e2e_test.go`** — BeforeAll creates a + per-file namespace + `managerRepo`. Specs 6–9. +3. **`watchrule_configmap_secret_e2e_test.go`** — BeforeAll creates a + per-file namespace + `managerRepo` + `gitprovider-normal`. Old #10 + becomes the BeforeAll. Each remaining spec keeps its own WR+GT + create/delete. The two `time.Sleep(5s)` hacks stay. +4. **`crd_lifecycle_e2e_test.go`** — BeforeAll creates a per-file + namespace + `managerRepo` + `gitprovider-normal`. Internal `Ordered` + carries specs 17 → 18 → 19 → 20 → 21. **Owns its own per-file CRD**; + AfterAll deletes it. + +**Risks:** + +- **Sequential runs may get slower** before Phase 2.5 lands. Each new + file adds a namespace teardown (`cleanupNamespace` in + [test/e2e/helpers.go:254](../../test/e2e/helpers.go#L254) hits + finalizer waits of 20–25 s). Three extra namespaces ≈ 60–75 s of + added wallclock if Phase 2.5 is not yet on. Phase 2 and Phase 2.5 + should land back-to-back; if shipping just Phase 2, advertise as a + prerequisite, not a speedup. +- Group A's audit-events spec (#5) reads global Prometheus counters + with a delta pattern. Preserve baseline-before-act ordering. +- CRD cleanup ownership moves to `crd_lifecycle_e2e_test.go` — verify no + other file still expects it to be present or absent at startup. +- N Gitea repos created per run instead of 1. Each ~1 s. Acceptable. + +**Exit criterion:** + +- 21 specs from the old `Manager` Describe run as 21 specs across 4 + files. No spec merged or removed. Suite green. +- Per-spec timings (from Phase 0 JSON report) unchanged within noise. +- No new shared state between the four Describes. + +### Phase 2.5 — Enable bounded Ginkgo parallelism + Serial registry + +Without this phase, Phase 2's wallclock is *worse* than today on +sequential runs. This phase is what realises the speedup. + +**Change:** + +- Add `E2E_GINKGO_PROCS` (default e.g. `2`) wired into + [test/e2e/Taskfile.yml:194](../../test/e2e/Taskfile.yml#L194) so + `go test ... -ginkgo.parallel -ginkgo.procs=$E2E_GINKGO_PROCS`. Local + default sane for a devcontainer; CI may pass a higher value when + Phase 6 (separately, not in active plan) is reconsidered. +- Establish a **suite-wide Serial registry** — a short list of test + containers marked `Serial` because they touch shared cluster-scoped + state that cannot be isolated by name alone. Created as a living + document `docs/design/e2e-serial-registry.md` in this phase. Initial + entries from the 2026-05-28 audit: + - `restart_snapshot_e2e_test.go` — restarts the controller deployment; + interferes with any other test reading or writing through the + controller while restarted. + - Anything else that performs `kubectl delete crd` cluster-wide or + rolls the controller. Phase 2.5 PR populates the initial list from + a grep audit; afterward, adding cluster-scoped writes requires + updating the registry in the same PR. + +**CRD isolation (the easier solution).** The shared +`icecreamorders.shop.example.com` CRD currently appears in: + +- [test/e2e/e2e_test.go](../../test/e2e/e2e_test.go) (Manager block; moves to + `crd_lifecycle_e2e_test.go` in Phase 2) +- [test/e2e/restart_snapshot_e2e_test.go:94-96](../../test/e2e/restart_snapshot_e2e_test.go#L94-L96) +- [test/e2e/bi_directional_e2e_test.go:130](../../test/e2e/bi_directional_e2e_test.go#L130) +- [test/e2e/e2e_suite_test.go](../../test/e2e/e2e_suite_test.go) (pre-run cleanup) + +Rather than serialise these three files against each other, **give each +its own CRD** in its own API group: + +| File | CRD group | Kind | +|---|---|---| +| `crd_lifecycle_e2e_test.go` | `crd-lifecycle.e2e.example.com` | IceCreamOrder | +| `restart_snapshot_e2e_test.go` | `restart-snapshot.e2e.example.com` | IceCreamOrder | +| `bi_directional_e2e_test.go` | `bi-directional.e2e.example.com` | IceCreamOrder | + +Implementation: take the existing +[test/e2e/templates/icecreamorder-crd.yaml](../../test/e2e/templates/icecreamorder-crd.yaml) +fixture and templatize the group + plural. Each test renders its own. +The pre-cleanup in `e2e_suite_test.go` deletes all three by name. + +This removes the cross-file shared cluster-scoped state and lets the +three files run in parallel without any Serial marking. The Serial +registry then only needs to capture genuinely process-level conflicts +(restart_snapshot at the controller-deployment level, future ones). + +**Risks:** + +- Hidden ordering assumptions only surface under `-procs > 1`. Run for + a stability window (say 10 consecutive clean runs) before flipping + the default. +- Some helpers may write to package-global state. Phase 2 already + removes the only obvious one (`managerRepo`); audit during this PR. + +**Exit criterion:** + +- `task test-e2e` with `E2E_GINKGO_PROCS=2` runs green across N + consecutive runs (N decided when starting; suggested 10). +- Smoke wallclock drops by at least the parallel-safe Group A+B+C share. +- Serial registry exists and is non-empty. + +### Phase 3 — Target-scoped reconcile-complete + drain signal for `Restart Snapshot Safety` + +**Why a pod-level gauge alone is insufficient** (per reviewer): a +pod-scoped "initial reconcile complete" can be set after +`ReconcileForRuleChange` returns +([internal/watch/manager.go:679](../../internal/watch/manager.go#L679)) +while snapshot writes for a specific GitTarget are still pending — they +flow through `MaybeReplaySnapshot` +([internal/watch/manager.go:1216](../../internal/watch/manager.go#L1216)) +and enqueue via the branch worker +([internal/watch/git_target_event_stream.go:139](../../internal/watch/git_target_event_stream.go#L139)) +which then pushes asynchronously. The restart test's negative assertion +("the mirror is not wiped after this restart") requires evidence that +*this GitTarget's* post-restart processing has fully drained, not just +that the pod started. + +**Change.** Pair two signals: + +1. **`gitopsreverser_target_initial_reconcile_complete{namespace,name}`** + — gauge per GitTarget, flips to `1` once its post-(re)start initial + reconcile pass has completed end-to-end (snapshot decision made, + queue submission completed). +2. **`gitopsreverser_branch_worker_queue_depth{namespace,name,branch}`** + — gauge of pending work items for this branch worker. Wait for `0` + to confirm the queue has drained. + +The metric package is `internal/telemetry/` (not `internal/metrics/` — +see `internal/telemetry/exporter.go`); add the new metrics alongside +the existing ones. + +**Restart test reframe:** + +1. Rollout-restart the controller deployment. +2. Wait for `target_initial_reconcile_complete{ns,name} == 1` for the + restart test's GitTarget. +3. Wait for `branch_worker_queue_depth{ns,name,branch} == 0`. +4. Apply a short stability window (a few seconds). +5. Assert no destructive commit happened. + +**Risks:** + +- The new metrics' contracts must be exact enough to be load-bearing in + a test. Document the lifecycle of each in code comments. Treat as a + public observability surface from day one — renaming costs cluster + rollouts. +- Worker-queue-depth must include in-flight items, not just queued ones, + or step 3 races with the worker. Verify against the actual + implementation when writing the metric. + +**Exit criterion:** + +- Spec runs in < 30 s (down from 128 s). +- Both metrics are scrapable via the existing Prometheus pipeline. +- Suite still green; no regressions in other restart-related specs. + +## Decisions + +- **File naming.** Files use the form `_e2e_test.go`. The CRD + lifecycle file is `crd_lifecycle_e2e_test.go` (consistent throughout + this doc; earlier drafts used `manager_icecream_lifecycle_test.go` + inconsistently — fixed). +- **Label.** All four new Describes keep `Label("manager")` so + `task test-e2e-manager` works unchanged. Renaming the label is a + separate decision not in this plan. +- **No spec merges in Phase 2.** The earlier draft proposed collapsing + old #18+#19+#20 into one create→update→delete spec; deferred to a + follow-up so the Phase 2 PR is purely mechanical and easy to review. +- **Metric package is `internal/telemetry`.** Confirmed against + `internal/telemetry/exporter.go`. No `internal/metrics` directory + exists. +- **CRD isolation over Serial marking.** Where multiple e2e files + install the same cluster-scoped CRD, the cheap fix is to give each + file its own CRD group (Phase 2.5). Serial marking is reserved for + conflicts that name-isolation can't fix (e.g. controller restart). + +## Reproducing the baseline + +```bash +task prepare-e2e +mkdir -p /tmp/e2e-baseline +go test -timeout 15m ./test/e2e/ \ + -ginkgo.v -ginkgo.label-filter=smoke \ + -ginkgo.json-report=/tmp/e2e-baseline/smoke.json +go run ./test/e2e/tools/spec-timings /tmp/e2e-baseline/smoke.json +``` + +(`test/e2e/tools/spec-timings/` is a Phase 0 deliverable.) diff --git a/docs/design/e2e-speedup-skipped.md b/docs/design/e2e-speedup-skipped.md new file mode 100644 index 000000000..f17fa0cd3 --- /dev/null +++ b/docs/design/e2e-speedup-skipped.md @@ -0,0 +1,122 @@ +# E2E Speedup — Items Considered and Skipped + +Companion to [e2e-speedup-plan.md](e2e-speedup-plan.md). This document records +proposals that were evaluated during the 2026-05-28 planning round and +*excluded* from the active plan, with the reasoning. The point is to make +future revisiting cheap: when someone asks "did we look at X?", the answer +and the *why* are here, not lost in chat history. + +## 1. Short-circuit `BeforeSuite` to skip warm `task prepare-e2e` + +**Proposal.** [test/e2e/e2e_suite_test.go:64](../../test/e2e/e2e_suite_test.go#L64) +unconditionally re-invokes `task prepare-e2e` on every `go test` run. Warm +cost is ~1.4 s. The idea was to `stat` the `prepare-e2e.ready` stamp from Go +and skip the subshell if it's newer than the relevant inputs. + +**Why skipped.** It duplicates logic that `task` already owns. The whole +point of the stamp files is that `task` itself decides whether to do work or +no-op. Re-implementing the freshness check in Go gives us two sources of +truth for the same thing, with the obvious failure mode that they drift. + +The 1–2 s warm cost is small enough that the better fix — if it ever +matters — is to make `task` cheaper to invoke for an all-no-op DAG (Phase 1 +of the active plan already moves in that direction by converting chains into +parallel `deps:`). + +**Revisit if:** warm prepare ever grows back past a few seconds in a way +that Phase 1's DAG refactor can't fix. + +## 2. CI matrix sharding (split the `full` e2e job by Ginkgo label) + +**Proposal.** [.github/workflows/ci.yml:261-313](../../.github/workflows/ci.yml#L261-L313) +runs the e2e suite as a 2-entry matrix `{quickstart, full}`. Splitting +`full` into 4–5 label-scoped shards (`smoke`, `signing`, `audit-redis`, +`bi-directional`, `image-refresh`, `aggregated-api`) would parallelize +across GitHub runners and drop wallclock to `max(shard)` instead of +`sum(shard)`. + +**Why skipped.** Empirically GitHub Actions chokes when this repo asks for +3+ concurrent runners — there are throttles in practice that make the +"free parallelism" assumption false. The shards would queue rather than +actually running in parallel, defeating the wallclock benefit and burning +more CPU minutes per run. + +**Revisit if:** + +- We move to a GitHub plan / runner pool with real concurrent capacity, or +- We move to a different CI provider where concurrent runners are not + throttled, or +- We adopt self-hosted runners (see item 3) where we control the + parallelism budget. + +## 3. CI cluster reuse via self-hosted runner or Tekton + +**Proposal.** Today every CI job creates a fresh k3d cluster; that boot is +the dominant CI cost. Two flavours of reuse were considered: + +- **Self-hosted GitHub runner** holding a long-lived k3d. PRs run against + namespace-scoped slices. +- **Tekton on a persistent Kubernetes cluster.** Pipelines run as + CRD-defined Tasks; the host cluster (or warm test fixtures within it) + survives across pipeline runs. + +**Why skipped (for now).** Both require taking on durable operational +responsibility: a VM or cluster we own, secure, upgrade, monitor, and pay +for. Multi-tenant isolation between concurrent PR pipelines (namespaces, +nested k3d, or vCluster) is its own design problem. Tekton additionally +requires running an image registry for CI-built images (`k3d image import` +no longer works once builds live in pods) and maintaining pipeline-as-CRD +definitions alongside GitHub Actions. + +The active plan's Phases 1–3 attack the *per-run* cost of the e2e harness. +That's the cheaper lever to pull first. We can re-measure CI wallclock +after they land, and only then decide whether the remaining cold-boot +cost justifies the ops investment. + +**Revisit if:** Phases 1–3 of the active plan are complete and CI wallclock +is *still* dominated by cluster boot, *and* the team is willing to take on +the ops burden of a persistent runner or cluster. + +## 4. Removing `Ordered` *within* the new Manager files (Ginkgo intra-file parallelism) + +**Proposal.** After Phase 2 of the active plan splits the `Manager` +Describe into four files, also drop the `Ordered` decorator inside each +new file so Ginkgo can parallelize specs within a single file. + +**Why skipped (for now).** Phase 2 already buys the parallelism that +matters: Ginkgo runs *across* top-level Describes in parallel, so the four +new files can run on four workers. Dropping `Ordered` inside each file +would only help if a single file has many slow specs — which the +post-split layout does not. Group D (IceCreamOrder lifecycle) genuinely +needs `Ordered`. Groups A/B/C have cheap specs (most < 5 s); the gain +from intra-file parallelism is small and the correctness risk (implicit +ordering not caught yet) is real. + +**Revisit after:** Phase 2 lands and we have stability data. If one of the +new files has a slow tail of independent specs, drop `Ordered` there +specifically. + +## 5. Move build to Tekton / Shipwright for cloud-native CI + +**Proposal.** Replace (or supplement) GitHub Actions with Tekton pipelines +and/or Shipwright build CRDs, on the theory that "cloud-native CI" would +naturally make iteration faster. + +**Why skipped.** The premise doesn't survive contact with the actual cost +model: + +- Iteration speed is dominated by *cluster boot* and *test execution*, not + by the *build* step. Shipwright only addresses image builds; Tekton is a + pipeline orchestrator, not a test accelerator. +- Pushing/pulling images through a registry on every build (Tekton can't do + `k3d image import` the way local `task` can) makes per-iteration cost go + *up*, not down. +- The legitimate angle — Tekton on a persistent cluster keeping test + infrastructure warm across pipeline runs — is the same problem as item 3 + with extra moving parts. Self-hosted GitHub runner gets ~80 % of the + benefit at ~10 % of the complexity. + +**Revisit if:** we have a separate reason to want pipeline-as-CRD (preview +environments, dynamic on-demand test infrastructure, multi-product +delivery) — in that case the iteration-speed benefit becomes a bonus rather +than the justification. diff --git a/docs/design/initial-seed-via-watch-list.md b/docs/design/initial-seed-via-watch-list.md new file mode 100644 index 000000000..92bb5cd5c --- /dev/null +++ b/docs/design/initial-seed-via-watch-list.md @@ -0,0 +1,241 @@ +# Initial Seed via client-go WatchListClient + +> Status: proposed — not yet implemented +> Scope: `internal/watch` snapshot path, dynamic informer construction, the +> client-go `WatchListClient` feature gate +> Related: [`docs/upgrade-finding.md`](../upgrade-finding.md), +> [`docs/cozystack-bugreport.md`](../cozystack-bugreport.md) + +## Goal + +Make initial cluster-state seeding flow through a single, cache-backed source of +truth — the dynamic informer cache that `internal/watch` already maintains — +and stop issuing a separate hand-rolled `LIST` per GVR per namespace whenever a +`GitTarget` needs a snapshot. + +Use the client-go `WatchListClient` feature (streaming-list, on by default +since client-go v1.35 / v0.36.x) as the *transport* that hydrates that cache, +rather than something to fight against. The bookmark-warning we observed in +production ([`docs/upgrade-finding.md`](../upgrade-finding.md)) was a hint that +we had silently opted into streaming-list without redesigning the seed path +around it; the right response is to redesign, not to disable the gate. + +## Assumption + +This plan assumes that every aggregated apiserver we watch implements the +WatchList contract correctly — i.e. honors `SendInitialEvents=true` and emits +the `k8s.io/initial-events-end` bookmark. Standard kube-apiserver has done +this since 1.27. Aggregated APIs are the risk surface; see the +[Aggregated-API caveat](#aggregated-api-caveat) section below. + +If the assumption proves false in a given cluster, the per-GVR fallback in +Phase B keeps the system functional for misbehaving resources without giving up +the benefits on the rest. + +## Today's state — duplicated seeding + +Two seed paths run side-by-side: + +1. **Implicit informer seed.** + [`internal/watch/manager.go:977-981`](../../internal/watch/manager.go#L977-L981) + creates a `dynamicinformer.NewDynamicSharedInformerFactory` per GVR / + namespace. client-go performs its own `LIST` (or, with `WatchListClient=true`, + a streaming-list) and hydrates an in-memory `cache.Store`. Only the + per-event callbacks are wired through `EventRouter`; the populated store is + otherwise unused. + +2. **Explicit by-hand LIST.** + [`GetClusterStateForGitDest`](../../internal/watch/manager.go#L461) → + [`listResourcesForGVR`](../../internal/watch/manager.go#L630-L672) calls + `dc.Resource(gvr).List(...)` again, per matching namespace, against the + apiserver. This fires every time a `GitTarget` is snapshotted — rule + changes, periodic re-snapshot, controller restart — and re-LISTs state that + the informer cache already holds. + +For any GVR that already has a running informer, this is two full +materializations of the same cluster state per snapshot: one cached and +unused, one re-LISTed and thrown away. + +## The decision + +- **Embrace `WatchListClient`.** Keep the client-go feature gate on. Do not + ship the `--disable-watch-list-client` flag that the earlier upgrade-finding + document sketched as a workaround. +- **Snapshots read from the informer cache, not the apiserver.** The + hand-rolled `LIST` path goes away. The dynamic informer becomes the single + source of truth for cluster state, hydrated by streaming-list semantics on + the wire. +- **Treat aggregated-API conformance as a known risk and verify it + explicitly.** See the caveat section; the work item is to audit every + aggregated API we watch before turning this loose in production. + +## Phase A — snapshot from the informer cache + +Pure refactor inside `internal/watch`. No client-go feature-gate change yet. + +- Replace + [`listResourcesForGVR`](../../internal/watch/manager.go#L630-L672) with a + `snapshotFromCache(gvr, namespaces)` helper that calls + `informer.GetStore().List()` (or a `cache.NewGenericLister`-backed `Lister` + so namespace filtering stays cheap). +- Snapshot-time work becomes O(in-memory walk), not O(`LIST` × + namespaces). +- If a snapshot is requested for a GVR with no active informer yet (rule just + added, reconcile hasn't run): start the informer on-demand, + `WaitForCacheSync`, then read the cache. The "initial events" stream *is* + the snapshot, which is exactly the shape streaming-list optimizes for in + Phase B. +- `dynamicClientFromConfig`'s seed-only role at + [`manager.go:309-326`](../../internal/watch/manager.go#L309-L326) goes away. + The informer factory has its own client; the dynamic client only remains + for *informer construction*. +- The "refusing to snapshot a partial cluster view" guard at + [`manager.go:535-540`](../../internal/watch/manager.go#L535-L540) is + preserved — it now triggers on cache-sync timeout or per-GVR informer + failure instead of `LIST` failure. + +Sketch of the seam: + +```go +// Before: re-LISTs the apiserver, per namespace, on every snapshot +list, err := dc.Resource(gvr).Namespace(ns).List(ctx, metav1.ListOptions{}) + +// After: reads the already-hydrated informer cache +objs, err := m.cacheForGVR(gvr).ByNamespace(ns).List(labels.Everything()) +``` + +This phase is independently shippable and is the largest correctness / +performance win. It does not depend on the feature-gate decision; it just +makes Phase B safe to land afterwards. + +## Phase B — turn `WatchListClient` on intentionally + +Once the snapshot path is cache-backed, streaming-list becomes net-positive +in three ways: + +- Cheaper initial sync: one chunked watch instead of a single mega-LIST + response that the apiserver materializes whole in memory. +- Apiserver-side relief, especially for large resources (Secrets, + ConfigMaps). +- The cache is consistent with watch events the moment `HasSynced` flips — + no race window where we re-LIST and miss in-flight deletes. + +Work items: + +- Keep the feature gate on for the standard API surface. Do not introduce a + binary-level kill switch. +- For aggregated APIs that we discover (at startup or runtime) to be + non-conformant: build the informer with a custom `ListWatch` that bypasses + `ToListWatcherWithWatchListSemantics`. That means dropping below + `dynamicinformer.NewDynamicSharedInformerFactory` for those GVRs and + constructing `cache.NewSharedIndexInformer` directly with a plain + `ListWatch`. +- Detection is **per-GVR**, not per-`APIService`. Cozystack proved why: + one aggregated server can host both conformant and non-conformant resources + in different API groups (see the bug report). +- Detection strategy: start the informer with watch-list semantics; if + `HasSynced` doesn't flip within a deadline (e.g. 60 s, with the + missing-bookmark warning as a strong signal), tear it down, fall back to a + plain `ListWatch` for that GVR, and remember the fallback in a process-local + cache so subsequent restarts use the safe path immediately. +- Add an e2e lane that watches at least one aggregated-API resource and + asserts the informer syncs and the snapshot is complete. + +## Phase C — delete the legacy seed code + +After A and B land: + +- `listResourcesForGVR`, the per-namespace LIST fan-out in + `getNamespacesForGVR`, the seed-only dynamic client, and the comments + about "by-hand" seeding all go. +- `GetClusterStateForGitDest` shrinks to roughly: resolve GVRs → ensure + informers running and synced → iterate caches. +- The reconciler-startup story becomes "rules → GVRs → start informers → + snapshot is implicit," which is what the existing design docs were already + pointing toward but never reached. + +## Aggregated-API caveat + +This plan **only works if the aggregated apiservers we watch implement the +WatchList contract correctly**. Specifically, each aggregated `Watch` +implementation must: + +1. Read `options.SendInitialEvents` from the internal `ListOptions`. +2. Deliver all matching existing objects as `ADDED` events. +3. Emit a final `watch.Bookmark` event whose object carries + `metadata.annotations["k8s.io/initial-events-end"] = "true"` and a + `ResourceVersion` equal to the last RV seen during the initial set. +4. Continue with live events thereafter. + +Standard kube-apiserver and resources backed by `genericregistry.Store` get +this for free. Hand-written `rest.Storage` implementations in aggregated +servers — exactly the kind that show up in projects like Cozystack — must +implement it themselves. The Cozystack bug report +([`docs/cozystack-bugreport.md`](../cozystack-bugreport.md)) is a worked +example: one API group conformant, one not, in the same binary. + +Before relying on Phase B in any production cluster, we need to audit every +aggregated API surface present in that cluster. The audit list per cluster is: + +- Enumerate `APIService` objects (`kubectl get apiservices`). +- For each non-local `APIService` we actually watch, confirm one of: + - the backing implementation is `genericregistry.Store`-based (built on + `k8s.io/apiserver`'s default registry — emits the bookmark by + construction), **or** + - the project has explicit `SendInitialEvents` handling in its `Watch` + method (grep for `SendInitialEvents` and + `k8s.io/initial-events-end` in their source), **or** + - the GVR is on our explicit fallback list and gets the plain-ListWatch + informer path from Phase B. + +This audit is not a one-time activity for the project; it's an operational +check the first time we deploy into a cluster with unfamiliar aggregated APIs. +The Phase B runtime fallback exists so a missed audit degrades gracefully +instead of hanging the reflector. + +## Open questions + +- **How aggressively should the Phase B detection probe?** A 60 s deadline is + generous for healthy apiservers but stalls every controller restart by that + amount when an aggregated API is broken. Options: a shorter deadline with a + faster fallback; or a configurable per-GVR override in the `WatchRule` / + `ClusterWatchRule` spec; or a one-time at-startup probe that records + conformance per `APIService`. +- **Where does the fallback memory live?** Process-local is enough for a + single-pod controller, but if we move to leader-elected HA the fallback + decision should arguably be a `Condition` on the `WatchRule` so a new pod + doesn't re-pay the detection cost on every failover. +- **Cache memory cost.** Today the snapshot LIST path holds objects only for + the duration of the snapshot. Cache-backed snapshots hold them for the life + of the informer. For resources with extreme cardinality (e.g. cluster-wide + Pod watches we'd never want), the `WatchRule` matching is already the gate; + this just makes it explicit that adding a rule has a steady-state memory + cost. + +## Out of scope + +- Multi-cluster snapshotting. The dynamic informer is per-cluster; multi- + cluster seed is a separate design. +- Server-side filtering beyond label selectors. `WatchListClient` does not + change the filter contract. +- The audit-ingestion pipeline. Snapshot path and audit path are independent; + this plan does not touch + [`docs/design/audit-metrics-overhaul-plan.md`](audit-metrics-overhaul-plan.md). +- A binary-level "disable WatchListClient" flag. Explicitly rejected by the + decision above. Operators who need to disable the gate in an emergency can + still set `KUBE_FEATURE_WatchListClient=false` via the chart's `.Values.env` + ([`charts/gitops-reverser/templates/deployment.yaml:107-125`](../../charts/gitops-reverser/templates/deployment.yaml#L107-L125)); + we just won't ship that as a first-class flag. + +## Suggested order of work + +1. **Phase A** as one PR — refactor `GetClusterStateForGitDest` onto informer + caches. Self-contained. Largest single win. +2. **Phase B detection + fallback** as a follow-up PR — per-GVR conformance + probe and plain-`ListWatch` fallback. Add an e2e lane that exercises a + known-good aggregated API. +3. **Phase C cleanup** PR — delete the legacy seed code now that nothing + calls it. +4. **Aggregated-API audit checklist** in `docs/` (or an extension of + [`docs/aggregated-api-guide.md`](../aggregated-api-guide.md)) so operators + know what to verify before turning Phase B loose in their cluster. diff --git a/docs/upgrade-finding.md b/docs/upgrade-finding.md new file mode 100644 index 000000000..62103c9fc --- /dev/null +++ b/docs/upgrade-finding.md @@ -0,0 +1,157 @@ +# gitops-reverser WatchList Bookmark Report + +Date: 2026-05-25 + +## Summary + +`apiservice-audit-proxy` 0.5.0 appears to have addressed the original +transport-level watch-stream problem, but the remaining production warning in +`gitops-reverser` points to a separate client-side watch-list behavior. + +`gitops-reverser` is built with `k8s.io/client-go v0.36.1`. In that version, +the `WatchListClient` feature defaults to enabled. Dynamic informers created by +`gitops-reverser` therefore use Kubernetes streaming-list semantics: + +```text +watch=true +sendInitialEvents=true +allowWatchBookmarks=true +resourceVersionMatch=NotOlderThan +``` + +client-go then waits for a special bookmark event annotated as the end of the +initial events stream. The production warning means that bookmark is not being +received. + +## Production Symptom + +Representative log line: + +```text +Warning: event bookmark expired err="pkg/mod/k8s.io/client-go@v0.36.1/tools/cache/reflector.go:343: hasn't received required bookmark event marking the end of initial events stream, received last event 59.998946102s ago" +``` + +This is not a generic watch timeout. It specifically means client-go is waiting +for the initial-events-end bookmark required by streaming-list mode. + +## Evidence + +`gitops-reverser` creates dynamic shared informer factories for watched +resources: + +- `external-resources/gitops-reverser/internal/watch/manager.go` +- `external-resources/gitops-reverser/internal/watch/manager_catalog.go` + +The relevant call sites are: + +```go +dynamicinformer.NewDynamicSharedInformerFactory(client, 0) +dynamicinformer.NewFilteredDynamicSharedInformerFactory(client, 0, ns, nil) +``` + +and: + +```go +dynamicinformer.NewDynamicSharedInformerFactory(dynamicClient, 0) +``` + +In `client-go v0.36.1`, dynamic informers wrap their list/watcher with +watch-list support: + +```go +cache.ToListWatcherWithWatchListSemantics(&cache.ListWatch{...}) +``` + +The reflector enables watch-list mode from the client-go feature gate: + +```go +r.useWatchList = clientfeatures.FeatureGates().Enabled(clientfeatures.WatchListClient) +``` + +`WatchListClient` defaults to true as of the v1.35 client-go feature defaults, +which are present in `v0.36.1`. + +When enabled, the reflector starts the initial stream with: + +```go +metav1.ListOptions{ + ResourceVersion: lastKnownRV, + AllowWatchBookmarks: true, + SendInitialEvents: ptr.To(true), + ResourceVersionMatch: metav1.ResourceVersionMatchNotOlderThan, + TimeoutSeconds: &timeoutSeconds, +} +``` + +It only considers the initial stream complete when it receives a bookmark with: + +```go +meta.GetAnnotations()[metav1.InitialEventsAnnotationKey] == "true" +``` + +The repeated production warning is emitted by client-go while waiting for that +bookmark. + +## Interpretation + +The proxy can pass watch events through, but it does not and should not +synthesize Kubernetes streaming-list bookmarks. The initial-events-end bookmark +must be produced by the Kubernetes API server or aggregated backend serving the +watched resource. + +So the remaining failure is most likely: + +1. `gitops-reverser` accidentally opted into streaming-list mode through the + client-go upgrade. +2. One or more watched API paths, especially aggregated APIs, do not produce + the required initial-events-end bookmark. +3. client-go keeps the reflector in its initial watch-list phase and logs the + missing-bookmark warning every 10 seconds. + +The design documents in `gitops-reverser` discuss future use of +`sendInitialEvents=true`, but the current implementation uses ordinary dynamic +informers. That makes this behavior an implicit dependency-default change, +rather than a fully implemented and tested watch-list design. + +## Immediate Workaround + +Disable the client-go `WatchListClient` feature in the `gitops-reverser` pod: + +```yaml +env: + - name: KUBE_FEATURE_WatchListClient + value: "false" +``` + +The Helm chart already exposes `.Values.env`, so this can be applied as a values +override. + +Example values fragment: + +```yaml +env: + - name: KUBE_FEATURE_WatchListClient + value: "false" +``` + +After rollout, the dynamic informers should return to the older LIST-then-WATCH +behavior and stop requiring the initial-events-end bookmark. + +## Recommended Follow-Up + +For `gitops-reverser`, make the choice explicit instead of relying on the +client-go default: + +- Disable `WatchListClient` by default in the manager process, or document and + set the environment override in the chart defaults. +- Add a regression test or e2e lane that watches resources through the + aggregated API path and confirms informer sync completes. +- Revisit streaming-list support later as an intentional feature, with explicit + tests for `sendInitialEvents=true`, `allowWatchBookmarks=true`, and the + `k8s.io/initial-events-end` bookmark. + +For `apiservice-audit-proxy`, the existing watch-stream work remains useful: +the proxy still must keep watch responses open, flush promptly, avoid audit +body spooling for watches, and avoid imposing write deadlines. This report is +about a different layer: client-go reflector semantics in the downstream +consumer. From a3914fe44afded232a2630fd6a00b0fae0397dd7 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Thu, 28 May 2026 15:14:39 +0000 Subject: [PATCH 02/40] chore: phase 0 --- Taskfile-build.yml | 2 +- test/e2e/E2E_DEBUGGING.md | 27 +++ test/e2e/Taskfile.yml | 29 ++- test/e2e/tools/spec-timings/main.go | 139 +++++++++++++ test/e2e/tools/spec-timings/main_test.go | 250 +++++++++++++++++++++++ test/e2e/tools/ts/main.go | 45 ++++ 6 files changed, 482 insertions(+), 10 deletions(-) create mode 100644 test/e2e/tools/spec-timings/main.go create mode 100644 test/e2e/tools/spec-timings/main_test.go create mode 100644 test/e2e/tools/ts/main.go diff --git a/Taskfile-build.yml b/Taskfile-build.yml index 1957b374e..8b737594c 100644 --- a/Taskfile-build.yml +++ b/Taskfile-build.yml @@ -121,7 +121,7 @@ tasks: cmds: - | export KUBEBUILDER_ASSETS="$({{.ENVTEST}} use {{.ENVTEST_K8S_VERSION}} --bin-dir "$(pwd)/bin" -p path)" - go test $(go list ./... | grep -v /e2e) -coverprofile cover.out + go test $(go list ./... | grep -Ev '/test/e2e$') -coverprofile cover.out lint: desc: Run golangci-lint diff --git a/test/e2e/E2E_DEBUGGING.md b/test/e2e/E2E_DEBUGGING.md index 4b573cc82..9c37dbf8c 100644 --- a/test/e2e/E2E_DEBUGGING.md +++ b/test/e2e/E2E_DEBUGGING.md @@ -126,6 +126,33 @@ Kind Cluster └─ Controller pods (2 replicas, HTTPS metrics) ``` +## Spec Timings + +Every e2e target writes a Ginkgo JSON report to +`.stamps/cluster///ginkgo-report-.json` (e.g. +`ginkgo-report-smoke.json` for `task test-e2e`). Render the per-spec +duration table from the latest run with: + +```bash +go run ./test/e2e/tools/spec-timings \ + .stamps/cluster/k3d-gitops-reverser-test-e2e/gitops-reverser/ginkgo-report-smoke.json +``` + +The companion `test/e2e/tools/ts` reads stdin and prefixes each line with +elapsed seconds + wallclock. Useful for decorating cold-run logs that +don't have structured JSON yet: + +```bash +task test-e2e 2>&1 | go run ./test/e2e/tools/ts | tee /tmp/e2e.log +``` + +If output looks batched (long pauses then a burst of timestamps), prefix +with `stdbuf -oL -eL` to force line-buffering on `task`/`go test`: + +```bash +stdbuf -oL -eL task test-e2e 2>&1 | go run ./test/e2e/tools/ts | tee /tmp/e2e.log +``` + ## Debugging Failed Tests 1. **Ensure port-forwards are running:** diff --git a/test/e2e/Taskfile.yml b/test/e2e/Taskfile.yml index 1852151c1..a332dc1dd 100644 --- a/test/e2e/Taskfile.yml +++ b/test/e2e/Taskfile.yml @@ -199,7 +199,8 @@ tasks: export INSTALL_MODE="{{.INSTALL_MODE}}" export NAMESPACE="{{.NAMESPACE}}" export E2E_AGE_KEY_FILE="{{.CS}}/age-key.txt" - go test -timeout "{{.E2E_GO_TEST_TIMEOUT}}" ./test/e2e/ -v -ginkgo.v -ginkgo.label-filter=smoke + go test -timeout "{{.E2E_GO_TEST_TIMEOUT}}" ./test/e2e/ -v -ginkgo.v -ginkgo.label-filter=smoke \ + -ginkgo.json-report="{{.CS}}/{{.NAMESPACE}}/ginkgo-report-smoke.json" test-e2e-full: desc: Run the full e2e suite, including slow and specialized scenarios @@ -209,7 +210,8 @@ tasks: export INSTALL_MODE="{{.INSTALL_MODE}}" export NAMESPACE="{{.NAMESPACE}}" export E2E_AGE_KEY_FILE="{{.CS}}/age-key.txt" - go test -timeout "{{.E2E_GO_TEST_TIMEOUT}}" ./test/e2e/ -v -ginkgo.v + go test -timeout "{{.E2E_GO_TEST_TIMEOUT}}" ./test/e2e/ -v -ginkgo.v \ + -ginkgo.json-report="{{.CS}}/{{.NAMESPACE}}/ginkgo-report-full.json" test-e2e-manager: desc: Run manager-focused e2e scenarios @@ -219,7 +221,8 @@ tasks: export INSTALL_MODE="{{.INSTALL_MODE}}" export NAMESPACE="{{.NAMESPACE}}" export E2E_AGE_KEY_FILE="{{.CS}}/age-key.txt" - go test -timeout "{{.E2E_GO_TEST_TIMEOUT}}" ./test/e2e/ -v -ginkgo.v -ginkgo.label-filter=manager + go test -timeout "{{.E2E_GO_TEST_TIMEOUT}}" ./test/e2e/ -v -ginkgo.v -ginkgo.label-filter=manager \ + -ginkgo.json-report="{{.CS}}/{{.NAMESPACE}}/ginkgo-report-manager.json" test-e2e-signing: desc: Run commit-signing e2e scenarios @@ -229,7 +232,8 @@ tasks: export INSTALL_MODE="{{.INSTALL_MODE}}" export NAMESPACE="{{.NAMESPACE}}" export E2E_AGE_KEY_FILE="{{.CS}}/age-key.txt" - go test -timeout "{{.E2E_GO_TEST_TIMEOUT}}" ./test/e2e/ -v -ginkgo.v -ginkgo.label-filter=signing + go test -timeout "{{.E2E_GO_TEST_TIMEOUT}}" ./test/e2e/ -v -ginkgo.v -ginkgo.label-filter=signing \ + -ginkgo.json-report="{{.CS}}/{{.NAMESPACE}}/ginkgo-report-signing.json" test-image-refresh: desc: Validate the image-refresh dependency chain @@ -239,7 +243,8 @@ tasks: export INSTALL_MODE="{{.INSTALL_MODE}}" export NAMESPACE="{{.NAMESPACE}}" export E2E_AGE_KEY_FILE="{{.CS}}/age-key.txt" - go test -timeout "{{.E2E_GO_TEST_TIMEOUT}}" ./test/e2e/ -v -ginkgo.v -ginkgo.label-filter=image-refresh + go test -timeout "{{.E2E_GO_TEST_TIMEOUT}}" ./test/e2e/ -v -ginkgo.v -ginkgo.label-filter=image-refresh \ + -ginkgo.json-report="{{.CS}}/{{.NAMESPACE}}/ginkgo-report-image-refresh.json" test-e2e-quickstart-helm: desc: Run quickstart smoke test with Helm install @@ -251,7 +256,8 @@ tasks: export E2E_ENABLE_QUICKSTART_FRAMEWORK=true export E2E_QUICKSTART_MODE=helm export E2E_AGE_KEY_FILE="{{.CS}}/age-key.txt" - go test -timeout "{{.E2E_GO_TEST_TIMEOUT}}" ./test/e2e/ -v -ginkgo.v -ginkgo.label-filter=quickstart-framework + go test -timeout "{{.E2E_GO_TEST_TIMEOUT}}" ./test/e2e/ -v -ginkgo.v -ginkgo.label-filter=quickstart-framework \ + -ginkgo.json-report="{{.CS}}/{{.NAMESPACE}}/ginkgo-report-quickstart-helm.json" test-e2e-quickstart-manifest: desc: Run quickstart smoke test with manifest install @@ -263,7 +269,8 @@ tasks: export E2E_ENABLE_QUICKSTART_FRAMEWORK=true export E2E_QUICKSTART_MODE=plain-manifests-file export E2E_AGE_KEY_FILE="{{.CS}}/age-key.txt" - go test -timeout "{{.E2E_GO_TEST_TIMEOUT}}" ./test/e2e/ -v -ginkgo.v -ginkgo.label-filter=quickstart-framework + go test -timeout "{{.E2E_GO_TEST_TIMEOUT}}" ./test/e2e/ -v -ginkgo.v -ginkgo.label-filter=quickstart-framework \ + -ginkgo.json-report="{{.CS}}/{{.NAMESPACE}}/ginkgo-report-quickstart-manifest.json" playground-bootstrap: desc: Bootstrap a reusable repo-backed playground @@ -443,7 +450,8 @@ tasks: export INSTALL_MODE="{{.INSTALL_MODE}}" export NAMESPACE="{{.NAMESPACE}}" export E2E_AGE_KEY_FILE="{{.CS}}/age-key.txt" - go test -timeout "{{.E2E_GO_TEST_TIMEOUT}}" ./test/e2e/ -v -ginkgo.v -ginkgo.label-filter=audit-redis + go test -timeout "{{.E2E_GO_TEST_TIMEOUT}}" ./test/e2e/ -v -ginkgo.v -ginkgo.label-filter=audit-redis \ + -ginkgo.json-report="{{.CS}}/{{.NAMESPACE}}/ginkgo-report-audit-redis.json" test-e2e-aggregated-api: desc: Run aggregated API server e2e scenarios @@ -453,8 +461,11 @@ tasks: export INSTALL_MODE="{{.INSTALL_MODE}}" export NAMESPACE="{{.NAMESPACE}}" export E2E_AGE_KEY_FILE="{{.CS}}/age-key.txt" - go test -timeout "{{.E2E_GO_TEST_TIMEOUT}}" ./test/e2e/ -v -ginkgo.v -ginkgo.label-filter=aggregated-api + go test -timeout "{{.E2E_GO_TEST_TIMEOUT}}" ./test/e2e/ -v -ginkgo.v -ginkgo.label-filter=aggregated-api \ + -ginkgo.json-report="{{.CS}}/{{.NAMESPACE}}/ginkgo-report-aggregated-api.json" + # test-e2e-demo intentionally has no -ginkgo.json-report flag: this target + # is a demo-setup runner, not a measurable test suite. test-e2e-demo: desc: Prepare a reusable demo repo and leave demo resources in place cmds: diff --git a/test/e2e/tools/spec-timings/main.go b/test/e2e/tools/spec-timings/main.go new file mode 100644 index 000000000..7b0f839ac --- /dev/null +++ b/test/e2e/tools/spec-timings/main.go @@ -0,0 +1,139 @@ +/* +SPDX-License-Identifier: Apache-2.0 + +Copyright 2025 ConfigButler + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// spec-timings reads a Ginkgo JSON report and prints a duration-sorted +// table of specs with each spec's share of the total wallclock. +package main + +import ( + "encoding/json" + "flag" + "fmt" + "io" + "os" + "sort" + "strings" + "time" + + "github.com/onsi/ginkgo/v2/types" +) + +const ( + exitUsage = 2 + percentScalar = 100.0 + usageInstruct = "usage: spec-timings " +) + +func main() { + os.Exit(mainRun(os.Args[1:], os.Stdout, os.Stderr)) +} + +func mainRun(args []string, stdout, stderr io.Writer) int { + fs := flag.NewFlagSet("spec-timings", flag.ContinueOnError) + fs.SetOutput(stderr) + if err := fs.Parse(args); err != nil { + return exitUsage + } + if fs.NArg() != 1 { + fmt.Fprintln(stderr, usageInstruct) + return exitUsage + } + if err := run(fs.Arg(0), stdout); err != nil { + fmt.Fprintln(stderr, err) + return 1 + } + return 0 +} + +type specRow struct { + name string + duration time.Duration +} + +func run(path string, out io.Writer) error { + f, err := os.Open(path) + if err != nil { + return fmt.Errorf("open %s: %w", path, err) + } + defer f.Close() + + rows, total, err := parseReport(f) + if err != nil { + return fmt.Errorf("parse %s: %w", path, err) + } + writeTable(out, rows, total) + return nil +} + +func parseReport(r io.Reader) ([]specRow, time.Duration, error) { + data, err := io.ReadAll(r) + if err != nil { + return nil, 0, err + } + var reports []types.Report + if err := json.Unmarshal(data, &reports); err != nil { + return nil, 0, err + } + + var rows []specRow + var total time.Duration + for _, rep := range reports { + for _, sr := range rep.SpecReports { + if sr.LeafNodeType != types.NodeTypeIt { + continue + } + if sr.State == types.SpecStateSkipped || sr.State == types.SpecStatePending { + continue + } + parts := append([]string{}, sr.ContainerHierarchyTexts...) + parts = append(parts, sr.LeafNodeText) + rows = append(rows, specRow{ + name: strings.Join(dedupeConsecutive(parts), " "), + duration: sr.RunTime, + }) + total += sr.RunTime + } + } + sort.SliceStable(rows, func(i, j int) bool { return rows[i].duration > rows[j].duration }) + return rows, total, nil +} + +func writeTable(out io.Writer, rows []specRow, total time.Duration) { + fmt.Fprintln(out, " Duration % Spec") + for _, row := range rows { + pct := 0.0 + if total > 0 { + pct = percentScalar * float64(row.duration) / float64(total) + } + fmt.Fprintf(out, "%8.1f s %4.1f %s\n", row.duration.Seconds(), pct, row.name) + } + fmt.Fprintf(out, "%8.1f s total\n", total.Seconds()) +} + +// dedupeConsecutive collapses adjacent equal entries — nested Describes +// with the same text otherwise print as "Manager Manager should ...". +func dedupeConsecutive(in []string) []string { + out := make([]string, 0, len(in)) + for _, s := range in { + if len(out) > 0 && out[len(out)-1] == s { + continue + } + out = append(out, s) + } + return out +} diff --git a/test/e2e/tools/spec-timings/main_test.go b/test/e2e/tools/spec-timings/main_test.go new file mode 100644 index 000000000..82db4b949 --- /dev/null +++ b/test/e2e/tools/spec-timings/main_test.go @@ -0,0 +1,250 @@ +/* +SPDX-License-Identifier: Apache-2.0 + +Copyright 2025 ConfigButler + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "bytes" + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/onsi/ginkgo/v2/types" +) + +func fixtureReports() []types.Report { + return []types.Report{{ + SpecReports: []types.SpecReport{ + { + LeafNodeType: types.NodeTypeBeforeSuite, + LeafNodeText: "BeforeSuite", + RunTime: time.Second, + }, + { + LeafNodeType: types.NodeTypeIt, + ContainerHierarchyTexts: []string{"Manager", "Manager"}, + LeafNodeText: "fast spec", + State: types.SpecStatePassed, + RunTime: 300 * time.Millisecond, + }, + { + LeafNodeType: types.NodeTypeIt, + ContainerHierarchyTexts: []string{"Restart Snapshot Safety"}, + LeafNodeText: "long spec", + State: types.SpecStateFailed, + RunTime: 90 * time.Second, + }, + { + LeafNodeType: types.NodeTypeIt, + ContainerHierarchyTexts: []string{"Audit Redis Queue"}, + LeafNodeText: "middle spec", + State: types.SpecStatePassed, + RunTime: 10 * time.Second, + }, + { + LeafNodeType: types.NodeTypeIt, + ContainerHierarchyTexts: []string{"Should Be Skipped"}, + LeafNodeText: "skipped", + State: types.SpecStateSkipped, + RunTime: 0, + }, + { + LeafNodeType: types.NodeTypeIt, + ContainerHierarchyTexts: []string{"Should Be Pending"}, + LeafNodeText: "pending", + State: types.SpecStatePending, + RunTime: 0, + }, + }, + }} +} + +func TestParseReport_FiltersAndSortsAndDedupes(t *testing.T) { + data, err := json.Marshal(fixtureReports()) + if err != nil { + t.Fatalf("marshal fixture: %v", err) + } + + rows, total, err := parseReport(bytes.NewReader(data)) + if err != nil { + t.Fatalf("parseReport: %v", err) + } + + wantNames := []string{ + "Restart Snapshot Safety long spec", + "Audit Redis Queue middle spec", + "Manager fast spec", + } + if len(rows) != len(wantNames) { + t.Fatalf("rows: want %d, got %d (%v)", len(wantNames), len(rows), rows) + } + for i, want := range wantNames { + if rows[i].name != want { + t.Errorf("row[%d]: want %q, got %q", i, want, rows[i].name) + } + } + + wantTotal := 90*time.Second + 10*time.Second + 300*time.Millisecond + if total != wantTotal { + t.Errorf("total: want %v, got %v", wantTotal, total) + } +} + +func TestWriteTable_FormatAndPercentages(t *testing.T) { + rows := []specRow{ + {name: "Restart Snapshot Safety long spec", duration: 90 * time.Second}, + {name: "Audit Redis Queue middle spec", duration: 10 * time.Second}, + {name: "Manager fast spec", duration: 300 * time.Millisecond}, + } + total := 90*time.Second + 10*time.Second + 300*time.Millisecond + + var buf bytes.Buffer + writeTable(&buf, rows, total) + + got := buf.String() + wantLines := []string{ + " Duration % Spec", + " 90.0 s 89.7 Restart Snapshot Safety long spec", + " 10.0 s 10.0 Audit Redis Queue middle spec", + " 0.3 s 0.3 Manager fast spec", + " 100.3 s total", + } + for _, line := range wantLines { + if !strings.Contains(got, line+"\n") { + t.Errorf("output missing line %q\nfull output:\n%s", line, got) + } + } +} + +func TestWriteTable_ZeroTotalAvoidsDivision(t *testing.T) { + var buf bytes.Buffer + writeTable(&buf, nil, 0) + got := buf.String() + if !strings.Contains(got, " Duration % Spec") { + t.Errorf("missing header: %s", got) + } + if !strings.Contains(got, " 0.0 s total") { + t.Errorf("missing total row: %s", got) + } +} + +func TestDedupeConsecutive(t *testing.T) { + cases := []struct { + in, want []string + }{ + {nil, []string{}}, + {[]string{"a"}, []string{"a"}}, + {[]string{"a", "a"}, []string{"a"}}, + {[]string{"a", "a", "b", "b", "a"}, []string{"a", "b", "a"}}, + {[]string{"Manager", "Manager", "spec"}, []string{"Manager", "spec"}}, + } + for _, tc := range cases { + got := dedupeConsecutive(tc.in) + if !equalStrings(got, tc.want) { + t.Errorf("dedupeConsecutive(%v) = %v, want %v", tc.in, got, tc.want) + } + } +} + +func equalStrings(a, b []string) bool { + if len(a) != len(b) { + return false + } + for i := range a { + if a[i] != b[i] { + return false + } + } + return true +} + +func TestRun_EndToEndWithFixtureFile(t *testing.T) { + data, err := json.Marshal(fixtureReports()) + if err != nil { + t.Fatalf("marshal fixture: %v", err) + } + path := filepath.Join(t.TempDir(), "report.json") + if err := os.WriteFile(path, data, 0o600); err != nil { + t.Fatalf("write fixture: %v", err) + } + + var buf bytes.Buffer + if err := run(path, &buf); err != nil { + t.Fatalf("run: %v", err) + } + got := buf.String() + if !strings.Contains(got, "Restart Snapshot Safety long spec") { + t.Errorf("missing top spec in output: %s", got) + } + if !strings.Contains(got, "100.3 s total") { + t.Errorf("missing total in output: %s", got) + } +} + +func TestRun_MissingFile(t *testing.T) { + err := run(filepath.Join(t.TempDir(), "does-not-exist.json"), &bytes.Buffer{}) + if err == nil { + t.Fatal("expected error for missing file") + } +} + +func TestRun_BadJSON(t *testing.T) { + path := filepath.Join(t.TempDir(), "bad.json") + if err := os.WriteFile(path, []byte("not json"), 0o600); err != nil { + t.Fatalf("write fixture: %v", err) + } + err := run(path, &bytes.Buffer{}) + if err == nil { + t.Fatal("expected error for invalid JSON") + } +} + +func TestMainRun_ExitCodes(t *testing.T) { + good := filepath.Join(t.TempDir(), "report.json") + data, err := json.Marshal(fixtureReports()) + if err != nil { + t.Fatalf("marshal fixture: %v", err) + } + if err := os.WriteFile(good, data, 0o600); err != nil { + t.Fatalf("write fixture: %v", err) + } + + cases := []struct { + name string + args []string + wantCode int + }{ + {name: "no args", args: nil, wantCode: exitUsage}, + {name: "too many args", args: []string{"a", "b"}, wantCode: exitUsage}, + {name: "bad flag", args: []string{"-not-a-flag"}, wantCode: exitUsage}, + {name: "missing file", args: []string{"/no/such/file"}, wantCode: 1}, + {name: "success", args: []string{good}, wantCode: 0}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + var stdout, stderr bytes.Buffer + got := mainRun(tc.args, &stdout, &stderr) + if got != tc.wantCode { + t.Errorf("mainRun: code=%d, want %d (stderr=%s)", got, tc.wantCode, stderr.String()) + } + }) + } +} diff --git a/test/e2e/tools/ts/main.go b/test/e2e/tools/ts/main.go new file mode 100644 index 000000000..81583b03d --- /dev/null +++ b/test/e2e/tools/ts/main.go @@ -0,0 +1,45 @@ +/* +SPDX-License-Identifier: Apache-2.0 + +Copyright 2025 ConfigButler + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// ts prefixes each stdin line with the elapsed seconds since startup +// and the current wallclock, then writes it to stdout. Useful when +// decorating long-running e2e logs. +package main + +import ( + "bufio" + "fmt" + "os" + "time" +) + +const maxLineBytes = 1 << 20 + +func main() { + start := time.Now() + scanner := bufio.NewScanner(os.Stdin) + scanner.Buffer(make([]byte, maxLineBytes), maxLineBytes) + for scanner.Scan() { + now := time.Now() + fmt.Printf("%7.3fs | %s | %s\n", now.Sub(start).Seconds(), now.Format("15:04:05"), scanner.Text()) + } + if err := scanner.Err(); err != nil { + fmt.Fprintln(os.Stderr, "ts:", err) + os.Exit(1) + } +} From 2827a5ee45b9d513b829bfaa3e2bf3caf4bad7f4 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Thu, 28 May 2026 20:03:22 +0000 Subject: [PATCH 03/40] chore: Let's dependOn --- Taskfile.yml | 5 ++ test/e2e/Taskfile.yml | 121 +++++++++++++++++++++--------------------- 2 files changed, 66 insertions(+), 60 deletions(-) diff --git a/Taskfile.yml b/Taskfile.yml index 4c73c3036..c8e992a2d 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -5,6 +5,11 @@ version: '3' # See .env.example for the available keys. dotenv: ['.env'] +# Default tasks to run once per invocation so the e2e DAG can fan tasks +# out via `deps:` without racing — e.g. `_cluster-ready` reachable through +# both `_image-loaded` and `_flux-installed` must not start twice. +run: once + set: - errexit - nounset diff --git a/test/e2e/Taskfile.yml b/test/e2e/Taskfile.yml index a332dc1dd..8619f7018 100644 --- a/test/e2e/Taskfile.yml +++ b/test/e2e/Taskfile.yml @@ -105,16 +105,11 @@ vars: tasks: prepare-e2e: desc: Prepare E2E prerequisites for Go tests + # portforward-ensure runs *after* _prepare-e2e-ready, not in parallel + # with it: _webhook-tls-ready inside the prepare graph restarts the + # k3d server node, which would kill any kubectl port-forwards started + # concurrently with it. cmds: - - task: install - - task: _project-image-ready - - task: _image-loaded - - task: _controller-deployed - - task: _age-key - - task: _sops-secret-yaml - - task: _sops-secret-applied - - task: _webhook-tls-ready - - task: _aggregated-api-ready - task: _prepare-e2e-ready - task: portforward-ensure @@ -126,11 +121,9 @@ tasks: portforward-ensure: desc: Ensure port-forwards are running + deps: + - _services-ready cmds: - - task: _cluster-ready - - task: _flux-installed - - task: _flux-setup-ready - - task: _services-ready - | export E2E_KUBECONTEXT="{{.CTX}}" export GITEA_PORT="{{.GITEA_PORT}}" @@ -142,54 +135,34 @@ tasks: install: desc: Install using the configured INSTALL_MODE cmds: + # Validate INSTALL_MODE before dispatching. The actual dispatch uses + # in-process task: invocation so the full dependency graph stays in + # one go-task run (the previous shell-subprocess form lost CTX and + # other task vars when not also exported as env). - | case "{{.INSTALL_MODE}}" in - helm) - task install-helm - ;; - plain-manifests-file) - task install-plain-manifests-file - ;; - config-dir) - task install-config-dir - ;; + helm|config-dir|plain-manifests-file) ;; *) echo "ERROR: INSTALL_MODE must be one of [config-dir helm plain-manifests-file]" >&2 exit 2 ;; esac + - task: 'install-{{.INSTALL_MODE}}' install-helm: - deps: - - helm-sync desc: Install via Helm - cmds: - - task: _cluster-ready - - task: _flux-installed - - task: _flux-setup-ready - - task: _services-ready - - task: _install-helm + deps: + - _install-helm install-plain-manifests-file: desc: Install via dist/install.yaml deps: - - dist-install - cmds: - - task: _cluster-ready - - task: _flux-installed - - task: _flux-setup-ready - - task: _services-ready - - task: _install-plain-manifests-file + - _install-plain-manifests-file install-config-dir: desc: Install via config directory - cmds: - - task: _cluster-ready - - task: _flux-installed - - task: _flux-setup-ready - - task: _services-ready - - task: manifests - - task: _install-config-dir + deps: + - _install-config-dir test-e2e: desc: Run the smoke e2e suite @@ -579,6 +552,8 @@ tasks: _flux-installed: method: timestamp + deps: + - _cluster-ready sources: - '{{.CS}}/ready' - '{{.FLUX_OPERATOR_DIR}}/**/*.yaml' @@ -594,7 +569,6 @@ tasks: -n "{{.FLUX_NAMESPACE}}" \ get fluxinstance "{{.FLUX_INSTANCE_NAME}}" >/dev/null cmds: - - task: _cluster-ready - | {{.FLUX_OPERATOR}} install \ --kube-context "{{.CTX}}" \ @@ -626,6 +600,8 @@ tasks: _flux-setup-ready: method: timestamp + deps: + - _flux-installed sources: - '{{.CS}}/flux.installed' - '{{.FLUX_SERVICES_DIR}}/**/*' @@ -634,7 +610,6 @@ tasks: status: - test -f "{{.CS}}/flux-setup.ready" cmds: - - task: _flux-installed - | kubectl --context "{{.CTX}}" apply -k "{{.FLUX_SERVICES_DIR}}" CTX="{{.CTX}}" FLUX_SERVICES_WAIT_TIMEOUT="{{.FLUX_SERVICES_WAIT_TIMEOUT}}" \ @@ -643,6 +618,8 @@ tasks: _services-ready: method: timestamp + deps: + - _flux-setup-ready sources: - '{{.CS}}/flux-setup.ready' - 'test/e2e/setup/manifests/**/*.yaml' @@ -651,7 +628,6 @@ tasks: status: - test -f "{{.CS}}/services.ready" cmds: - - task: _flux-setup-ready - | kubectl --context "{{.CTX}}" wait --for=condition=Established \ crd/prometheuses.monitoring.coreos.com \ @@ -700,6 +676,8 @@ tasks: _project-image-ready-local: method: timestamp + deps: + - _controller-image-id sources: - '{{.IS}}/controller.id' generates: @@ -708,7 +686,6 @@ tasks: - test -f "{{.IS}}/project-image.ready" - test "$(<"{{.IS}}/project-image.ready")" = "{{.PROJECT_IMAGE}}" cmds: - - task: _controller-image-id - mkdir -p "{{.IS}}" - printf '%s\n' "{{.PROJECT_IMAGE}}" > "{{.IS}}/project-image.ready" @@ -727,17 +704,17 @@ tasks: printf '%s\n' "{{.PROJECT_IMAGE}}" > "{{.IS}}/project-image.ready" _project-image-ready: + # In-process dispatch so the leaf task runs inside the same go-task + # graph as its caller — preserves CTX and lets it run in parallel with + # the cluster/Flux chain via deps on _image-loaded. cmds: - - | - if [ -n "{{.PROJECT_IMAGE_PROVIDED}}" ]; then - task _project-image-ready-provided - else - task _controller-image-id - task _project-image-ready-local - fi + - task: '{{if .PROJECT_IMAGE_PROVIDED}}_project-image-ready-provided{{else}}_project-image-ready-local{{end}}' _image-loaded: method: timestamp + deps: + - _cluster-ready + - _project-image-ready sources: - '{{.CS}}/ready' - '{{.IS}}/project-image.ready' @@ -746,7 +723,6 @@ tasks: status: - test -f "{{.CS}}/image.loaded" cmds: - - task: _cluster-ready - mkdir -p "{{.CS}}" - | CTX="{{.CTX}}" CLUSTER_NAME="{{.CLUSTER_NAME}}" PROJECT_IMAGE="{{.PROJECT_IMAGE}}" \ @@ -757,6 +733,9 @@ tasks: bash hack/e2e/load-image.sh _install-helm: + deps: + - _services-ready + - helm-sync sources: - '{{.CS}}/services.ready' - charts/gitops-reverser/Chart.yaml @@ -800,6 +779,9 @@ tasks: fi _install-config-dir: + deps: + - _services-ready + - manifests sources: - '{{.CS}}/services.ready' - config/**/* @@ -817,6 +799,9 @@ tasks: _install-plain-manifests-file: method: timestamp + deps: + - _services-ready + - dist-install sources: - '{{.CS}}/services.ready' - dist/install.yaml @@ -838,6 +823,9 @@ tasks: _controller-deployed: method: timestamp + deps: + - install + - _image-loaded sources: - '{{.CS}}/{{.NAMESPACE}}/{{.INSTALL_MODE}}/install.yaml' - '{{.CS}}/image.loaded' @@ -861,6 +849,8 @@ tasks: _webhook-tls-ready: method: timestamp + deps: + - _controller-deployed sources: - '{{.CS}}/{{.NAMESPACE}}/controller.deployed' generates: @@ -876,6 +866,9 @@ tasks: _aggregated-api-webhook-kubeconfig-ready: method: timestamp + deps: + - _services-ready + - _webhook-tls-ready sources: - '{{.CS}}/services.ready' - '{{.CS}}/{{.NAMESPACE}}/webhook-tls.ready' @@ -886,8 +879,6 @@ tasks: status: - test -f "{{.CS}}/aggregated-api-webhook-kubeconfig.ready" cmds: - - task: _services-ready - - task: _webhook-tls-ready - | CTX="{{.CTX}}" NAMESPACE="{{.NAMESPACE}}" PROXY_NAMESPACE="aggregated-api" \ bash hack/e2e/prepare-aggregated-api-webhook-kubeconfig.sh @@ -895,6 +886,8 @@ tasks: _aggregated-api-ready: method: timestamp + deps: + - _aggregated-api-webhook-kubeconfig-ready sources: - '{{.CS}}/services.ready' - '{{.CS}}/aggregated-api-webhook-kubeconfig.ready' @@ -904,8 +897,6 @@ tasks: status: - test -f "{{.CS}}/aggregated-api.ready" cmds: - - task: _services-ready - - task: _aggregated-api-webhook-kubeconfig-ready - | kubectl --context "{{.CTX}}" -n aggregated-api wait certificate/audit-pass-through-proxy-serving-cert \ --for=condition=Ready --timeout=180s @@ -945,6 +936,8 @@ tasks: _sops-secret-yaml: method: timestamp + deps: + - _age-key sources: - '{{.CS}}/age-key.txt' - test/e2e/tools/gen-age-key/main.go @@ -969,6 +962,9 @@ tasks: _sops-secret-applied: method: timestamp + deps: + - _sops-secret-yaml + - install sources: - '{{.CS}}/{{.NAMESPACE}}/sops-secret.yaml' - '{{.CS}}/{{.NAMESPACE}}/{{.INSTALL_MODE}}/install.yaml' @@ -983,6 +979,11 @@ tasks: _prepare-e2e-ready: method: timestamp + deps: + - _controller-deployed + - _webhook-tls-ready + - _sops-secret-applied + - _aggregated-api-ready sources: - '{{.CS}}/{{.NAMESPACE}}/{{.INSTALL_MODE}}/install.yaml' - '{{.CS}}/image.loaded' From 3c96ae8975f93f74ea61150744e189781158ddfa Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Thu, 28 May 2026 20:52:38 +0000 Subject: [PATCH 04/40] chore: Would this work? --- test/e2e/Taskfile.yml | 41 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 38 insertions(+), 3 deletions(-) diff --git a/test/e2e/Taskfile.yml b/test/e2e/Taskfile.yml index 8619f7018..a15659e6b 100644 --- a/test/e2e/Taskfile.yml +++ b/test/e2e/Taskfile.yml @@ -732,10 +732,39 @@ tasks: CONTAINER_TOOL="{{.CONTAINER_TOOL}}" K3D="{{.K3D}}" STAMP_FILE="{{.CS}}/image.loaded" \ bash hack/e2e/load-image.sh + _install-cleanup: + # Wipe prior install state so a new install mode can take over cleanly. + # Runs cleanup-installs.sh, which deletes any existing install.yaml-owned + # resources, drops the install namespace, and rm -rf's every stamp under + # .stamps/cluster/// except git-* fixtures. + # + # This used to live inline at the top of every _install- task. As + # its own DAG node it can be a parallel-safe prerequisite for both the + # active _install- and for tasks like _sops-secret-yaml that write + # into the same namespace dir. + # + # Status: if the install.yaml for the *current* INSTALL_MODE already + # exists we're either warm or just re-running the same mode, and the + # in-place apply/upgrade does not need a wipe. Cleanup is only required + # when switching modes or on a brand-new CTX where the file is missing. + method: timestamp + deps: + - _cluster-ready + generates: + - '{{.CS}}/install-cleanup.done' + status: + - test -f "{{.CS}}/{{.NAMESPACE}}/{{.INSTALL_MODE}}/install.yaml" + cmds: + - | + CTX="{{.CTX}}" KUBECTL="{{.KUBECTL}}" bash hack/cleanup-installs.sh + mkdir -p "{{.CS}}" + touch "{{.CS}}/install-cleanup.done" + _install-helm: deps: - _services-ready - helm-sync + - _install-cleanup sources: - '{{.CS}}/services.ready' - charts/gitops-reverser/Chart.yaml @@ -747,7 +776,6 @@ tasks: - '{{.CS}}/{{.NAMESPACE}}/helm/install.yaml' cmds: - | - CTX="{{.CTX}}" KUBECTL="{{.KUBECTL}}" bash hack/cleanup-installs.sh mkdir -p "{{.CS}}/{{.NAMESPACE}}/helm" skip_crds_arg="" if kubectl --context "{{.CTX}}" get crd \ @@ -782,6 +810,7 @@ tasks: deps: - _services-ready - manifests + - _install-cleanup sources: - '{{.CS}}/services.ready' - config/**/* @@ -789,7 +818,6 @@ tasks: - '{{.CS}}/{{.NAMESPACE}}/config-dir/install.yaml' cmds: - | - CTX="{{.CTX}}" KUBECTL="{{.KUBECTL}}" bash hack/cleanup-installs.sh mkdir -p "{{.CS}}/{{.NAMESPACE}}/config-dir" CTX="{{.CTX}}" NAMESPACE="{{.NAMESPACE}}" PROJECT_IMAGE="{{.PROJECT_IMAGE}}" \ E2E_VALKEY_PASSWORD="{{.E2E_VALKEY_PASSWORD}}" \ @@ -802,6 +830,7 @@ tasks: deps: - _services-ready - dist-install + - _install-cleanup sources: - '{{.CS}}/services.ready' - dist/install.yaml @@ -809,7 +838,6 @@ tasks: - '{{.CS}}/{{.NAMESPACE}}/plain-manifests-file/install.yaml' cmds: - | - CTX="{{.CTX}}" KUBECTL="{{.KUBECTL}}" bash hack/cleanup-installs.sh mkdir -p "{{.CS}}/{{.NAMESPACE}}/plain-manifests-file" CTX="{{.CTX}}" NAMESPACE="{{.NAMESPACE}}" E2E_VALKEY_PASSWORD="{{.E2E_VALKEY_PASSWORD}}" \ bash hack/e2e/ensure-valkey-auth.sh @@ -936,8 +964,15 @@ tasks: _sops-secret-yaml: method: timestamp + # _install-cleanup runs first: it wipes everything under + # .stamps/cluster/// except git-* fixtures, including this + # task's output. Without this dep, status: was evaluated in parallel + # with the cleanup, found a stale file from a prior run, reported "up + # to date", and then _sops-secret-applied tried to kubectl apply a + # file the cleanup had just deleted. deps: - _age-key + - _install-cleanup sources: - '{{.CS}}/age-key.txt' - test/e2e/tools/gen-age-key/main.go From 5d9c8bcdb78da86fa93ff3a0caabd066faf22416 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Thu, 28 May 2026 21:28:19 +0000 Subject: [PATCH 05/40] =?UTF-8?q?chore:=20phase=202=20=E2=80=94=20split=20?= =?UTF-8?q?Manager=20Describe=20into=20four=20files?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Mechanical split of the 21-spec Manager Describe in test/e2e/e2e_test.go into four files, each its own top-level Describe with file-local fixtures: - controller_basics_e2e_test.go (specs 1–5) - gitprovider_validation_e2e_test.go (specs 6–9) - watchrule_configmap_secret_e2e_test.go (specs 10–16) - crd_lifecycle_e2e_test.go (specs 17–21) All four keep Label("manager") so `task test-e2e-manager` still selects all 21. Each file owns its own namespace + RepoArtifacts; the package-global `managerRepo` is gone. CRD ownership now lives in crd_lifecycle_e2e_test.go's AfterAll. No spec merged, no Ordered removed, no parallelism enabled, no fix for the two 5s sleeps — those are later phases per docs/design/e2e-speedup-plan.md. Validated: task lint, task test, task test-e2e-manager (21/21 green in 194 s on the warm dev cluster). Co-Authored-By: Claude Opus 4.7 --- test/e2e/controller_basics_e2e_test.go | 273 +++ test/e2e/crd_lifecycle_e2e_test.go | 699 +++++++ test/e2e/e2e_test.go | 1697 ----------------- test/e2e/gitprovider_validation_e2e_test.go | 162 ++ .../watchrule_configmap_secret_e2e_test.go | 757 ++++++++ 5 files changed, 1891 insertions(+), 1697 deletions(-) create mode 100644 test/e2e/controller_basics_e2e_test.go create mode 100644 test/e2e/crd_lifecycle_e2e_test.go create mode 100644 test/e2e/gitprovider_validation_e2e_test.go create mode 100644 test/e2e/watchrule_configmap_secret_e2e_test.go diff --git a/test/e2e/controller_basics_e2e_test.go b/test/e2e/controller_basics_e2e_test.go new file mode 100644 index 000000000..6ac98a377 --- /dev/null +++ b/test/e2e/controller_basics_e2e_test.go @@ -0,0 +1,273 @@ +/* +SPDX-License-Identifier: Apache-2.0 + +Copyright 2025 ConfigButler + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "fmt" + "strings" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/ConfigButler/gitops-reverser/test/utils" +) + +var _ = Describe("Manager Controller Basics", Label("manager"), Ordered, func() { + var controllerPodName string // Name of first controller pod for logging + + BeforeAll(func() { + By("setting up Prometheus client for metrics testing") + setupPrometheusClient() + verifyPrometheusAvailable() + }) + + SetDefaultEventuallyTimeout(30 * time.Second) + SetDefaultEventuallyPollingInterval(time.Second) + + It("should run successfully", Label("smoke"), func() { + By("validating that the gitops-reverser pods are running as expected") + verifyControllerUp := func(g Gomega) { + // Get the names of the gitops-reverser pods + podOutput, err := kubectlRunInNamespace( + namespace, + "get", + "pods", + "-l", + "control-plane=gitops-reverser", + "-o", "go-template={{ range .items }}"+ + "{{ if not .metadata.deletionTimestamp }}"+ + "{{ .metadata.name }}"+ + "{{ \"\\n\" }}{{ end }}{{ end }}", + ) + g.Expect(err).NotTo(HaveOccurred(), "Failed to retrieve gitops-reverser pod information") + podNames := utils.GetNonEmptyLines(podOutput) + g.Expect(podNames).To(HaveLen(1), "expected exactly 1 controller pod running") + controllerPodName = podNames[0] // Use first pod for logging + g.Expect(controllerPodName).To(ContainSubstring("gitops-reverser")) + + // Validate all pods' status + for _, podName := range podNames { + output, err := kubectlRunInNamespace( + namespace, + "get", + "pods", + podName, + "-o", + "jsonpath={.status.phase}", + ) + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(output).To(Equal("Running"), fmt.Sprintf("Incorrect status for pod %s", podName)) + } + } + Eventually(verifyControllerUp).Should(Succeed()) + }) + + It("should expose the controller service", Label("smoke"), func() { + By("verifying controller service exists") + _, err := kubectlRunInNamespace(namespace, "get", "svc", controllerServiceName) + Expect(err).NotTo(HaveOccurred(), "Controller service should exist") + + By("verifying controller service routes to the controller pod") + Eventually(func(g Gomega) { + output, endpointsErr := kubectlRunInNamespace( + namespace, + "get", + "endpoints", + controllerServiceName, + "-o", + "jsonpath={.subsets[*].addresses[*].targetRef.name}", + ) + g.Expect(endpointsErr).NotTo(HaveOccurred(), "Failed to get controller service endpoints") + + lines := utils.GetNonEmptyLines(output) + podSet := map[string]struct{}{} + for _, line := range lines { + if !strings.HasPrefix(line, "Warning:") && + !strings.Contains(line, "deprecated") && + strings.Contains(line, "gitops-reverser") { + podSet[line] = struct{}{} + } + } + var podNames []string + for podName := range podSet { + podNames = append(podNames, podName) + } + + g.Expect(podNames).To(HaveLen(1), "controller service should route to exactly 1 pod") + g.Expect(podNames[0]).To(Equal(controllerPodName), "controller service should route to controller pod") + }, 30*time.Second).Should(Succeed()) + }) + + It("should expose the audit service separately", func() { + By("verifying audit service exists") + _, err := kubectlRunInNamespace(namespace, "get", "svc", auditServiceName) + Expect(err).NotTo(HaveOccurred(), "Audit service should exist") + + By("verifying audit service routes to the controller pod") + Eventually(func(g Gomega) { + output, endpointsErr := kubectlRunInNamespace( + namespace, + "get", + "endpoints", + auditServiceName, + "-o", + "jsonpath={.subsets[*].addresses[*].targetRef.name}", + ) + g.Expect(endpointsErr).NotTo(HaveOccurred(), "Failed to get audit service endpoints") + + lines := utils.GetNonEmptyLines(output) + var podNames []string + for _, line := range lines { + if strings.HasPrefix(line, "Warning:") || strings.Contains(line, "deprecated") { + continue + } + podNames = append(podNames, line) + } + + g.Expect(podNames).To(HaveLen(1), "audit service should route to exactly 1 pod") + g.Expect(podNames[0]).To(Equal(controllerPodName), "audit service should route to controller pod") + }, 30*time.Second).Should(Succeed()) + }) + + It("should ensure the metrics endpoint is serving metrics", Label("smoke"), func() { + By("validating that the controller service is available for metrics") + _, err := kubectlRunInNamespace(namespace, "get", "service", controllerServiceName) + Expect(err).NotTo(HaveOccurred(), "Controller service should exist") + + By("waiting for the metrics endpoint to be ready") + verifyMetricsEndpointReady := func(g Gomega) { + output, err := kubectlRunInNamespace(namespace, "get", "endpoints", controllerServiceName) + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(output).To(ContainSubstring("8443"), "Metrics endpoint is not ready") + } + Eventually(verifyMetricsEndpointReady).Should(Succeed()) + + By("verifying that the controller manager is serving the metrics server") + verifyMetricsServerStarted := func(g Gomega) { + output, err := kubectlRunInNamespace(namespace, "logs", controllerPodName) + g.Expect(err).NotTo(HaveOccurred()) + jsonMetricsLogLine := "\"logger\":\"controller-runtime.metrics\"," + + "\"msg\":\"Serving metrics server\"" + g.Expect(output).To( + SatisfyAny( + ContainSubstring("controller-runtime.metrics\tServing metrics server"), + ContainSubstring(jsonMetricsLogLine), + ), + "Metrics server not yet started", + ) + } + Eventually(verifyMetricsServerStarted).Should(Succeed()) + + By("waiting for Prometheus to scrape controller metrics") + waitForMetric("sum(up{job='gitops-reverser'})", + func(v float64) bool { return v == 1 }, + "metrics endpoint should be up") + + By("verifying basic process metrics are exposed") + waitForMetric("sum(process_cpu_seconds_total{job='gitops-reverser'})", + func(v float64) bool { return v > 0 }, + "process metrics should exist") + + By("verifying metrics from the controller pod") + podCount, err := queryPrometheus("sum(up{job='gitops-reverser'})") + Expect(err).NotTo(HaveOccurred()) + Expect(podCount).To(BeEquivalentTo(1), "Should scrape from 1 controller pod") + + fmt.Printf("✅ Metrics collection verified from %.0f pods\n", podCount) + fmt.Printf("📊 Inspect metrics: %s\n", getPrometheusURL()) + }) + + It("should receive audit webhook events from kube-apiserver", Label("smoke"), func() { + By("recording baseline audit event count") + baselineAuditEvents, err := queryPrometheus("sum(gitopsreverser_audit_events_received_total) or vector(0)") + Expect(err).NotTo(HaveOccurred()) + fmt.Printf("📊 Baseline audit events: %.0f\n", baselineAuditEvents) + baselineOfficialAuditEvents, err := queryPrometheus( + "sum(gitopsreverser_audit_events_received_total{source='official'}) or vector(0)", + ) + Expect(err).NotTo(HaveOccurred()) + fmt.Printf("📊 Baseline official audit events: %.0f\n", baselineOfficialAuditEvents) + + By("creating a ConfigMap to trigger audit events") + _, err = kubectlRunInNamespace( + namespace, + "create", + "configmap", + "audit-test-cm", + "--from-literal=test=audit", + ) + Expect(err).NotTo(HaveOccurred(), "ConfigMap creation should succeed") + _, err = kubectlRunInNamespace( + namespace, + "patch", + "configmap", + "audit-test-cm", + "--type=merge", + "--patch", + `{"data":{"test":"audit-updated"}}`, + ) + Expect(err).NotTo(HaveOccurred(), "ConfigMap update should succeed") + + By("waiting for audit event metric to increment") + waitForMetricWithTimeout("sum(gitopsreverser_audit_events_received_total) or vector(0)", + func(v float64) bool { return v > baselineAuditEvents }, + "audit events should increment", 2*time.Minute) + waitForMetricWithTimeout( + "sum(gitopsreverser_audit_events_received_total{source='official'}) or vector(0)", + func(v float64) bool { return v > baselineOfficialAuditEvents }, + "audit events should increment for source=official", 2*time.Minute, + ) + + By("verifying audit events were received") + currentAuditEvents, err := queryPrometheus("sum(gitopsreverser_audit_events_received_total) or vector(0)") + Expect(err).NotTo(HaveOccurred()) + Expect(currentAuditEvents).To(BeNumerically(">", baselineAuditEvents), + "Should have received audit events from kube-apiserver") + + By("verifying the EventList ingress metric appears for source=official") + waitForMetricWithTimeout( + "sum(gitopsreverser_audit_eventlists_total{source='official'}) or vector(0)", + func(v float64) bool { return v > 0 }, + "EventList ingress requests should be counted for source=official", 2*time.Minute, + ) + + By("verifying no removed cluster/gvr/action label remains on any audit series") + for _, removed := range []string{"cluster", "gvr", "action"} { + stale, queryErr := queryPrometheus(fmt.Sprintf( + "sum({__name__=~'gitopsreverser_audit_.+', %s=~'.+'}) or vector(0)", removed)) + Expect(queryErr).NotTo(HaveOccurred()) + Expect(stale).To(BeNumerically("==", 0), + fmt.Sprintf("no audit series should still carry the %q label", removed)) + } + + newEvents := currentAuditEvents - baselineAuditEvents + fmt.Printf("✅ Received %.0f new audit events from kube-apiserver\n", newEvents) + fmt.Printf("📊 Total audit events: %.0f\n", currentAuditEvents) + + By("cleaning up audit test resources") + _, _ = kubectlRunInNamespace( + namespace, + "delete", + "configmap", + "audit-test-cm", + "--ignore-not-found=true", + ) + }) +}) diff --git a/test/e2e/crd_lifecycle_e2e_test.go b/test/e2e/crd_lifecycle_e2e_test.go new file mode 100644 index 000000000..65d894c9a --- /dev/null +++ b/test/e2e/crd_lifecycle_e2e_test.go @@ -0,0 +1,699 @@ +/* +SPDX-License-Identifier: Apache-2.0 + +Copyright 2025 ConfigButler + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "fmt" + "os" + "os/exec" + "path/filepath" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { + var ( + testNs string + crdLifecycleRepo *RepoArtifacts + ) + + BeforeAll(func() { + By("creating CRD lifecycle test namespace") + testNs = testNamespaceFor("manager-crd") + _, _ = kubectlRun("create", "namespace", testNs) // idempotent; ignore AlreadyExists + + By("setting up Gitea repo and credentials for CRD lifecycle tests") + crdLifecycleRepo = SetupRepo( + resolveE2EContext(), + testNs, + fmt.Sprintf("e2e-manager-crd-%d", GinkgoRandomSeed()), + ) + + By("applying git secrets to test namespace") + _, err := kubectlRunInNamespace(testNs, "apply", "-f", crdLifecycleRepo.SecretsYAML) + Expect(err).NotTo(HaveOccurred(), "failed to apply git secrets to test namespace") + applySOPSAgeKeyToNamespace(testNs) + + By("creating shared gitprovider-normal for CRD lifecycle specs") + createGitProviderWithURLInNamespace( + "gitprovider-normal", + testNs, + crdLifecycleRepo.GitSecretHTTP, + crdLifecycleRepo.RepoURLHTTP, + ) + verifyResourceStatus( + "gitprovider", "gitprovider-normal", testNs, + "True", "Ready", "Repository connectivity validated", + ) + }) + + AfterAll(func() { + By("cleaning up CRD lifecycle test resources") + + // Clean up IceCreamOrder CRD (cluster-scoped — must delete explicitly) + _, _ = kubectlRun( + "delete", + "crd", + "icecreamorders.shop.example.com", + "--ignore-not-found=true", + ) + + cleanupNamespace(testNs) + + By("test infrastructure still running for debugging") + fmt.Printf("\n") + fmt.Printf("═══════════════════════════════════════════════════════════\n") + fmt.Printf("📊 E2E Infrastructure kept running for debugging purposes:\n") + fmt.Printf("═══════════════════════════════════════════════════════════\n") + fmt.Printf(" Prometheus: http://localhost:19090\n") + fmt.Printf(" Gitea: http://localhost:13000\n") + fmt.Printf(" Flux UI: http://localhost:19080\n") + fmt.Printf("\n") + fmt.Printf("═══════════════════════════════════════════════════════════\n") + fmt.Printf("\n") + }) + + SetDefaultEventuallyTimeout(30 * time.Second) + SetDefaultEventuallyPollingInterval(time.Second) + + It("should create Git commit when IceCreamOrder CRD is installed via ClusterWatchRule", Label("smoke"), func() { + gitProviderName := "gitprovider-normal" + clusterWatchRuleName := "clusterwatchrule-crd-install" + crdName := "icecreamorders.shop.example.com" + + By("creating ClusterWatchRule with Cluster scope for CRDs") + destName := clusterWatchRuleName + "-dest" + createGitTarget(destName, testNs, gitProviderName, "e2e/crd-install-test", "main") + + clusterWatchRuleData := struct { + Name string + DestinationName string + Namespace string + }{ + Name: clusterWatchRuleName, + DestinationName: destName, + Namespace: testNs, + } + + err := applyFromTemplate("test/e2e/templates/manager/clusterwatchrule-crd.tmpl", clusterWatchRuleData, "") + Expect(err).NotTo(HaveOccurred(), "Failed to apply ClusterWatchRule for CRDs") + + By("verifying ClusterWatchRule is ready") + verifyResourceStatus("clusterwatchrule", clusterWatchRuleName, "", "True", "Ready", "") + verifyResourceStatus("gittarget", destName, testNs, "True", "Ready", "") + + By("installing the IceCreamOrder CRD to trigger Git commit") + _, err = kubectlRun("apply", "-f", "test/e2e/templates/icecreamorder-crd.yaml") + Expect(err).NotTo(HaveOccurred(), "Failed to install CRD") + + By("waiting for CRD to be established") + verifyCRDEstablished := func(g Gomega) { + output, err := kubectlRun( + "get", + "crd", + crdName, + "-o", + "jsonpath={.status.conditions[?(@.type=='Established')].status}", + ) + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(output).To(Equal("True")) + } + Eventually(verifyCRDEstablished).Should(Succeed()) + + By("verifying CRD YAML file exists in Git repository (NO namespace in path - cluster resource)") + verifyGitCommit := func(g Gomega) { + pullCmd := exec.Command("git", "pull") + pullCmd.Dir = crdLifecycleRepo.CheckoutDir + pullOutput, pullErr := pullCmd.CombinedOutput() + if pullErr != nil { + g.Expect(pullErr).NotTo(HaveOccurred(), + fmt.Sprintf("Should successfully pull latest changes. Output: %s", string(pullOutput))) + } + + // CRDs are cluster-scoped, so path should NOT include namespace + expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, + "e2e/crd-install-test", + "apiextensions.k8s.io/v1/customresourcedefinitions/icecreamorders.shop.example.com.yaml") + fileInfo, statErr := os.Stat(expectedFile) + g.Expect(statErr).NotTo(HaveOccurred(), fmt.Sprintf("CRD file should exist at %s", expectedFile)) + g.Expect(fileInfo.Size()).To(BeNumerically(">", 0), "CRD file should not be empty") + + // Verify file content + content, readErr := os.ReadFile(expectedFile) + g.Expect(readErr).NotTo(HaveOccurred()) + g.Expect(string(content)).To(ContainSubstring("kind: CustomResourceDefinition"), + "File should contain CRD kind") + g.Expect(string(content)).To(ContainSubstring("name: icecreamorders.shop.example.com"), + "File should contain CRD name") + } + Eventually(verifyGitCommit). + WithTimeout(60 * time.Second). + WithPolling(2 * time.Second). + Should(Succeed()) + + By("cleaning up test resources") + cleanupClusterWatchRule(clusterWatchRuleName) + cleanupGitTarget(destName, testNs) + // Keep CRD installed for subsequent tests + + By("✅ CRD installation via ClusterWatchRule E2E test passed") + }) + + It("should create Git commit when IceCreamOrder is added via WatchRule", func() { + gitProviderName := "gitprovider-normal" + watchRuleName := "watchrule-icecream-orders" + + By("installing the IceCreamOrder CRD first (needed for custom resource tests)") + _, err := kubectlRun("apply", "-f", "test/e2e/templates/icecreamorder-crd.yaml") + Expect(err).NotTo(HaveOccurred(), "Failed to install sample CRD") + + By("waiting for CRD to be established") + verifyCRDEstablished := func(g Gomega) { + output, err := kubectlRun( + "get", + "crd", + "icecreamorders.shop.example.com", + "-o", + "jsonpath={.status.conditions[?(@.type=='Established')].status}", + ) + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(output).To(Equal("True")) + } + Eventually(verifyCRDEstablished, 30*time.Second, time.Second).Should(Succeed()) + + crdInstanceName := "alices-order" + uniqueRepoName := crdLifecycleRepo.RepoName + + By("creating WatchRule that monitors IceCreamOrder resources") + destName := watchRuleName + "-dest" + createGitTarget(destName, testNs, gitProviderName, "e2e/icecream-test", "main") + + data := struct { + Name string + Namespace string + DestinationName string + }{ + Name: watchRuleName, + Namespace: testNs, + DestinationName: destName, + } + + err2 := applyFromTemplate("test/e2e/templates/watchrule-crd.tmpl", data, testNs) + Expect(err2).NotTo(HaveOccurred(), "Failed to apply WatchRule for CRDs") + + By("verifying WatchRule is ready") + verifyResourceStatus("watchrule", watchRuleName, testNs, "True", "Ready", "") + verifyResourceStatus("gittarget", destName, testNs, "True", "Ready", "") + + By("creating CR with labels and annotations to trigger Git commit") + crdInstanceData := struct { + Name string + Namespace string + Labels map[string]string + Annotations map[string]string + CustomerName string + Container string + Scoops []struct { + Flavor string + Quantity int + } + Toppings []string + }{ + Name: crdInstanceName, + Namespace: testNs, + Labels: map[string]string{ + "environment": "test", + "team": "engineering", + }, + Annotations: map[string]string{ + "description": "Alice's favorite ice cream", + "priority": "high", + "kubectl.kubernetes.io/last-applied-configuration": "should-be-filtered", + "deployment.kubernetes.io/revision": "should-also-be-filtered", + }, + CustomerName: "Alice", + Container: "Cone", + Scoops: []struct { + Flavor string + Quantity int + }{ + {Flavor: "Vanilla", Quantity: 2}, + {Flavor: "Chocolate", Quantity: 1}, + }, + Toppings: []string{"Sprinkles", "HotFudge"}, + } + + err3 := applyFromTemplate("test/e2e/templates/icecreamorder-instance.tmpl", crdInstanceData, testNs) + Expect(err3).NotTo(HaveOccurred(), "Failed to apply CRD instance") + + By("waiting for controller reconciliation of CRD instance event") + verifyReconciliationLogs := func(g Gomega) { + output, err := kubectlRunInNamespace( + namespace, + "logs", + "-l", + "control-plane=gitops-reverser", + "--tail=500", + "--prefix=true", + ) + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(output).To(ContainSubstring("git commit"), + "Should see git commit operation in logs") + } + Eventually(verifyReconciliationLogs, 45*time.Second, 2*time.Second).Should(Succeed()) + + By("verifying CRD instance YAML file exists in Gitea repository") + verifyGitCommit := func(g Gomega) { + pullCmd := exec.Command("git", "pull") + pullCmd.Dir = crdLifecycleRepo.CheckoutDir + pullOutput, pullErr := pullCmd.CombinedOutput() + if pullErr != nil { + g.Expect(pullErr).NotTo(HaveOccurred(), + fmt.Sprintf("Should successfully pull latest changes. Output: %s", string(pullOutput))) + } + + expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, + "e2e/icecream-test", + fmt.Sprintf("shop.example.com/v1/icecreamorders/%s/%s.yaml", testNs, crdInstanceName)) + fileInfo, statErr := os.Stat(expectedFile) + g.Expect(statErr). + NotTo(HaveOccurred(), fmt.Sprintf("CRD instance file should exist at %s", expectedFile)) + g.Expect(fileInfo.Size()).To(BeNumerically(">", 0), "CRD instance file should not be empty") + + content, readErr := os.ReadFile(expectedFile) + g.Expect(readErr).NotTo(HaveOccurred()) + contentStr := string(content) + g.Expect(contentStr).To(ContainSubstring("kind: IceCreamOrder"), + "CRD instance file should contain IceCreamOrder kind") + g.Expect(contentStr).To(ContainSubstring("customerName: Alice"), + "CRD instance file should contain customer name") + g.Expect(contentStr).To(ContainSubstring("container: Cone"), + "CRD instance file should contain container type") + g.Expect(contentStr).To(ContainSubstring("flavor: Vanilla"), + "CRD instance file should contain ice cream flavors") + + // Verify labels are present + g.Expect(contentStr).To(ContainSubstring("environment: test"), + "CRD instance file should contain environment label") + g.Expect(contentStr).To(ContainSubstring("team: engineering"), + "CRD instance file should contain team label") + + // Verify user annotations are present + g.Expect(contentStr).To(ContainSubstring("description: Alice's favorite ice cream"), + "CRD instance file should contain description annotation") + g.Expect(contentStr).To(ContainSubstring("priority: high"), + "CRD instance file should contain priority annotation") + + // Verify filtered annotations are NOT present + g.Expect(contentStr).NotTo(ContainSubstring("kubectl.kubernetes.io/last-applied-configuration"), + "CRD instance file should NOT contain kubectl annotation") + g.Expect(contentStr).NotTo(ContainSubstring("deployment.kubernetes.io/revision"), + "CRD instance file should NOT contain deployment annotation") + + // Verify status field is NOT present in Git + g.Expect(contentStr).NotTo(ContainSubstring("status:"), + "CRD instance file should NOT contain status field") + } + Eventually(verifyGitCommit). + WithTimeout(60 * time.Second). + WithPolling(2 * time.Second). + Should(Succeed()) + + By("verifying the IceCreamOrder still exists before status update") + verifyCRDInstanceExists := func(g Gomega) { + _, err := kubectlRunInNamespace(testNs, "get", "icecreamorder", crdInstanceName) + g.Expect(err).NotTo(HaveOccurred(), "IceCreamOrder should exist before status patch") + } + Eventually(verifyCRDInstanceExists, 15*time.Second, time.Second).Should(Succeed()) + + By("applying status update to the IceCreamOrder CR") + statusPatch := `{"status":{"phase":"Preparing","cost":12.5,"message":"Queued for pickup"}}` + statusOutput, statusErr := kubectlRunInNamespace( + testNs, + "patch", + "icecreamorder", + crdInstanceName, + "--type=merge", + "--subresource=status", + "-p", + statusPatch, + ) + Expect(statusErr).NotTo(HaveOccurred(), "Status subresource patch should succeed") + By(fmt.Sprintf("status patched successfully: %s", statusOutput)) + + By("getting current git commit hash") + gitRevCmd := exec.Command("git", "rev-parse", "HEAD") + gitRevCmd.Dir = crdLifecycleRepo.CheckoutDir + beforeStatusCommit, _ := gitRevCmd.Output() + + By("waiting to ensure no new commit is created from status update") + time.Sleep(10 * time.Second) + + By("verifying no new commit was created and status is not in Git") + verifyStatusNotCommitted := func(g Gomega) { + pullCmd := exec.Command("git", "pull") + pullCmd.Dir = crdLifecycleRepo.CheckoutDir + _, _ = pullCmd.CombinedOutput() + + // Check that commit hash hasn't changed + gitRevCmd := exec.Command("git", "rev-parse", "HEAD") + gitRevCmd.Dir = crdLifecycleRepo.CheckoutDir + afterStatusCommit, err := gitRevCmd.Output() + g.Expect(err).NotTo(HaveOccurred()) + + By(fmt.Sprintf("Commit before status: %s", string(beforeStatusCommit))) + By(fmt.Sprintf("Commit after status: %s", string(afterStatusCommit))) + + // Read the file again to ensure status is still not present + expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, + "e2e/icecream-test", + fmt.Sprintf("shop.example.com/v1/icecreamorders/%s/%s.yaml", testNs, crdInstanceName)) + content, readErr := os.ReadFile(expectedFile) + g.Expect(readErr).NotTo(HaveOccurred()) + g.Expect(string(content)).NotTo(ContainSubstring("status:"), + "CRD instance file should still NOT contain status field after status update") + g.Expect(string(content)).NotTo(ContainSubstring("phase:"), + "CRD instance file should NOT contain status phase content") + g.Expect(string(content)).NotTo(ContainSubstring("Queued for pickup"), + "CRD instance file should NOT contain status content") + } + Eventually(verifyStatusNotCommitted).Should(Succeed()) + + By("✅ Status update verified - no Git commit created and status not in file") + + By("cleaning up IceCreamOrder instance") + _, _ = kubectlRunInNamespace(testNs, "delete", "icecreamorder", crdInstanceName) + + By("Note: GitTarget, WatchRule, GitProvider, and CRD kept for subsequent tests") + + By("✅ IceCreamOrder to Git commit E2E test passed") + fmt.Printf("✅ IceCreamOrder '%s' successfully triggered Git commit in repo '%s'\n", + crdInstanceName, uniqueRepoName) + }) + + It("should update Git file when IceCreamOrder is modified via WatchRule", func() { + crdInstanceName := "bobs-order" + uniqueRepoName := crdLifecycleRepo.RepoName + + By("creating initial IceCreamOrder instance") + crdInstanceData := struct { + Name string + Namespace string + Labels map[string]string + Annotations map[string]string + CustomerName string + Container string + Scoops []struct { + Flavor string + Quantity int + } + Toppings []string + }{ + Name: crdInstanceName, + Namespace: testNs, + Labels: nil, + Annotations: nil, + CustomerName: "Bob", + Container: "Cup", + Scoops: []struct { + Flavor string + Quantity int + }{ + {Flavor: "Strawberry", Quantity: 1}, + }, + Toppings: []string{"WhippedCream"}, + } + + err := applyFromTemplate("test/e2e/templates/icecreamorder-instance.tmpl", crdInstanceData, testNs) + Expect(err).NotTo(HaveOccurred(), "Failed to apply initial CRD instance") + + By("waiting for initial CRD instance file to appear in Git") + verifyInitialFile := func(g Gomega) { + pullCmd := exec.Command("git", "pull") + pullCmd.Dir = crdLifecycleRepo.CheckoutDir + _, _ = pullCmd.CombinedOutput() + + expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, + "e2e/icecream-test", + fmt.Sprintf("shop.example.com/v1/icecreamorders/%s/%s.yaml", testNs, crdInstanceName)) + content, readErr := os.ReadFile(expectedFile) + g.Expect(readErr).NotTo(HaveOccurred()) + g.Expect(string(content)).To(ContainSubstring("customerName: Bob")) + g.Expect(string(content)).To(ContainSubstring("flavor: Strawberry")) + } + Eventually(verifyInitialFile).Should(Succeed()) + + By("updating CRD instance with new values") + updatedCRDData := struct { + Name string + Namespace string + Labels map[string]string + Annotations map[string]string + CustomerName string + Container string + Scoops []struct { + Flavor string + Quantity int + } + Toppings []string + }{ + Name: crdInstanceName, + Namespace: testNs, + Labels: nil, + Annotations: nil, + CustomerName: "Bob", + Container: "WaffleBowl", + Scoops: []struct { + Flavor string + Quantity int + }{ + {Flavor: "RockyRoad", Quantity: 3}, + {Flavor: "MintChip", Quantity: 2}, + }, + Toppings: []string{"HotFudge", "Caramel", "Sprinkles"}, + } + + err = applyFromTemplate("test/e2e/templates/icecreamorder-instance.tmpl", updatedCRDData, testNs) + Expect(err).NotTo(HaveOccurred(), "Failed to update CRD instance") + + By("verifying updated CRD instance content in Git") + verifyUpdatedFile := func(g Gomega) { + pullCmd := exec.Command("git", "pull") + pullCmd.Dir = crdLifecycleRepo.CheckoutDir + _, _ = pullCmd.CombinedOutput() + + expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, + "e2e/icecream-test", + fmt.Sprintf("shop.example.com/v1/icecreamorders/%s/%s.yaml", testNs, crdInstanceName)) + content, readErr := os.ReadFile(expectedFile) + g.Expect(readErr).NotTo(HaveOccurred()) + g.Expect(string(content)).To(ContainSubstring("container: WaffleBowl"), + "Updated file should contain new container type") + g.Expect(string(content)).To(ContainSubstring("flavor: RockyRoad"), + "Updated file should contain new flavor") + g.Expect(string(content)).To(ContainSubstring("quantity: 3"), + "Updated file should contain new quantity") + } + Eventually(verifyUpdatedFile).Should(Succeed()) + + By("cleaning up IceCreamOrder instance") + _, _ = kubectlRunInNamespace(testNs, "delete", "icecreamorder", crdInstanceName) + + By("Note: GitTarget, WatchRule, GitProvider, and CRD kept for subsequent tests") + + By("✅ IceCreamOrder update E2E test passed") + fmt.Printf("✅ IceCreamOrder '%s' update successfully reflected in Git repo '%s'\n", + crdInstanceName, uniqueRepoName) + }) + + It("should delete Git file when IceCreamOrder is deleted via WatchRule", func() { + crdInstanceName := "charlies-order" + uniqueRepoName := crdLifecycleRepo.RepoName + + By("creating IceCreamOrder instance") + crdInstanceData := struct { + Name string + Namespace string + Labels map[string]string + Annotations map[string]string + CustomerName string + Container string + Scoops []struct { + Flavor string + Quantity int + } + Toppings []string + }{ + Name: crdInstanceName, + Namespace: testNs, + Labels: nil, + Annotations: nil, + CustomerName: "Charlie", + Container: "Cone", + Scoops: []struct { + Flavor string + Quantity int + }{ + {Flavor: "Chocolate", Quantity: 2}, + }, + Toppings: []string{"Sprinkles"}, + } + + err := applyFromTemplate("test/e2e/templates/icecreamorder-instance.tmpl", crdInstanceData, testNs) + Expect(err).NotTo(HaveOccurred(), "Failed to apply CR") + + By("waiting for CR file to appear in Git repository") + verifyFileCreated := func(g Gomega) { + pullCmd := exec.Command("git", "pull") + pullCmd.Dir = crdLifecycleRepo.CheckoutDir + _, _ = pullCmd.CombinedOutput() + + expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, + "e2e/icecream-test", + fmt.Sprintf("shop.example.com/v1/icecreamorders/%s/%s.yaml", testNs, crdInstanceName)) + fileInfo, statErr := os.Stat(expectedFile) + g.Expect(statErr). + NotTo(HaveOccurred(), fmt.Sprintf("CRD instance file should exist at %s", expectedFile)) + g.Expect(fileInfo.Size()).To(BeNumerically(">", 0), "CRD instance file should not be empty") + } + Eventually(verifyFileCreated).Should(Succeed()) + + By("deleting the CR to trigger DELETE operation") + _, err = kubectlRunInNamespace(testNs, "delete", "icecreamorder", crdInstanceName) + Expect(err).NotTo(HaveOccurred(), "CRD instance deletion should succeed") + + By("verifying CRD instance file is deleted from Git repository") + verifyFileDeleted := func(g Gomega) { + pullCmd := exec.Command("git", "pull") + pullCmd.Dir = crdLifecycleRepo.CheckoutDir + _, _ = pullCmd.CombinedOutput() + + expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, + "e2e/icecream-test", + fmt.Sprintf("shop.example.com/v1/icecreamorders/%s/%s.yaml", testNs, crdInstanceName)) + _, statErr := os.Stat(expectedFile) + g.Expect(statErr). + To(HaveOccurred(), fmt.Sprintf("CRD instance file should NOT exist at %s", expectedFile)) + g.Expect(os.IsNotExist(statErr)).To(BeTrue(), "Error should be 'file does not exist'") + + By("verifying git log shows DELETE commit") + gitLogCmd := exec.Command("git", "log", "--oneline", "-n", "5") + gitLogCmd.Dir = crdLifecycleRepo.CheckoutDir + logOutput, logErr := gitLogCmd.CombinedOutput() + g.Expect(logErr).NotTo(HaveOccurred(), "Should be able to read git log") + g.Expect(string(logOutput)).To(ContainSubstring("DELETE"), + "Git log should contain DELETE operation") + } + Eventually(verifyFileDeleted).Should(Succeed()) + + By("✅ IceCreamOrder deletion E2E test passed") + fmt.Printf("✅ IceCreamOrder '%s' deletion successfully removed file from Git repo '%s'\n", + crdInstanceName, uniqueRepoName) + }) + + It("should delete Git file when IceCreamOrder CRD is deleted via ClusterWatchRule", func() { + gitProviderName := "gitprovider-normal" + clusterWatchRuleName := "clusterwatchrule-crd-delete" + crdName := "icecreamorders.shop.example.com" + + By("creating ClusterWatchRule with Cluster scope for CRDs") + destName := clusterWatchRuleName + "-dest" + createGitTarget(destName, testNs, gitProviderName, "e2e/crd-delete-test", "main") + verifyResourceStatus("gittarget", destName, testNs, "True", "Ready", "") + + clusterWatchRuleData := struct { + Name string + DestinationName string + Namespace string + }{ + Name: clusterWatchRuleName, + DestinationName: destName, + Namespace: testNs, + } + + err := applyFromTemplate("test/e2e/templates/manager/clusterwatchrule-crd.tmpl", clusterWatchRuleData, "") + Expect(err).NotTo(HaveOccurred(), "Failed to apply ClusterWatchRule for CRDs") + + By("verifying ClusterWatchRule is ready") + + verifyResourceStatus("clusterwatchrule", clusterWatchRuleName, "", "True", "Ready", "") + + By("verifying CRD file exists in Git before deletion") + verifyFileExists := func(g Gomega) { + pullCmd := exec.Command("git", "pull") + pullCmd.Dir = crdLifecycleRepo.CheckoutDir + _, _ = pullCmd.CombinedOutput() + + expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, + "e2e/crd-delete-test", + "apiextensions.k8s.io/v1/customresourcedefinitions/icecreamorders.shop.example.com.yaml") + _, statErr := os.Stat(expectedFile) + g.Expect(statErr).NotTo(HaveOccurred(), "CRD file should exist before deletion") + } + Eventually(verifyFileExists).Should(Succeed()) + + By("deleting the CRD to trigger DELETE operation") + _, deleteErr := kubectlRun("delete", "crd", crdName) + Expect(deleteErr).NotTo(HaveOccurred(), "CRD deletion should succeed") + By("verifying CRD file is deleted from Git repository") + verifyFileDeleted := func(g Gomega) { + pullCmd := exec.Command("git", "pull") + pullCmd.Dir = crdLifecycleRepo.CheckoutDir + _, _ = pullCmd.CombinedOutput() + + expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, + "e2e/crd-delete-test", + "apiextensions.k8s.io/v1/customresourcedefinitions/icecreamorders.shop.example.com.yaml") + _, statErr := os.Stat(expectedFile) + g.Expect(statErr).To(HaveOccurred(), "CRD file should NOT exist after deletion") + g.Expect(os.IsNotExist(statErr)).To(BeTrue(), "Error should be 'file does not exist'") + + // Verify git log shows DELETE commit + gitLogCmd := exec.Command("git", "log", "--oneline", "-n", "5") + gitLogCmd.Dir = crdLifecycleRepo.CheckoutDir + logOutput, logErr := gitLogCmd.CombinedOutput() + g.Expect(logErr).NotTo(HaveOccurred(), "Should be able to read git log") + g.Expect(string(logOutput)).To(ContainSubstring("DELETE"), + "Git log should contain DELETE operation") + } + Eventually(verifyFileDeleted, "60s", "1s").Should(Succeed()) + + By("verifying the deleted CRD file does not reappear after terminating updates") + Consistently(func(g Gomega) { + pullCmd := exec.Command("git", "pull") + pullCmd.Dir = crdLifecycleRepo.CheckoutDir + _, _ = pullCmd.CombinedOutput() + + expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, + "e2e/crd-delete-test", + "apiextensions.k8s.io/v1/customresourcedefinitions/icecreamorders.shop.example.com.yaml") + _, statErr := os.Stat(expectedFile) + g.Expect(statErr).To(HaveOccurred(), "CRD file must stay deleted after CRD termination updates") + g.Expect(os.IsNotExist(statErr)).To(BeTrue(), "CRD file must not reappear in Git") + }, "15s", "1s").Should(Succeed()) + + By("cleaning up test resources") + cleanupClusterWatchRule(clusterWatchRuleName) + cleanupGitTarget(destName, testNs) + + By("✅ CRD deletion via ClusterWatchRule E2E test passed") + }) +}) diff --git a/test/e2e/e2e_test.go b/test/e2e/e2e_test.go index 248efcd79..53ed522d5 100644 --- a/test/e2e/e2e_test.go +++ b/test/e2e/e2e_test.go @@ -21,16 +21,11 @@ package e2e import ( "bytes" "context" - "encoding/base64" "encoding/json" "fmt" "os" - "os/exec" - "path" - "path/filepath" "sort" "strings" - "time" "filippo.io/age" . "github.com/onsi/ginkgo/v2" @@ -42,1698 +37,6 @@ import ( const defaultE2EAgeKeyPath = "/tmp/e2e-age-key.txt" -// managerRepo holds the file-local repo fixtures for the Manager describe block. -var managerRepo *RepoArtifacts - -var _ = Describe("Manager", Label("manager"), Ordered, func() { - var controllerPodName string // Name of first controller pod for logging - var testNs string - - // Before running the tests, set up per-run test fixtures. - BeforeAll(func() { - By("setting up Prometheus client for metrics testing") - setupPrometheusClient() - verifyPrometheusAvailable() - - By("creating test namespace") - testNs = testNamespaceFor("manager") - _, _ = kubectlRun("create", "namespace", testNs) // idempotent; ignore AlreadyExists - - By("setting up Gitea repo and credentials for manager tests") - managerRepo = SetupRepo( - resolveE2EContext(), - testNs, - fmt.Sprintf("e2e-manager-%d", GinkgoRandomSeed()), - ) - - By("applying git secrets to test namespace") - _, err := kubectlRunInNamespace(testNs, "apply", "-f", managerRepo.SecretsYAML) - Expect(err).NotTo(HaveOccurred(), "failed to apply git secrets to test namespace") - applySOPSAgeKeyToNamespace(testNs) - }) - - // After all tests have been executed, clean up the test namespace - AfterAll(func() { - cleanupNamespace(testNs) - - By("test infrastructure still running for debugging") - fmt.Printf("\n") - fmt.Printf("═══════════════════════════════════════════════════════════\n") - fmt.Printf("📊 E2E Infrastructure kept running for debugging purposes:\n") - fmt.Printf("═══════════════════════════════════════════════════════════\n") - fmt.Printf(" Prometheus: http://localhost:19090\n") - fmt.Printf(" Gitea: http://localhost:13000\n") - fmt.Printf(" Flux UI: http://localhost:19080\n") - fmt.Printf("\n") - fmt.Printf("═══════════════════════════════════════════════════════════\n") - fmt.Printf("\n") - }) - - // Optimize timeouts for faster test execution - SetDefaultEventuallyTimeout(30 * time.Second) - SetDefaultEventuallyPollingInterval(time.Second) - - Context("Manager", func() { - - It("should run successfully", Label("smoke"), func() { - By("validating that the gitops-reverser pods are running as expected") - verifyControllerUp := func(g Gomega) { - // Get the names of the gitops-reverser pods - podOutput, err := kubectlRunInNamespace( - namespace, - "get", - "pods", - "-l", - "control-plane=gitops-reverser", - "-o", "go-template={{ range .items }}"+ - "{{ if not .metadata.deletionTimestamp }}"+ - "{{ .metadata.name }}"+ - "{{ \"\\n\" }}{{ end }}{{ end }}", - ) - g.Expect(err).NotTo(HaveOccurred(), "Failed to retrieve gitops-reverser pod information") - podNames := utils.GetNonEmptyLines(podOutput) - g.Expect(podNames).To(HaveLen(1), "expected exactly 1 controller pod running") - controllerPodName = podNames[0] // Use first pod for logging - g.Expect(controllerPodName).To(ContainSubstring("gitops-reverser")) - - // Validate all pods' status - for _, podName := range podNames { - output, err := kubectlRunInNamespace( - namespace, - "get", - "pods", - podName, - "-o", - "jsonpath={.status.phase}", - ) - g.Expect(err).NotTo(HaveOccurred()) - g.Expect(output).To(Equal("Running"), fmt.Sprintf("Incorrect status for pod %s", podName)) - } - } - Eventually(verifyControllerUp).Should(Succeed()) - }) - - It("should expose the controller service", Label("smoke"), func() { - By("verifying controller service exists") - _, err := kubectlRunInNamespace(namespace, "get", "svc", controllerServiceName) - Expect(err).NotTo(HaveOccurred(), "Controller service should exist") - - By("verifying controller service routes to the controller pod") - Eventually(func(g Gomega) { - output, endpointsErr := kubectlRunInNamespace( - namespace, - "get", - "endpoints", - controllerServiceName, - "-o", - "jsonpath={.subsets[*].addresses[*].targetRef.name}", - ) - g.Expect(endpointsErr).NotTo(HaveOccurred(), "Failed to get controller service endpoints") - - lines := utils.GetNonEmptyLines(output) - podSet := map[string]struct{}{} - for _, line := range lines { - if !strings.HasPrefix(line, "Warning:") && - !strings.Contains(line, "deprecated") && - strings.Contains(line, "gitops-reverser") { - podSet[line] = struct{}{} - } - } - var podNames []string - for podName := range podSet { - podNames = append(podNames, podName) - } - - g.Expect(podNames).To(HaveLen(1), "controller service should route to exactly 1 pod") - g.Expect(podNames[0]).To(Equal(controllerPodName), "controller service should route to controller pod") - }, 30*time.Second).Should(Succeed()) - }) - - It("should expose the audit service separately", func() { - By("verifying audit service exists") - _, err := kubectlRunInNamespace(namespace, "get", "svc", auditServiceName) - Expect(err).NotTo(HaveOccurred(), "Audit service should exist") - - By("verifying audit service routes to the controller pod") - Eventually(func(g Gomega) { - output, endpointsErr := kubectlRunInNamespace( - namespace, - "get", - "endpoints", - auditServiceName, - "-o", - "jsonpath={.subsets[*].addresses[*].targetRef.name}", - ) - g.Expect(endpointsErr).NotTo(HaveOccurred(), "Failed to get audit service endpoints") - - lines := utils.GetNonEmptyLines(output) - var podNames []string - for _, line := range lines { - if strings.HasPrefix(line, "Warning:") || strings.Contains(line, "deprecated") { - continue - } - podNames = append(podNames, line) - } - - g.Expect(podNames).To(HaveLen(1), "audit service should route to exactly 1 pod") - g.Expect(podNames[0]).To(Equal(controllerPodName), "audit service should route to controller pod") - }, 30*time.Second).Should(Succeed()) - }) - - It("should ensure the metrics endpoint is serving metrics", Label("smoke"), func() { - By("validating that the controller service is available for metrics") - _, err := kubectlRunInNamespace(namespace, "get", "service", controllerServiceName) - Expect(err).NotTo(HaveOccurred(), "Controller service should exist") - - By("waiting for the metrics endpoint to be ready") - verifyMetricsEndpointReady := func(g Gomega) { - output, err := kubectlRunInNamespace(namespace, "get", "endpoints", controllerServiceName) - g.Expect(err).NotTo(HaveOccurred()) - g.Expect(output).To(ContainSubstring("8443"), "Metrics endpoint is not ready") - } - Eventually(verifyMetricsEndpointReady).Should(Succeed()) - - By("verifying that the controller manager is serving the metrics server") - verifyMetricsServerStarted := func(g Gomega) { - output, err := kubectlRunInNamespace(namespace, "logs", controllerPodName) - g.Expect(err).NotTo(HaveOccurred()) - jsonMetricsLogLine := "\"logger\":\"controller-runtime.metrics\"," + - "\"msg\":\"Serving metrics server\"" - g.Expect(output).To( - SatisfyAny( - ContainSubstring("controller-runtime.metrics\tServing metrics server"), - ContainSubstring(jsonMetricsLogLine), - ), - "Metrics server not yet started", - ) - } - Eventually(verifyMetricsServerStarted).Should(Succeed()) - - By("waiting for Prometheus to scrape controller metrics") - waitForMetric("sum(up{job='gitops-reverser'})", - func(v float64) bool { return v == 1 }, - "metrics endpoint should be up") - - By("verifying basic process metrics are exposed") - waitForMetric("sum(process_cpu_seconds_total{job='gitops-reverser'})", - func(v float64) bool { return v > 0 }, - "process metrics should exist") - - By("verifying metrics from the controller pod") - podCount, err := queryPrometheus("sum(up{job='gitops-reverser'})") - Expect(err).NotTo(HaveOccurred()) - Expect(podCount).To(BeEquivalentTo(1), "Should scrape from 1 controller pod") - - fmt.Printf("✅ Metrics collection verified from %.0f pods\n", podCount) - fmt.Printf("📊 Inspect metrics: %s\n", getPrometheusURL()) - }) - - It("should receive audit webhook events from kube-apiserver", Label("smoke"), func() { - By("recording baseline audit event count") - baselineAuditEvents, err := queryPrometheus("sum(gitopsreverser_audit_events_received_total) or vector(0)") - Expect(err).NotTo(HaveOccurred()) - fmt.Printf("📊 Baseline audit events: %.0f\n", baselineAuditEvents) - baselineOfficialAuditEvents, err := queryPrometheus( - "sum(gitopsreverser_audit_events_received_total{source='official'}) or vector(0)", - ) - Expect(err).NotTo(HaveOccurred()) - fmt.Printf("📊 Baseline official audit events: %.0f\n", baselineOfficialAuditEvents) - - By("creating a ConfigMap to trigger audit events") - _, err = kubectlRunInNamespace( - namespace, - "create", - "configmap", - "audit-test-cm", - "--from-literal=test=audit", - ) - Expect(err).NotTo(HaveOccurred(), "ConfigMap creation should succeed") - _, err = kubectlRunInNamespace( - namespace, - "patch", - "configmap", - "audit-test-cm", - "--type=merge", - "--patch", - `{"data":{"test":"audit-updated"}}`, - ) - Expect(err).NotTo(HaveOccurred(), "ConfigMap update should succeed") - - By("waiting for audit event metric to increment") - waitForMetricWithTimeout("sum(gitopsreverser_audit_events_received_total) or vector(0)", - func(v float64) bool { return v > baselineAuditEvents }, - "audit events should increment", 2*time.Minute) - waitForMetricWithTimeout( - "sum(gitopsreverser_audit_events_received_total{source='official'}) or vector(0)", - func(v float64) bool { return v > baselineOfficialAuditEvents }, - "audit events should increment for source=official", 2*time.Minute, - ) - - By("verifying audit events were received") - currentAuditEvents, err := queryPrometheus("sum(gitopsreverser_audit_events_received_total) or vector(0)") - Expect(err).NotTo(HaveOccurred()) - Expect(currentAuditEvents).To(BeNumerically(">", baselineAuditEvents), - "Should have received audit events from kube-apiserver") - - By("verifying the EventList ingress metric appears for source=official") - waitForMetricWithTimeout( - "sum(gitopsreverser_audit_eventlists_total{source='official'}) or vector(0)", - func(v float64) bool { return v > 0 }, - "EventList ingress requests should be counted for source=official", 2*time.Minute, - ) - - By("verifying no removed cluster/gvr/action label remains on any audit series") - for _, removed := range []string{"cluster", "gvr", "action"} { - stale, queryErr := queryPrometheus(fmt.Sprintf( - "sum({__name__=~'gitopsreverser_audit_.+', %s=~'.+'}) or vector(0)", removed)) - Expect(queryErr).NotTo(HaveOccurred()) - Expect(stale).To(BeNumerically("==", 0), - fmt.Sprintf("no audit series should still carry the %q label", removed)) - } - - newEvents := currentAuditEvents - baselineAuditEvents - fmt.Printf("✅ Received %.0f new audit events from kube-apiserver\n", newEvents) - fmt.Printf("📊 Total audit events: %.0f\n", currentAuditEvents) - - By("cleaning up audit test resources") - _, _ = kubectlRunInNamespace( - namespace, - "delete", - "configmap", - "audit-test-cm", - "--ignore-not-found=true", - ) - }) - - It("should validate GitProvider with real Gitea repository", Label("smoke"), func() { - gitProviderName := "gitprovider-e2e-test" - - By("showing initial controller logs") - showControllerLogs("before creating GitProvider") - - createGitProviderWithURLInNamespace( - gitProviderName, - testNs, - managerRepo.GitSecretHTTP, - managerRepo.RepoURLHTTP, - ) - - By("showing controller logs after GitProvider creation") - showControllerLogs("after creating GitProvider") - - verifyResourceStatus( - "gitprovider", gitProviderName, testNs, - "True", "Ready", "Repository connectivity validated", - ) - - By("showing final controller logs") - showControllerLogs("after status verification") - - _, _ = kubectlRunInNamespace(testNs, "delete", "gitprovider", gitProviderName, "--ignore-not-found=true") - }) - - It("should handle GitProvider with invalid credentials", func() { - gitProviderName := "gitprovider-invalid-test" - createGitProviderWithURLInNamespace( - gitProviderName, - testNs, - managerRepo.GitSecretInvalid, - managerRepo.RepoURLHTTP, - ) - verifyResourceStatus("gitprovider", gitProviderName, testNs, "False", "ConnectionFailed", "") - _, _ = kubectlRunInNamespace(testNs, "delete", "gitprovider", gitProviderName, "--ignore-not-found=true") - }) - - It("should handle GitTarget with nonexistent branch pattern", func() { - gitProviderName := "gitprovider-branch-test" - - // GitProvider should be Ready=True (validates connectivity, not branch existence) - createGitProviderWithURLInNamespace( - gitProviderName, - testNs, - managerRepo.GitSecretHTTP, - managerRepo.RepoURLHTTP, - ) - verifyResourceStatus( - "gitprovider", gitProviderName, testNs, "True", "Ready", "Repository connectivity validated", - ) - - // GitTarget with branch not matching any pattern should fail - destName := "dest-invalid-branch" - createGitTarget(destName, testNs, gitProviderName, "test/invalid", "different-branch") - - By("verifying GitTarget fails branch validation") - verifyResourceStatus("gittarget", destName, testNs, "False", "ValidationFailed", "BranchNotAllowed") - - cleanupGitTarget(destName, testNs) - _, _ = kubectlRunInNamespace(testNs, "delete", "gitprovider", gitProviderName, "--ignore-not-found=true") - }) - - It("should validate GitProvider with SSH authentication", func() { - gitProviderName := "gitprovider-ssh-test" - - By("🔐 Starting SSH authentication test") - showControllerLogs("before SSH test") - - By("📋 Checking SSH secret exists") - secretOutput, err := kubectlRunInNamespace(testNs, "get", "secret", managerRepo.GitSecretSSH, "-o", "yaml") - if err != nil { - fmt.Printf("❌ SSH secret not found: %v\n", err) - } else { - previewLen := minInt(300, len(secretOutput)) - fmt.Printf( - "✅ SSH secret exists - showing first %d chars:\n%s...\n", - previewLen, - secretOutput[:previewLen], - ) - } - - createGitProviderWithURLInNamespace( - gitProviderName, - testNs, - managerRepo.GitSecretSSH, - managerRepo.RepoURLSSH, - ) - - By("🔍 Controller logs after SSH GitProvider creation") - showControllerLogs("after SSH GitProvider creation") - - verifyResourceStatus( - "gitprovider", gitProviderName, testNs, "True", "Ready", "Repository connectivity validated", - ) - - By("✅ Final SSH test logs") - showControllerLogs("SSH test completion") - - _, _ = kubectlRunInNamespace(testNs, "delete", "gitprovider", gitProviderName, "--ignore-not-found=true") - }) - - It("should handle a normal and healthy GitProvider", Label("smoke"), func() { - gitProviderName := "gitprovider-normal" - createGitProviderWithURLInNamespace( - gitProviderName, - testNs, - managerRepo.GitSecretHTTP, - managerRepo.RepoURLHTTP, - ) - verifyResourceStatus( - "gitprovider", gitProviderName, testNs, "True", "Ready", "Repository connectivity validated", - ) - }) - - It("should reconcile a WatchRule CR", func() { - gitProviderName := "gitprovider-normal" - watchRuleName := "watchrule-test" - - By("creating a WatchRule that references the working GitProvider") - destName := watchRuleName + "-dest" - createGitTarget(destName, testNs, gitProviderName, getBaseFolder(), "main") - - data := struct { - Name string - Namespace string - DestinationName string - }{ - Name: watchRuleName, - Namespace: testNs, - DestinationName: destName, - } - - err := applyFromTemplate("test/e2e/templates/watchrule.tmpl", data, testNs) - Expect(err).NotTo(HaveOccurred(), "Failed to apply WatchRule") - - By("verifying the WatchRule is reconciled") - verifyResourceStatus("watchrule", watchRuleName, testNs, "True", "Ready", "") - - By("cleaning up test resources") - cleanupWatchRule(watchRuleName, testNs) - cleanupGitTarget(destName, testNs) - }) - - It("should commit encrypted Secret manifests when WatchRule includes secrets", Label("smoke"), func() { - gitProviderName := "gitprovider-normal" - watchRuleName := "watchrule-secret-encryption-test" - secretName := "test-secret-encryption" - - By("creating WatchRule that includes secrets") - destName := watchRuleName + "-dest" - createGitTarget(destName, testNs, gitProviderName, "e2e/secret-encryption-test", "main") - - data := struct { - Name string - Namespace string - DestinationName string - }{ - Name: watchRuleName, - Namespace: testNs, - DestinationName: destName, - } - - err := applyFromTemplate("test/e2e/templates/watchrule-secret.tmpl", data, testNs) - Expect(err).NotTo(HaveOccurred(), "Failed to apply WatchRule") - verifyResourceStatus("gittarget", destName, testNs, "True", "Ready", "") - verifyResourceStatus("watchrule", watchRuleName, testNs, "True", "Ready", "") - verifyResourceStatus("gittarget", destName, testNs, "True", "Ready", "") - - By("creating Secret in watched namespace") - _, _ = kubectlRunInNamespace(testNs, "delete", "secret", secretName, "--ignore-not-found=true") - - _, err = kubectlRunInNamespace( - testNs, - "create", - "secret", - "generic", - secretName, - "--from-literal=password=do-not-commit", - ) - Expect(err).NotTo(HaveOccurred(), "Secret creation should succeed") - - By("patching Secret once to avoid informer start race and force an update event") - _, err = kubectlRunInNamespace( - testNs, - "patch", - "secret", - secretName, - "--type=merge", - "--patch", - `{"stringData":{"password":"never-commit-this"}}`, - ) - Expect(err).NotTo(HaveOccurred(), "Secret patch should succeed") - - By("verifying Secret file is committed and does not contain plaintext data") - verifyEncryptedSecretCommitted := func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = managerRepo.CheckoutDir - pullOutput, pullErr := pullCmd.CombinedOutput() - if pullErr != nil { - g.Expect(pullErr).NotTo(HaveOccurred(), - fmt.Sprintf("Should successfully pull latest changes. Output: %s", string(pullOutput))) - } - - expectedFile := filepath.Join(managerRepo.CheckoutDir, - "e2e/secret-encryption-test", - fmt.Sprintf("v1/secrets/%s/%s.sops.yaml", testNs, secretName)) - content, readErr := os.ReadFile(expectedFile) - g.Expect(readErr).NotTo(HaveOccurred(), fmt.Sprintf("Secret file must exist at %s", expectedFile)) - g.Expect(string(content)).To(ContainSubstring("sops:")) - g.Expect(string(content)).NotTo(ContainSubstring("do-not-commit")) - - bootstrapSOPSFile := filepath.Join(managerRepo.CheckoutDir, "e2e/secret-encryption-test", ".sops.yaml") - bootstrapContent, bootstrapErr := os.ReadFile(bootstrapSOPSFile) - g.Expect(bootstrapErr).NotTo( - HaveOccurred(), - fmt.Sprintf(".sops.yaml must exist at %s", bootstrapSOPSFile), - ) - ageKey, ageKeyErr := readSOPSAgeKeyFromFile(getE2EAgeKeyPath()) - g.Expect(ageKeyErr).NotTo(HaveOccurred(), "Should read age private key") - recipient, recipientErr := deriveAgeRecipient(ageKey) - g.Expect(recipientErr).NotTo(HaveOccurred(), "Should derive age recipient") - g.Expect(string(bootstrapContent)).To(ContainSubstring(recipient)) - - decryptedOutput, decryptErr := decryptWithControllerSOPS(content, ageKey) - g.Expect(decryptErr).NotTo(HaveOccurred(), "Should decrypt committed secret via controller sops binary") - g.Expect(decryptedOutput).To(ContainSubstring("bmV2ZXItY29tbWl0LXRoaXM=")) - } - Eventually(verifyEncryptedSecretCommitted, "30s", "2s").Should(Succeed()) - - By("cleaning up test resources") - _, _ = kubectlRunInNamespace(testNs, "delete", "secret", secretName, "--ignore-not-found=true") - cleanupWatchRule(watchRuleName, testNs) - cleanupGitTarget(destName, testNs) - }) - - It("should generate missing SOPS age secret when age.recipients.generateWhenMissing is enabled", func() { - gitProviderName := "gitprovider-normal" - watchRuleName := "watchrule-secret-autogen-test" - secretName := "test-secret-autogen" - generatedSecretName := "sops-age-key-autogen" - - By("ensuring generated encryption secret does not exist before test") - _, _ = kubectlRunInNamespace( - testNs, - "delete", - "secret", - generatedSecretName, - "--ignore-not-found=true", - ) - - By("creating GitTarget with age recipient auto-generation enabled") - destName := watchRuleName + "-dest" - createGitTargetWithEncryptionOptions( - destName, - testNs, - gitProviderName, - "e2e/secret-autogen-test", - "main", - generatedSecretName, - true, - ) - - data := struct { - Name string - Namespace string - DestinationName string - }{ - Name: watchRuleName, - Namespace: testNs, - DestinationName: destName, - } - - err := applyFromTemplate("test/e2e/templates/watchrule-secret.tmpl", data, testNs) - Expect(err).NotTo(HaveOccurred(), "Failed to apply WatchRule") - verifyResourceStatus("watchrule", watchRuleName, testNs, "True", "Ready", "") - - By("validating generated encryption secret has recipient and warning annotations") - var generatedAgeKey string - var generatedRecipient string - Eventually(func(g Gomega) { - output, getErr := kubectlRunInNamespace( - testNs, - "get", - "secret", - generatedSecretName, - "-o", - "json", - ) - g.Expect(getErr).NotTo(HaveOccurred()) - - var secretObj map[string]interface{} - unmarshalErr := json.Unmarshal([]byte(output), &secretObj) - g.Expect(unmarshalErr).NotTo(HaveOccurred()) - - annotations, _, annoErr := unstructured.NestedStringMap(secretObj, "metadata", "annotations") - g.Expect(annoErr).NotTo(HaveOccurred()) - g.Expect(annotations).To(HaveKey("configbutler.ai/age-recipient")) - g.Expect(annotations["configbutler.ai/age-recipient"]).To(HavePrefix("age1")) - g.Expect(annotations).To(HaveKeyWithValue("configbutler.ai/backup-warning", "REMOVE_AFTER_BACKUP")) - generatedRecipient = annotations["configbutler.ai/age-recipient"] - - secretData, found, keyErr := unstructured.NestedStringMap(secretObj, "data") - g.Expect(keyErr).NotTo(HaveOccurred()) - g.Expect(found).To(BeTrue()) - var sopsAgeKeyB64 string - for key, value := range secretData { - if strings.HasSuffix(key, ".agekey") { - sopsAgeKeyB64 = value - break - } - } - g.Expect(sopsAgeKeyB64).NotTo(BeEmpty()) - - keyBytes, decodeErr := base64.StdEncoding.DecodeString(sopsAgeKeyB64) - g.Expect(decodeErr).NotTo(HaveOccurred()) - generatedAgeKey = strings.TrimSpace(string(keyBytes)) - g.Expect(generatedAgeKey).To(HavePrefix("AGE-SECRET-KEY-")) - }, "30s", "2s").Should(Succeed()) - - By("waiting for auto-generated target bootstrap file to be present") - Eventually(func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = managerRepo.CheckoutDir - pullOutput, pullErr := pullCmd.CombinedOutput() - if pullErr != nil { - g.Expect(pullErr).NotTo(HaveOccurred(), - fmt.Sprintf("Should successfully pull latest changes. Output: %s", string(pullOutput))) - } - - bootstrapSOPSFile := filepath.Join(managerRepo.CheckoutDir, "e2e/secret-autogen-test", ".sops.yaml") - bootstrapContent, bootstrapErr := os.ReadFile(bootstrapSOPSFile) - g.Expect(bootstrapErr).NotTo(HaveOccurred(), - fmt.Sprintf(".sops.yaml must exist at %s", bootstrapSOPSFile)) - g.Expect(string(bootstrapContent)).To(ContainSubstring(generatedRecipient)) - }, "30s", "2s").Should(Succeed()) - - By("creating Secret in watched namespace") - _, _ = kubectlRunInNamespace(testNs, "delete", "secret", secretName, "--ignore-not-found=true") - _, err = kubectlRunInNamespace( - testNs, - "create", - "secret", - "generic", - secretName, - "--from-literal=password=do-not-commit", - ) - Expect(err).NotTo(HaveOccurred(), "Secret creation should succeed") - - By("patching Secret once to avoid informer start race and force an update event") - _, err = kubectlRunInNamespace( - testNs, - "patch", - "secret", - secretName, - "--type=merge", - "--patch", - `{"stringData":{"password":"autogen-never-commit-this"}}`, - ) - Expect(err).NotTo(HaveOccurred(), "Secret patch should succeed") - - By("verifying committed secret is encrypted and decryptable with generated key") - verifyEncryptedSecretCommitted := func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = managerRepo.CheckoutDir - pullOutput, pullErr := pullCmd.CombinedOutput() - if pullErr != nil { - g.Expect(pullErr).NotTo(HaveOccurred(), - fmt.Sprintf("Should successfully pull latest changes. Output: %s", string(pullOutput))) - } - - expectedFile := filepath.Join(managerRepo.CheckoutDir, - "e2e/secret-autogen-test", - fmt.Sprintf("v1/secrets/%s/%s.sops.yaml", testNs, secretName)) - content, readErr := os.ReadFile(expectedFile) - g.Expect(readErr).NotTo(HaveOccurred(), fmt.Sprintf("Secret file must exist at %s", expectedFile)) - g.Expect(string(content)).To(ContainSubstring("sops:")) - g.Expect(string(content)).NotTo(ContainSubstring("autogen-never-commit-this")) - - bootstrapSOPSFile := filepath.Join(managerRepo.CheckoutDir, "e2e/secret-autogen-test", ".sops.yaml") - bootstrapContent, bootstrapErr := os.ReadFile(bootstrapSOPSFile) - g.Expect(bootstrapErr).NotTo( - HaveOccurred(), - fmt.Sprintf(".sops.yaml must exist at %s", bootstrapSOPSFile), - ) - g.Expect(string(bootstrapContent)).To(ContainSubstring(generatedRecipient)) - - decryptedOutput, decryptErr := decryptWithControllerSOPS(content, generatedAgeKey) - g.Expect(decryptErr).NotTo(HaveOccurred(), "Should decrypt committed secret via generated age key") - g.Expect(decryptedOutput).To(ContainSubstring("YXV0b2dlbi1uZXZlci1jb21taXQtdGhpcw==")) - } - Eventually(verifyEncryptedSecretCommitted, "30s", "2s").Should(Succeed()) - - By("cleaning up test resources") - _, _ = kubectlRunInNamespace(testNs, "delete", "secret", secretName, "--ignore-not-found=true") - _, _ = kubectlRunInNamespace( - testNs, - "delete", - "secret", - generatedSecretName, - "--ignore-not-found=true", - ) - cleanupWatchRule(watchRuleName, testNs) - cleanupGitTarget(destName, testNs) - }) - - It("should create Git commit when ConfigMap is added via WatchRule", Label("smoke"), func() { - gitProviderName := "gitprovider-normal" - watchRuleName := "watchrule-configmap-test" - configMapName := "test-configmap" - uniqueRepoName := managerRepo.RepoName - - By("creating WatchRule that monitors ConfigMaps") - destName := watchRuleName + "-dest" - createGitTarget(destName, testNs, gitProviderName, "e2e/configmap-test", "main") - - data := struct { - Name string - Namespace string - DestinationName string - }{ - Name: watchRuleName, - Namespace: testNs, - DestinationName: destName, - } - - err2 := applyFromTemplate("test/e2e/templates/manager/watchrule-configmap.tmpl", data, testNs) - Expect(err2).NotTo(HaveOccurred(), "Failed to apply WatchRule") - - By("verifying WatchRule is ready") - verifyResourceStatus("watchrule", watchRuleName, testNs, "True", "Ready", "") - verifyResourceStatus("gittarget", destName, testNs, "True", "Ready", "") - - // Let any in-flight reconciles from prior specs drain before triggering - // our event. Without this, a stale WatchRule still in informer cache can - // double-commit the same ConfigMap under a different commitPath and end - // up on HEAD, masking our commit. - time.Sleep(5 * time.Second) - - By("creating test ConfigMap to trigger Git commit") - configMapData := struct { - Name string - Namespace string - }{ - Name: configMapName, - Namespace: testNs, - } - - err3 := applyFromTemplate( - "test/e2e/templates/manager/configmap.tmpl", - configMapData, - testNs, - "--as=jane@acme.com", // Important: we validate later if the user is included in the git commit! - ) - Expect(err3).NotTo(HaveOccurred(), "Failed to apply ConfigMap") - - By("waiting for controller reconciliation of ConfigMap event") - verifyReconciliationLogs := func(g Gomega) { - output, err := controllerLogs(500) - g.Expect(err).NotTo(HaveOccurred()) - - // Check for git commit operation in logs - g.Expect(output).To(ContainSubstring("git commit"), - "Should see git commit operation in controller logs") - } - Eventually(verifyReconciliationLogs).Should(Succeed()) - - By("verifying ConfigMap YAML file exists in Gitea repository") - verifyGitCommit := func(g Gomega) { - // Use the pre-checked out repository directory - By("using pre-checked out repository for verification") - - // Pull latest changes from the remote repository - By("pulling latest changes from remote repository") - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = managerRepo.CheckoutDir - // Don't use utils.Run() here because it overwrites cmd.Dir with the project directory - pullOutput, pullErr := pullCmd.CombinedOutput() - if pullErr != nil { - g.Expect(pullErr).NotTo(HaveOccurred(), - fmt.Sprintf("Should successfully pull latest changes. Output: %s", string(pullOutput))) - } - - // Check for the expected ConfigMap file (new API-aligned path) - expectedFile := filepath.Join(managerRepo.CheckoutDir, - "e2e/configmap-test", - fmt.Sprintf("v1/configmaps/%s/%s.yaml", testNs, configMapName)) - fileInfo, statErr := os.Stat(expectedFile) - g.Expect(statErr).NotTo(HaveOccurred(), fmt.Sprintf("ConfigMap file should exist at %s", expectedFile)) - g.Expect(fileInfo.Size()).To(BeNumerically(">", 0), "ConfigMap file should not be empty") - - expectedRepoPath := path.Join( - "e2e/configmap-test", - fmt.Sprintf("v1/configmaps/%s/%s.yaml", testNs, configMapName), - ) - assertLatestCommitForPathTouchesOnlyWithOptional( - g, - managerRepo.CheckoutDir, - expectedRepoPath, - []string{expectedRepoPath}, - []string{ - path.Join("e2e/configmap-test", "README.md"), - path.Join("e2e/configmap-test", ".sops.yaml"), - }, - ) - assertLatestCommitTouchesNoNamespaces( - g, - managerRepo.CheckoutDir, - "e2e/configmap-test/v1/configmaps", - []string{ - "gitops-reverser", - "flux-system", - "kube-system", - "tilt-playground", - }, - ) - - // Verify file content contains expected ConfigMap data - content, readErr := os.ReadFile(expectedFile) - g.Expect(readErr).NotTo(HaveOccurred()) - g.Expect(string(content)).To(ContainSubstring("test-key: test-value"), - "ConfigMap file should contain expected data") - - // Verify latest commit message contains operation, resource path - gitLogCmd := exec.Command("git", "log", "-1", "--pretty=%B") - gitLogCmd.Dir = managerRepo.CheckoutDir - commitMsg, commitErr := gitLogCmd.CombinedOutput() - - if commitErr != nil { - g.Expect(commitErr).NotTo(HaveOccurred(), - fmt.Sprintf("Should read latest commit message. Output: %s", string(commitMsg))) - } - - msg := string(commitMsg) - g.Expect(msg).To(ContainSubstring("[CREATE]"), - "Latest commit message should include operation [CREATE]") - g.Expect(msg).To(ContainSubstring(fmt.Sprintf("v1/configmaps/%s", configMapName)), - "Latest commit message should include resource path") - - gitLogCmd = exec.Command("git", "log", "-1", "--pretty=%an") - gitLogCmd.Dir = managerRepo.CheckoutDir - authorMsg, commitErr := gitLogCmd.CombinedOutput() - if commitErr != nil { - g.Expect(commitErr).NotTo(HaveOccurred(), - fmt.Sprintf("Should read latest commit author. Output: %s", string(authorMsg))) - } - - author := string(authorMsg) - g.Expect(author).To(ContainSubstring("jane@acme.com")) - } - Eventually(verifyGitCommit).Should(Succeed()) - - By("cleaning up test resources") - _, _ = kubectlRunInNamespace(testNs, "delete", "configmap", configMapName, "--ignore-not-found=true") - cleanupWatchRule(watchRuleName, testNs) - cleanupGitTarget(destName, testNs) - - By("✅ ConfigMap to Git commit E2E test passed - verified actual file creation and commit") - fmt.Printf("✅ ConfigMap '%s' successfully triggered Git commit with YAML file in repo '%s'\n", - configMapName, uniqueRepoName) - }) - - // Regression net for the rule-add backfill gap: a ConfigMap that - // already exists in the cluster *before* a WatchRule is applied must - // land in git once the rule becomes Ready. Today this fails because - // ReconcileForRuleChange takes the early-return at manager.go:660 - // when the new rule's GVR is already covered by other rules (and on - // fresh-target installs the snapshot output can be dropped because no - // FolderReconciler is registered yet). The unit-level coverage lives - // in internal/watch/rule_change_snapshot_test.go; this spec locks the - // observable in at the user-visible layer so a future revamp of the - // snapshot trigger logic can't silently regress it. - // - It("should backfill pre-existing ConfigMap when WatchRule is added afterwards", Label("smoke"), func() { - gitProviderName := "gitprovider-normal" - watchRuleName := "watchrule-backfill-test" - configMapName := "preexisting-configmap" - uniqueRepoName := managerRepo.RepoName - - By("creating GitTarget but no WatchRule yet") - destName := watchRuleName + "-dest" - createGitTarget(destName, testNs, gitProviderName, "e2e/backfill-rule-add", "main") - verifyResourceStatus("gittarget", destName, testNs, "True", "Ready", "") - - By("creating the ConfigMap BEFORE the rule that should select it") - configMapData := struct { - Name string - Namespace string - }{ - Name: configMapName, - Namespace: testNs, - } - err := applyFromTemplate( - "test/e2e/templates/manager/configmap.tmpl", - configMapData, - testNs, - "--as=jane@acme.com", - ) - Expect(err).NotTo(HaveOccurred(), "Failed to pre-create ConfigMap") - - // Confirm nothing committed yet — no rule exists, so the ConfigMap - // must not appear in the repo before we apply the WatchRule. - expectedRepoPath := path.Join( - "e2e/backfill-rule-add", - fmt.Sprintf("v1/configmaps/%s/%s.yaml", testNs, configMapName), - ) - expectedFile := filepath.Join(managerRepo.CheckoutDir, expectedRepoPath) - - By("applying the WatchRule (rule arrives after the resource)") - watchRuleData := struct { - Name string - Namespace string - DestinationName string - }{ - Name: watchRuleName, - Namespace: testNs, - DestinationName: destName, - } - err = applyFromTemplate("test/e2e/templates/manager/watchrule-configmap.tmpl", watchRuleData, testNs) - Expect(err).NotTo(HaveOccurred(), "Failed to apply WatchRule") - verifyResourceStatus("watchrule", watchRuleName, testNs, "True", "Ready", "") - - By("waiting for the pre-existing ConfigMap to be backfilled into git") - verifyBackfill := func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = managerRepo.CheckoutDir - pullOutput, pullErr := pullCmd.CombinedOutput() - if pullErr != nil { - g.Expect(pullErr).NotTo(HaveOccurred(), - fmt.Sprintf("Should successfully pull latest changes. Output: %s", string(pullOutput))) - } - - fileInfo, statErr := os.Stat(expectedFile) - g.Expect(statErr).NotTo(HaveOccurred(), - fmt.Sprintf("Pre-existing ConfigMap must be backfilled at %s after the rule lands", expectedFile)) - g.Expect(fileInfo.Size()).To(BeNumerically(">", 0), "Backfilled ConfigMap file should not be empty") - - content, readErr := os.ReadFile(expectedFile) - g.Expect(readErr).NotTo(HaveOccurred()) - g.Expect(string(content)).To(ContainSubstring("test-key: test-value"), - "Backfilled file should contain the ConfigMap data that existed before the rule") - } - Eventually(verifyBackfill). - WithTimeout(60 * time.Second). - WithPolling(2 * time.Second). - Should(Succeed()) - - By("cleaning up test resources") - _, _ = kubectlRunInNamespace(testNs, "delete", "configmap", configMapName, "--ignore-not-found=true") - cleanupWatchRule(watchRuleName, testNs) - cleanupGitTarget(destName, testNs) - - By("✅ rule-add backfill E2E test passed") - fmt.Printf("✅ Pre-existing ConfigMap '%s' was backfilled to repo '%s' after the WatchRule landed\n", - configMapName, uniqueRepoName) - }) - - It("should delete Git file when ConfigMap is deleted via WatchRule", Label("smoke"), func() { - gitProviderName := "gitprovider-normal" - watchRuleName := "watchrule-delete-test" - configMapName := "test-configmap-to-delete" - uniqueRepoName := managerRepo.RepoName - - By("creating WatchRule that monitors ConfigMaps") - destName := watchRuleName + "-dest" - createGitTarget(destName, testNs, gitProviderName, "e2e/delete-test", "main") - data := struct { - Name string - Namespace string - DestinationName string - }{ - Name: watchRuleName, - Namespace: testNs, - DestinationName: destName, - } - - err2 := applyFromTemplate("test/e2e/templates/manager/watchrule-configmap.tmpl", data, testNs) - Expect(err2).NotTo(HaveOccurred(), "Failed to apply WatchRule") - - By("verifying WatchRule is ready") - verifyResourceStatus("watchrule", watchRuleName, testNs, "True", "Ready", "") - verifyResourceStatus("gittarget", destName, testNs, "True", "Ready", "") - - // See "should create Git commit when ConfigMap is added": prior specs' - // WatchRules can still be in the controller's event router and pile on - // extra commits at unrelated commitPaths, knocking HEAD off our commit. - time.Sleep(5 * time.Second) - - By("creating test ConfigMap to trigger Git commit") - configMapData := struct { - Name string - Namespace string - }{ - Name: configMapName, - Namespace: testNs, - } - - err3 := applyFromTemplate("test/e2e/templates/manager/configmap.tmpl", configMapData, testNs) - Expect(err3).NotTo(HaveOccurred(), "Failed to apply ConfigMap") - - By("waiting for ConfigMap file to appear in Git repository") - verifyFileCreated := func(g Gomega) { - // Pull latest changes from the remote repository - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = managerRepo.CheckoutDir - pullOutput, pullErr := pullCmd.CombinedOutput() - if pullErr != nil { - g.Expect(pullErr).NotTo(HaveOccurred(), - fmt.Sprintf("Should successfully pull latest changes. Output: %s", string(pullOutput))) - } - - // Check for the expected ConfigMap file (new API-aligned path) - expectedFile := filepath.Join(managerRepo.CheckoutDir, - "e2e/delete-test", - fmt.Sprintf("v1/configmaps/%s/%s.yaml", testNs, configMapName)) - fileInfo, statErr := os.Stat(expectedFile) - g.Expect(statErr).NotTo(HaveOccurred(), fmt.Sprintf("ConfigMap file should exist at %s", expectedFile)) - g.Expect(fileInfo.Size()).To(BeNumerically(">", 0), "ConfigMap file should not be empty") - } - Eventually(verifyFileCreated).Should(Succeed()) - - By("deleting the ConfigMap to trigger DELETE operation") - _, err := kubectlRunInNamespace(testNs, "delete", "configmap", configMapName) - Expect(err).NotTo(HaveOccurred(), "ConfigMap deletion should succeed") - - By("verifying ConfigMap file is deleted from Git repository") - verifyFileDeleted := func(g Gomega) { - // Pull latest changes from the remote repository - By("pulling latest changes after deletion") - pullLatestRepoState(g, managerRepo.CheckoutDir) - - // Check that the ConfigMap file no longer exists (new API-aligned path) - expectedRelativePath := path.Join( - "e2e/delete-test", - fmt.Sprintf("v1/configmaps/%s/%s.yaml", testNs, configMapName), - ) - expectedFile := filepath.Join(managerRepo.CheckoutDir, expectedRelativePath) - _, statErr := os.Stat(expectedFile) - g.Expect(statErr).To(HaveOccurred(), fmt.Sprintf("ConfigMap file should NOT exist at %s", expectedFile)) - g.Expect(os.IsNotExist(statErr)).To(BeTrue(), "Error should be 'file does not exist'") - - assertLatestCommitForPathTouchesOnlyWithOptional( - g, - managerRepo.CheckoutDir, - expectedRelativePath, - []string{expectedRelativePath}, - nil, - ) - assertLatestCommitTouchesNoNamespaces( - g, - managerRepo.CheckoutDir, - "e2e/delete-test/v1/configmaps", - []string{ - "gitops-reverser", - "flux-system", - "kube-system", - "tilt-playground", - }, - ) - - // Verify git log shows DELETE commit - By("verifying git log shows DELETE operation") - gitLogCmd := exec.Command("git", "log", "--oneline", "-n", "5") - gitLogCmd.Dir = managerRepo.CheckoutDir - logOutput, logErr := gitLogCmd.CombinedOutput() - g.Expect(logErr).NotTo(HaveOccurred(), "Should be able to read git log") - g.Expect(string(logOutput)).To(ContainSubstring("DELETE"), - "Git log should contain DELETE operation") - } - Eventually(verifyFileDeleted).Should(Succeed()) - - By("cleaning up test resources") - cleanupWatchRule(watchRuleName, testNs) - cleanupGitTarget(destName, testNs) - - By("✅ ConfigMap deletion E2E test passed - verified file removal from Git") - fmt.Printf("✅ ConfigMap '%s' deletion successfully triggered Git commit removing file from repo '%s'\n", - configMapName, uniqueRepoName) - }) - - It("should create Git commit when IceCreamOrder CRD is installed via ClusterWatchRule", Label("smoke"), func() { - gitProviderName := "gitprovider-normal" - clusterWatchRuleName := "clusterwatchrule-crd-install" - crdName := "icecreamorders.shop.example.com" - - By("creating ClusterWatchRule with Cluster scope for CRDs") - destName := clusterWatchRuleName + "-dest" - createGitTarget(destName, testNs, gitProviderName, "e2e/crd-install-test", "main") - - clusterWatchRuleData := struct { - Name string - DestinationName string - Namespace string - }{ - Name: clusterWatchRuleName, - DestinationName: destName, - Namespace: testNs, - } - - err := applyFromTemplate("test/e2e/templates/manager/clusterwatchrule-crd.tmpl", clusterWatchRuleData, "") - Expect(err).NotTo(HaveOccurred(), "Failed to apply ClusterWatchRule for CRDs") - - By("verifying ClusterWatchRule is ready") - verifyResourceStatus("clusterwatchrule", clusterWatchRuleName, "", "True", "Ready", "") - verifyResourceStatus("gittarget", destName, testNs, "True", "Ready", "") - - By("installing the IceCreamOrder CRD to trigger Git commit") - _, err = kubectlRun("apply", "-f", "test/e2e/templates/icecreamorder-crd.yaml") - Expect(err).NotTo(HaveOccurred(), "Failed to install CRD") - - By("waiting for CRD to be established") - verifyCRDEstablished := func(g Gomega) { - output, err := kubectlRun( - "get", - "crd", - crdName, - "-o", - "jsonpath={.status.conditions[?(@.type=='Established')].status}", - ) - g.Expect(err).NotTo(HaveOccurred()) - g.Expect(output).To(Equal("True")) - } - Eventually(verifyCRDEstablished).Should(Succeed()) - - By("verifying CRD YAML file exists in Git repository (NO namespace in path - cluster resource)") - verifyGitCommit := func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = managerRepo.CheckoutDir - pullOutput, pullErr := pullCmd.CombinedOutput() - if pullErr != nil { - g.Expect(pullErr).NotTo(HaveOccurred(), - fmt.Sprintf("Should successfully pull latest changes. Output: %s", string(pullOutput))) - } - - // CRDs are cluster-scoped, so path should NOT include namespace - expectedFile := filepath.Join(managerRepo.CheckoutDir, - "e2e/crd-install-test", - "apiextensions.k8s.io/v1/customresourcedefinitions/icecreamorders.shop.example.com.yaml") - fileInfo, statErr := os.Stat(expectedFile) - g.Expect(statErr).NotTo(HaveOccurred(), fmt.Sprintf("CRD file should exist at %s", expectedFile)) - g.Expect(fileInfo.Size()).To(BeNumerically(">", 0), "CRD file should not be empty") - - // Verify file content - content, readErr := os.ReadFile(expectedFile) - g.Expect(readErr).NotTo(HaveOccurred()) - g.Expect(string(content)).To(ContainSubstring("kind: CustomResourceDefinition"), - "File should contain CRD kind") - g.Expect(string(content)).To(ContainSubstring("name: icecreamorders.shop.example.com"), - "File should contain CRD name") - } - Eventually(verifyGitCommit). - WithTimeout(60 * time.Second). - WithPolling(2 * time.Second). - Should(Succeed()) - - By("cleaning up test resources") - cleanupClusterWatchRule(clusterWatchRuleName) - cleanupGitTarget(destName, testNs) - // Keep CRD installed for subsequent tests - - By("✅ CRD installation via ClusterWatchRule E2E test passed") - }) - - It("should create Git commit when IceCreamOrder is added via WatchRule", func() { - gitProviderName := "gitprovider-normal" - watchRuleName := "watchrule-icecream-orders" - - By("installing the IceCreamOrder CRD first (needed for custom resource tests)") - _, err := kubectlRun("apply", "-f", "test/e2e/templates/icecreamorder-crd.yaml") - Expect(err).NotTo(HaveOccurred(), "Failed to install sample CRD") - - By("waiting for CRD to be established") - verifyCRDEstablished := func(g Gomega) { - output, err := kubectlRun( - "get", - "crd", - "icecreamorders.shop.example.com", - "-o", - "jsonpath={.status.conditions[?(@.type=='Established')].status}", - ) - g.Expect(err).NotTo(HaveOccurred()) - g.Expect(output).To(Equal("True")) - } - Eventually(verifyCRDEstablished, 30*time.Second, time.Second).Should(Succeed()) - - crdInstanceName := "alices-order" - uniqueRepoName := managerRepo.RepoName - - By("creating WatchRule that monitors IceCreamOrder resources") - destName := watchRuleName + "-dest" - createGitTarget(destName, testNs, gitProviderName, "e2e/icecream-test", "main") - - data := struct { - Name string - Namespace string - DestinationName string - }{ - Name: watchRuleName, - Namespace: testNs, - DestinationName: destName, - } - - err2 := applyFromTemplate("test/e2e/templates/watchrule-crd.tmpl", data, testNs) - Expect(err2).NotTo(HaveOccurred(), "Failed to apply WatchRule for CRDs") - - By("verifying WatchRule is ready") - verifyResourceStatus("watchrule", watchRuleName, testNs, "True", "Ready", "") - verifyResourceStatus("gittarget", destName, testNs, "True", "Ready", "") - - By("creating CR with labels and annotations to trigger Git commit") - crdInstanceData := struct { - Name string - Namespace string - Labels map[string]string - Annotations map[string]string - CustomerName string - Container string - Scoops []struct { - Flavor string - Quantity int - } - Toppings []string - }{ - Name: crdInstanceName, - Namespace: testNs, - Labels: map[string]string{ - "environment": "test", - "team": "engineering", - }, - Annotations: map[string]string{ - "description": "Alice's favorite ice cream", - "priority": "high", - "kubectl.kubernetes.io/last-applied-configuration": "should-be-filtered", - "deployment.kubernetes.io/revision": "should-also-be-filtered", - }, - CustomerName: "Alice", - Container: "Cone", - Scoops: []struct { - Flavor string - Quantity int - }{ - {Flavor: "Vanilla", Quantity: 2}, - {Flavor: "Chocolate", Quantity: 1}, - }, - Toppings: []string{"Sprinkles", "HotFudge"}, - } - - err3 := applyFromTemplate("test/e2e/templates/icecreamorder-instance.tmpl", crdInstanceData, testNs) - Expect(err3).NotTo(HaveOccurred(), "Failed to apply CRD instance") - - By("waiting for controller reconciliation of CRD instance event") - verifyReconciliationLogs := func(g Gomega) { - output, err := kubectlRunInNamespace( - namespace, - "logs", - "-l", - "control-plane=gitops-reverser", - "--tail=500", - "--prefix=true", - ) - g.Expect(err).NotTo(HaveOccurred()) - g.Expect(output).To(ContainSubstring("git commit"), - "Should see git commit operation in logs") - } - Eventually(verifyReconciliationLogs, 45*time.Second, 2*time.Second).Should(Succeed()) - - By("verifying CRD instance YAML file exists in Gitea repository") - verifyGitCommit := func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = managerRepo.CheckoutDir - pullOutput, pullErr := pullCmd.CombinedOutput() - if pullErr != nil { - g.Expect(pullErr).NotTo(HaveOccurred(), - fmt.Sprintf("Should successfully pull latest changes. Output: %s", string(pullOutput))) - } - - expectedFile := filepath.Join(managerRepo.CheckoutDir, - "e2e/icecream-test", - fmt.Sprintf("shop.example.com/v1/icecreamorders/%s/%s.yaml", testNs, crdInstanceName)) - fileInfo, statErr := os.Stat(expectedFile) - g.Expect(statErr). - NotTo(HaveOccurred(), fmt.Sprintf("CRD instance file should exist at %s", expectedFile)) - g.Expect(fileInfo.Size()).To(BeNumerically(">", 0), "CRD instance file should not be empty") - - content, readErr := os.ReadFile(expectedFile) - g.Expect(readErr).NotTo(HaveOccurred()) - contentStr := string(content) - g.Expect(contentStr).To(ContainSubstring("kind: IceCreamOrder"), - "CRD instance file should contain IceCreamOrder kind") - g.Expect(contentStr).To(ContainSubstring("customerName: Alice"), - "CRD instance file should contain customer name") - g.Expect(contentStr).To(ContainSubstring("container: Cone"), - "CRD instance file should contain container type") - g.Expect(contentStr).To(ContainSubstring("flavor: Vanilla"), - "CRD instance file should contain ice cream flavors") - - // Verify labels are present - g.Expect(contentStr).To(ContainSubstring("environment: test"), - "CRD instance file should contain environment label") - g.Expect(contentStr).To(ContainSubstring("team: engineering"), - "CRD instance file should contain team label") - - // Verify user annotations are present - g.Expect(contentStr).To(ContainSubstring("description: Alice's favorite ice cream"), - "CRD instance file should contain description annotation") - g.Expect(contentStr).To(ContainSubstring("priority: high"), - "CRD instance file should contain priority annotation") - - // Verify filtered annotations are NOT present - g.Expect(contentStr).NotTo(ContainSubstring("kubectl.kubernetes.io/last-applied-configuration"), - "CRD instance file should NOT contain kubectl annotation") - g.Expect(contentStr).NotTo(ContainSubstring("deployment.kubernetes.io/revision"), - "CRD instance file should NOT contain deployment annotation") - - // Verify status field is NOT present in Git - g.Expect(contentStr).NotTo(ContainSubstring("status:"), - "CRD instance file should NOT contain status field") - } - Eventually(verifyGitCommit). - WithTimeout(60 * time.Second). - WithPolling(2 * time.Second). - Should(Succeed()) - - By("verifying the IceCreamOrder still exists before status update") - verifyCRDInstanceExists := func(g Gomega) { - _, err := kubectlRunInNamespace(testNs, "get", "icecreamorder", crdInstanceName) - g.Expect(err).NotTo(HaveOccurred(), "IceCreamOrder should exist before status patch") - } - Eventually(verifyCRDInstanceExists, 15*time.Second, time.Second).Should(Succeed()) - - By("applying status update to the IceCreamOrder CR") - statusPatch := `{"status":{"phase":"Preparing","cost":12.5,"message":"Queued for pickup"}}` - statusOutput, statusErr := kubectlRunInNamespace( - testNs, - "patch", - "icecreamorder", - crdInstanceName, - "--type=merge", - "--subresource=status", - "-p", - statusPatch, - ) - Expect(statusErr).NotTo(HaveOccurred(), "Status subresource patch should succeed") - By(fmt.Sprintf("status patched successfully: %s", statusOutput)) - - By("getting current git commit hash") - gitRevCmd := exec.Command("git", "rev-parse", "HEAD") - gitRevCmd.Dir = managerRepo.CheckoutDir - beforeStatusCommit, _ := gitRevCmd.Output() - - By("waiting to ensure no new commit is created from status update") - time.Sleep(10 * time.Second) - - By("verifying no new commit was created and status is not in Git") - verifyStatusNotCommitted := func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = managerRepo.CheckoutDir - _, _ = pullCmd.CombinedOutput() - - // Check that commit hash hasn't changed - gitRevCmd := exec.Command("git", "rev-parse", "HEAD") - gitRevCmd.Dir = managerRepo.CheckoutDir - afterStatusCommit, err := gitRevCmd.Output() - g.Expect(err).NotTo(HaveOccurred()) - - By(fmt.Sprintf("Commit before status: %s", string(beforeStatusCommit))) - By(fmt.Sprintf("Commit after status: %s", string(afterStatusCommit))) - - // Read the file again to ensure status is still not present - expectedFile := filepath.Join(managerRepo.CheckoutDir, - "e2e/icecream-test", - fmt.Sprintf("shop.example.com/v1/icecreamorders/%s/%s.yaml", testNs, crdInstanceName)) - content, readErr := os.ReadFile(expectedFile) - g.Expect(readErr).NotTo(HaveOccurred()) - g.Expect(string(content)).NotTo(ContainSubstring("status:"), - "CRD instance file should still NOT contain status field after status update") - g.Expect(string(content)).NotTo(ContainSubstring("phase:"), - "CRD instance file should NOT contain status phase content") - g.Expect(string(content)).NotTo(ContainSubstring("Queued for pickup"), - "CRD instance file should NOT contain status content") - } - Eventually(verifyStatusNotCommitted).Should(Succeed()) - - By("✅ Status update verified - no Git commit created and status not in file") - - By("cleaning up IceCreamOrder instance") - _, _ = kubectlRunInNamespace(testNs, "delete", "icecreamorder", crdInstanceName) - - By("Note: GitTarget, WatchRule, GitProvider, and CRD kept for subsequent tests") - - By("✅ IceCreamOrder to Git commit E2E test passed") - fmt.Printf("✅ IceCreamOrder '%s' successfully triggered Git commit in repo '%s'\n", - crdInstanceName, uniqueRepoName) - }) - - It("should update Git file when IceCreamOrder is modified via WatchRule", func() { - crdInstanceName := "bobs-order" - uniqueRepoName := managerRepo.RepoName - - By("creating initial IceCreamOrder instance") - crdInstanceData := struct { - Name string - Namespace string - Labels map[string]string - Annotations map[string]string - CustomerName string - Container string - Scoops []struct { - Flavor string - Quantity int - } - Toppings []string - }{ - Name: crdInstanceName, - Namespace: testNs, - Labels: nil, - Annotations: nil, - CustomerName: "Bob", - Container: "Cup", - Scoops: []struct { - Flavor string - Quantity int - }{ - {Flavor: "Strawberry", Quantity: 1}, - }, - Toppings: []string{"WhippedCream"}, - } - - err := applyFromTemplate("test/e2e/templates/icecreamorder-instance.tmpl", crdInstanceData, testNs) - Expect(err).NotTo(HaveOccurred(), "Failed to apply initial CRD instance") - - By("waiting for initial CRD instance file to appear in Git") - verifyInitialFile := func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = managerRepo.CheckoutDir - _, _ = pullCmd.CombinedOutput() - - expectedFile := filepath.Join(managerRepo.CheckoutDir, - "e2e/icecream-test", - fmt.Sprintf("shop.example.com/v1/icecreamorders/%s/%s.yaml", testNs, crdInstanceName)) - content, readErr := os.ReadFile(expectedFile) - g.Expect(readErr).NotTo(HaveOccurred()) - g.Expect(string(content)).To(ContainSubstring("customerName: Bob")) - g.Expect(string(content)).To(ContainSubstring("flavor: Strawberry")) - } - Eventually(verifyInitialFile).Should(Succeed()) - - By("updating CRD instance with new values") - updatedCRDData := struct { - Name string - Namespace string - Labels map[string]string - Annotations map[string]string - CustomerName string - Container string - Scoops []struct { - Flavor string - Quantity int - } - Toppings []string - }{ - Name: crdInstanceName, - Namespace: testNs, - Labels: nil, - Annotations: nil, - CustomerName: "Bob", - Container: "WaffleBowl", - Scoops: []struct { - Flavor string - Quantity int - }{ - {Flavor: "RockyRoad", Quantity: 3}, - {Flavor: "MintChip", Quantity: 2}, - }, - Toppings: []string{"HotFudge", "Caramel", "Sprinkles"}, - } - - err = applyFromTemplate("test/e2e/templates/icecreamorder-instance.tmpl", updatedCRDData, testNs) - Expect(err).NotTo(HaveOccurred(), "Failed to update CRD instance") - - By("verifying updated CRD instance content in Git") - verifyUpdatedFile := func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = managerRepo.CheckoutDir - _, _ = pullCmd.CombinedOutput() - - expectedFile := filepath.Join(managerRepo.CheckoutDir, - "e2e/icecream-test", - fmt.Sprintf("shop.example.com/v1/icecreamorders/%s/%s.yaml", testNs, crdInstanceName)) - content, readErr := os.ReadFile(expectedFile) - g.Expect(readErr).NotTo(HaveOccurred()) - g.Expect(string(content)).To(ContainSubstring("container: WaffleBowl"), - "Updated file should contain new container type") - g.Expect(string(content)).To(ContainSubstring("flavor: RockyRoad"), - "Updated file should contain new flavor") - g.Expect(string(content)).To(ContainSubstring("quantity: 3"), - "Updated file should contain new quantity") - } - Eventually(verifyUpdatedFile).Should(Succeed()) - - By("cleaning up IceCreamOrder instance") - _, _ = kubectlRunInNamespace(testNs, "delete", "icecreamorder", crdInstanceName) - - By("Note: GitTarget, WatchRule, GitProvider, and CRD kept for subsequent tests") - - By("✅ IceCreamOrder update E2E test passed") - fmt.Printf("✅ IceCreamOrder '%s' update successfully reflected in Git repo '%s'\n", - crdInstanceName, uniqueRepoName) - }) - - It("should delete Git file when IceCreamOrder is deleted via WatchRule", func() { - crdInstanceName := "charlies-order" - uniqueRepoName := managerRepo.RepoName - - By("creating IceCreamOrder instance") - crdInstanceData := struct { - Name string - Namespace string - Labels map[string]string - Annotations map[string]string - CustomerName string - Container string - Scoops []struct { - Flavor string - Quantity int - } - Toppings []string - }{ - Name: crdInstanceName, - Namespace: testNs, - Labels: nil, - Annotations: nil, - CustomerName: "Charlie", - Container: "Cone", - Scoops: []struct { - Flavor string - Quantity int - }{ - {Flavor: "Chocolate", Quantity: 2}, - }, - Toppings: []string{"Sprinkles"}, - } - - err := applyFromTemplate("test/e2e/templates/icecreamorder-instance.tmpl", crdInstanceData, testNs) - Expect(err).NotTo(HaveOccurred(), "Failed to apply CR") - - By("waiting for CR file to appear in Git repository") - verifyFileCreated := func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = managerRepo.CheckoutDir - _, _ = pullCmd.CombinedOutput() - - expectedFile := filepath.Join(managerRepo.CheckoutDir, - "e2e/icecream-test", - fmt.Sprintf("shop.example.com/v1/icecreamorders/%s/%s.yaml", testNs, crdInstanceName)) - fileInfo, statErr := os.Stat(expectedFile) - g.Expect(statErr). - NotTo(HaveOccurred(), fmt.Sprintf("CRD instance file should exist at %s", expectedFile)) - g.Expect(fileInfo.Size()).To(BeNumerically(">", 0), "CRD instance file should not be empty") - } - Eventually(verifyFileCreated).Should(Succeed()) - - By("deleting the CR to trigger DELETE operation") - _, err = kubectlRunInNamespace(testNs, "delete", "icecreamorder", crdInstanceName) - Expect(err).NotTo(HaveOccurred(), "CRD instance deletion should succeed") - - By("verifying CRD instance file is deleted from Git repository") - verifyFileDeleted := func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = managerRepo.CheckoutDir - _, _ = pullCmd.CombinedOutput() - - expectedFile := filepath.Join(managerRepo.CheckoutDir, - "e2e/icecream-test", - fmt.Sprintf("shop.example.com/v1/icecreamorders/%s/%s.yaml", testNs, crdInstanceName)) - _, statErr := os.Stat(expectedFile) - g.Expect(statErr). - To(HaveOccurred(), fmt.Sprintf("CRD instance file should NOT exist at %s", expectedFile)) - g.Expect(os.IsNotExist(statErr)).To(BeTrue(), "Error should be 'file does not exist'") - - By("verifying git log shows DELETE commit") - gitLogCmd := exec.Command("git", "log", "--oneline", "-n", "5") - gitLogCmd.Dir = managerRepo.CheckoutDir - logOutput, logErr := gitLogCmd.CombinedOutput() - g.Expect(logErr).NotTo(HaveOccurred(), "Should be able to read git log") - g.Expect(string(logOutput)).To(ContainSubstring("DELETE"), - "Git log should contain DELETE operation") - } - Eventually(verifyFileDeleted).Should(Succeed()) - - By("✅ IceCreamOrder deletion E2E test passed") - fmt.Printf("✅ IceCreamOrder '%s' deletion successfully removed file from Git repo '%s'\n", - crdInstanceName, uniqueRepoName) - }) - - It("should delete Git file when IceCreamOrder CRD is deleted via ClusterWatchRule", func() { - gitProviderName := "gitprovider-normal" - clusterWatchRuleName := "clusterwatchrule-crd-delete" - crdName := "icecreamorders.shop.example.com" - - By("creating ClusterWatchRule with Cluster scope for CRDs") - destName := clusterWatchRuleName + "-dest" - createGitTarget(destName, testNs, gitProviderName, "e2e/crd-delete-test", "main") - verifyResourceStatus("gittarget", destName, testNs, "True", "Ready", "") - - clusterWatchRuleData := struct { - Name string - DestinationName string - Namespace string - }{ - Name: clusterWatchRuleName, - DestinationName: destName, - Namespace: testNs, - } - - err := applyFromTemplate("test/e2e/templates/manager/clusterwatchrule-crd.tmpl", clusterWatchRuleData, "") - Expect(err).NotTo(HaveOccurred(), "Failed to apply ClusterWatchRule for CRDs") - - By("verifying ClusterWatchRule is ready") - - verifyResourceStatus("clusterwatchrule", clusterWatchRuleName, "", "True", "Ready", "") - - By("verifying CRD file exists in Git before deletion") - verifyFileExists := func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = managerRepo.CheckoutDir - _, _ = pullCmd.CombinedOutput() - - expectedFile := filepath.Join(managerRepo.CheckoutDir, - "e2e/crd-delete-test", - "apiextensions.k8s.io/v1/customresourcedefinitions/icecreamorders.shop.example.com.yaml") - _, statErr := os.Stat(expectedFile) - g.Expect(statErr).NotTo(HaveOccurred(), "CRD file should exist before deletion") - } - Eventually(verifyFileExists).Should(Succeed()) - - By("deleting the CRD to trigger DELETE operation") - _, deleteErr := kubectlRun("delete", "crd", crdName) - Expect(deleteErr).NotTo(HaveOccurred(), "CRD deletion should succeed") - By("verifying CRD file is deleted from Git repository") - verifyFileDeleted := func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = managerRepo.CheckoutDir - _, _ = pullCmd.CombinedOutput() - - expectedFile := filepath.Join(managerRepo.CheckoutDir, - "e2e/crd-delete-test", - "apiextensions.k8s.io/v1/customresourcedefinitions/icecreamorders.shop.example.com.yaml") - _, statErr := os.Stat(expectedFile) - g.Expect(statErr).To(HaveOccurred(), "CRD file should NOT exist after deletion") - g.Expect(os.IsNotExist(statErr)).To(BeTrue(), "Error should be 'file does not exist'") - - // Verify git log shows DELETE commit - gitLogCmd := exec.Command("git", "log", "--oneline", "-n", "5") - gitLogCmd.Dir = managerRepo.CheckoutDir - logOutput, logErr := gitLogCmd.CombinedOutput() - g.Expect(logErr).NotTo(HaveOccurred(), "Should be able to read git log") - g.Expect(string(logOutput)).To(ContainSubstring("DELETE"), - "Git log should contain DELETE operation") - } - Eventually(verifyFileDeleted, "60s", "1s").Should(Succeed()) - - By("verifying the deleted CRD file does not reappear after terminating updates") - Consistently(func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = managerRepo.CheckoutDir - _, _ = pullCmd.CombinedOutput() - - expectedFile := filepath.Join(managerRepo.CheckoutDir, - "e2e/crd-delete-test", - "apiextensions.k8s.io/v1/customresourcedefinitions/icecreamorders.shop.example.com.yaml") - _, statErr := os.Stat(expectedFile) - g.Expect(statErr).To(HaveOccurred(), "CRD file must stay deleted after CRD termination updates") - g.Expect(os.IsNotExist(statErr)).To(BeTrue(), "CRD file must not reappear in Git") - }, "15s", "1s").Should(Succeed()) - - By("cleaning up test resources") - cleanupClusterWatchRule(clusterWatchRuleName) - cleanupGitTarget(destName, testNs) - - By("✅ CRD deletion via ClusterWatchRule E2E test passed") - }) - - AfterAll(func() { - By("cleaning up shared test resources") - - // Clean up WatchRule from IceCreamOrder tests - cleanupWatchRule("watchrule-icecream-orders", testNs) - - // Clean up GitTarget from IceCreamOrder tests - cleanupGitTarget("watchrule-icecream-orders-dest", testNs) - - // Clean up GitProvider from IceCreamOrder tests - cleanupNamespacedResource(testNs, "gitprovider", "gitprovider-normal") - - // Clean up IceCreamOrder CRD - _, _ = kubectlRun( - "delete", - "crd", - "icecreamorders.shop.example.com", - "--ignore-not-found=true", - ) - }) - - // +kubebuilder:scaffold:e2e-webhooks-checks - }) -}) - func readSOPSAgeKeyFromFile(path string) (string, error) { content, err := os.ReadFile(path) if err != nil { diff --git a/test/e2e/gitprovider_validation_e2e_test.go b/test/e2e/gitprovider_validation_e2e_test.go new file mode 100644 index 000000000..2431c4635 --- /dev/null +++ b/test/e2e/gitprovider_validation_e2e_test.go @@ -0,0 +1,162 @@ +/* +SPDX-License-Identifier: Apache-2.0 + +Copyright 2025 ConfigButler + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "fmt" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("Manager GitProvider Validation", Label("manager"), Ordered, func() { + var ( + testNs string + validationRepo *RepoArtifacts + ) + + BeforeAll(func() { + By("creating GitProvider validation test namespace") + testNs = testNamespaceFor("manager-gitprovider") + _, _ = kubectlRun("create", "namespace", testNs) // idempotent; ignore AlreadyExists + + By("setting up Gitea repo and credentials for GitProvider validation tests") + validationRepo = SetupRepo( + resolveE2EContext(), + testNs, + fmt.Sprintf("e2e-manager-gitprovider-%d", GinkgoRandomSeed()), + ) + + By("applying git secrets to test namespace") + _, err := kubectlRunInNamespace(testNs, "apply", "-f", validationRepo.SecretsYAML) + Expect(err).NotTo(HaveOccurred(), "failed to apply git secrets to test namespace") + applySOPSAgeKeyToNamespace(testNs) + }) + + AfterAll(func() { + cleanupNamespace(testNs) + }) + + SetDefaultEventuallyTimeout(30 * time.Second) + SetDefaultEventuallyPollingInterval(time.Second) + + It("should validate GitProvider with real Gitea repository", Label("smoke"), func() { + gitProviderName := "gitprovider-e2e-test" + + By("showing initial controller logs") + showControllerLogs("before creating GitProvider") + + createGitProviderWithURLInNamespace( + gitProviderName, + testNs, + validationRepo.GitSecretHTTP, + validationRepo.RepoURLHTTP, + ) + + By("showing controller logs after GitProvider creation") + showControllerLogs("after creating GitProvider") + + verifyResourceStatus( + "gitprovider", gitProviderName, testNs, + "True", "Ready", "Repository connectivity validated", + ) + + By("showing final controller logs") + showControllerLogs("after status verification") + + _, _ = kubectlRunInNamespace(testNs, "delete", "gitprovider", gitProviderName, "--ignore-not-found=true") + }) + + It("should handle GitProvider with invalid credentials", func() { + gitProviderName := "gitprovider-invalid-test" + createGitProviderWithURLInNamespace( + gitProviderName, + testNs, + validationRepo.GitSecretInvalid, + validationRepo.RepoURLHTTP, + ) + verifyResourceStatus("gitprovider", gitProviderName, testNs, "False", "ConnectionFailed", "") + _, _ = kubectlRunInNamespace(testNs, "delete", "gitprovider", gitProviderName, "--ignore-not-found=true") + }) + + It("should handle GitTarget with nonexistent branch pattern", func() { + gitProviderName := "gitprovider-branch-test" + + // GitProvider should be Ready=True (validates connectivity, not branch existence) + createGitProviderWithURLInNamespace( + gitProviderName, + testNs, + validationRepo.GitSecretHTTP, + validationRepo.RepoURLHTTP, + ) + verifyResourceStatus( + "gitprovider", gitProviderName, testNs, "True", "Ready", "Repository connectivity validated", + ) + + // GitTarget with branch not matching any pattern should fail + destName := "dest-invalid-branch" + createGitTarget(destName, testNs, gitProviderName, "test/invalid", "different-branch") + + By("verifying GitTarget fails branch validation") + verifyResourceStatus("gittarget", destName, testNs, "False", "ValidationFailed", "BranchNotAllowed") + + cleanupGitTarget(destName, testNs) + _, _ = kubectlRunInNamespace(testNs, "delete", "gitprovider", gitProviderName, "--ignore-not-found=true") + }) + + It("should validate GitProvider with SSH authentication", func() { + gitProviderName := "gitprovider-ssh-test" + + By("🔐 Starting SSH authentication test") + showControllerLogs("before SSH test") + + By("📋 Checking SSH secret exists") + secretOutput, err := kubectlRunInNamespace(testNs, "get", "secret", validationRepo.GitSecretSSH, "-o", "yaml") + if err != nil { + fmt.Printf("❌ SSH secret not found: %v\n", err) + } else { + previewLen := minInt(300, len(secretOutput)) + fmt.Printf( + "✅ SSH secret exists - showing first %d chars:\n%s...\n", + previewLen, + secretOutput[:previewLen], + ) + } + + createGitProviderWithURLInNamespace( + gitProviderName, + testNs, + validationRepo.GitSecretSSH, + validationRepo.RepoURLSSH, + ) + + By("🔍 Controller logs after SSH GitProvider creation") + showControllerLogs("after SSH GitProvider creation") + + verifyResourceStatus( + "gitprovider", gitProviderName, testNs, "True", "Ready", "Repository connectivity validated", + ) + + By("✅ Final SSH test logs") + showControllerLogs("SSH test completion") + + _, _ = kubectlRunInNamespace(testNs, "delete", "gitprovider", gitProviderName, "--ignore-not-found=true") + }) +}) diff --git a/test/e2e/watchrule_configmap_secret_e2e_test.go b/test/e2e/watchrule_configmap_secret_e2e_test.go new file mode 100644 index 000000000..d284d0d06 --- /dev/null +++ b/test/e2e/watchrule_configmap_secret_e2e_test.go @@ -0,0 +1,757 @@ +/* +SPDX-License-Identifier: Apache-2.0 + +Copyright 2025 ConfigButler + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "encoding/base64" + "encoding/json" + "fmt" + "os" + "os/exec" + "path" + "path/filepath" + "strings" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" +) + +var _ = Describe("Manager WatchRule ConfigMap and Secret", Label("manager"), Ordered, func() { + var ( + testNs string + watchRuleRepo *RepoArtifacts + ) + + BeforeAll(func() { + By("creating WatchRule ConfigMap/Secret test namespace") + testNs = testNamespaceFor("manager-watchrule") + _, _ = kubectlRun("create", "namespace", testNs) // idempotent; ignore AlreadyExists + + By("setting up Gitea repo and credentials for WatchRule tests") + watchRuleRepo = SetupRepo( + resolveE2EContext(), + testNs, + fmt.Sprintf("e2e-manager-watchrule-%d", GinkgoRandomSeed()), + ) + + By("applying git secrets to test namespace") + _, err := kubectlRunInNamespace(testNs, "apply", "-f", watchRuleRepo.SecretsYAML) + Expect(err).NotTo(HaveOccurred(), "failed to apply git secrets to test namespace") + applySOPSAgeKeyToNamespace(testNs) + + By("creating shared gitprovider-normal for WatchRule specs") + createGitProviderWithURLInNamespace( + "gitprovider-normal", + testNs, + watchRuleRepo.GitSecretHTTP, + watchRuleRepo.RepoURLHTTP, + ) + verifyResourceStatus( + "gitprovider", "gitprovider-normal", testNs, + "True", "Ready", "Repository connectivity validated", + ) + }) + + AfterAll(func() { + cleanupNamespace(testNs) + }) + + SetDefaultEventuallyTimeout(30 * time.Second) + SetDefaultEventuallyPollingInterval(time.Second) + + It("should handle a normal and healthy GitProvider", Label("smoke"), func() { + gitProviderName := "gitprovider-normal" + createGitProviderWithURLInNamespace( + gitProviderName, + testNs, + watchRuleRepo.GitSecretHTTP, + watchRuleRepo.RepoURLHTTP, + ) + verifyResourceStatus( + "gitprovider", gitProviderName, testNs, "True", "Ready", "Repository connectivity validated", + ) + }) + + It("should reconcile a WatchRule CR", func() { + gitProviderName := "gitprovider-normal" + watchRuleName := "watchrule-test" + + By("creating a WatchRule that references the working GitProvider") + destName := watchRuleName + "-dest" + createGitTarget(destName, testNs, gitProviderName, getBaseFolder(), "main") + + data := struct { + Name string + Namespace string + DestinationName string + }{ + Name: watchRuleName, + Namespace: testNs, + DestinationName: destName, + } + + err := applyFromTemplate("test/e2e/templates/watchrule.tmpl", data, testNs) + Expect(err).NotTo(HaveOccurred(), "Failed to apply WatchRule") + + By("verifying the WatchRule is reconciled") + verifyResourceStatus("watchrule", watchRuleName, testNs, "True", "Ready", "") + + By("cleaning up test resources") + cleanupWatchRule(watchRuleName, testNs) + cleanupGitTarget(destName, testNs) + }) + + It("should commit encrypted Secret manifests when WatchRule includes secrets", Label("smoke"), func() { + gitProviderName := "gitprovider-normal" + watchRuleName := "watchrule-secret-encryption-test" + secretName := "test-secret-encryption" + + By("creating WatchRule that includes secrets") + destName := watchRuleName + "-dest" + createGitTarget(destName, testNs, gitProviderName, "e2e/secret-encryption-test", "main") + + data := struct { + Name string + Namespace string + DestinationName string + }{ + Name: watchRuleName, + Namespace: testNs, + DestinationName: destName, + } + + err := applyFromTemplate("test/e2e/templates/watchrule-secret.tmpl", data, testNs) + Expect(err).NotTo(HaveOccurred(), "Failed to apply WatchRule") + verifyResourceStatus("gittarget", destName, testNs, "True", "Ready", "") + verifyResourceStatus("watchrule", watchRuleName, testNs, "True", "Ready", "") + verifyResourceStatus("gittarget", destName, testNs, "True", "Ready", "") + + By("creating Secret in watched namespace") + _, _ = kubectlRunInNamespace(testNs, "delete", "secret", secretName, "--ignore-not-found=true") + + _, err = kubectlRunInNamespace( + testNs, + "create", + "secret", + "generic", + secretName, + "--from-literal=password=do-not-commit", + ) + Expect(err).NotTo(HaveOccurred(), "Secret creation should succeed") + + By("patching Secret once to avoid informer start race and force an update event") + _, err = kubectlRunInNamespace( + testNs, + "patch", + "secret", + secretName, + "--type=merge", + "--patch", + `{"stringData":{"password":"never-commit-this"}}`, + ) + Expect(err).NotTo(HaveOccurred(), "Secret patch should succeed") + + By("verifying Secret file is committed and does not contain plaintext data") + verifyEncryptedSecretCommitted := func(g Gomega) { + pullCmd := exec.Command("git", "pull") + pullCmd.Dir = watchRuleRepo.CheckoutDir + pullOutput, pullErr := pullCmd.CombinedOutput() + if pullErr != nil { + g.Expect(pullErr).NotTo(HaveOccurred(), + fmt.Sprintf("Should successfully pull latest changes. Output: %s", string(pullOutput))) + } + + expectedFile := filepath.Join(watchRuleRepo.CheckoutDir, + "e2e/secret-encryption-test", + fmt.Sprintf("v1/secrets/%s/%s.sops.yaml", testNs, secretName)) + content, readErr := os.ReadFile(expectedFile) + g.Expect(readErr).NotTo(HaveOccurred(), fmt.Sprintf("Secret file must exist at %s", expectedFile)) + g.Expect(string(content)).To(ContainSubstring("sops:")) + g.Expect(string(content)).NotTo(ContainSubstring("do-not-commit")) + + bootstrapSOPSFile := filepath.Join(watchRuleRepo.CheckoutDir, "e2e/secret-encryption-test", ".sops.yaml") + bootstrapContent, bootstrapErr := os.ReadFile(bootstrapSOPSFile) + g.Expect(bootstrapErr).NotTo( + HaveOccurred(), + fmt.Sprintf(".sops.yaml must exist at %s", bootstrapSOPSFile), + ) + ageKey, ageKeyErr := readSOPSAgeKeyFromFile(getE2EAgeKeyPath()) + g.Expect(ageKeyErr).NotTo(HaveOccurred(), "Should read age private key") + recipient, recipientErr := deriveAgeRecipient(ageKey) + g.Expect(recipientErr).NotTo(HaveOccurred(), "Should derive age recipient") + g.Expect(string(bootstrapContent)).To(ContainSubstring(recipient)) + + decryptedOutput, decryptErr := decryptWithControllerSOPS(content, ageKey) + g.Expect(decryptErr).NotTo(HaveOccurred(), "Should decrypt committed secret via controller sops binary") + g.Expect(decryptedOutput).To(ContainSubstring("bmV2ZXItY29tbWl0LXRoaXM=")) + } + Eventually(verifyEncryptedSecretCommitted, "30s", "2s").Should(Succeed()) + + By("cleaning up test resources") + _, _ = kubectlRunInNamespace(testNs, "delete", "secret", secretName, "--ignore-not-found=true") + cleanupWatchRule(watchRuleName, testNs) + cleanupGitTarget(destName, testNs) + }) + + It("should generate missing SOPS age secret when age.recipients.generateWhenMissing is enabled", func() { + gitProviderName := "gitprovider-normal" + watchRuleName := "watchrule-secret-autogen-test" + secretName := "test-secret-autogen" + generatedSecretName := "sops-age-key-autogen" + + By("ensuring generated encryption secret does not exist before test") + _, _ = kubectlRunInNamespace( + testNs, + "delete", + "secret", + generatedSecretName, + "--ignore-not-found=true", + ) + + By("creating GitTarget with age recipient auto-generation enabled") + destName := watchRuleName + "-dest" + createGitTargetWithEncryptionOptions( + destName, + testNs, + gitProviderName, + "e2e/secret-autogen-test", + "main", + generatedSecretName, + true, + ) + + data := struct { + Name string + Namespace string + DestinationName string + }{ + Name: watchRuleName, + Namespace: testNs, + DestinationName: destName, + } + + err := applyFromTemplate("test/e2e/templates/watchrule-secret.tmpl", data, testNs) + Expect(err).NotTo(HaveOccurred(), "Failed to apply WatchRule") + verifyResourceStatus("watchrule", watchRuleName, testNs, "True", "Ready", "") + + By("validating generated encryption secret has recipient and warning annotations") + var generatedAgeKey string + var generatedRecipient string + Eventually(func(g Gomega) { + output, getErr := kubectlRunInNamespace( + testNs, + "get", + "secret", + generatedSecretName, + "-o", + "json", + ) + g.Expect(getErr).NotTo(HaveOccurred()) + + var secretObj map[string]interface{} + unmarshalErr := json.Unmarshal([]byte(output), &secretObj) + g.Expect(unmarshalErr).NotTo(HaveOccurred()) + + annotations, _, annoErr := unstructured.NestedStringMap(secretObj, "metadata", "annotations") + g.Expect(annoErr).NotTo(HaveOccurred()) + g.Expect(annotations).To(HaveKey("configbutler.ai/age-recipient")) + g.Expect(annotations["configbutler.ai/age-recipient"]).To(HavePrefix("age1")) + g.Expect(annotations).To(HaveKeyWithValue("configbutler.ai/backup-warning", "REMOVE_AFTER_BACKUP")) + generatedRecipient = annotations["configbutler.ai/age-recipient"] + + secretData, found, keyErr := unstructured.NestedStringMap(secretObj, "data") + g.Expect(keyErr).NotTo(HaveOccurred()) + g.Expect(found).To(BeTrue()) + var sopsAgeKeyB64 string + for key, value := range secretData { + if strings.HasSuffix(key, ".agekey") { + sopsAgeKeyB64 = value + break + } + } + g.Expect(sopsAgeKeyB64).NotTo(BeEmpty()) + + keyBytes, decodeErr := base64.StdEncoding.DecodeString(sopsAgeKeyB64) + g.Expect(decodeErr).NotTo(HaveOccurred()) + generatedAgeKey = strings.TrimSpace(string(keyBytes)) + g.Expect(generatedAgeKey).To(HavePrefix("AGE-SECRET-KEY-")) + }, "30s", "2s").Should(Succeed()) + + By("waiting for auto-generated target bootstrap file to be present") + Eventually(func(g Gomega) { + pullCmd := exec.Command("git", "pull") + pullCmd.Dir = watchRuleRepo.CheckoutDir + pullOutput, pullErr := pullCmd.CombinedOutput() + if pullErr != nil { + g.Expect(pullErr).NotTo(HaveOccurred(), + fmt.Sprintf("Should successfully pull latest changes. Output: %s", string(pullOutput))) + } + + bootstrapSOPSFile := filepath.Join(watchRuleRepo.CheckoutDir, "e2e/secret-autogen-test", ".sops.yaml") + bootstrapContent, bootstrapErr := os.ReadFile(bootstrapSOPSFile) + g.Expect(bootstrapErr).NotTo(HaveOccurred(), + fmt.Sprintf(".sops.yaml must exist at %s", bootstrapSOPSFile)) + g.Expect(string(bootstrapContent)).To(ContainSubstring(generatedRecipient)) + }, "30s", "2s").Should(Succeed()) + + By("creating Secret in watched namespace") + _, _ = kubectlRunInNamespace(testNs, "delete", "secret", secretName, "--ignore-not-found=true") + _, err = kubectlRunInNamespace( + testNs, + "create", + "secret", + "generic", + secretName, + "--from-literal=password=do-not-commit", + ) + Expect(err).NotTo(HaveOccurred(), "Secret creation should succeed") + + By("patching Secret once to avoid informer start race and force an update event") + _, err = kubectlRunInNamespace( + testNs, + "patch", + "secret", + secretName, + "--type=merge", + "--patch", + `{"stringData":{"password":"autogen-never-commit-this"}}`, + ) + Expect(err).NotTo(HaveOccurred(), "Secret patch should succeed") + + By("verifying committed secret is encrypted and decryptable with generated key") + verifyEncryptedSecretCommitted := func(g Gomega) { + pullCmd := exec.Command("git", "pull") + pullCmd.Dir = watchRuleRepo.CheckoutDir + pullOutput, pullErr := pullCmd.CombinedOutput() + if pullErr != nil { + g.Expect(pullErr).NotTo(HaveOccurred(), + fmt.Sprintf("Should successfully pull latest changes. Output: %s", string(pullOutput))) + } + + expectedFile := filepath.Join(watchRuleRepo.CheckoutDir, + "e2e/secret-autogen-test", + fmt.Sprintf("v1/secrets/%s/%s.sops.yaml", testNs, secretName)) + content, readErr := os.ReadFile(expectedFile) + g.Expect(readErr).NotTo(HaveOccurred(), fmt.Sprintf("Secret file must exist at %s", expectedFile)) + g.Expect(string(content)).To(ContainSubstring("sops:")) + g.Expect(string(content)).NotTo(ContainSubstring("autogen-never-commit-this")) + + bootstrapSOPSFile := filepath.Join(watchRuleRepo.CheckoutDir, "e2e/secret-autogen-test", ".sops.yaml") + bootstrapContent, bootstrapErr := os.ReadFile(bootstrapSOPSFile) + g.Expect(bootstrapErr).NotTo( + HaveOccurred(), + fmt.Sprintf(".sops.yaml must exist at %s", bootstrapSOPSFile), + ) + g.Expect(string(bootstrapContent)).To(ContainSubstring(generatedRecipient)) + + decryptedOutput, decryptErr := decryptWithControllerSOPS(content, generatedAgeKey) + g.Expect(decryptErr).NotTo(HaveOccurred(), "Should decrypt committed secret via generated age key") + g.Expect(decryptedOutput).To(ContainSubstring("YXV0b2dlbi1uZXZlci1jb21taXQtdGhpcw==")) + } + Eventually(verifyEncryptedSecretCommitted, "30s", "2s").Should(Succeed()) + + By("cleaning up test resources") + _, _ = kubectlRunInNamespace(testNs, "delete", "secret", secretName, "--ignore-not-found=true") + _, _ = kubectlRunInNamespace( + testNs, + "delete", + "secret", + generatedSecretName, + "--ignore-not-found=true", + ) + cleanupWatchRule(watchRuleName, testNs) + cleanupGitTarget(destName, testNs) + }) + + It("should create Git commit when ConfigMap is added via WatchRule", Label("smoke"), func() { + gitProviderName := "gitprovider-normal" + watchRuleName := "watchrule-configmap-test" + configMapName := "test-configmap" + uniqueRepoName := watchRuleRepo.RepoName + + By("creating WatchRule that monitors ConfigMaps") + destName := watchRuleName + "-dest" + createGitTarget(destName, testNs, gitProviderName, "e2e/configmap-test", "main") + + data := struct { + Name string + Namespace string + DestinationName string + }{ + Name: watchRuleName, + Namespace: testNs, + DestinationName: destName, + } + + err2 := applyFromTemplate("test/e2e/templates/manager/watchrule-configmap.tmpl", data, testNs) + Expect(err2).NotTo(HaveOccurred(), "Failed to apply WatchRule") + + By("verifying WatchRule is ready") + verifyResourceStatus("watchrule", watchRuleName, testNs, "True", "Ready", "") + verifyResourceStatus("gittarget", destName, testNs, "True", "Ready", "") + + // Let any in-flight reconciles from prior specs drain before triggering + // our event. Without this, a stale WatchRule still in informer cache can + // double-commit the same ConfigMap under a different commitPath and end + // up on HEAD, masking our commit. + time.Sleep(5 * time.Second) + + By("creating test ConfigMap to trigger Git commit") + configMapData := struct { + Name string + Namespace string + }{ + Name: configMapName, + Namespace: testNs, + } + + err3 := applyFromTemplate( + "test/e2e/templates/manager/configmap.tmpl", + configMapData, + testNs, + "--as=jane@acme.com", // Important: we validate later if the user is included in the git commit! + ) + Expect(err3).NotTo(HaveOccurred(), "Failed to apply ConfigMap") + + By("waiting for controller reconciliation of ConfigMap event") + verifyReconciliationLogs := func(g Gomega) { + output, err := controllerLogs(500) + g.Expect(err).NotTo(HaveOccurred()) + + // Check for git commit operation in logs + g.Expect(output).To(ContainSubstring("git commit"), + "Should see git commit operation in controller logs") + } + Eventually(verifyReconciliationLogs).Should(Succeed()) + + By("verifying ConfigMap YAML file exists in Gitea repository") + verifyGitCommit := func(g Gomega) { + // Use the pre-checked out repository directory + By("using pre-checked out repository for verification") + + // Pull latest changes from the remote repository + By("pulling latest changes from remote repository") + pullCmd := exec.Command("git", "pull") + pullCmd.Dir = watchRuleRepo.CheckoutDir + // Don't use utils.Run() here because it overwrites cmd.Dir with the project directory + pullOutput, pullErr := pullCmd.CombinedOutput() + if pullErr != nil { + g.Expect(pullErr).NotTo(HaveOccurred(), + fmt.Sprintf("Should successfully pull latest changes. Output: %s", string(pullOutput))) + } + + // Check for the expected ConfigMap file (new API-aligned path) + expectedFile := filepath.Join(watchRuleRepo.CheckoutDir, + "e2e/configmap-test", + fmt.Sprintf("v1/configmaps/%s/%s.yaml", testNs, configMapName)) + fileInfo, statErr := os.Stat(expectedFile) + g.Expect(statErr).NotTo(HaveOccurred(), fmt.Sprintf("ConfigMap file should exist at %s", expectedFile)) + g.Expect(fileInfo.Size()).To(BeNumerically(">", 0), "ConfigMap file should not be empty") + + expectedRepoPath := path.Join( + "e2e/configmap-test", + fmt.Sprintf("v1/configmaps/%s/%s.yaml", testNs, configMapName), + ) + assertLatestCommitForPathTouchesOnlyWithOptional( + g, + watchRuleRepo.CheckoutDir, + expectedRepoPath, + []string{expectedRepoPath}, + []string{ + path.Join("e2e/configmap-test", "README.md"), + path.Join("e2e/configmap-test", ".sops.yaml"), + }, + ) + assertLatestCommitTouchesNoNamespaces( + g, + watchRuleRepo.CheckoutDir, + "e2e/configmap-test/v1/configmaps", + []string{ + "gitops-reverser", + "flux-system", + "kube-system", + "tilt-playground", + }, + ) + + // Verify file content contains expected ConfigMap data + content, readErr := os.ReadFile(expectedFile) + g.Expect(readErr).NotTo(HaveOccurred()) + g.Expect(string(content)).To(ContainSubstring("test-key: test-value"), + "ConfigMap file should contain expected data") + + // Verify latest commit message contains operation, resource path + gitLogCmd := exec.Command("git", "log", "-1", "--pretty=%B") + gitLogCmd.Dir = watchRuleRepo.CheckoutDir + commitMsg, commitErr := gitLogCmd.CombinedOutput() + + if commitErr != nil { + g.Expect(commitErr).NotTo(HaveOccurred(), + fmt.Sprintf("Should read latest commit message. Output: %s", string(commitMsg))) + } + + msg := string(commitMsg) + g.Expect(msg).To(ContainSubstring("[CREATE]"), + "Latest commit message should include operation [CREATE]") + g.Expect(msg).To(ContainSubstring(fmt.Sprintf("v1/configmaps/%s", configMapName)), + "Latest commit message should include resource path") + + gitLogCmd = exec.Command("git", "log", "-1", "--pretty=%an") + gitLogCmd.Dir = watchRuleRepo.CheckoutDir + authorMsg, commitErr := gitLogCmd.CombinedOutput() + if commitErr != nil { + g.Expect(commitErr).NotTo(HaveOccurred(), + fmt.Sprintf("Should read latest commit author. Output: %s", string(authorMsg))) + } + + author := string(authorMsg) + g.Expect(author).To(ContainSubstring("jane@acme.com")) + } + Eventually(verifyGitCommit).Should(Succeed()) + + By("cleaning up test resources") + _, _ = kubectlRunInNamespace(testNs, "delete", "configmap", configMapName, "--ignore-not-found=true") + cleanupWatchRule(watchRuleName, testNs) + cleanupGitTarget(destName, testNs) + + By("✅ ConfigMap to Git commit E2E test passed - verified actual file creation and commit") + fmt.Printf("✅ ConfigMap '%s' successfully triggered Git commit with YAML file in repo '%s'\n", + configMapName, uniqueRepoName) + }) + + // Regression net for the rule-add backfill gap: a ConfigMap that + // already exists in the cluster *before* a WatchRule is applied must + // land in git once the rule becomes Ready. Today this fails because + // ReconcileForRuleChange takes the early-return at manager.go:660 + // when the new rule's GVR is already covered by other rules (and on + // fresh-target installs the snapshot output can be dropped because no + // FolderReconciler is registered yet). The unit-level coverage lives + // in internal/watch/rule_change_snapshot_test.go; this spec locks the + // observable in at the user-visible layer so a future revamp of the + // snapshot trigger logic can't silently regress it. + // + It("should backfill pre-existing ConfigMap when WatchRule is added afterwards", Label("smoke"), func() { + gitProviderName := "gitprovider-normal" + watchRuleName := "watchrule-backfill-test" + configMapName := "preexisting-configmap" + uniqueRepoName := watchRuleRepo.RepoName + + By("creating GitTarget but no WatchRule yet") + destName := watchRuleName + "-dest" + createGitTarget(destName, testNs, gitProviderName, "e2e/backfill-rule-add", "main") + verifyResourceStatus("gittarget", destName, testNs, "True", "Ready", "") + + By("creating the ConfigMap BEFORE the rule that should select it") + configMapData := struct { + Name string + Namespace string + }{ + Name: configMapName, + Namespace: testNs, + } + err := applyFromTemplate( + "test/e2e/templates/manager/configmap.tmpl", + configMapData, + testNs, + "--as=jane@acme.com", + ) + Expect(err).NotTo(HaveOccurred(), "Failed to pre-create ConfigMap") + + // Confirm nothing committed yet — no rule exists, so the ConfigMap + // must not appear in the repo before we apply the WatchRule. + expectedRepoPath := path.Join( + "e2e/backfill-rule-add", + fmt.Sprintf("v1/configmaps/%s/%s.yaml", testNs, configMapName), + ) + expectedFile := filepath.Join(watchRuleRepo.CheckoutDir, expectedRepoPath) + + By("applying the WatchRule (rule arrives after the resource)") + watchRuleData := struct { + Name string + Namespace string + DestinationName string + }{ + Name: watchRuleName, + Namespace: testNs, + DestinationName: destName, + } + err = applyFromTemplate("test/e2e/templates/manager/watchrule-configmap.tmpl", watchRuleData, testNs) + Expect(err).NotTo(HaveOccurred(), "Failed to apply WatchRule") + verifyResourceStatus("watchrule", watchRuleName, testNs, "True", "Ready", "") + + By("waiting for the pre-existing ConfigMap to be backfilled into git") + verifyBackfill := func(g Gomega) { + pullCmd := exec.Command("git", "pull") + pullCmd.Dir = watchRuleRepo.CheckoutDir + pullOutput, pullErr := pullCmd.CombinedOutput() + if pullErr != nil { + g.Expect(pullErr).NotTo(HaveOccurred(), + fmt.Sprintf("Should successfully pull latest changes. Output: %s", string(pullOutput))) + } + + fileInfo, statErr := os.Stat(expectedFile) + g.Expect(statErr).NotTo(HaveOccurred(), + fmt.Sprintf("Pre-existing ConfigMap must be backfilled at %s after the rule lands", expectedFile)) + g.Expect(fileInfo.Size()).To(BeNumerically(">", 0), "Backfilled ConfigMap file should not be empty") + + content, readErr := os.ReadFile(expectedFile) + g.Expect(readErr).NotTo(HaveOccurred()) + g.Expect(string(content)).To(ContainSubstring("test-key: test-value"), + "Backfilled file should contain the ConfigMap data that existed before the rule") + } + Eventually(verifyBackfill). + WithTimeout(60 * time.Second). + WithPolling(2 * time.Second). + Should(Succeed()) + + By("cleaning up test resources") + _, _ = kubectlRunInNamespace(testNs, "delete", "configmap", configMapName, "--ignore-not-found=true") + cleanupWatchRule(watchRuleName, testNs) + cleanupGitTarget(destName, testNs) + + By("✅ rule-add backfill E2E test passed") + fmt.Printf("✅ Pre-existing ConfigMap '%s' was backfilled to repo '%s' after the WatchRule landed\n", + configMapName, uniqueRepoName) + }) + + It("should delete Git file when ConfigMap is deleted via WatchRule", Label("smoke"), func() { + gitProviderName := "gitprovider-normal" + watchRuleName := "watchrule-delete-test" + configMapName := "test-configmap-to-delete" + uniqueRepoName := watchRuleRepo.RepoName + + By("creating WatchRule that monitors ConfigMaps") + destName := watchRuleName + "-dest" + createGitTarget(destName, testNs, gitProviderName, "e2e/delete-test", "main") + data := struct { + Name string + Namespace string + DestinationName string + }{ + Name: watchRuleName, + Namespace: testNs, + DestinationName: destName, + } + + err2 := applyFromTemplate("test/e2e/templates/manager/watchrule-configmap.tmpl", data, testNs) + Expect(err2).NotTo(HaveOccurred(), "Failed to apply WatchRule") + + By("verifying WatchRule is ready") + verifyResourceStatus("watchrule", watchRuleName, testNs, "True", "Ready", "") + verifyResourceStatus("gittarget", destName, testNs, "True", "Ready", "") + + // See "should create Git commit when ConfigMap is added": prior specs' + // WatchRules can still be in the controller's event router and pile on + // extra commits at unrelated commitPaths, knocking HEAD off our commit. + time.Sleep(5 * time.Second) + + By("creating test ConfigMap to trigger Git commit") + configMapData := struct { + Name string + Namespace string + }{ + Name: configMapName, + Namespace: testNs, + } + + err3 := applyFromTemplate("test/e2e/templates/manager/configmap.tmpl", configMapData, testNs) + Expect(err3).NotTo(HaveOccurred(), "Failed to apply ConfigMap") + + By("waiting for ConfigMap file to appear in Git repository") + verifyFileCreated := func(g Gomega) { + // Pull latest changes from the remote repository + pullCmd := exec.Command("git", "pull") + pullCmd.Dir = watchRuleRepo.CheckoutDir + pullOutput, pullErr := pullCmd.CombinedOutput() + if pullErr != nil { + g.Expect(pullErr).NotTo(HaveOccurred(), + fmt.Sprintf("Should successfully pull latest changes. Output: %s", string(pullOutput))) + } + + // Check for the expected ConfigMap file (new API-aligned path) + expectedFile := filepath.Join(watchRuleRepo.CheckoutDir, + "e2e/delete-test", + fmt.Sprintf("v1/configmaps/%s/%s.yaml", testNs, configMapName)) + fileInfo, statErr := os.Stat(expectedFile) + g.Expect(statErr).NotTo(HaveOccurred(), fmt.Sprintf("ConfigMap file should exist at %s", expectedFile)) + g.Expect(fileInfo.Size()).To(BeNumerically(">", 0), "ConfigMap file should not be empty") + } + Eventually(verifyFileCreated).Should(Succeed()) + + By("deleting the ConfigMap to trigger DELETE operation") + _, err := kubectlRunInNamespace(testNs, "delete", "configmap", configMapName) + Expect(err).NotTo(HaveOccurred(), "ConfigMap deletion should succeed") + + By("verifying ConfigMap file is deleted from Git repository") + verifyFileDeleted := func(g Gomega) { + // Pull latest changes from the remote repository + By("pulling latest changes after deletion") + pullLatestRepoState(g, watchRuleRepo.CheckoutDir) + + // Check that the ConfigMap file no longer exists (new API-aligned path) + expectedRelativePath := path.Join( + "e2e/delete-test", + fmt.Sprintf("v1/configmaps/%s/%s.yaml", testNs, configMapName), + ) + expectedFile := filepath.Join(watchRuleRepo.CheckoutDir, expectedRelativePath) + _, statErr := os.Stat(expectedFile) + g.Expect(statErr).To(HaveOccurred(), fmt.Sprintf("ConfigMap file should NOT exist at %s", expectedFile)) + g.Expect(os.IsNotExist(statErr)).To(BeTrue(), "Error should be 'file does not exist'") + + assertLatestCommitForPathTouchesOnlyWithOptional( + g, + watchRuleRepo.CheckoutDir, + expectedRelativePath, + []string{expectedRelativePath}, + nil, + ) + assertLatestCommitTouchesNoNamespaces( + g, + watchRuleRepo.CheckoutDir, + "e2e/delete-test/v1/configmaps", + []string{ + "gitops-reverser", + "flux-system", + "kube-system", + "tilt-playground", + }, + ) + + // Verify git log shows DELETE commit + By("verifying git log shows DELETE operation") + gitLogCmd := exec.Command("git", "log", "--oneline", "-n", "5") + gitLogCmd.Dir = watchRuleRepo.CheckoutDir + logOutput, logErr := gitLogCmd.CombinedOutput() + g.Expect(logErr).NotTo(HaveOccurred(), "Should be able to read git log") + g.Expect(string(logOutput)).To(ContainSubstring("DELETE"), + "Git log should contain DELETE operation") + } + Eventually(verifyFileDeleted).Should(Succeed()) + + By("cleaning up test resources") + cleanupWatchRule(watchRuleName, testNs) + cleanupGitTarget(destName, testNs) + + By("✅ ConfigMap deletion E2E test passed - verified file removal from Git") + fmt.Printf("✅ ConfigMap '%s' deletion successfully triggered Git commit removing file from repo '%s'\n", + configMapName, uniqueRepoName) + }) +}) From ddd5f77c38ca0944ff70bdbf6c4ed9fba53ddb16 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 29 May 2026 07:29:08 +0000 Subject: [PATCH 06/40] chore: bump deps --- .devcontainer/Dockerfile | 21 ++++-- .devcontainer/devcontainer-lock.json | 16 ++--- .devcontainer/devcontainer.json | 4 +- docs/design/e2e-speedup-plan.md | 97 ++++++++++++++++++++++++++++ 4 files changed, 123 insertions(+), 15 deletions(-) diff --git a/.devcontainer/Dockerfile b/.devcontainer/Dockerfile index e5539a94c..ee3d6953e 100644 --- a/.devcontainer/Dockerfile +++ b/.devcontainer/Dockerfile @@ -39,16 +39,28 @@ RUN apt-get update \ # Tool versions - centralized for easy updates. # This devcontainer currently supports amd64 tool downloads only. +# Finding the latest versions: +# KUBECTL_VERSION -> https://github.com/kubernetes/kubernetes/releases +# KUSTOMIZE_VERSION -> https://github.com/kubernetes-sigs/kustomize/releases +# KUBEBUILDER_VERSION -> https://github.com/kubernetes-sigs/kubebuilder +# GOLANGCI_LINT_VERSION -> https://github.com/golangci/golangci-lint/releases +# K3D_VERSION -> https://github.com/k3d-io/k3d (seems a bit stsale?) +# FLUX_VERSION -> https://github.com/fluxcd/flux2 +# FLUX_OPERATOR_VERSION -> https://github.com/controlplaneio-fluxcd/flux-operator +# TASK_VERSION -> https://github.com/go-task/task/releases +# TILT_VERSION -> https://github.com/tilt-dev/tilt/releases + ENV PATH="/go/bin:/usr/local/go/bin:${PATH}" \ KUBECTL_VERSION=v1.36.1 \ KUSTOMIZE_VERSION=5.8.1 \ - KUBEBUILDER_VERSION=4.14.0 \ + KUBEBUILDER_VERSION=4.14.1 \ GOLANGCI_LINT_VERSION=v2.12.2 \ HELM_VERSION=v4.2.0 \ K3D_VERSION=v5.8.3 \ - FLUX_VERSION=2.8.6 \ - FLUX_OPERATOR_VERSION=0.49.0 \ - TASK_VERSION=v3.50.0 + FLUX_VERSION=2.8.8 \ + FLUX_OPERATOR_VERSION=0.50.0 \ + TASK_VERSION=v3.51.1 \ + TILT_VERSION=v0.37.3 # https://github.com/fluxcd/flux2/releases # https://fluxoperator.dev/ @@ -192,7 +204,6 @@ RUN printf '%s\n' \ >> /etc/bash.bashrc # Install Tilt CLI (local dev loop orchestration; dev stage only) -ENV TILT_VERSION=v0.37.1 RUN arch="$(dpkg --print-architecture)" && \ case "${arch}" in \ amd64) tilt_arch="x86_64" ;; \ diff --git a/.devcontainer/devcontainer-lock.json b/.devcontainer/devcontainer-lock.json index 9d81b9292..e2a33b7be 100644 --- a/.devcontainer/devcontainer-lock.json +++ b/.devcontainer/devcontainer-lock.json @@ -1,14 +1,14 @@ { "features": { - "ghcr.io/devcontainers/features/common-utils:2.5.8": { - "version": "2.5.8", - "resolved": "ghcr.io/devcontainers/features/common-utils@sha256:c42fdefe6d737a3a6f61cc52b23c7c9a565d08cc4d9c303669a7cf2ee5fd81fc", - "integrity": "sha256:c42fdefe6d737a3a6f61cc52b23c7c9a565d08cc4d9c303669a7cf2ee5fd81fc" + "ghcr.io/devcontainers/features/common-utils:2.5.9": { + "version": "2.5.9", + "resolved": "ghcr.io/devcontainers/features/common-utils@sha256:cb0c4d3c276f157eed17935747e364178d75fee17f55c4e129966f64633deb3a", + "integrity": "sha256:cb0c4d3c276f157eed17935747e364178d75fee17f55c4e129966f64633deb3a" }, - "ghcr.io/devcontainers/features/docker-outside-of-docker:1": { - "version": "1.9.1", - "resolved": "ghcr.io/devcontainers/features/docker-outside-of-docker@sha256:dc89605f01ff2f24252c61f7c8ba2a58ccdbc14f2ebf87a7952d9e2b89834850", - "integrity": "sha256:dc89605f01ff2f24252c61f7c8ba2a58ccdbc14f2ebf87a7952d9e2b89834850" + "ghcr.io/devcontainers/features/docker-outside-of-docker:1.10.0": { + "version": "1.10.0", + "resolved": "ghcr.io/devcontainers/features/docker-outside-of-docker@sha256:c2c2cf829505ead8e4892c88c31b6594ae94a2bbb209e16e1fac456c1a3a624e", + "integrity": "sha256:c2c2cf829505ead8e4892c88c31b6594ae94a2bbb209e16e1fac456c1a3a624e" }, "ghcr.io/devcontainers/features/git:1": { "version": "1.3.5", diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json index f70523b9b..fd1ab6935 100644 --- a/.devcontainer/devcontainer.json +++ b/.devcontainer/devcontainer.json @@ -8,8 +8,8 @@ "workspaceMount": "source=${localWorkspaceFolder},target=/workspaces/${localWorkspaceFolderBasename},type=bind", "workspaceFolder": "/workspaces/${localWorkspaceFolderBasename}", "features": { - "ghcr.io/devcontainers/features/docker-outside-of-docker:1": {}, - "ghcr.io/devcontainers/features/common-utils:2.5.8": { + "ghcr.io/devcontainers/features/docker-outside-of-docker:1.10.0": {}, + "ghcr.io/devcontainers/features/common-utils:2.5.9": { "userUid": "automatic", "userGid": "automatic", "username": "vscode" diff --git a/docs/design/e2e-speedup-plan.md b/docs/design/e2e-speedup-plan.md index 3ec497a86..dd767f740 100644 --- a/docs/design/e2e-speedup-plan.md +++ b/docs/design/e2e-speedup-plan.md @@ -123,6 +123,103 @@ execution. - Cold `task prepare-e2e` ≥ 25 % faster than the sequential baseline. - Warm `task prepare-e2e` still ≈ 1.4 s. +**DAG (as implemented).** Two regressions surfaced during validation and +shaped the final shape: + +- `portforward-ensure` had to stay *strictly sequential after* + `_prepare-e2e-ready` (not a `deps:` peer): the k3d server-node restart + inside `_webhook-tls-ready` kills any kubectl port-forwards started + concurrently with it. +- `cleanup-installs.sh` (which wipes `.stamps/cluster///`) + needed to be its own DAG node, `_install-cleanup`, so + `_sops-secret-yaml` can depend on it. Previously the cleanup ran inside + `_install-{mode}` cmds, and in parallel mode `_sops-secret-yaml`'s + status was evaluated *before* the cleanup, reported "up to date" from a + prior run's stamp, and then the cleanup deleted the file out from + under `_sops-secret-applied`. `_install-cleanup`'s status check + (`test -f /install.yaml`) lets it stay idempotent across + warm runs while still firing on a mode switch or a fresh CTX. + +```mermaid +flowchart TD + cluster([_cluster-ready]):::root + age([_age-key]):::root + imageid([_controller-image-id]):::root + + imageid --> imageready["_project-image-ready-local
+ _project-image-ready"] + + cluster --> cleanup[_install-cleanup] + cluster --> flux[_flux-installed] + cluster --> imageloaded[_image-loaded] + imageready --> imageloaded + + flux --> fluxsetup[_flux-setup-ready] + fluxsetup --> services[_services-ready] + + age --> sopsyaml[_sops-secret-yaml] + cleanup --> sopsyaml + + services --> install["install dispatcher
+ _install-{mode}"] + cleanup --> install + + install --> ctrl[_controller-deployed] + imageloaded --> ctrl + ctrl --> webhook[_webhook-tls-ready] + + services --> aggkube["_aggregated-api-
webhook-kubeconfig-ready"] + webhook --> aggkube + aggkube --> aggapi[_aggregated-api-ready] + + sopsyaml --> sopsapplied[_sops-secret-applied] + install --> sopsapplied + + ctrl --> ready[_prepare-e2e-ready] + webhook --> ready + sopsapplied --> ready + aggapi --> ready + + services --> pf[portforward-ensure] + ready ==> prepare([prepare-e2e]):::top + pf ==> prepare + + classDef root fill:#e8f4f8,stroke:#0066cc,stroke-width:2px + classDef top fill:#f0e8f8,stroke:#6600cc,stroke-width:2px +``` + +Legend: + +- Thin arrows are go-task `deps:` — parallel-safe; a task starts as soon + as all incoming arrows have completed. +- Thick arrows are sequential `cmds:` ordering inside `prepare-e2e` + itself (the only place where ordering is not expressed via `deps:`). +- Blue nodes (`_cluster-ready`, `_age-key`, `_controller-image-id`) have + no deps. Purple is the user-facing entry point. +- The `install` node collapses three steps that act as a single unit: + the templated dispatcher `install` (`task: 'install-{{.INSTALL_MODE}}'`), + the per-mode wrapper `install-{helm,config-dir,plain-manifests-file}`, + and the leaf `_install-{mode}` it depends on. Each `_install-{mode}` + carries its own build-side dep from `Taskfile-build.yml` + (`helm-sync`, `manifests`, or `dist-install`); those are elided here + to keep the diagram readable. + +The critical path on a cold run is +`_cluster-ready → _flux-installed → _flux-setup-ready → _services-ready +→ install → _controller-deployed → _webhook-tls-ready +→ _aggregated-api-webhook-kubeconfig-ready → _aggregated-api-ready +→ _prepare-e2e-ready → portforward-ensure`. Everything else folds onto +that path before its outputs are needed. + +**Measured impact:** cold `task prepare-e2e` 230 s → 213 s (7.4 %), well +short of the 25 % target. Honest accounting: the Flux ramp-up is a +strictly serial ~150 s of that budget and the docker image was cached in +this run, so the parallel branches (`_age-key`/`_sops-secret-yaml`, +`_image-loaded`, `_install-cleanup`, `manifests`) only saved the ~17 s +of work they would have spent on the critical path sequentially. With a +fresh Go build (the scenario the original plan estimate assumed), +`_project-image-ready` would absorb tens of seconds of additional cluster +ramp-up and the gain should approach the original estimate. Warm +`task prepare-e2e` stays at 1 s — no regression. + ### Phase 2 — Mechanical split of the `Manager` Describe The `Manager` Describe in From 8377d7ebc713294e0d6d0f632cc49660708db2db Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 29 May 2026 08:10:26 +0000 Subject: [PATCH 07/40] chore: Let's see if this is also making our build greend --- test/e2e/Taskfile.yml | 6 +++++- test/e2e/crd_lifecycle_e2e_test.go | 14 +++++++++++++- 2 files changed, 18 insertions(+), 2 deletions(-) diff --git a/test/e2e/Taskfile.yml b/test/e2e/Taskfile.yml index a15659e6b..6f4ea9dc8 100644 --- a/test/e2e/Taskfile.yml +++ b/test/e2e/Taskfile.yml @@ -177,13 +177,17 @@ tasks: test-e2e-full: desc: Run the full e2e suite, including slow and specialized scenarios + vars: + # 47 specs at ~22s avg + ~6m cold prepare ≈ 25m; the smoke-tuned 15m default + # times out mid-run and gives a goroutine dump instead of a useful report. + E2E_FULL_TIMEOUT: '{{.E2E_FULL_TIMEOUT | default "30m"}}' cmds: - | export CTX="{{.CTX}}" export INSTALL_MODE="{{.INSTALL_MODE}}" export NAMESPACE="{{.NAMESPACE}}" export E2E_AGE_KEY_FILE="{{.CS}}/age-key.txt" - go test -timeout "{{.E2E_GO_TEST_TIMEOUT}}" ./test/e2e/ -v -ginkgo.v \ + go test -timeout "{{.E2E_FULL_TIMEOUT}}" ./test/e2e/ -v -ginkgo.v \ -ginkgo.json-report="{{.CS}}/{{.NAMESPACE}}/ginkgo-report-full.json" test-e2e-manager: diff --git a/test/e2e/crd_lifecycle_e2e_test.go b/test/e2e/crd_lifecycle_e2e_test.go index 65d894c9a..0f4fdfb0e 100644 --- a/test/e2e/crd_lifecycle_e2e_test.go +++ b/test/e2e/crd_lifecycle_e2e_test.go @@ -199,6 +199,16 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { } Eventually(verifyCRDEstablished, 30*time.Second, time.Second).Should(Succeed()) + // Established=True only means the apiserver registered the CRD; the + // controller's reflector may still be in "Failed to watch" backoff for + // a few seconds. If we create a CR before that recovers, the create + // event is missed and the next spec sees no file in git. Block until + // the resource is actually listable end-to-end. + Eventually(func(g Gomega) { + _, err := kubectlRun("get", "icecreamorders.shop.example.com", "-A") + g.Expect(err).NotTo(HaveOccurred(), "icecreamorders must be servable before creating instances") + }, 30*time.Second, time.Second).Should(Succeed()) + crdInstanceName := "alices-order" uniqueRepoName := crdLifecycleRepo.RepoName @@ -574,7 +584,9 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { NotTo(HaveOccurred(), fmt.Sprintf("CRD instance file should exist at %s", expectedFile)) g.Expect(fileInfo.Size()).To(BeNumerically(">", 0), "CRD instance file should not be empty") } - Eventually(verifyFileCreated).Should(Succeed()) + // 60s tolerates controller-reflector re-establishment if the deployment + // happened to roll during the prior spec (default 30s was racing it). + Eventually(verifyFileCreated, 60*time.Second, 2*time.Second).Should(Succeed()) By("deleting the CR to trigger DELETE operation") _, err = kubectlRunInNamespace(testNs, "delete", "icecreamorder", crdInstanceName) From 12053690cc262917ef4f861baf4b5dbd65ad4a8c Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 29 May 2026 13:41:56 +0000 Subject: [PATCH 08/40] chore: A first start, probably red yet --- docs/design/e2e-serial-registry.md | 56 ++++ test/e2e/Taskfile.yml | 28 +- test/e2e/aggregated_apiserver_e2e_test.go | 5 +- test/e2e/audit_redis_e2e_test.go | 9 +- test/e2e/bi_directional_e2e_test.go | 29 ++- test/e2e/commit_request_e2e_test.go | 5 +- test/e2e/commit_window_batching_e2e_test.go | 240 +++++++++--------- test/e2e/crd_lifecycle_e2e_test.go | 56 ++-- test/e2e/e2e_suite_test.go | 51 +++- test/e2e/icecream.go | 61 +++++ test/e2e/image_refresh_test.go | 5 +- test/e2e/restart_snapshot_e2e_test.go | 23 +- ...morder-crd.yaml => icecreamorder-crd.tmpl} | 4 +- .../e2e/templates/icecreamorder-instance.tmpl | 2 +- .../restart/clusterwatchrule-wildcard.tmpl | 2 +- test/e2e/templates/watchrule-crd.tmpl | 2 +- 16 files changed, 387 insertions(+), 191 deletions(-) create mode 100644 docs/design/e2e-serial-registry.md create mode 100644 test/e2e/icecream.go rename test/e2e/templates/{icecreamorder-crd.yaml => icecreamorder-crd.tmpl} (98%) diff --git a/docs/design/e2e-serial-registry.md b/docs/design/e2e-serial-registry.md new file mode 100644 index 000000000..44212b4c9 --- /dev/null +++ b/docs/design/e2e-serial-registry.md @@ -0,0 +1,56 @@ +# E2E Serial Registry + +This is a **living document**. It is the authoritative list of e2e test +containers that must run `Serial` (i.e. never concurrently with any other spec) +under bounded Ginkgo parallelism (Phase 2.5 of +[e2e-speedup-plan.md](e2e-speedup-plan.md)). + +## Rule + +A test container is `Serial` only when it mutates shared, **process-level or +cluster-wide** state that cannot be isolated by name. Prefer name isolation +(e.g. per-file CRD groups — see [test/e2e/icecream.go](../../test/e2e/icecream.go)) +over `Serial`; reserve `Serial` for conflicts name isolation cannot fix. + +**When you add a spec that performs a cluster-scoped write (rolls/scales the +controller, installs an `APIService`/CRD/cluster RBAC, or otherwise affects +shared apiserver state), update this registry in the same PR** — either by +isolating the state by name or by marking the container `Serial` and adding a +row below. + +## Serial containers + +| File | Container | Shared state it touches | Why name isolation can't fix it | +|---|---|---|---| +| [test/e2e/restart_snapshot_e2e_test.go](../../test/e2e/restart_snapshot_e2e_test.go) | `Restart Snapshot Safety` | Rollout-restarts the controller deployment. | The controller is a singleton; restarting it disrupts in-flight reconciles/commits for every other spec. | +| [test/e2e/image_refresh_test.go](../../test/e2e/image_refresh_test.go) | `image refresh dependency chain` | Changes the controller image / redeploys the controller. | Same singleton controller; an image swap perturbs all concurrent specs. | +| [test/e2e/aggregated_apiserver_e2e_test.go](../../test/e2e/aggregated_apiserver_e2e_test.go) | `Aggregated API server` | Installs/removes a cluster-scoped `APIService` (aggregation layer). | Registering/removing an aggregated API briefly disrupts apiserver discovery for **every** client, including unrelated kubectl calls in other specs. | +| [test/e2e/audit_redis_e2e_test.go](../../test/e2e/audit_redis_e2e_test.go) | `Audit Redis Queue`, `Audit Redis Consumer` | The single global audit pipeline: audit webhook → shared Redis stream → consumer. | The consumer is a cluster-wide singleton processing one shared stream; these specs assert on commit *exclusivity*, which any concurrent audit traffic violates. | +| [test/e2e/commit_window_batching_e2e_test.go](../../test/e2e/commit_window_batching_e2e_test.go) | `Commit Window Batching` | Same global audit pipeline (labelled `audit-redis`). | Bursts of ConfigMap events flow through the shared stream/consumer and leak into other audit specs' commits. | +| [test/e2e/commit_request_e2e_test.go](../../test/e2e/commit_request_e2e_test.go) | `Commit Request` | Same global audit pipeline (labelled `audit-redis`). | Shares the stream/consumer; commit-window/queue semantics interleave under concurrency. | + +## Watched, but intentionally NOT Serial + +- **`crd_lifecycle_e2e_test.go`** installs a `ClusterWatchRule` that watches all + CRDs cluster-wide (`ClusterResourceRule` has no per-object-name filter, see + [api/v1alpha1/clusterwatchrule_types.go](../../api/v1alpha1/clusterwatchrule_types.go)). + While that rule is live it mirrors *other* files' CRD definitions into its own + repo. This is harmless: the rule is only active during the two CRD-lifecycle + specs (install/delete), it is torn down before the "no new commit" assertion, + and every assertion checks a *specific*, group-scoped file path. Extra mirrored + files are cosmetic. Kept parallel; revisit if the stability window shows flakes. + +All four `audit-redis`-labelled containers above were moved to `Serial` after the +first procs=2 smoke run: `Audit Redis Consumer` asserted its commit touched only +its own file but picked up `Commit Window Batching`'s files from the shared +consumer. Because Ginkgo runs `Serial` specs alone (after all parallel specs), +the audit pipeline now only ever has one client at a time. + +## Triage list (watch during stability runs) + +`controller_basics_e2e_test.go` spec #5 ("should receive audit webhook events") +reads a **global Prometheus audit counter** with a baseline/delta pattern. It +remained parallel and passed, because the delta only needs the counter to +*increase*. If a future change makes that read exact, concurrent audit traffic +from other parallel specs could break it — harden the read (filter by +namespace/resource) or move the container into the Serial table above. diff --git a/test/e2e/Taskfile.yml b/test/e2e/Taskfile.yml index 6f4ea9dc8..33675cf94 100644 --- a/test/e2e/Taskfile.yml +++ b/test/e2e/Taskfile.yml @@ -93,6 +93,10 @@ vars: FLUX_WAIT_TIMEOUT: '{{.FLUX_WAIT_TIMEOUT | default "500s"}}' FLUX_SERVICES_WAIT_TIMEOUT: '{{.FLUX_SERVICES_WAIT_TIMEOUT | default "600s"}}' E2E_GO_TEST_TIMEOUT: '{{.E2E_GO_TEST_TIMEOUT | default "15m"}}' + # Number of parallel Ginkgo processes for the e2e suite. >1 enables Ginkgo + # parallelism (Phase 2.5 of docs/design/e2e-speedup-plan.md). Files that touch + # shared cluster-scoped state are marked Serial (see docs/design/e2e-serial-registry.md). + E2E_GINKGO_PROCS: '{{.E2E_GINKGO_PROCS | default "2"}}' FLUX_SERVICES_DIR: '{{.FLUX_SERVICES_DIR | default "test/e2e/setup/flux"}}' DEMO_MANIFESTS_DIR: 'test/e2e/setup/demo-only' DEMO_TUNNEL_CREDENTIALS: '{{.DEMO_MANIFESTS_DIR}}/cloudflared-public/tunnel-credentials.yaml' @@ -172,8 +176,15 @@ tasks: export INSTALL_MODE="{{.INSTALL_MODE}}" export NAMESPACE="{{.NAMESPACE}}" export E2E_AGE_KEY_FILE="{{.CS}}/age-key.txt" - go test -timeout "{{.E2E_GO_TEST_TIMEOUT}}" ./test/e2e/ -v -ginkgo.v -ginkgo.label-filter=smoke \ - -ginkgo.json-report="{{.CS}}/{{.NAMESPACE}}/ginkgo-report-smoke.json" + # Ginkgo parallelism (--procs) is orchestrated by the ginkgo CLI, not by + # `go test`. Run it via `go run` so the CLI version matches the pinned + # github.com/onsi/ginkgo/v2 module (a globally-installed ginkgo can differ + # and refuse to run on a version mismatch). + go run github.com/onsi/ginkgo/v2/ginkgo \ + --procs={{.E2E_GINKGO_PROCS}} --timeout="{{.E2E_GO_TEST_TIMEOUT}}" -v \ + --label-filter=smoke \ + --output-dir="{{.CS}}/{{.NAMESPACE}}" --json-report=ginkgo-report-smoke.json \ + ./test/e2e/ test-e2e-full: desc: Run the full e2e suite, including slow and specialized scenarios @@ -187,8 +198,10 @@ tasks: export INSTALL_MODE="{{.INSTALL_MODE}}" export NAMESPACE="{{.NAMESPACE}}" export E2E_AGE_KEY_FILE="{{.CS}}/age-key.txt" - go test -timeout "{{.E2E_FULL_TIMEOUT}}" ./test/e2e/ -v -ginkgo.v \ - -ginkgo.json-report="{{.CS}}/{{.NAMESPACE}}/ginkgo-report-full.json" + go run github.com/onsi/ginkgo/v2/ginkgo \ + --procs={{.E2E_GINKGO_PROCS}} --timeout="{{.E2E_FULL_TIMEOUT}}" -v \ + --output-dir="{{.CS}}/{{.NAMESPACE}}" --json-report=ginkgo-report-full.json \ + ./test/e2e/ test-e2e-manager: desc: Run manager-focused e2e scenarios @@ -198,8 +211,11 @@ tasks: export INSTALL_MODE="{{.INSTALL_MODE}}" export NAMESPACE="{{.NAMESPACE}}" export E2E_AGE_KEY_FILE="{{.CS}}/age-key.txt" - go test -timeout "{{.E2E_GO_TEST_TIMEOUT}}" ./test/e2e/ -v -ginkgo.v -ginkgo.label-filter=manager \ - -ginkgo.json-report="{{.CS}}/{{.NAMESPACE}}/ginkgo-report-manager.json" + go run github.com/onsi/ginkgo/v2/ginkgo \ + --procs={{.E2E_GINKGO_PROCS}} --timeout="{{.E2E_GO_TEST_TIMEOUT}}" -v \ + --label-filter=manager \ + --output-dir="{{.CS}}/{{.NAMESPACE}}" --json-report=ginkgo-report-manager.json \ + ./test/e2e/ test-e2e-signing: desc: Run commit-signing e2e scenarios diff --git a/test/e2e/aggregated_apiserver_e2e_test.go b/test/e2e/aggregated_apiserver_e2e_test.go index e9ed2617b..7cbf1d0b9 100644 --- a/test/e2e/aggregated_apiserver_e2e_test.go +++ b/test/e2e/aggregated_apiserver_e2e_test.go @@ -33,7 +33,10 @@ import ( "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" ) -var _ = Describe("Aggregated API server", Label("aggregated-api"), Ordered, func() { +// Serial: installs/removes a cluster-scoped APIService (aggregation layer), +// which perturbs apiserver discovery for every other client while it is in +// flight. See docs/design/e2e-serial-registry.md. +var _ = Describe("Aggregated API server", Label("aggregated-api"), Serial, Ordered, func() { var ( testNs string repo *RepoArtifacts diff --git a/test/e2e/audit_redis_e2e_test.go b/test/e2e/audit_redis_e2e_test.go index 138b22649..e89cac519 100644 --- a/test/e2e/audit_redis_e2e_test.go +++ b/test/e2e/audit_redis_e2e_test.go @@ -66,7 +66,10 @@ func ensureAuditRedisRepo() *RepoArtifacts { return auditRedisRepo } -var _ = Describe("Audit Redis Queue", Label("audit-redis", "smoke"), Ordered, func() { +// Serial: shares the single global audit pipeline (audit webhook → Redis stream +// → consumer) with every other audit-redis spec; concurrent audit traffic +// pollutes its exclusivity assertions. See docs/design/e2e-serial-registry.md. +var _ = Describe("Audit Redis Queue", Label("audit-redis", "smoke"), Serial, Ordered, func() { var testNs string BeforeAll(func() { @@ -160,7 +163,9 @@ var _ = Describe("Audit Redis Queue", Label("audit-redis", "smoke"), Ordered, fu // auditConsumerDescribe holds the consumer-path e2e test. // Label: audit-redis (runs together with the producer test via task test-e2e-audit-redis). -var _ = Describe("Audit Redis Consumer", Label("audit-redis", "smoke"), Ordered, func() { +// Serial: shares the single global audit pipeline with every other audit-redis +// spec. See docs/design/e2e-serial-registry.md. +var _ = Describe("Audit Redis Consumer", Label("audit-redis", "smoke"), Serial, Ordered, func() { var ( testNs string valkeyClient *redis.Client diff --git a/test/e2e/bi_directional_e2e_test.go b/test/e2e/bi_directional_e2e_test.go index cf153ea99..232c72384 100644 --- a/test/e2e/bi_directional_e2e_test.go +++ b/test/e2e/bi_directional_e2e_test.go @@ -127,7 +127,7 @@ var _ = Describe("Bi Directional", Label("bi-directional"), Ordered, func() { // Namespace deletion is enough for ordinary namespaced leftovers like test Secrets and // IceCreamOrder instances. Keep explicit cleanup only for Flux resources and cluster-scoped // resources that live outside the test namespace. - cleanupClusterResource("crd", "icecreamorders.shop.example.com") + cleanupClusterResource("crd", iceCreamCRDName(crdGroupBiDirectional)) cleanupNamespace(testNs) }) @@ -167,10 +167,12 @@ var _ = Describe("Bi Directional", Label("bi-directional"), Ordered, func() { Name string Namespace string DestinationName string + Group string }{ Name: run.watchRuleName, Namespace: testNs, DestinationName: run.gitTargetName, + Group: crdGroupBiDirectional, }, testNs) Expect(err).NotTo(HaveOccurred(), "failed to apply bi-directional WatchRule") verifyResourceStatus("gitprovider", run.gitProviderName, testNs, "True", "Ready", "") @@ -217,7 +219,7 @@ var _ = Describe("Bi Directional", Label("bi-directional"), Ordered, func() { _, err = kubectlRunInNamespace( testNs, "patch", - "icecreamorder", + iceCreamCRDName(crdGroupBiDirectional), run.firstOrderName, "--type=merge", "--patch", @@ -377,7 +379,7 @@ func (r biDirectionalRun) waitForCRDEstablished() { output, err := kubectlRun( "get", "crd", - "icecreamorders.shop.example.com", + iceCreamCRDName(crdGroupBiDirectional), "-o", "jsonpath={.status.conditions[?(@.type=='Established')].status}", ) @@ -394,9 +396,7 @@ func (r biDirectionalRun) repoPath(parts ...string) string { func (r biDirectionalRun) liveOrderPath(name string) string { return r.repoPath( r.livePath, - "shop.example.com", - "v1", - "icecreamorders", + iceCreamInstanceDir(crdGroupBiDirectional), r.testNs, name+".yaml", ) @@ -413,8 +413,11 @@ func (r biDirectionalRun) liveSecretPath(name string) string { } func (r biDirectionalRun) writeCRDToRepo() { - content, err := os.ReadFile("test/e2e/templates/icecreamorder-crd.yaml") - Expect(err).NotTo(HaveOccurred(), "failed to read IceCreamOrder CRD fixture") + content, err := renderTemplate( + "test/e2e/templates/icecreamorder-crd.tmpl", + struct{ Group string }{Group: crdGroupBiDirectional}, + ) + Expect(err).NotTo(HaveOccurred(), "failed to render IceCreamOrder CRD fixture") Expect( os.MkdirAll( r.repoPath(r.crdPath, "apiextensions.k8s.io", "v1", "customresourcedefinitions"), @@ -428,9 +431,9 @@ func (r biDirectionalRun) writeCRDToRepo() { "apiextensions.k8s.io", "v1", "customresourcedefinitions", - "icecreamorders.shop.example.com.yaml", + iceCreamCRDMirrorFile(crdGroupBiDirectional), ), - content, + []byte(content), 0o644, ), ).To(Succeed()) @@ -446,6 +449,7 @@ func (r biDirectionalRun) writeLiveOrder(order iceCreamOrderFile) { Container string Scoops []iceCreamScoop Toppings []string + Group string }{ Name: order.Name, Namespace: order.Namespace, @@ -455,6 +459,7 @@ func (r biDirectionalRun) writeLiveOrder(order iceCreamOrderFile) { Container: order.Container, Scoops: order.Scoops, Toppings: order.Toppings, + Group: crdGroupBiDirectional, }) Expect(err).NotTo(HaveOccurred(), "failed to render IceCreamOrder manifest") Expect(os.MkdirAll(filepath.Dir(r.liveOrderPath(order.Name)), 0o755)).To(Succeed()) @@ -831,7 +836,7 @@ func (r biDirectionalRun) waitForOrderSpec(name, container, flavor, topping stri output, err := kubectlRunInNamespace( r.testNs, "get", - "icecreamorder", + iceCreamCRDName(crdGroupBiDirectional), name, "-o", "json", @@ -1039,7 +1044,7 @@ func (r biDirectionalRun) waitForSecretValue(name, key, expectedValue string) { func (r biDirectionalRun) waitForOrderDeleted(name string) { Eventually(func(g Gomega) { - _, err := kubectlRunInNamespace(r.testNs, "get", "icecreamorder", name, "-o", "json") + _, err := kubectlRunInNamespace(r.testNs, "get", iceCreamCRDName(crdGroupBiDirectional), name, "-o", "json") g.Expect(err).To(HaveOccurred()) g.Expect(strings.ToLower(err.Error())).To(ContainSubstring("not found")) }, biEventuallyTimeout, biPollInterval).Should(Succeed()) diff --git a/test/e2e/commit_request_e2e_test.go b/test/e2e/commit_request_e2e_test.go index dec1a3364..d5db3cdc6 100644 --- a/test/e2e/commit_request_e2e_test.go +++ b/test/e2e/commit_request_e2e_test.go @@ -36,7 +36,10 @@ import ( // a deliberately long commitWindow so that, without the CommitRequest, the // edit would not be committed for minutes — observing the commit promptly is // what proves the commit-request path works. -var _ = Describe("Commit Request", Label("commit-request", "audit-redis", "smoke"), Ordered, func() { +// Serial: shares the single global audit pipeline (audit webhook → Redis stream +// → consumer) with every other audit-redis spec. See +// docs/design/e2e-serial-registry.md. +var _ = Describe("Commit Request", Label("commit-request", "audit-redis", "smoke"), Serial, Ordered, func() { var ( testNs string gitProvName string diff --git a/test/e2e/commit_window_batching_e2e_test.go b/test/e2e/commit_window_batching_e2e_test.go index a0eda207a..e011dea89 100644 --- a/test/e2e/commit_window_batching_e2e_test.go +++ b/test/e2e/commit_window_batching_e2e_test.go @@ -35,131 +35,135 @@ import ( // and one push. The audit-redis consumer pipeline is the real path under test // — events flow kubectl → audit webhook → Valkey stream → consumer → // BranchWorker.commitGroups → BranchWorker.pushPendingCommits. -var _ = Describe("Commit Window Batching", Label("commit-window-batching", "audit-redis", "smoke"), Ordered, func() { - var ( - testNs string - gitProvName string - gitTargetName string - watchRuleName string - ) - - const commitWindow = "3s" - - BeforeAll(func() { - By("creating commit-window-batching test namespace and applying git secrets") - testNs = testNamespaceFor("commit-window-batching") - _, _ = kubectlRun("create", "namespace", testNs) - repo := ensureAuditRedisRepo() - _, err := kubectlRunInNamespace(testNs, "apply", "-f", repo.SecretsYAML) - Expect(err).NotTo(HaveOccurred(), "failed to apply git secrets to namespace") - applySOPSAgeKeyToNamespace(testNs) - - seed := GinkgoRandomSeed() - gitProvName = fmt.Sprintf("commit-window-gitprovider-%d", seed) - gitTargetName = fmt.Sprintf("commit-window-gittarget-%d", seed) - watchRuleName = fmt.Sprintf("commit-window-watchrule-%d", seed) - - By(fmt.Sprintf("creating GitProvider with commitWindow=%s", commitWindow)) - createGitProviderWithCommitWindow( - gitProvName, - testNs, - repo.GitSecretHTTP, - repo.RepoURLHTTP, - commitWindow, +// Serial: shares the single global audit pipeline (audit webhook → Redis stream +// → consumer) with every other audit-redis spec. See +// docs/design/e2e-serial-registry.md. +var _ = Describe("Commit Window Batching", + Label("commit-window-batching", "audit-redis", "smoke"), Serial, Ordered, func() { + var ( + testNs string + gitProvName string + gitTargetName string + watchRuleName string ) - verifyResourceStatus("gitprovider", gitProvName, testNs, "True", "Ready", "") - - createGitTarget(gitTargetName, testNs, gitProvName, "e2e/commit-window-test", "main") - verifyResourceStatus("gittarget", gitTargetName, testNs, "True", "Ready", "") - - watchRuleData := struct { - Name string - Namespace string - DestinationName string - }{ - Name: watchRuleName, - Namespace: testNs, - DestinationName: gitTargetName, - } - err = applyFromTemplate("test/e2e/templates/manager/watchrule-configmap.tmpl", watchRuleData, testNs) - Expect(err).NotTo(HaveOccurred(), "failed to apply WatchRule") - verifyResourceStatus("watchrule", watchRuleName, testNs, "True", "Ready", "") - }) - - AfterAll(func() { - cleanupWatchRule(watchRuleName, testNs) - cleanupGitTarget(gitTargetName, testNs) - cleanupNamespacedResource(testNs, "gitprovider", gitProvName) - cleanupNamespace(testNs) - }) - It("collapses a burst of events into one grouped commit and one push", func() { - const burstSize = 4 - - repo := auditRedisRepo - basePath := "e2e/commit-window-test" - seed := GinkgoRandomSeed() - burstPrefix := fmt.Sprintf("commit-window-burst-%d", seed) - - // Seed: ensure the repo has at least one commit for this target so - // commit-count math is meaningful and not racing against bootstrap. - seedCM := fmt.Sprintf("%s-seed", burstPrefix) - applyConfigMap(testNs, seedCM) - Eventually(func(g Gomega) { - pullLatestRepoState(g, repo.CheckoutDir) - expected := filepath.Join(repo.CheckoutDir, basePath, - "v1", "configmaps", testNs, seedCM+".yaml") - _, statErr := os.Stat(expected) - g.Expect(statErr).NotTo(HaveOccurred(), - fmt.Sprintf("seed ConfigMap should land before the burst: %s", expected)) - }, 90*time.Second, 2*time.Second).Should(Succeed()) - - commitsBefore := mustCommitCount(repo.CheckoutDir) - - By(fmt.Sprintf("creating %d ConfigMaps in rapid succession", burstSize)) - burstStart := time.Now() - burstNames := make([]string, burstSize) - for i := range burstSize { - name := fmt.Sprintf("%s-%d", burstPrefix, i) - burstNames[i] = name - applyConfigMap(testNs, name) - } - burstSubmitted := time.Since(burstStart) - // Burst must fit comfortably inside the commit window so the - // batching property is what we are actually exercising. - Expect(burstSubmitted).To(BeNumerically("<", 2*time.Second), - "burst should be submitted faster than the commit window") - - By("waiting for every burst ConfigMap to land in the repository") - Eventually(func(g Gomega) { - pullLatestRepoState(g, repo.CheckoutDir) - for _, name := range burstNames { + const commitWindow = "3s" + + BeforeAll(func() { + By("creating commit-window-batching test namespace and applying git secrets") + testNs = testNamespaceFor("commit-window-batching") + _, _ = kubectlRun("create", "namespace", testNs) + repo := ensureAuditRedisRepo() + _, err := kubectlRunInNamespace(testNs, "apply", "-f", repo.SecretsYAML) + Expect(err).NotTo(HaveOccurred(), "failed to apply git secrets to namespace") + applySOPSAgeKeyToNamespace(testNs) + + seed := GinkgoRandomSeed() + gitProvName = fmt.Sprintf("commit-window-gitprovider-%d", seed) + gitTargetName = fmt.Sprintf("commit-window-gittarget-%d", seed) + watchRuleName = fmt.Sprintf("commit-window-watchrule-%d", seed) + + By(fmt.Sprintf("creating GitProvider with commitWindow=%s", commitWindow)) + createGitProviderWithCommitWindow( + gitProvName, + testNs, + repo.GitSecretHTTP, + repo.RepoURLHTTP, + commitWindow, + ) + verifyResourceStatus("gitprovider", gitProvName, testNs, "True", "Ready", "") + + createGitTarget(gitTargetName, testNs, gitProvName, "e2e/commit-window-test", "main") + verifyResourceStatus("gittarget", gitTargetName, testNs, "True", "Ready", "") + + watchRuleData := struct { + Name string + Namespace string + DestinationName string + }{ + Name: watchRuleName, + Namespace: testNs, + DestinationName: gitTargetName, + } + err = applyFromTemplate("test/e2e/templates/manager/watchrule-configmap.tmpl", watchRuleData, testNs) + Expect(err).NotTo(HaveOccurred(), "failed to apply WatchRule") + verifyResourceStatus("watchrule", watchRuleName, testNs, "True", "Ready", "") + }) + + AfterAll(func() { + cleanupWatchRule(watchRuleName, testNs) + cleanupGitTarget(gitTargetName, testNs) + cleanupNamespacedResource(testNs, "gitprovider", gitProvName) + cleanupNamespace(testNs) + }) + + It("collapses a burst of events into one grouped commit and one push", func() { + const burstSize = 4 + + repo := auditRedisRepo + basePath := "e2e/commit-window-test" + seed := GinkgoRandomSeed() + burstPrefix := fmt.Sprintf("commit-window-burst-%d", seed) + + // Seed: ensure the repo has at least one commit for this target so + // commit-count math is meaningful and not racing against bootstrap. + seedCM := fmt.Sprintf("%s-seed", burstPrefix) + applyConfigMap(testNs, seedCM) + Eventually(func(g Gomega) { + pullLatestRepoState(g, repo.CheckoutDir) expected := filepath.Join(repo.CheckoutDir, basePath, - "v1", "configmaps", testNs, name+".yaml") + "v1", "configmaps", testNs, seedCM+".yaml") _, statErr := os.Stat(expected) g.Expect(statErr).NotTo(HaveOccurred(), - fmt.Sprintf("ConfigMap file should exist: %s", expected)) + fmt.Sprintf("seed ConfigMap should land before the burst: %s", expected)) + }, 90*time.Second, 2*time.Second).Should(Succeed()) + + commitsBefore := mustCommitCount(repo.CheckoutDir) + + By(fmt.Sprintf("creating %d ConfigMaps in rapid succession", burstSize)) + burstStart := time.Now() + burstNames := make([]string, burstSize) + for i := range burstSize { + name := fmt.Sprintf("%s-%d", burstPrefix, i) + burstNames[i] = name + applyConfigMap(testNs, name) } - }, 90*time.Second, 2*time.Second).Should(Succeed()) - - commitsAfter := mustCommitCount(repo.CheckoutDir) - commitsAdded := commitsAfter - commitsBefore - - // Phase 3 groups same-author events in the window into one commit. The - // batching property is observable as one new remote commit touching the - // whole burst after the commitWindow expires. - Expect(commitsAdded).To(Equal(1), - fmt.Sprintf("expected 1 grouped commit for the burst, got %d", commitsAdded)) - - assertBurstFilesAreGroupedIntoLatestCommit(repo.CheckoutDir, burstNames, basePath, testNs) - - By("cleaning up burst ConfigMaps") - for _, name := range burstNames { - _, _ = kubectlRunInNamespace(testNs, "delete", "configmap", name, "--ignore-not-found=true") - } - _, _ = kubectlRunInNamespace(testNs, "delete", "configmap", seedCM, "--ignore-not-found=true") + burstSubmitted := time.Since(burstStart) + // Burst must fit comfortably inside the commit window so the + // batching property is what we are actually exercising. + Expect(burstSubmitted).To(BeNumerically("<", 2*time.Second), + "burst should be submitted faster than the commit window") + + By("waiting for every burst ConfigMap to land in the repository") + Eventually(func(g Gomega) { + pullLatestRepoState(g, repo.CheckoutDir) + for _, name := range burstNames { + expected := filepath.Join(repo.CheckoutDir, basePath, + "v1", "configmaps", testNs, name+".yaml") + _, statErr := os.Stat(expected) + g.Expect(statErr).NotTo(HaveOccurred(), + fmt.Sprintf("ConfigMap file should exist: %s", expected)) + } + }, 90*time.Second, 2*time.Second).Should(Succeed()) + + commitsAfter := mustCommitCount(repo.CheckoutDir) + commitsAdded := commitsAfter - commitsBefore + + // Phase 3 groups same-author events in the window into one commit. The + // batching property is observable as one new remote commit touching the + // whole burst after the commitWindow expires. + Expect(commitsAdded).To(Equal(1), + fmt.Sprintf("expected 1 grouped commit for the burst, got %d", commitsAdded)) + + assertBurstFilesAreGroupedIntoLatestCommit(repo.CheckoutDir, burstNames, basePath, testNs) + + By("cleaning up burst ConfigMaps") + for _, name := range burstNames { + _, _ = kubectlRunInNamespace(testNs, "delete", "configmap", name, "--ignore-not-found=true") + } + _, _ = kubectlRunInNamespace(testNs, "delete", "configmap", seedCM, "--ignore-not-found=true") + }) }) -}) // applyConfigMap renders the standard manager configmap template and applies // it. The template carries the ConfigMap data the burst-event audit pipeline diff --git a/test/e2e/crd_lifecycle_e2e_test.go b/test/e2e/crd_lifecycle_e2e_test.go index 0f4fdfb0e..9c475c35b 100644 --- a/test/e2e/crd_lifecycle_e2e_test.go +++ b/test/e2e/crd_lifecycle_e2e_test.go @@ -72,7 +72,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { _, _ = kubectlRun( "delete", "crd", - "icecreamorders.shop.example.com", + iceCreamCRDName(crdGroupCRDLifecycle), "--ignore-not-found=true", ) @@ -97,7 +97,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { It("should create Git commit when IceCreamOrder CRD is installed via ClusterWatchRule", Label("smoke"), func() { gitProviderName := "gitprovider-normal" clusterWatchRuleName := "clusterwatchrule-crd-install" - crdName := "icecreamorders.shop.example.com" + crdName := iceCreamCRDName(crdGroupCRDLifecycle) By("creating ClusterWatchRule with Cluster scope for CRDs") destName := clusterWatchRuleName + "-dest" @@ -121,7 +121,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { verifyResourceStatus("gittarget", destName, testNs, "True", "Ready", "") By("installing the IceCreamOrder CRD to trigger Git commit") - _, err = kubectlRun("apply", "-f", "test/e2e/templates/icecreamorder-crd.yaml") + err = applyIceCreamCRD(crdGroupCRDLifecycle) Expect(err).NotTo(HaveOccurred(), "Failed to install CRD") By("waiting for CRD to be established") @@ -151,7 +151,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { // CRDs are cluster-scoped, so path should NOT include namespace expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, "e2e/crd-install-test", - "apiextensions.k8s.io/v1/customresourcedefinitions/icecreamorders.shop.example.com.yaml") + "apiextensions.k8s.io/v1/customresourcedefinitions/"+iceCreamCRDMirrorFile(crdGroupCRDLifecycle)) fileInfo, statErr := os.Stat(expectedFile) g.Expect(statErr).NotTo(HaveOccurred(), fmt.Sprintf("CRD file should exist at %s", expectedFile)) g.Expect(fileInfo.Size()).To(BeNumerically(">", 0), "CRD file should not be empty") @@ -161,7 +161,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { g.Expect(readErr).NotTo(HaveOccurred()) g.Expect(string(content)).To(ContainSubstring("kind: CustomResourceDefinition"), "File should contain CRD kind") - g.Expect(string(content)).To(ContainSubstring("name: icecreamorders.shop.example.com"), + g.Expect(string(content)).To(ContainSubstring("name: "+iceCreamCRDName(crdGroupCRDLifecycle)), "File should contain CRD name") } Eventually(verifyGitCommit). @@ -182,7 +182,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { watchRuleName := "watchrule-icecream-orders" By("installing the IceCreamOrder CRD first (needed for custom resource tests)") - _, err := kubectlRun("apply", "-f", "test/e2e/templates/icecreamorder-crd.yaml") + err := applyIceCreamCRD(crdGroupCRDLifecycle) Expect(err).NotTo(HaveOccurred(), "Failed to install sample CRD") By("waiting for CRD to be established") @@ -190,7 +190,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { output, err := kubectlRun( "get", "crd", - "icecreamorders.shop.example.com", + iceCreamCRDName(crdGroupCRDLifecycle), "-o", "jsonpath={.status.conditions[?(@.type=='Established')].status}", ) @@ -205,7 +205,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { // event is missed and the next spec sees no file in git. Block until // the resource is actually listable end-to-end. Eventually(func(g Gomega) { - _, err := kubectlRun("get", "icecreamorders.shop.example.com", "-A") + _, err := kubectlRun("get", iceCreamCRDName(crdGroupCRDLifecycle), "-A") g.Expect(err).NotTo(HaveOccurred(), "icecreamorders must be servable before creating instances") }, 30*time.Second, time.Second).Should(Succeed()) @@ -220,10 +220,12 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { Name string Namespace string DestinationName string + Group string }{ Name: watchRuleName, Namespace: testNs, DestinationName: destName, + Group: crdGroupCRDLifecycle, } err2 := applyFromTemplate("test/e2e/templates/watchrule-crd.tmpl", data, testNs) @@ -246,6 +248,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { Quantity int } Toppings []string + Group string }{ Name: crdInstanceName, Namespace: testNs, @@ -259,6 +262,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { "kubectl.kubernetes.io/last-applied-configuration": "should-be-filtered", "deployment.kubernetes.io/revision": "should-also-be-filtered", }, + Group: crdGroupCRDLifecycle, CustomerName: "Alice", Container: "Cone", Scoops: []struct { @@ -302,7 +306,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, "e2e/icecream-test", - fmt.Sprintf("shop.example.com/v1/icecreamorders/%s/%s.yaml", testNs, crdInstanceName)) + fmt.Sprintf("%s/%s/%s.yaml", iceCreamInstanceDir(crdGroupCRDLifecycle), testNs, crdInstanceName)) fileInfo, statErr := os.Stat(expectedFile) g.Expect(statErr). NotTo(HaveOccurred(), fmt.Sprintf("CRD instance file should exist at %s", expectedFile)) @@ -349,7 +353,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { By("verifying the IceCreamOrder still exists before status update") verifyCRDInstanceExists := func(g Gomega) { - _, err := kubectlRunInNamespace(testNs, "get", "icecreamorder", crdInstanceName) + _, err := kubectlRunInNamespace(testNs, "get", iceCreamCRDName(crdGroupCRDLifecycle), crdInstanceName) g.Expect(err).NotTo(HaveOccurred(), "IceCreamOrder should exist before status patch") } Eventually(verifyCRDInstanceExists, 15*time.Second, time.Second).Should(Succeed()) @@ -359,7 +363,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { statusOutput, statusErr := kubectlRunInNamespace( testNs, "patch", - "icecreamorder", + iceCreamCRDName(crdGroupCRDLifecycle), crdInstanceName, "--type=merge", "--subresource=status", @@ -395,7 +399,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { // Read the file again to ensure status is still not present expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, "e2e/icecream-test", - fmt.Sprintf("shop.example.com/v1/icecreamorders/%s/%s.yaml", testNs, crdInstanceName)) + fmt.Sprintf("%s/%s/%s.yaml", iceCreamInstanceDir(crdGroupCRDLifecycle), testNs, crdInstanceName)) content, readErr := os.ReadFile(expectedFile) g.Expect(readErr).NotTo(HaveOccurred()) g.Expect(string(content)).NotTo(ContainSubstring("status:"), @@ -410,7 +414,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { By("✅ Status update verified - no Git commit created and status not in file") By("cleaning up IceCreamOrder instance") - _, _ = kubectlRunInNamespace(testNs, "delete", "icecreamorder", crdInstanceName) + _, _ = kubectlRunInNamespace(testNs, "delete", iceCreamCRDName(crdGroupCRDLifecycle), crdInstanceName) By("Note: GitTarget, WatchRule, GitProvider, and CRD kept for subsequent tests") @@ -436,11 +440,13 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { Quantity int } Toppings []string + Group string }{ Name: crdInstanceName, Namespace: testNs, Labels: nil, Annotations: nil, + Group: crdGroupCRDLifecycle, CustomerName: "Bob", Container: "Cup", Scoops: []struct { @@ -463,7 +469,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, "e2e/icecream-test", - fmt.Sprintf("shop.example.com/v1/icecreamorders/%s/%s.yaml", testNs, crdInstanceName)) + fmt.Sprintf("%s/%s/%s.yaml", iceCreamInstanceDir(crdGroupCRDLifecycle), testNs, crdInstanceName)) content, readErr := os.ReadFile(expectedFile) g.Expect(readErr).NotTo(HaveOccurred()) g.Expect(string(content)).To(ContainSubstring("customerName: Bob")) @@ -484,11 +490,13 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { Quantity int } Toppings []string + Group string }{ Name: crdInstanceName, Namespace: testNs, Labels: nil, Annotations: nil, + Group: crdGroupCRDLifecycle, CustomerName: "Bob", Container: "WaffleBowl", Scoops: []struct { @@ -512,7 +520,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, "e2e/icecream-test", - fmt.Sprintf("shop.example.com/v1/icecreamorders/%s/%s.yaml", testNs, crdInstanceName)) + fmt.Sprintf("%s/%s/%s.yaml", iceCreamInstanceDir(crdGroupCRDLifecycle), testNs, crdInstanceName)) content, readErr := os.ReadFile(expectedFile) g.Expect(readErr).NotTo(HaveOccurred()) g.Expect(string(content)).To(ContainSubstring("container: WaffleBowl"), @@ -525,7 +533,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { Eventually(verifyUpdatedFile).Should(Succeed()) By("cleaning up IceCreamOrder instance") - _, _ = kubectlRunInNamespace(testNs, "delete", "icecreamorder", crdInstanceName) + _, _ = kubectlRunInNamespace(testNs, "delete", iceCreamCRDName(crdGroupCRDLifecycle), crdInstanceName) By("Note: GitTarget, WatchRule, GitProvider, and CRD kept for subsequent tests") @@ -551,11 +559,13 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { Quantity int } Toppings []string + Group string }{ Name: crdInstanceName, Namespace: testNs, Labels: nil, Annotations: nil, + Group: crdGroupCRDLifecycle, CustomerName: "Charlie", Container: "Cone", Scoops: []struct { @@ -578,7 +588,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, "e2e/icecream-test", - fmt.Sprintf("shop.example.com/v1/icecreamorders/%s/%s.yaml", testNs, crdInstanceName)) + fmt.Sprintf("%s/%s/%s.yaml", iceCreamInstanceDir(crdGroupCRDLifecycle), testNs, crdInstanceName)) fileInfo, statErr := os.Stat(expectedFile) g.Expect(statErr). NotTo(HaveOccurred(), fmt.Sprintf("CRD instance file should exist at %s", expectedFile)) @@ -589,7 +599,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { Eventually(verifyFileCreated, 60*time.Second, 2*time.Second).Should(Succeed()) By("deleting the CR to trigger DELETE operation") - _, err = kubectlRunInNamespace(testNs, "delete", "icecreamorder", crdInstanceName) + _, err = kubectlRunInNamespace(testNs, "delete", iceCreamCRDName(crdGroupCRDLifecycle), crdInstanceName) Expect(err).NotTo(HaveOccurred(), "CRD instance deletion should succeed") By("verifying CRD instance file is deleted from Git repository") @@ -600,7 +610,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, "e2e/icecream-test", - fmt.Sprintf("shop.example.com/v1/icecreamorders/%s/%s.yaml", testNs, crdInstanceName)) + fmt.Sprintf("%s/%s/%s.yaml", iceCreamInstanceDir(crdGroupCRDLifecycle), testNs, crdInstanceName)) _, statErr := os.Stat(expectedFile) g.Expect(statErr). To(HaveOccurred(), fmt.Sprintf("CRD instance file should NOT exist at %s", expectedFile)) @@ -624,7 +634,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { It("should delete Git file when IceCreamOrder CRD is deleted via ClusterWatchRule", func() { gitProviderName := "gitprovider-normal" clusterWatchRuleName := "clusterwatchrule-crd-delete" - crdName := "icecreamorders.shop.example.com" + crdName := iceCreamCRDName(crdGroupCRDLifecycle) By("creating ClusterWatchRule with Cluster scope for CRDs") destName := clusterWatchRuleName + "-dest" @@ -656,7 +666,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, "e2e/crd-delete-test", - "apiextensions.k8s.io/v1/customresourcedefinitions/icecreamorders.shop.example.com.yaml") + "apiextensions.k8s.io/v1/customresourcedefinitions/"+iceCreamCRDMirrorFile(crdGroupCRDLifecycle)) _, statErr := os.Stat(expectedFile) g.Expect(statErr).NotTo(HaveOccurred(), "CRD file should exist before deletion") } @@ -673,7 +683,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, "e2e/crd-delete-test", - "apiextensions.k8s.io/v1/customresourcedefinitions/icecreamorders.shop.example.com.yaml") + "apiextensions.k8s.io/v1/customresourcedefinitions/"+iceCreamCRDMirrorFile(crdGroupCRDLifecycle)) _, statErr := os.Stat(expectedFile) g.Expect(statErr).To(HaveOccurred(), "CRD file should NOT exist after deletion") g.Expect(os.IsNotExist(statErr)).To(BeTrue(), "Error should be 'file does not exist'") @@ -696,7 +706,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, "e2e/crd-delete-test", - "apiextensions.k8s.io/v1/customresourcedefinitions/icecreamorders.shop.example.com.yaml") + "apiextensions.k8s.io/v1/customresourcedefinitions/"+iceCreamCRDMirrorFile(crdGroupCRDLifecycle)) _, statErr := os.Stat(expectedFile) g.Expect(statErr).To(HaveOccurred(), "CRD file must stay deleted after CRD termination updates") g.Expect(os.IsNotExist(statErr)).To(BeTrue(), "CRD file must not reappear in Git") diff --git a/test/e2e/e2e_suite_test.go b/test/e2e/e2e_suite_test.go index bca687baf..8fa61dcb9 100644 --- a/test/e2e/e2e_suite_test.go +++ b/test/e2e/e2e_suite_test.go @@ -47,7 +47,11 @@ func TestE2E(t *testing.T) { RunSpecs(t, "e2e suite") } -var _ = BeforeSuite(func() { +// SynchronizedBeforeSuite splits bootstrap into work that must happen exactly +// once (the Task prepare flow and the cluster-scoped CRD pre-cleanup, run on +// parallel process #1) and per-process configuration (the E2E_AGE_KEY_FILE +// fallback, which relies on os.Setenv and therefore must run in every process). +var _ = SynchronizedBeforeSuite(func() []byte { defer func() { if recovered := recover(); recovered != nil { markSuiteWidePreservation("BeforeSuite failed") @@ -62,14 +66,20 @@ var _ = BeforeSuite(func() { } By("preparing e2e cluster prerequisites via Task target") - ensureE2EPrepared() + prepareE2EClusterOnce() + return nil +}, func(_ []byte) { + configureE2EProcess() }) var _ = AfterEach(func() { dumpFailureDiagnostics() }) -func ensureE2EPrepared() { +// prepareE2EClusterOnce runs the expensive, cluster-mutating bootstrap exactly +// once (parallel process #1): the Task prepare flow plus the cluster-scoped CRD +// pre-cleanup. It is safe to run concurrently with nothing else. +func prepareE2EClusterOnce() { ctx := resolveE2EContext() setE2EKubectlContext(ctx) ns := resolveE2ENamespace() @@ -93,20 +103,35 @@ func ensureE2EPrepared() { Expect(err).NotTo(HaveOccurred(), "failed to set kubectl context for e2e run") _, _ = fmt.Fprintf(GinkgoWriter, "%s", output) - By("ensuring IceCreamOrder CRD is removed before tests") - output, err = kubectlRun( - "delete", "crd", "icecreamorders.shop.example.com", - "--ignore-not-found=true", - ) - Expect(err).NotTo(HaveOccurred(), "failed to delete IceCreamOrder CRD before tests") - _, _ = fmt.Fprintf(GinkgoWriter, "%s", output) + // Each CRD-installing e2e file now owns its own IceCreamOrder CRD group + // (see test/e2e/icecream.go). Remove all of them, plus the legacy shared + // group, so a warm/reused cluster starts clean. + By("ensuring IceCreamOrder CRDs are removed before tests") + for _, group := range []string{ + crdGroupCRDLifecycle, + crdGroupRestartSnapshot, + crdGroupBiDirectional, + "shop.example.com", // legacy pre-isolation group + } { + output, err = kubectlRun( + "delete", "crd", iceCreamCRDName(group), + "--ignore-not-found=true", + ) + Expect(err).NotTo(HaveOccurred(), + "failed to delete IceCreamOrder CRD %q before tests", iceCreamCRDName(group)) + _, _ = fmt.Fprintf(GinkgoWriter, "%s", output) + } +} - // The Task prepare flow writes the age key under the stamp directory. When running `go test` directly - // (without `task test-e2e`), ensure the suite uses that prepared key file. +// configureE2EProcess runs in every parallel process. The Task prepare flow +// writes the age key under the stamp directory; when running `go test` directly +// (without `task test-e2e`), point the suite at that prepared key file. This +// relies on os.Setenv, so it must run per-process rather than once. +func configureE2EProcess() { if strings.TrimSpace(os.Getenv("E2E_AGE_KEY_FILE")) == "" { wd, err := os.Getwd() Expect(err).NotTo(HaveOccurred(), "failed to get working directory for e2e run") - ageKeyPath := filepath.Join(wd, ".stamps", "cluster", ctx, "age-key.txt") + ageKeyPath := filepath.Join(wd, ".stamps", "cluster", resolveE2EContext(), "age-key.txt") Expect(os.Setenv("E2E_AGE_KEY_FILE", ageKeyPath)).To(Succeed()) } } diff --git a/test/e2e/icecream.go b/test/e2e/icecream.go new file mode 100644 index 000000000..f4ccd1c49 --- /dev/null +++ b/test/e2e/icecream.go @@ -0,0 +1,61 @@ +/* +SPDX-License-Identifier: Apache-2.0 + +Copyright 2025 ConfigButler + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +// Per-file IceCreamOrder CRD groups. Every e2e file that installs the +// IceCreamOrder CRD owns its own API group so the three files no longer share +// cluster-scoped state and can run on parallel Ginkgo processes (Phase 2.5 of +// docs/design/e2e-speedup-plan.md). The kind, plural and singular stay the same +// across all three; only the group differs. Because the plural ("icecreamorders") +// is shared, kubectl resource references MUST be qualified as +// "icecreamorders." (see iceCreamCRDName) to avoid ambiguity when more +// than one of these CRDs is installed concurrently. +const ( + crdGroupCRDLifecycle = "crd-lifecycle.e2e.example.com" + crdGroupRestartSnapshot = "restart-snapshot.e2e.example.com" + crdGroupBiDirectional = "bi-directional.e2e.example.com" +) + +// iceCreamCRDName returns the fully-qualified CRD name for the given group. This +// doubles as the unambiguous kubectl resource selector (e.g. +// "icecreamorders.crd-lifecycle.e2e.example.com"). +func iceCreamCRDName(group string) string { + return "icecreamorders." + group +} + +// iceCreamCRDMirrorFile returns the filename gitops-reverser writes for the CRD +// under apiextensions.k8s.io/v1/customresourcedefinitions/. +func iceCreamCRDMirrorFile(group string) string { + return iceCreamCRDName(group) + ".yaml" +} + +// iceCreamInstanceDir returns the mirror path prefix for IceCreamOrder instances +// of the given group: "/v1/icecreamorders". +func iceCreamInstanceDir(group string) string { + return group + "/v1/icecreamorders" +} + +// applyIceCreamCRD renders and applies the IceCreamOrder CRD for the given group. +func applyIceCreamCRD(group string) error { + return applyFromTemplate( + "test/e2e/templates/icecreamorder-crd.tmpl", + struct{ Group string }{Group: group}, + "", + ) +} diff --git a/test/e2e/image_refresh_test.go b/test/e2e/image_refresh_test.go index ac309fe17..026dda39b 100644 --- a/test/e2e/image_refresh_test.go +++ b/test/e2e/image_refresh_test.go @@ -40,7 +40,10 @@ import ( // or via: // // task test-image-refresh -var _ = Describe("image refresh dependency chain", Label("image-refresh"), Ordered, func() { +// +// Serial: redeploys / changes the controller image, perturbing the shared +// controller for any concurrent spec. See docs/design/e2e-serial-registry.md. +var _ = Describe("image refresh dependency chain", Label("image-refresh"), Serial, Ordered, func() { var ( projectDir string deployedStamp string diff --git a/test/e2e/restart_snapshot_e2e_test.go b/test/e2e/restart_snapshot_e2e_test.go index be54ede87..e5bb12e4a 100644 --- a/test/e2e/restart_snapshot_e2e_test.go +++ b/test/e2e/restart_snapshot_e2e_test.go @@ -45,7 +45,9 @@ import ( // IMPORTANT: the wildcard rule is what makes this test meaningful. A test that // watches with a concrete `apiVersions: ["v1"]` rule passes even with the bug // present — which is exactly why the existing e2e suite never caught this. -var _ = Describe("Restart Snapshot Safety", Label("restart-snapshot", "smoke"), Ordered, func() { +// Serial: rolls the controller deployment, which disrupts any spec running +// concurrently on another process. See docs/design/e2e-serial-registry.md. +var _ = Describe("Restart Snapshot Safety", Label("restart-snapshot", "smoke"), Serial, Ordered, func() { var ( testNs string restartRepo *RepoArtifacts @@ -92,11 +94,11 @@ var _ = Describe("Restart Snapshot Safety", Label("restart-snapshot", "smoke"), It("keeps the git mirror intact when the controller restarts", func() { By("installing the IceCreamOrder CRD") - _, err := kubectlRun("apply", "-f", "test/e2e/templates/icecreamorder-crd.yaml") + err := applyIceCreamCRD(crdGroupRestartSnapshot) Expect(err).NotTo(HaveOccurred(), "failed to install IceCreamOrder CRD") Eventually(func(g Gomega) { output, getErr := kubectlRun( - "get", "crd", "icecreamorders.shop.example.com", + "get", "crd", iceCreamCRDName(crdGroupRestartSnapshot), "-o", "jsonpath={.status.conditions[?(@.type=='Established')].status}", ) g.Expect(getErr).NotTo(HaveOccurred()) @@ -111,10 +113,12 @@ var _ = Describe("Restart Snapshot Safety", Label("restart-snapshot", "smoke"), Name string DestinationName string Namespace string + Group string }{ Name: clusterWatchRuleName, DestinationName: gitTargetName, Namespace: testNs, + Group: crdGroupRestartSnapshot, } Expect(applyFromTemplate( "test/e2e/templates/restart/clusterwatchrule-wildcard.tmpl", cwrData, "", @@ -125,13 +129,13 @@ var _ = Describe("Restart Snapshot Safety", Label("restart-snapshot", "smoke"), By("creating quiet IceCreamOrder resources to build up the git mirror") for _, name := range orderNames { - createIceCreamOrder(testNs, name) + createIceCreamOrder(crdGroupRestartSnapshot, testNs, name) } expectedFiles := make([]string, 0, len(orderNames)) for _, name := range orderNames { expectedFiles = append(expectedFiles, filepath.Join( - gitTargetPath, "shop.example.com", "v1", "icecreamorders", testNs, name+".yaml", + gitTargetPath, iceCreamInstanceDir(crdGroupRestartSnapshot), testNs, name+".yaml", )) } @@ -176,10 +180,11 @@ var _ = Describe("Restart Snapshot Safety", Label("restart-snapshot", "smoke"), }) }) -// createIceCreamOrder applies a minimal IceCreamOrder custom resource. -func createIceCreamOrder(ns, name string) { +// createIceCreamOrder applies a minimal IceCreamOrder custom resource for the +// given CRD group. +func createIceCreamOrder(group, ns, name string) { By(fmt.Sprintf("creating IceCreamOrder '%s/%s'", ns, name)) - manifest := fmt.Sprintf(`apiVersion: shop.example.com/v1 + manifest := fmt.Sprintf(`apiVersion: %s/v1 kind: IceCreamOrder metadata: name: %s @@ -190,7 +195,7 @@ spec: scoops: - flavor: Vanilla quantity: 1 -`, name, ns, name) +`, group, name, ns, name) _, err := kubectlRunWithStdin(ns, manifest, "apply", "-f", "-") Expect(err).NotTo(HaveOccurred(), "failed to apply IceCreamOrder %s/%s", ns, name) } diff --git a/test/e2e/templates/icecreamorder-crd.yaml b/test/e2e/templates/icecreamorder-crd.tmpl similarity index 98% rename from test/e2e/templates/icecreamorder-crd.yaml rename to test/e2e/templates/icecreamorder-crd.tmpl index 4d87345e6..1f4b2ad86 100644 --- a/test/e2e/templates/icecreamorder-crd.yaml +++ b/test/e2e/templates/icecreamorder-crd.tmpl @@ -2,10 +2,10 @@ apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: # name must match the spec fields below, and be in the form: . - name: icecreamorders.shop.example.com + name: icecreamorders.{{ .Group }} spec: # group name to use for REST API: /apis// - group: shop.example.com + group: {{ .Group }} # list of versions supported by this CRD versions: - name: v1 diff --git a/test/e2e/templates/icecreamorder-instance.tmpl b/test/e2e/templates/icecreamorder-instance.tmpl index 2cb5789d7..9e65af3b6 100644 --- a/test/e2e/templates/icecreamorder-instance.tmpl +++ b/test/e2e/templates/icecreamorder-instance.tmpl @@ -1,4 +1,4 @@ -apiVersion: shop.example.com/v1 +apiVersion: {{ .Group }}/v1 kind: IceCreamOrder metadata: name: {{ .Name }} diff --git a/test/e2e/templates/restart/clusterwatchrule-wildcard.tmpl b/test/e2e/templates/restart/clusterwatchrule-wildcard.tmpl index aec9ab522..41b12e82f 100644 --- a/test/e2e/templates/restart/clusterwatchrule-wildcard.tmpl +++ b/test/e2e/templates/restart/clusterwatchrule-wildcard.tmpl @@ -13,6 +13,6 @@ spec: # restart snapshots an empty cluster and deletes the whole tracked tree. - scope: Namespaced operations: [CREATE, UPDATE, DELETE] - apiGroups: ["shop.example.com"] + apiGroups: ["{{ .Group }}"] apiVersions: ["*"] resources: ["icecreamorders"] diff --git a/test/e2e/templates/watchrule-crd.tmpl b/test/e2e/templates/watchrule-crd.tmpl index 5d4a85e2e..939c8e264 100644 --- a/test/e2e/templates/watchrule-crd.tmpl +++ b/test/e2e/templates/watchrule-crd.tmpl @@ -7,6 +7,6 @@ spec: targetRef: name: {{ .DestinationName }} rules: - - apiGroups: ["shop.example.com"] + - apiGroups: ["{{ .Group }}"] apiVersions: ["v1"] resources: ["icecreamorders"] From cad990369a274f511045e9a795b203ab866b4243 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 29 May 2026 13:43:47 +0000 Subject: [PATCH 09/40] =?UTF-8?q?chore:=20phase=202.5=20=E2=80=94=20mark?= =?UTF-8?q?=20bi-directional=20+=20audit-pipeline=20specs=20Serial?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two failures surfaced under procs=2 parallelism: - Bi Directional asserts exact remote commit counts to prove no commit loop; concurrent controller activity added 2 commits. Mark Serial. - Manager WatchRule ConfigMap delete: the DELETE commit lagged past the 30s default under a busier shared controller. Bump that Eventually to 60s (matches the crd-lifecycle precedent). The audit-pipeline specs (audit-redis queue/consumer, commit-window, commit-request) were already moved to Serial in the prior commit after the consumer picked up commit-window's files from the shared Redis stream. Serial registry updated. Co-Authored-By: Claude Opus 4.8 --- docs/design/e2e-serial-registry.md | 1 + test/e2e/bi_directional_e2e_test.go | 7 ++++++- test/e2e/watchrule_configmap_secret_e2e_test.go | 5 ++++- 3 files changed, 11 insertions(+), 2 deletions(-) diff --git a/docs/design/e2e-serial-registry.md b/docs/design/e2e-serial-registry.md index 44212b4c9..5a4a5373a 100644 --- a/docs/design/e2e-serial-registry.md +++ b/docs/design/e2e-serial-registry.md @@ -28,6 +28,7 @@ row below. | [test/e2e/audit_redis_e2e_test.go](../../test/e2e/audit_redis_e2e_test.go) | `Audit Redis Queue`, `Audit Redis Consumer` | The single global audit pipeline: audit webhook → shared Redis stream → consumer. | The consumer is a cluster-wide singleton processing one shared stream; these specs assert on commit *exclusivity*, which any concurrent audit traffic violates. | | [test/e2e/commit_window_batching_e2e_test.go](../../test/e2e/commit_window_batching_e2e_test.go) | `Commit Window Batching` | Same global audit pipeline (labelled `audit-redis`). | Bursts of ConfigMap events flow through the shared stream/consumer and leak into other audit specs' commits. | | [test/e2e/commit_request_e2e_test.go](../../test/e2e/commit_request_e2e_test.go) | `Commit Request` | Same global audit pipeline (labelled `audit-redis`). | Shares the stream/consumer; commit-window/queue semantics interleave under concurrency. | +| [test/e2e/bi_directional_e2e_test.go](../../test/e2e/bi_directional_e2e_test.go) | `Bi Directional` | Whole-cluster Flux↔gitops-reverser round-trip; asserts on **exact** remote commit counts to prove no commit loop. | Any concurrent controller activity adds/reorders commits and breaks the exact-count loop assertions. Passes sequentially; +2 commits only under parallelism. | ## Watched, but intentionally NOT Serial diff --git a/test/e2e/bi_directional_e2e_test.go b/test/e2e/bi_directional_e2e_test.go index 232c72384..ca857e9ef 100644 --- a/test/e2e/bi_directional_e2e_test.go +++ b/test/e2e/bi_directional_e2e_test.go @@ -95,7 +95,12 @@ type iceCreamScoop struct { // biDirectionalRepo holds the file-local repo fixtures for the Bi Directional describe block. var biDirectionalRepo *RepoArtifacts -var _ = Describe("Bi Directional", Label("bi-directional"), Ordered, func() { +// Serial: a whole-cluster bidirectional behavioral test (Flux applies from git +// while gitops-reverser mirrors live state back) that asserts on *exact* commit +// counts to prove no commit loop. Concurrent controller activity from other +// processes perturbs those counts. Passes sequentially; +2 commits only under +// parallelism. See docs/design/e2e-serial-registry.md. +var _ = Describe("Bi Directional", Label("bi-directional"), Serial, Ordered, func() { var run biDirectionalRun var testNs string diff --git a/test/e2e/watchrule_configmap_secret_e2e_test.go b/test/e2e/watchrule_configmap_secret_e2e_test.go index d284d0d06..bfacd1778 100644 --- a/test/e2e/watchrule_configmap_secret_e2e_test.go +++ b/test/e2e/watchrule_configmap_secret_e2e_test.go @@ -744,7 +744,10 @@ var _ = Describe("Manager WatchRule ConfigMap and Secret", Label("manager"), Ord g.Expect(string(logOutput)).To(ContainSubstring("DELETE"), "Git log should contain DELETE operation") } - Eventually(verifyFileDeleted).Should(Succeed()) + // 60s (vs the 30s default) tolerates a busier shared controller under + // Ginkgo parallelism: the DELETE commit can lag when other processes are + // driving reconciles concurrently. + Eventually(verifyFileDeleted, 60*time.Second, 2*time.Second).Should(Succeed()) By("cleaning up test resources") cleanupWatchRule(watchRuleName, testNs) From 9942e0cb23c0b970dd7166abf1e4a8c4d8acff62 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 29 May 2026 14:13:35 +0000 Subject: [PATCH 10/40] =?UTF-8?q?chore:=20phase=202.5=20=E2=80=94=20widen?= =?UTF-8?q?=20resource-readiness=20timeout=20for=20slow-CI=20parallelism?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI E2E (full) failed on crd_lifecycle spec #18: the WatchRule never reached Ready within the 30s default. Root cause (from CI logs): after a fresh CRD install the controller's discovery cache lags tens of seconds before it serves the new GVR ("resource is not served" / reflector "Failed to watch"), so the dependent WatchRule can't be planned. kubectl already serves the CRD, so the spec's servable-wait passes, but the controller hasn't caught up. The 30s readiness window expired; the subsequent "GitProvider not found" was just the AfterAll tearing down the namespace after the failure. Bump the shared verifyResourceStatus Eventually 30s → 90s (2s poll). The happy path still returns as soon as Ready; this only buys headroom on slow/contended CI runners under procs=2. Local fast runs don't reproduce the lag (cold full run passed 45/45). Co-Authored-By: Claude Opus 4.8 --- test/e2e/e2e_test.go | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/test/e2e/e2e_test.go b/test/e2e/e2e_test.go index 53ed522d5..a28bd9302 100644 --- a/test/e2e/e2e_test.go +++ b/test/e2e/e2e_test.go @@ -217,7 +217,11 @@ func verifyResourceStatus(resourceType, name, ns, expectedStatus, expectedReason g.Expect(readyMessage).To(ContainSubstring(expectedMessageContains)) } } - Eventually(verifyStatus).Should(Succeed()) + // 90s (vs the 30s default) absorbs slow-CI and parallel load: in particular, + // after a fresh CRD install the controller's discovery cache can lag tens of + // seconds before it serves the new GVR and a dependent WatchRule can be + // planned and reach Ready. The happy path still returns as soon as Ready. + Eventually(verifyStatus, "90s", "2s").Should(Succeed()) } // showControllerLogs displays the current controller logs to help with debugging during test execution. From dba83ac962c749e744788416419625daefabedda Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 29 May 2026 14:14:46 +0000 Subject: [PATCH 11/40] docs: record Phase 2.5 implementation status and measured impact Co-Authored-By: Claude Opus 4.8 --- docs/design/e2e-speedup-plan.md | 34 +++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/docs/design/e2e-speedup-plan.md b/docs/design/e2e-speedup-plan.md index dd767f740..c66c9afc1 100644 --- a/docs/design/e2e-speedup-plan.md +++ b/docs/design/e2e-speedup-plan.md @@ -418,6 +418,40 @@ registry then only needs to capture genuinely process-level conflicts - Smoke wallclock drops by at least the parallel-safe Group A+B+C share. - Serial registry exists and is non-empty. +**Implementation status (2026-05-29).** Landed on branch `refactors`. + +- **Parallelism is driven by the `ginkgo` CLI, not `go test`.** `go test` + has no `-ginkgo.procs` flag (it only accepts the `-ginkgo.parallel.*` + flags that the CLI sets on its workers). The three suite tasks now run + `go run github.com/onsi/ginkgo/v2/ginkgo --procs={{.E2E_GINKGO_PROCS}}` + (pinned to the module version to avoid a CLI/library mismatch). + `E2E_GINKGO_PROCS` defaults to `2`. +- **`SynchronizedBeforeSuite`** runs `prepare-e2e` + CRD pre-cleanup once + on process #1; the `E2E_AGE_KEY_FILE` fallback runs per-process. +- **CRD isolation:** each of `crd_lifecycle`, `restart_snapshot`, + `bi_directional` renders its own group (`*.e2e.example.com`) via + `test/e2e/icecream.go`; the shared `icecreamorders` plural means every + kubectl reference is qualified `icecreamorders.`. +- **Final Serial set** (see [e2e-serial-registry.md](e2e-serial-registry.md)) + ended up larger than the initial audit. Beyond `restart_snapshot`, + `image_refresh` and `aggregated_api` (controller/APIService level), the + four `audit-redis`-labelled containers (`Audit Redis Queue/Consumer`, + `Commit Window Batching`, `Commit Request`) had to go Serial — they + share one global audit pipeline (webhook → Redis stream → consumer) and + cross-contaminated each other's commits. `bi_directional` is Serial too: + it asserts exact commit counts to prove no commit loop, which any + concurrent controller activity breaks. +- **Timeouts widened for parallel/slow-CI load:** the shared + `verifyResourceStatus` readiness wait went 30s → 90s (after a fresh CRD + install the controller's discovery cache lags before it serves the new + GVR, so a dependent WatchRule can't reach Ready inside 30s on contended + runners); the manager ConfigMap-delete `Eventually` went 30s → 60s. +- **Measured:** smoke (`task test-e2e`, procs=2) **352 s** vs the 418.7 s + sequential baseline. Cold full run (fresh cluster, all 47 specs) green: + 45 passed / 0 failed / 2 skipped in ~20 min wallclock. The dominant + remaining smoke cost is the Serial `Restart Snapshot Safety` spec + (~128 s) — that is exactly what Phase 3 targets. + ### Phase 3 — Target-scoped reconcile-complete + drain signal for `Restart Snapshot Safety` **Why a pod-level gauge alone is insufficient** (per reviewer): a From 1294814c6cdaa02fc50561194cb13bdbe4f82cd7 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 29 May 2026 16:18:46 +0000 Subject: [PATCH 12/40] =?UTF-8?q?chore:=20e2e=20=E2=80=94=20raise=20k3d=20?= =?UTF-8?q?agents=201=E2=86=922=20to=20spread=20parallel=20load?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Under E2E_GINKGO_PROCS=2 the GitHub runners flaked: different specs hit 30s Eventually timeouts each run because all per-test pods plus the controller/Gitea/Flux/Redis contend on a single k3d agent node. Spread them across 2 worker nodes. Agents were 3 historically, dropped to 1 when 1+3 nodes exhausted the default fs.inotify.max_user_instances (128) and broke containerd. ensure_inotify_limits already raises that to 512, so 1 server + 2 agents has ample headroom. Co-Authored-By: Claude Opus 4.8 --- test/e2e/cluster/start-cluster.sh | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/test/e2e/cluster/start-cluster.sh b/test/e2e/cluster/start-cluster.sh index b71970fd9..a5a6426e0 100644 --- a/test/e2e/cluster/start-cluster.sh +++ b/test/e2e/cluster/start-cluster.sh @@ -172,7 +172,13 @@ create_cluster() { k3d cluster create "${CLUSTER_NAME}" --image rancher/k3s:v1.35.2-k3s1 --servers 1 - --agents 1 + # 2 agents (raised from 1) spread the per-test pods across more worker + # nodes so the controller/Gitea/Flux/Redis aren't all contending on a + # single agent under Ginkgo parallelism (E2E_GINKGO_PROCS>1). This was + # 3 historically but dropped to 1 when 1+3 nodes exhausted the default + # fs.inotify.max_user_instances (128); ensure_inotify_limits now raises + # that to 512, leaving ample headroom for the extra node. + --agents 2 --kubeconfig-update-default --kubeconfig-switch-context -v "${audit_host_dir}:/etc/kubernetes/audit@server:0" From 452961c3da3ebd8043772fe3a324c509e52f5d3b Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 29 May 2026 16:40:40 +0000 Subject: [PATCH 13/40] =?UTF-8?q?chore:=20e2e=20=E2=80=94=20widen=20signin?= =?UTF-8?q?g=20commit-wait=20timeouts=2060s=E2=86=9290s=20for=20parallel?= =?UTF-8?q?=20load?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Commit Signing specs do the slowest per-event work (commit + SSH sign + push). Under Ginkgo parallelism the shared controller occasionally pushed the per-event commit past the explicit 60s waits (seen once on a contended 2-agent run). Raise the three explicit 60s waits to 90s to match the file's default and the verifyResourceStatus readiness window. Co-Authored-By: Claude Opus 4.8 --- test/e2e/signing_e2e_test.go | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/test/e2e/signing_e2e_test.go b/test/e2e/signing_e2e_test.go index 73cc95060..8827f71b8 100644 --- a/test/e2e/signing_e2e_test.go +++ b/test/e2e/signing_e2e_test.go @@ -184,7 +184,7 @@ var _ = Describe("Commit Signing", Label("signing"), Ordered, func() { g.Expect(rawObj).To(ContainSubstring("-----BEGIN SSH SIGNATURE-----"), "commit should carry an SSH signature") commitHash = h - }, "60s", "3s").Should(Succeed()) + }, "90s", "3s").Should(Succeed()) By("verifying the commit locally with ssh-keygen and git verify-commit") assertLocalSSHVerification(signingRepo.CheckoutDir, commitHash, signingPublicKey, committerEmail) @@ -306,7 +306,7 @@ var _ = Describe("Commit Signing", Label("signing"), Ordered, func() { g.Expect(rawErr).NotTo(HaveOccurred()) g.Expect(rawObj).To(ContainSubstring("-----BEGIN SSH SIGNATURE-----")) commitHash = h - }, "60s", "3s").Should(Succeed()) + }, "90s", "3s").Should(Succeed()) By("verifying the BYOK commit locally") assertLocalSSHVerification(signingRepo.CheckoutDir, commitHash, publicKeyStr, committerEmail) @@ -387,7 +387,10 @@ var _ = Describe("Commit Signing", Label("signing"), Ordered, func() { g.Expect(parts[1]).To(Equal(customEmail), "committer email should match configured value") g.Expect(parts[2]).To(HavePrefix("e2e:"), "commit subject should use custom template prefix") g.Expect(parts[2]).To(ContainSubstring("configmaps"), "commit subject should include resource type") - }, "60s", "3s").Should(Succeed()) + // 90s: signing does the slowest per-event work (commit + SSH sign + + // push); under Ginkgo parallelism the shared controller can push this + // past 60s on a contended run. + }, "90s", "3s").Should(Succeed()) }) // ── Test 4: batch/atomic commit uses batch message template ───────────── From b7a00b48486e27b292ad7c08618757239bb14706 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 29 May 2026 20:32:49 +0000 Subject: [PATCH 14/40] =?UTF-8?q?chore:=20e2e=20=E2=80=94=20raise=20k3d=20?= =?UTF-8?q?agents=202=E2=86=923=20to=20reduce=20controller=20CPU=20starvat?= =?UTF-8?q?ion?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit procs=2 on stock GitHub runners still intermittently failed the Commit Signing spec even at a 90s wait: the controller's CPU-heavy SSH-signing + git work gets starved when co-located with a heavy test pod. 2 agents helped (one green run) but didn't eliminate it. Go back to 3 agents (the original count, safe now that ensure_inotify_limits raises fs.inotify.max_user_instances to 512) to spread pods further and give the controller more isolated CPU. Co-Authored-By: Claude Opus 4.8 --- test/e2e/cluster/start-cluster.sh | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/test/e2e/cluster/start-cluster.sh b/test/e2e/cluster/start-cluster.sh index a5a6426e0..9925334dd 100644 --- a/test/e2e/cluster/start-cluster.sh +++ b/test/e2e/cluster/start-cluster.sh @@ -172,13 +172,14 @@ create_cluster() { k3d cluster create "${CLUSTER_NAME}" --image rancher/k3s:v1.35.2-k3s1 --servers 1 - # 2 agents (raised from 1) spread the per-test pods across more worker - # nodes so the controller/Gitea/Flux/Redis aren't all contending on a - # single agent under Ginkgo parallelism (E2E_GINKGO_PROCS>1). This was - # 3 historically but dropped to 1 when 1+3 nodes exhausted the default - # fs.inotify.max_user_instances (128); ensure_inotify_limits now raises - # that to 512, leaving ample headroom for the extra node. - --agents 2 + # 3 agents spread the per-test pods across more worker nodes so the + # controller (which does CPU-heavy SSH signing + git work) is less likely + # to be co-located with — and starved by — a heavy test pod under Ginkgo + # parallelism (E2E_GINKGO_PROCS>1). This was dropped to 1 when 1+3 nodes + # exhausted the default fs.inotify.max_user_instances (128); the cause is + # now mitigated by ensure_inotify_limits raising that to 512, which leaves + # ample headroom for 4 nodes. + --agents 3 --kubeconfig-update-default --kubeconfig-switch-context -v "${audit_host_dir}:/etc/kubernetes/audit@server:0" From b285ac5e84a936143f3040e5f9b66a7de8b36764 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 29 May 2026 20:47:07 +0000 Subject: [PATCH 15/40] docs: Phase 2.5 status + Monday resume plan (3-agent CI green once; stability TBD) Co-Authored-By: Claude Opus 4.8 --- docs/design/e2e-speedup-plan.md | 108 +++++++++++++++++++++++++------- 1 file changed, 84 insertions(+), 24 deletions(-) diff --git a/docs/design/e2e-speedup-plan.md b/docs/design/e2e-speedup-plan.md index c66c9afc1..af6442ac0 100644 --- a/docs/design/e2e-speedup-plan.md +++ b/docs/design/e2e-speedup-plan.md @@ -418,39 +418,99 @@ registry then only needs to capture genuinely process-level conflicts - Smoke wallclock drops by at least the parallel-safe Group A+B+C share. - Serial registry exists and is non-empty. -**Implementation status (2026-05-29).** Landed on branch `refactors`. +**Implementation status & where we are (2026-05-29 — read this first on resume).** +Landed on branch `refactors` (PR #159). `gh` works directly now — no +`unset GH_TOKEN` needed. + +**What is implemented (and green locally at `procs=2`):** - **Parallelism is driven by the `ginkgo` CLI, not `go test`.** `go test` has no `-ginkgo.procs` flag (it only accepts the `-ginkgo.parallel.*` - flags that the CLI sets on its workers). The three suite tasks now run + flags the CLI sets on its workers). `test-e2e`, `test-e2e-full` and + `test-e2e-manager` now run `go run github.com/onsi/ginkgo/v2/ginkgo --procs={{.E2E_GINKGO_PROCS}}` (pinned to the module version to avoid a CLI/library mismatch). `E2E_GINKGO_PROCS` defaults to `2`. - **`SynchronizedBeforeSuite`** runs `prepare-e2e` + CRD pre-cleanup once on process #1; the `E2E_AGE_KEY_FILE` fallback runs per-process. -- **CRD isolation:** each of `crd_lifecycle`, `restart_snapshot`, - `bi_directional` renders its own group (`*.e2e.example.com`) via - `test/e2e/icecream.go`; the shared `icecreamorders` plural means every +- **Per-file CRD group isolation** via `test/e2e/icecream.go`: + `crd_lifecycle`/`restart_snapshot`/`bi_directional` each own a group + (`*.e2e.example.com`); the shared `icecreamorders` plural means every kubectl reference is qualified `icecreamorders.`. -- **Final Serial set** (see [e2e-serial-registry.md](e2e-serial-registry.md)) - ended up larger than the initial audit. Beyond `restart_snapshot`, - `image_refresh` and `aggregated_api` (controller/APIService level), the - four `audit-redis`-labelled containers (`Audit Redis Queue/Consumer`, - `Commit Window Batching`, `Commit Request`) had to go Serial — they - share one global audit pipeline (webhook → Redis stream → consumer) and - cross-contaminated each other's commits. `bi_directional` is Serial too: - it asserts exact commit counts to prove no commit loop, which any - concurrent controller activity breaks. -- **Timeouts widened for parallel/slow-CI load:** the shared - `verifyResourceStatus` readiness wait went 30s → 90s (after a fresh CRD - install the controller's discovery cache lags before it serves the new - GVR, so a dependent WatchRule can't reach Ready inside 30s on contended - runners); the manager ConfigMap-delete `Eventually` went 30s → 60s. -- **Measured:** smoke (`task test-e2e`, procs=2) **352 s** vs the 418.7 s - sequential baseline. Cold full run (fresh cluster, all 47 specs) green: - 45 passed / 0 failed / 2 skipped in ~20 min wallclock. The dominant - remaining smoke cost is the Serial `Restart Snapshot Safety` spec - (~128 s) — that is exactly what Phase 3 targets. +- **Serial registry** ([e2e-serial-registry.md](e2e-serial-registry.md)) + is larger than the initial audit: `restart_snapshot`, `image_refresh`, + `aggregated_api` (controller/APIService level) **plus** the four + `audit-redis`-labelled containers (`Audit Redis Queue/Consumer`, + `Commit Window Batching`, `Commit Request`) — they share one global + audit pipeline (webhook → Redis → consumer) and cross-contaminated each + other's commits — **plus** `bi_directional` (asserts exact commit counts + to prove no commit loop; broken by any concurrent controller activity). +- **Timeouts widened for parallel/slow load:** `verifyResourceStatus` + readiness 30s → 90s (post-CRD-install the controller's discovery cache + lags before serving the new GVR, so a dependent WatchRule can't reach + Ready in 30s); manager ConfigMap-delete `Eventually` 30s → 60s; signing + commit-waits 60s → 90s. + +**Commits on `refactors` since the `1205369` squash:** `cad9903` (Serial: +bi-directional + audit pipeline), `9942e0c` (readiness 90s), `dba83ac` +(this doc), `1294814` (k3d agents 1→2), `452961c` (signing 90s), `b7a00b4` +(k3d agents 2→3). + +**Measured (local devcontainer, `procs=2`):** smoke **352 s** vs 418.7 s +sequential baseline (~16 %); cold full **45/45** in ~20 min; warm full +**45/45** in ~17 min. Rock solid locally across smoke + 2 full runs. The +full-suite gain is modest *by design*: the heavy specs are `Serial` for +correctness and the single biggest spec, `Restart Snapshot Safety` +(~128 s), is Serial and untouched until Phase 3. Smoke's ~16 % is the +honest gain today; the bigger win is Phase 3. + +**⚠️ Open problem — CI is flaky at `procs=2` (the only thing blocking +merge).** On the stock `ubuntu-latest` GitHub runners the single +controller is **CPU-starved** under two concurrent test streams plus its +own CPU-heavy SSH-signing/git work. Symptom: a *different* spec times out +each run (crd_lifecycle WatchRule readiness; watchrule "create"; signing — +the last failed even at a 90 s wait, i.e. genuine starvation, not a tight +timeout). It is **not** a logic bug — local is consistently green. +Progression tried: +- 1 agent → red. +- **2 agents** (`1294814`) → E2E (full) went **green once**, then **red** + on the next run (`452961c`, signing at 90 s). Reduced but didn't + eliminate. +- **3 agents** (`b7a00b4`) → E2E (full) went **green** (run + `26660720283`). But this is **one** green run, and 2 agents also went + green-once-then-red, so stability is **not yet proven** — needs a few + consecutive green reruns before we trust it. The 3→1 drop was originally + due to `fs.inotify.max_user_instances` (default 128) exhaustion; that is + now mitigated to 512 by `ensure_inotify_limits`, so 1 server + 3 agents + is safe. Confirmed the suite uses a **single** cluster (the + `audit-pass-through-e2e` cluster seen locally is an unrelated leftover). + +**▶ Resume Monday — do this:** + +1. Check the 3-agent result: + `gh pr view 159 --json statusCheckRollup,mergeable`. Look at + **E2E (full)**. +2. **If green:** re-run it 1–2× to confirm it's stable, not lucky + (`gh run rerun ` after it completes, or push a trivial change). + If stable → Phase 2.5 is **done**; merge #159. +3. **If still flaky (likely):** apply the agreed fallback — set + `E2E_GINKGO_PROCS=1` for the **CI E2E (full) job only** in + [.github/workflows/ci.yml](../../.github/workflows/ci.yml) (the `e2e` + job, `full` matrix entry, ~L262–280; add `E2E_GINKGO_PROCS: 1` to that + job/step env or prefix the `task test-e2e-full` script). This keeps the + local/agent `procs=2` speedup while giving CI the reliable sequential + behaviour (pre-2.5). All Phase 2.5 isolation work still ships and is + correct. Then push, confirm green, merge. (Alternative if CI speedup is + wanted: move the e2e job to a larger runner with real CPU for + `procs=2` — needs larger-runner access/quota.) +4. **Consider reverting the agent bump to 2** (`1294814`) and dropping + `b7a00b4` if CI goes to `procs=1`: with sequential CI, 1 agent is + enough and fewer nodes = less runner pressure. Keep the higher agent + count only if CI stays at `procs=2`. +5. Then start **Phase 3** (below) on a *separate* branch — it touches + production controller code (`internal/telemetry` + `internal/watch`) + and cuts the Serial `Restart Snapshot Safety` spec (~128 s, the + dominant remaining cost). ### Phase 3 — Target-scoped reconcile-complete + drain signal for `Restart Snapshot Safety` From 7834c42ff6cf9479ec5464fef375060e24f67cf3 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Sat, 30 May 2026 22:00:20 +0000 Subject: [PATCH 16/40] chore: Would this resolve it? --- .github/workflows/ci.yml | 50 +++++++++++++++++++++++++++ docs/design/e2e-speedup-plan.md | 57 +++++++++++++++---------------- test/e2e/cluster/start-cluster.sh | 48 ++++++++++++++++++++------ 3 files changed, 115 insertions(+), 40 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1f2735e45..8080d9203 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -273,9 +273,13 @@ jobs: && export INSTALL_MODE=plain-manifests-file && task test-e2e-quickstart-manifest needs_artifact: true + e2e_ginkgo_procs: "2" + k3d_agent_count: "1" - name: full script: "task test-e2e-full" needs_artifact: false + e2e_ginkgo_procs: "1" + k3d_agent_count: "1" env: PROJECT_IMAGE: ${{ needs.docker-build.outputs.image }} CI_CONTAINER: ${{ needs.build-ci-container.outputs.image }} @@ -305,6 +309,8 @@ jobs: -w "${{ env.CI_WORKDIR }}" \ -e PROJECT_IMAGE=${{ env.PROJECT_IMAGE }} \ -e IMAGE_DELIVERY_MODE=${{ env.IMAGE_DELIVERY_MODE }} \ + -e E2E_GINKGO_PROCS=${{ matrix.e2e_ginkgo_procs }} \ + -e K3D_AGENT_COUNT=${{ matrix.k3d_agent_count }} \ -e HOST_PROJECT_PATH=${{ github.workspace }} \ ${{ env.CI_CONTAINER }} \ bash -c " @@ -312,6 +318,50 @@ jobs: ${{ matrix.script }} " + - name: Summarize E2E timing reports + if: always() + run: | + summary_file=".stamps/e2e-timing-summary-${{ matrix.name }}.md" + docker run --rm \ + -v "${GITHUB_WORKSPACE}:${{ env.CI_WORKDIR }}" \ + -w "${{ env.CI_WORKDIR }}" \ + -e SUMMARY_FILE="${summary_file}" \ + ${{ env.CI_CONTAINER }} \ + bash -c ' + set -euo pipefail + mkdir -p .stamps + { + echo "### E2E timing reports (${{ matrix.name }})" + reports="$(find .stamps -path "*/ginkgo-report-*.json" -type f | sort)" + if [ -z "${reports}" ]; then + echo + echo "No Ginkgo JSON reports were produced." + exit 0 + fi + while IFS= read -r report; do + [ -n "${report}" ] || continue + echo + echo "#### ${report}" + echo + echo "```" + go run ./test/e2e/tools/spec-timings "${report}" || true + echo "```" + done <<<"${reports}" + } > "${SUMMARY_FILE}" + ' + cat "${summary_file}" >> "${GITHUB_STEP_SUMMARY}" + + - name: Upload E2E timing reports + if: always() + uses: actions/upload-artifact@v7 + with: + name: e2e-ginkgo-reports-${{ matrix.name }} + path: | + .stamps/**/ginkgo-report-*.json + .stamps/e2e-timing-summary-${{ matrix.name }}.md + if-no-files-found: ignore + retention-days: 7 + # Release job only runs on push to main after tests pass release-please: name: Release Please diff --git a/docs/design/e2e-speedup-plan.md b/docs/design/e2e-speedup-plan.md index af6442ac0..71f38a5db 100644 --- a/docs/design/e2e-speedup-plan.md +++ b/docs/design/e2e-speedup-plan.md @@ -464,13 +464,32 @@ correctness and the single biggest spec, `Restart Snapshot Safety` (~128 s), is Serial and untouched until Phase 3. Smoke's ~16 % is the honest gain today; the bigger win is Phase 3. -**⚠️ Open problem — CI is flaky at `procs=2` (the only thing blocking -merge).** On the stock `ubuntu-latest` GitHub runners the single +**Resolution for merge — CI full is sequential, local remains parallel +(2026-05-30).** On the stock `ubuntu-latest` GitHub runners the single controller is **CPU-starved** under two concurrent test streams plus its own CPU-heavy SSH-signing/git work. Symptom: a *different* spec times out each run (crd_lifecycle WatchRule readiness; watchrule "create"; signing — the last failed even at a 90 s wait, i.e. genuine starvation, not a tight -timeout). It is **not** a logic bug — local is consistently green. +timeout). The follow-up run after 3 agents also failed during setup while +waiting for `helmrelease/kro`, so the blocking CI signal is broader than +spec logic: runner/resource pressure plus external chart/registry +readiness. + +GitHub API throttling was checked with `gh api rate_limit` and was not the +limiting factor (4967/5000 core requests remaining at review time). The +workflow now uploads any Ginkgo JSON reports and writes a per-spec timing +summary to the GitHub job summary so build-server timings are visible. + +Applied fallback: + +- CI **E2E (full)** sets `E2E_GINKGO_PROCS=1`. +- Local/default `task test-e2e*` remains `E2E_GINKGO_PROCS=2`. +- k3d local/default agents remain 3 for `procs=2`; CI passes + `K3D_AGENT_COUNT=1` because the full job is sequential. +- Warm k3d cluster reuse now verifies the expected node count and Ready + condition instead of trusting `kubectl get ns` plus stamps. + +History that led here: Progression tried: - 1 agent → red. - **2 agents** (`1294814`) → E2E (full) went **green once**, then **red** @@ -482,35 +501,13 @@ Progression tried: consecutive green reruns before we trust it. The 3→1 drop was originally due to `fs.inotify.max_user_instances` (default 128) exhaustion; that is now mitigated to 512 by `ensure_inotify_limits`, so 1 server + 3 agents - is safe. Confirmed the suite uses a **single** cluster (the + is possible but no longer the CI default. Confirmed the suite uses a **single** cluster (the `audit-pass-through-e2e` cluster seen locally is an unrelated leftover). -**▶ Resume Monday — do this:** - -1. Check the 3-agent result: - `gh pr view 159 --json statusCheckRollup,mergeable`. Look at - **E2E (full)**. -2. **If green:** re-run it 1–2× to confirm it's stable, not lucky - (`gh run rerun ` after it completes, or push a trivial change). - If stable → Phase 2.5 is **done**; merge #159. -3. **If still flaky (likely):** apply the agreed fallback — set - `E2E_GINKGO_PROCS=1` for the **CI E2E (full) job only** in - [.github/workflows/ci.yml](../../.github/workflows/ci.yml) (the `e2e` - job, `full` matrix entry, ~L262–280; add `E2E_GINKGO_PROCS: 1` to that - job/step env or prefix the `task test-e2e-full` script). This keeps the - local/agent `procs=2` speedup while giving CI the reliable sequential - behaviour (pre-2.5). All Phase 2.5 isolation work still ships and is - correct. Then push, confirm green, merge. (Alternative if CI speedup is - wanted: move the e2e job to a larger runner with real CPU for - `procs=2` — needs larger-runner access/quota.) -4. **Consider reverting the agent bump to 2** (`1294814`) and dropping - `b7a00b4` if CI goes to `procs=1`: with sequential CI, 1 agent is - enough and fewer nodes = less runner pressure. Keep the higher agent - count only if CI stays at `procs=2`. -5. Then start **Phase 3** (below) on a *separate* branch — it touches - production controller code (`internal/telemetry` + `internal/watch`) - and cuts the Serial `Restart Snapshot Safety` spec (~128 s, the - dominant remaining cost). +**Next after merge:** start **Phase 3** (below) on a *separate* branch — +it touches production controller code (`internal/telemetry` + +`internal/watch`) and cuts the Serial `Restart Snapshot Safety` spec +(~128 s, the dominant remaining cost). ### Phase 3 — Target-scoped reconcile-complete + drain signal for `Restart Snapshot Safety` diff --git a/test/e2e/cluster/start-cluster.sh b/test/e2e/cluster/start-cluster.sh index 9925334dd..1a0df850f 100644 --- a/test/e2e/cluster/start-cluster.sh +++ b/test/e2e/cluster/start-cluster.sh @@ -11,6 +11,7 @@ DISABLE_K3S_TRAEFIK="${DISABLE_K3S_TRAEFIK:-true}" DISABLE_K3S_SERVICELB="${DISABLE_K3S_SERVICELB:-true}" KUBE_APISERVER_MAX_REQUESTS_INFLIGHT="${KUBE_APISERVER_MAX_REQUESTS_INFLIGHT:-800}" KUBE_APISERVER_MAX_MUTATING_REQUESTS_INFLIGHT="${KUBE_APISERVER_MAX_MUTATING_REQUESTS_INFLIGHT:-400}" +K3D_AGENT_COUNT="${K3D_AGENT_COUNT:-3}" AUDIT_DIR_REL="${AUDIT_DIR_REL:-test/e2e/cluster/audit}" K3D_CREATE_LOG_FILE="${TMPDIR:-/tmp}/k3d-create-${CLUSTER_NAME}.log" REPO_PWD="$(pwd -P)" @@ -74,7 +75,41 @@ cluster_is_healthy() { return 1 fi - kubectl --context "${context_name}" --request-timeout=10s get ns >/dev/null 2>&1 + kubectl --context "${context_name}" --request-timeout=10s get ns >/dev/null 2>&1 || return 1 + + local expected_nodes + expected_nodes="$((K3D_AGENT_COUNT + 1))" + kubectl --context "${context_name}" --request-timeout=10s get nodes \ + -o jsonpath='{range .items[*]}{.metadata.name}{" "}{range .status.conditions[?(@.type=="Ready")]}{.status}{end}{"\n"}{end}' \ + 2>/dev/null \ + | awk -v expected="${expected_nodes}" ' + BEGIN { total = 0; ready = 0 } + NF >= 2 { + total++ + if ($2 == "True") { + ready++ + } + } + END { + exit !(total == expected && ready == expected) + } + ' +} + +wait_cluster_healthy() { + local context_name attempt + context_name="$(cluster_context_name)" + + for attempt in $(seq 1 60); do + if cluster_is_healthy; then + return 0 + fi + sleep 2 + done + + echo "Current node status for ${context_name}:" >&2 + kubectl --context "${context_name}" get nodes -o wide >&2 || true + return 1 } ensure_k3d_stat_compat_path() { local host_project_path="$1" @@ -172,14 +207,7 @@ create_cluster() { k3d cluster create "${CLUSTER_NAME}" --image rancher/k3s:v1.35.2-k3s1 --servers 1 - # 3 agents spread the per-test pods across more worker nodes so the - # controller (which does CPU-heavy SSH signing + git work) is less likely - # to be co-located with — and starved by — a heavy test pod under Ginkgo - # parallelism (E2E_GINKGO_PROCS>1). This was dropped to 1 when 1+3 nodes - # exhausted the default fs.inotify.max_user_instances (128); the cause is - # now mitigated by ensure_inotify_limits raising that to 512, which leaves - # ample headroom for 4 nodes. - --agents 3 + --agents "${K3D_AGENT_COUNT}" --kubeconfig-update-default --kubeconfig-switch-context -v "${audit_host_dir}:/etc/kubernetes/audit@server:0" @@ -270,7 +298,7 @@ main() { merge_kubeconfig rewrite_kubeconfig_for_devcontainer - if ! cluster_is_healthy; then + if ! wait_cluster_healthy; then echo "❌ Existing k3d cluster '${CLUSTER_NAME}' is unhealthy." >&2 echo "Refusing to auto-recreate it because cluster-scoped stamps may now be inconsistent." >&2 echo "" >&2 From d531c204988dba65c8cb4889b56702df4867cc31 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Sun, 31 May 2026 07:57:25 +0000 Subject: [PATCH 17/40] chore: should it be serial as well? --- docs/design/e2e-serial-registry.md | 12 +----------- docs/design/e2e-speedup-plan.md | 15 ++++++++++----- test/e2e/crd_lifecycle_e2e_test.go | 2 +- 3 files changed, 12 insertions(+), 17 deletions(-) diff --git a/docs/design/e2e-serial-registry.md b/docs/design/e2e-serial-registry.md index 5a4a5373a..cf341808f 100644 --- a/docs/design/e2e-serial-registry.md +++ b/docs/design/e2e-serial-registry.md @@ -29,17 +29,7 @@ row below. | [test/e2e/commit_window_batching_e2e_test.go](../../test/e2e/commit_window_batching_e2e_test.go) | `Commit Window Batching` | Same global audit pipeline (labelled `audit-redis`). | Bursts of ConfigMap events flow through the shared stream/consumer and leak into other audit specs' commits. | | [test/e2e/commit_request_e2e_test.go](../../test/e2e/commit_request_e2e_test.go) | `Commit Request` | Same global audit pipeline (labelled `audit-redis`). | Shares the stream/consumer; commit-window/queue semantics interleave under concurrency. | | [test/e2e/bi_directional_e2e_test.go](../../test/e2e/bi_directional_e2e_test.go) | `Bi Directional` | Whole-cluster Flux↔gitops-reverser round-trip; asserts on **exact** remote commit counts to prove no commit loop. | Any concurrent controller activity adds/reorders commits and breaks the exact-count loop assertions. Passes sequentially; +2 commits only under parallelism. | - -## Watched, but intentionally NOT Serial - -- **`crd_lifecycle_e2e_test.go`** installs a `ClusterWatchRule` that watches all - CRDs cluster-wide (`ClusterResourceRule` has no per-object-name filter, see - [api/v1alpha1/clusterwatchrule_types.go](../../api/v1alpha1/clusterwatchrule_types.go)). - While that rule is live it mirrors *other* files' CRD definitions into its own - repo. This is harmless: the rule is only active during the two CRD-lifecycle - specs (install/delete), it is torn down before the "no new commit" assertion, - and every assertion checks a *specific*, group-scoped file path. Extra mirrored - files are cosmetic. Kept parallel; revisit if the stability window shows flakes. +| [test/e2e/crd_lifecycle_e2e_test.go](../../test/e2e/crd_lifecycle_e2e_test.go) | `Manager CRD Lifecycle` | Installs/deletes a CRD and a wildcard `ClusterWatchRule`, changing cluster-wide discovery/GVR catalog state. | Per-file CRD groups prevent name collisions, but discovery changes still force unrelated GitTargets to resnapshot. That can hide exact `[CREATE]`/`[DELETE]` event-commit assertions in other WatchRule specs behind later `reconcile: sync ...` commits. | All four `audit-redis`-labelled containers above were moved to `Serial` after the first procs=2 smoke run: `Audit Redis Consumer` asserted its commit touched only diff --git a/docs/design/e2e-speedup-plan.md b/docs/design/e2e-speedup-plan.md index 71f38a5db..501726602 100644 --- a/docs/design/e2e-speedup-plan.md +++ b/docs/design/e2e-speedup-plan.md @@ -444,7 +444,10 @@ Landed on branch `refactors` (PR #159). `gh` works directly now — no `Commit Window Batching`, `Commit Request`) — they share one global audit pipeline (webhook → Redis → consumer) and cross-contaminated each other's commits — **plus** `bi_directional` (asserts exact commit counts - to prove no commit loop; broken by any concurrent controller activity). + to prove no commit loop; broken by any concurrent controller activity) + and `crd_lifecycle` (CRD/ClusterWatchRule changes are name-isolated, but + still perturb cluster discovery and can trigger unrelated GitTarget + resnapshots). - **Timeouts widened for parallel/slow load:** `verifyResourceStatus` readiness 30s → 90s (post-CRD-install the controller's discovery cache lags before serving the new GVR, so a dependent WatchRule can't reach @@ -579,10 +582,12 @@ the existing ones. - **Metric package is `internal/telemetry`.** Confirmed against `internal/telemetry/exporter.go`. No `internal/metrics` directory exists. -- **CRD isolation over Serial marking.** Where multiple e2e files - install the same cluster-scoped CRD, the cheap fix is to give each - file its own CRD group (Phase 2.5). Serial marking is reserved for - conflicts that name-isolation can't fix (e.g. controller restart). +- **CRD isolation before Serial marking.** Where multiple e2e files + install the same cluster-scoped CRD, the cheap first fix is to give + each file its own CRD group (Phase 2.5). Serial marking is still + required when the remaining cluster-wide side effect cannot be + isolated by name, such as controller restarts or CRD discovery/GVR + catalog churn. ## Reproducing the baseline diff --git a/test/e2e/crd_lifecycle_e2e_test.go b/test/e2e/crd_lifecycle_e2e_test.go index 9c475c35b..5653bc176 100644 --- a/test/e2e/crd_lifecycle_e2e_test.go +++ b/test/e2e/crd_lifecycle_e2e_test.go @@ -29,7 +29,7 @@ import ( . "github.com/onsi/gomega" ) -var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { +var _ = Describe("Manager CRD Lifecycle", Label("manager"), Serial, Ordered, func() { var ( testNs string crdLifecycleRepo *RepoArtifacts From d74a1e3fa37912b34bb59e420ce980fafe26bc84 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Sun, 31 May 2026 17:55:42 +0000 Subject: [PATCH 18/40] chore: round 1 --- docs/design/e2e-speedup-plan.md | 64 ++++++++++- internal/git/branch_worker.go | 49 +++++++++ internal/git/branch_worker_metrics_test.go | 104 ++++++++++++++++++ internal/telemetry/exporter.go | 24 ++++ internal/watch/manager.go | 20 ++++ .../watch/target_reconcile_metric_test.go | 56 ++++++++++ test/e2e/restart_snapshot_e2e_test.go | 50 ++++++++- 7 files changed, 356 insertions(+), 11 deletions(-) create mode 100644 internal/git/branch_worker_metrics_test.go create mode 100644 internal/watch/target_reconcile_metric_test.go diff --git a/docs/design/e2e-speedup-plan.md b/docs/design/e2e-speedup-plan.md index 501726602..9d16cee5f 100644 --- a/docs/design/e2e-speedup-plan.md +++ b/docs/design/e2e-speedup-plan.md @@ -530,11 +530,11 @@ that the pod started. **Change.** Pair two signals: -1. **`gitopsreverser_target_initial_reconcile_complete{namespace,name}`** +1. **`gitopsreverser_target_initial_reconcile_complete{gittarget_namespace,gittarget_name}`** — gauge per GitTarget, flips to `1` once its post-(re)start initial reconcile pass has completed end-to-end (snapshot decision made, queue submission completed). -2. **`gitopsreverser_branch_worker_queue_depth{namespace,name,branch}`** +2. **`gitopsreverser_branch_worker_queue_depth{provider_namespace,provider_name,branch}`** — gauge of pending work items for this branch worker. Wait for `0` to confirm the queue has drained. @@ -542,15 +542,32 @@ The metric package is `internal/telemetry/` (not `internal/metrics/` — see `internal/telemetry/exporter.go`); add the new metrics alongside the existing ones. +> **Label naming — do not use `namespace`/`name`.** The e2e Prometheus +> scrapes the controller as a pod target with `honor_labels: false` +> ([test/e2e/setup/manifests/prometheus/servicemonitor.yaml](../../test/e2e/setup/manifests/prometheus/servicemonitor.yaml)). +> A pod scrape attaches the scraped pod's own `namespace`/`pod`/`job` +> target labels; with honor disabled, any metric attribute literally named +> `namespace` is overwritten and the original is moved to +> `exported_namespace`. A `{namespace=...}` selector would then silently +> match nothing. Hence the prefixed keys above +> (`gittarget_namespace`, `provider_namespace`, …), matching the existing +> convention where every gitopsreverser metric avoids bare `namespace`. + **Restart test reframe:** 1. Rollout-restart the controller deployment. -2. Wait for `target_initial_reconcile_complete{ns,name} == 1` for the - restart test's GitTarget. -3. Wait for `branch_worker_queue_depth{ns,name,branch} == 0`. +2. Wait for `max(target_initial_reconcile_complete{gittarget_*}) == 1` + for the restart test's GitTarget. +3. Wait for `sum(branch_worker_queue_depth{provider_*,branch}) == 0`. 4. Apply a short stability window (a few seconds). 5. Assert no destructive commit happened. +The `max()`/`sum()` aggregation in steps 2–3 collapses the brief window +during a rollout where Prometheus still holds the old pod's last sample +(distinct `pod` target label) next to the new pod's, so the assertion sees +the freshly started pod's value rather than whichever series the query +returns first. + **Risks:** - The new metrics' contracts must be exact enough to be load-bearing in @@ -559,7 +576,12 @@ the existing ones. rollouts. - Worker-queue-depth must include in-flight items, not just queued ones, or step 3 races with the worker. Verify against the actual - implementation when writing the metric. + implementation when writing the metric. **Implemented:** the event loop + republishes the gauge once per iteration from its authoritative state + (`openWindow != nil || len(pendingWrites) > 0`, surfaced via an + `atomic.Bool` the enqueue paths also re-read), so the gauge counts the + open commit window and retained-for-replay writes — not just channel + depth — and only reaches 0 after a successful push clears `pendingWrites`. **Exit criterion:** @@ -567,6 +589,36 @@ the existing ones. - Both metrics are scrapable via the existing Prometheus pipeline. - Suite still green; no regressions in other restart-related specs. +**Implementation status (2026-05-31).** Landed on branch `refactors`. + +- New gauges in [internal/telemetry/exporter.go](../../internal/telemetry/exporter.go): + `gitopsreverser_target_initial_reconcile_complete` (set in + `emitSnapshotForRuleChange`'s per-target success branch, + [internal/watch/manager.go](../../internal/watch/manager.go)) and + `gitopsreverser_branch_worker_queue_depth` (recorded from the branch + worker event loop + enqueue paths, + [internal/git/branch_worker.go](../../internal/git/branch_worker.go)). +- Unit tests: + [internal/git/branch_worker_metrics_test.go](../../internal/git/branch_worker_metrics_test.go), + [internal/watch/target_reconcile_metric_test.go](../../internal/watch/target_reconcile_metric_test.go). +- **First e2e run failed** — the query used a bare `namespace` label and + matched nothing (the honor_labels collision above). Renaming to the + prefixed keys and adding `max()`/`sum()` aggregation fixed it. The + controller-side delivery was correct on the first run (7 snapshot emits, + zero "No reconciler registered"); the bug was purely in the metric's + label contract. +- **Measured:** restart spec `Restart Snapshot Safety` **57.0 s** total + wallclock (down from ~128 s), full smoke suite green (20/20, 444 s vs + the earlier 537 s run that included the failing attempt). Both metric + waits resolved within the first poll after the rollout settled — the + remaining 57 s is BeforeAll setup (Prometheus client, Gitea repo, CRD + install, building the mirror, the rollout restart itself) plus the 15 s + stability window, **not** the old 75 s blind negative-assertion wait. + The signal-driven wait does not hit the <30 s exit-criterion target on + its own because the per-spec figure is dominated by fixed setup, not the + drain wait that Phase 3 removed; the drain wait itself is now ~0 s. + Tightening the residual setup/stability cost is follow-up work. + ## Decisions - **File naming.** Files use the form `_e2e_test.go`. The CRD diff --git a/internal/git/branch_worker.go b/internal/git/branch_worker.go index 5dda18c21..d5b0bba62 100644 --- a/internal/git/branch_worker.go +++ b/internal/git/branch_worker.go @@ -28,12 +28,15 @@ import ( "path/filepath" "strings" "sync" + "sync/atomic" "time" gogit "github.com/go-git/go-git/v5" "github.com/go-git/go-git/v5/plumbing" "github.com/go-git/go-git/v5/plumbing/transport" "github.com/go-logr/logr" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" "k8s.io/apimachinery/pkg/types" "sigs.k8s.io/controller-runtime/pkg/client" @@ -124,6 +127,13 @@ type BranchWorker struct { // firsts surfaces the first successful commit and push at default verbosity. firsts branchWorkerLogFirsts + + // hasUnpushedWork mirrors whether the event loop is currently holding a live + // open window or any committed-but-not-yet-pushed pending writes. The event + // loop is the only writer; recordQueueDepth (callable from enqueue goroutines) + // is the only other reader, hence the atomic. It lets the depth gauge account + // for in-flight work the channel length alone cannot see. + hasUnpushedWork atomic.Bool } // branchWorkerLogFirsts logs the first successful commit and push of a worker's @@ -249,6 +259,7 @@ func (w *BranchWorker) EnqueueFinalize(signal *FinalizeSignal) { select { case w.eventQueue <- WorkItem{Finalize: signal}: w.Log.V(1).Info("Finalize signal enqueued") + w.recordQueueDepth() default: w.Log.Error(nil, "Event queue full, finalize signal dropped") signal.reply(FinalizeResult{Branch: w.Branch, Err: ErrFinalizeQueueFull}) @@ -266,6 +277,7 @@ func (w *BranchWorker) enqueueRequest(request *WriteRequest) { "events", len(request.Events), "mode", request.CommitMode, "gitTarget", request.GitTargetName) + w.recordQueueDepth() default: w.Log.Error(nil, "Event queue full, request dropped", "events", len(request.Events), @@ -274,6 +286,31 @@ func (w *BranchWorker) enqueueRequest(request *WriteRequest) { } } +// recordQueueDepth publishes the current pending-work depth for this worker: +// the number of items still sitting in the event queue plus one when the event +// loop is holding unpushed work (a live open window or retained pending +// writes). It reads 0 only when the worker has fully drained. Safe to call from +// any goroutine: it reads the channel length and an atomic flag. No-op until the +// gauge is registered (e.g. in unit tests that never init the exporter). +func (w *BranchWorker) recordQueueDepth() { + if telemetry.BranchWorkerQueueDepth == nil { + return + } + depth := int64(len(w.eventQueue)) + if w.hasUnpushedWork.Load() { + depth++ + } + ctx := w.ctx + if ctx == nil { + ctx = context.Background() + } + telemetry.BranchWorkerQueueDepth.Record(ctx, depth, metric.WithAttributes( + attribute.String("provider_namespace", w.GitProviderNamespace), + attribute.String("provider_name", w.GitProviderRef), + attribute.String("branch", w.Branch), + )) +} + // ListResourcesInPath returns resource identifiers found in a Git folder. // This is a synchronous service method called by EventRouter. func (w *BranchWorker) ListResourcesInPath(path string) ([]itypes.ResourceIdentifier, error) { @@ -521,11 +558,13 @@ func newBranchWorkerEventLoop(w *BranchWorker, commitWindow time.Duration) *bran func (l *branchWorkerEventLoop) run() { defer l.stopTimers() + l.syncQueueDepthMetric() for { commitC, pushC := l.timerChannels() select { case <-l.w.ctx.Done(): l.handleShutdown() + l.syncQueueDepthMetric() return case item := <-l.w.eventQueue: l.handleQueueItem(item) @@ -537,9 +576,19 @@ func (l *branchWorkerEventLoop) run() { l.pushTimer = nil l.pushPending() } + l.syncQueueDepthMetric() } } +// syncQueueDepthMetric refreshes the worker's unpushed-work flag from the loop's +// authoritative state and republishes the depth gauge. Called once per loop +// iteration (the loop goroutine is the only writer of hasUnpushedWork), so the +// gauge converges to 0 once the queue is empty and nothing is retained. +func (l *branchWorkerEventLoop) syncQueueDepthMetric() { + l.w.hasUnpushedWork.Store(l.openWindow != nil || len(l.pendingWrites) > 0) + l.w.recordQueueDepth() +} + func (l *branchWorkerEventLoop) timerChannels() (<-chan time.Time, <-chan time.Time) { var commitC, pushC <-chan time.Time if l.commitTimer != nil { diff --git a/internal/git/branch_worker_metrics_test.go b/internal/git/branch_worker_metrics_test.go new file mode 100644 index 000000000..f5d7e2029 --- /dev/null +++ b/internal/git/branch_worker_metrics_test.go @@ -0,0 +1,104 @@ +/* +SPDX-License-Identifier: Apache-2.0 + +Copyright 2025 ConfigButler + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package git + +import ( + "context" + "testing" + + "github.com/go-logr/logr" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/ConfigButler/gitops-reverser/internal/telemetry" + itypes "github.com/ConfigButler/gitops-reverser/internal/types" +) + +const branchWorkerQueueDepthMetric = "gitopsreverser_branch_worker_queue_depth" + +func newMetricsTestWorker() *BranchWorker { + return &BranchWorker{ + GitProviderRef: "test-provider", + GitProviderNamespace: "test-ns", + Branch: "main", + Log: logr.Discard(), + ctx: context.Background(), + contentWriter: newContentWriter(itypes.SensitiveResourcePolicy{}), + eventQueue: make(chan WorkItem, branchWorkerQueueSize), + branchBufferMaxBytes: DefaultBranchBufferMaxBytes, + } +} + +func queueDepthLabels() map[string]string { + return map[string]string{ + "provider_namespace": "test-ns", + "provider_name": "test-provider", + "branch": "main", + } +} + +// recordQueueDepth must report 0 for a freshly drained worker: empty queue and +// no retained unpushed work. +func TestRecordQueueDepth_DrainedReportsZero(t *testing.T) { + reader, err := telemetry.InitTestExporter() + require.NoError(t, err) + + w := newMetricsTestWorker() + w.recordQueueDepth() + + depth, ok := telemetry.CollectInt64Sum(reader, branchWorkerQueueDepthMetric, queueDepthLabels()) + require.True(t, ok, "expected a branch_worker_queue_depth sample") + assert.Equal(t, int64(0), depth) +} + +// recordQueueDepth must count both queued items and the in-flight unpushed-work +// flag, so the gauge reflects work the channel length alone cannot see. +func TestRecordQueueDepth_CountsQueuedAndInFlight(t *testing.T) { + reader, err := telemetry.InitTestExporter() + require.NoError(t, err) + + w := newMetricsTestWorker() + // Two items sitting in the queue, plus retained unpushed work. + w.eventQueue <- WorkItem{} + w.eventQueue <- WorkItem{} + w.hasUnpushedWork.Store(true) + + w.recordQueueDepth() + + depth, ok := telemetry.CollectInt64Sum(reader, branchWorkerQueueDepthMetric, queueDepthLabels()) + require.True(t, ok, "expected a branch_worker_queue_depth sample") + assert.Equal(t, int64(3), depth) +} + +// enqueueRequest must publish a non-zero depth as soon as the item lands on the +// queue, closing the race where a poller reads a stale drained value. +func TestEnqueueRequest_RecordsDepth(t *testing.T) { + reader, err := telemetry.InitTestExporter() + require.NoError(t, err) + + w := newMetricsTestWorker() + w.enqueueRequest(&WriteRequest{ + Events: []Event{{}}, + CommitMode: CommitModePerEvent, + }) + + depth, ok := telemetry.CollectInt64Sum(reader, branchWorkerQueueDepthMetric, queueDepthLabels()) + require.True(t, ok, "expected a branch_worker_queue_depth sample after enqueue") + assert.Equal(t, int64(1), depth) +} diff --git a/internal/telemetry/exporter.go b/internal/telemetry/exporter.go index 487ca3d8c..c16119bba 100644 --- a/internal/telemetry/exporter.go +++ b/internal/telemetry/exporter.go @@ -62,6 +62,28 @@ var ( // RepoBranchQueueDepth is a gauge for per-repo-branch queue depth. RepoBranchQueueDepth metric.Int64UpDownCounter + // TargetInitialReconcileComplete latches to 1 once a GitTarget's post-(re)start + // initial reconcile pass has completed end-to-end in this process: the snapshot + // decision was made and its write request was submitted to the branch worker + // queue. Labelled by {gittarget_namespace, gittarget_name}. The labels avoid + // the reserved `namespace`/`name` keys on purpose: a Prometheus pod scrape with + // honor_labels=false overwrites a metric's `namespace` attribute with the + // scraped pod's own namespace, so a per-GitTarget `namespace` label would be + // silently unqueryable. It is a per-process latch — a controller restart starts + // a fresh process whose gauge is unset until the first snapshot delivery sets + // it. Load-bearing for the restart-snapshot e2e spec; treat the name/labels as + // a public observability contract. + TargetInitialReconcileComplete metric.Int64Gauge + // BranchWorkerQueueDepth gauges pending work for a single branch worker: + // queued work items plus any committed-but-not-yet-pushed work the worker is + // still holding. It reads 0 only when the worker has fully drained (empty + // queue and nothing retained for replay). Labelled by {provider_namespace, + // provider_name, branch}; the namespace/name keys are prefixed to avoid the + // reserved Prometheus pod-scrape target labels (see + // TargetInitialReconcileComplete). Load-bearing for the restart-snapshot e2e + // spec's drain wait; treat the name/labels as a public observability contract. + BranchWorkerQueueDepth metric.Int64Gauge + // WatchDuplicatesSkippedTotal counts watch events skipped due to duplicate sanitized content. WatchDuplicatesSkippedTotal metric.Int64Counter // AuditEventsReceivedTotal counts audit events received from Kubernetes API server. @@ -284,6 +306,8 @@ func registerInstruments() error { {"gitopsreverser_audit_queue_oldest_entry_age_seconds", &AuditQueueOldestEntryAgeSeconds}, {"gitopsreverser_audit_queue_oldest_pending_age_seconds", &AuditQueueOldestPendingAgeSeconds}, {"gitopsreverser_audit_debug_stream_length", &AuditDebugStreamLength}, + {"gitopsreverser_target_initial_reconcile_complete", &TargetInitialReconcileComplete}, + {"gitopsreverser_branch_worker_queue_depth", &BranchWorkerQueueDepth}, } for _, s := range gauges { v, err := otelMeter.Int64Gauge(s.name) diff --git a/internal/watch/manager.go b/internal/watch/manager.go index c7544ae7e..5ff22eb03 100644 --- a/internal/watch/manager.go +++ b/internal/watch/manager.go @@ -41,12 +41,15 @@ import ( "github.com/cespare/xxhash/v2" "github.com/go-logr/logr" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" "sigs.k8s.io/controller-runtime/pkg/client" configv1alpha1 "github.com/ConfigButler/gitops-reverser/api/v1alpha1" "github.com/ConfigButler/gitops-reverser/internal/events" "github.com/ConfigButler/gitops-reverser/internal/rulestore" "github.com/ConfigButler/gitops-reverser/internal/sanitize" + "github.com/ConfigButler/gitops-reverser/internal/telemetry" "github.com/ConfigButler/gitops-reverser/internal/types" ) @@ -1293,6 +1296,7 @@ func (m *Manager) emitSnapshotForRuleChange( continue } m.markRuleSetSnapshotDelivered(target) + m.recordTargetInitialReconcileComplete(gitDest) emitted = true } if emitted { @@ -1300,6 +1304,22 @@ func (m *Manager) emitSnapshotForRuleChange( } } +// recordTargetInitialReconcileComplete latches the per-GitTarget gauge to 1 once +// its snapshot decision has been made and the resulting write request submitted +// to the branch worker. On a controller restart this is the signal that the +// target's post-restart initial reconcile pass has reached the git write path — +// paired with a drained BranchWorkerQueueDepth it proves the post-restart +// snapshot has fully landed. No-op until the gauge is registered. +func (m *Manager) recordTargetInitialReconcileComplete(gitDest types.ResourceReference) { + if telemetry.TargetInitialReconcileComplete == nil { + return + } + telemetry.TargetInitialReconcileComplete.Record(context.Background(), 1, metric.WithAttributes( + attribute.String("gittarget_namespace", gitDest.Namespace), + attribute.String("gittarget_name", gitDest.Name), + )) +} + func (m *Manager) completeReconciliationForTargets(targets []ruleSetSnapshotTarget, log logr.Logger) { if m.EventRouter == nil { return diff --git a/internal/watch/target_reconcile_metric_test.go b/internal/watch/target_reconcile_metric_test.go new file mode 100644 index 000000000..64196b2ab --- /dev/null +++ b/internal/watch/target_reconcile_metric_test.go @@ -0,0 +1,56 @@ +/* +SPDX-License-Identifier: Apache-2.0 + +Copyright 2025 ConfigButler + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package watch + +import ( + "testing" + + "github.com/go-logr/logr" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/ConfigButler/gitops-reverser/internal/telemetry" + "github.com/ConfigButler/gitops-reverser/internal/types" +) + +const targetInitialReconcileCompleteMetric = "gitopsreverser_target_initial_reconcile_complete" + +// recordTargetInitialReconcileComplete must latch the per-GitTarget gauge to 1 +// under the target's namespace/name labels. +func TestRecordTargetInitialReconcileComplete_LatchesToOne(t *testing.T) { + reader, err := telemetry.InitTestExporter() + require.NoError(t, err) + + m := &Manager{Log: logr.Discard()} + gitDest := types.NewResourceReference("my-target", "my-ns") + + m.recordTargetInitialReconcileComplete(gitDest) + + value, ok := telemetry.CollectInt64Sum(reader, targetInitialReconcileCompleteMetric, + map[string]string{"gittarget_namespace": "my-ns", "gittarget_name": "my-target"}) + require.True(t, ok, "expected a target_initial_reconcile_complete sample") + assert.Equal(t, int64(1), value) + + // Re-recording is idempotent: the latch stays at 1. + m.recordTargetInitialReconcileComplete(gitDest) + value, ok = telemetry.CollectInt64Sum(reader, targetInitialReconcileCompleteMetric, + map[string]string{"gittarget_namespace": "my-ns", "gittarget_name": "my-target"}) + require.True(t, ok) + assert.Equal(t, int64(1), value) +} diff --git a/test/e2e/restart_snapshot_e2e_test.go b/test/e2e/restart_snapshot_e2e_test.go index e5bb12e4a..f64bdbc0e 100644 --- a/test/e2e/restart_snapshot_e2e_test.go +++ b/test/e2e/restart_snapshot_e2e_test.go @@ -66,6 +66,10 @@ var _ = Describe("Restart Snapshot Safety", Label("restart-snapshot", "smoke"), orderNames := []string{"restart-order-alpha", "restart-order-bravo", "restart-order-charlie"} BeforeAll(func() { + By("setting up the Prometheus client for drain-signal metrics") + setupPrometheusClient() + verifyPrometheusAvailable() + By("creating the restart-snapshot test namespace") testNs = testNamespaceFor("restart-snapshot") _, _ = kubectlRun("create", "namespace", testNs) // idempotent; ignore AlreadyExists @@ -154,10 +158,46 @@ var _ = Describe("Restart Snapshot Safety", Label("restart-snapshot", "smoke"), By("restarting the controller deployment (plain rollout restart)") restartControllerDeployment() - // The destructive snapshot commit lands within seconds of the new pod - // starting. A quiet order, once wiped, never comes back — so once the - // mirror is intact for a sustained window after the restart settles, it - // will stay intact. + // Phase 3 drain signals replace a 75 s blind wait. The new pod's startup + // reconcile sets target_initial_reconcile_complete=1 once its snapshot + // decision is made and submitted, and branch_worker_queue_depth returns + // to 0 once that submission has been committed and pushed. Waiting on + // both confirms the post-restart snapshot has fully drained through the + // git write path — the exact moment any destructive commit would have + // landed — instead of guessing with a fixed sleep. + // Both queries aggregate across pods: a rollout briefly leaves the old + // pod's last sample in Prometheus alongside the new pod's, each carrying + // its own pod target label. max()/sum() collapse them so the freshly + // started pod's value is what the assertion sees, rather than whichever + // series queryPrometheus happens to return first. + By("waiting for the post-restart initial reconcile to complete") + waitForMetricWithTimeout( + fmt.Sprintf( + `max(gitopsreverser_target_initial_reconcile_complete`+ + `{gittarget_namespace=%q,gittarget_name=%q}) or vector(0)`, + testNs, gitTargetName, + ), + func(v float64) bool { return v == 1 }, + "GitTarget initial reconcile complete after restart", + 90*time.Second, + ) + + By("waiting for the branch worker queue to drain") + waitForMetricWithTimeout( + fmt.Sprintf( + `sum(gitopsreverser_branch_worker_queue_depth`+ + `{provider_namespace=%q,provider_name=%q,branch="main"}) or vector(0)`, + testNs, providerName, + ), + func(v float64) bool { return v == 0 }, + "branch worker queue drained after restart", + 90*time.Second, + ) + + // A short stability window guards against a delayed second reconcile and + // against any Prometheus cross-restart sample staleness in the gates + // above: a quiet order, once wiped, never comes back, so a mirror that + // stays intact for this window after the drain will stay intact. By("verifying the git mirror is NOT wiped by the restart") Consistently(func(g Gomega) { pullLatestRepoState(g, restartRepo.CheckoutDir) @@ -166,7 +206,7 @@ var _ = Describe("Restart Snapshot Safety", Label("restart-snapshot", "smoke"), g.Expect(statErr).NotTo(HaveOccurred(), "file %q disappeared after the controller restart — startup snapshot wiped the mirror", relPath) } - }, 75*time.Second, 5*time.Second).Should(Succeed()) + }, 15*time.Second, 5*time.Second).Should(Succeed()) By("confirming no commit since the restart deleted tracked files") deletions, logErr := gitRun( From 54842213ac6869a85e1e27bccdfc0c73496664b0 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Sun, 31 May 2026 19:43:37 +0000 Subject: [PATCH 19/40] chore: metric to prevent the long wait --- docs/design/e2e-speedup-plan.md | 103 ++++++++++++------ internal/telemetry/exporter.go | 34 +++--- internal/watch/manager.go | 27 +++-- .../watch/target_reconcile_metric_test.go | 32 ++++-- test/e2e/restart_snapshot_e2e_test.go | 51 +++++---- 5 files changed, 155 insertions(+), 92 deletions(-) diff --git a/docs/design/e2e-speedup-plan.md b/docs/design/e2e-speedup-plan.md index 9d16cee5f..4bc2e14f3 100644 --- a/docs/design/e2e-speedup-plan.md +++ b/docs/design/e2e-speedup-plan.md @@ -530,14 +530,36 @@ that the pod started. **Change.** Pair two signals: -1. **`gitopsreverser_target_initial_reconcile_complete{gittarget_namespace,gittarget_name}`** - — gauge per GitTarget, flips to `1` once its post-(re)start initial - reconcile pass has completed end-to-end (snapshot decision made, - queue submission completed). +1. **`gitopsreverser_target_reconcile_completed_total{gittarget_namespace,gittarget_name,trigger}`** + — **counter** per GitTarget, incremented once each rule-set snapshot + reconcile pass completes end-to-end (snapshot decision made, queue + submission completed). `trigger` is `rule_change` (the GVR/rule reconcile + path) or `startup_replay` (a snapshot replayed once a FolderReconciler is + created). A counter, **not a latched gauge**: see the stale-latch note + below. 2. **`gitopsreverser_branch_worker_queue_depth{provider_namespace,provider_name,branch}`** — gauge of pending work items for this branch worker. Wait for `0` to confirm the queue has drained. +> **Why a counter, not a latched gauge, and why gate per-pod.** An earlier +> draft used a gauge `gitopsreverser_target_initial_reconcile_complete` that +> latched to `1`. During a rollout Prometheus can still hold the **old** +> pod's series (distinct `pod` target label) alongside the new pod's, so a +> `max(...) == 1` check is satisfied by the dying pod's stale `1` and proves +> nothing about the new pod — re-weakening the safety gate toward the old +> blind wait. A counter is the right shape, but a *cross-pod* +> `sum(...) > baseline` delta still has a failure mode: counters reset to `0` +> on the fresh pod, so once Prometheus marks the old pod's series stale, the +> new pod's first increment can bring the cross-pod sum back to the old total +> (e.g. old=1 → stale → new=1 → sum stays 1) and `> baseline` never passes. +> The robust gate is therefore **per-pod**: discover the new pod's name after +> the rollout completes and assert `…{pod=""} > 0`. A counter on a +> freshly started pod starts at `0`, so any value `> 0` proves *that pod* +> completed its own post-restart reconcile. The counter also has long-term +> value — `increase(...[5m])` surfaces excessive/looping reconciles. The name +> reflects what it actually instruments: rule-set snapshot delivery/replay, +> not a literal "initial" controller path. + The metric package is `internal/telemetry/` (not `internal/metrics/` — see `internal/telemetry/exporter.go`); add the new metrics alongside the existing ones. @@ -555,18 +577,20 @@ the existing ones. **Restart test reframe:** -1. Rollout-restart the controller deployment. -2. Wait for `max(target_initial_reconcile_complete{gittarget_*}) == 1` - for the restart test's GitTarget. -3. Wait for `sum(branch_worker_queue_depth{provider_*,branch}) == 0`. -4. Apply a short stability window (a few seconds). -5. Assert no destructive commit happened. - -The `max()`/`sum()` aggregation in steps 2–3 collapses the brief window -during a rollout where Prometheus still holds the old pod's last sample -(distinct `pod` target label) next to the new pod's, so the assertion sees -the freshly started pod's value rather than whichever series the query -returns first. +1. Rollout-restart the controller deployment (`restartControllerDeployment` + blocks until the rollout completes). +2. Discover the new controller pod — after the rollout there is exactly one + non-terminating pod matching `control-plane=gitops-reverser`. +3. Wait for `sum(target_reconcile_completed_total{gittarget_*,pod=""}) + > 0` for the restart test's GitTarget. +4. Wait for `sum(branch_worker_queue_depth{provider_*,branch,pod=""}) + == 0`. +5. Apply a short stability window (a few seconds). +6. Assert no destructive commit happened. + +Both gates pin the `pod` target label to the freshly started pod, so neither +the old pod's lingering samples nor a counter reset can produce a false +pass/timeout during the rollout's brief two-series overlap. **Risks:** @@ -591,10 +615,11 @@ returns first. **Implementation status (2026-05-31).** Landed on branch `refactors`. -- New gauges in [internal/telemetry/exporter.go](../../internal/telemetry/exporter.go): - `gitopsreverser_target_initial_reconcile_complete` (set in +- New metrics in [internal/telemetry/exporter.go](../../internal/telemetry/exporter.go): + counter `gitopsreverser_target_reconcile_completed_total` (incremented in `emitSnapshotForRuleChange`'s per-target success branch, - [internal/watch/manager.go](../../internal/watch/manager.go)) and + [internal/watch/manager.go](../../internal/watch/manager.go), with the + `trigger` passed by each caller) and gauge `gitopsreverser_branch_worker_queue_depth` (recorded from the branch worker event loop + enqueue paths, [internal/git/branch_worker.go](../../internal/git/branch_worker.go)). @@ -603,21 +628,31 @@ returns first. [internal/watch/target_reconcile_metric_test.go](../../internal/watch/target_reconcile_metric_test.go). - **First e2e run failed** — the query used a bare `namespace` label and matched nothing (the honor_labels collision above). Renaming to the - prefixed keys and adding `max()`/`sum()` aggregation fixed it. The - controller-side delivery was correct on the first run (7 snapshot emits, - zero "No reconciler registered"); the bug was purely in the metric's - label contract. -- **Measured:** restart spec `Restart Snapshot Safety` **57.0 s** total - wallclock (down from ~128 s), full smoke suite green (20/20, 444 s vs - the earlier 537 s run that included the failing attempt). Both metric - waits resolved within the first poll after the rollout settled — the - remaining 57 s is BeforeAll setup (Prometheus client, Gitea repo, CRD - install, building the mirror, the rollout restart itself) plus the 15 s - stability window, **not** the old 75 s blind negative-assertion wait. - The signal-driven wait does not hit the <30 s exit-criterion target on - its own because the per-spec figure is dominated by fixed setup, not the - drain wait that Phase 3 removed; the drain wait itself is now ~0 s. - Tightening the residual setup/stability cost is follow-up work. + prefixed keys and adding aggregation fixed it. The controller-side + delivery was correct on the first run (7 snapshot emits, zero "No + reconciler registered"); the bug was purely in the metric's label + contract. +- **Review follow-up 1:** the reconcile signal started as a latched gauge + read with `max(...) == 1`; review flagged that the old pod's stale `1` + could satisfy that during a rollout. Replaced with the counter. +- **Review follow-up 2:** a cross-pod `sum(...) > baseline` delta was then + flagged as still race-prone — counters reset on the new pod, so once the + old series goes stale the new pod's first increment can return the sum to + the baseline rather than above it, causing a false timeout. Replaced with + a **per-pod** gate: discover the new pod after the rollout and assert + `…{pod=""} > 0` (and `branch_worker_queue_depth{…,pod=""} == 0`). +- **Measured (per-pod gate, 2026-05-31):** restart spec `Restart Snapshot + Safety` ~**67 s** total wallclock (down from ~128 s; earlier gauge/baseline + runs measured 57–67 s — the spread is fixed-setup variance, not the gates), + full smoke suite green (20/20). Both gates resolved within the first poll + after the rollout settled — the remaining time is BeforeAll setup + (Prometheus client, Gitea repo, CRD install, building the mirror, the + rollout restart itself) plus the 15 s stability window, **not** the old + 75 s blind negative-assertion wait. The signal-driven wait does not hit the + <30 s exit-criterion target on its own because the per-spec figure is + dominated by fixed setup, not the drain wait that Phase 3 removed; the + drain wait itself is now ~0 s. Tightening the residual setup/stability cost + is follow-up work. ## Decisions diff --git a/internal/telemetry/exporter.go b/internal/telemetry/exporter.go index c16119bba..c1b45594e 100644 --- a/internal/telemetry/exporter.go +++ b/internal/telemetry/exporter.go @@ -62,25 +62,31 @@ var ( // RepoBranchQueueDepth is a gauge for per-repo-branch queue depth. RepoBranchQueueDepth metric.Int64UpDownCounter - // TargetInitialReconcileComplete latches to 1 once a GitTarget's post-(re)start - // initial reconcile pass has completed end-to-end in this process: the snapshot - // decision was made and its write request was submitted to the branch worker - // queue. Labelled by {gittarget_namespace, gittarget_name}. The labels avoid - // the reserved `namespace`/`name` keys on purpose: a Prometheus pod scrape with - // honor_labels=false overwrites a metric's `namespace` attribute with the - // scraped pod's own namespace, so a per-GitTarget `namespace` label would be - // silently unqueryable. It is a per-process latch — a controller restart starts - // a fresh process whose gauge is unset until the first snapshot delivery sets - // it. Load-bearing for the restart-snapshot e2e spec; treat the name/labels as - // a public observability contract. - TargetInitialReconcileComplete metric.Int64Gauge + // TargetReconcileCompletedTotal counts completed rule-set snapshot reconcile + // passes per GitTarget: each increment marks one pass where the snapshot + // decision was made and its write request submitted to the branch worker + // queue. Labelled by {gittarget_namespace, gittarget_name, trigger} where + // trigger is `rule_change` (the GVR/rule reconcile path) or `startup_replay` + // (a snapshot replayed once a FolderReconciler is created). A counter, not a + // latched gauge, on purpose: a counter resets to 0 on a fresh pod, so a + // per-pod `{pod=""} > 0` check after a rollout proves the new pod did + // its own reconcile — robust to the old pod's stale series that a Prometheus + // pod scrape may still be holding during the rollout, which a latched gauge + // (or a cross-pod sum-over-baseline) cannot distinguish. + // The label keys avoid the reserved `namespace`/`name`: a pod scrape with + // honor_labels=false would overwrite a metric's `namespace` attribute with the + // scraped pod's own namespace, making a per-GitTarget `namespace` selector + // silently match nothing. Load-bearing for the restart-snapshot e2e spec and + // useful long-term for spotting excessive reconciles via increase(...[5m]); + // treat the name/labels as a public observability contract. + TargetReconcileCompletedTotal metric.Int64Counter // BranchWorkerQueueDepth gauges pending work for a single branch worker: // queued work items plus any committed-but-not-yet-pushed work the worker is // still holding. It reads 0 only when the worker has fully drained (empty // queue and nothing retained for replay). Labelled by {provider_namespace, // provider_name, branch}; the namespace/name keys are prefixed to avoid the // reserved Prometheus pod-scrape target labels (see - // TargetInitialReconcileComplete). Load-bearing for the restart-snapshot e2e + // TargetReconcileCompletedTotal). Load-bearing for the restart-snapshot e2e // spec's drain wait; treat the name/labels as a public observability contract. BranchWorkerQueueDepth metric.Int64Gauge @@ -258,6 +264,7 @@ func registerInstruments() error { {"gitopsreverser_secret_encryption_failures_total", &SecretEncryptionFailuresTotal}, {"gitopsreverser_secret_encryption_cache_hits_total", &SecretEncryptionCacheHitsTotal}, {"gitopsreverser_secret_encryption_marker_skips_total", &SecretEncryptionMarkerSkipsTotal}, + {"gitopsreverser_target_reconcile_completed_total", &TargetReconcileCompletedTotal}, } for _, s := range counters { v, err := otelMeter.Int64Counter(s.name) @@ -306,7 +313,6 @@ func registerInstruments() error { {"gitopsreverser_audit_queue_oldest_entry_age_seconds", &AuditQueueOldestEntryAgeSeconds}, {"gitopsreverser_audit_queue_oldest_pending_age_seconds", &AuditQueueOldestPendingAgeSeconds}, {"gitopsreverser_audit_debug_stream_length", &AuditDebugStreamLength}, - {"gitopsreverser_target_initial_reconcile_complete", &TargetInitialReconcileComplete}, {"gitopsreverser_branch_worker_queue_depth", &BranchWorkerQueueDepth}, } for _, s := range gauges { diff --git a/internal/watch/manager.go b/internal/watch/manager.go index 5ff22eb03..c806c8aaa 100644 --- a/internal/watch/manager.go +++ b/internal/watch/manager.go @@ -741,7 +741,7 @@ func (m *Manager) ReconcileForRuleChange(ctx context.Context) error { // Emit RequestClusterState for each affected GitTarget so that a single // "reconcile: sync N resources" commit is produced instead of N individual // [CREATE] commits from the informer ADDED events buffered above. - m.emitSnapshotForRuleChange(ctx, log, targets) + m.emitSnapshotForRuleChange(ctx, log, targets, "rule_change") // Transition streams back to LIVE_PROCESSING and flush buffered events. // startInformersForGVRs already waited for cache sync (WaitForCacheSync), @@ -1238,7 +1238,7 @@ func (m *Manager) MaybeReplaySnapshot(ctx context.Context, gitDest types.Resourc target := ruleSetSnapshotTarget{gitDest: gitDest, hash: hash} log := m.Log.WithName("reconcile") m.EventRouter.BeginReconciliationForStream(gitDest) - m.emitSnapshotForRuleChange(ctx, log, []ruleSetSnapshotTarget{target}) + m.emitSnapshotForRuleChange(ctx, log, []ruleSetSnapshotTarget{target}, "startup_replay") m.EventRouter.CompleteReconciliationForStream(gitDest) } @@ -1259,6 +1259,7 @@ func (m *Manager) emitSnapshotForRuleChange( ctx context.Context, log logr.Logger, targets []ruleSetSnapshotTarget, + trigger string, ) { if m.EventRouter == nil { log.Info("EventRouter not set, skipping snapshot emission") @@ -1296,7 +1297,7 @@ func (m *Manager) emitSnapshotForRuleChange( continue } m.markRuleSetSnapshotDelivered(target) - m.recordTargetInitialReconcileComplete(gitDest) + m.recordTargetReconcileCompleted(gitDest, trigger) emitted = true } if emitted { @@ -1304,19 +1305,21 @@ func (m *Manager) emitSnapshotForRuleChange( } } -// recordTargetInitialReconcileComplete latches the per-GitTarget gauge to 1 once -// its snapshot decision has been made and the resulting write request submitted -// to the branch worker. On a controller restart this is the signal that the -// target's post-restart initial reconcile pass has reached the git write path — -// paired with a drained BranchWorkerQueueDepth it proves the post-restart -// snapshot has fully landed. No-op until the gauge is registered. -func (m *Manager) recordTargetInitialReconcileComplete(gitDest types.ResourceReference) { - if telemetry.TargetInitialReconcileComplete == nil { +// recordTargetReconcileCompleted increments the per-GitTarget reconcile counter +// once its snapshot decision has been made and the resulting write request +// submitted to the branch worker, tagged with the trigger that drove the pass. +// On a controller restart a delta over a pre-restart baseline is the signal that +// the new pod's snapshot reconcile reached the git write path — paired with a +// drained BranchWorkerQueueDepth it proves the post-restart snapshot has fully +// landed. No-op until the counter is registered. +func (m *Manager) recordTargetReconcileCompleted(gitDest types.ResourceReference, trigger string) { + if telemetry.TargetReconcileCompletedTotal == nil { return } - telemetry.TargetInitialReconcileComplete.Record(context.Background(), 1, metric.WithAttributes( + telemetry.TargetReconcileCompletedTotal.Add(context.Background(), 1, metric.WithAttributes( attribute.String("gittarget_namespace", gitDest.Namespace), attribute.String("gittarget_name", gitDest.Name), + attribute.String("trigger", trigger), )) } diff --git a/internal/watch/target_reconcile_metric_test.go b/internal/watch/target_reconcile_metric_test.go index 64196b2ab..a67667bef 100644 --- a/internal/watch/target_reconcile_metric_test.go +++ b/internal/watch/target_reconcile_metric_test.go @@ -29,28 +29,36 @@ import ( "github.com/ConfigButler/gitops-reverser/internal/types" ) -const targetInitialReconcileCompleteMetric = "gitopsreverser_target_initial_reconcile_complete" +const targetReconcileCompletedMetric = "gitopsreverser_target_reconcile_completed_total" -// recordTargetInitialReconcileComplete must latch the per-GitTarget gauge to 1 -// under the target's namespace/name labels. -func TestRecordTargetInitialReconcileComplete_LatchesToOne(t *testing.T) { +// recordTargetReconcileCompleted must increment the per-GitTarget counter and +// carry the gittarget_* and trigger labels. +func TestRecordTargetReconcileCompleted_IncrementsPerTrigger(t *testing.T) { reader, err := telemetry.InitTestExporter() require.NoError(t, err) m := &Manager{Log: logr.Discard()} gitDest := types.NewResourceReference("my-target", "my-ns") - m.recordTargetInitialReconcileComplete(gitDest) + m.recordTargetReconcileCompleted(gitDest, "rule_change") - value, ok := telemetry.CollectInt64Sum(reader, targetInitialReconcileCompleteMetric, - map[string]string{"gittarget_namespace": "my-ns", "gittarget_name": "my-target"}) - require.True(t, ok, "expected a target_initial_reconcile_complete sample") + value, ok := telemetry.CollectInt64Sum(reader, targetReconcileCompletedMetric, + map[string]string{"gittarget_namespace": "my-ns", "gittarget_name": "my-target", "trigger": "rule_change"}) + require.True(t, ok, "expected a target_reconcile_completed_total sample") assert.Equal(t, int64(1), value) - // Re-recording is idempotent: the latch stays at 1. - m.recordTargetInitialReconcileComplete(gitDest) - value, ok = telemetry.CollectInt64Sum(reader, targetInitialReconcileCompleteMetric, - map[string]string{"gittarget_namespace": "my-ns", "gittarget_name": "my-target"}) + // A second pass increments the counter, proving a delta over a baseline + // distinguishes a fresh reconcile from a stale latched value. + m.recordTargetReconcileCompleted(gitDest, "rule_change") + value, ok = telemetry.CollectInt64Sum(reader, targetReconcileCompletedMetric, + map[string]string{"gittarget_namespace": "my-ns", "gittarget_name": "my-target", "trigger": "rule_change"}) + require.True(t, ok) + assert.Equal(t, int64(2), value) + + // A different trigger is a distinct series, not folded into the first. + m.recordTargetReconcileCompleted(gitDest, "startup_replay") + value, ok = telemetry.CollectInt64Sum(reader, targetReconcileCompletedMetric, + map[string]string{"gittarget_namespace": "my-ns", "gittarget_name": "my-target", "trigger": "startup_replay"}) require.True(t, ok) assert.Equal(t, int64(1), value) } diff --git a/test/e2e/restart_snapshot_e2e_test.go b/test/e2e/restart_snapshot_e2e_test.go index f64bdbc0e..b34129eb1 100644 --- a/test/e2e/restart_snapshot_e2e_test.go +++ b/test/e2e/restart_snapshot_e2e_test.go @@ -158,27 +158,38 @@ var _ = Describe("Restart Snapshot Safety", Label("restart-snapshot", "smoke"), By("restarting the controller deployment (plain rollout restart)") restartControllerDeployment() - // Phase 3 drain signals replace a 75 s blind wait. The new pod's startup - // reconcile sets target_initial_reconcile_complete=1 once its snapshot - // decision is made and submitted, and branch_worker_queue_depth returns - // to 0 once that submission has been committed and pushed. Waiting on - // both confirms the post-restart snapshot has fully drained through the - // git write path — the exact moment any destructive commit would have - // landed — instead of guessing with a fixed sleep. - // Both queries aggregate across pods: a rollout briefly leaves the old - // pod's last sample in Prometheus alongside the new pod's, each carrying - // its own pod target label. max()/sum() collapse them so the freshly - // started pod's value is what the assertion sees, rather than whichever - // series queryPrometheus happens to return first. - By("waiting for the post-restart initial reconcile to complete") + // Anchor the reconcile gate to the NEW pod. restartControllerDeployment + // blocks until the rollout completes, so exactly one non-terminating + // controller pod remains — the freshly started one. controllerPodNames + // already filters out pods with a deletionTimestamp. + var newControllerPod string + Eventually(func(g Gomega) { + pods, podErr := controllerPodNames() + g.Expect(podErr).NotTo(HaveOccurred()) + g.Expect(pods).To(HaveLen(1), "expected exactly one non-terminating controller pod after rollout") + newControllerPod = pods[0] + }, 30*time.Second, 2*time.Second).Should(Succeed()) + By(fmt.Sprintf("new controller pod after restart: %s", newControllerPod)) + + // Phase 3 drain signals replace a 75 s blind wait. The reconcile counter + // is scoped to the new pod via its `pod` target label: a counter resets to + // 0 on a fresh pod, so `{pod=""} > 0` proves the new pod completed its + // own post-restart snapshot reconcile. A sum() over a pre-restart baseline + // cannot prove this — once Prometheus marks the old pod's series stale, the + // new pod's first increment can bring the cross-pod sum back to the old + // total rather than above it, so `> baseline` could never pass. + // branch_worker_queue_depth returning to 0 then confirms that submission + // has been committed and pushed — the exact moment any destructive commit + // would have landed — instead of guessing with a fixed sleep. + By("waiting for the new pod to complete its post-restart reconcile") waitForMetricWithTimeout( fmt.Sprintf( - `max(gitopsreverser_target_initial_reconcile_complete`+ - `{gittarget_namespace=%q,gittarget_name=%q}) or vector(0)`, - testNs, gitTargetName, + `sum(gitopsreverser_target_reconcile_completed_total`+ + `{gittarget_namespace=%q,gittarget_name=%q,pod=%q}) or vector(0)`, + testNs, gitTargetName, newControllerPod, ), - func(v float64) bool { return v == 1 }, - "GitTarget initial reconcile complete after restart", + func(v float64) bool { return v > 0 }, + "GitTarget reconcile completed on the new controller pod", 90*time.Second, ) @@ -186,8 +197,8 @@ var _ = Describe("Restart Snapshot Safety", Label("restart-snapshot", "smoke"), waitForMetricWithTimeout( fmt.Sprintf( `sum(gitopsreverser_branch_worker_queue_depth`+ - `{provider_namespace=%q,provider_name=%q,branch="main"}) or vector(0)`, - testNs, providerName, + `{provider_namespace=%q,provider_name=%q,branch="main",pod=%q}) or vector(0)`, + testNs, providerName, newControllerPod, ), func(v float64) bool { return v == 0 }, "branch worker queue drained after restart", From 796d7b61cc80bf0d94b2bb47ca56cff6b3fabfa2 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Sun, 31 May 2026 20:15:20 +0000 Subject: [PATCH 20/40] chore: Insights in parallel runs with Allure --- .github/workflows/ci.yml | 50 +++ docs/ci/e2e-allure-reporting.md | 31 ++ docs/ci/findings.md | 1 + docs/design/e2e-speedup-plan.md | 28 +- internal/git/branch_worker.go | 41 ++- internal/git/branch_worker_metrics_test.go | 32 +- internal/telemetry/exporter.go | 8 +- internal/watch/manager.go | 9 +- test/e2e/tools/ginkgo-allure/main.go | 305 +++++++++++++++++ test/e2e/tools/ginkgo-allure/main_test.go | 361 +++++++++++++++++++++ 10 files changed, 838 insertions(+), 28 deletions(-) create mode 100644 docs/ci/e2e-allure-reporting.md create mode 100644 test/e2e/tools/ginkgo-allure/main.go create mode 100644 test/e2e/tools/ginkgo-allure/main_test.go diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8080d9203..d34b6469a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -362,6 +362,56 @@ jobs: if-no-files-found: ignore retention-days: 7 + - name: Generate Allure results + id: allure-results + if: always() + run: | + docker run --rm \ + -v "${GITHUB_WORKSPACE}:${{ env.CI_WORKDIR }}" \ + -w "${{ env.CI_WORKDIR }}" \ + ${{ env.CI_CONTAINER }} \ + bash -c ' + set -euo pipefail + mapfile -t reports < <(find .stamps -path "*/ginkgo-report-*.json" -type f | sort) + if [ "${#reports[@]}" -eq 0 ]; then + echo "No Ginkgo JSON reports were produced." + exit 0 + fi + rm -rf allure-results + mkdir -p allure-results + go run ./test/e2e/tools/ginkgo-allure --output-dir allure-results "${reports[@]}" + ' + if find allure-results -name '*-result.json' -type f | grep -q .; then + echo "has_results=true" >> "${GITHUB_OUTPUT}" + else + echo "has_results=false" >> "${GITHUB_OUTPUT}" + fi + + - name: Prepare Allure history directory + if: always() && steps.allure-results.outputs.has_results == 'true' + run: | + mkdir -p gh-pages allure-history + + - name: Generate Allure report + if: always() && steps.allure-results.outputs.has_results == 'true' + uses: simple-elf/allure-report-action@v1.13 + with: + allure_results: allure-results + allure_report: allure-report + gh_pages: gh-pages + allure_history: allure-history + + - name: Upload Allure report artifact + if: always() && steps.allure-results.outputs.has_results == 'true' + uses: actions/upload-artifact@v7 + with: + name: e2e-allure-report-${{ matrix.name }} + path: | + allure-report + allure-results + if-no-files-found: ignore + retention-days: 7 + # Release job only runs on push to main after tests pass release-please: name: Release Please diff --git a/docs/ci/e2e-allure-reporting.md b/docs/ci/e2e-allure-reporting.md new file mode 100644 index 000000000..3010bc5ff --- /dev/null +++ b/docs/ci/e2e-allure-reporting.md @@ -0,0 +1,31 @@ +# E2E Allure Reporting + +The CI e2e job produces a visual Allure HTML report for each matrix entry. + +The e2e tasks still run through the repository Taskfile so local and CI execution +stay aligned. Each task writes Ginkgo JSON reports under +`.stamps/cluster///ginkgo-report-.json`. After the e2e +run, CI converts those Ginkgo JSON files into Allure result files with: + +```bash +go run ./test/e2e/tools/ginkgo-allure --output-dir allure-results ... +``` + +Then `simple-elf/allure-report-action` renders `allure-report`, and CI uploads +the generated HTML as the `e2e-allure-report-` artifact. + +This is intentionally a post-run report. During an active e2e run, GitHub Actions +logs remain the live view. After the run finishes, the Allure artifact provides a +visual timeline, status breakdown, labels, durations, and captured Ginkgo output. + +The converter is deliberately small and one-way: + +- Ginkgo `passed` maps to Allure `passed`. +- Ginkgo `failed` maps to Allure `failed`. +- Ginkgo `pending` and `skipped` map to Allure `skipped`. +- Ginkgo timeout, interrupt, abort, and panic states map to Allure `broken`. +- Captured `GinkgoWriter` and parallel stdout/stderr output are attached as + plain-text test artifacts. + +The existing raw Ginkgo JSON and timing-summary artifacts are still uploaded. +Those files remain the source of truth for local tooling and timing analysis. diff --git a/docs/ci/findings.md b/docs/ci/findings.md index 6ae8f7458..611b53923 100644 --- a/docs/ci/findings.md +++ b/docs/ci/findings.md @@ -89,3 +89,4 @@ kubectl get nodes - `windows-devcontainer.md` - Windows-specific mount behavior and expected differences - `ci-root-user.md` - why CI containers run as root - `git-safe-dir-explained.md` - why `safe.directory` is required in containerized CI +- `e2e-allure-reporting.md` - how CI renders visual Allure reports from Ginkgo JSON diff --git a/docs/design/e2e-speedup-plan.md b/docs/design/e2e-speedup-plan.md index 4bc2e14f3..0958209ef 100644 --- a/docs/design/e2e-speedup-plan.md +++ b/docs/design/e2e-speedup-plan.md @@ -599,13 +599,18 @@ pass/timeout during the rollout's brief two-series overlap. public observability surface from day one — renaming costs cluster rollouts. - Worker-queue-depth must include in-flight items, not just queued ones, - or step 3 races with the worker. Verify against the actual - implementation when writing the metric. **Implemented:** the event loop - republishes the gauge once per iteration from its authoritative state - (`openWindow != nil || len(pendingWrites) > 0`, surfaced via an - `atomic.Bool` the enqueue paths also re-read), so the gauge counts the - open commit window and retained-for-replay writes — not just channel - depth — and only reaches 0 after a successful push clears `pendingWrites`. + or step 4 races with the worker. Verify against the actual + implementation when writing the metric. **Implemented:** depth = + `inflightItems + (hasUnpushedWork ? 1 : 0)`, where `inflightItems` + (`atomic.Int64`) is incremented *before* each `eventQueue` send and + decremented only *after* `handleQueueItem` returns, and `hasUnpushedWork` + (`atomic.Bool`, refreshed each loop iteration) tracks an open commit + window or retained pending writes. Using `len(eventQueue)` would read 0 + the instant the loop receives an item — before it is handled — so a scrape + in that window could satisfy the drain gate mid-commit; counting accepted + items instead closes that race. The gauge reaches 0 only once every + accepted item is fully handled and a successful push has cleared + `pendingWrites`. See the in-flight follow-up below. **Exit criterion:** @@ -641,6 +646,15 @@ pass/timeout during the rollout's brief two-series overlap. the baseline rather than above it, causing a false timeout. Replaced with a **per-pod** gate: discover the new pod after the rollout and assert `…{pod=""} > 0` (and `branch_worker_queue_depth{…,pod=""} == 0`). +- **Review follow-up 3:** the queue-depth gauge originally derived its + pending count from `len(eventQueue)` plus an `atomic.Bool`. Review flagged + that the channel length drops to 0 the moment the loop *receives* an item, + before `handleQueueItem` finishes, so a scrape in that window could satisfy + the drain gate while a commit/push was still in flight. Replaced the + channel-length read with an `inflightItems atomic.Int64` incremented before + the send and decremented after handling, so an accepted-but-unhandled item + keeps depth `> 0`. Covered by `TestRecordQueueDepth_InflightItemDequeuedButUnhandled` + and verified under `go test -race`. - **Measured (per-pod gate, 2026-05-31):** restart spec `Restart Snapshot Safety` ~**67 s** total wallclock (down from ~128 s; earlier gauge/baseline runs measured 57–67 s — the spread is fixed-setup variance, not the gates), diff --git a/internal/git/branch_worker.go b/internal/git/branch_worker.go index d5b0bba62..5a628d44a 100644 --- a/internal/git/branch_worker.go +++ b/internal/git/branch_worker.go @@ -132,8 +132,18 @@ type BranchWorker struct { // open window or any committed-but-not-yet-pushed pending writes. The event // loop is the only writer; recordQueueDepth (callable from enqueue goroutines) // is the only other reader, hence the atomic. It lets the depth gauge account - // for in-flight work the channel length alone cannot see. + // for retained work the channel length alone cannot see. hasUnpushedWork atomic.Bool + + // inflightItems counts work items accepted onto eventQueue but not yet fully + // handled. It is incremented before the channel send (so it can never lag the + // loop's receive) and decremented only after handleQueueItem returns. Using + // len(eventQueue) instead would read 0 in the window between the loop + // receiving an item and finishing it — a scrape there could satisfy a drain + // gate while a commit/push is still in flight. The depth gauge sums this with + // hasUnpushedWork, so it reads 0 only once every accepted item has been + // handled and nothing is retained. + inflightItems atomic.Int64 } // branchWorkerLogFirsts logs the first successful commit and push of a worker's @@ -256,11 +266,15 @@ func (w *BranchWorker) EnqueueFinalize(signal *FinalizeSignal) { if signal == nil { return } + // Increment before the send so inflightItems can never lag the loop's + // receive; roll back if the queue is full and the item is dropped. + w.inflightItems.Add(1) select { case w.eventQueue <- WorkItem{Finalize: signal}: w.Log.V(1).Info("Finalize signal enqueued") w.recordQueueDepth() default: + w.inflightItems.Add(-1) w.Log.Error(nil, "Event queue full, finalize signal dropped") signal.reply(FinalizeResult{Branch: w.Branch, Err: ErrFinalizeQueueFull}) } @@ -271,6 +285,9 @@ func (w *BranchWorker) enqueueRequest(request *WriteRequest) { return } item := WorkItem{Request: request} + // Increment before the send so inflightItems can never lag the loop's + // receive; roll back if the queue is full and the item is dropped. + w.inflightItems.Add(1) select { case w.eventQueue <- item: w.Log.V(1).Info("Write request enqueued", @@ -279,6 +296,7 @@ func (w *BranchWorker) enqueueRequest(request *WriteRequest) { "gitTarget", request.GitTargetName) w.recordQueueDepth() default: + w.inflightItems.Add(-1) w.Log.Error(nil, "Event queue full, request dropped", "events", len(request.Events), "mode", request.CommitMode, @@ -287,16 +305,18 @@ func (w *BranchWorker) enqueueRequest(request *WriteRequest) { } // recordQueueDepth publishes the current pending-work depth for this worker: -// the number of items still sitting in the event queue plus one when the event -// loop is holding unpushed work (a live open window or retained pending -// writes). It reads 0 only when the worker has fully drained. Safe to call from -// any goroutine: it reads the channel length and an atomic flag. No-op until the -// gauge is registered (e.g. in unit tests that never init the exporter). +// the number of accepted-but-not-yet-handled items (queued or actively being +// processed) plus one when the event loop is holding retained unpushed work (a +// live open window or pending writes). It reads 0 only when the worker has +// fully drained — every accepted item handled and nothing retained — so a +// drain gate cannot be satisfied while a commit/push is still in flight. Safe +// to call from any goroutine: it reads two atomics. No-op until the gauge is +// registered (e.g. in unit tests that never init the exporter). func (w *BranchWorker) recordQueueDepth() { if telemetry.BranchWorkerQueueDepth == nil { return } - depth := int64(len(w.eventQueue)) + depth := w.inflightItems.Load() if w.hasUnpushedWork.Load() { depth++ } @@ -568,6 +588,10 @@ func (l *branchWorkerEventLoop) run() { return case item := <-l.w.eventQueue: l.handleQueueItem(item) + // Decrement only after the item is fully handled; the post-handling + // open-window/pending-writes state is captured by syncQueueDepthMetric + // below, so there is no window where depth drops to 0 prematurely. + l.w.inflightItems.Add(-1) case <-commitC: l.commitTimer = nil l.finalizeOpenWindow() @@ -583,7 +607,8 @@ func (l *branchWorkerEventLoop) run() { // syncQueueDepthMetric refreshes the worker's unpushed-work flag from the loop's // authoritative state and republishes the depth gauge. Called once per loop // iteration (the loop goroutine is the only writer of hasUnpushedWork), so the -// gauge converges to 0 once the queue is empty and nothing is retained. +// gauge converges to 0 once every accepted item has been handled (inflightItems +// == 0) and nothing is retained (no open window, no pending writes). func (l *branchWorkerEventLoop) syncQueueDepthMetric() { l.w.hasUnpushedWork.Store(l.openWindow != nil || len(l.pendingWrites) > 0) l.w.recordQueueDepth() diff --git a/internal/git/branch_worker_metrics_test.go b/internal/git/branch_worker_metrics_test.go index f5d7e2029..5e799d90b 100644 --- a/internal/git/branch_worker_metrics_test.go +++ b/internal/git/branch_worker_metrics_test.go @@ -67,16 +67,16 @@ func TestRecordQueueDepth_DrainedReportsZero(t *testing.T) { assert.Equal(t, int64(0), depth) } -// recordQueueDepth must count both queued items and the in-flight unpushed-work -// flag, so the gauge reflects work the channel length alone cannot see. -func TestRecordQueueDepth_CountsQueuedAndInFlight(t *testing.T) { +// recordQueueDepth must count both accepted-but-unhandled items (in flight) and +// the retained unpushed-work flag, so the gauge reflects work the channel length +// alone cannot see — including an item already dequeued and being processed. +func TestRecordQueueDepth_CountsInflightAndRetained(t *testing.T) { reader, err := telemetry.InitTestExporter() require.NoError(t, err) w := newMetricsTestWorker() - // Two items sitting in the queue, plus retained unpushed work. - w.eventQueue <- WorkItem{} - w.eventQueue <- WorkItem{} + // Two items accepted but not yet fully handled, plus retained unpushed work. + w.inflightItems.Store(2) w.hasUnpushedWork.Store(true) w.recordQueueDepth() @@ -86,6 +86,26 @@ func TestRecordQueueDepth_CountsQueuedAndInFlight(t *testing.T) { assert.Equal(t, int64(3), depth) } +// recordQueueDepth must still report > 0 for an item that has been dequeued from +// the channel but is still being handled — the exact window where len(eventQueue) +// would read 0 and a drain gate could be falsely satisfied mid-commit. +func TestRecordQueueDepth_InflightItemDequeuedButUnhandled(t *testing.T) { + reader, err := telemetry.InitTestExporter() + require.NoError(t, err) + + w := newMetricsTestWorker() + // Simulate the loop having received the item (channel empty) while it is + // still being processed: inflight accounts for it, retained flag not yet set. + w.inflightItems.Store(1) + w.hasUnpushedWork.Store(false) + + w.recordQueueDepth() + + depth, ok := telemetry.CollectInt64Sum(reader, branchWorkerQueueDepthMetric, queueDepthLabels()) + require.True(t, ok, "expected a branch_worker_queue_depth sample") + assert.Equal(t, int64(1), depth, "an in-flight item must keep depth > 0 even with an empty channel") +} + // enqueueRequest must publish a non-zero depth as soon as the item lands on the // queue, closing the race where a poller reads a stale drained value. func TestEnqueueRequest_RecordsDepth(t *testing.T) { diff --git a/internal/telemetry/exporter.go b/internal/telemetry/exporter.go index c1b45594e..2f5288e8d 100644 --- a/internal/telemetry/exporter.go +++ b/internal/telemetry/exporter.go @@ -81,9 +81,11 @@ var ( // treat the name/labels as a public observability contract. TargetReconcileCompletedTotal metric.Int64Counter // BranchWorkerQueueDepth gauges pending work for a single branch worker: - // queued work items plus any committed-but-not-yet-pushed work the worker is - // still holding. It reads 0 only when the worker has fully drained (empty - // queue and nothing retained for replay). Labelled by {provider_namespace, + // accepted-but-not-yet-handled items (queued or actively being processed) + // plus any committed-but-not-yet-pushed work the worker is still holding. It + // reads 0 only when the worker has fully drained (every accepted item handled + // and nothing retained for replay), so it never reports drained while a + // commit/push is still in flight. Labelled by {provider_namespace, // provider_name, branch}; the namespace/name keys are prefixed to avoid the // reserved Prometheus pod-scrape target labels (see // TargetReconcileCompletedTotal). Load-bearing for the restart-snapshot e2e diff --git a/internal/watch/manager.go b/internal/watch/manager.go index c806c8aaa..c329f2448 100644 --- a/internal/watch/manager.go +++ b/internal/watch/manager.go @@ -1308,10 +1308,11 @@ func (m *Manager) emitSnapshotForRuleChange( // recordTargetReconcileCompleted increments the per-GitTarget reconcile counter // once its snapshot decision has been made and the resulting write request // submitted to the branch worker, tagged with the trigger that drove the pass. -// On a controller restart a delta over a pre-restart baseline is the signal that -// the new pod's snapshot reconcile reached the git write path — paired with a -// drained BranchWorkerQueueDepth it proves the post-restart snapshot has fully -// landed. No-op until the counter is registered. +// On a controller restart the new pod's counter starts at 0, so a per-pod +// `{pod=""} > 0` reading is the signal that the new pod's snapshot +// reconcile reached the git write path — paired with a drained +// BranchWorkerQueueDepth it proves the post-restart snapshot has fully landed. +// No-op until the counter is registered. func (m *Manager) recordTargetReconcileCompleted(gitDest types.ResourceReference, trigger string) { if telemetry.TargetReconcileCompletedTotal == nil { return diff --git a/test/e2e/tools/ginkgo-allure/main.go b/test/e2e/tools/ginkgo-allure/main.go new file mode 100644 index 000000000..76a8224b4 --- /dev/null +++ b/test/e2e/tools/ginkgo-allure/main.go @@ -0,0 +1,305 @@ +/* +SPDX-License-Identifier: Apache-2.0 + +Copyright 2025 ConfigButler + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// ginkgo-allure converts Ginkgo JSON reports into Allure result files. +package main + +import ( + "crypto/sha256" + "encoding/hex" + "encoding/json" + "flag" + "fmt" + "io" + "os" + "path/filepath" + "strconv" + "strings" + "time" + + "github.com/onsi/ginkgo/v2/types" +) + +const ( + exitUsage = 2 + usageInstruct = "usage: ginkgo-allure --output-dir ..." +) + +func main() { + os.Exit(mainRun(os.Args[1:], os.Stdout, os.Stderr)) +} + +func mainRun(args []string, stdout, stderr io.Writer) int { + fs := flag.NewFlagSet("ginkgo-allure", flag.ContinueOnError) + fs.SetOutput(stderr) + outputDir := fs.String("output-dir", "allure-results", "directory for generated Allure result files") + if err := fs.Parse(args); err != nil { + return exitUsage + } + if *outputDir == "" || fs.NArg() == 0 { + fmt.Fprintln(stderr, usageInstruct) + return exitUsage + } + count, err := run(*outputDir, fs.Args()) + if err != nil { + fmt.Fprintln(stderr, err) + return 1 + } + fmt.Fprintf(stdout, "wrote %d Allure result files to %s\n", count, *outputDir) + return 0 +} + +type allureResult struct { + UUID string `json:"uuid"` + HistoryID string `json:"historyId"` + TestCaseID string `json:"testCaseId"` + FullName string `json:"fullName"` + Name string `json:"name"` + Status string `json:"status"` + StatusDetails *allureStatusDetails `json:"statusDetails,omitempty"` + Stage string `json:"stage"` + Start int64 `json:"start,omitempty"` + Stop int64 `json:"stop,omitempty"` + Labels []allureLabel `json:"labels,omitempty"` + Parameters []allureParameter `json:"parameters,omitempty"` + Attachments []allureAttachment `json:"attachments,omitempty"` +} + +type allureStatusDetails struct { + Message string `json:"message,omitempty"` + Trace string `json:"trace,omitempty"` +} + +type allureLabel struct { + Name string `json:"name"` + Value string `json:"value"` +} + +type allureParameter struct { + Name string `json:"name"` + Value string `json:"value"` +} + +type allureAttachment struct { + Name string `json:"name"` + Source string `json:"source"` + Type string `json:"type"` +} + +func run(outputDir string, paths []string) (int, error) { + if err := os.MkdirAll(outputDir, 0o750); err != nil { + return 0, fmt.Errorf("create output dir %s: %w", outputDir, err) + } + + count := 0 + for _, path := range paths { + reports, err := readReports(path) + if err != nil { + return count, err + } + for reportIndex, report := range reports { + for specIndex, spec := range report.SpecReports { + if spec.LeafNodeType != types.NodeTypeIt { + continue + } + result := convertSpec(path, reportIndex, specIndex, report, spec) + if err := writeResult(outputDir, result, spec.CombinedOutput()); err != nil { + return count, err + } + count++ + } + } + } + return count, nil +} + +func readReports(path string) ([]types.Report, error) { + data, err := os.ReadFile(path) + if err != nil { + return nil, fmt.Errorf("open %s: %w", path, err) + } + var reports []types.Report + if err := json.Unmarshal(data, &reports); err != nil { + return nil, fmt.Errorf("parse %s: %w", path, err) + } + return reports, nil +} + +func convertSpec(path string, reportIndex, specIndex int, report types.Report, spec types.SpecReport) allureResult { + fullName := spec.FullText() + if fullName == "" { + fullName = spec.LeafNodeText + } + if fullName == "" { + fullName = fmt.Sprintf("%s#%d", filepath.Base(path), specIndex) + } + + start, stop := specTimes(report, spec) + result := allureResult{ + UUID: uuidFromParts(path, reportIndex, specIndex, fullName), + HistoryID: hashString("history", fullName), + TestCaseID: hashString("test-case", fullName), + FullName: fullName, + Name: resultName(spec), + Status: allureStatus(spec.State), + Stage: "finished", + Start: unixMillis(start), + Stop: unixMillis(stop), + Labels: labels(report, spec), + Parameters: []allureParameter{ + {Name: "ginkgo.parallel_process", Value: strconv.Itoa(spec.ParallelProcess)}, + {Name: "ginkgo.random_seed", Value: strconv.FormatInt(report.SuiteConfig.RandomSeed, 10)}, + }, + } + if details := statusDetails(spec); details != nil { + result.StatusDetails = details + } + return result +} + +func writeResult(outputDir string, result allureResult, combinedOutput string) error { + if strings.TrimSpace(combinedOutput) != "" { + source := result.UUID + "-attachment.txt" + if err := os.WriteFile(filepath.Join(outputDir, source), []byte(combinedOutput), 0o600); err != nil { + return fmt.Errorf("write attachment %s: %w", source, err) + } + result.Attachments = append(result.Attachments, allureAttachment{ + Name: "Ginkgo output", + Source: source, + Type: "text/plain", + }) + } + + data, err := json.MarshalIndent(result, "", " ") + if err != nil { + return fmt.Errorf("marshal result %s: %w", result.UUID, err) + } + name := result.UUID + "-result.json" + if err := os.WriteFile(filepath.Join(outputDir, name), append(data, '\n'), 0o600); err != nil { + return fmt.Errorf("write result %s: %w", name, err) + } + return nil +} + +func resultName(spec types.SpecReport) string { + if spec.LeafNodeText != "" { + return spec.LeafNodeText + } + return spec.FullText() +} + +func labels(report types.Report, spec types.SpecReport) []allureLabel { + out := []allureLabel{ + {Name: "language", Value: "go"}, + {Name: "framework", Value: "ginkgo"}, + } + if report.SuiteDescription != "" { + out = append(out, allureLabel{Name: "suite", Value: report.SuiteDescription}) + } + if report.SuitePath != "" { + out = append(out, allureLabel{Name: "package", Value: filepath.Base(report.SuitePath)}) + } + if len(spec.ContainerHierarchyTexts) > 0 { + out = append(out, allureLabel{Name: "parentSuite", Value: spec.ContainerHierarchyTexts[0]}) + } + if len(spec.ContainerHierarchyTexts) > 1 { + out = append(out, allureLabel{Name: "subSuite", Value: strings.Join(spec.ContainerHierarchyTexts[1:], " ")}) + } + for _, label := range spec.Labels() { + out = append(out, allureLabel{Name: "tag", Value: label}) + } + if spec.IsSerial { + out = append(out, allureLabel{Name: "tag", Value: "Serial"}) + } + return out +} + +func allureStatus(state types.SpecState) string { + switch { + case state == types.SpecStatePassed: + return "passed" + case state == types.SpecStateSkipped || state == types.SpecStatePending: + return "skipped" + case state == types.SpecStateFailed: + return "failed" + case state.Is(types.SpecStateAborted | types.SpecStatePanicked | types.SpecStateInterrupted | types.SpecStateTimedout): + return "broken" + default: + return "unknown" + } +} + +func statusDetails(spec types.SpecReport) *allureStatusDetails { + if spec.Failure.IsZero() { + return nil + } + trace := spec.Failure.Location.FullStackTrace + if trace == "" && spec.Failure.Location != (types.CodeLocation{}) { + trace = spec.Failure.Location.String() + } + if spec.Failure.ForwardedPanic != "" { + if trace != "" { + trace += "\n" + } + trace += spec.Failure.ForwardedPanic + } + return &allureStatusDetails{ + Message: spec.Failure.Message, + Trace: trace, + } +} + +func specTimes(report types.Report, spec types.SpecReport) (time.Time, time.Time) { + start := spec.StartTime + if start.IsZero() { + start = report.StartTime + } + stop := spec.EndTime + if stop.IsZero() && !start.IsZero() { + stop = start.Add(spec.RunTime) + } + return start, stop +} + +func unixMillis(t time.Time) int64 { + if t.IsZero() { + return 0 + } + return t.UnixNano() / int64(time.Millisecond) +} + +func uuidFromParts(parts ...any) string { + hash := hashParts(parts...) + hexed := hex.EncodeToString(hash[:16]) + return fmt.Sprintf("%s-%s-%s-%s-%s", hexed[0:8], hexed[8:12], hexed[12:16], hexed[16:20], hexed[20:32]) +} + +func hashString(parts ...any) string { + hash := hashParts(parts...) + return hex.EncodeToString(hash[:16]) +} + +func hashParts(parts ...any) [32]byte { + h := sha256.New() + for _, part := range parts { + _, _ = fmt.Fprintf(h, "%v\x00", part) + } + var out [32]byte + copy(out[:], h.Sum(nil)) + return out +} diff --git a/test/e2e/tools/ginkgo-allure/main_test.go b/test/e2e/tools/ginkgo-allure/main_test.go new file mode 100644 index 000000000..9e77f90ab --- /dev/null +++ b/test/e2e/tools/ginkgo-allure/main_test.go @@ -0,0 +1,361 @@ +/* +SPDX-License-Identifier: Apache-2.0 + +Copyright 2025 ConfigButler + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "bytes" + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/onsi/ginkgo/v2/types" +) + +func fixtureReport() []types.Report { + start := time.Date(2026, 5, 31, 12, 0, 0, 0, time.UTC) + return []types.Report{{ + SuitePath: "/workspaces/gitops-reverser/test/e2e", + SuiteDescription: "E2E Suite", + StartTime: start, + SuiteConfig: types.SuiteConfig{ + RandomSeed: 12345, + }, + SpecReports: []types.SpecReport{ + { + LeafNodeType: types.NodeTypeBeforeSuite, + LeafNodeText: "BeforeSuite", + }, + { + ContainerHierarchyTexts: []string{"Manager", "Smoke"}, + ContainerHierarchyLabels: [][]string{{"manager"}}, + LeafNodeType: types.NodeTypeIt, + LeafNodeText: "passes", + LeafNodeLabels: []string{"smoke"}, + State: types.SpecStatePassed, + StartTime: start, + EndTime: start.Add(2 * time.Second), + RunTime: 2 * time.Second, + ParallelProcess: 2, + CapturedGinkgoWriterOutput: "useful output\n", + }, + { + ContainerHierarchyTexts: []string{"Manager"}, + LeafNodeType: types.NodeTypeIt, + LeafNodeText: "fails", + State: types.SpecStateFailed, + StartTime: start.Add(3 * time.Second), + RunTime: time.Second, + Failure: types.Failure{ + Message: "expected success", + Location: types.CodeLocation{ + FileName: "test/e2e/example_test.go", + LineNumber: 42, + FullStackTrace: "stack trace", + }, + }, + }, + { + ContainerHierarchyTexts: []string{"Manager"}, + LeafNodeType: types.NodeTypeIt, + LeafNodeText: "times out", + State: types.SpecStateTimedout, + }, + { + ContainerHierarchyTexts: []string{"Manager"}, + LeafNodeType: types.NodeTypeIt, + LeafNodeText: "skips", + State: types.SpecStateSkipped, + }, + }, + }} +} + +func TestRun_ConvertsGinkgoJSONToAllureResults(t *testing.T) { + reportPath := writeFixtureReport(t, fixtureReport()) + outputDir := filepath.Join(t.TempDir(), "allure-results") + + count, err := run(outputDir, []string{reportPath}) + if err != nil { + t.Fatalf("run: %v", err) + } + if count != 4 { + t.Fatalf("count: got %d, want 4", count) + } + + results := readAllureResults(t, outputDir) + if len(results) != 4 { + t.Fatalf("results: got %d, want 4", len(results)) + } + + assertPassingResult(t, outputDir, findResult(t, results, "passes")) + assertFailedResult(t, findResult(t, results, "fails")) + + timedOut := findResult(t, results, "times out") + if timedOut.Status != "broken" { + t.Errorf("timed out status: got %q", timedOut.Status) + } + + skipped := findResult(t, results, "skips") + if skipped.Status != "skipped" { + t.Errorf("skipped status: got %q", skipped.Status) + } +} + +func assertPassingResult(t *testing.T, outputDir string, result allureResult) { + t.Helper() + if result.Status != "passed" { + t.Errorf("passed status: got %q", result.Status) + } + if result.FullName != "Manager Smoke passes" { + t.Errorf("fullName: got %q", result.FullName) + } + assertHasLabel(t, result, "framework", "ginkgo") + assertHasLabel(t, result, "language", "go") + assertHasLabel(t, result, "suite", "E2E Suite") + assertHasLabel(t, result, "package", "e2e") + assertHasLabel(t, result, "parentSuite", "Manager") + assertHasLabel(t, result, "subSuite", "Smoke") + assertHasLabel(t, result, "tag", "manager") + assertHasLabel(t, result, "tag", "smoke") + assertHasParameter(t, result, "ginkgo.parallel_process", "2") + assertHasParameter(t, result, "ginkgo.random_seed", "12345") + if result.Start == 0 || result.Stop == 0 || result.Stop <= result.Start { + t.Errorf("bad timing: start=%d stop=%d", result.Start, result.Stop) + } + if len(result.Attachments) != 1 { + t.Fatalf("attachments: got %d, want 1", len(result.Attachments)) + } + attachment, err := os.ReadFile(filepath.Join(outputDir, result.Attachments[0].Source)) + if err != nil { + t.Fatalf("read attachment: %v", err) + } + if string(attachment) != "useful output\n" { + t.Errorf("attachment: got %q", string(attachment)) + } +} + +func assertFailedResult(t *testing.T, result allureResult) { + t.Helper() + if result.Status != "failed" { + t.Errorf("failed status: got %q", result.Status) + } + if result.StatusDetails == nil || result.StatusDetails.Message != "expected success" { + t.Fatalf("status details missing failure message: %#v", result.StatusDetails) + } + if result.StatusDetails.Trace != "stack trace" { + t.Errorf("trace: got %q", result.StatusDetails.Trace) + } +} + +func TestMainRun_ExitCodes(t *testing.T) { + good := writeFixtureReport(t, fixtureReport()) + cases := []struct { + name string + args []string + wantCode int + }{ + {name: "no args", args: nil, wantCode: exitUsage}, + {name: "empty output dir", args: []string{"--output-dir", "", good}, wantCode: exitUsage}, + {name: "bad flag", args: []string{"-not-a-flag"}, wantCode: exitUsage}, + {name: "missing report", args: []string{"--output-dir", t.TempDir(), "/no/such/file"}, wantCode: 1}, + {name: "success", args: []string{"--output-dir", t.TempDir(), good}, wantCode: 0}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + var stdout, stderr bytes.Buffer + got := mainRun(tc.args, &stdout, &stderr) + if got != tc.wantCode { + t.Errorf("mainRun: code=%d, want %d (stderr=%s)", got, tc.wantCode, stderr.String()) + } + }) + } +} + +func TestRun_BadJSON(t *testing.T) { + path := filepath.Join(t.TempDir(), "bad.json") + if err := os.WriteFile(path, []byte("not json"), 0o600); err != nil { + t.Fatalf("write bad fixture: %v", err) + } + _, err := run(t.TempDir(), []string{path}) + if err == nil { + t.Fatal("expected invalid JSON error") + } +} + +func TestRun_OutputDirCreateError(t *testing.T) { + reportPath := writeFixtureReport(t, fixtureReport()) + outputDir := filepath.Join(t.TempDir(), "not-a-dir") + if err := os.WriteFile(outputDir, []byte("file"), 0o600); err != nil { + t.Fatalf("write output placeholder: %v", err) + } + _, err := run(outputDir, []string{reportPath}) + if err == nil { + t.Fatal("expected output dir creation error") + } +} + +func TestConvertSpec_FallbacksAndFailureDetails(t *testing.T) { + spec := types.SpecReport{ + LeafNodeType: types.NodeTypeIt, + State: types.SpecStatePanicked, + Failure: types.Failure{ + Message: "boom", + Location: types.CodeLocation{ + FileName: "test/e2e/example_test.go", + LineNumber: 42, + }, + ForwardedPanic: "panic payload", + }, + } + + result := convertSpec("/tmp/ginkgo-report-empty.json", 0, 7, types.Report{}, spec) + + if result.FullName != "ginkgo-report-empty.json#7" { + t.Errorf("fullName: got %q", result.FullName) + } + if result.Name != "" { + t.Errorf("name: got %q, want empty fallback", result.Name) + } + if result.Status != "broken" { + t.Errorf("status: got %q", result.Status) + } + if result.Start != 0 || result.Stop != 0 { + t.Errorf("timing: start=%d stop=%d, want zero values", result.Start, result.Stop) + } + if result.StatusDetails == nil { + t.Fatal("missing status details") + } + if result.StatusDetails.Message != "boom" { + t.Errorf("message: got %q", result.StatusDetails.Message) + } + if result.StatusDetails.Trace != "test/e2e/example_test.go:42\npanic payload" { + t.Errorf("trace: got %q", result.StatusDetails.Trace) + } +} + +func TestWriteResult_NoAttachment(t *testing.T) { + outputDir := t.TempDir() + result := allureResult{ + UUID: "no-attachment", + Name: "passes", + Status: "passed", + Stage: "finished", + } + if err := writeResult(outputDir, result, " \n\t"); err != nil { + t.Fatalf("writeResult: %v", err) + } + + results := readAllureResults(t, outputDir) + if len(results) != 1 { + t.Fatalf("results: got %d, want 1", len(results)) + } + if len(results[0].Attachments) != 0 { + t.Errorf("attachments: got %#v, want none", results[0].Attachments) + } +} + +func TestStatusMappings(t *testing.T) { + cases := map[types.SpecState]string{ + types.SpecStatePassed: "passed", + types.SpecStateSkipped: "skipped", + types.SpecStatePending: "skipped", + types.SpecStateFailed: "failed", + types.SpecStatePanicked: "broken", + types.SpecStateAborted: "broken", + types.SpecStateInterrupted: "broken", + types.SpecStateTimedout: "broken", + types.SpecStateInvalid: "unknown", + } + for state, want := range cases { + if got := allureStatus(state); got != want { + t.Errorf("allureStatus(%s) = %q, want %q", state, got, want) + } + } +} + +func writeFixtureReport(t *testing.T, reports []types.Report) string { + t.Helper() + data, err := json.Marshal(reports) + if err != nil { + t.Fatalf("marshal report: %v", err) + } + path := filepath.Join(t.TempDir(), "ginkgo-report.json") + if err := os.WriteFile(path, data, 0o600); err != nil { + t.Fatalf("write report: %v", err) + } + return path +} + +func readAllureResults(t *testing.T, outputDir string) []allureResult { + t.Helper() + entries, err := os.ReadDir(outputDir) + if err != nil { + t.Fatalf("read output dir: %v", err) + } + results := make([]allureResult, 0, len(entries)) + for _, entry := range entries { + if !strings.HasSuffix(entry.Name(), "-result.json") { + continue + } + data, err := os.ReadFile(filepath.Join(outputDir, entry.Name())) + if err != nil { + t.Fatalf("read result: %v", err) + } + var result allureResult + if err := json.Unmarshal(data, &result); err != nil { + t.Fatalf("parse result: %v", err) + } + results = append(results, result) + } + return results +} + +func findResult(t *testing.T, results []allureResult, name string) allureResult { + t.Helper() + for _, result := range results { + if result.Name == name { + return result + } + } + t.Fatalf("missing result %q in %#v", name, results) + return allureResult{} +} + +func assertHasLabel(t *testing.T, result allureResult, name, value string) { + t.Helper() + for _, label := range result.Labels { + if label.Name == name && label.Value == value { + return + } + } + t.Errorf("missing label %s=%s in %#v", name, value, result.Labels) +} + +func assertHasParameter(t *testing.T, result allureResult, name, value string) { + t.Helper() + for _, parameter := range result.Parameters { + if parameter.Name == name && parameter.Value == value { + return + } + } + t.Errorf("missing parameter %s=%s in %#v", name, value, result.Parameters) +} From 3d14322dc7d1113854a7e4b4ef63eff5dcce32c5 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Sun, 31 May 2026 20:32:58 +0000 Subject: [PATCH 21/40] chore: will this fix the broken build? --- .github/workflows/ci.yml | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d34b6469a..397b4b050 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -401,6 +401,15 @@ jobs: gh_pages: gh-pages allure_history: allure-history + - name: Fix Allure artifact permissions + if: always() && steps.allure-results.outputs.has_results == 'true' + run: | + # allure-results and allure-report are produced by root-owned + # containers (docker run / the Docker-based report action), so the + # native upload-artifact step cannot read them. Hand ownership back + # to the runner user before uploading. + sudo chown -R "$(id -u):$(id -g)" allure-results allure-report + - name: Upload Allure report artifact if: always() && steps.allure-results.outputs.has_results == 'true' uses: actions/upload-artifact@v7 From d90e38a9ce683678509c78d78d07d0d35310d1a1 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Mon, 1 Jun 2026 06:58:19 +0000 Subject: [PATCH 22/40] chore: attacking that instability with more logs to learn why it's happening --- .devcontainer/Dockerfile | 16 +- .github/workflows/ci.yml | 60 ++++ AGENTS.md | 10 + Taskfile-build.yml | 5 + docs/design/e2e-ci-stability-report.md | 369 +++++++++++++++++++++++++ 5 files changed, 459 insertions(+), 1 deletion(-) create mode 100644 docs/design/e2e-ci-stability-report.md diff --git a/.devcontainer/Dockerfile b/.devcontainer/Dockerfile index ee3d6953e..14a5087f2 100644 --- a/.devcontainer/Dockerfile +++ b/.devcontainer/Dockerfile @@ -49,6 +49,7 @@ RUN apt-get update \ # FLUX_OPERATOR_VERSION -> https://github.com/controlplaneio-fluxcd/flux-operator # TASK_VERSION -> https://github.com/go-task/task/releases # TILT_VERSION -> https://github.com/tilt-dev/tilt/releases +# ACTIONLINT_VERSION -> https://github.com/rhysd/actionlint/releases ENV PATH="/go/bin:/usr/local/go/bin:${PATH}" \ KUBECTL_VERSION=v1.36.1 \ @@ -60,7 +61,8 @@ ENV PATH="/go/bin:/usr/local/go/bin:${PATH}" \ FLUX_VERSION=2.8.8 \ FLUX_OPERATOR_VERSION=0.50.0 \ TASK_VERSION=v3.51.1 \ - TILT_VERSION=v0.37.3 + TILT_VERSION=v0.37.3 \ + ACTIONLINT_VERSION=1.7.12 # https://github.com/fluxcd/flux2/releases # https://fluxoperator.dev/ @@ -119,6 +121,18 @@ RUN curl -fsSL "https://github.com/go-task/task/releases/download/${TASK_VERSION | tar -xzO task > /usr/local/bin/task \ && chmod +x /usr/local/bin/task +# Install actionlint (static linter for GitHub Actions workflows) +RUN asset="actionlint_${ACTIONLINT_VERSION}_linux_amd64.tar.gz" \ + && base="https://github.com/rhysd/actionlint/releases/download/v${ACTIONLINT_VERSION}" \ + && tmpdir="$(mktemp -d)" \ + && curl -fsSL "${base}/${asset}" -o "${tmpdir}/${asset}" \ + && curl -fsSL "${base}/actionlint_${ACTIONLINT_VERSION}_checksums.txt" -o "${tmpdir}/checksums.txt" \ + && cd "${tmpdir}" \ + && grep " ${asset}$" checksums.txt | sha256sum -c - \ + && tar -xzf "${asset}" actionlint \ + && install -m 0755 actionlint /usr/local/bin/actionlint \ + && rm -rf "${tmpdir}" + # Set working directory WORKDIR /workspaces diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 397b4b050..2c17accaf 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -300,6 +300,40 @@ jobs: run: | echo "${{ secrets.GITHUB_TOKEN }}" | docker login ${{ env.REGISTRY }} -u ${{ github.actor }} --password-stdin + - name: Start disk usage sampler + run: | + # Sample free space on / every 10s for the whole e2e run so we can see + # the peak (not just the post-run value). /var/lib/docker — k3d node + # volumes, pulled images, build cache — lives on this filesystem. + ( while true; do + printf '%s avail=%s\n' "$(date -u +%H:%M:%S)" \ + "$(df --output=avail -BM / | tail -1 | tr -d ' ')" + sleep 10 + done ) > "df-samples-${{ matrix.name }}.log" 2>/dev/null & + echo "SAMPLER_PID=$!" >> "$GITHUB_ENV" + + - name: Bring up cluster + Flux services (explicit, streaming logs) + run: | + # Install-mode-/image-independent slice of prepare-e2e. Runs the + # slow, hang-prone cluster + Flux + services bring-up here so its + # logs stream live, instead of being buffered inside Ginkgo's + # SynchronizedBeforeSuite (where a stall just looks like a silent + # 15m hang ending in a goroutine dump). The in-suite prepare-e2e + # no-ops this part because the stamps already exist. + docker run --rm \ + --network host \ + -v "${GITHUB_WORKSPACE}:${{ env.CI_WORKDIR }}" \ + -v /var/run/docker.sock:/var/run/docker.sock \ + -w "${{ env.CI_WORKDIR }}" \ + -e IMAGE_DELIVERY_MODE=${{ env.IMAGE_DELIVERY_MODE }} \ + -e K3D_AGENT_COUNT=${{ matrix.k3d_agent_count }} \ + -e HOST_PROJECT_PATH=${{ github.workspace }} \ + ${{ env.CI_CONTAINER }} \ + bash -c " + git config --global --add safe.directory ${{ env.CI_WORKDIR }} + task _services-ready + " + - name: Run E2E tests in CI container run: | docker run --rm \ @@ -318,6 +352,32 @@ jobs: ${{ matrix.script }} " + - name: Report disk usage (peak + final) + if: always() + run: | + kill "${SAMPLER_PID}" 2>/dev/null || true + samples="df-samples-${{ matrix.name }}.log" + { + echo "### Disk usage (${{ matrix.name }})" + echo + echo "Lowest free space on \`/\` during the run:" + echo '```' + sort -t= -k2 -n "${samples}" 2>/dev/null | head -1 || echo "(no samples captured)" + echo '```' + echo "Final state:" + echo '```' + df -h / + echo + docker system df + echo '```' + echo "
Full df sample timeline" + echo + echo '```' + cat "${samples}" 2>/dev/null || echo "(no samples captured)" + echo '```' + echo "
" + } >> "$GITHUB_STEP_SUMMARY" + - name: Summarize E2E timing reports if: always() run: | diff --git a/AGENTS.md b/AGENTS.md index b0322c4f3..773315704 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,6 +24,10 @@ task test # Must pass all unit tests with >90% coverage task test-e2e # Must pass end-to-end tests ``` +If you change a GitHub Actions workflow, also run `task lint-actions`, which lints +`.github/workflows/ci.yml` with `actionlint`. Both `actionlint` and `golangci-lint` +ship in the devcontainer image. + ## PRE-IMPLEMENTATION BEHAVIOR 1. **Check Docker availability for e2e tests**: Before running `task test-e2e`, verify Docker is running with `docker info` or ask user to start Docker daemon if needed @@ -64,6 +68,12 @@ task test-e2e # Must pass end-to-end tests - Add race condition protection - Use temporary directories for testing +### CI Workflows (`.github/workflows/`) +- After editing any workflow, run `task lint-actions` to catch errors with + `actionlint` before pushing +- A workflow-only change still counts as a CI/config change, so it is **not** + covered by the markdown/docs-only validation exception + ## TESTING REQUIREMENTS - Write unit tests with >90% coverage diff --git a/Taskfile-build.yml b/Taskfile-build.yml index 8b737594c..022dfff04 100644 --- a/Taskfile-build.yml +++ b/Taskfile-build.yml @@ -128,6 +128,11 @@ tasks: cmds: - '{{.GOLANGCI_LINT}} run' + lint-actions: + desc: Lint GitHub Actions workflows with actionlint + cmds: + - actionlint .github/workflows/ci.yml + lint-fix: desc: Run golangci-lint and perform fixes cmds: diff --git a/docs/design/e2e-ci-stability-report.md b/docs/design/e2e-ci-stability-report.md new file mode 100644 index 000000000..07c2c1cf4 --- /dev/null +++ b/docs/design/e2e-ci-stability-report.md @@ -0,0 +1,369 @@ +# E2E CI stability — how the setup works & where it hurts + +A walkthrough of how the e2e job in [`.github/workflows/ci.yml`](../../.github/workflows/ci.yml) +runs today, a check of your assumptions, and concrete answers to the two +questions (run `prepare-e2e` up front; observe disk usage). + +--- + +## 1. How a run is wired today + +### 1.1 The `e2e` job and its matrix + +``` +e2e (runs-on: ubuntu-latest) + ├─ matrix leg "quickstart" → E2E_GINKGO_PROCS=2, K3D_AGENT_COUNT=1, needs_artifact=true + │ script: INSTALL_MODE=helm task test-e2e-quickstart-helm + │ → task test-image-refresh + │ → INSTALL_MODE=plain-manifests-file task test-e2e-quickstart-manifest + └─ matrix leg "full" → E2E_GINKGO_PROCS=1, K3D_AGENT_COUNT=1, needs_artifact=false + script: task test-e2e-full +``` + +`needs: [build-ci-container, docker-build, lint-helm]` — so by the time e2e +starts, the CI base image, the project image, and the release bundle +(`install.yaml` + `gitops-reverser.tgz`) already exist. + +Each matrix leg runs the actual suite inside the CI container via +**Docker-outside-of-Docker**: + +```yaml +docker run --rm --network host \ + -v "${GITHUB_WORKSPACE}:/workspaces/gitops-reverser" \ + -v /var/run/docker.sock:/var/run/docker.sock \ + ... ${CI_CONTAINER} bash -c "" +``` + +The container talks to the **runner's** Docker daemon (the mounted socket), and +that same daemon is what k3d uses to create the cluster nodes. So everything — +the CI container, k3d node containers, the project image, and every Flux-managed +service image — lands on the **one** runner VM's disk. + +### 1.2 What `task test-e2e-*` actually does + +The Task targets ([`test/e2e/Taskfile.yml`](../../test/e2e/Taskfile.yml)) don't +prepare the cluster themselves — they just invoke Ginkgo: + +``` +go run .../ginkgo --procs=$E2E_GINKGO_PROCS --timeout=$E2E_GO_TEST_TIMEOUT ... ./test/e2e/ +``` + +Cluster prep happens **inside** the suite. `SynchronizedBeforeSuite` (process #1 +only) calls `prepareE2EClusterOnce()` +([`e2e_suite_test.go:82`](../../test/e2e/e2e_suite_test.go#L82)), which shells out +to `task prepare-e2e` with the leg's `CTX / INSTALL_MODE / NAMESPACE / +INSTALL_NAME`. + +### 1.3 The `prepare-e2e` dependency graph + +`prepare-e2e` is a go-task DAG, every node gated by a stamp file under +`.stamps/cluster//`. Stamps make it **idempotent**: a node whose stamp +already exists and whose sources are unchanged is skipped. The chain +(leaf → root): + +``` +prepare-e2e + ├─ _prepare-e2e-ready + │ ├─ _controller-deployed ── _image-loaded ── _project-image-ready (load-image.sh; pull mode in CI) + │ │ └─ _cluster-ready ── start-cluster.sh (k3d create) + │ ├─ _webhook-tls-ready ─┐ + │ ├─ _sops-secret-applied ┤ + │ └─ _aggregated-api-ready┘ + │ └─ _services-ready ── _flux-setup-ready ── _flux-installed ── _cluster-ready + └─ portforward-ensure (runs after the ready stamp, never in parallel with it) +``` + +The genuinely slow / hang-prone nodes: + +| Node | What it waits on | Timeout | +|------|------------------|---------| +| `_cluster-ready` | `start-cluster.sh`: k3d create, all nodes `Ready` | 60×2s health loop | +| `_flux-installed` | flux-operator install + FluxInstance + Flux deployments `Available` | `FLUX_WAIT_TIMEOUT=500s` | +| `_flux-setup-ready` | every HelmRelease/Kustomization (gitea, valkey, prometheus, traefik…) `Ready` | `FLUX_SERVICES_WAIT_TIMEOUT=600s` per resource | + +The whole Ginkgo run — prepare **plus** the specs — is bounded by one timeout: +`E2E_GO_TEST_TIMEOUT=15m` (smoke/quickstart) or `E2E_FULL_TIMEOUT=30m` (full). + +### 1.4 Why it "goes silent for ~15 min, then green on retry" + +This is the most important mechanical detail. `prepare-e2e` runs **inside +Ginkgo's `SynchronizedBeforeSuite`**, and its output is written to +`GinkgoWriter`. In `-v` (non-streaming) mode Ginkgo **buffers** that writer and +only flushes it when the node finishes or fails. So while the cluster + Flux + +services are coming up, the CI log shows nothing. + +If any readiness wait stalls (a HelmRelease that never reconciles, a node that +never goes `Ready`, a slow/stalled image pull), prepare sits there until the +outer **15m Ginkgo timeout** fires — at which point you get a goroutine dump +instead of a useful "service X never became ready" message. A re-run starts from +a clean VM and usually wins the race, which is exactly the signature of a +**flaky readiness/resource-contention problem**, not a hard deterministic +failure. + +--- + +## 2. Your assumptions, checked + +> **Every job (incl. every matrix leg) gets a new runner.** ✅ Correct. +GitHub provisions a fresh VM per job, and each matrix combination is its own job. +The two e2e legs never share disk, Docker daemon, or `.stamps`. + +> **Public-repo runners: max 20 concurrent, 4 vCPU / 16 GB / 14 GB SSD.** ✅ +Matches the current standard free Linux runner (4 vCPU, 16 GB RAM, 14 GB SSD) +and the 20-concurrent-job ceiling for the free tier. The **14 GB SSD is the +squeeze** — the runner image already pre-consumes a chunk of it, and *everything* +(CI container, k3d nodes, project image, all Flux service images, buildx/Go +caches) shares that one disk. + +> **Biggest chance is running out of disk.** ⚠️ Plausible but **not yet proven — +and the symptom argues against it.** A full disk almost always fails *loudly and +fast* ("no space left on device"), not with a 15-minute silent hang that +disappears on retry. The silent-hang-then-green pattern points more at a +**readiness wait stalling** (Flux service contention, image-pull stall, node not +Ready). Disk pressure can *cause* those stalls (e.g. kubelet +`DiskPressure` taint → pods won't schedule → HelmRelease never Ready), so it's +worth ruling in or out — but **measure before you commit to that theory.** That's +exactly what §4 adds. + +> **Set `K3D_AGENT_COUNT=1`.** ✅ Already done — both matrix legs pass +`k3d_agent_count: "1"` (1 server + 1 agent = 2 nodes). You could go to **0 +agents** (single-node; k3s schedules workloads on the server) to shave one more +node-container's worth of RAM/disk/inotify. Trade-off: less parallelism headroom +for `--procs=2`. Worth trying on the quickstart leg if measurements show +resource pressure. + +> **Spin up 4 e2e legs and run more in parallel.** ◑ Mixed. More legs = more +*isolated* runners, so wall-clock can drop **if you split specs across them**. +But it does **not** reduce per-leg disk/RAM pressure — each new leg re-pulls the +full image set onto its own 14 GB disk. If disk/resource pressure is the real +root cause, adding legs spreads load but each leg is just as likely to hit it. +Fix the per-leg footprint first (measure → free disk → maybe 0 agents), *then* +fan out for speed. + +--- + +## 3. Question 1 — bring the cluster up explicitly first + +**Recommended: yes — and target `_services-ready`, not the full `prepare-e2e`.** +Add an explicit cluster-bring-up step *before* the test `docker run`, in the same +container + workspace volume, with logs streaming live. Keep the in-Ginkgo +`prepareE2EClusterOnce()` call as-is — because the DAG is stamp-gated, it then +**no-ops** the parts already done. + +### Why `_services-ready` is the right target (not `prepare-e2e`) + +The hang is in *waiting for the Flux install + Flux-managed services to become +ready*. `_services-ready` is precisely that subtree and nothing more: + +``` +_services-ready + └─ _flux-setup-ready ── _flux-installed ── _cluster-ready (k3d create) +``` + +It deliberately stops short of `_image-loaded`, `_controller-deployed`, and the +whole install-mode tail. That gives two concrete advantages over running +`prepare-e2e`: + +- **No install-mode variables to decide.** The entire `_services-ready` chain is + independent of `INSTALL_MODE` / `NAMESPACE` / `INSTALL_NAME`; it needs only + `CTX`, which already defaults to `k3d-gitops-reverser-test-e2e`. So the step is + the same for *both* matrix legs — no `helm` vs `plain-manifests-file` vs + `config-dir` branching, and no risk of priming the wrong install tail. +- **It's exactly the hang surface.** Cluster create + Flux operator/instance + + every HelmRelease/Kustomization `Ready` wait — the three slow nodes from §1.3 + — and nothing you don't care about debugging right now. + +The image load and controller deploy stay inside Ginkgo's `prepare-e2e`, which is +fine: those aren't where it hangs, and `_cluster-ready`/`_flux-*`/`_services-ready` +are already stamped by the time Ginkgo runs, so it skips straight to them. + +Why this kills the symptom either way: run **outside** Ginkgo, the output +**streams** instead of being buffered to `GinkgoWriter`, so you see *which* node +hangs in real time instead of a 15-min silence + goroutine dump — and a genuine +bring-up failure fails a short, clearly-named CI step instead of masquerading as +a test timeout. + +### Two ways to invoke it + +`task _services-ready` works **today** — no task in +[`test/e2e/Taskfile.yml`](../../test/e2e/Taskfile.yml) is marked +`internal: true`, so the leading `_` is only a naming convention, not a runtime +guard. But depending on a `_`-prefixed target from CI is fragile (a future +`internal: true` or rename breaks CI silently). Cheapest hardening is a one-line +public wrapper: + +```yaml +# test/e2e/Taskfile.yml + prepare-cluster: + desc: Bring up the e2e cluster + Flux + Flux-managed services (install-mode independent) + cmds: + - task: _services-ready +``` + +CI step (new, before "Run E2E tests in CI container"), identical for both legs: + +```yaml +- name: Bring up cluster + Flux services (explicit, streaming logs) + run: | + docker run --rm --network host \ + -v "${GITHUB_WORKSPACE}:${{ env.CI_WORKDIR }}" \ + -v /var/run/docker.sock:/var/run/docker.sock \ + -w "${{ env.CI_WORKDIR }}" \ + -e IMAGE_DELIVERY_MODE=${{ env.IMAGE_DELIVERY_MODE }} \ + -e K3D_AGENT_COUNT=${{ matrix.k3d_agent_count }} \ + -e HOST_PROJECT_PATH=${{ github.workspace }} \ + ${{ env.CI_CONTAINER }} \ + bash -c " + git config --global --add safe.directory ${{ env.CI_WORKDIR }} + task prepare-cluster # or: task _services-ready + " +``` + +Note this step needs **no `PROJECT_IMAGE`, no `INSTALL_MODE`** — that's the whole +point of stopping at `_services-ready`. Keep `CTX` at its default (or pass the +same value the suite uses) so the stamps match and Ginkgo skips the work. + +### 3.1 Why the boundary stays at `_services-ready` (and not `_aggregated-api-ready`) + +A natural question is whether to push the explicit pre-step one node further, to +`_aggregated-api-ready`. **No — that node is not install-mode- or +image-independent.** Its full subtree: + +``` +_aggregated-api-ready + └─ _aggregated-api-webhook-kubeconfig-ready + ├─ _services-ready ← cluster + Flux + all flux services + └─ _webhook-tls-ready + └─ _controller-deployed ← install-mode-specific + project image + ├─ install (INSTALL_MODE: helm / plain-manifests-file / config-dir) + └─ _image-loaded (needs PROJECT_IMAGE) +``` + +Folding `_aggregated-api-ready` into the pre-step would re-introduce exactly the +two things stopping at `_services-ready` lets us avoid — choosing `INSTALL_MODE` +and passing `PROJECT_IMAGE` — and would tie the step to one matrix leg's mode +(the quickstart leg switches modes mid-run). So `_services-ready` is the last +node that is both **install-mode-agnostic** and **image-agnostic**; one step +further and that property is gone. The aggregated-api bring-up keeps running +inside the in-Ginkgo `prepare-e2e`, where it belongs. + +### 3.2 Observation: `_services-ready` is an all-or-nothing barrier (future refactor) + +`_aggregated-api-ready` *functionally* needs only **cert-manager** and the +**aggregated-api** HelmRelease — its commands only wait on the two cert-manager +`Certificate`s, the `aggregated-api` namespace deployments, and the `wardle` +APIService/discovery. It does **not** need gitea, valkey, prometheus-operator, +kro, reflector, or ingress. But it depends on `_services-ready`, and +[`hack/e2e/wait-flux-services.sh`](../../hack/e2e/wait-flux-services.sh) holds +that single gate until **every** HelmRelease/Kustomization is `Ready`. + +Consequence for stability: `_services-ready` is a coarse barrier. If *any one* +Flux service is slow or flaky to reconcile, it blocks aggregated-api, the +controller install, and effectively the whole suite — even specs that never +touch that service. A finer-grained dependency (e.g. aggregated-api waiting only +on cert-manager + its own release) would isolate such failures. + +**For now we are not refactoring this** — start with the immediate +`_services-ready` pre-step and let the streaming logs + disk metrics tell us +whether a specific service is the culprit. The barrier-narrowing is a candidate +follow-up *if* the data shows a single slow service is gating everything. + +--- + +## 4. Question 2 — observe disk usage (after the run, on failure, and the peak) + +Three layers, cheap to add, all `if: always()` so they fire on failure too. + +### 4.1 Snapshot after the run (and on failure) + +```yaml +- name: Disk & Docker usage (post-run) + if: always() + run: | + echo "## Disk after ${{ matrix.name }}" >> "$GITHUB_STEP_SUMMARY" + { echo '```'; df -h; echo; docker system df -v; echo '```'; } >> "$GITHUB_STEP_SUMMARY" + echo "### Biggest consumers" + sudo du -xh --max-depth=1 /var/lib/docker 2>/dev/null | sort -rh | head + du -sh .stamps 2>/dev/null || true +``` + +`df -h` tells you headroom; `docker system df -v` breaks it down by +images / containers / build cache / **volumes** (k3d node data lives here). + +### 4.2 Capture the *peak* with a background sampler + +A post-run snapshot misses the peak (k3d image imports + Flux pulls spike +mid-prepare, then settle). Sample throughout instead: + +```yaml +- name: Start disk sampler + run: | + ( while true; do + printf '%s %s\n' "$(date -u +%H:%M:%S)" \ + "$(df --output=avail -BM / | tail -1 | tr -d ' ')" + sleep 10 + done ) > df-samples-${{ matrix.name }}.log & + echo "SAMPLER_PID=$!" >> "$GITHUB_ENV" + +# ... prepare + test steps ... + +- name: Stop sampler & report peak + if: always() + run: | + kill "${SAMPLER_PID}" 2>/dev/null || true + echo "### Min free space on / during ${{ matrix.name }}" >> "$GITHUB_STEP_SUMMARY" + sort -k2 -n df-samples-${{ matrix.name }}.log | head -1 >> "$GITHUB_STEP_SUMMARY" + { echo '```'; cat df-samples-${{ matrix.name }}.log; echo '```'; } >> "$GITHUB_STEP_SUMMARY" + +- name: Upload disk samples + if: always() + uses: actions/upload-artifact@v7 + with: + name: e2e-disk-samples-${{ matrix.name }} + path: df-samples-${{ matrix.name }}.log + if-no-files-found: ignore +``` + +Lowest "avail" line = your true low-water mark. If it never drops below a few +hundred MB, disk is **not** your problem and you should look at Flux/service +readiness instead. + +### 4.3 If §4 confirms disk pressure — reclaim space first + +GitHub runners ship with large unused toolchains. A one-liner up front buys +~10–30 GB before the cluster is even created: + +```yaml +- name: Free disk space + run: | + sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc \ + /opt/hostedtoolcache/CodeQL /usr/local/.ghcup + sudo docker image prune -af + df -h / +``` + +(or use `jlumbroso/free-disk-space` / `easimon/maximize-build-space`.) + +--- + +## 5. Recommended order of attack + +1. **Add the instrumentation (§4.1–4.2) — do this first.** One run tells you + whether disk is actually the bottleneck. Cheap, non-invasive, settles the + open question instead of guessing. +2. **Add the explicit streaming `task _services-ready` / `prepare-cluster` step + (§3).** Independently valuable: turns the silent 15-min hang into a readable, + correctly-attributed failure, needs no install-mode variables, and the + in-Ginkgo `prepare-e2e` call still no-ops the cluster/Flux/services part. +3. **Then, based on the data:** + - Disk-bound → reclaim space (§4.3), consider `K3D_AGENT_COUNT=0`. + - Readiness-bound → look at *which* HelmRelease/Kustomization stalls (now + visible thanks to step 2) and tune that service or its wait. +4. **Only after the per-leg run is stable**, consider fanning out into more + matrix legs for wall-clock speed (§2) — splitting specs, not duplicating the + whole suite. + +The throughline: the "green on retry" behaviour is a **flaky readiness race**, +and the current design **hides the evidence** by burying prepare inside a +buffered Ginkgo writer under a single coarse 15-minute timeout. Steps 1 and 2 +make the evidence visible; everything else is tuning once you can see it. From 09e7bcbd2e0853ea9b100464630e295828f05be5 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Mon, 1 Jun 2026 07:23:31 +0000 Subject: [PATCH 23/40] chore: let's keep kubeconfig arround --- .github/workflows/ci.yml | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 2c17accaf..cf94ec51d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -328,9 +328,20 @@ jobs: -e IMAGE_DELIVERY_MODE=${{ env.IMAGE_DELIVERY_MODE }} \ -e K3D_AGENT_COUNT=${{ matrix.k3d_agent_count }} \ -e HOST_PROJECT_PATH=${{ github.workspace }} \ + -e KUBECONFIG=${{ env.CI_WORKDIR }}/.stamps/cluster/k3d-gitops-reverser-test-e2e/.kube/config \ ${{ env.CI_CONTAINER }} \ bash -c " git config --global --add safe.directory ${{ env.CI_WORKDIR }} + # The cluster is created here but reused by the test container + # below. start-cluster.sh writes the k3d context into KUBECONFIG; + # point it at a file on the mounted workspace so the next + # container inherits the context (otherwise it is lost with this + # ephemeral container and kubectl reports 'context does not exist'). + # Living under .stamps/cluster//.kube keeps it tied to the + # cluster lifecycle: cleanup-installs.sh (per-namespace) leaves it + # alone, while clean-cluster (rm -rf .stamps/cluster/) clears + # it together with the cluster it points at. + mkdir -p .stamps/cluster/k3d-gitops-reverser-test-e2e/.kube task _services-ready " @@ -346,6 +357,7 @@ jobs: -e E2E_GINKGO_PROCS=${{ matrix.e2e_ginkgo_procs }} \ -e K3D_AGENT_COUNT=${{ matrix.k3d_agent_count }} \ -e HOST_PROJECT_PATH=${{ github.workspace }} \ + -e KUBECONFIG=${{ env.CI_WORKDIR }}/.stamps/cluster/k3d-gitops-reverser-test-e2e/.kube/config \ ${{ env.CI_CONTAINER }} \ bash -c " git config --global --add safe.directory ${{ env.CI_WORKDIR }} From 82715f2e900aa2e877256331c031a4bcd20756ee Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Mon, 1 Jun 2026 07:45:48 +0000 Subject: [PATCH 24/40] chore: local allure and 0 agents (to speed up startup). --- .devcontainer/devcontainer.json | 7 +++- .github/workflows/ci.yml | 6 +-- docs/ci/e2e-allure-reporting.md | 33 +++++++++++++++ test/e2e/Taskfile.yml | 72 +++++++++++++++++++++++++++++++++ 4 files changed, 114 insertions(+), 4 deletions(-) diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json index fd1ab6935..bfb22bbd9 100644 --- a/.devcontainer/devcontainer.json +++ b/.devcontainer/devcontainer.json @@ -25,7 +25,8 @@ 19090, 10350, 16379, - 19080 + 19080, + 19081 ], "portsAttributes": { "13000": { @@ -47,6 +48,10 @@ "19080": { "label": "Flux", "onAutoForward": "notify" + }, + "19081": { + "label": "Allure", + "onAutoForward": "notify" } }, "customizations": { diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index cf94ec51d..d3f7c0d5f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -274,12 +274,12 @@ jobs: && task test-e2e-quickstart-manifest needs_artifact: true e2e_ginkgo_procs: "2" - k3d_agent_count: "1" + k3d_agent_count: "0" - name: full script: "task test-e2e-full" needs_artifact: false e2e_ginkgo_procs: "1" - k3d_agent_count: "1" + k3d_agent_count: "0" env: PROJECT_IMAGE: ${{ needs.docker-build.outputs.image }} CI_CONTAINER: ${{ needs.build-ci-container.outputs.image }} @@ -388,7 +388,7 @@ jobs: cat "${samples}" 2>/dev/null || echo "(no samples captured)" echo '```' echo "" - } >> "$GITHUB_STEP_SUMMARY" + } | tee -a "$GITHUB_STEP_SUMMARY" - name: Summarize E2E timing reports if: always() diff --git a/docs/ci/e2e-allure-reporting.md b/docs/ci/e2e-allure-reporting.md index 3010bc5ff..d9b60584d 100644 --- a/docs/ci/e2e-allure-reporting.md +++ b/docs/ci/e2e-allure-reporting.md @@ -29,3 +29,36 @@ The converter is deliberately small and one-way: The existing raw Ginkgo JSON and timing-summary artifacts are still uploaded. Those files remain the source of truth for local tooling and timing analysis. + +## Local devcontainer use + +The devcontainer does not install Java or the Allure CLI. Local report rendering +uses Docker to run Allure in an on-demand Java runtime container, keeping the +devcontainer image smaller. + +Run any e2e task that writes a Ginkgo JSON report, for example: + +```bash +task test-e2e +``` + +Then generate a local Allure HTML report: + +```bash +task allure-e2e-report +``` + +The report is written to `.stamps/allure-report`. To serve it from inside the +devcontainer on the forwarded Allure port: + +```bash +task allure-e2e-open +``` + +This serves the report on port `19081`. The same tasks can reuse older local +Ginkgo JSON reports under `.stamps`, so you can generate the report after a +failed run without rerunning the suite. + +The first local render pulls the `eclipse-temurin:17-jre` image and downloads the +Allure CLI into `.stamps/allure-cli`. Later renders reuse both Docker's image +cache and the local Allure CLI cache. diff --git a/test/e2e/Taskfile.yml b/test/e2e/Taskfile.yml index 33675cf94..be410caca 100644 --- a/test/e2e/Taskfile.yml +++ b/test/e2e/Taskfile.yml @@ -93,6 +93,12 @@ vars: FLUX_WAIT_TIMEOUT: '{{.FLUX_WAIT_TIMEOUT | default "500s"}}' FLUX_SERVICES_WAIT_TIMEOUT: '{{.FLUX_SERVICES_WAIT_TIMEOUT | default "600s"}}' E2E_GO_TEST_TIMEOUT: '{{.E2E_GO_TEST_TIMEOUT | default "15m"}}' + ALLURE_RESULTS_DIR: '{{.ALLURE_RESULTS_DIR | default ".stamps/allure-results"}}' + ALLURE_REPORT_DIR: '{{.ALLURE_REPORT_DIR | default ".stamps/allure-report"}}' + ALLURE_CLI_DIR: '{{.ALLURE_CLI_DIR | default ".stamps/allure-cli"}}' + ALLURE_VERSION: '{{.ALLURE_VERSION | default "2.42.0"}}' + ALLURE_CONTAINER_IMAGE: '{{.ALLURE_CONTAINER_IMAGE | default "eclipse-temurin:17-jre"}}' + ALLURE_PORT: '{{.ALLURE_PORT | default "19081"}}' # Number of parallel Ginkgo processes for the e2e suite. >1 enables Ginkgo # parallelism (Phase 2.5 of docs/design/e2e-speedup-plan.md). Files that touch # shared cluster-scoped state are marked Serial (see docs/design/e2e-serial-registry.md). @@ -203,6 +209,72 @@ tasks: --output-dir="{{.CS}}/{{.NAMESPACE}}" --json-report=ginkgo-report-full.json \ ./test/e2e/ + allure-e2e-results: + desc: Convert local e2e Ginkgo JSON reports into Allure results + aliases: + - e2e-allure-results + cmds: + - | + reports_file="$(mktemp)" + trap 'rm -f "${reports_file}"' EXIT + if [ -d .stamps/cluster ]; then + find .stamps/cluster -path "*/ginkgo-report-*.json" -type f | sort > "${reports_file}" + fi + if [ ! -s "${reports_file}" ]; then + echo "No Ginkgo JSON reports found under .stamps/cluster. Run an e2e task first." >&2 + exit 1 + fi + + rm -rf "{{.ALLURE_RESULTS_DIR}}" + mkdir -p "{{.ALLURE_RESULTS_DIR}}" + xargs go run ./test/e2e/tools/ginkgo-allure --output-dir "{{.ALLURE_RESULTS_DIR}}" < "${reports_file}" + echo "Wrote Allure results to {{.ALLURE_RESULTS_DIR}}" + + allure-e2e-report: + desc: Generate local Allure HTML from e2e Ginkgo reports + aliases: + - e2e-allure-report + cmds: + - task: allure-e2e-results + - | + if ! command -v docker >/dev/null 2>&1; then + echo "Docker is required to render Allure without adding Java to the devcontainer." >&2 + exit 1 + fi + + workspace_host="${HOST_PROJECT_PATH:-$(pwd)}" + workspace_container="$(pwd)" + mkdir -p "{{.ALLURE_CLI_DIR}}" + docker run --rm \ + --user "$(id -u):$(id -g)" \ + -e HOME=/tmp \ + -e ALLURE_CLI_DIR="{{.ALLURE_CLI_DIR}}" \ + -e ALLURE_REPORT_DIR="{{.ALLURE_REPORT_DIR}}" \ + -e ALLURE_RESULTS_DIR="{{.ALLURE_RESULTS_DIR}}" \ + -e ALLURE_VERSION="{{.ALLURE_VERSION}}" \ + -v "${workspace_host}:${workspace_container}" \ + -w "${workspace_container}" \ + "{{.ALLURE_CONTAINER_IMAGE}}" \ + sh -eu -c ' + allure_home="${ALLURE_CLI_DIR}/allure-${ALLURE_VERSION}" + if [ ! -x "${allure_home}/bin/allure" ]; then + rm -rf "${allure_home}" + mkdir -p "${ALLURE_CLI_DIR}" + curl -fsSL "https://github.com/allure-framework/allure2/releases/download/${ALLURE_VERSION}/allure-${ALLURE_VERSION}.tgz" \ + | tar -xz -C "${ALLURE_CLI_DIR}" + fi + "${allure_home}/bin/allure" generate "${ALLURE_RESULTS_DIR}" --clean -o "${ALLURE_REPORT_DIR}" + ' + echo "Wrote Allure HTML report to {{.ALLURE_REPORT_DIR}}" + + allure-e2e-open: + desc: Generate and serve the local e2e Allure report + aliases: + - e2e-allure-open + cmds: + - task: allure-e2e-report + - python3 -m http.server "{{.ALLURE_PORT}}" --bind 0.0.0.0 --directory "{{.ALLURE_REPORT_DIR}}" + test-e2e-manager: desc: Run manager-focused e2e scenarios cmds: From de3dc0955c7397dffc9d8a072fbdd0c3aaddb75b Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Mon, 1 Jun 2026 08:11:56 +0000 Subject: [PATCH 25/40] chore: hardening and cleaning always printed comments --- test/e2e/Taskfile.yml | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/test/e2e/Taskfile.yml b/test/e2e/Taskfile.yml index be410caca..dcdb1feaa 100644 --- a/test/e2e/Taskfile.yml +++ b/test/e2e/Taskfile.yml @@ -176,16 +176,16 @@ tasks: test-e2e: desc: Run the smoke e2e suite + # Ginkgo parallelism (--procs) is orchestrated by the ginkgo CLI, not by + # `go test`. Run it via `go run` so the CLI version matches the pinned + # github.com/onsi/ginkgo/v2 module (a globally-installed ginkgo can differ + # and refuse to run on a version mismatch). cmds: - | export CTX="{{.CTX}}" export INSTALL_MODE="{{.INSTALL_MODE}}" export NAMESPACE="{{.NAMESPACE}}" export E2E_AGE_KEY_FILE="{{.CS}}/age-key.txt" - # Ginkgo parallelism (--procs) is orchestrated by the ginkgo CLI, not by - # `go test`. Run it via `go run` so the CLI version matches the pinned - # github.com/onsi/ginkgo/v2 module (a globally-installed ginkgo can differ - # and refuse to run on a version mismatch). go run github.com/onsi/ginkgo/v2/ginkgo \ --procs={{.E2E_GINKGO_PROCS}} --timeout="{{.E2E_GO_TEST_TIMEOUT}}" -v \ --label-filter=smoke \ @@ -610,13 +610,16 @@ tasks: clean-cluster: desc: Tear down the E2E cluster used for tests and remove its stamps cmds: + # Delete unconditionally rather than guarding on `k3d cluster list`. An + # interrupted create can leave orphaned artifacts (e.g. a `-tools` + # container and `-images` volume) that `k3d cluster list` does not show + # but `k3d cluster create` still rejects as "a cluster with that name + # already exists". `k3d cluster delete` cleans up that partial state and + # exits 0 when there is genuinely nothing to delete, so it is safe to run + # every time. - | - if {{.K3D}} cluster list 2>/dev/null | awk '{print $1}' | grep -q "^{{.CLUSTER_NAME}}$"; then - echo "Deleting k3d cluster '{{.CLUSTER_NAME}}'" - {{.K3D}} cluster delete "{{.CLUSTER_NAME}}" - else - echo "k3d cluster '{{.CLUSTER_NAME}}' does not exist; skipping cleanup" - fi + echo "Deleting k3d cluster '{{.CLUSTER_NAME}}' (including any orphaned partial state)" + {{.K3D}} cluster delete "{{.CLUSTER_NAME}}" - rm -rf "{{.CS}}" _cluster-ready: From 03b95abfae8a52552729872231441c1e1ac2cbc5 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Mon, 1 Jun 2026 08:40:42 +0000 Subject: [PATCH 26/40] fix: race condition in HelmRelease causing unstable e2e tests --- hack/e2e/wait-flux-services.sh | 4 ++-- test/e2e/Taskfile.yml | 2 +- test/e2e/setup/flux/releases/aggregated-api.yaml | 10 ++++++++++ 3 files changed, 13 insertions(+), 3 deletions(-) diff --git a/hack/e2e/wait-flux-services.sh b/hack/e2e/wait-flux-services.sh index 276e5e33a..1125eb723 100755 --- a/hack/e2e/wait-flux-services.sh +++ b/hack/e2e/wait-flux-services.sh @@ -6,10 +6,10 @@ set -euo pipefail # # Inputs (env): # - CTX (required): kube context -# - FLUX_SERVICES_WAIT_TIMEOUT (optional): per-resource wait timeout; default 600s +# - FLUX_SERVICES_WAIT_TIMEOUT (optional): per-resource wait timeout; default 120s : "${CTX:?CTX is required}" -FLUX_SERVICES_WAIT_TIMEOUT="${FLUX_SERVICES_WAIT_TIMEOUT:-600s}" +FLUX_SERVICES_WAIT_TIMEOUT="${FLUX_SERVICES_WAIT_TIMEOUT:-120s}" flux_ready_count=0 echo "⏳ Waiting for Flux-managed installations to become ready..." diff --git a/test/e2e/Taskfile.yml b/test/e2e/Taskfile.yml index dcdb1feaa..bb03262b2 100644 --- a/test/e2e/Taskfile.yml +++ b/test/e2e/Taskfile.yml @@ -91,7 +91,7 @@ vars: FLUX_INSTANCE_FILE: '{{.FLUX_INSTANCE_FILE | default "test/e2e/setup/flux-operator/flux-instance.yaml"}}' FLUX_OPERATOR_DIR: '{{.FLUX_OPERATOR_DIR | default "test/e2e/setup/flux-operator"}}' FLUX_WAIT_TIMEOUT: '{{.FLUX_WAIT_TIMEOUT | default "500s"}}' - FLUX_SERVICES_WAIT_TIMEOUT: '{{.FLUX_SERVICES_WAIT_TIMEOUT | default "600s"}}' + FLUX_SERVICES_WAIT_TIMEOUT: '{{.FLUX_SERVICES_WAIT_TIMEOUT | default "120s"}}' E2E_GO_TEST_TIMEOUT: '{{.E2E_GO_TEST_TIMEOUT | default "15m"}}' ALLURE_RESULTS_DIR: '{{.ALLURE_RESULTS_DIR | default ".stamps/allure-results"}}' ALLURE_REPORT_DIR: '{{.ALLURE_REPORT_DIR | default ".stamps/allure-report"}}' diff --git a/test/e2e/setup/flux/releases/aggregated-api.yaml b/test/e2e/setup/flux/releases/aggregated-api.yaml index c0ad69406..0dce37166 100644 --- a/test/e2e/setup/flux/releases/aggregated-api.yaml +++ b/test/e2e/setup/flux/releases/aggregated-api.yaml @@ -20,6 +20,16 @@ metadata: spec: interval: 60m timeout: 10m + # This release renders cert-manager Issuer/Certificate resources (tls.mode: + # cert-manager below), so it must not install until cert-manager's CRDs and + # webhook are ready. Without this, kubectl apply -k fires both releases in + # parallel and aggregated-api's helm install races cert-manager: if the CRDs + # aren't established yet the install fails, exhausts its 3 remediation + # retries, and the release stays not-Ready until the next 60m interval — + # which times out wait-flux-services.sh on helmreleases/aggregated-api. + dependsOn: + - name: cert-manager + namespace: cert-manager chartRef: kind: OCIRepository name: apiservice-audit-proxy From 0787b1b68bd9eb47ea3b816712f8eb09861f6d6b Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Mon, 1 Jun 2026 09:47:58 +0000 Subject: [PATCH 27/40] fix: improve and test GitTarget isolation behaviour --- .github/workflows/ci.yml | 6 +- docs/design/e2e-ci-stability-report.md | 40 +- docs/design/e2e-speedup-plan.md | 32 +- .../gittarget-isolation-on-rule-change.md | 422 +++++++++++++++++ internal/rulestore/store_test.go | 103 +++++ internal/watch/catalog_test_helpers_test.go | 32 -- internal/watch/manager.go | 182 ++++++-- internal/watch/rule_change_snapshot_test.go | 426 +++++++++++++----- test/e2e/Taskfile.yml | 5 +- test/e2e/gittarget_isolation_e2e_test.go | 203 +++++++++ .../manager/watchrule-resources.tmpl | 11 + 11 files changed, 1251 insertions(+), 211 deletions(-) create mode 100644 docs/design/gittarget-isolation-on-rule-change.md create mode 100644 test/e2e/gittarget_isolation_e2e_test.go create mode 100644 test/e2e/templates/manager/watchrule-resources.tmpl diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d3f7c0d5f..8422c32c0 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -278,7 +278,11 @@ jobs: - name: full script: "task test-e2e-full" needs_artifact: false - e2e_ginkgo_procs: "1" + # Bumped 1 -> 4: the full suite was serialized to avoid the + # cross-target rule-change snapshot coupling, now fixed (per-target + # effective watch plan — docs/design/gittarget-isolation-on-rule-change.md). + # Validated locally at --procs=4; trialing the same in CI. + e2e_ginkgo_procs: "4" k3d_agent_count: "0" env: PROJECT_IMAGE: ${{ needs.docker-build.outputs.image }} diff --git a/docs/design/e2e-ci-stability-report.md b/docs/design/e2e-ci-stability-report.md index 07c2c1cf4..ba1ed1ff8 100644 --- a/docs/design/e2e-ci-stability-report.md +++ b/docs/design/e2e-ci-stability-report.md @@ -12,11 +12,11 @@ questions (run `prepare-e2e` up front; observe disk usage). ``` e2e (runs-on: ubuntu-latest) - ├─ matrix leg "quickstart" → E2E_GINKGO_PROCS=2, K3D_AGENT_COUNT=1, needs_artifact=true + ├─ matrix leg "quickstart" → E2E_GINKGO_PROCS=2, K3D_AGENT_COUNT=0, needs_artifact=true │ script: INSTALL_MODE=helm task test-e2e-quickstart-helm │ → task test-image-refresh │ → INSTALL_MODE=plain-manifests-file task test-e2e-quickstart-manifest - └─ matrix leg "full" → E2E_GINKGO_PROCS=1, K3D_AGENT_COUNT=1, needs_artifact=false + └─ matrix leg "full" → E2E_GINKGO_PROCS=1, K3D_AGENT_COUNT=0, needs_artifact=false script: task test-e2e-full ``` @@ -41,19 +41,25 @@ service image — lands on the **one** runner VM's disk. ### 1.2 What `task test-e2e-*` actually does -The Task targets ([`test/e2e/Taskfile.yml`](../../test/e2e/Taskfile.yml)) don't -prepare the cluster themselves — they just invoke Ginkgo: +The Task targets ([`test/e2e/Taskfile.yml`](../../test/e2e/Taskfile.yml)) still +invoke Ginkgo: ``` go run .../ginkgo --procs=$E2E_GINKGO_PROCS --timeout=$E2E_GO_TEST_TIMEOUT ... ./test/e2e/ ``` -Cluster prep happens **inside** the suite. `SynchronizedBeforeSuite` (process #1 -only) calls `prepareE2EClusterOnce()` +For local runs, cluster prep happens **inside** the suite. `SynchronizedBeforeSuite` +(process #1 only) calls `prepareE2EClusterOnce()` ([`e2e_suite_test.go:82`](../../test/e2e/e2e_suite_test.go#L82)), which shells out to `task prepare-e2e` with the leg's `CTX / INSTALL_MODE / NAMESPACE / INSTALL_NAME`. +In CI, the workflow now runs the install-mode-independent `_services-ready` +slice first, outside Ginkgo, so cluster + Flux + Flux-managed service logs +stream in their own step. The suite still calls `prepare-e2e`; because the +cluster/service stamps already exist, it skips that expensive subtree and +continues with the install-mode-specific controller/image tail. + ### 1.3 The `prepare-e2e` dependency graph `prepare-e2e` is a go-task DAG, every node gated by a stamp file under @@ -81,8 +87,11 @@ The genuinely slow / hang-prone nodes: | `_flux-installed` | flux-operator install + FluxInstance + Flux deployments `Available` | `FLUX_WAIT_TIMEOUT=500s` | | `_flux-setup-ready` | every HelmRelease/Kustomization (gitea, valkey, prometheus, traefik…) `Ready` | `FLUX_SERVICES_WAIT_TIMEOUT=600s` per resource | -The whole Ginkgo run — prepare **plus** the specs — is bounded by one timeout: -`E2E_GO_TEST_TIMEOUT=15m` (smoke/quickstart) or `E2E_FULL_TIMEOUT=30m` (full). +For local runs, the whole Ginkgo run — prepare **plus** the specs — is bounded by +one timeout: `E2E_GO_TEST_TIMEOUT=15m` (smoke/quickstart) or +`E2E_FULL_TIMEOUT=30m` (full). In CI, `_services-ready` is outside that Ginkgo +timeout; the timeout now covers the stamped/no-op cluster prep, the +install-mode-specific tail, and the specs. ### 1.4 Why it "goes silent for ~15 min, then green on retry" @@ -125,20 +134,19 @@ Ready). Disk pressure can *cause* those stalls (e.g. kubelet worth ruling in or out — but **measure before you commit to that theory.** That's exactly what §4 adds. -> **Set `K3D_AGENT_COUNT=1`.** ✅ Already done — both matrix legs pass -`k3d_agent_count: "1"` (1 server + 1 agent = 2 nodes). You could go to **0 -agents** (single-node; k3s schedules workloads on the server) to shave one more -node-container's worth of RAM/disk/inotify. Trade-off: less parallelism headroom -for `--procs=2`. Worth trying on the quickstart leg if measurements show -resource pressure. +> **Set `K3D_AGENT_COUNT=1`.** Superseded. Both matrix legs now pass +`k3d_agent_count: "0"` (single-node k3d; k3s schedules workloads on the server). +This reduces node-container RAM/disk/inotify footprint, but leaves less +scheduling headroom for the quickstart leg's `--procs=2`. Keep watching the +quickstart timing and resource samples before treating `0` agents as permanent. > **Spin up 4 e2e legs and run more in parallel.** ◑ Mixed. More legs = more *isolated* runners, so wall-clock can drop **if you split specs across them**. But it does **not** reduce per-leg disk/RAM pressure — each new leg re-pulls the full image set onto its own 14 GB disk. If disk/resource pressure is the real root cause, adding legs spreads load but each leg is just as likely to hit it. -Fix the per-leg footprint first (measure → free disk → maybe 0 agents), *then* -fan out for speed. +Fix the per-leg footprint first (measure → single-node/0-agent if it keeps +passing), *then* fan out for speed. --- diff --git a/docs/design/e2e-speedup-plan.md b/docs/design/e2e-speedup-plan.md index 0958209ef..876ecd7f5 100644 --- a/docs/design/e2e-speedup-plan.md +++ b/docs/design/e2e-speedup-plan.md @@ -487,10 +487,13 @@ Applied fallback: - CI **E2E (full)** sets `E2E_GINKGO_PROCS=1`. - Local/default `task test-e2e*` remains `E2E_GINKGO_PROCS=2`. -- k3d local/default agents remain 3 for `procs=2`; CI passes - `K3D_AGENT_COUNT=1` because the full job is sequential. +- k3d local/default agents remain 3 for `procs=2`; CI currently passes + `K3D_AGENT_COUNT=0` for both e2e legs to reduce per-leg runner footprint. - Warm k3d cluster reuse now verifies the expected node count and Ready condition instead of trusting `kubectl get ns` plus stamps. +- CI pre-runs `_services-ready` outside Ginkgo so cluster + Flux + service + readiness logs stream live; the in-suite `prepare-e2e` call then reuses the + stamps and only performs the install-mode-specific tail. History that led here: Progression tried: @@ -504,13 +507,24 @@ Progression tried: consecutive green reruns before we trust it. The 3→1 drop was originally due to `fs.inotify.max_user_instances` (default 128) exhaustion; that is now mitigated to 512 by `ensure_inotify_limits`, so 1 server + 3 agents - is possible but no longer the CI default. Confirmed the suite uses a **single** cluster (the - `audit-pass-through-e2e` cluster seen locally is an unrelated leftover). - -**Next after merge:** start **Phase 3** (below) on a *separate* branch — -it touches production controller code (`internal/telemetry` + -`internal/watch`) and cuts the Serial `Restart Snapshot Safety` spec -(~128 s, the dominant remaining cost). + is possible but no longer the CI default. +- **0 agents** (`82715f2`/`de3dc09` era) → the branch has produced multiple + green CI runs with the explicit `_services-ready` pre-step and Allure/timing + artifacts, but this is still below the preferred stability window. Treat it + as promising, not proven, until there are 5-10 consecutive green full e2e + runs on the same shape. + +Confirmed the suite uses a **single** cluster (the `audit-pass-through-e2e` +cluster seen locally is an unrelated leftover). + +**Current best next work (2026-06-01):** do not raise the full-suite timeout +again yet. `test-e2e-full` already has a 30 minute timeout, and recent green CI +full legs complete well below it. The next useful work is a confidence pass: +rerun the current CI shape enough times to build a 5-10 green-run window, then +use the uploaded timing summaries, disk samples, and Allure reports to decide +whether `K3D_AGENT_COUNT=0` is stable enough to merge. If it is stable, the next +speed target is the residual fixed setup in `Restart Snapshot Safety` and the +coarse `_services-ready` barrier, not more timeout. ### Phase 3 — Target-scoped reconcile-complete + drain signal for `Restart Snapshot Safety` diff --git a/docs/design/gittarget-isolation-on-rule-change.md b/docs/design/gittarget-isolation-on-rule-change.md new file mode 100644 index 000000000..e905ffe3f --- /dev/null +++ b/docs/design/gittarget-isolation-on-rule-change.md @@ -0,0 +1,422 @@ +# GitTarget Isolation on Rule Changes + +> **Status: ✅ Implemented and shipped.** Snapshot selection is now driven purely +> by a per-target *effective watch plan* hash (resolved GVR + scope + unioned +> operations + destination); the global `force` flag is gone. Validated by unit +> tests, a non-Serial e2e regression, and a `--procs=4` smoke run (21/21). See +> the Rollout Plan below for the per-step landing notes. The sections before it +> are kept as the original design rationale. + +## Problem + +`GitTarget` should be a user-facing isolation boundary: rules for one target +should not change the commit behavior of another target. + +The parallel e2e smoke run exposed a cross-target coupling: + +- `Manager WatchRule ConfigMap and Secret should create Git commit when + ConfigMap is added via WatchRule` created a ConfigMap and expected an event + commit such as `[CREATE] v1/configmaps/...`. +- Another spec changed WatchRules/GVRs at roughly the same time. +- The manager target entered rule-change snapshot mode even though its own + user-visible rule set had not changed. +- The ConfigMap was written correctly, but the latest commit was + `reconcile: sync 2 resources`, not `[CREATE] ...`. + +That is more than a brittle test assertion. It means unrelated GitTargets can +affect each other's audit/commit semantics. + +## Why It Matters + +GitTarget isolation keeps the system predictable: + +- Auditability: an event commit preserves operation, resource path, and author + intent. A snapshot commit is less specific. +- Multi-tenancy: one team or namespace changing rules should not perturb another + target's repo history. +- Testability: parallel e2e should reveal real isolation problems, not require + broad `Serial` markings to hide them. +- Operational clarity: rule-change behavior is easier to reason about when it is + scoped to the target whose effective watch plan changed. + +## Current Behavior + +Per-target snapshot tracking **already exists**. `currentRuleSetSnapshots` +(`internal/watch/manager.go`) computes a per-GitTarget hash, and +`snapshotTargetsNeedingDelivery` already compares each target's current hash +against its last-delivered hash and skips targets whose hash is unchanged. So +the machinery for target-scoped snapshots is in place. + +The coupling comes from a single global override. `ReconcileForRuleChange` +computes global GVR additions/removals and passes them as a `force` flag: + +```go +targets := m.snapshotTargetsNeedingDelivery(len(added) > 0 || len(removed) > 0) +``` + +`added`/`removed` are *global* deltas. When `force=true`, +`snapshotTargetsNeedingDelivery` **bypasses the per-target hash comparison** and +selects every current GitTarget — even targets whose effective rule set did not +change. That one wire is the cross-target coupling. + +There is a second, subtler reason the global `force` exists, and it constrains +the fix. The per-target hash is computed over the **rule text** +(`rule.ResourceRules`, source, provider, branch, path) — *not* over the +**resolved GVR set**. So if target A has a wildcard rule and a newly installed +CRD makes that wildcard match a new served resource, A's *rule text* is +unchanged, A's hash is unchanged, and without `force` A would never snapshot — +letting the new informer's initial `ADDED` events leak as individual `[CREATE]` +commits. The global `force` was masking this by snapshotting everyone. + +Shared informers are fine. The coupling is purely that global informer churn is +wired into per-target snapshot selection as a `force` override, and that the +per-target hash is not yet sensitive to the resolved watch set. + +## Desired Behavior + +Only GitTargets whose effective watched resource set changed should receive a +rule-change snapshot. + +Examples: + +- If target A watches ConfigMaps in namespace A, and target B adds a signing + WatchRule, target A should keep processing live events normally. +- If target A's own WatchRule changes, target A should snapshot. +- If target A has a wildcard rule and a newly installed CRD makes that wildcard + match a new served resource, target A should snapshot. +- If a global informer has to start because target B now needs a GVR, only + targets that depend on that new watch should be protected from its initial + ADDED events. + +## Component Overview + +```mermaid +flowchart TD + Controllers["WatchRule / ClusterWatchRule controllers
(reconcile on rule CRD create/update/delete)"] + Controllers -->|ReconcileForRuleChange| Manager + + subgraph Manager["watch.Manager (manager.go)"] + direction TB + RuleStore["RuleStore
effective rules per GitTarget"] + Catalog["RefreshAPIResourceCatalog
what the cluster serves"] + Resolver["rule_gvr_resolver +
computeRequestedGVRs()"] + RuleStore --> Resolver + Catalog --> Resolver + + Resolver --> Global["GLOBAL informer lifecycle
compareGVRs(added, removed)
start/stopInformer per GVR
(SHARED across targets)"] + Resolver --> Scoped["TARGET-SCOPED snapshot selection
snapshotTargetsNeedingDelivery(force)
currentRuleSetSnapshots()
per-target hash"] + + Global -.->|"len(added)+len(removed) > 0
=> force=true ⟵ BUG"| Scoped + Global --> Active["activeInformers map"] + Scoped --> Targets["targets[] needing snapshot"] + end + + Active -->|"ADDED / UPDATED / DELETED events"| Router + Targets -->|"Begin/CompleteReconciliationForStream
+ RequestRepoState / RequestClusterState"| Router + + subgraph Router["EventRouter (event_router.go)"] + States["per-GitTarget stream state machine
LIVE_PROCESSING ⇄ RECONCILING (buffers events)"] + end + + Router -->|ProcessControlEvent| Reconciler + + subgraph Reconciler["ReconcilerManager → FolderReconciler"] + Commit["per GitTarget → branch worker → Git commit
[CREATE] event commit  vs  'reconcile: sync N resources' snapshot"] + end +``` + +Key takeaway: informers are legitimately **global/shared** (left branch); +snapshot selection is already **per-target** (right branch). The bug is purely +that the global left branch (`added`/`removed`) is wired into the right branch +as a `force` override. The fix is to cut that wire and make the per-target hash +sensitive to the *resolved* watch set so the right branch can stand on its own. + +## How the Data Structures Work Together + +The behavior above is the product of four data structures that are computed +fresh (or compared) on every `ReconcileForRuleChange`. Understanding where each +one comes from makes it clear why the coupling exists and what the fix touches. + +**1. The rule set — `RuleStore` (`internal/rulestore`).** +The controllers for `WatchRule` and `ClusterWatchRule` write the user's rules +into a shared `RuleStore`. It is the single source of *intent*: "target T wants +to watch these resources, write them to this provider/branch/path." Everything +downstream is derived from `RuleStore.SnapshotWatchRules()` and +`SnapshotClusterWatchRules()`. + +**2. What the cluster actually serves — `APIResourceCatalog`.** +`RefreshAPIResourceCatalog` queries discovery and records which GVRs are served, +listable, and watchable. Rules can name wildcards or resources that do not exist +yet, so intent (1) must be intersected with reality (2) before anything can be +watched. + +**3. The desired GVR set — `computeRequestedGVRs()`.** +This is the *global* join of (1) and (2): rules resolved against the catalog, +producing the flat set of GVRs that informers must cover across all targets. +`compareGVRs` diffs this against `activeInformers` to produce `added`/`removed`. +This drives informer lifecycle and is deliberately **target-agnostic** — +informers are shared, so it does not matter *which* target needs a GVR, only +that *some* target does. + +``` +RuleStore (intent) ─┐ + ├─► computeRequestedGVRs() ─► added/removed ─► start/stopInformer +APIResourceCatalog ─┘ (global, shared) +``` + +**4. The per-target snapshot ledger — `currentRuleSetSnapshots()` + +`lastDeliveredRuleSetHash` / `pendingRuleSetHash`.** +This is the target-scoped half. `currentRuleSetSnapshots()` walks the same +`RuleStore`, but instead of flattening everything into one GVR set it groups +entries **by GitTarget** (`gitDest.Key()`) and hashes each group. Today it hashes +the raw rule text; after the fix it hashes the target's *effective watch plan* +(resolved GVR + scope + operations + destination — see Section 2): + +``` +RuleStore ─► currentRuleSetSnapshots() ─► []ruleSetSnapshotTarget{gitDest, hash} + │ + snapshotTargetsNeedingDelivery(...) compares each target's + hash against lastDeliveredRuleSetHash[key]: + hash unchanged → skip (target already in sync) + hash changed → select, record in pendingRuleSetHash[key] +``` + +Two maps keyed by `gitDest.Key()` hold the ledger state across reconciles: + +- `lastDeliveredRuleSetHash[key]` — the hash whose snapshot was last successfully + emitted for that target. Updated by `markRuleSetSnapshotDelivered` only after + the snapshot actually reaches a reconciler. +- `pendingRuleSetHash[key]` — targets selected this pass but not yet confirmed + delivered. (The unit tests read this map to observe *which* targets were + selected, without wiring a full `EventRouter`.) + +**Where the two halves meet — and where it goes wrong.** +`ReconcileForRuleChange` is the only place that uses both halves, and it crosses +the wires here: + +```go +targets := m.snapshotTargetsNeedingDelivery(len(added) > 0 || len(removed) > 0) +``` + +The argument is the *global* GVR delta from structure (3); the function it feeds +is the *per-target* ledger from structure (4). When `force=true`, the +per-target hash comparison is skipped entirely and **every** target is selected. +So a global informer change — even one caused by an unrelated target — overrides +the otherwise-correct per-target isolation. + +The second flaw is upstream of the ledger: structure (4) hashes only **rule +text**, never the **resolved GVR** from structure (3). So a wildcard rule whose +*resolved* set grows (a new CRD appears) does not change the target's hash. The +`force` flag was the crude compensation for that blind spot. The fix is to feed +the resolved GVR into structure (4)'s hash so the ledger sees real watch-surface +changes, after which structure (3)'s global delta no longer needs to override +it. + +## Proposed Approach + +### 1. Test the Invariant First — ✅ done + +The invariant is pinned in `internal/watch/rule_change_snapshot_test.go`. The +first test was written **red against the old `force` behavior** and is now green +under the fix: + +- `TestReconcileForRuleChange_UnrelatedTargetNotSnapshotted_OnGlobalGVRChurn` — + two GitTargets A and B both watch ConfigMaps, both in steady state. Only B's + effective plan changes (it starts watching secrets) while unrelated global + informer churn happens concurrently (a stale GVR is dropped). It asserts B is + selected *and* A is not. This failed against the global `force` flag and passes + now that selection is purely per-target. +- `TestSnapshotTargetsNeedingDelivery_PerTargetHashIsolatesTargets` — exercises + the selection function directly and confirms only the changed target is + returned. +- `TestReconcileForRuleChange_RedundantDuplicateRule_DoesNotSnapshot` — the flip + side: a duplicate rule resolving to the same surface does not snapshot (what + dropping source identity buys). +- `TestReconcileForRuleChange_TargetGainsAlreadyWatchedGVR_Snapshots` — issue + #146 under the new model: a target gaining coverage of an already-watched GVR + still snapshots, with no informer churn. +- `TestCurrentRuleSetSnapshots_NamespacedWatchRulePlanByNamespace` — the + namespace dimension of the plan hash for namespaced WatchRules. + +How A's selection is observed without a full `EventRouter`: +`snapshotTargetsNeedingDelivery` records every selected target in +`pendingRuleSetHash` before emission, and with `EventRouter == nil` the emit +path is a guarded no-op that never clears it — so `pendingRuleSetHash` is a +faithful record of who was selected (see structure 4 above). + +### 2. Make the Per-Target Hash Reflect the Resolved Watch Set + +This is the core change. `currentRuleSetSnapshots` already produces a per-target +hash, but it hashes the **rule text** verbatim — the raw `apiGroups` / +`apiVersions` / `resources` patterns, scope, operations, destination, *and* the +source rule's namespace/name. The fix is to hash the target's **effective watch +plan** — what it actually watches after rule resolution and API discovery — not +what the rule literally says: + +```go +type targetWatchPlan struct { + GitDest types.ResourceReference + Entries []targetWatchPlanEntry + Hash uint64 +} +``` + +Not all of the rule text is equal. Split each field by whether GVR resolution +already captures it: + +| Rule field | In the plan hash? | Why | +|---|---|---| +| `apiGroups` / `apiVersions` / `resources` (incl. wildcards) | **Replace** with the resolved GVR set | These are *inputs* to resolution; the resolved GVR is the *output*, intersected with the catalog. The resolved set is strictly more accurate — it catches wildcard-meets-new-CRD, which the raw patterns miss. Hashing the raw patterns is exactly the blind spot that forced `force` to exist. | +| `scope` (namespaced/cluster) | **Replace** with resolved namespace scope | An input to namespace resolution; the resolved scope subsumes it. | +| `operations` | **Keep** | Not part of GVR resolution. Changes *which events* become commits (the existing narrowing test depends on an operations-only change still emitting a snapshot). | +| destination: `provider` / `branch` / `path` | **Keep** | Not part of GVR resolution. Changes *where* writes land. | +| source rule identity (namespace/name) | **Drop** | Pure identity — it does not change the effective watch surface. A redundant duplicate rule resolving to the same plan should not trigger a no-op snapshot. | + +So a `targetWatchPlanEntry` is built from the *resolved* watch surface plus the +non-resolvable fields that still affect writes: + +- **resolved GVR** (the missing ingredient today) +- resolved namespace / cluster scope +- operations +- destination details that affect writes: provider, branch, and path + +When more than one rule for the same target resolves to the same GVR, the plan +entry's `operations` is the **union** across those rules — operations add up, +there is no first-wins precedence. This matches how matching already works: +`RuleStore.GetMatchingRules` evaluates per-operation and returns every rule whose +operation set covers it, so a resource covered by rule A=`[CREATE]` and rule +B=`[UPDATE]` is effectively watched for both. The plan builder must preserve that +union rather than letting one rule's operations shadow another's. This invariant +is pinned by `TestGetMatchingRules_OverlappingRulesUnionOperations` in +`internal/rulestore/store_test.go`. + +Then: + +- The union of all target plans determines which informers must run (this is + effectively what `computeRequestedGVRs` already does — see Open Questions on + whether to wrap or replace it). +- The per-target plan hash determines which GitTargets need snapshots. +- Global GVR churn no longer forces unrelated targets into snapshot mode. + +Why the resolved GVR matters: once it is in the hash, the wildcard-meets-new-CRD +case (target A's effective set grows even though its rule text is unchanged) +naturally changes A's hash and selects A — which is exactly the case that breaks +if you only remove the `force` flag without enriching the hash. This is what +lets the global `force` go away safely. + +This should simplify the mental model: one resolved plan drives both informer +lifecycle and snapshot decisions. + +### 3. Keep Informers Global, But Snapshots Target-Scoped + +Informer management can remain global and shared: + +- start informers needed by the union of all target plans +- stop informers no longer needed by any target + +Snapshot management should be target-scoped: + +- call `BeginReconciliationForStream` +- emit `RequestRepoState` +- emit `RequestClusterState` +- call `CompleteReconciliationForStream` + +only for targets whose effective plan hash changed, or targets directly affected +by a newly available resolved resource. + +### 4. Preserve Protection Against Initial ADDED Events + +The current snapshot path exists for a good reason. When a new informer starts, +its cache emits initial ADDED events. Those should not become a burst of +individual `[CREATE]` commits. + +The refined rule should be: + +- if a newly started informer is relevant to target A, buffer/snapshot target A +- if that informer is irrelevant to target B, do not move target B into + `RECONCILING` + +That preserves safety without globally disturbing every target. + +Note that if Section 2 is done — the per-target hash includes resolved GVRs — +this protection mostly falls out for free and may not need a separate +informer→target index. A new informer only starts because some target's resolved +GVR set grew; that target's hash therefore changes and it is already selected for +a snapshot. The hash diff and the "which targets depend on this new informer" +question have the same answer. Prefer the resolved-plan-hash mechanism over +maintaining a second informer→target mapping; treat the two as alternatives, not +complementary layers. + +### 5. Add an E2E Regression + +After unit coverage proves the target selection logic, add or adapt an e2e +regression: + +- run two non-Serial targets concurrently +- trigger a rule change for target B +- create a ConfigMap for target A +- assert target A still receives the expected event commit, not a snapshot commit + +This should be smaller and more intentional than relying on incidental +cross-spec timing in the full smoke suite. + +## Non-Goals + +- Do not make every manager spec `Serial` as the primary fix. That hides the + product coupling. +- Do not remove shared informers. Sharing informers is efficient and not the + core problem. +- Do not weaken event-commit assertions unless the product decision is that + snapshot commits are acceptable for unrelated rule changes. + +## Rollout Plan + +1. ✅ Done — `rule_change_snapshot_test.go` has a two-target isolation test plus + a control test; alongside them + `TestReconcileForRuleChange_TargetGainsAlreadyWatchedGVR_Snapshots` (issue + #146 under the new model) and + `TestCurrentRuleSetSnapshots_NamespacedWatchRulePlanByNamespace` (namespace + dimension) pin the surrounding contract. +2. ✅ Done — `currentRuleSetSnapshots` now hashes the effective watch plan via a + `targetWatchPlan`: it **replaces** the raw resource-matching patterns and + scope with the resolved GVR + namespace scope, **keeps** operations + (unioned across rules per GVR) and the destination (provider/branch/path), + and **drops** source rule identity. A target with no resolvable rules has an + empty plan and is omitted. +3. ✅ Done — `snapshotTargetsNeedingDelivery` no longer takes a `force` argument; + selection relies solely on per-target hash diffs. +4. ✅ Done — the new-informer ADDED-burst protection falls out of the enriched + hash (a new informer only starts when some target's resolved GVR set grew, so + that target's hash changes and it is selected). No separate informer→target + index was needed. +5. ✅ Done — `test/e2e/gittarget_isolation_e2e_test.go` ("Manager GitTarget + Isolation", non-Serial, smoke) runs two GitTargets in one repo at separate + paths, repeatedly churns target B's rule set while creating ConfigMaps for + target A, and asserts target A's path-scoped commit stays a `[CREATE]` event + commit (never `reconcile: sync`). +6. ✅ `task fmt`, `task vet`, `task lint`, `task test`, `task generate`, and + `task manifests` all pass with no drift. `task test-e2e` (smoke) green at the + default 2 procs (20/20) and at `--procs=4` (21/21, isolation spec included and + passing under real parallel contention — the scenario that originally flaked). + +## Open Questions + +- ~~Should path/provider/branch changes be part of the same watch-plan hash?~~ + They already are (`currentRuleSetSnapshots` hashes provider/branch/path today), + and they should stay: changing the write destination genuinely requires a + fresh snapshot to populate the new branch/path. Keep them in the hash. +- For removed rules, should the affected target always snapshot to remove stale + files, or should deletion cleanup be handled by a more explicit target-state + transition? +- Should the resolved-GVR plan builder replace `computeRequestedGVRs` / + `currentRuleSetSnapshots`, or wrap them? Recommendation: wrap initially. + `computeRequestedGVRs` drives the tested informer lifecycle; derive the + per-target resolved entries alongside it and feed them into the hash rather + than rewriting the resolver, to keep the change low-risk. + +## Recommendation + +Work on this before further e2e speed work. Parallel e2e uncovered a user-facing +isolation issue: unrelated GitTargets can change each other's commit semantics. +The cleanest end state is a per-target effective watch plan that drives both +global informer lifecycle and target-scoped snapshot decisions. diff --git a/internal/rulestore/store_test.go b/internal/rulestore/store_test.go index ff8aa25df..0212c75c8 100644 --- a/internal/rulestore/store_test.go +++ b/internal/rulestore/store_test.go @@ -386,6 +386,109 @@ func TestGetMatchingRules(t *testing.T) { } } +// TestGetMatchingRules_OverlappingRulesUnionOperations pins the semantics for +// "two WatchRules match the same resource": operations *add up* (union), there +// is no first-wins precedence and no single rule that "owns" the resource. +// +// Matching is evaluated per-operation: GetMatchingRules returns every rule whose +// operation set includes the operation being tested (an empty set means all). +// So for pods covered by rule A=[CREATE] and rule B=[UPDATE]: +// - a CREATE matches A only, +// - an UPDATE matches B only, +// - a DELETE matches neither (the union is exactly {CREATE, UPDATE}, not "all"), +// - and when two rules both cover CREATE, *both* are returned (additive, not +// deduped to one). +// +// This matters for the per-target effective watch plan +// (docs/design/gittarget-isolation-on-rule-change.md): a target's watched +// operation set for a GVR is the union across its rules, and each returned +// CompiledRule carries its own GitTarget, so two complementary rules pointing at +// the same target make that target watch the union. +func TestGetMatchingRules_OverlappingRulesUnionOperations(t *testing.T) { + store := NewStore() + + podsRule := func(name string, ops ...configv1alpha1.OperationType) configv1alpha1.WatchRule { + r := configv1alpha1.WatchRule{ + Spec: configv1alpha1.WatchRuleSpec{ + Rules: []configv1alpha1.ResourceRule{{ + Operations: ops, + APIGroups: []string{""}, + APIVersions: []string{"v1"}, + Resources: []string{"pods"}, + }}, + }, + } + r.Name = name + r.Namespace = "default" + return r + } + + // Two rules covering the same resource with complementary operations, plus a + // second CREATE rule to prove matches are additive rather than first-wins. + store.AddOrUpdateWatchRule( + podsRule("pod-create", configv1alpha1.OperationCreate), + "dest-a", "default", "repo", "gitops-system", "main", "a") + store.AddOrUpdateWatchRule( + podsRule("pod-update", configv1alpha1.OperationUpdate), + "dest-a", "default", "repo", "gitops-system", "main", "a") + store.AddOrUpdateWatchRule( + podsRule("pod-create-2", configv1alpha1.OperationCreate), + "dest-b", "default", "repo", "gitops-system", "main", "b") + + tests := []struct { + name string + operation configv1alpha1.OperationType + expectedNames []string + }{ + { + name: "CREATE returns both CREATE rules (additive, not first-wins)", + operation: configv1alpha1.OperationCreate, + expectedNames: []string{"pod-create", "pod-create-2"}, + }, + { + name: "UPDATE returns the UPDATE rule (operations union across rules)", + operation: configv1alpha1.OperationUpdate, + expectedNames: []string{"pod-update"}, + }, + { + name: "DELETE matches nothing (union is exactly CREATE+UPDATE, not all)", + operation: configv1alpha1.OperationDelete, + expectedNames: []string{}, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + matches := store.GetMatchingRules(nil, "pods", tt.operation, "", "v1", false) + + gotNames := make(map[string]struct{}, len(matches)) + for _, m := range matches { + gotNames[m.Source.Name] = struct{}{} + } + + if len(matches) != len(tt.expectedNames) { + t.Errorf("expected %d matches %v, got %d %v", + len(tt.expectedNames), tt.expectedNames, len(matches), keys(gotNames)) + } + for _, want := range tt.expectedNames { + if _, ok := gotNames[want]; !ok { + t.Errorf("expected rule %q to match %s, got %v", + want, tt.operation, keys(gotNames)) + } + } + }) + } +} + +// keys returns the keys of a set, for readable test failure messages. +func keys(m map[string]struct{}) []string { + out := make([]string, 0, len(m)) + for k := range m { + out = append(out, k) + } + return out +} + func TestRuleStore_Readiness(t *testing.T) { store := NewStore() if store.IsReady() { diff --git a/internal/watch/catalog_test_helpers_test.go b/internal/watch/catalog_test_helpers_test.go index d84c2c933..e2e719fed 100644 --- a/internal/watch/catalog_test_helpers_test.go +++ b/internal/watch/catalog_test_helpers_test.go @@ -120,35 +120,3 @@ func commonTestDiscoveryClient() func() (apiResourceDiscovery, error) { return newCommonTestDiscovery(), nil } } - -// snapshotDeliveryTestDiscovery returns a trusted-but-minimal discovery that -// serves only an unrelated resource. The snapshot-delivery tests register -// configmaps rules purely to exercise rule-set-change snapshot emission; with -// configmaps absent from trusted discovery, rule resolution requests no informer -// GVRs, so those tests never plan real informers. This replaces the old -// discoveryFilter stub that returned no GVRs. -func snapshotDeliveryTestDiscovery() staticCatalogDiscovery { - return staticCatalogDiscovery{ - groups: []*metav1.APIGroup{testAPIGroup("", "v1")}, - resources: []*metav1.APIResourceList{{ - GroupVersion: "v1", - APIResources: []metav1.APIResource{ - {Name: "nodes", Kind: "Node", Verbs: metav1.Verbs{"get", "list", "watch"}}, - }, - }}, - } -} - -func newSnapshotDeliveryTestCatalog(t *testing.T) *APIResourceCatalog { - t.Helper() - catalog := NewAPIResourceCatalog() - _, err := catalog.Refresh(snapshotDeliveryTestDiscovery()) - require.NoError(t, err) - return catalog -} - -func snapshotDeliveryTestDiscoveryClient() func() (apiResourceDiscovery, error) { - return func() (apiResourceDiscovery, error) { - return snapshotDeliveryTestDiscovery(), nil - } -} diff --git a/internal/watch/manager.go b/internal/watch/manager.go index c329f2448..70cfe4724 100644 --- a/internal/watch/manager.go +++ b/internal/watch/manager.go @@ -123,6 +123,64 @@ type ruleSetSnapshotTarget struct { hash uint64 } +// targetWatchPlan accumulates a single GitTarget's effective watch surface while +// currentRuleSetSnapshots walks the rule set. entries maps an entry key +// ("group/version/resource|scope|ns") to the union of operation tokens watched +// for it; dest holds the write destination. hash() folds both into a stable +// per-target hash. See currentRuleSetSnapshots for the rationale on what is and +// is not included. +type targetWatchPlan struct { + gitDest types.ResourceReference + entries map[string]map[string]struct{} + dest string +} + +// addEntry records that the target watches gvr in the given namespace ("" means +// all namespaces / cluster-scoped) for the given operations. An empty or +// wildcard operation set is canonicalised to "*" (all operations), and once "*" +// is present it subsumes any explicit operations during hashing. +func (p *targetWatchPlan) addEntry(gvr GVR, namespace string, ops []configv1alpha1.OperationType) { + entryKey := fmt.Sprintf("%s/%s/%s|scope=%s|ns=%s", + gvr.Group, gvr.Version, gvr.Resource, gvr.Scope, namespace) + opsSet := p.entries[entryKey] + if opsSet == nil { + opsSet = make(map[string]struct{}) + p.entries[entryKey] = opsSet + } + if len(ops) == 0 { + opsSet["*"] = struct{}{} + return + } + for _, op := range ops { + if op == configv1alpha1.OperationAll { + opsSet["*"] = struct{}{} + continue + } + opsSet[string(op)] = struct{}{} + } +} + +// hash returns a stable hash of the plan: destination plus each entry with its +// resolved operation set, sorted for determinism. +func (p *targetWatchPlan) hash() uint64 { + entryStrings := make([]string, 0, len(p.entries)) + for entryKey, opsSet := range p.entries { + var ops []string + if _, all := opsSet["*"]; all { + ops = []string{"*"} + } else { + ops = make([]string, 0, len(opsSet)) + for op := range opsSet { + ops = append(ops, op) + } + sort.Strings(ops) + } + entryStrings = append(entryStrings, entryKey+"|ops="+strings.Join(ops, ",")) + } + sort.Strings(entryStrings) + return xxhash.Sum64String(p.dest + "\x00" + strings.Join(entryStrings, "\x00")) +} + const ( heartbeatInterval = 30 * time.Second periodicReconcileInterval = 30 * time.Second @@ -705,7 +763,7 @@ func (m *Manager) ReconcileForRuleChange(ctx context.Context) error { activeCount := len(m.activeInformers) m.informersMu.Unlock() - targets := m.snapshotTargetsNeedingDelivery(len(added) > 0 || len(removed) > 0) + targets := m.snapshotTargetsNeedingDelivery() if len(added) == 0 && len(removed) == 0 && len(targets) == 0 { log.V(1).Info("No GVR changes detected, skipping reconciliation", "activeGVRs", activeCount) @@ -1103,7 +1161,13 @@ func splitResourceKey(key string) []string { return parts } -func (m *Manager) snapshotTargetsNeedingDelivery(force bool) []ruleSetSnapshotTarget { +// snapshotTargetsNeedingDelivery returns the GitTargets whose effective watch +// plan hash differs from the one last delivered — i.e. targets whose watched +// resource surface actually changed. Selection is purely per-target: global GVR +// churn for an unrelated target no longer drags every target into a snapshot. +// Targets whose hash is unchanged keep processing live events. See +// docs/design/gittarget-isolation-on-rule-change.md. +func (m *Manager) snapshotTargetsNeedingDelivery() []ruleSetSnapshotTarget { current := m.currentRuleSetSnapshots() currentKeys := make(map[string]struct{}, len(current)) @@ -1115,10 +1179,8 @@ func (m *Manager) snapshotTargetsNeedingDelivery(force bool) []ruleSetSnapshotTa for _, target := range current { key := target.gitDest.Key() currentKeys[key] = struct{}{} - if !force { - if lastDelivered, ok := m.lastDeliveredRuleSetHash[key]; ok && lastDelivered == target.hash { - continue - } + if lastDelivered, ok := m.lastDeliveredRuleSetHash[key]; ok && lastDelivered == target.hash { + continue } m.pendingRuleSetHash[key] = target.hash targets = append(targets, target) @@ -1139,58 +1201,88 @@ func (m *Manager) snapshotTargetsNeedingDelivery(force bool) []ruleSetSnapshotTa return targets } +// currentRuleSetSnapshots returns, per GitTarget, a hash of that target's +// *effective watch plan* — what it actually watches after rule resolution and +// API discovery, not the literal rule text. The hash drives snapshot selection +// in snapshotTargetsNeedingDelivery: a target is re-snapshotted only when its +// plan hash changes. +// +// A plan is the set of (resolved GVR, namespace scope, union of operations) +// entries the target watches, plus the write destination (provider/branch/path). +// Deliberately excluded: source rule identity (namespace/name) and the raw +// apiGroups/apiVersions/resources patterns. Those are inputs to resolution, not +// the resolved surface — keeping them caused unrelated churn (a redundant +// duplicate rule) and missed real churn (a wildcard newly matching a CRD). See +// docs/design/gittarget-isolation-on-rule-change.md. +// +// Operations add up across rules: when two rules for the same target resolve to +// the same GVR, the entry's operation set is their union (see +// rulestore TestGetMatchingRules_OverlappingRulesUnionOperations). A target with +// no resolvable rules has an empty plan and is omitted entirely. func (m *Manager) currentRuleSetSnapshots() []ruleSetSnapshotTarget { - entriesByTarget := make(map[string][]string) - refsByTarget := make(map[string]types.ResourceReference) + plans := make(map[string]*targetWatchPlan) + resolver := m.ruleGVRResolver() - for _, rule := range m.RuleStore.SnapshotWatchRules() { - ref := types.NewResourceReference(rule.GitTargetRef, rule.GitTargetNamespace) + plan := func(ref types.ResourceReference, providerNS, provider, branch, path string) *targetWatchPlan { key := ref.Key() - refsByTarget[key] = ref - entriesByTarget[key] = append(entriesByTarget[key], fmt.Sprintf( - "watch|source=%s/%s|target=%s/%s|provider=%s/%s|branch=%q|path=%q|rules=%#v", - rule.Source.Namespace, - rule.Source.Name, - rule.GitTargetNamespace, - rule.GitTargetRef, - rule.GitProviderNamespace, - rule.GitProviderRef, - rule.Branch, - rule.Path, - rule.ResourceRules, - )) + p := plans[key] + if p == nil { + p = &targetWatchPlan{gitDest: ref, entries: make(map[string]map[string]struct{})} + plans[key] = p + } + // Destination is a property of the GitTarget, so it is identical across + // that target's rules; recording it on each rule is harmless. + p.dest = fmt.Sprintf("provider=%s/%s|branch=%q|path=%q", providerNS, provider, branch, path) + return p + } + + // Namespaced WatchRules watch their own namespace (rule.Source.Namespace), + // so the namespace is part of the resolved scope even though the rule name is + // not part of the plan. + for _, rule := range m.RuleStore.SnapshotWatchRules() { + p := plan( + types.NewResourceReference(rule.GitTargetRef, rule.GitTargetNamespace), + rule.GitProviderNamespace, rule.GitProviderRef, rule.Branch, rule.Path, + ) + for _, rr := range rule.ResourceRules { + gvrs, _ := resolver.Resolve(rr.APIGroups, rr.APIVersions, rr.Resources, + configv1alpha1.ResourceScopeNamespaced) + for _, gvr := range gvrs { + p.addEntry(gvr, rule.Source.Namespace, rr.Operations) + } + } } + // ClusterWatchRules carry per-rule scope and watch all namespaces (or are + // cluster-scoped), so the plan namespace is empty. for _, rule := range m.RuleStore.SnapshotClusterWatchRules() { - ref := types.NewResourceReference(rule.GitTargetRef, rule.GitTargetNamespace) - key := ref.Key() - refsByTarget[key] = ref - entriesByTarget[key] = append(entriesByTarget[key], fmt.Sprintf( - "cluster|source=%s|target=%s/%s|provider=%s/%s|branch=%q|path=%q|rules=%#v", - rule.Source.Name, - rule.GitTargetNamespace, - rule.GitTargetRef, - rule.GitProviderNamespace, - rule.GitProviderRef, - rule.Branch, - rule.Path, - rule.Rules, - )) - } - - keys := make([]string, 0, len(entriesByTarget)) - for key := range entriesByTarget { + p := plan( + types.NewResourceReference(rule.GitTargetRef, rule.GitTargetNamespace), + rule.GitProviderNamespace, rule.GitProviderRef, rule.Branch, rule.Path, + ) + for _, rr := range rule.Rules { + gvrs, _ := resolver.Resolve(rr.APIGroups, rr.APIVersions, rr.Resources, rr.Scope) + for _, gvr := range gvrs { + p.addEntry(gvr, "", rr.Operations) + } + } + } + + keys := make([]string, 0, len(plans)) + for key := range plans { keys = append(keys, key) } sort.Strings(keys) targets := make([]ruleSetSnapshotTarget, 0, len(keys)) for _, key := range keys { - entries := entriesByTarget[key] - sort.Strings(entries) + p := plans[key] + if len(p.entries) == 0 { + continue + } targets = append(targets, ruleSetSnapshotTarget{ - gitDest: refsByTarget[key], - hash: xxhash.Sum64String(strings.Join(entries, "\x00")), + gitDest: p.gitDest, + hash: p.hash(), }) } return targets diff --git a/internal/watch/rule_change_snapshot_test.go b/internal/watch/rule_change_snapshot_test.go index 44fda13d3..0fc33bd87 100644 --- a/internal/watch/rule_change_snapshot_test.go +++ b/internal/watch/rule_change_snapshot_test.go @@ -49,17 +49,17 @@ import ( // Each test asserts the contract directly: the snapshot count goes up when the // rule-set changes, regardless of whether the GVR set changed. -// makeRuleChangeTestManager constructs a Manager configured for these tests: -// fake controller-runtime client, real RuleStore, a trusted-but-minimal API -// resource catalog that does not serve configmaps (so rule resolution plans no -// informer GVRs and no real informers start), and a target GitTarget already in -// the fake client. +// makeRuleChangeTestManager constructs a single-GitTarget Manager for these +// tests: fake controller-runtime client, real RuleStore, the common catalog +// (which serves configmaps and secrets, so configmaps rules resolve to a real +// GVR and the target's effective watch plan is non-empty), and a target +// GitTarget already in the fake client. // -// The configmaps GVR is also pre-populated in activeInformers with a no-op -// cancel so compareGVRs sees the GVR as "already watched". This is the -// steady-state we want to model: an informer for that GVR is logically -// running, so adding a second rule for it should not cause an informer churn, -// but *should* re-trigger a snapshot. +// The configmaps GVR is pre-populated in activeInformers (cluster-wide) so +// compareGVRs reports added=0, removed=0 for it: the informer is logically +// already running, and no real informer is started. This isolates the snapshot +// *decision* (driven by the per-target effective-watch-plan hash) from informer +// lifecycle. func makeRuleChangeTestManager(t *testing.T) (*Manager, *rulestore.RuleStore) { t.Helper() @@ -82,36 +82,27 @@ func makeRuleChangeTestManager(t *testing.T) (*Manager, *rulestore.RuleStore) { Build() store := rulestore.NewStore() - configMapGVR := GVR{ - Group: "", - Version: "v1", - Resource: "configmaps", - Scope: configv1alpha1.ResourceScopeNamespaced, - } manager := &Manager{ - Client: fakeClient, - Log: logr.Discard(), - RuleStore: store, - // The catalog does not serve configmaps, so these tests exercise - // snapshot delivery after rule-set changes without planning real - // informers. - resourceCatalog: newSnapshotDeliveryTestCatalog(t), - discoveryClient: snapshotDeliveryTestDiscoveryClient(), + Client: fakeClient, + Log: logr.Discard(), + RuleStore: store, + resourceCatalog: newCommonTestCatalog(t), + discoveryClient: commonTestDiscoveryClient(), // Pretend the configmaps informer is already running cluster-wide, // so compareGVRs reports added=0, removed=0 for that GVR. No real // informers are started. activeInformers: map[GVR]map[string]context.CancelFunc{ - configMapGVR: {"": func() {}}, + namespacedGVR("configmaps"): {"": func() {}}, }, } return manager, store } -// configMapRuleForTarget builds a ClusterWatchRule that selects core/v1 -// configmaps, scoped at the given GitTarget (always in the shared "test-ns" -// namespace used across these tests). -func configMapRuleForTarget(name, gitTargetName string) configv1alpha1.ClusterWatchRule { +// clusterRuleForResource builds a ClusterWatchRule that selects a single core/v1 +// namespaced resource, scoped at the given GitTarget (always in the shared +// "test-ns" namespace used across these tests). +func clusterRuleForResource(name, gitTargetName, resource string) configv1alpha1.ClusterWatchRule { return configv1alpha1.ClusterWatchRule{ ObjectMeta: metav1.ObjectMeta{Name: name}, Spec: configv1alpha1.ClusterWatchRuleSpec{ @@ -122,67 +113,72 @@ func configMapRuleForTarget(name, gitTargetName string) configv1alpha1.ClusterWa Rules: []configv1alpha1.ClusterResourceRule{{ APIGroups: []string{""}, APIVersions: []string{"v1"}, - Resources: []string{"configmaps"}, + Resources: []string{resource}, Scope: configv1alpha1.ResourceScopeNamespaced, }}, }, } } -// TestReconcileForRuleChange_AddingSecondRuleSameGVR_EmitsSnapshot exercises -// the central case. After the first rule is registered and reconciled, the -// informer set is steady-state. Adding a *second* rule that names the same GVR -// does not change the GVR set — only the rule-set changes. The contract: any -// rule-set change must still emit a snapshot for affected GitTargets, so -// resources that already match the new rule are backfilled to git. -func TestReconcileForRuleChange_AddingSecondRuleSameGVR_EmitsSnapshot(t *testing.T) { - manager, store := makeRuleChangeTestManager(t) - ctx := context.Background() +// configMapRuleForTarget builds a configmaps ClusterWatchRule for the target. +func configMapRuleForTarget(name, gitTargetName string) configv1alpha1.ClusterWatchRule { + return clusterRuleForResource(name, gitTargetName, "configmaps") +} + +// TestReconcileForRuleChange_TargetGainsAlreadyWatchedGVR_Snapshots is the +// issue #146 case under the per-target plan model. A GVR can already be watched +// globally (because another target's rule covers it) while a particular target +// has never received it. When that target gains a rule for the GVR, no informer +// churn happens (added=0, removed=0 for that GVR), yet the target's effective +// watch plan grows — so it must still snapshot to backfill the already-present +// resources into its repo. +// +// Here target-a watches configmaps and target-b watches secrets. configmaps is +// already active. target-b then also starts watching configmaps: target-b's +// plan grows and it must be selected, even though no new configmaps informer is +// needed. +func TestReconcileForRuleChange_TargetGainsAlreadyWatchedGVR_Snapshots(t *testing.T) { + manager, store := makeTwoTargetRuleChangeManager(t) - // First rule: register, reconcile once. This brings the manager into a - // stable state where the configmaps informer is logically active. store.AddOrUpdateClusterWatchRule( - configMapRuleForTarget("rule-1", "test-target"), - "test-target", "test-ns", - "test-provider", "test-ns", - "main", "test-path", + clusterRuleForResource("rule-a", "target-a", "configmaps"), + "target-a", "test-ns", "test-provider", "test-ns", "main", "path-a", ) - require.NoError(t, manager.ReconcileForRuleChange(ctx)) - - // Reset counter so the assertion targets only the *second* reconcile. - manager.snapshotEmitCount.Store(0) + store.AddOrUpdateClusterWatchRule( + clusterRuleForResource("rule-b", "target-b", "secrets"), + "target-b", "test-ns", "test-provider", "test-ns", "main", "path-b", + ) + seedDeliveredBaseline(manager) - // Second rule for the same GVR. The GVR diff is empty; only the rule-set - // changes — ReconcileForRuleChange must still emit a snapshot. + // target-b now also watches configmaps — already watched by target-a, so no + // new informer, but target-b's plan grows. store.AddOrUpdateClusterWatchRule( - configMapRuleForTarget("rule-2", "test-target"), - "test-target", "test-ns", - "test-provider", "test-ns", - "main", "test-path", + clusterRuleForResource("rule-b-configmaps", "target-b", "configmaps"), + "target-b", "test-ns", "test-provider", "test-ns", "main", "path-b", ) - require.NoError(t, manager.ReconcileForRuleChange(ctx)) - assert.Positive(t, - manager.SnapshotEmitCount(), - "adding a ClusterWatchRule should emit a snapshot even when no new GVR is introduced; "+ - "existing matching resources must be backfilled to git") + require.NoError(t, manager.ReconcileForRuleChange(ctx())) + + assert.True(t, targetPending(manager, "target-b"), + "target-b gained coverage of an already-watched GVR; it must snapshot to backfill "+ + "those resources even though no informer churn occurred (issue #146)") + assert.False(t, targetPending(manager, "target-a"), + "target-a's plan is unchanged and must not snapshot") } -// TestReconcileForRuleChange_NarrowingSelectorOnExistingRule_EmitsSnapshot -// covers a sibling gap: editing a rule's selectors without changing its GVR. -// A common operator action is "I added a label selector to my existing rule to -// scope it down" — the GVR set does not change, but the matched resource set -// does. Same root cause as the test above. Uses a distinct GitTarget name -// from the prior test so the same helper exercises more than one input. -func TestReconcileForRuleChange_NarrowingSelectorOnExistingRule_EmitsSnapshot(t *testing.T) { +// TestReconcileForRuleChange_NarrowingOperationsOnExistingRule_Snapshots covers +// editing a rule's operations without changing its GVR. The resolved GVR is +// unchanged, but operations are part of the effective watch plan, so the plan +// hash changes and the target must snapshot. +func TestReconcileForRuleChange_NarrowingOperationsOnExistingRule_Snapshots(t *testing.T) { manager, store := makeRuleChangeTestManager(t) ctx := context.Background() - // Initial rule: selects all configmaps via narrow-target. - initial := configMapRuleForTarget("rule-1", "narrow-target") + // Initial rule: all operations on configmaps. + initial := configMapRuleForTarget("rule-1", "test-target") store.AddOrUpdateClusterWatchRule( initial, - "narrow-target", "test-ns", + "test-target", "test-ns", "test-provider", "test-ns", "main", "test-path", ) @@ -190,15 +186,15 @@ func TestReconcileForRuleChange_NarrowingSelectorOnExistingRule_EmitsSnapshot(t manager.snapshotEmitCount.Store(0) - // Edit the rule — narrow its selectors. The GVR (core/v1/configmaps) is - // unchanged; only the rule body changes. - narrowed := initial + // Narrow operations to UPDATE only. The GVR (core/v1/configmaps) is + // unchanged; only the operation set in the plan changes. + narrowed := configMapRuleForTarget("rule-1", "test-target") narrowed.Spec.Rules[0].Operations = []configv1alpha1.OperationType{ configv1alpha1.OperationUpdate, } store.AddOrUpdateClusterWatchRule( narrowed, - "narrow-target", "test-ns", + "test-target", "test-ns", "test-provider", "test-ns", "main", "test-path", ) @@ -206,8 +202,8 @@ func TestReconcileForRuleChange_NarrowingSelectorOnExistingRule_EmitsSnapshot(t assert.Positive(t, manager.SnapshotEmitCount(), - "editing a ClusterWatchRule should emit a snapshot when the matched resource set may change, "+ - "even if the GVR set is unchanged") + "narrowing a rule's operations changes the effective watch plan and must emit a snapshot, "+ + "even though the resolved GVR set is unchanged") } // watchRuleForTarget builds a namespaced WatchRule selecting core/v1 @@ -226,38 +222,48 @@ func watchRuleForTarget(name, gitTargetName, namespace string) configv1alpha1.Wa } } -// TestReconcileForRuleChange_AddingSecondWatchRuleSameGVR_EmitsSnapshot is the -// namespaced-WatchRule analogue of the ClusterWatchRule test above. The -// underlying code path is the same (rules feed into ComputeRequestedGVRs and -// collectAffectedGitTargets via the shared RuleStore), so the same gap -// applies. Documenting it as a separate test makes the regression coverage -// for both rule kinds explicit. -func TestReconcileForRuleChange_AddingSecondWatchRuleSameGVR_EmitsSnapshot(t *testing.T) { +// TestCurrentRuleSetSnapshots_NamespacedWatchRulePlanByNamespace exercises the +// namespace dimension of the effective watch plan for namespaced WatchRules, +// directly at the hash level (no informer lifecycle). A WatchRule watches its +// own namespace, so the same GVR in a different namespace is a different watch +// surface and must change the target's plan hash — while a redundant duplicate +// WatchRule in an already-watched namespace must not. +func TestCurrentRuleSetSnapshots_NamespacedWatchRulePlanByNamespace(t *testing.T) { manager, store := makeRuleChangeTestManager(t) - ctx := context.Background() + hashForTarget := func() uint64 { + for _, target := range manager.currentRuleSetSnapshots() { + if target.gitDest.Name == "test-target" { + return target.hash + } + } + t.Fatal("test-target not present in rule-set snapshots") + return 0 + } + + // WatchRule watching configmaps in ns-a. store.AddOrUpdateWatchRule( - watchRuleForTarget("rule-1", "test-target", "test-ns"), - "test-target", "test-ns", - "test-provider", "test-ns", - "main", "test-path", + watchRuleForTarget("rule-a", "test-target", "ns-a"), + "test-target", "test-ns", "test-provider", "test-ns", "main", "test-path", ) - require.NoError(t, manager.ReconcileForRuleChange(ctx)) - - manager.snapshotEmitCount.Store(0) + hashA := hashForTarget() + // Second WatchRule in a different namespace (ns-b): the plan grows. store.AddOrUpdateWatchRule( - watchRuleForTarget("rule-2", "test-target", "test-ns"), - "test-target", "test-ns", - "test-provider", "test-ns", - "main", "test-path", + watchRuleForTarget("rule-b", "test-target", "ns-b"), + "test-target", "test-ns", "test-provider", "test-ns", "main", "test-path", ) - require.NoError(t, manager.ReconcileForRuleChange(ctx)) + hashAB := hashForTarget() + assert.NotEqual(t, hashA, hashAB, + "a WatchRule adding the same GVR in a new namespace expands the watch surface and must change the plan hash") - assert.Positive(t, - manager.SnapshotEmitCount(), - "adding a second WatchRule should emit a snapshot even when no new GVR is introduced; "+ - "the bug is symmetric across WatchRule and ClusterWatchRule") + // Redundant duplicate WatchRule in an already-watched namespace (ns-a). + store.AddOrUpdateWatchRule( + watchRuleForTarget("rule-a-dup", "test-target", "ns-a"), + "test-target", "test-ns", "test-provider", "test-ns", "main", "test-path", + ) + assert.Equal(t, hashAB, hashForTarget(), + "a redundant duplicate WatchRule in an already-watched namespace must not change the plan hash") } // TestReconcileForRuleChange_RestartLikeBootstrap_NoSnapshotDrops models the @@ -328,14 +334,16 @@ func TestReconcileForRuleChange_RestartLikeBootstrap_NoSnapshotDrops(t *testing. Log: logr.Discard(), RuleStore: store, dynamicClient: dynamicfake.NewSimpleDynamicClient(scheme, preexistingCM), - // The catalog does not serve configmaps, so the rule's GVR resolves to - // nothing and does not hit startInformersForGVRs. Combined with a stale - // entry in activeInformers, compareGVRs reports removed > 0 and the - // snapshot-emit path runs. - resourceCatalog: newSnapshotDeliveryTestCatalog(t), - discoveryClient: snapshotDeliveryTestDiscoveryClient(), + // The common catalog serves configmaps, so the rule resolves to a real + // GVR and the target's effective watch plan is non-empty (the target is + // selected for a snapshot). configmaps is pre-activated so no informer is + // started; the stale GVR is removed so compareGVRs avoids the + // no-change early-return. + resourceCatalog: newCommonTestCatalog(t), + discoveryClient: commonTestDiscoveryClient(), activeInformers: map[GVR]map[string]context.CancelFunc{ - staleGVR: {"": func() {}}, + namespacedGVR("configmaps"): {"": func() {}}, + staleGVR: {"": func() {}}, }, } @@ -360,6 +368,210 @@ func TestReconcileForRuleChange_RestartLikeBootstrap_NoSnapshotDrops(t *testing. "emission with reconciler registration, or retry, so pre-existing matching resources land in git") } +// makeTwoTargetRuleChangeManager builds a Manager with two GitTargets, +// "target-a" and "target-b", both in "test-ns". +// +// The common catalog serves configmaps and secrets, so rules for them resolve +// to real GVRs and each target's effective watch plan is non-empty. Those +// resolved GVRs are pre-populated in activeInformers, so compareGVRs reports +// them as already watched (added=0) and no real informer is started. A stale +// apps/v1 deployments GVR that no rule wants is also pre-populated: the next +// ReconcileForRuleChange drops it (removed>0), modelling unrelated *global* +// informer churn happening concurrently — again without starting any informer. +// +// EventRouter is left nil. snapshotTargetsNeedingDelivery records every selected +// target in pendingRuleSetHash *before* emission, and with no EventRouter the +// emit path is a no-op that never marks targets delivered, so pendingRuleSetHash +// is a faithful record of which targets were selected for a rule-change snapshot. +func makeTwoTargetRuleChangeManager(t *testing.T) (*Manager, *rulestore.RuleStore) { + t.Helper() + + scheme := runtime.NewScheme() + require.NoError(t, clientgoscheme.AddToScheme(scheme)) + require.NoError(t, configv1alpha1.AddToScheme(scheme)) + + targetA := &configv1alpha1.GitTarget{ + ObjectMeta: metav1.ObjectMeta{Name: "target-a", Namespace: "test-ns"}, + Spec: configv1alpha1.GitTargetSpec{ + ProviderRef: configv1alpha1.GitProviderReference{Name: "test-provider"}, + Branch: "main", + Path: "path-a", + }, + } + targetB := &configv1alpha1.GitTarget{ + ObjectMeta: metav1.ObjectMeta{Name: "target-b", Namespace: "test-ns"}, + Spec: configv1alpha1.GitTargetSpec{ + ProviderRef: configv1alpha1.GitProviderReference{Name: "test-provider"}, + Branch: "main", + Path: "path-b", + }, + } + + fakeClient := fake.NewClientBuilder(). + WithScheme(scheme). + WithObjects(targetA, targetB). + Build() + + staleGVR := GVR{ + Group: "apps", Version: "v1", Resource: "deployments", + Scope: configv1alpha1.ResourceScopeNamespaced, + } + manager := &Manager{ + Client: fakeClient, + Log: logr.Discard(), + RuleStore: rulestore.NewStore(), + resourceCatalog: newCommonTestCatalog(t), + discoveryClient: commonTestDiscoveryClient(), + activeInformers: map[GVR]map[string]context.CancelFunc{ + namespacedGVR("configmaps"): {"": func() {}}, + namespacedGVR("secrets"): {"": func() {}}, + staleGVR: {"": func() {}}, + }, + } + + return manager, manager.RuleStore +} + +// namespacedGVR is a terse constructor for a core/v1 namespaced GVR. +func namespacedGVR(resource string) GVR { + return GVR{Group: "", Version: "v1", Resource: resource, Scope: configv1alpha1.ResourceScopeNamespaced} +} + +// seedDeliveredBaseline registers the current rule set's hashes as already +// delivered, modelling the steady state where every target's snapshot is up to +// date. After this, only a target whose hash *changes* should be re-selected. +func seedDeliveredBaseline(m *Manager) { + for _, target := range m.currentRuleSetSnapshots() { + m.markRuleSetSnapshotDelivered(target) + } +} + +// targetPending reports whether the given GitTarget (in the shared "test-ns" +// namespace these tests use) was selected for a rule-change snapshot on the last +// reconcile. +func targetPending(m *Manager, name string) bool { + key := types.NewResourceReference(name, "test-ns").Key() + m.ruleSetSnapshotMu.Lock() + defer m.ruleSetSnapshotMu.Unlock() + _, ok := m.pendingRuleSetHash[key] + return ok +} + +// TestReconcileForRuleChange_UnrelatedTargetNotSnapshotted_OnGlobalGVRChurn is +// the GitTarget-isolation invariant from +// docs/design/gittarget-isolation-on-rule-change.md. +// +// Two GitTargets, A and B, both watch configmaps and are in steady state (their +// snapshots already delivered). Then *only* target B's effective watch plan +// changes — it starts watching secrets too — while at the same moment unrelated +// global informer churn occurs (a stale GVR is dropped). Target A's plan is +// untouched. +// +// Expected: only B is selected for a rule-change snapshot; A keeps processing +// live events and is not dragged into RECONCILING. +// +// This is the regression the design doc set out to remove: previously +// ReconcileForRuleChange passed `len(added) > 0 || len(removed) > 0` as a global +// `force` flag that bypassed the per-target hash check and selected *every* +// target, so unrelated churn dragged A into a snapshot too. Snapshot selection +// is now purely per-target, so this passes. +func TestReconcileForRuleChange_UnrelatedTargetNotSnapshotted_OnGlobalGVRChurn(t *testing.T) { + manager, store := makeTwoTargetRuleChangeManager(t) + + // Both targets watch configmaps. + store.AddOrUpdateClusterWatchRule( + clusterRuleForResource("rule-a", "target-a", "configmaps"), + "target-a", "test-ns", "test-provider", "test-ns", "main", "path-a", + ) + store.AddOrUpdateClusterWatchRule( + clusterRuleForResource("rule-b", "target-b", "configmaps"), + "target-b", "test-ns", "test-provider", "test-ns", "main", "path-b", + ) + + // Steady state: both snapshots already delivered. + seedDeliveredBaseline(manager) + + // Change ONLY target B's effective plan: it now also watches secrets. + // A is untouched. + store.AddOrUpdateClusterWatchRule( + clusterRuleForResource("rule-b-secrets", "target-b", "secrets"), + "target-b", "test-ns", "test-provider", "test-ns", "main", "path-b", + ) + + require.NoError(t, manager.ReconcileForRuleChange(ctx())) + + assert.True(t, targetPending(manager, "target-b"), + "target B's effective watch plan changed (now watches secrets), so B must be selected") + assert.False(t, targetPending(manager, "target-a"), + "target A's effective watch plan did not change; unrelated global GVR churn must not "+ + "drag A into a rule-change snapshot. GitTargets are an isolation boundary — see "+ + "docs/design/gittarget-isolation-on-rule-change.md") +} + +// TestReconcileForRuleChange_RedundantDuplicateRule_DoesNotSnapshot pins the +// flip side of the isolation fix: because the hash is over the *effective watch +// plan* (resolved GVR + scope + operations), a second rule that resolves to the +// same surface a target already watches does not change the plan and must not +// trigger a no-op snapshot. This is what dropping source rule identity buys. +func TestReconcileForRuleChange_RedundantDuplicateRule_DoesNotSnapshot(t *testing.T) { + manager, store := makeTwoTargetRuleChangeManager(t) + + store.AddOrUpdateClusterWatchRule( + clusterRuleForResource("rule-a", "target-a", "configmaps"), + "target-a", "test-ns", "test-provider", "test-ns", "main", "path-a", + ) + seedDeliveredBaseline(manager) + + // A second rule that resolves to the exact same surface (configmaps) for the + // same target — a redundant duplicate. The effective plan is unchanged. + store.AddOrUpdateClusterWatchRule( + clusterRuleForResource("rule-a-dup", "target-a", "configmaps"), + "target-a", "test-ns", "test-provider", "test-ns", "main", "path-a", + ) + + require.NoError(t, manager.ReconcileForRuleChange(ctx())) + + assert.False(t, targetPending(manager, "target-a"), + "a redundant duplicate rule does not change the effective watch plan and must not snapshot") +} + +// TestSnapshotTargetsNeedingDelivery_PerTargetHashIsolatesTargets exercises the +// selection function directly (no full reconcile): with two targets in steady +// state and only B's plan changing, only B is returned. This is the unit-level +// guard for the per-target hash diff that snapshot selection now relies on. +func TestSnapshotTargetsNeedingDelivery_PerTargetHashIsolatesTargets(t *testing.T) { + manager, store := makeTwoTargetRuleChangeManager(t) + + store.AddOrUpdateClusterWatchRule( + clusterRuleForResource("rule-a", "target-a", "configmaps"), + "target-a", "test-ns", "test-provider", "test-ns", "main", "path-a", + ) + store.AddOrUpdateClusterWatchRule( + clusterRuleForResource("rule-b", "target-b", "configmaps"), + "target-b", "test-ns", "test-provider", "test-ns", "main", "path-b", + ) + seedDeliveredBaseline(manager) + + store.AddOrUpdateClusterWatchRule( + clusterRuleForResource("rule-b-secrets", "target-b", "secrets"), + "target-b", "test-ns", "test-provider", "test-ns", "main", "path-b", + ) + + selected := manager.snapshotTargetsNeedingDelivery() + + keyA := types.NewResourceReference("target-a", "test-ns").Key() + keyB := types.NewResourceReference("target-b", "test-ns").Key() + selectedKeys := make(map[string]struct{}, len(selected)) + for _, target := range selected { + selectedKeys[target.gitDest.Key()] = struct{}{} + } + + assert.Contains(t, selectedKeys, keyB, + "B (whose effective plan changed) is correctly selected") + assert.NotContains(t, selectedKeys, keyA, + "A (whose effective plan is unchanged) is correctly skipped") +} + // ctx returns a background context. Wrapped for terse use in test setup. func ctx() context.Context { return context.Background() } diff --git a/test/e2e/Taskfile.yml b/test/e2e/Taskfile.yml index bb03262b2..dc6636e23 100644 --- a/test/e2e/Taskfile.yml +++ b/test/e2e/Taskfile.yml @@ -102,7 +102,10 @@ vars: # Number of parallel Ginkgo processes for the e2e suite. >1 enables Ginkgo # parallelism (Phase 2.5 of docs/design/e2e-speedup-plan.md). Files that touch # shared cluster-scoped state are marked Serial (see docs/design/e2e-serial-registry.md). - E2E_GINKGO_PROCS: '{{.E2E_GINKGO_PROCS | default "2"}}' + # Default 4 locally: GitTarget isolation on rule changes is now per-target + # (docs/design/gittarget-isolation-on-rule-change.md), so higher parallelism no + # longer trips cross-target snapshot coupling. + E2E_GINKGO_PROCS: '{{.E2E_GINKGO_PROCS | default "4"}}' FLUX_SERVICES_DIR: '{{.FLUX_SERVICES_DIR | default "test/e2e/setup/flux"}}' DEMO_MANIFESTS_DIR: 'test/e2e/setup/demo-only' DEMO_TUNNEL_CREDENTIALS: '{{.DEMO_MANIFESTS_DIR}}/cloudflared-public/tunnel-credentials.yaml' diff --git a/test/e2e/gittarget_isolation_e2e_test.go b/test/e2e/gittarget_isolation_e2e_test.go new file mode 100644 index 000000000..d9b1ff68d --- /dev/null +++ b/test/e2e/gittarget_isolation_e2e_test.go @@ -0,0 +1,203 @@ +/* +SPDX-License-Identifier: Apache-2.0 + +Copyright 2025 ConfigButler + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "fmt" + "os/exec" + "path" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +// This spec is the e2e regression for GitTarget isolation on rule changes (see +// docs/design/gittarget-isolation-on-rule-change.md). The original symptom was a +// parallel-run flake: one GitTarget's ConfigMap event commit ("[CREATE] ...") +// was replaced by a snapshot commit ("reconcile: sync N resources") because an +// unrelated spec changed a *different* target's rules at the same time, dragging +// every target into rule-change snapshot mode. +// +// It is deliberately NOT Serial — the whole point is that isolation must hold +// under parallel execution. It is Label("smoke") so it runs in the same suite +// where the flake originally appeared. +var _ = Describe("Manager GitTarget Isolation", Label("manager"), Label("smoke"), Ordered, func() { + const ( + providerName = "gitprovider-iso" + targetA = "iso-target-a" + targetB = "iso-target-b" + ruleA = "iso-rule-a" + ruleB = "iso-rule-b" + pathA = "e2e/iso-a" + pathB = "e2e/iso-b" + ) + + var ( + testNs string + isoRepo *RepoArtifacts + ) + + BeforeAll(func() { + By("creating GitTarget isolation test namespace") + testNs = testNamespaceFor("manager-isolation") + _, _ = kubectlRun("create", "namespace", testNs) // idempotent; ignore AlreadyExists + + By("setting up Gitea repo and credentials for isolation tests") + isoRepo = SetupRepo( + resolveE2EContext(), + testNs, + fmt.Sprintf("e2e-manager-isolation-%d", GinkgoRandomSeed()), + ) + + By("applying git secrets to test namespace") + _, err := kubectlRunInNamespace(testNs, "apply", "-f", isoRepo.SecretsYAML) + Expect(err).NotTo(HaveOccurred(), "failed to apply git secrets to test namespace") + applySOPSAgeKeyToNamespace(testNs) + + By("creating shared GitProvider for isolation specs") + createGitProviderWithURLInNamespace( + providerName, + testNs, + isoRepo.GitSecretHTTP, + isoRepo.RepoURLHTTP, + ) + verifyResourceStatus( + "gitprovider", providerName, testNs, + "True", "Ready", "Repository connectivity validated", + ) + + By("creating two independent GitTargets writing to separate paths in the same repo") + createGitTarget(targetA, testNs, providerName, pathA, "main") + createGitTarget(targetB, testNs, providerName, pathB, "main") + verifyResourceStatus("gittarget", targetA, testNs, "True", "Ready", "") + verifyResourceStatus("gittarget", targetB, testNs, "True", "Ready", "") + + By("target A and target B both watch ConfigMaps (baseline steady state)") + applyIsolationWatchRule(ruleA, testNs, targetA, `"configmaps"`) + applyIsolationWatchRule(ruleB, testNs, targetB, `"configmaps"`) + verifyResourceStatus("watchrule", ruleA, testNs, "True", "Ready", "") + verifyResourceStatus("watchrule", ruleB, testNs, "True", "Ready", "") + }) + + AfterAll(func() { + cleanupNamespace(testNs) + }) + + SetDefaultEventuallyTimeout(30 * time.Second) + SetDefaultEventuallyPollingInterval(time.Second) + + It("keeps target A's commits as events while target B's rules churn", Label("smoke"), func() { + // Let any in-flight reconciles from setup drain before we begin, so the + // baseline snapshot commits are settled and only our event commits land + // next on target A's path. + time.Sleep(5 * time.Second) + + // Each iteration changes target B's effective watch plan (toggling an + // extra GVR on/off, which also churns the global informer set) and then + // creates a brand-new ConfigMap that target A must commit. If isolation + // holds, target A always produces an event commit; pre-fix, target B's + // churn would force target A into a "reconcile: sync" snapshot. + for i := range 3 { + cmName := fmt.Sprintf("iso-cm-%d", i) + + By(fmt.Sprintf("churning target B's rule set (iteration %d)", i)) + if i%2 == 0 { + applyIsolationWatchRule(ruleB, testNs, targetB, `"configmaps", "services"`) + } else { + applyIsolationWatchRule(ruleB, testNs, targetB, `"configmaps"`) + } + + By(fmt.Sprintf("creating ConfigMap %q for target A", cmName)) + applyIsolationConfigMap(cmName, testNs) + + By("asserting target A commits it as an event commit, not a snapshot") + relPath := path.Join(pathA, fmt.Sprintf("v1/configmaps/%s/%s.yaml", testNs, cmName)) + assertEventCommit := func(g Gomega) { + gitPull(g, isoRepo.CheckoutDir) + + msg := lastCommitMessageForPath(g, isoRepo.CheckoutDir, relPath) + g.Expect(msg).To(ContainSubstring("[CREATE]"), + "target A's commit for %s must be a [CREATE] event commit", cmName) + g.Expect(msg).To(ContainSubstring(fmt.Sprintf("v1/configmaps/%s", cmName)), + "target A's commit message must name the resource path") + g.Expect(msg).NotTo(ContainSubstring("reconcile: sync"), + "target A must not enter snapshot mode because of target B's unrelated rule change "+ + "(GitTarget isolation — see docs/design/gittarget-isolation-on-rule-change.md)") + } + Eventually(assertEventCommit).Should(Succeed()) + } + }) +}) + +// applyIsolationWatchRule applies a WatchRule selecting the given resources list +// (a raw YAML array body, e.g. `"configmaps", "services"`) for the target. +func applyIsolationWatchRule(name, namespace, target, resources string) { + data := struct { + Name string + Namespace string + DestinationName string + Resources string + }{ + Name: name, + Namespace: namespace, + DestinationName: target, + Resources: resources, + } + Expect(applyFromTemplate("test/e2e/templates/manager/watchrule-resources.tmpl", data, namespace)). + To(Succeed(), "failed to apply isolation WatchRule %q", name) +} + +// applyIsolationConfigMap creates a watched ConfigMap, attributed to a fixed +// user so the commit is a genuine event commit. +func applyIsolationConfigMap(name, namespace string) { + data := struct { + Name string + Namespace string + }{ + Name: name, + Namespace: namespace, + } + Expect(applyFromTemplate( + "test/e2e/templates/manager/configmap.tmpl", + data, + namespace, + "--as=jane@acme.com", + )).To(Succeed(), "failed to apply isolation ConfigMap %q", name) +} + +// gitPull pulls the latest changes into a checkout. It does not use utils.Run, +// which would override cmd.Dir with the project directory. +func gitPull(g Gomega, checkoutDir string) { + cmd := exec.Command("git", "pull") + cmd.Dir = checkoutDir + out, err := cmd.CombinedOutput() + g.Expect(err).NotTo(HaveOccurred(), "git pull failed: %s", string(out)) +} + +// lastCommitMessageForPath returns the body of the most recent commit that +// touched the given repo-relative path. Scoping by path keeps each GitTarget's +// history unambiguous even when several targets share one repo. +func lastCommitMessageForPath(g Gomega, checkoutDir, relPath string) string { + cmd := exec.Command("git", "log", "-1", "--pretty=%B", "--", relPath) + cmd.Dir = checkoutDir + out, err := cmd.CombinedOutput() + g.Expect(err).NotTo(HaveOccurred(), "git log failed: %s", string(out)) + return string(out) +} diff --git a/test/e2e/templates/manager/watchrule-resources.tmpl b/test/e2e/templates/manager/watchrule-resources.tmpl new file mode 100644 index 000000000..6715a2170 --- /dev/null +++ b/test/e2e/templates/manager/watchrule-resources.tmpl @@ -0,0 +1,11 @@ +apiVersion: configbutler.ai/v1alpha1 +kind: WatchRule +metadata: + name: {{.Name}} + namespace: {{.Namespace}} +spec: + targetRef: + kind: GitTarget + name: {{.DestinationName}} + rules: + - resources: [{{.Resources}}] From eb0e06afdef81e903fcd0c9ab49790284c15a1bd Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Mon, 1 Jun 2026 13:03:36 +0000 Subject: [PATCH 28/40] chore: get allure report to work locally --- docs/ci/e2e-allure-reporting.md | 14 +- docs/design/refactors-branch-review.md | 186 ++++++++++++++++++++++ test/e2e/Taskfile.yml | 17 +- test/e2e/tools/ginkgo-allure/main.go | 44 +++-- test/e2e/tools/ginkgo-allure/main_test.go | 76 +++++++++ 5 files changed, 321 insertions(+), 16 deletions(-) create mode 100644 docs/design/refactors-branch-review.md diff --git a/docs/ci/e2e-allure-reporting.md b/docs/ci/e2e-allure-reporting.md index d9b60584d..ccedd5005 100644 --- a/docs/ci/e2e-allure-reporting.md +++ b/docs/ci/e2e-allure-reporting.md @@ -55,9 +55,17 @@ devcontainer on the forwarded Allure port: task allure-e2e-open ``` -This serves the report on port `19081`. The same tasks can reuse older local -Ginkgo JSON reports under `.stamps`, so you can generate the report after a -failed run without rerunning the suite. +This serves the report on port `19081`. By default, the local Allure task reads +the newest Ginkgo JSON report from the current e2e context and namespace: +`.stamps/cluster///ginkgo-report-*.json`. + +To render a different set of reports, set `ALLURE_GINKGO_REPORTS_DIR` to another +directory or set `ALLURE_GINKGO_REPORTS` to an explicit whitespace-separated +list of report files. To render every report in the selected directory, set +`ALLURE_INCLUDE_ALL_GINKGO_REPORTS=true`. + +The same tasks can reuse an older local Ginkgo JSON report in that directory, so +you can generate the report after a failed run without rerunning the suite. The first local render pulls the `eclipse-temurin:17-jre` image and downloads the Allure CLI into `.stamps/allure-cli`. Later renders reuse both Docker's image diff --git a/docs/design/refactors-branch-review.md b/docs/design/refactors-branch-review.md new file mode 100644 index 000000000..bf1fdb6a2 --- /dev/null +++ b/docs/design/refactors-branch-review.md @@ -0,0 +1,186 @@ +# `refactors` branch — review & action checklist + +> Review of `main...refactors` (57 files, ~5k added / ~2k deleted) plus the +> uncommitted Allure/Taskfile working-tree changes. Generated 2026-06-01. +> Covers: (1) code-review findings, (2) docs triage, (3) refactor advice on the +> two e2e docs and the stability-enabled follow-ups. + +The big test-file split (`e2e_test.go` → four `*_e2e_test.go` files) verified +clean: all 23 Manager specs are re-created, assertion counts match, and +CRD/namespace cleanup was correctly redistributed. Findings concentrate in the +new Phase-3 drain metrics and the new e2e helpers. + +--- + +## 1. Code review + +### Correctness (ranked) + +- [ ] **1. Stale queue-depth gauge race — can hang the restart drain gate** + — [branch_worker.go:315-331](../../internal/git/branch_worker.go#L315-L331). + `recordQueueDepth()` runs from enqueue goroutines *and* the event loop; it + does `Load(inflightItems)` → … → `Record()` non-atomically. An enqueue + goroutine can load `depth=1`, get preempted while the loop drains the item and + `Record(0)` then idles, and then publish the stale `1`. OTel gauge is + last-writer-wins and the idle loop never republishes → gauge latches at `1`, + so the restart-snapshot gate `branch_worker_queue_depth … == 0` times out at + 90s though the worker is drained. **Fix:** publish the gauge *only* from the + loop goroutine (drop `recordQueueDepth()` from the enqueue paths; the loop + already republishes each iteration via `syncQueueDepthMetric`). + +- [ ] **2. Discovery-lag causes spurious full re-snapshots** + — [manager.go:1248](../../internal/watch/manager.go#L1248), + [:1264](../../internal/watch/manager.go#L1264), + [:1280](../../internal/watch/manager.go#L1280), + [:1189-1191](../../internal/watch/manager.go#L1189-L1191). + `currentRuleSetSnapshots` hashes the *resolved* GVR plan and swallows the + resolve error (`gvrs, _ := resolver.Resolve(...)`); an empty-plan target is + dropped and `snapshotTargetsNeedingDelivery` then deletes its + `lastDeliveredRuleSetHash`. A transient API-discovery gap makes a target + vanish and, on recovery, get re-snapshotted (full repo-state diff). New + regression vs the old rule-text hash; it's the same mechanism that forces + `crd_lifecycle` Serial. **Fix:** log the resolve miss instead of discarding, + and don't evict last-delivered state on an *empty* plan (distinguish "no + rules" from "rules that didn't resolve yet"). + +- [ ] **3. Reconcile counter skipped on transient error / fired on no-op** + — [manager.go:1384-1392](../../internal/watch/manager.go#L1384-L1392). + `recordTargetReconcileCompleted` fires only after **both** `RequestRepoState` + and `RequestClusterState` succeed. A transient error on the second `continue`s + (counter never increments though partial state was enqueued) → restart + `{pod=…} > 0` gate can time out at 90s on a one-off error. It also fires on a + no-op snapshot, so `increase(target_reconcile_completed_total)` over-counts + for non-test consumers. + +- [ ] **4. Shutdown leaves the depth gauge non-zero** + — [branch_worker.go:585](../../internal/git/branch_worker.go#L585), + [:704](../../internal/git/branch_worker.go#L704). + On `ctx.Done()` the loop finalizes + pushes and returns **without draining + items still buffered in `eventQueue`**; each was counted into `inflightItems` + at enqueue, so the final `recordQueueDepth` publishes a non-zero depth for the + exiting pod that never clears. Shielded today by the per-pod gate; latent leak + for any cross-pod use. + +### Cleanup / efficiency + +- [ ] **5. ~13 inlined `git pull` blocks duplicate `pullLatestRepoState`** + — [watchrule_configmap_secret_e2e_test.go:174](../../test/e2e/watchrule_configmap_secret_e2e_test.go#L174) + (+~6 more) and ~7 in `crd_lifecycle_e2e_test.go`. The canonical helper + (fetch + checkout -B + reset --hard) is already used at + [:707](../../test/e2e/watchrule_configmap_secret_e2e_test.go#L707) and is more + robust than a bare `git pull` (which can flake on a diverged branch). +- [ ] **6. `gitPull` / `lastCommitMessageForPath` reinvent existing helpers** + — [gittarget_isolation_e2e_test.go:187](../../test/e2e/gittarget_isolation_e2e_test.go#L187). + Third pull implementation (no fetch/reset) + manual `git log` where + `gitRun(dir, args...)` already exists in helpers.go. +- [ ] **7. Hand-rolled `minInt`** — [e2e_test.go:264](../../test/e2e/e2e_test.go#L264); + used once, Go 1.26 builtin `min` covers it. +- [ ] **8. Redundant "healthy GitProvider" spec** + — [watchrule_configmap_secret_e2e_test.go:80](../../test/e2e/watchrule_configmap_secret_e2e_test.go#L80) + re-creates/re-verifies `gitprovider-normal` already built in `BeforeAll` + ([:60-70](../../test/e2e/watchrule_configmap_secret_e2e_test.go#L60-L70)). +- [ ] **9. Duplicated endpoint→pod filtering** + — [controller_basics_e2e_test.go:99](../../test/e2e/controller_basics_e2e_test.go#L99) + vs [:135](../../test/e2e/controller_basics_e2e_test.go#L135); extract + `expectServiceRoutesToPod`. +- [ ] **10. Wasted work** — CI "Summarize E2E timing reports" `go run`s + spec-timings per report in a loop ([ci.yml:388](../../.github/workflows/ci.yml#L388)), + recompiling N+1× (`go build -o` once instead); `currentRuleSetSnapshots` + resolves GVRs twice per reconcile ([manager.go:1224](../../internal/watch/manager.go#L1224)); + `syncQueueDepthMetric` records every loop iteration rather than on change. + +> Uncommitted working tree (Allure env-var overrides + `specSkippedByFilter` + +> per-process thread label) looks good. Caveat: `specSkippedByFilter` +> ([ginkgo-allure/main.go:263](../../test/e2e/tools/ginkgo-allure/main.go#L263)) +> infers "filtered-out" from `LabelFilter!="" && Skipped && EndTime.IsZero() && +> RunTime==0`; a real `Skip()` recording zero timing would also be dropped. Low +> risk — a heuristic, not a guarantee. + +--- + +## 2. Docs triage + +> `docs/finished/` already exists (empty) but isn't in the README — adopt it as +> the "done" folder and document it in [docs/README.md](../README.md). + +### Move to `docs/finished/` — shipped/resolved + +- [ ] [design/gittarget-isolation-on-rule-change.md](gittarget-isolation-on-rule-change.md) — ✅ shipped (this branch) +- [ ] [design/commit-window-refactor.md](commit-window-refactor.md) — implemented +- [ ] [design/e2e-cleanup-scoping.md](e2e-cleanup-scoping.md) — implemented +- [ ] [design/partial-object-audit-event-handling.md](partial-object-audit-event-handling.md) — implemented +- [ ] [design/path-scoped-bootstrap-template-design.md](path-scoped-bootstrap-template-design.md) — implemented +- [ ] [design/remove-gittarget-admission-webhook.md](remove-gittarget-admission-webhook.md) — webhook file gone → done +- [ ] [design/watch-gvr-resolution-cleanup.md](watch-gvr-resolution-cleanup.md) — `APIResourceCatalog` cleanup landed +- [ ] [design/watchrule-gvr-resolution-plan.md](watchrule-gvr-resolution-plan.md) — implemented +- [ ] [design/aggregated-api-requestheader-ca-miswire-report.md](aggregated-api-requestheader-ca-miswire-report.md) — resolved report +- [ ] [design/audit-consumer-username-attribution-gap.md](audit-consumer-username-attribution-gap.md) — resolved +- [ ] [design/e2e-runtime-state-vs-stamps.md](e2e-runtime-state-vs-stamps.md) — resolved failure mode +- [ ] [design/shallow-audit-event-misclassification.md](shallow-audit-event-misclassification.md) — root cause confirmed +- [ ] [design/e2e-test-review.md](e2e-test-review.md) — point-in-time review +- [ ] [git-author.md](../git-author.md) — OIDC author plan, implemented +- [ ] [future/design-commit-request-api.md](../future/design-commit-request-api.md) — implemented +- [ ] [future/design-rule-change-snapshot-trigger.md](../future/design-rule-change-snapshot-trigger.md) — implemented +- [ ] [future/design-audit-ingestion-hardening.md](../future/design-audit-ingestion-hardening.md) — implemented +- [ ] [future/design-commit-context-api.md](../future/design-commit-context-api.md) — superseded +- [ ] [future/idea-audit-enrichment-side-channel.md](../future/idea-audit-enrichment-side-channel.md) — superseded + +### Consider removing / relocating + +- [ ] [cozystack-bugreport.md](../cozystack-bugreport.md) — upstream bug report against + CozyStack, not our docs. If filed upstream, replace with a link. Referenced by + `initial-seed-via-watch-list.md`, so update that link; `serious-bug/` is the + safe relocation. +- [ ] [ci/findings.md](../ci/findings.md) — stale (2026-02-13), overlaps + [e2e-ci-stability-report.md](e2e-ci-stability-report.md). Fold true bits in, drop it. +- [ ] Empty `docs/finished/`, `docs/serious-bug/`, `docs/contributing/` — populate + (per above) or remove the empty scaffolding. + +### Keep on the active design surface + +Still-relevant plans/proposals: [e2e-speedup-plan.md](e2e-speedup-plan.md), +[e2e-serial-registry.md](e2e-serial-registry.md), +[e2e-ci-stability-report.md](e2e-ci-stability-report.md), +[e2e-speedup-skipped.md](e2e-speedup-skipped.md), +[audit-metrics-overhaul-plan.md](audit-metrics-overhaul-plan.md) (proposed), +[initial-seed-via-watch-list.md](initial-seed-via-watch-list.md) (proposed) + +[upgrade-finding.md](../upgrade-finding.md), the sensitive-resource plan + +follow-up, and [e2e-watchrule-cross-spec-interference.md](e2e-watchrule-cross-spec-interference.md) +(status: **open**). All `future/idea-*.md` stay. Reference guides +(`kubernetes-*`, `crd-relationships`, `status-conditions-guide`, audit-webhook +designs, lifecycle/architecture) stay. All top-level user guides + `ci/*` +explainers + `demo/*` + `audit-setup/*` stay. + +--- + +## 3. The two e2e docs + stability-enabled refactors + +**[e2e-allure-reporting.md](../ci/e2e-allure-reporting.md)** — accurate and +current. Only gap: + +- [ ] Document the new per-process **`thread` Allure label** the converter now + emits (the point of the parallel-insight work). + +**[e2e-serial-registry.md](e2e-serial-registry.md)** — the Serial entries are +**correctness-driven (shared state), not flakiness-driven**, so improved +stability doesn't justify de-serializing directly. It *does* unlock the deeper +refactors the registry is implicitly waiting for: + +- [ ] **Isolate the audit pipeline → de-serialize 4 of 8 entries.** `Audit Redis + Queue/Consumer`, `Commit Window Batching`, `Commit Request` are Serial only + because the webhook→Redis-stream→consumer pipeline is a global singleton. + Per-test stream keys / consumer groups (same "isolate by name" move as the + icecream CRD groups) lets all four run parallel. **Highest-leverage remaining + speedup.** +- [ ] **Fix the discovery-lag re-snapshot (code finding #2) → de-serialize + `crd_lifecycle`.** That spec is Serial because CRD install/delete forces + unrelated GitTargets to resnapshot — the `manager.go` empty-plan / swallowed + resolve-miss behavior. Fix at the source; name-isolated CRD groups then make + the spec parallel-safe. +- `bi_directional` (exact commit-count) and `restart_snapshot` / `image_refresh` + (singleton controller) are genuinely serial — leave them. + +- [ ] **Confidence-window pass (the speedup plan's open thread).** Now that the + race fixes landed (`03b95ab`, `0787b1b`), decide whether CI can move back off + `K3D_AGENT_COUNT=0` / `procs=1` toward parallelism, using the uploaded timing + summaries + Allure reports. diff --git a/test/e2e/Taskfile.yml b/test/e2e/Taskfile.yml index dc6636e23..8b0bee84e 100644 --- a/test/e2e/Taskfile.yml +++ b/test/e2e/Taskfile.yml @@ -220,11 +220,22 @@ tasks: - | reports_file="$(mktemp)" trap 'rm -f "${reports_file}"' EXIT - if [ -d .stamps/cluster ]; then - find .stamps/cluster -path "*/ginkgo-report-*.json" -type f | sort > "${reports_file}" + if [ -n "${ALLURE_GINKGO_REPORTS:-}" ]; then + printf '%s\n' ${ALLURE_GINKGO_REPORTS} | sort > "${reports_file}" + else + reports_root="${ALLURE_GINKGO_REPORTS_DIR:-{{.CS}}/{{.NAMESPACE}}}" + if [ -d "${reports_root}" ]; then + if [ "${ALLURE_INCLUDE_ALL_GINKGO_REPORTS:-}" = "true" ]; then + find "${reports_root}" -maxdepth 1 -name "ginkgo-report-*.json" -type f | sort > "${reports_file}" + else + find "${reports_root}" -maxdepth 1 -name "ginkgo-report-*.json" -type f -printf '%T@ %p\n' \ + | sort -rn \ + | sed -n '1s/^[^ ]* //p' > "${reports_file}" + fi + fi fi if [ ! -s "${reports_file}" ]; then - echo "No Ginkgo JSON reports found under .stamps/cluster. Run an e2e task first." >&2 + echo "No Ginkgo JSON reports found for the current e2e context/namespace. Run an e2e task first." >&2 exit 1 fi diff --git a/test/e2e/tools/ginkgo-allure/main.go b/test/e2e/tools/ginkgo-allure/main.go index 76a8224b4..f8815136b 100644 --- a/test/e2e/tools/ginkgo-allure/main.go +++ b/test/e2e/tools/ginkgo-allure/main.go @@ -112,17 +112,30 @@ func run(outputDir string, paths []string) (int, error) { if err != nil { return count, err } - for reportIndex, report := range reports { - for specIndex, spec := range report.SpecReports { - if spec.LeafNodeType != types.NodeTypeIt { - continue - } - result := convertSpec(path, reportIndex, specIndex, report, spec) - if err := writeResult(outputDir, result, spec.CombinedOutput()); err != nil { - return count, err - } - count++ + reportCount, err := convertReports(outputDir, path, reports) + if err != nil { + return count, err + } + count += reportCount + } + return count, nil +} + +func convertReports(outputDir, path string, reports []types.Report) (int, error) { + count := 0 + for reportIndex, report := range reports { + if report.SuiteConfig.DryRun { + continue + } + for specIndex, spec := range report.SpecReports { + if spec.LeafNodeType != types.NodeTypeIt || specSkippedByFilter(report, spec) { + continue + } + result := convertSpec(path, reportIndex, specIndex, report, spec) + if err := writeResult(outputDir, result, spec.CombinedOutput()); err != nil { + return count, err } + count++ } } return count, nil @@ -208,6 +221,9 @@ func labels(report types.Report, spec types.SpecReport) []allureLabel { {Name: "language", Value: "go"}, {Name: "framework", Value: "ginkgo"}, } + if spec.ParallelProcess > 0 { + out = append(out, allureLabel{Name: "thread", Value: fmt.Sprintf("ginkgo-process-%d", spec.ParallelProcess)}) + } if report.SuiteDescription != "" { out = append(out, allureLabel{Name: "suite", Value: report.SuiteDescription}) } @@ -244,6 +260,14 @@ func allureStatus(state types.SpecState) string { } } +func specSkippedByFilter(report types.Report, spec types.SpecReport) bool { + return report.SuiteConfig.LabelFilter != "" && + spec.State == types.SpecStateSkipped && + spec.EndTime.IsZero() && + spec.RunTime == 0 && + spec.Failure.IsZero() +} + func statusDetails(spec types.SpecReport) *allureStatusDetails { if spec.Failure.IsZero() { return nil diff --git a/test/e2e/tools/ginkgo-allure/main_test.go b/test/e2e/tools/ginkgo-allure/main_test.go index 9e77f90ab..c7cf8c34b 100644 --- a/test/e2e/tools/ginkgo-allure/main_test.go +++ b/test/e2e/tools/ginkgo-allure/main_test.go @@ -130,6 +130,7 @@ func assertPassingResult(t *testing.T, outputDir string, result allureResult) { } assertHasLabel(t, result, "framework", "ginkgo") assertHasLabel(t, result, "language", "go") + assertHasLabel(t, result, "thread", "ginkgo-process-2") assertHasLabel(t, result, "suite", "E2E Suite") assertHasLabel(t, result, "package", "e2e") assertHasLabel(t, result, "parentSuite", "Manager") @@ -201,6 +202,81 @@ func TestRun_BadJSON(t *testing.T) { } } +func TestRun_IgnoresDryRunReports(t *testing.T) { + reports := fixtureReport() + reports[0].SuiteConfig.DryRun = true + reportPath := writeFixtureReport(t, reports) + outputDir := filepath.Join(t.TempDir(), "allure-results") + + count, err := run(outputDir, []string{reportPath}) + if err != nil { + t.Fatalf("run: %v", err) + } + if count != 0 { + t.Fatalf("count: got %d, want 0", count) + } + if results := readAllureResults(t, outputDir); len(results) != 0 { + t.Fatalf("results: got %d, want 0", len(results)) + } +} + +func TestRun_IgnoresSpecsSkippedByLabelFilter(t *testing.T) { + reports := fixtureReport() + reports[0].SuiteConfig.LabelFilter = "smoke" + reports[0].SpecReports = []types.SpecReport{ + { + ContainerHierarchyTexts: []string{"Manager"}, + LeafNodeType: types.NodeTypeIt, + LeafNodeText: "runs", + State: types.SpecStatePassed, + StartTime: reports[0].StartTime, + EndTime: reports[0].StartTime.Add(time.Second), + RunTime: time.Second, + ParallelProcess: 1, + }, + { + ContainerHierarchyTexts: []string{"Manager"}, + LeafNodeType: types.NodeTypeIt, + LeafNodeText: "filtered out", + State: types.SpecStateSkipped, + StartTime: reports[0].StartTime.Add(time.Second), + ParallelProcess: 1, + }, + { + ContainerHierarchyTexts: []string{"Manager"}, + LeafNodeType: types.NodeTypeIt, + LeafNodeText: "skips intentionally", + State: types.SpecStateSkipped, + StartTime: reports[0].StartTime.Add(2 * time.Second), + Failure: types.Failure{ + Message: "operator skipped", + }, + ParallelProcess: 1, + }, + } + reportPath := writeFixtureReport(t, reports) + outputDir := filepath.Join(t.TempDir(), "allure-results") + + count, err := run(outputDir, []string{reportPath}) + if err != nil { + t.Fatalf("run: %v", err) + } + if count != 2 { + t.Fatalf("count: got %d, want 2", count) + } + results := readAllureResults(t, outputDir) + if len(results) != 2 { + t.Fatalf("results: got %d, want 2", len(results)) + } + findResult(t, results, "runs") + findResult(t, results, "skips intentionally") + for _, result := range results { + if result.Name == "filtered out" { + t.Fatal("filtered-out spec was converted") + } + } +} + func TestRun_OutputDirCreateError(t *testing.T) { reportPath := writeFixtureReport(t, fixtureReport()) outputDir := filepath.Join(t.TempDir(), "not-a-dir") From 145e11a3b2c86c50c45a8acd5ba3b29fe4c115bb Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Tue, 2 Jun 2026 08:37:09 +0000 Subject: [PATCH 29/40] chore: cleaing / updating docs and minor fixes --- docs/architecture.md | 2 +- .../rule-set-snapshot-discovery-lag-fix.md | 235 +++++++++ docs/design/watch-and-catalog-architecture.md | 459 ++++++++++++++++++ ...hrule-wildcard-and-resolution-semantics.md | 131 +++++ ...ted-api-requestheader-ca-miswire-report.md | 0 ...audit-consumer-username-attribution-gap.md | 0 .../commit-window-refactor.md | 0 .../design-audit-ingestion-hardening.md | 0 .../design-commit-context-api.md | 8 +- .../design-commit-request-api.md | 10 +- .../design-rule-change-snapshot-trigger.md | 2 +- .../e2e-cleanup-scoping.md | 4 +- .../e2e-runtime-state-vs-stamps.md | 0 docs/{design => finished}/e2e-test-review.md | 0 docs/{ => finished}/git-author.md | 0 .../gittarget-isolation-on-rule-change.md | 0 .../idea-audit-enrichment-side-channel.md | 0 .../partial-object-audit-event-handling.md | 0 .../path-scoped-bootstrap-template-design.md | 0 .../remove-gittarget-admission-webhook.md | 0 .../shallow-audit-event-misclassification.md | 0 .../watch-gvr-resolution-cleanup.md | 0 .../watchrule-gvr-resolution-plan.md | 2 +- docs/future/design-commit-request-phase-2.md | 2 +- .../design-snapshot-engine-evolution.md | 6 +- docs/future/idea-end-user-commit-messages.md | 2 +- docs/future/idea-unify-pending-write-kinds.md | 2 +- docs/interpreting-metrics.md | 2 +- 28 files changed, 846 insertions(+), 21 deletions(-) create mode 100644 docs/design/rule-set-snapshot-discovery-lag-fix.md create mode 100644 docs/design/watch-and-catalog-architecture.md create mode 100644 docs/design/watchrule-wildcard-and-resolution-semantics.md rename docs/{design => finished}/aggregated-api-requestheader-ca-miswire-report.md (100%) rename docs/{design => finished}/audit-consumer-username-attribution-gap.md (100%) rename docs/{design => finished}/commit-window-refactor.md (100%) rename docs/{future => finished}/design-audit-ingestion-hardening.md (100%) rename docs/{future => finished}/design-commit-context-api.md (99%) rename docs/{future => finished}/design-commit-request-api.md (96%) rename docs/{future => finished}/design-rule-change-snapshot-trigger.md (98%) rename docs/{design => finished}/e2e-cleanup-scoping.md (96%) rename docs/{design => finished}/e2e-runtime-state-vs-stamps.md (100%) rename docs/{design => finished}/e2e-test-review.md (100%) rename docs/{ => finished}/git-author.md (100%) rename docs/{design => finished}/gittarget-isolation-on-rule-change.md (100%) rename docs/{future => finished}/idea-audit-enrichment-side-channel.md (100%) rename docs/{design => finished}/partial-object-audit-event-handling.md (100%) rename docs/{design => finished}/path-scoped-bootstrap-template-design.md (100%) rename docs/{design => finished}/remove-gittarget-admission-webhook.md (100%) rename docs/{design => finished}/shallow-audit-event-misclassification.md (100%) rename docs/{design => finished}/watch-gvr-resolution-cleanup.md (100%) rename docs/{design => finished}/watchrule-gvr-resolution-plan.md (99%) diff --git a/docs/architecture.md b/docs/architecture.md index cbb5e34d9..5a6c76f34 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -884,7 +884,7 @@ For deeper context on specific decisions: - [GitTarget lifecycle and repo architecture](design/gittarget-lifecycle-and-repo-architecture.md) — GitTarget state machine details - [Watch and audit rule matching improvement](design/watch-audit-rule-matching-improvement.md) — known issues with namespace matching and startup bootstrap - [Kubernetes API resource catalog](design/kubernetes-api-resource-catalog.md) — the served-resource discovery model behind `APIResourceCatalog` -- [WatchRule GVR resolution](design/watchrule-gvr-resolution-plan.md) — how bare rule resources resolve to concrete GVRs, and the snapshot trust model +- [WatchRule GVR resolution](finished/watchrule-gvr-resolution-plan.md) — how bare rule resources resolve to concrete GVRs, and the snapshot trust model - [Multi-cluster audit ingestion implications](design/multi-cluster-audit-ingestion-implications.md) — what multi-cluster means beyond ingestion - [SOPS/age key management](design/sops-repo-bootstrap-and-key-management-architecture.md) — encryption architecture - [Audit webhook TLS design](design/audit-webhook-tls-design.md) — webhook transport security diff --git a/docs/design/rule-set-snapshot-discovery-lag-fix.md b/docs/design/rule-set-snapshot-discovery-lag-fix.md new file mode 100644 index 000000000..0c1b1d259 --- /dev/null +++ b/docs/design/rule-set-snapshot-discovery-lag-fix.md @@ -0,0 +1,235 @@ +# Rule-Set Snapshot Discovery-Lag Fix + +## Problem + +Finding 2 in [refactors-branch-review.md](refactors-branch-review.md) is about +the new per-GitTarget effective-watch-plan hash: + +- `currentRuleSetSnapshots()` resolves rules through the API resource catalog. +- It ignores `ResolveMiss` values from `resolver.Resolve(...)` + (`gvrs, _ := resolver.Resolve(...)`). +- If a target's rules temporarily resolve to zero GVRs, the target is omitted + entirely ([manager.go:1280-1282](../../internal/watch/manager.go#L1280-L1282)). +- `snapshotTargetsNeedingDelivery()` treats an omitted target as "this target no + longer has rules" and deletes `lastDeliveredRuleSetHash` + ([manager.go:1189-1194](../../internal/watch/manager.go#L1189-L1194)). +- When discovery recovers, the same target appears again with the same real + watch plan but no remembered baseline, so it gets a full rule-change snapshot + even though the user's rules did not change. + +That makes a transient discovery gap look like a real watch-plan change. It can +hide event commits behind later `reconcile: sync ...` commits and is the reason +`crd_lifecycle` still needs `Serial` protection. + +## Two facts that shape the fix + +Before designing anything, two existing properties narrow what the fix actually +has to do: + +1. **Partial snapshots are already refused downstream.** + `RequestClusterState` aborts whenever `blockingSnapshotMisses(...)` is + non-empty — "refusing to snapshot a partial cluster view" + ([manager.go:596-601](../../internal/watch/manager.go#L596-L601)). So a + half-resolved plan can never *emit* a snapshot. The selection layer does not + need to be perfect about blocked plans for **correctness** — only to avoid + noisy select-then-abort retry loops. + +2. **"No rules" vs "rules that didn't resolve" needs no miss classification.** + It is already encoded in the RuleStore. `currentRuleSetSnapshots()` only + builds a plan for a target that appears in `SnapshotWatchRules()` / + `SnapshotClusterWatchRules()`. A target with zero rules never gets a plan and + falls out of `currentKeys` naturally. The bug exists solely because the + `if len(p.entries) == 0 { continue }` guard drops a target that *still has + rules* but momentarily resolved to nothing. + +Together these mean the headline bug has a small, safe fix, and the richer +classification is a set of optional refinements layered on top. + +## Expected states — and when they happen + +A target's resolved plan can be in one of four states. The two that produce zero +GVRs (states 3 and 4) look identical to the old code, which is the root cause. + +| State | Trigger | Stable? | Action | +|---|---|---|---| +| 1. No rules | The last WatchRule/ClusterWatchRule for the GitTarget was deleted, or its `gitTargetRef` was repointed. | — | Prune remembered state. | +| 2. Resolved plan | One or more rules resolve to served, allowed GVRs. The normal steady state. | yes | Compare hash to last settled; emit a snapshot only if it changed. | +| 3. Stable empty plan | Rules exist but resolve to zero watchable GVRs for a reason that will not change on its own: `NotServed`, `Ambiguous`, `Disallowed`. | yes | No snapshot (nothing to watch). Optionally record a no-watch settled hash (Refinement B). | +| 4. Blocked / uncertain | Rules exist and *would* resolve, but discovery can't answer authoritatively right now: `CatalogUnavailable`, `DiscoveryDegraded`, `WildcardGroup`, `WildcardResource`. | no | Do not snapshot, do not mutate remembered state. Retry next cycle. | + +Concrete triggers: + +- **State 1 — No rules.** `kubectl delete watchrule`, or editing a rule to point + at a different GitTarget. The target no longer exists in the RuleStore + snapshot, so no plan is built for it. +- **State 2 — Resolved plan.** Steady state, every reconcile. A real change is + e.g. a rule that newly adds `secrets` alongside `configmaps`, or a wildcard + that begins matching a freshly installed CRD. +- **State 3 — Stable empty.** The user wrote a rule for a CRD that was never + installed, typo'd a resource name, omitted `apiGroups` in a way that matches + multiple groups, or named a resource the operator is configured to refuse. + Re-running discovery returns the same answer, so this is settled. +- **State 4 — Blocked.** **This is the bug's trigger.** A CRD is mid + install/delete (the `crd_lifecycle` spec), the controller just started and + discovery is still warming up, or the aggregation layer briefly 503s. Unlike + state 3, the answer **changes on retry**. + +The crucial pair is **3 vs 4**: both yield zero GVRs, but 3 is a settled fact and +4 is a temporary "ask again later." The original code collapsed both into "target +removed," which is what turned a routine CRD install (state 4) into a spurious +full re-snapshot of every unrelated target. + +## Fix (ship this) + +The headline bug is fixed by a near-trivial change. No new types, no miss +classification, no rename. + +1. In `currentRuleSetSnapshots()`, **stop dropping empty-entry plans.** Keep + every target that has at least one rule, even if it currently resolves to zero + GVRs. (Delete the `if len(p.entries) == 0 { continue }` guard.) +2. In `snapshotTargetsNeedingDelivery()`, **if a plan has no entries, `continue` + — skip selection and leave `lastDeliveredRuleSetHash` untouched.** There is + nothing to snapshot, and the remembered baseline must survive the gap. +3. The existing prune loop now fires only for targets with genuinely zero rules. + That is already correct — no change needed. + +Walk the transient-gap case through it: `valid -> (gap, empty) -> valid`. During +the gap the target is kept but skipped, `lastDeliveredRuleSetHash` stays at the +valid hash; on recovery the hash matches, so no snapshot is emitted. Bug gone. + +This alone fixes finding #2 and unblocks de-serializing `crd_lifecycle`. + +### What the Fix gives up + +Exactly **one** behavioral difference versus a fully classified design: the +`valid -> stable-empty -> valid` *edit* sequence (a user edits a rule to +something non-watchable, then edits it back). The Fix keeps the baseline at the +valid hash the whole time, so on return it sees "unchanged" and skips the +recovery snapshot — even though no live events were watched during the empty +window. Refinement B below addresses exactly this case if you want it. + +## Refinement A (optional): skip blocked plans instead of aborting them + +The Fix is safe for the **mixed** case (one resolved GVR plus one blocking miss → +non-empty entries) because the downstream abort at +[manager.go:596](../../internal/watch/manager.go#L596) refuses the partial +snapshot. But it does so by *selecting* the target every cycle and aborting it, +which logs an error each reconcile until discovery recovers. + +To avoid that noise, capture misses in the plan and add a `blocked` flag: + +```go +// fields added to the existing targetWatchPlan, not a parallel type +blocked bool +blockingMisses []ResolveMiss +``` + +Set `blocked` when any rule for the target produces a miss in +`blockingSnapshotMisses(...)` (`CatalogUnavailable`, `DiscoveryDegraded`, +`WildcardGroup`, `WildcardResource`). Then in `snapshotTargetsNeedingDelivery()`: + +```go +if plan.blocked { + // Rules exist, but the current resolved plan is not authoritative. + // Keep lastDeliveredRuleSetHash and pendingRuleSetHash as-is; + // discovery/catalog recovery will retry. + continue +} +``` + +This is a quality-of-noise optimization, not a correctness fix — the Fix is +already safe without it. + +## Refinement B (optional): settle a stable-empty hash + +This is the *only* mechanism that fixes the `valid -> stable-empty -> valid` edit +case, and the only reason to compute a hash for an empty plan. + +Include the **stable** misses (`NotServed`, `Ambiguous`, `Disallowed`) in the +hash so a target can settle into a known "no-watch" plan and later transition +back to a resolved plan: + +``` +valid hash -> stable empty hash -> valid hash +``` + +The settled empty hash differs from the valid hash, so the final transition is +detected and the target is re-snapshotted. In `snapshotTargetsNeedingDelivery()`: + +```go +if !plan.hasEntries { // hasEntries == len(plan.entries) > 0 + // Stable empty/no-watch plan. No cluster snapshot to emit, but the + // rule-set state did advance. Mark it settled so a later empty -> resolved + // transition emits a snapshot. + lastDeliveredRuleSetHash[key] = plan.hash + delete(pendingRuleSetHash, key) + continue +} +``` + +Do **not** include blocking misses in the hash — a blocked plan is not an +authoritative plan, and a blocked target must hit the Refinement A path +(`continue` without mutating state) before this branch. + +When is this worth it? Only when a *served* resource churns during a window in +which the user deliberately edited the rule to be non-watchable and back. Rare, +and the worst case without it is a missed recovery snapshot that self-heals on +the next real rule change. For `NotServed` resources there are by definition no +objects to miss, so it is often moot. + +## Implementation notes + +- Extend the existing `targetWatchPlan` with the `blocked` / `blockingMisses` + fields rather than introducing a parallel `ruleSetSnapshotPlan` type. Express + `hasEntries` as `len(entries) > 0`, not a stored field. +- Reuse `blockingSnapshotMisses(...)` for the classification. If the name reads + oddly in a planning context, extract a thin alias such as + `blockingRuleSetPlanMisses(...)`. +- If Refinement B lands, `lastDeliveredRuleSetHash` would be more accurately + named `lastSettledRuleSetHash`, because stable empty plans are settled without + a delivered snapshot. Rename only if the patch stays readable; otherwise add a + comment explaining the broadened meaning. +- Log blocked plans at `V(1)` with `FormatResolveMisses(...)`. Avoid + default-verbosity logs on every periodic reconcile; catalog transition logs + already cover the noisy discovery edges. +- Do not mark a blocked plan as pending. Pending is for a snapshot that should be + emitted once a reconciler exists, not for a plan that is not yet authoritative. + +## Test plan + +Tests map onto the tiers, so you can land the Fix and its three tests and drop +`Serial` without ever building Refinement A or B. Add them near +[rule_change_snapshot_test.go](../../internal/watch/rule_change_snapshot_test.go): + +**Fix:** + +- `TestSnapshotTargets_DiscoveryUnavailableKeepsDeliveredHash`: seed a delivered + configmaps plan, swap the manager catalog to unavailable, call + `snapshotTargetsNeedingDelivery()`, and assert the target is not selected and + `lastDeliveredRuleSetHash` is not pruned. +- `TestSnapshotTargets_DiscoveryRecoveryDoesNotResnapshotUnchangedPlan`: after + the unavailable-catalog pass above, restore the common catalog and assert the + unchanged target is still not selected. +- `TestSnapshotTargets_RuleRemovalPrunesDeliveredHash`: remove the rule from the + RuleStore and assert the remembered hash is pruned. This guards against + confusing "blocked" (state 4) with "deleted" (state 1). + +**Refinement A:** + +- `TestSnapshotTargets_MixedResolvedAndBlockingMissDoesNotSnapshotPartialPlan`: + give one target a resolved GVR plus a discovery-degraded lookup. Assert the + target is not selected and no hash is advanced. +- `TestSnapshotTargets_BlockedTargetDoesNotAffectOtherTargets`: target A is + blocked by discovery, target B changes from configmaps to configmaps+secrets. + Assert B is selected and A's remembered state is preserved. + +**Refinement B:** + +- `TestSnapshotTargets_StableEmptyPlanSettlesAndRecoverySnapshots`: seed a + delivered configmaps plan, change the rule to a definitely not-served resource, + assert no snapshot is selected but the settled hash changes, then change back + to configmaps and assert the target is selected. + +The e2e follow-up is to remove `Serial` from `Manager CRD Lifecycle` only after +these unit tests pass and at least one parallel CI confidence window shows that +CRD install/delete no longer drags unrelated GitTargets into snapshots. diff --git a/docs/design/watch-and-catalog-architecture.md b/docs/design/watch-and-catalog-architecture.md new file mode 100644 index 000000000..22451ebb4 --- /dev/null +++ b/docs/design/watch-and-catalog-architecture.md @@ -0,0 +1,459 @@ +# Watch & Catalog Architecture — Requirements, Current State, Target Design + +Status: **vision / proposed** — supersedes the framing in +[watchrule-wildcard-support-plan.md](watchrule-wildcard-support-plan.md), +[watchrule-wildcard-and-resolution-semantics.md](watchrule-wildcard-and-resolution-semantics.md), +and [rule-set-snapshot-discovery-lag-fix.md](rule-set-snapshot-discovery-lag-fix.md); +those become implementation details of the model described here. + +This document is in three parts: + +1. **Requirements** — how it *should* work, including requirements not yet + written down anywhere. +2. **Current architecture** — what exists today and where it falls short. +3. **Target architecture** — one clean model and how we get there. A large + refactor is acceptable; correctness and simplicity win over diff size. + +--- + +# Part 1 — Requirements: how it should work + +## 1.0 One sentence + +The application keeps a **live, authoritative model of the cluster's API +surface**, and every `GitTarget` mirrors a **deterministic projection** of that +surface (selected by its rules) into git, reacting to surface changes +**incrementally** and **never writing an uncertain or partial view**. + +## 1.1 The catalog is the single source of truth + +There must be exactly one component that knows "what does this cluster currently +serve, and can I trust that knowledge right now." Everything else +(resolution, informers, snapshots, status) asks the catalog; nothing else talks +to discovery or guesses. + +The catalog must: + +- **Maintain last-known-good.** A `GroupVersion` that blips out during an + apiserver rollout or CRD upgrade must not vanish from the model the instant + discovery hiccups. The catalog keeps the prior entries and marks the GV + *degraded*, not *gone*. +- **Distinguish the three reasons a resource can be "absent":** + 1. *Never served* / not yet served (stable). + 2. *Degraded* — discovery temporarily can't confirm it (transient). + 3. *Withdrawn* — confirmed removed from the surface (e.g. CRD uninstalled). + These three drive completely different downstream actions, so the catalog must + label them, not collapse them to "empty". +- **Debounce withdrawal.** A GV is only promoted from *degraded* to *withdrawn* + after a confirmation window / repeated clean discovery that still omits it. + This is what "abstracts away shortly-unavailable CRDs" means concretely. +- **Expose confidence.** A consumer can always ask "is the surface authoritative + enough to make a destructive (delete) decision right now?" +- **Emit deltas.** Consumers react to *changes* (GVR added / withdrawn / degraded + / recovered), not by recomputing the world each tick. + +## 1.2 Wildcards behave like Kubernetes policy APIs (non-negotiable) + +Skipping wildcards is **not** an option. `WatchRule`/`ClusterWatchRule` selectors +must follow the admission-webhook / RBAC / audit-policy idiom that users already +know: + +- `apiGroups: ["*"]` matches all groups; `resources: ["*"]` matches all + resources; `apiVersions: ["*"]` matches all versions. +- A `*` rule is a *live* selection: when a new CRD appears that matches it, the + target starts watching it automatically; when one is withdrawn, it stops. +- The selector is a **projection of the catalog**, evaluated continuously — not a + one-time literal expansion frozen at apply time. + +Decisions to lock (see open questions): the meaning of *empty* vs `*` (proposal: +empty `apiGroups` = "resolve to the single served group, else surface an error"; +`["*"]` = "all served groups" — a strict mode and a wildcard mode, both useful), +and that *subresources* (`pods/log`, `pods/*`) are **out of scope** for the +list/watch+mirror model and must not be advertised. + +## 1.3 Incremental, relaxed reconcile + +An active `GitTarget` must respond to a newly added or removed CRD by changing +**only the part that changed**: + +- New GVR matches a target's plan → start one informer for that GVR, seed **only + that GVR's objects** into the mirror, done. No re-snapshot of the rest of the + target. +- GVR withdrawn → stop that one informer; apply the configured withdrawal policy + to only that GVR's files. + +The unit of reconciliation is the **(GitTarget, GVR) cell**, not the whole +GitTarget. Rationale (the user's, and worth stating as a requirement): because +live audit/event coverage is high-quality, a full-target re-list is rarely needed +— the only thing a new GVR genuinely needs is its *initial seed*; steady state is +maintained by events. A full re-snapshot should be a **fallback**, not the +normal path. + +## 1.4 The authoritative-write invariant (the core safety property) + +The mirror must **never** be written from a partial or uncertain view, because a +resource that is merely *unconfirmed* is indistinguishable from a *deleted* one, +and writing the difference would mirror phantom deletions into git. + +Concretely: + +- A delete is only ever written for a cell whose source-of-truth was + **authoritatively observed** (a successful list of that GVR, or a real DELETE + event for that object) **or** whose removal was **intent-driven** (the rules no + longer select it — see 1.6). +- If a GVR's list fails, its GV is degraded, or the controller lacks permission, + that cell is **left untouched and retried** — an *inability to observe* must + never be read as a deletion. +- The dividing line is **intent vs observability**: a cell leaving the desired + set because of a rule/scope/policy change is an intent-driven mutation and may + delete; a cell that is merely unobservable right now must hold. +- This invariant is per-cell, so one flaky GVR cannot corrupt an entire target. + +## 1.5 Convergence, determinism, idempotency + +- **Deterministic projection.** A target's desired mirror is a pure function of + (its rules, the authoritative catalog surface, cluster object state). Same + inputs → same files. This is what makes the plan-hash and tests trustworthy. +- **Convergence.** After any transient disruption (discovery gap, list failure, + permission blip), the system returns to the correct mirror automatically within + a bounded time, with no operator action and no spurious churn commits. +- **Idempotency.** Re-running a reconcile with unchanged inputs produces no + commit. + +## 1.6 A cell can disappear for several reasons; the cause decides the action + +This is the most important semantic to get right. "Cell removed" is **not** one +case — and collapsing them all into one policy is wrong. A cell can leave a +target's desired set because of an **intent** change or because of an +**observability** change, and the mirror must treat those oppositely: + +| Cause | Class | Mirror action | +|---|---|---| +| Rule deleted / narrowed / ops changed so the cell is no longer selected | intent | **Delete** the cell's managed files — required to keep the mirror a deterministic projection of the rules. | +| Namespace / scope change | intent | Old cell deselected → delete its managed files; new cell selected → seed it. | +| GitTarget rewiring (provider/branch/path change) | intent | Projection moves: delete from the old path, seed at the new one. | +| Resource policy now excludes the kind | intent/config | Likely **delete** (open question 6) — it is a config-driven deselection, not an outage. | +| Surface withdrawal (CRD uninstalled) | observability | **Keep** files (proposed) and record withdrawal in status/log. Absence is an operator surface change, not GitOps intent; deleting hundreds of files on a CRD upgrade/reinstall would be destructive and noisy. | +| Object deleted while kind still served | observability (authoritative) | Real deletion, driven by DELETE event / authoritative re-list → **delete** the file. | +| GV degraded / list failed / permission denied | observability (non-authoritative) | **Hold** the cell — never delete (see 1.4). | + +The principle: **only intent-driven removals and authoritatively-observed +deletions mutate the mirror; non-authoritative absences hold.** The reconciler +must therefore know *why* a cell left the desired set, not merely *that* it did. + +This requires a notion of **managed projection** (see 1.7): the set of files a +given cell owns, so "delete the cell's files" is well-defined and scoped. + +## 1.7 Requirements you may be missing + +Beyond the four beliefs above, a complete design needs: + +- **Deterministic file ownership (managed projection).** There must be a + deterministic mapping from a cell `{GitTarget, GVR, scope, namespaces}` to the + set of repo paths it owns, and a way to enumerate/diff *only* those paths. This + underpins two things at once: deleting exactly a deselected cell's files (1.6), + and **cell-scoped repo diffing** so a per-cell reconcile never has to read or + rewrite the whole GitTarget path (which would smuggle whole-target behavior + back in). Files not owned by any current cell are "unmanaged" and are never + touched unless an intent change brought them into scope. + +- **RBAC / least privilege.** "Watch everything" means the controller's + ServiceAccount must `list`/`watch` every selected GVR. A wildcard implies broad + RBAC. Required: either a documented broad ClusterRole, or per-GVR + permission-denied handling that degrades that *cell* gracefully (treat as + "cannot observe" → never delete, surface in status) instead of failing the + target. +- **Scale bounds & backpressure.** One informer per selected GVR across the whole + surface is real memory + apiserver load, and a CRD burst can cause a list + storm. Required: a soft cap with observability, and coalescing of surface + bursts (the existing single-slot refresh channel is a start). +- **Startup gating.** On cold start, do not snapshot (and certainly do not delete) + until the catalog has reached initial trust — otherwise an empty early surface + could wipe the mirror. +- **Observability / status conditions.** `WatchRule`/`ClusterWatchRule`/`GitTarget` + status must show: resolved GVR count (incl. wildcard expansion), degraded + surface, withdrawn types, permission failures, and unseeded/retrying cells. +- **Version selection.** A resource served at multiple versions must mirror + exactly one (preferred) to avoid duplicate files; define wildcard-version + behavior (proposal: still collapse to preferred unless a specific version is + named). +- **Resource policy under wildcard.** The default denylist + (pods/events/leases/…) still applies to `*`; define whether users can opt + excluded kinds back in. +- **Sensitive resources.** Wildcards will select Secrets and similar. This must + compose with the existing sensitive-resource classification work + ([sensitive-resource-classification-plan.md](sensitive-resource-classification-plan.md)). +- **Migration / backward compatibility.** Changing selector semantics changes + what *existing* rules watch. Required: no silent expansion of an existing user's + scope on upgrade — a migration story or explicit opt-in. +- **Causal ordering per cell.** A GVR's initial seed must be reconciled before its + live events are flushed, so the seed isn't overwritten or double-counted. Today + this is done per-target; it must hold per-cell. + +--- + +# Part 2 — Current architecture + +## 2.1 Components + +- **`APIResourceCatalog`** ([api_resource_catalog.go](../../internal/watch/api_resource_catalog.go)) + — in-memory index of served resources (`byGVR`, `byResource`, `byGroupRes`, + `byGroupVer`) with per-GV state (`trusted`/`degraded`) and a `generation` + counter. Refreshed from discovery; **already preserves entries for degraded + GVs** (`markFailedGroupVersions`) and removes confirmed-gone GVs only when + discovery is complete (`removeUndiscoveredGroupVersions`). +- **`RuleGVRResolver`** ([rule_gvr_resolver.go](../../internal/watch/rule_gvr_resolver.go)) + — maps one rule selector to concrete GVRs via the catalog, returning + `ResolveMiss` values classified by reason. **Refuses wildcards** in + `preflightMisses`. +- **`Manager`** ([manager.go](../../internal/watch/manager.go)) — owns the + reconcile loop, the active informer set (`activeInformers` keyed by + `GVR → namespace → cancel`), GVR resolution for both the watch plan and the + snapshot, and snapshot emission. +- **Event surface triggers** ([manager_catalog.go:278-303](../../internal/watch/manager_catalog.go#L278-L303)) + — informers on CRDs and APIServices that coalesce changes into + `catalogRefreshCh`. +- **`EventRouter` / per-GitTarget streams / reconcilers** — translate informer + events and control events (`RequestRepoState`, `RequestClusterState`) into git + commits, with a RECONCILING→LIVE buffering handshake per target. + +## 2.2 Control flow today + +``` +discovery / CRD trigger / 30s tick + │ + ▼ +ReconcileForRuleChange (manager.go:740) + ├─ RefreshAPIResourceCatalog → catalog updated + ├─ computeRequestedGVRs → resolve ALL rules → desired GVR set + ├─ compareGVRs → added / removed (per GVR+namespace) + ├─ snapshotTargetsNeedingDelivery → per-target plan-hash diff + ├─ stopInformer(removed) + ├─ beginReconciliationForTargets → buffer live events + ├─ startInformersForGVRs(added) → seed via informer cache + ├─ emitSnapshotForRuleChange → RequestRepoState + RequestClusterState + │ └─ whole-target cluster list, ABORTS on any blocking miss/list error + └─ completeReconciliationForTargets → flush buffered events +``` + +## 2.3 What already works in our favor + +- **Dynamic surface reaction exists.** The CRD/APIService triggers + reconcile + loop already start/stop informers as the surface changes. Reacting to a new + GVR is the steady state, not a new capability. +- **Per-(GVR, namespace) informer diffing exists** (`compareGVRs`, + `findAddedGVRs`/`findRemovedGVRs`). +- **Degraded-aware catalog exists** — last-known-good is already preserved. +- **Per-target snapshot isolation exists** (plan-hash selection). + +## 2.4 Where it falls short (the gaps Part 3 must close) + +1. **Two resolution call sites, two truths.** The watch-plan path + (`currentRuleSetSnapshots`, manager.go:1248/1264) and the snapshot path + (manager.go:559-594) both resolve rules independently — duplicated logic that + has already drifted (the bug that birthed + [watchrule-gvr-resolution-plan.md](../finished/watchrule-gvr-resolution-plan.md)). +2. **Snapshots are whole-target, not per-cell.** Adding one GVR re-lists and + re-diffs the *entire* target (`emitSnapshotForRuleChange` → + `RequestClusterState` over all GVRs). Violates requirement 1.3. +3. **One failure aborts the whole target.** A single GVR list failure or blocking + miss aborts the entire target snapshot (manager.go:596-628). Safe but coarse; + at wildcard scale it means a target may rarely snapshot. Violates the + *per-cell* form of requirement 1.4. +4. **Wildcards refused** at planning (requirement 1.2 unmet) — and the CRD field + docs advertise the opposite, so a `*` rule applies cleanly and silently + watches nothing. +5. **Degraded handling is bolted on, not central.** The discovery-lag fix + reintroduces "is this authoritative?" logic at the snapshot-selection layer + because the catalog doesn't expose confidence/deltas as first-class outputs. + Many `ResolveMiss` reasons + special cases = the exception sprawl the user + wants gone. +6. **Withdrawal semantics are emergent**, not decided (requirement 1.6). + +--- + +# Part 3 — Target architecture + +The strategy: make the **catalog** a proper live model that emits deltas and +confidence, make the **watch plan** a single deterministic projection consumed in +one place, and make **reconciliation per-cell and incremental**. This removes the +duplicate resolution, the whole-target snapshots, and most of the miss/degraded +special-casing in one move. + +## 3.1 Layer 1 — `ClusterSurface` (evolve the catalog) + +Promote `APIResourceCatalog` into the authoritative, observable +`ClusterSurface`: + +- **State per GVR:** `served | degraded | withdrawn`, plus the existing + allowed/listable/scope/preferred metadata. (`served`/`degraded` already exist; + add explicit `withdrawn` with a debounce before a degraded GV is declared + withdrawn.) +- **Per-cell confidence, not a global flag.** Expose + `AuthoritativeFor(GVR) Confidence` (and a startup-trust gate), **not** a global + `Authoritative() bool`. A single boolean would either block healthy targets + because one unrelated APIService is degraded, or get bypassed and become unsafe. + Confidence travels with each planned cell so destructive decisions are gated + per cell (1.4). Degradation is observed at GV granularity (discovery fails per + GroupVersion) and projected down to the GVRs in that GV — see open question 7 + on GV-vs-GVR granularity. +- **Current surface + generation is the source of truth; deltas are only + triggers.** Expose a queryable current surface snapshot (with its `generation`) + as the authority, and a `Subscribe() <-chan SurfaceDelta` + (`{GVR, Change: Added|Withdrawn|Degraded|Recovered}`) purely as an + *optimization* that tells consumers *when* and *roughly what* to re-evaluate. + Consumers must establish correctness by **re-diffing desired cells against + active cells** against the current surface, never by replaying deltas as if they + were complete — deltas can be missed across restart, backpressure, or + subscription churn. Deltas make reconvergence cheap; the re-diff makes it + correct. +- **Absorbs transience centrally.** No consumer ever sees a flapping CRD as + add/remove churn; the surface only reports `Withdrawn` after the debounce. This + is requirement 1.1, in one place, deleting the need for scattered + `CatalogUnavailable`/`DiscoveryDegraded` handling elsewhere. + +## 3.2 Layer 2 — `WatchPlan` (one resolver, projection semantics) + +A single `WatchPlan` projects rules onto the surface, used by **both** informers +and snapshots (kills gap 2.4.1): + +- `Plan(rules, surface) → map[GitTarget]TargetPlan`, where a `TargetPlan` is a + set of **cells**: `{GVR, scope, namespaces, ops}`. +- **Wildcards are projection, not expansion-at-apply** (requirement 1.2): `*` + selects all matching `served` cells from the surface *now*; the plan is + recomputed on every surface delta, so membership tracks the cluster live. +- **Empty vs `*`**: empty = strict unique-or-error; `*` = all. One small, explicit + rule — not a pile of miss reasons. +- **Resolution errors still exist and still matter.** Strict-empty ambiguity, + not-served, and disallowed are real errors the resolver must report. The + critical rule: an unresolved or errored selector must mark its cells + **unresolved/held**, never collapse them into an *empty desired set* — otherwise + a failed resolve looks like "the user wants nothing here" and the reconciler + deletes the projection (1.4). **Resolution failure ≠ empty desired state.** + What this design removes is the *duplicated, drifting* blocking-miss handling + spread across two resolution sites and the snapshot layer — not error handling + itself, which consolidates into one place with one meaning. + +## 3.3 Layer 3 — per-cell reconciler (incremental) + +Replace whole-target snapshotting with a **cell reconciler** driven by two delta +sources — rule changes and surface deltas — joined against current plans: + +``` +SurfaceDelta{GVR, …} ─┐ (triggers only) +RuleChange{target} ─┼─► RE-DIFF desired cells (WatchPlan over current +periodic tick ─┘ surface+generation) vs active cells + managed files + │ + ▼ + per-cell actions: + cell appeared → start informer + SEED ONLY THIS GVR (cell-scoped) → additions + cell removed → classify cause (1.6): + intent (rule/scope/policy) → stop informer + DELETE the + cell's managed files + surface withdrawal → stop informer + KEEP files, + record in status + non-authoritative (degraded/ + denied/list-failed) → HOLD: do not stop as removed, + do not delete, retry + cell unchanged → nothing +``` + +Two things this diagram makes explicit, both from review: + +- **Correctness is the re-diff, not the delta.** Desired cells are recomputed + from the `WatchPlan` over the *current* surface snapshot (and its generation) + and diffed against active cells and the managed-file set. Deltas and the + periodic tick only *schedule* a re-diff; missing a delta can never cause + divergence because the next re-diff is authoritative (§3.1). +- **Repo-side scoping must match.** Seeding/diffing/deleting a cell operates only + on that cell's managed files (1.7). `RequestRepoState` must be answerable at + cell granularity — list/diff just `{target, GVR, scope, namespaces}` — or the + per-cell path silently degrades back into whole-target snapshotting. + +Properties this gives us for free: + +- **Incremental** (requirement 1.3): a new CRD seeds exactly one GVR's objects on + the targets that select it; everything else is untouched. +- **Per-cell safety** (requirement 1.4): seeding lists only that GVR; on failure + the cell stays `unseeded/retrying` and writes nothing — no other cell, no + deletions. The "partial view = phantom deletes" failure mode becomes + structurally impossible because deletes are scoped to a cell that just listed + successfully. +- **Convergence** (requirement 1.5): a failed/degraded cell is simply retried on + the next relevant delta or tick; no target-wide churn. +- **Full re-snapshot becomes the fallback**, triggered only by explicit + resync/repair, not routine surface changes. + +Per-cell causal ordering reuses the existing RECONCILING→LIVE buffering, scoped to +the cell (requirement 1.7). + +## 3.4 How the existing docs fold in + +- **Discovery-lag fix** → disappears as a special case: the surface never reports + a degraded GVR as withdrawn, so a transient gap can't drop a cell or erase a + baseline. The plan-hash machinery is replaced by cell-level delta diffing. +- **Wildcard semantics doc** → becomes the spec for `WatchPlan` projection. +- **Wildcard support plan** → its phases map onto building Layers 1–3; the + "snapshot robustness" risk (its Phase 2) is resolved structurally by per-cell + seeding rather than needing a bespoke list-failure policy. +- **Initial-seed-via-watch-list** → the per-cell seed is exactly where the + cache-backed `WatchListClient` seed belongs. + +## 3.5 Suggested build order + +Each step is independently shippable and leaves the system correct. + +1. **Unify resolution** behind one `WatchPlan` used by both informer and snapshot + paths (removes drift; no behavior change). Pure refactor, fully unit-testable. +2. **Add surface state + deltas** to the catalog (`served/degraded/withdrawn`, + `Authoritative()`, `Subscribe()`), with withdrawal debounce. Internal; no + consumer change yet. +3. **Per-cell seeding** for *added* cells (incremental seed), keeping whole-target + snapshot as the fallback path. Closes requirement 1.3 for additions. +4. **Per-cell withdrawal policy** (requirement 1.6) for removed cells. +5. **Enable wildcards** in `WatchPlan` projection (requirement 1.2) — now safe + because Layers 1–3 handle dynamic membership, per-cell safety, and scale. +6. **Fix the field docs + admission validation** to match (and add status + conditions, RBAC guidance, scale caps). + +## 3.6 Cleanliness check (the user's steer) + +This design *removes* code as much as it adds: one resolver instead of two; one +authority for "is the surface trustworthy" instead of miss-reason handling spread +across resolver, snapshot, and selection layers; structural per-cell safety +instead of the whole-target abort + discovery-lag patch. The number of distinct +"cases" a maintainer must hold in their head drops from "many `ResolveMiss` +reasons × two resolution sites × whole-target abort rules × discovery-lag +bookkeeping" to "a surface with three states (queried as the source of truth, +with deltas as cheap triggers), projected by one resolver into cells that each +reconcile independently and classify their own removal cause." + +--- + +## Open decisions (need a human call before building) + +1. **Empty vs `*`**: adopt the strict-`[]` / wildcard-`*` split, or make `[]` + also mean "all" to match RBAC exactly? (Proposal: strict `[]`, wildcard `*`.) +2. **Surface-withdrawal default**: keep files on CRD uninstall (proposed) or + delete them? Per-GitTarget override? (This is the *observability* class in 1.6.) +3. **RBAC model for wildcard**: ship a broad ClusterRole, or require users to + grant it and degrade cells gracefully on permission-denied? +4. **Migration**: how do existing rules behave on upgrade — frozen semantics, or + opt-in to the new projection? +5. **Scale cap**: hard limit on informer count per controller, or + observe-and-warn only? +6. **Resource-policy deselection** (ref 1.6): when the default denylist newly + excludes a kind that was being mirrored, delete its managed files (treat as a + config-driven intent change) or keep them (treat as observability)? +7. **`withdrawn` granularity** (ref 3.1): is withdrawal a GV-level state, a + GVR-level state, or both? Degradation is naturally GV-level (discovery fails + per GroupVersion) while deltas are GVR-level — the model must state how the two + compose rather than switching between them. +8. **Preferred-version change**: when a resource's preferred version changes, is + that `Withdrawn(old)+Added(new)` (risks a delete+reseed and file churn), an + `Updated` delta, or an explicit **migration** action (move/rewrite files, + never delete)? Affects whether a routine apiserver upgrade churns the mirror. +9. **Rule-deselection disposition** (ref 1.6): when a rule stops selecting a cell, + delete its managed files immediately (proposed — preserves the deterministic + mirror) or leave them as *unmanaged historical* files? If the latter, define + how unmanaged files are marked and ever reclaimed. diff --git a/docs/design/watchrule-wildcard-and-resolution-semantics.md b/docs/design/watchrule-wildcard-and-resolution-semantics.md new file mode 100644 index 000000000..c0e818335 --- /dev/null +++ b/docs/design/watchrule-wildcard-and-resolution-semantics.md @@ -0,0 +1,131 @@ +# WatchRule Wildcard & Resolution Semantics + +How GitOps Reverser turns a `WatchRule`/`ClusterWatchRule` resource selector into +a concrete set of watchable GVRs, what it deliberately refuses, and why there is +no "watch everything" rule. + +Source of truth: `RuleGVRResolver` in +[rule_gvr_resolver.go](../../internal/watch/rule_gvr_resolver.go), backed by the +[APIResourceCatalog](../../internal/watch/api_resource_catalog.go). See also +[watchrule-gvr-resolution-plan.md](../finished/watchrule-gvr-resolution-plan.md). + +> **Note:** this documents *current* behavior. The CRD field docs promise +> admission-webhook-style wildcard semantics that this resolver does not yet +> implement — see +> [watchrule-wildcard-support-plan.md](watchrule-wildcard-support-plan.md) for +> that finding and the plan to resolve it. + +## Two directions, treated differently + +Resource selectors are used in two distinct places, and they handle wildcards +**oppositely**: + +| Direction | Code | Question it answers | `*` behavior | +|---|---|---|---| +| **Planning / resolution** | `RuleGVRResolver.Resolve(...)` ([rule_gvr_resolver.go:67](../../internal/watch/rule_gvr_resolver.go#L67)) | "Which concrete GVRs should I set up informers for and snapshot?" | **Refused** (`Wildcard*` miss). | +| **Matching** | `matchesAPIGroups`/`matchesResources` ([manager.go:480-513](../../internal/watch/manager.go#L480-L513)) | "Does this concrete incoming event's GVR match a rule?" | **Honored** (`"*"` matches anything). | + +The planning direction is the only one that creates informers and drives +snapshots (`gvrFromCompiledRule → Resolve`, +[gvr.go:79-92](../../internal/watch/gvr.go#L79-L92)). The matching direction only +runs against a GVR an informer **already delivered** +([collectWatchRuleNamespaces](../../internal/watch/manager.go#L416-L424)). + +**Consequence:** a `*` rule with no concrete companion rule sets up zero +informers and therefore watches nothing. The matching-side `*` only ever matches +events some *other* concrete rule already brought in. A lone `resources: ["*"]` +rule looks like "watch everything" but is effectively a no-op (it logs a +`WildcardResource` miss at `V(1)`). + +## The preflight gate + +`Resolve` calls `resolveResource` per resource name, which first runs +`preflightMisses` ([rule_gvr_resolver.go:108-131](../../internal/watch/rule_gvr_resolver.go#L108-L131)). +This gate short-circuits **before the catalog is ever queried**: + +| Condition | Result | Meaning | +|---|---|---| +| `resource == ""` | dropped, no miss | Empty resource entry is ignored. | +| `resource == "*"` | `WildcardResource` | `*` in `resources` — expansion unsupported. | +| `hasWildcard(groups)` | `WildcardGroup` | `*` in `apiGroups` — expansion unsupported. | +| `strings.Contains(resource, "/")` | `NotServed` | Subresource (e.g. `pods/status`) planning unsupported. | +| catalog nil or not `Ready()` | `CatalogUnavailable` | Discovery hasn't populated a catalog yet. | +| otherwise | proceed | Look the resource up in the catalog. | + +`WildcardResource` vs `WildcardGroup` is purely *which axis* held the `*`. +Resource is checked first, so `apiGroups:["*"], resources:["*"]` reports +`WildcardResource`. + +## What the resolver *does* expand from the catalog + +Two narrower, **bounded** expansions are supported — neither is an open wildcard: + +- **Empty `apiGroups`** (not `*`): `resourceCandidates` looks the resource name + up across *all* groups via + [`entriesForResource`](../../internal/watch/api_resource_catalog.go#L286-L290), + but [`ambiguityMiss`](../../internal/watch/rule_gvr_resolver.go#L141-L152) + refuses with `Ambiguous` if more than one group serves that name. So it is + "find this one kind without naming its group," requiring a unique answer — not + a fan-out. +- **Empty or `*` `apiVersions`**: + [`choosePreferredVersions`](../../internal/watch/rule_gvr_resolver.go#L212-L235) + collapses to the single **preferred** version per group-resource. + +Note the asymmetry: a version wildcard *is* accepted (it resolves to exactly one +version), but group/resource wildcards are refused. The dividing line is +**boundedness**: does the selector resolve to a fixed, knowable set? + +## There is no "watch everything" rule + +By design, no single rule can watch the whole cluster: + +- `resources: ["*"]` → `WildcardResource`, refused. +- `apiGroups: ["*"]` → `WildcardGroup`, refused. +- Empty `apiGroups` → resolves *one* kind across groups, and only if unambiguous. +- Empty `resources` → the `Resolve` loop iterates `resources`, so it yields zero + GVRs and watches nothing. + +The only way to watch broadly is to **enumerate each concrete resource kind +explicitly**. That list is static: a newly installed CRD is not picked up until a +rule names it (or a wildcard pattern would match it — which is exactly what is +refused). Dynamic "all current and future kinds" is intentionally not +expressible. + +## Why this is intentional + +1. **Bounded vs unbounded.** A version wildcard collapses to one preferred + version. A group/resource wildcard expands to an open-ended set whose + membership depends on what is installed right now. The resolver only expands + selectors that resolve to a fixed set. +2. **Plan stability.** A wildcard plan's membership would change every time *any* + CRD is installed or removed, churning the per-GitTarget effective-watch-plan + hash on unrelated cluster activity — the same spurious re-snapshot problem + described in + [rule-set-snapshot-discovery-lag-fix.md](rule-set-snapshot-discovery-lag-fix.md), + but permanent and amplified. Refusing wildcards keeps watch plans + deterministic. +3. **Snapshot safety.** Snapshotting `*` would commit the entire cluster (minus + policy-excluded kinds) to git on every rule-change snapshot — heavy, + surprising, and far more exposed to the partial-view aborts that + `RequestClusterState` enforces ([manager.go:596-601](../../internal/watch/manager.go#L596-L601)). +4. **A terse alternative already exists.** Empty `apiGroups` lets a rule name a + resource without its group as long as the name is unambiguous, so users are + not pushed toward wildcards just to stay concise. + +## Relationship to the snapshot miss classification + +`blockingSnapshotMisses` ([manager.go:657-669](../../internal/watch/manager.go#L657-L669)) +groups `WildcardGroup`/`WildcardResource` with `CatalogUnavailable`/ +`DiscoveryDegraded` as "blocking" — all four prevent emitting a snapshot. But the +two families differ in nature: + +- **Transient-blocking** (`CatalogUnavailable`, `DiscoveryDegraded`): the answer + changes once discovery recovers. Genuinely "ask again later." +- **Stable-blocking** (`WildcardGroup`, `WildcardResource`): a fixed property of + the rule text. Retrying never changes it; it is effectively a configuration + error to surface to the user. + +This distinction matters for the state model in +[rule-set-snapshot-discovery-lag-fix.md](rule-set-snapshot-discovery-lag-fix.md): +only the transient family should drive "retry next cycle"; the wildcard family +will not resolve on its own. diff --git a/docs/design/aggregated-api-requestheader-ca-miswire-report.md b/docs/finished/aggregated-api-requestheader-ca-miswire-report.md similarity index 100% rename from docs/design/aggregated-api-requestheader-ca-miswire-report.md rename to docs/finished/aggregated-api-requestheader-ca-miswire-report.md diff --git a/docs/design/audit-consumer-username-attribution-gap.md b/docs/finished/audit-consumer-username-attribution-gap.md similarity index 100% rename from docs/design/audit-consumer-username-attribution-gap.md rename to docs/finished/audit-consumer-username-attribution-gap.md diff --git a/docs/design/commit-window-refactor.md b/docs/finished/commit-window-refactor.md similarity index 100% rename from docs/design/commit-window-refactor.md rename to docs/finished/commit-window-refactor.md diff --git a/docs/future/design-audit-ingestion-hardening.md b/docs/finished/design-audit-ingestion-hardening.md similarity index 100% rename from docs/future/design-audit-ingestion-hardening.md rename to docs/finished/design-audit-ingestion-hardening.md diff --git a/docs/future/design-commit-context-api.md b/docs/finished/design-commit-context-api.md similarity index 99% rename from docs/future/design-commit-context-api.md rename to docs/finished/design-commit-context-api.md index edfff086a..183416e00 100644 --- a/docs/future/design-commit-context-api.md +++ b/docs/finished/design-commit-context-api.md @@ -8,7 +8,7 @@ > for the deep-dive on the aggregated-API audit gap; everything else here is reframed there. > > Original status: design — concrete pass at Option 3 from -> [idea-end-user-commit-messages.md](idea-end-user-commit-messages.md). +> [idea-end-user-commit-messages.md](../future/idea-end-user-commit-messages.md). > Date: 2026-05-07 ## Goals @@ -138,7 +138,7 @@ These are listed to validate that the resource shape generalises. None of them a initial implementation. - `commitNow` — finalize the matching open commit window immediately instead of waiting for the - silence timer. Open questions are tracked in [idea-end-user-commit-messages.md](idea-end-user-commit-messages.md) + silence timer. Open questions are tracked in [idea-end-user-commit-messages.md](../future/idea-end-user-commit-messages.md) under "Commit now." - `proposeAsPullRequest` — instruct the branch worker to push the in-progress commit window to a feature branch and open a pull request via `GitProvider`, instead of pushing to the target @@ -763,9 +763,9 @@ namespace `team-a`. ## References - Parent design exploration: - [idea-end-user-commit-messages.md](idea-end-user-commit-messages.md) + [idea-end-user-commit-messages.md](../future/idea-end-user-commit-messages.md) - Audit-carried transport addendum (alternatives this design supersedes for v1alpha1): - [addendum-end-user-commit-messages-audit-transports.md](addendum-end-user-commit-messages-audit-transports.md) + [addendum-end-user-commit-messages-audit-transports.md](../future/addendum-end-user-commit-messages-audit-transports.md) - Existing audit pipeline reference: `docs/design/audit-consumer-next-steps.md` - The aggregated-API audit gap and the related proxy: [external-sources/apiservice-audit-proxy/README.md](../../external-sources/apiservice-audit-proxy/README.md) diff --git a/docs/future/design-commit-request-api.md b/docs/finished/design-commit-request-api.md similarity index 96% rename from docs/future/design-commit-request-api.md rename to docs/finished/design-commit-request-api.md index 9a1128ecd..6ea9f957d 100644 --- a/docs/future/design-commit-request-api.md +++ b/docs/finished/design-commit-request-api.md @@ -1,7 +1,7 @@ # Design: `CommitRequest` CRD > Status: **implemented**. Namespaced CRD in API group `configbutler.ai/v1alpha1`. -> Deferred work is tracked in [design-commit-request-phase-2.md](design-commit-request-phase-2.md). +> Deferred work is tracked in [design-commit-request-phase-2.md](../future/design-commit-request-phase-2.md). > Date: 2026-05-18 ## What it is @@ -168,7 +168,7 @@ every event already commits immediately, so the signal simply finds no open wind ## Alternatives considered -The parent exploration ([idea-end-user-commit-messages.md](idea-end-user-commit-messages.md)) and +The parent exploration ([idea-end-user-commit-messages.md](../future/idea-end-user-commit-messages.md)) and the superseded aggregated-API design ([design-commit-context-api.md](design-commit-context-api.md)) weighed several transports. They are summarized here so the choice is not relitigated. @@ -209,15 +209,15 @@ clean success/SHA signal. Garbage collection, multi-target "save everything I edited," retry for transient finalize failures, and outcome metrics are intentionally out of this first cut. They are tracked — with the object-lifecycle question front and center — in -[design-commit-request-phase-2.md](design-commit-request-phase-2.md). +[design-commit-request-phase-2.md](../future/design-commit-request-phase-2.md). ## References -- Deferred / phase-2 work: [design-commit-request-phase-2.md](design-commit-request-phase-2.md) +- Deferred / phase-2 work: [design-commit-request-phase-2.md](../future/design-commit-request-phase-2.md) - Superseded aggregated-API design, kept for the audit-gap deep-dive: [design-commit-context-api.md](design-commit-context-api.md) - Parent exploration — transport options and the audit-stream-as-source-of-truth principle: - [idea-end-user-commit-messages.md](idea-end-user-commit-messages.md) + [idea-end-user-commit-messages.md](../future/idea-end-user-commit-messages.md) - Audit ingestion pipeline this design rides on: [design-audit-ingestion-hardening.md](design-audit-ingestion-hardening.md) - Implementation: `api/v1alpha1/commitrequest_types.go`, diff --git a/docs/future/design-rule-change-snapshot-trigger.md b/docs/finished/design-rule-change-snapshot-trigger.md similarity index 98% rename from docs/future/design-rule-change-snapshot-trigger.md rename to docs/finished/design-rule-change-snapshot-trigger.md index 669a77342..fc05fed3a 100644 --- a/docs/future/design-rule-change-snapshot-trigger.md +++ b/docs/finished/design-rule-change-snapshot-trigger.md @@ -213,7 +213,7 @@ The `EventRouter.SnapshotDeliveryDrops()` counter stays at 0 in steady state und ## References -- Trigger bug summary: [idea-cross-kind-dependency-watches.md](idea-cross-kind-dependency-watches.md) (related coordination concern around dependency-watches). +- Trigger bug summary: [idea-cross-kind-dependency-watches.md](../future/idea-cross-kind-dependency-watches.md) (related coordination concern around dependency-watches). - The old early-return: `ReconcileForRuleChange` in [manager.go](../../internal/watch/manager.go). - The snapshot consumer drop instrumentation: `RouteRepoStateEvent` and `RouteClusterStateEvent` in [event_router.go](../../internal/watch/event_router.go). - The GitTarget restart gate: `evaluateSnapshotGate` in [gittarget_controller.go](../../internal/controller/gittarget_controller.go). diff --git a/docs/design/e2e-cleanup-scoping.md b/docs/finished/e2e-cleanup-scoping.md similarity index 96% rename from docs/design/e2e-cleanup-scoping.md rename to docs/finished/e2e-cleanup-scoping.md index abb5514da..3f1d8b1f3 100644 --- a/docs/design/e2e-cleanup-scoping.md +++ b/docs/finished/e2e-cleanup-scoping.md @@ -15,7 +15,7 @@ The e2e suite originally had one global flag, `e2ePreserveResources`, that decid 2. The registered Gomega fail handler and the SIGINT/SIGTERM handler — both fed the same global preservation bit, so a normal assertion failure was effectively treated like a suite-wide abort. 3. The playground bootstrap, which flipped it deliberately to keep the playground live for the developer after the suite finishes. -Once the flag was on, `skipCleanupBecauseResourcesArePreserved` returned `true` for every cleanup call until the process exited. That meant a single failure poisoned the rest of the run: every spec that ran afterward leaked its WatchRules, GitTargets, ConfigMaps, and namespaces. Those leftovers then interfered with subsequent specs through the controller's own informer (most visibly, the signing batch spec — see [e2e-watchrule-cross-spec-interference.md](e2e-watchrule-cross-spec-interference.md)). +Once the flag was on, `skipCleanupBecauseResourcesArePreserved` returned `true` for every cleanup call until the process exited. That meant a single failure poisoned the rest of the run: every spec that ran afterward leaked its WatchRules, GitTargets, ConfigMaps, and namespaces. Those leftovers then interfered with subsequent specs through the controller's own informer (most visibly, the signing batch spec — see [e2e-watchrule-cross-spec-interference.md](../design/e2e-watchrule-cross-spec-interference.md)). ## Vision and design choice @@ -128,7 +128,7 @@ CI runs always pick up event dumps and controller logs from `dumpFailureDiagnost ## What this does not solve -This change does not fix the underlying ghost-WatchRule bug in the controller (described in [e2e-watchrule-cross-spec-interference.md](e2e-watchrule-cross-spec-interference.md) §"Open root causes" #1). The manager create/delete specs would still occasionally see ghost commits from prior specs that ran inside the same describe block, because those specs share the same `managerRepo` and namespace. Those need either: +This change does not fix the underlying ghost-WatchRule bug in the controller (described in [e2e-watchrule-cross-spec-interference.md](../design/e2e-watchrule-cross-spec-interference.md) §"Open root causes" #1). The manager create/delete specs would still occasionally see ghost commits from prior specs that ran inside the same describe block, because those specs share the same `managerRepo` and namespace. Those need either: - the controller-side fix (drain event-router state on WatchRule deletion), or - a unique-namespace-per-spec restructuring of the Manager describe block. diff --git a/docs/design/e2e-runtime-state-vs-stamps.md b/docs/finished/e2e-runtime-state-vs-stamps.md similarity index 100% rename from docs/design/e2e-runtime-state-vs-stamps.md rename to docs/finished/e2e-runtime-state-vs-stamps.md diff --git a/docs/design/e2e-test-review.md b/docs/finished/e2e-test-review.md similarity index 100% rename from docs/design/e2e-test-review.md rename to docs/finished/e2e-test-review.md diff --git a/docs/git-author.md b/docs/finished/git-author.md similarity index 100% rename from docs/git-author.md rename to docs/finished/git-author.md diff --git a/docs/design/gittarget-isolation-on-rule-change.md b/docs/finished/gittarget-isolation-on-rule-change.md similarity index 100% rename from docs/design/gittarget-isolation-on-rule-change.md rename to docs/finished/gittarget-isolation-on-rule-change.md diff --git a/docs/future/idea-audit-enrichment-side-channel.md b/docs/finished/idea-audit-enrichment-side-channel.md similarity index 100% rename from docs/future/idea-audit-enrichment-side-channel.md rename to docs/finished/idea-audit-enrichment-side-channel.md diff --git a/docs/design/partial-object-audit-event-handling.md b/docs/finished/partial-object-audit-event-handling.md similarity index 100% rename from docs/design/partial-object-audit-event-handling.md rename to docs/finished/partial-object-audit-event-handling.md diff --git a/docs/design/path-scoped-bootstrap-template-design.md b/docs/finished/path-scoped-bootstrap-template-design.md similarity index 100% rename from docs/design/path-scoped-bootstrap-template-design.md rename to docs/finished/path-scoped-bootstrap-template-design.md diff --git a/docs/design/remove-gittarget-admission-webhook.md b/docs/finished/remove-gittarget-admission-webhook.md similarity index 100% rename from docs/design/remove-gittarget-admission-webhook.md rename to docs/finished/remove-gittarget-admission-webhook.md diff --git a/docs/design/shallow-audit-event-misclassification.md b/docs/finished/shallow-audit-event-misclassification.md similarity index 100% rename from docs/design/shallow-audit-event-misclassification.md rename to docs/finished/shallow-audit-event-misclassification.md diff --git a/docs/design/watch-gvr-resolution-cleanup.md b/docs/finished/watch-gvr-resolution-cleanup.md similarity index 100% rename from docs/design/watch-gvr-resolution-cleanup.md rename to docs/finished/watch-gvr-resolution-cleanup.md diff --git a/docs/design/watchrule-gvr-resolution-plan.md b/docs/finished/watchrule-gvr-resolution-plan.md similarity index 99% rename from docs/design/watchrule-gvr-resolution-plan.md rename to docs/finished/watchrule-gvr-resolution-plan.md index 2c31b7f57..eba04e8a7 100644 --- a/docs/design/watchrule-gvr-resolution-plan.md +++ b/docs/finished/watchrule-gvr-resolution-plan.md @@ -147,7 +147,7 @@ its own discovery lookup or applying its own core `/v1` defaults. The detailed discovery model, refresh triggers, trust rules, and streaming-list note live in -[`kubernetes-api-resource-catalog.md`](./kubernetes-api-resource-catalog.md). +[`kubernetes-api-resource-catalog.md`](../design/kubernetes-api-resource-catalog.md). ### Catalog responsibilities diff --git a/docs/future/design-commit-request-phase-2.md b/docs/future/design-commit-request-phase-2.md index e277b039c..d15918271 100644 --- a/docs/future/design-commit-request-phase-2.md +++ b/docs/future/design-commit-request-phase-2.md @@ -1,6 +1,6 @@ # Phase 2: `CommitRequest` — deferred work -> Status: proposed. Follow-up to [design-commit-request-api.md](design-commit-request-api.md). +> Status: proposed. Follow-up to [design-commit-request-api.md](../finished/design-commit-request-api.md). > Date: 2026-05-18 The first cut of `CommitRequest` is intentionally minimal. This doc tracks what was deliberately diff --git a/docs/future/design-snapshot-engine-evolution.md b/docs/future/design-snapshot-engine-evolution.md index 776e8ec18..7c16c0d4d 100644 --- a/docs/future/design-snapshot-engine-evolution.md +++ b/docs/future/design-snapshot-engine-evolution.md @@ -1,7 +1,7 @@ # Design: snapshot engine evolution > Status: proposed — continuation of -> [design-rule-change-snapshot-trigger.md](design-rule-change-snapshot-trigger.md). +> [design-rule-change-snapshot-trigger.md](../finished/design-rule-change-snapshot-trigger.md). > Date: 2026-05-21 > Author context: written after a critical review of the rule-change snapshot > trigger implementation ([ed3a1db](#) "fix: better reconcile"). @@ -402,7 +402,7 @@ per-target. This removes a whole-cluster re-list amplification on CRD install. ### 1.5 Doc hygiene -[design-rule-change-snapshot-trigger.md](design-rule-change-snapshot-trigger.md) +[design-rule-change-snapshot-trigger.md](../finished/design-rule-change-snapshot-trigger.md) still cites `manager.go:660` for the old early-return; it is now line 696. Fix the stale reference since the doc is marked "implemented". @@ -675,7 +675,7 @@ flowchart LR ## References - Predecessor: - [design-rule-change-snapshot-trigger.md](design-rule-change-snapshot-trigger.md). + [design-rule-change-snapshot-trigger.md](../finished/design-rule-change-snapshot-trigger.md). - API discovery background (discoverability and verbs): [docs/design/kubernetes-api-discovery.md](../design/kubernetes-api-discovery.md). - API server capacity tuning: diff --git a/docs/future/idea-end-user-commit-messages.md b/docs/future/idea-end-user-commit-messages.md index d849d825b..39bf74df0 100644 --- a/docs/future/idea-end-user-commit-messages.md +++ b/docs/future/idea-end-user-commit-messages.md @@ -422,7 +422,7 @@ this comparison underweighted on the first pass. The aggregated API is also futu future temporary actions (`commitNow`, "propose as PR") fit the same shape with no new transports, and the close-off pattern is a clean fit for parallel-write frontends. -The detailed design lives in [design-commit-context-api.md](design-commit-context-api.md). +The detailed design lives in [design-commit-context-api.md](../finished/design-commit-context-api.md). The audit-carried transports remain valuable as a fallback for clusters that already have the infrastructure in place (impersonation in regular use, or OIDC `AuthenticationConfiguration` diff --git a/docs/future/idea-unify-pending-write-kinds.md b/docs/future/idea-unify-pending-write-kinds.md index 058fb7b60..f7c875232 100644 --- a/docs/future/idea-unify-pending-write-kinds.md +++ b/docs/future/idea-unify-pending-write-kinds.md @@ -19,7 +19,7 @@ unset. The temptation is to make that the dispatch and delete the `Kind` enum. ## Use case walkthrough -The four use cases in [commit-window-refactor.md](../design/commit-window-refactor.md#use-cases): +The four use cases in [commit-window-refactor.md](../finished/commit-window-refactor.md#use-cases): 1. **Burst collapse.** Per-event-only behavior (atomic is one commit by definition). Unification neither helps nor breaks this. diff --git a/docs/interpreting-metrics.md b/docs/interpreting-metrics.md index 009ab03da..820767f16 100644 --- a/docs/interpreting-metrics.md +++ b/docs/interpreting-metrics.md @@ -191,7 +191,7 @@ topk(15, sum by (resource, subresource, verb) ( Any row with a non-empty `subresource` is received-then-dropped at ingress (subresources are not mirrorable top-level objects). A large `pods`/`exec` row is the canonical example — see -[shallow-audit-event-misclassification.md](design/shallow-audit-event-misclassification.md). To +[shallow-audit-event-misclassification.md](finished/shallow-audit-event-misclassification.md). To look only at the dropped subresource traffic: ```promql From feeee67255c624d13d510624ebee231eca3d922f Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Tue, 2 Jun 2026 08:56:37 +0000 Subject: [PATCH 30/40] chore: fixing review issues and adding diagnostics on still failing flux setup --- .../design/watchrule-wildcard-support-plan.md | 192 ++++++++++++++++ docs/finished/e2e-test-review.md | 2 +- hack/e2e/wait-flux-services.sh | 52 ++++- internal/git/branch_worker.go | 47 +++- internal/git/branch_worker_metrics_test.go | 70 +++++- internal/git/finalize_signal.go | 5 + internal/watch/manager.go | 54 +++-- internal/watch/rule_change_snapshot_test.go | 208 ++++++++++++------ 8 files changed, 539 insertions(+), 91 deletions(-) create mode 100644 docs/design/watchrule-wildcard-support-plan.md diff --git a/docs/design/watchrule-wildcard-support-plan.md b/docs/design/watchrule-wildcard-support-plan.md new file mode 100644 index 000000000..6d984474b --- /dev/null +++ b/docs/design/watchrule-wildcard-support-plan.md @@ -0,0 +1,192 @@ +# WatchRule Wildcard Support — Finding & Plan + +Status: **proposed** (not started) + +> **Superseded framing:** the finding below stands, but the recommended way to +> resolve it is the unified model in +> [watch-and-catalog-architecture.md](watch-and-catalog-architecture.md). Treat +> this document as the focused finding + a tactical (Phase-0-first) fallback if +> the larger refactor is deferred. + +## Finding: documented behavior contradicts implemented behavior + +The `WatchRule`/`ClusterWatchRule` CRD field docs promise Kubernetes +admission-webhook / RBAC wildcard semantics, but the GVR resolver refuses them. + +The field comments in [watchrule_types.go:71-118](../../api/v1alpha1/watchrule_types.go#L71-L118) +(shipped as the CRD OpenAPI schema → `kubectl explain`) state: + +- "All fields except Resources are optional and default to matching all when not + specified." +- `apiGroups`: "Wildcards supported: `*` matches all groups", "`["*"]` or `[]` + matches all groups". +- `apiVersions`: "If empty, matches all versions", "`*` matches all versions". +- `resources`: "Wildcard semantics follow Kubernetes admission webhook patterns: + `*` matches all resources", "`pods/*` matches all pod subresources", + "`pods/log` matches specific subresource". + +The resolver's preflight gate +([rule_gvr_resolver.go:108-131](../../internal/watch/rule_gvr_resolver.go#L108-L131)) +does the opposite: + +| User writes | Field doc promises | Resolver actually does | +|---|---|---| +| `resources: ["*"]` | all resources | `WildcardResource` miss — **refused** | +| `apiGroups: ["*"]` | all groups | `WildcardGroup` miss — **refused** | +| `apiVersions: []` / `["*"]` | all versions | single **preferred** version only | +| `apiGroups: []` (ambiguous resource) | all groups | `Ambiguous` miss — **refused** | +| `resources: ["pods/log"]`, `["pods/*"]` | subresource match | `NotServed` — **refused** | + +There is **no admission validation** rejecting these (only `Operations` has an +enum), so a user can `kubectl apply` a `resources: ["*"]` rule, get no error, and +**silently watch nothing** — only a `V(1)` resolve-miss log. See +[watchrule-wildcard-and-resolution-semantics.md](watchrule-wildcard-and-resolution-semantics.md) +for the full current semantics. + +### Why it matters + +The field docs were written in the Kubernetes admission-webhook / RBAC / +audit-policy idiom (the comment says so verbatim). **That is what users will +expect** — `*` and "empty = all" are universal in those APIs. The current +behavior is a silent, surprising divergence from a well-known contract. + +### Why it diverges today + +RBAC and audit policy evaluate selectors **lazily, per request**: a concrete GVR +already exists on the incoming request and the rule is just a predicate. GitOps +Reverser must **eagerly materialize** the watch set — an informer per GVR plus +authoritative `list` snapshots — so it needs concrete GVRs *before* any event +exists. The resolver was added to fix a concrete expansion bug +([watchrule-gvr-resolution-plan.md](../finished/watchrule-gvr-resolution-plan.md)) and, in +making expansion safe, narrowed the contract without updating the docs. Notably +the *matching* layer ([manager.go:480-513](../../internal/watch/manager.go#L480-L513)) +*does* honor `*` lazily like RBAC — but it only ever runs against GVRs an +informer already delivers, so it cannot rescue a wildcard rule that planned +nothing. + +## What already exists (good news for difficulty) + +The hard part of dynamic wildcard support — reacting to a changing API surface — +is **already built and running**: + +- API-surface trigger informers watch CRDs and APIServices and coalesce + add/update/delete into `catalogRefreshCh` + ([manager_catalog.go:278-303](../../internal/watch/manager_catalog.go#L278-L303)). +- The manager loop re-runs `ReconcileForRuleChange` on that signal and every 30s + ([manager.go:218-245](../../internal/watch/manager.go#L218-L245)). +- `ReconcileForRuleChange` re-resolves desired GVRs, diffs added/removed against + active informers, and starts/stops informers live + ([manager.go:751-794](../../internal/watch/manager.go#L751-L794)). +- Per-GitTarget snapshot isolation already scopes which targets re-snapshot on a + plan change + ([gittarget-isolation-on-rule-change.md](../finished/gittarget-isolation-on-rule-change.md), + [rule-set-snapshot-discovery-lag-fix.md](rule-set-snapshot-discovery-lag-fix.md)). + +So "we'd have to redo reconciles when a GVR is added" is **already the steady +state** for explicit rules. Wildcard support mostly means letting the resolver +emit a larger, catalog-derived set; the reconcile/informer/snapshot plumbing +downstream is unchanged. The churn cost is real but **scoped to the targets that +actually use a wildcard** — explicit-rule targets keep full isolation. + +The default watch policy is a small **denylist** +([resource_policy.go:30-47](../../internal/watch/resource_policy.go#L30-L47)): +pods, events, endpoints, endpointslices, leases, controllerrevisions, jobs, +cronjobs, flowcontrol. So "watch everything" really means "every served, allowed, +listable, watchable resource minus ~12 noisy kinds" — still large (every CRD +kind in the cluster), which drives the scale concerns below. + +## Difficulty assessment + +Ordered low → high effort/risk. + +### A. Resolver wildcard expansion — **moderate, low risk** + +Replace the preflight refusals with catalog enumeration: + +- `apiGroups: ["*"]` → enumerate all groups serving the named resource(s). +- `resources: ["*"]` → enumerate all resources in the named group(s) (or all + groups if combined with group `*`). +- Both `*` → every entry in the catalog. + +The catalog already holds the data (`byGVR`, `byGroupVer`, `byResource`); add +enumeration accessors (e.g. `allEntries()`, `entriesForGroup(group)`). Reuse the +existing `Allowed` / `Supports("list","watch")` / scope filters in +`gvrsForCandidates` so policy and listability still apply. Bounded, mechanical. + +### B. Empty-vs-wildcard semantics — **small, but a real decision** + +Today empty `apiGroups` demands a *unique* group (`Ambiguous` otherwise). A +wildcard must mean "**all** matches", not "exactly one". Decision: keep `[]` = +"resolve to the single served group, else error" (current, safe) and make `["*"]` += "all served groups". This preserves a useful strict mode while honoring the +documented wildcard. Same split for versions: `[]` = preferred (current), +`["*"]` = all served versions (new). + +### C. Subresources — **small; mostly a docs fix, not a feature** + +`pods/log` / `pods/*` cannot be `list`/`watch`-ed as a snapshot surface; the +list/watch+mirror model can't represent them. Recommend **dropping the +subresource promise from the field docs** rather than implementing it, and +keeping the `NotServed` classification. (Subresources still flow as audit events +through a different path; this is only about WatchRule planning.) + +### D. Snapshot robustness at wildcard scale — **moderate→large, the real risk** + +A wildcard target lists *every* allowed GVR. Two existing behaviors get stressed: + +- **Partial-view abort.** `RequestClusterState` aborts the whole target snapshot + if any single GVR `list` fails + ([manager.go:596-628](../../internal/watch/manager.go#L596-L628)) — correct + today because a missing list looks like deletions. Across hundreds of GVRs the + probability that *one* fails is much higher, so a wildcard target could rarely + produce a complete snapshot. Likely needs a per-GVR resilience strategy that + still distinguishes "failed to list" from "genuinely empty" without mirroring + spurious deletions. This is the hardest correctness question. +- **Informer scale / memory.** One informer per GVR across the whole surface is a + real resource cost; worth a soft cap or opt-in guardrail. + +### E. Plan-hash churn — **small mechanically, a policy choice** + +A wildcard target's effective-watch-plan hash changes on every CRD install/delete +→ it re-snapshots each time. Per-target isolation already contains this to +wildcard targets. Options: accept it (simplest; the user opted into "everything"), +or debounce/coalesce CRD bursts. The existing `catalogRefreshCh` already coalesces +to depth 1, so bursts are partly absorbed. **Probably acceptable as-is for now.** + +### F. Status & validation surface — **small** + +- `ResolveWatchRuleResources` / `ResolveClusterWatchRuleResources` + ([manager_catalog.go:230-265](../../internal/watch/manager_catalog.go#L230-L265)) + should report "wildcard expanded to N GVRs" in status so users see the effect. +- Correct the CRD field docs (requires `task generate` + `task manifests`). +- Optionally add a `DiscoveryDegraded`-aware guard: a wildcard expanded while a + relevant group's discovery is degraded is an incomplete plan and should block + (consistent with current snapshot-safety rules). + +## Proposed phased plan + +1. **Phase 0 — stop the silent lie (smallest, ship independently).** Either make + the docs match reality (state that `*`/subresources are rejected, `[]` + resolves uniquely/preferred) *or* add admission validation that rejects `*` + with a clear message. This removes the "applies cleanly, watches nothing" + trap regardless of whether full support lands. +2. **Phase 1 — resolver expansion (A + B + C).** Implement `*` enumeration with + the all-vs-unique split, drop the subresource promise, surface expansion in + status (F, partial). Gated behind the existing reconcile machinery; no + snapshot changes yet. Wildcard targets now watch live events correctly. +3. **Phase 2 — snapshot robustness (D).** Decide and implement per-GVR list + resilience so a wildcard target can take a complete, safe cluster snapshot. + This is the gating item for declaring wildcard support "done". +4. **Phase 3 — guardrails (D scale, E debounce).** Informer-count cap/observability + and optional CRD-burst debounce if churn proves costly in practice. + +## Open decisions + +- Phase 0: fix docs, or add rejecting validation, or both? (Both is cleanest.) +- D: what is the correct "one GVR failed to list" behavior for a wildcard target + — skip that GVR for this cycle (risk: looks like deletions) vs abort the whole + target (risk: wildcard targets rarely snapshot)? This needs its own mini-design. +- Should "watch everything" remain policy-filtered by the default denylist, or + should wildcard rules be able to opt into the excluded noisy kinds? +- Is the plan-hash churn for wildcard targets acceptable for v1, or is debounce a + prerequisite? diff --git a/docs/finished/e2e-test-review.md b/docs/finished/e2e-test-review.md index 7dea1ab06..33045eb4f 100644 --- a/docs/finished/e2e-test-review.md +++ b/docs/finished/e2e-test-review.md @@ -1,6 +1,6 @@ # E2E Test Review: Gomega/Ginkgo Implementation -Related investigation: [E2E Full-Suite Shared-State Investigation](./e2e-full-suite-shared-state-investigation.md) +Related investigation: [E2E Full-Suite Shared-State Investigation](../design/e2e-full-suite-shared-state-investigation.md) ## What's Good diff --git a/hack/e2e/wait-flux-services.sh b/hack/e2e/wait-flux-services.sh index 1125eb723..0ff0873dd 100755 --- a/hack/e2e/wait-flux-services.sh +++ b/hack/e2e/wait-flux-services.sh @@ -11,6 +11,50 @@ set -euo pipefail : "${CTX:?CTX is required}" FLUX_SERVICES_WAIT_TIMEOUT="${FLUX_SERVICES_WAIT_TIMEOUT:-120s}" +# On a wait timeout we otherwise get a single opaque "timed out" line and have to +# guess. This dumps the resource's conditions, the source it pulls from +# (OCIRepository/HelmRepository), the workloads/events in its target namespace and +# recent Flux controller logs so the *cause* is visible in the CI log. Best-effort: +# every command is guarded so diagnostics never mask the original failure. +dump_diagnostics() { + local kind="$1" namespace="$2" name="$3" + + echo "::group::diagnostics ${kind}/${name} (ns=${namespace})" >&2 + echo "----- ${kind}/${name} conditions -----" >&2 + kubectl --context "${CTX}" -n "${namespace}" get "${kind}/${name}" -o wide >&2 2>&1 || true + kubectl --context "${CTX}" -n "${namespace}" get "${kind}/${name}" \ + -o jsonpath='{range .status.conditions[*]}{.type}={.status} reason={.reason} msg={.message}{"\n"}{end}' >&2 2>&1 || true + + if [[ "${kind}" == helmreleases.* ]]; then + # Resolve the chart source (OCIRepository/HelmRepository) and its readiness. + local src_kind src_name src_ns target_ns + src_kind="$(kubectl --context "${CTX}" -n "${namespace}" get "${kind}/${name}" -o jsonpath='{.spec.chartRef.kind}' 2>/dev/null || true)" + src_name="$(kubectl --context "${CTX}" -n "${namespace}" get "${kind}/${name}" -o jsonpath='{.spec.chartRef.name}' 2>/dev/null || true)" + src_ns="$(kubectl --context "${CTX}" -n "${namespace}" get "${kind}/${name}" -o jsonpath='{.spec.chartRef.namespace}' 2>/dev/null || true)" + src_ns="${src_ns:-${namespace}}" + if [[ -n "${src_kind}" && -n "${src_name}" ]]; then + echo "----- source ${src_kind}/${src_name} (ns=${src_ns}) -----" >&2 + kubectl --context "${CTX}" -n "${src_ns}" get "${src_kind}" "${src_name}" -o wide >&2 2>&1 || true + kubectl --context "${CTX}" -n "${src_ns}" get "${src_kind}" "${src_name}" \ + -o jsonpath='{range .status.conditions[*]}{.type}={.status} reason={.reason} msg={.message}{"\n"}{end}' >&2 2>&1 || true + fi + + target_ns="$(kubectl --context "${CTX}" -n "${namespace}" get "${kind}/${name}" -o jsonpath='{.spec.targetNamespace}' 2>/dev/null || true)" + target_ns="${target_ns:-${namespace}}" + echo "----- pods in target namespace ${target_ns} -----" >&2 + kubectl --context "${CTX}" -n "${target_ns}" get pods -o wide >&2 2>&1 || true + echo "----- recent events in ${target_ns} -----" >&2 + { kubectl --context "${CTX}" -n "${target_ns}" get events --sort-by=.lastTimestamp 2>&1 | tail -n 30; } >&2 || true + fi + + echo "----- recent flux controller logs (source/helm/kustomize) -----" >&2 + for ctrl in source-controller helm-controller kustomize-controller; do + echo "--- ${ctrl} ---" >&2 + kubectl --context "${CTX}" -n flux-system logs "deploy/${ctrl}" --tail=40 >&2 2>&1 || true + done + echo "::endgroup::" >&2 +} + flux_ready_count=0 echo "⏳ Waiting for Flux-managed installations to become ready..." @@ -36,10 +80,14 @@ do while IFS=' ' read -r namespace name; do [[ -n "${namespace}" ]] || continue - kubectl --context "${CTX}" -n "${namespace}" \ + if ! kubectl --context "${CTX}" -n "${namespace}" \ wait "${kind}/${name}" \ --for=condition=Ready \ - --timeout="${FLUX_SERVICES_WAIT_TIMEOUT}" + --timeout="${FLUX_SERVICES_WAIT_TIMEOUT}"; then + echo "ERROR: ${kind}/${name} (ns=${namespace}) did not become Ready within ${FLUX_SERVICES_WAIT_TIMEOUT}" >&2 + dump_diagnostics "${kind}" "${namespace}" "${name}" + exit 1 + fi done <<<"${resources}" done diff --git a/internal/git/branch_worker.go b/internal/git/branch_worker.go index 5a628d44a..6e3ada1b2 100644 --- a/internal/git/branch_worker.go +++ b/internal/git/branch_worker.go @@ -130,9 +130,9 @@ type BranchWorker struct { // hasUnpushedWork mirrors whether the event loop is currently holding a live // open window or any committed-but-not-yet-pushed pending writes. The event - // loop is the only writer; recordQueueDepth (callable from enqueue goroutines) - // is the only other reader, hence the atomic. It lets the depth gauge account - // for retained work the channel length alone cannot see. + // loop is the only writer and, via syncQueueDepthMetric, the only reader; the + // atomic guards the store/load and lets the depth gauge account for retained + // work the channel length alone cannot see. hasUnpushedWork atomic.Bool // inflightItems counts work items accepted onto eventQueue but not yet fully @@ -272,7 +272,9 @@ func (w *BranchWorker) EnqueueFinalize(signal *FinalizeSignal) { select { case w.eventQueue <- WorkItem{Finalize: signal}: w.Log.V(1).Info("Finalize signal enqueued") - w.recordQueueDepth() + // Depth is published only from the loop goroutine (syncQueueDepthMetric); + // the loop republishes on every received item, so the gauge converges + // without an enqueue-side write that could latch a stale value. default: w.inflightItems.Add(-1) w.Log.Error(nil, "Event queue full, finalize signal dropped") @@ -294,7 +296,9 @@ func (w *BranchWorker) enqueueRequest(request *WriteRequest) { "events", len(request.Events), "mode", request.CommitMode, "gitTarget", request.GitTargetName) - w.recordQueueDepth() + // Depth is published only from the loop goroutine (syncQueueDepthMetric); + // the loop republishes on every received item, so the gauge converges + // without an enqueue-side write that could latch a stale value. default: w.inflightItems.Add(-1) w.Log.Error(nil, "Event queue full, request dropped", @@ -309,9 +313,11 @@ func (w *BranchWorker) enqueueRequest(request *WriteRequest) { // processed) plus one when the event loop is holding retained unpushed work (a // live open window or pending writes). It reads 0 only when the worker has // fully drained — every accepted item handled and nothing retained — so a -// drain gate cannot be satisfied while a commit/push is still in flight. Safe -// to call from any goroutine: it reads two atomics. No-op until the gauge is -// registered (e.g. in unit tests that never init the exporter). +// drain gate cannot be satisfied while a commit/push is still in flight. Called +// only from the loop goroutine (via syncQueueDepthMetric) so the OTel gauge — +// last-writer-wins — can never latch a stale depth from an enqueue goroutine +// that raced the loop's drain. No-op until the gauge is registered (e.g. in +// unit tests that never init the exporter). func (w *BranchWorker) recordQueueDepth() { if telemetry.BranchWorkerQueueDepth == nil { return @@ -709,6 +715,31 @@ func (l *branchWorkerEventLoop) handleShutdown() { // the worker exits, even if a push was just sent. l.pushPending() } + l.drainUnhandledQueueItems() +} + +// drainUnhandledQueueItems clears items still buffered on eventQueue that the +// exiting loop will never handle. Each was counted into inflightItems at enqueue +// and is decremented only by the loop after handling, so without this drain the +// final syncQueueDepthMetric would publish a non-zero depth for the exiting +// worker that never clears — the loop has stopped, so nothing republishes a +// corrected value. A buffered finalize signal is answered with +// ErrWorkerShuttingDown so its caller unblocks instead of waiting out its +// timeout. The depth gauge then settles to 0 once the open window is finalized +// and pending writes are pushed (any genuinely-unpushed work keeps it non-zero, +// which is correct). +func (l *branchWorkerEventLoop) drainUnhandledQueueItems() { + for { + select { + case item := <-l.w.eventQueue: + if item.Finalize != nil { + item.Finalize.reply(FinalizeResult{Branch: l.w.Branch, Err: ErrWorkerShuttingDown}) + } + l.w.inflightItems.Add(-1) + default: + return + } + } } func (l *branchWorkerEventLoop) resetCommitTimer() { diff --git a/internal/git/branch_worker_metrics_test.go b/internal/git/branch_worker_metrics_test.go index 5e799d90b..c30a99657 100644 --- a/internal/git/branch_worker_metrics_test.go +++ b/internal/git/branch_worker_metrics_test.go @@ -21,6 +21,7 @@ package git import ( "context" "testing" + "time" "github.com/go-logr/logr" "github.com/stretchr/testify/assert" @@ -106,9 +107,12 @@ func TestRecordQueueDepth_InflightItemDequeuedButUnhandled(t *testing.T) { assert.Equal(t, int64(1), depth, "an in-flight item must keep depth > 0 even with an empty channel") } -// enqueueRequest must publish a non-zero depth as soon as the item lands on the -// queue, closing the race where a poller reads a stale drained value. -func TestEnqueueRequest_RecordsDepth(t *testing.T) { +// enqueueRequest must account for the accepted item in inflightItems but must +// NOT publish the depth gauge itself: the gauge is last-writer-wins, so an +// enqueue goroutine that raced the loop's drain could latch a stale value and +// hang the restart drain gate. Publication is the loop's job; once the loop +// observes the inflight item it reports the non-zero depth. +func TestEnqueueRequest_AccountsInflightWithoutPublishing(t *testing.T) { reader, err := telemetry.InitTestExporter() require.NoError(t, err) @@ -118,7 +122,65 @@ func TestEnqueueRequest_RecordsDepth(t *testing.T) { CommitMode: CommitModePerEvent, }) + // The item is accounted for, but enqueue did not touch the gauge. + assert.Equal(t, int64(1), w.inflightItems.Load()) + _, ok := telemetry.CollectInt64Sum(reader, branchWorkerQueueDepthMetric, queueDepthLabels()) + assert.False(t, ok, "enqueue must not publish the depth gauge; only the loop does") + + // The loop's publication (modelled here by a direct record) then reports the + // inflight item as non-zero depth. + w.recordQueueDepth() depth, ok := telemetry.CollectInt64Sum(reader, branchWorkerQueueDepthMetric, queueDepthLabels()) - require.True(t, ok, "expected a branch_worker_queue_depth sample after enqueue") + require.True(t, ok, "expected a branch_worker_queue_depth sample after the loop publishes") assert.Equal(t, int64(1), depth) } + +// On shutdown the loop must drain items still buffered on eventQueue so the depth +// gauge settles to 0. Each buffered item was counted into inflightItems at +// enqueue; left undrained, the exiting worker's final publish would latch a +// non-zero depth that never clears. +func TestHandleShutdown_DrainsBufferedItemsToZeroDepth(t *testing.T) { + reader, err := telemetry.InitTestExporter() + require.NoError(t, err) + + w := newMetricsTestWorker() + // Two write requests accepted onto the queue but never handled by the loop. + w.enqueueRequest(&WriteRequest{Events: []Event{{}}, CommitMode: CommitModePerEvent}) + w.enqueueRequest(&WriteRequest{Events: []Event{{}}, CommitMode: CommitModePerEvent}) + require.Equal(t, int64(2), w.inflightItems.Load()) + + loop := newBranchWorkerEventLoop(w, time.Second) + loop.handleShutdown() + // run() publishes once more after handleShutdown; model that here. + loop.syncQueueDepthMetric() + + assert.Equal(t, int64(0), w.inflightItems.Load(), "buffered items must be drained from the inflight count") + depth, ok := telemetry.CollectInt64Sum(reader, branchWorkerQueueDepthMetric, queueDepthLabels()) + require.True(t, ok, "expected a branch_worker_queue_depth sample after shutdown") + assert.Equal(t, int64(0), depth, "a drained, exiting worker must publish depth 0") +} + +// A finalize signal still buffered at shutdown must be answered with +// ErrWorkerShuttingDown so its caller unblocks instead of waiting out its +// timeout, and its inflight slot must be released. +func TestHandleShutdown_RepliesToBufferedFinalize(t *testing.T) { + _, err := telemetry.InitTestExporter() + require.NoError(t, err) + + w := newMetricsTestWorker() + result := make(chan FinalizeResult, 1) + w.EnqueueFinalize(&FinalizeSignal{Result: result}) + require.Equal(t, int64(1), w.inflightItems.Load()) + + loop := newBranchWorkerEventLoop(w, time.Second) + loop.handleShutdown() + + assert.Equal(t, int64(0), w.inflightItems.Load()) + select { + case res := <-result: + require.ErrorIs(t, res.Err, ErrWorkerShuttingDown) + assert.Equal(t, "main", res.Branch) + default: + t.Fatal("expected a finalize reply on shutdown, got none") + } +} diff --git a/internal/git/finalize_signal.go b/internal/git/finalize_signal.go index 481b881a5..c04f30795 100644 --- a/internal/git/finalize_signal.go +++ b/internal/git/finalize_signal.go @@ -24,6 +24,11 @@ import "errors" // because the worker's event queue is saturated. var ErrFinalizeQueueFull = errors.New("branch worker event queue full; finalize signal dropped") +// ErrWorkerShuttingDown is reported for a finalize signal that was still buffered +// on the event queue when the worker began shutting down, so its caller unblocks +// immediately instead of waiting out its timeout. +var ErrWorkerShuttingDown = errors.New("branch worker shutting down; finalize signal not processed") + // FinalizeOutcome is the terminal result of processing a FinalizeSignal. type FinalizeOutcome string diff --git a/internal/watch/manager.go b/internal/watch/manager.go index 70cfe4724..c1321d366 100644 --- a/internal/watch/manager.go +++ b/internal/watch/manager.go @@ -119,8 +119,9 @@ func (m *Manager) SnapshotEmitCount() int64 { } type ruleSetSnapshotTarget struct { - gitDest types.ResourceReference - hash uint64 + gitDest types.ResourceReference + hash uint64 + hasEntries bool } // targetWatchPlan accumulates a single GitTarget's effective watch surface while @@ -799,15 +800,23 @@ func (m *Manager) ReconcileForRuleChange(ctx context.Context) error { // Emit RequestClusterState for each affected GitTarget so that a single // "reconcile: sync N resources" commit is produced instead of N individual // [CREATE] commits from the informer ADDED events buffered above. - m.emitSnapshotForRuleChange(ctx, log, targets, "rule_change") + deliveryErr := m.emitSnapshotForRuleChange(ctx, log, targets, "rule_change") // Transition streams back to LIVE_PROCESSING and flush buffered events. // startInformersForGVRs already waited for cache sync (WaitForCacheSync), // so all initial ADDED events are guaranteed to be buffered before this point. // The flushed events are no-ops at the git level because the snapshot batch - // just wrote those files. + // just wrote those files. Run this even when a target failed delivery so the + // streams that did snapshot leave the buffering state. m.completeReconciliationForTargets(targets, log) + if deliveryErr != nil { + // A transient emit failure left at least one target pending. Surface it so + // the controller requeues with backoff and retries promptly instead of + // waiting for the next periodic reconcile. + return deliveryErr + } + log.V(1).Info("Watch manager reconciliation completed", "addedGVRs", len(added), "removedGVRs", len(removed)) @@ -1179,6 +1188,9 @@ func (m *Manager) snapshotTargetsNeedingDelivery() []ruleSetSnapshotTarget { for _, target := range current { key := target.gitDest.Key() currentKeys[key] = struct{}{} + if !target.hasEntries { + continue + } if lastDelivered, ok := m.lastDeliveredRuleSetHash[key]; ok && lastDelivered == target.hash { continue } @@ -1218,7 +1230,8 @@ func (m *Manager) snapshotTargetsNeedingDelivery() []ruleSetSnapshotTarget { // Operations add up across rules: when two rules for the same target resolve to // the same GVR, the entry's operation set is their union (see // rulestore TestGetMatchingRules_OverlappingRulesUnionOperations). A target with -// no resolvable rules has an empty plan and is omitted entirely. +// rules that currently resolve to nothing is kept as an empty plan so transient +// discovery gaps do not look like rule removal. func (m *Manager) currentRuleSetSnapshots() []ruleSetSnapshotTarget { plans := make(map[string]*targetWatchPlan) resolver := m.ruleGVRResolver() @@ -1277,12 +1290,10 @@ func (m *Manager) currentRuleSetSnapshots() []ruleSetSnapshotTarget { targets := make([]ruleSetSnapshotTarget, 0, len(keys)) for _, key := range keys { p := plans[key] - if len(p.entries) == 0 { - continue - } targets = append(targets, ruleSetSnapshotTarget{ - gitDest: p.gitDest, - hash: p.hash(), + gitDest: p.gitDest, + hash: p.hash(), + hasEntries: len(p.entries) > 0, }) } return targets @@ -1330,7 +1341,11 @@ func (m *Manager) MaybeReplaySnapshot(ctx context.Context, gitDest types.Resourc target := ruleSetSnapshotTarget{gitDest: gitDest, hash: hash} log := m.Log.WithName("reconcile") m.EventRouter.BeginReconciliationForStream(gitDest) - m.emitSnapshotForRuleChange(ctx, log, []ruleSetSnapshotTarget{target}, "startup_replay") + if err := m.emitSnapshotForRuleChange(ctx, log, []ruleSetSnapshotTarget{target}, "startup_replay"); err != nil { + // The target stays pending (not marked delivered), so the next reconcile + // or replay retries it; just record why this attempt did not land. + log.Error(err, "snapshot replay did not complete, leaving target pending", "gitDest", gitDest.String()) + } m.EventRouter.CompleteReconciliationForStream(gitDest) } @@ -1347,21 +1362,31 @@ func (m *Manager) beginReconciliationForTargets(targets []ruleSetSnapshotTarget, // emitSnapshotForRuleChange emits fresh repo and cluster state requests for every affected // GitTarget so FolderReconciler diffs against current repository contents rather than a // stale cached repo snapshot from an earlier reconcile. +// +// A target is marked delivered and counted only once *both* its repo- and +// cluster-state requests land on the worker queue: a partial emission would have +// the reconciler diff against an incomplete cluster view. A transient failure on +// either request leaves that target pending and is returned as an error so the +// caller requeues with backoff and retries it promptly, rather than waiting out +// the 30s periodic reconcile — which matters for the per-pod restart gate that +// blocks on the reconcile counter reaching the new pod. Other targets in the +// batch are still attempted so one bad target cannot starve the rest. func (m *Manager) emitSnapshotForRuleChange( ctx context.Context, log logr.Logger, targets []ruleSetSnapshotTarget, trigger string, -) { +) error { if m.EventRouter == nil { log.Info("EventRouter not set, skipping snapshot emission") if len(targets) > 0 { m.snapshotEmitCount.Add(1) } - return + return nil } log.Info("Emitting fresh repo and cluster state for affected GitTargets after rule change", "count", len(targets)) emitted := false + var errs []error for _, target := range targets { gitDest := target.gitDest if m.EventRouter.ReconcilerManager == nil { @@ -1379,6 +1404,7 @@ func (m *Manager) emitSnapshotForRuleChange( GitDest: gitDest, }); err != nil { log.Error(err, "failed to emit RequestRepoState for rule change", "gitDest", gitDest) + errs = append(errs, fmt.Errorf("emit RequestRepoState for %s: %w", gitDest, err)) continue } if err := m.EventRouter.ProcessControlEvent(ctx, events.ControlEvent{ @@ -1386,6 +1412,7 @@ func (m *Manager) emitSnapshotForRuleChange( GitDest: gitDest, }); err != nil { log.Error(err, "failed to emit RequestClusterState for rule change", "gitDest", gitDest) + errs = append(errs, fmt.Errorf("emit RequestClusterState for %s: %w", gitDest, err)) continue } m.markRuleSetSnapshotDelivered(target) @@ -1395,6 +1422,7 @@ func (m *Manager) emitSnapshotForRuleChange( if emitted { m.snapshotEmitCount.Add(1) } + return errors.Join(errs...) } // recordTargetReconcileCompleted increments the per-GitTarget reconcile counter diff --git a/internal/watch/rule_change_snapshot_test.go b/internal/watch/rule_change_snapshot_test.go index 0fc33bd87..d97876fe9 100644 --- a/internal/watch/rule_change_snapshot_test.go +++ b/internal/watch/rule_change_snapshot_test.go @@ -25,10 +25,9 @@ import ( "github.com/go-logr/logr" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" - corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" - dynamicfake "k8s.io/client-go/dynamic/fake" + k8stypes "k8s.io/apimachinery/pkg/types" clientgoscheme "k8s.io/client-go/kubernetes/scheme" "sigs.k8s.io/controller-runtime/pkg/client/fake" @@ -36,6 +35,7 @@ import ( "github.com/ConfigButler/gitops-reverser/internal/git" "github.com/ConfigButler/gitops-reverser/internal/reconcile" "github.com/ConfigButler/gitops-reverser/internal/rulestore" + "github.com/ConfigButler/gitops-reverser/internal/telemetry" "github.com/ConfigButler/gitops-reverser/internal/types" ) @@ -120,9 +120,9 @@ func clusterRuleForResource(name, gitTargetName, resource string) configv1alpha1 } } -// configMapRuleForTarget builds a configmaps ClusterWatchRule for the target. -func configMapRuleForTarget(name, gitTargetName string) configv1alpha1.ClusterWatchRule { - return clusterRuleForResource(name, gitTargetName, "configmaps") +// configMapRule builds the standard single-target configmaps ClusterWatchRule. +func configMapRule() configv1alpha1.ClusterWatchRule { + return clusterRuleForResource("rule-1", "test-target", "configmaps") } // TestReconcileForRuleChange_TargetGainsAlreadyWatchedGVR_Snapshots is the @@ -175,7 +175,7 @@ func TestReconcileForRuleChange_NarrowingOperationsOnExistingRule_Snapshots(t *t ctx := context.Background() // Initial rule: all operations on configmaps. - initial := configMapRuleForTarget("rule-1", "test-target") + initial := configMapRule() store.AddOrUpdateClusterWatchRule( initial, "test-target", "test-ns", @@ -188,7 +188,7 @@ func TestReconcileForRuleChange_NarrowingOperationsOnExistingRule_Snapshots(t *t // Narrow operations to UPDATE only. The GVR (core/v1/configmaps) is // unchanged; only the operation set in the plan changes. - narrowed := configMapRuleForTarget("rule-1", "test-target") + narrowed := configMapRule() narrowed.Spec.Rules[0].Operations = []configv1alpha1.OperationType{ configv1alpha1.OperationUpdate, } @@ -266,29 +266,11 @@ func TestCurrentRuleSetSnapshots_NamespacedWatchRulePlanByNamespace(t *testing.T "a redundant duplicate WatchRule in an already-watched namespace must not change the plan hash") } -// TestReconcileForRuleChange_RestartLikeBootstrap_NoSnapshotDrops models the -// "restart with already-Ready GitTarget" scenario. The user-visible symptom: -// -// - On controller restart, GitTarget.Status.SnapshotSynced is already True -// from the prior run, so GitTargetReconciler takes the early-return at -// gittarget_controller.go:384 and creates no FolderReconciler. -// - WatchManager.Start runs its initial ReconcileForRuleChange before the -// GitTargetReconciler reaches the target, so the snapshot is emitted -// while no reconciler is registered. RouteClusterStateEvent / RouteRepoStateEvent -// find no receiver and silently drop the snapshot output. -// - Live update events still flow through (informer is running, the eventual -// stream registration brings it to LiveProcessing), so the user sees -// "updates write, but pre-existing resources never land in git." -// -// This test guards the fix: it emits a snapshot with no pre-registered -// FolderReconciler and asserts the drop counter stays at zero, so snapshot -// emission stays coordinated with reconciler registration. -// -// The setup populates activeInformers with a stale GVR and uses a catalog that -// does not serve configmaps, so the rule's GVR resolves to nothing and -// compareGVRs reports added=[], removed=[stale-gvr]. That bypasses the -// early-return without requiring any real informers to start. -func TestReconcileForRuleChange_RestartLikeBootstrap_NoSnapshotDrops(t *testing.T) { +// TestReconcileForRuleChange_NoReconcilerLeavesSnapshotPending verifies that +// rule-change reconciliation does not emit state events until a FolderReconciler +// exists. The target remains pending so MaybeReplaySnapshot can retry when the +// GitTarget reconciler registers the receiver. +func TestReconcileForRuleChange_NoReconcilerLeavesSnapshotPending(t *testing.T) { scheme := runtime.NewScheme() require.NoError(t, clientgoscheme.AddToScheme(scheme)) require.NoError(t, configv1alpha1.AddToScheme(scheme)) @@ -300,16 +282,6 @@ func TestReconcileForRuleChange_RestartLikeBootstrap_NoSnapshotDrops(t *testing. Branch: "main", Path: "test-path", }, - Status: configv1alpha1.GitTargetStatus{ - Conditions: []metav1.Condition{ - // Simulate the post-restart state that triggers the - // gittarget_controller.go:384 early-return. - {Type: "SnapshotSynced", Status: metav1.ConditionTrue, Reason: "Completed"}, - }, - }, - } - preexistingCM := &corev1.ConfigMap{ - ObjectMeta: metav1.ObjectMeta{Name: "preexisting", Namespace: "test-ns"}, } fakeK8s := fake.NewClientBuilder(). @@ -319,7 +291,7 @@ func TestReconcileForRuleChange_RestartLikeBootstrap_NoSnapshotDrops(t *testing. store := rulestore.NewStore() store.AddOrUpdateClusterWatchRule( - configMapRuleForTarget("rule-1", "test-target"), + configMapRule(), "test-target", "test-ns", "test-provider", "test-ns", "main", "test-path", @@ -330,15 +302,9 @@ func TestReconcileForRuleChange_RestartLikeBootstrap_NoSnapshotDrops(t *testing. Scope: configv1alpha1.ResourceScopeNamespaced, } manager := &Manager{ - Client: fakeK8s, - Log: logr.Discard(), - RuleStore: store, - dynamicClient: dynamicfake.NewSimpleDynamicClient(scheme, preexistingCM), - // The common catalog serves configmaps, so the rule resolves to a real - // GVR and the target's effective watch plan is non-empty (the target is - // selected for a snapshot). configmaps is pre-activated so no informer is - // started; the stale GVR is removed so compareGVRs avoids the - // no-change early-return. + Client: fakeK8s, + Log: logr.Discard(), + RuleStore: store, resourceCatalog: newCommonTestCatalog(t), discoveryClient: commonTestDiscoveryClient(), activeInformers: map[GVR]map[string]context.CancelFunc{ @@ -347,9 +313,8 @@ func TestReconcileForRuleChange_RestartLikeBootstrap_NoSnapshotDrops(t *testing. }, } - // Wire EventRouter with empty-but-non-nil sub-components. This is the - // state immediately after WatchManager.Start: workers and reconcilers - // have not yet been registered by their respective controllers. + // Wire EventRouter with no registered FolderReconciler. This is the state + // before the GitTarget controller has created the per-target receiver. manager.EventRouter = &EventRouter{ WorkerManager: git.NewWorkerManager(fakeK8s, logr.Discard(), 0, types.SensitiveResourcePolicy{}), ReconcilerManager: reconcile.NewReconcilerManager(nil, logr.Discard()), @@ -361,11 +326,10 @@ func TestReconcileForRuleChange_RestartLikeBootstrap_NoSnapshotDrops(t *testing. require.NoError(t, manager.ReconcileForRuleChange(ctx())) - assert.Zero(t, - manager.EventRouter.SnapshotDeliveryDrops(), - "on a restart-like bootstrap (no FolderReconciler registered yet), the snapshot is emitted "+ - "but RouteClusterStateEvent/RouteRepoStateEvent silently drop it; the system must coordinate "+ - "emission with reconciler registration, or retry, so pre-existing matching resources land in git") + assert.True(t, targetPending(manager, "test-target"), + "target must stay pending until a FolderReconciler exists") + assert.Zero(t, manager.EventRouter.SnapshotDeliveryDrops(), + "state events must not be emitted before a FolderReconciler exists") } // makeTwoTargetRuleChangeManager builds a Manager with two GitTargets, @@ -457,6 +421,19 @@ func targetPending(m *Manager, name string) bool { return ok } +func deliveredHash(m *Manager, name string) (uint64, bool) { + key := types.NewResourceReference(name, "test-ns").Key() + m.ruleSetSnapshotMu.Lock() + defer m.ruleSetSnapshotMu.Unlock() + hash, ok := m.lastDeliveredRuleSetHash[key] + return hash, ok +} + +func targetDelivered(m *Manager, name string) bool { + _, ok := deliveredHash(m, name) + return ok +} + // TestReconcileForRuleChange_UnrelatedTargetNotSnapshotted_OnGlobalGVRChurn is // the GitTarget-isolation invariant from // docs/design/gittarget-isolation-on-rule-change.md. @@ -572,10 +549,115 @@ func TestSnapshotTargetsNeedingDelivery_PerTargetHashIsolatesTargets(t *testing. "A (whose effective plan is unchanged) is correctly skipped") } +func TestSnapshotTargets_DiscoveryUnavailableKeepsDeliveredHash(t *testing.T) { + manager, store := makeRuleChangeTestManager(t) + + store.AddOrUpdateClusterWatchRule( + configMapRule(), + "test-target", "test-ns", "test-provider", "test-ns", "main", "test-path", + ) + seedDeliveredBaseline(manager) + baseline, ok := deliveredHash(manager, "test-target") + require.True(t, ok, "expected delivered baseline") + + manager.resourceCatalog = NewAPIResourceCatalog() + selected := manager.snapshotTargetsNeedingDelivery() + + assert.Empty(t, selected, "an unavailable catalog produces an empty plan, not a snapshot") + current, ok := deliveredHash(manager, "test-target") + require.True(t, ok, "empty plan must not prune delivered state while rules still exist") + assert.Equal(t, baseline, current) + assert.False(t, targetPending(manager, "test-target"), "empty plans are not pending snapshots") +} + +func TestSnapshotTargets_DiscoveryRecoveryDoesNotResnapshotUnchangedPlan(t *testing.T) { + manager, store := makeRuleChangeTestManager(t) + + store.AddOrUpdateClusterWatchRule( + configMapRule(), + "test-target", "test-ns", "test-provider", "test-ns", "main", "test-path", + ) + seedDeliveredBaseline(manager) + + manager.resourceCatalog = NewAPIResourceCatalog() + require.Empty(t, manager.snapshotTargetsNeedingDelivery()) + + manager.resourceCatalog = newCommonTestCatalog(t) + assert.Empty(t, manager.snapshotTargetsNeedingDelivery(), + "the recovered plan matches the remembered delivered hash, so no resnapshot is needed") +} + +func TestSnapshotTargets_RuleRemovalPrunesDeliveredHash(t *testing.T) { + manager, store := makeRuleChangeTestManager(t) + + store.AddOrUpdateClusterWatchRule( + configMapRule(), + "test-target", "test-ns", "test-provider", "test-ns", "main", "test-path", + ) + seedDeliveredBaseline(manager) + require.True(t, targetDelivered(manager, "test-target"), "expected delivered baseline") + + store.DeleteClusterWatchRule(k8stypes.NamespacedName{Name: "rule-1"}) + assert.Empty(t, manager.snapshotTargetsNeedingDelivery()) + assert.False(t, targetDelivered(manager, "test-target"), + "when the target truly has no rules, delivered state is pruned") +} + +// A transient emit failure (here: RequestRepoState fails because no worker is +// registered) must NOT mark the target delivered or bump the reconcile counter, +// and must be returned to the caller so it requeues and retries promptly. Before +// the fix the error was swallowed (counter and delivery were skipped silently), +// so the per-pod restart gate could wait out its 90s timeout on a one-off error. +func TestEmitSnapshotForRuleChange_TransientFailureReturnsErrorAndStaysPending(t *testing.T) { + reader, err := telemetry.InitTestExporter() + require.NoError(t, err) + + scheme := runtime.NewScheme() + require.NoError(t, clientgoscheme.AddToScheme(scheme)) + require.NoError(t, configv1alpha1.AddToScheme(scheme)) + + gitTarget := &configv1alpha1.GitTarget{ + ObjectMeta: metav1.ObjectMeta{Name: "test-target", Namespace: "test-ns"}, + Spec: configv1alpha1.GitTargetSpec{ + ProviderRef: configv1alpha1.GitProviderReference{Name: "test-provider"}, + Branch: "main", + Path: "test-path", + }, + } + fakeK8s := fake.NewClientBuilder().WithScheme(scheme).WithObjects(gitTarget).Build() + + manager := &Manager{Client: fakeK8s, Log: logr.Discard()} + gitDest := types.NewResourceReference("test-target", "test-ns") + + // A FolderReconciler exists (so emission is attempted), but no BranchWorker is + // registered, so RequestRepoState fails with "no worker". + reconcilerManager := reconcile.NewReconcilerManager(nil, logr.Discard()) + reconcilerManager.CreateReconciler(ctx(), gitDest, nil) + manager.EventRouter = &EventRouter{ + WorkerManager: git.NewWorkerManager(fakeK8s, logr.Discard(), 0, types.SensitiveResourcePolicy{}), + ReconcilerManager: reconcilerManager, + WatchManager: manager, + Client: fakeK8s, + Log: logr.Discard(), + gitTargetStreams: map[string]*reconcile.GitTargetEventStream{}, + } + + target := ruleSetSnapshotTarget{gitDest: gitDest, hash: 0xABCD} + emitErr := manager.emitSnapshotForRuleChange(ctx(), logr.Discard(), []ruleSetSnapshotTarget{target}, "rule_change") + + require.Error(t, emitErr, "a failed emission must be returned so the caller requeues") + + // The target was not marked delivered, so the next reconcile retries it. + manager.ruleSetSnapshotMu.Lock() + delivered, ok := manager.lastDeliveredRuleSetHash[gitDest.Key()] + manager.ruleSetSnapshotMu.Unlock() + assert.False(t, ok && delivered == target.hash, "a failed target must stay pending, not be marked delivered") + + // The reconcile counter did not fire for this failed pass. + _, counted := telemetry.CollectInt64Sum(reader, targetReconcileCompletedMetric, + map[string]string{"gittarget_namespace": "test-ns", "gittarget_name": "test-target", "trigger": "rule_change"}) + assert.False(t, counted, "the reconcile counter must not fire when emission failed") +} + // ctx returns a background context. Wrapped for terse use in test setup. func ctx() context.Context { return context.Background() } - -// Compile-time sanity: corev1 is used by tests that may extend this file with -// pre-populated ConfigMap objects to assert backfill content. Today only the -// snapshot-emit counter is asserted; keep the import for cheap follow-up tests. -var _ = corev1.ConfigMap{} From 63b8b95b5283a20531ff2c0f613b998aaab92987 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Tue, 2 Jun 2026 09:17:46 +0000 Subject: [PATCH 31/40] chore: removing allure from ci path, and reworking small findings --- .github/workflows/ci.yml | 85 ++------ docs/README.md | 3 + docs/ci/e2e-allure-reporting.md | 38 +++- docs/ci/findings.md | 24 ++- docs/design/initial-seed-via-watch-list.md | 4 +- docs/design/refactors-branch-review.md | 184 +++++++++++++----- docs/{ => serious-bug}/cozystack-bugreport.md | 0 test/e2e/controller_basics_e2e_test.go | 50 +---- test/e2e/crd_lifecycle_e2e_test.go | 48 +---- test/e2e/e2e_test.go | 8 - test/e2e/gitprovider_validation_e2e_test.go | 2 +- test/e2e/gittarget_isolation_e2e_test.go | 20 +- test/e2e/helpers.go | 32 +++ .../watchrule_configmap_secret_e2e_test.go | 61 +----- 14 files changed, 253 insertions(+), 306 deletions(-) rename docs/{ => serious-bug}/cozystack-bugreport.md (100%) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8422c32c0..1f2c48dad 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -406,28 +406,30 @@ jobs: bash -c ' set -euo pipefail mkdir -p .stamps + reports="$(find .stamps -path "*/ginkgo-report-*.json" -type f | sort)" { echo "### E2E timing reports (${{ matrix.name }})" - reports="$(find .stamps -path "*/ginkgo-report-*.json" -type f | sort)" if [ -z "${reports}" ]; then echo echo "No Ginkgo JSON reports were produced." - exit 0 + else + # Compile the tool once instead of recompiling per report. + go build -o /tmp/spec-timings ./test/e2e/tools/spec-timings + while IFS= read -r report; do + [ -n "${report}" ] || continue + echo + echo "#### ${report}" + echo + echo "```" + /tmp/spec-timings "${report}" || true + echo "```" + done <<<"${reports}" fi - while IFS= read -r report; do - [ -n "${report}" ] || continue - echo - echo "#### ${report}" - echo - echo "```" - go run ./test/e2e/tools/spec-timings "${report}" || true - echo "```" - done <<<"${reports}" } > "${SUMMARY_FILE}" ' cat "${summary_file}" >> "${GITHUB_STEP_SUMMARY}" - - name: Upload E2E timing reports + - name: Upload E2E raw Ginkgo JSON reports if: always() uses: actions/upload-artifact@v7 with: @@ -438,65 +440,6 @@ jobs: if-no-files-found: ignore retention-days: 7 - - name: Generate Allure results - id: allure-results - if: always() - run: | - docker run --rm \ - -v "${GITHUB_WORKSPACE}:${{ env.CI_WORKDIR }}" \ - -w "${{ env.CI_WORKDIR }}" \ - ${{ env.CI_CONTAINER }} \ - bash -c ' - set -euo pipefail - mapfile -t reports < <(find .stamps -path "*/ginkgo-report-*.json" -type f | sort) - if [ "${#reports[@]}" -eq 0 ]; then - echo "No Ginkgo JSON reports were produced." - exit 0 - fi - rm -rf allure-results - mkdir -p allure-results - go run ./test/e2e/tools/ginkgo-allure --output-dir allure-results "${reports[@]}" - ' - if find allure-results -name '*-result.json' -type f | grep -q .; then - echo "has_results=true" >> "${GITHUB_OUTPUT}" - else - echo "has_results=false" >> "${GITHUB_OUTPUT}" - fi - - - name: Prepare Allure history directory - if: always() && steps.allure-results.outputs.has_results == 'true' - run: | - mkdir -p gh-pages allure-history - - - name: Generate Allure report - if: always() && steps.allure-results.outputs.has_results == 'true' - uses: simple-elf/allure-report-action@v1.13 - with: - allure_results: allure-results - allure_report: allure-report - gh_pages: gh-pages - allure_history: allure-history - - - name: Fix Allure artifact permissions - if: always() && steps.allure-results.outputs.has_results == 'true' - run: | - # allure-results and allure-report are produced by root-owned - # containers (docker run / the Docker-based report action), so the - # native upload-artifact step cannot read them. Hand ownership back - # to the runner user before uploading. - sudo chown -R "$(id -u):$(id -g)" allure-results allure-report - - - name: Upload Allure report artifact - if: always() && steps.allure-results.outputs.has_results == 'true' - uses: actions/upload-artifact@v7 - with: - name: e2e-allure-report-${{ matrix.name }} - path: | - allure-report - allure-results - if-no-files-found: ignore - retention-days: 7 - # Release job only runs on push to main after tests pass release-please: name: Release Please diff --git a/docs/README.md b/docs/README.md index 4c8bae344..02b429544 100644 --- a/docs/README.md +++ b/docs/README.md @@ -27,8 +27,11 @@ These directories are useful when changing internals, but they are not the prima docs surface: - [`design/`](design/): architecture notes, investigations, and implementation plans +- [`finished/`](finished/): design docs and reports for work that has shipped or been resolved, + kept for historical context - [`ci/`](ci/): CI/devcontainer rationale and troubleshooting - [`future/`](future/): ideas that are intentionally deferred +- [`serious-bug/`](serious-bug/): worked bug reports against upstream projects - [`audit-setup/`](audit-setup/): cluster-specific audit delivery notes and examples These root-level files are also working notes rather than polished user docs: diff --git a/docs/ci/e2e-allure-reporting.md b/docs/ci/e2e-allure-reporting.md index ccedd5005..532d54da2 100644 --- a/docs/ci/e2e-allure-reporting.md +++ b/docs/ci/e2e-allure-reporting.md @@ -1,22 +1,27 @@ # E2E Allure Reporting -The CI e2e job produces a visual Allure HTML report for each matrix entry. +The CI e2e job uploads raw Ginkgo JSON reports for each matrix entry. Developers +can download those artifacts and render a visual Allure HTML report locally when +the timeline view is useful. The e2e tasks still run through the repository Taskfile so local and CI execution stay aligned. Each task writes Ginkgo JSON reports under `.stamps/cluster///ginkgo-report-.json`. After the e2e -run, CI converts those Ginkgo JSON files into Allure result files with: +run, CI uploads those files, plus the timing summary, as the +`e2e-ginkgo-reports-` artifact. + +The raw Ginkgo JSON files are the source of truth for local tooling and timing +analysis. To render an Allure report from downloaded or locally produced Ginkgo +JSON files, use: ```bash go run ./test/e2e/tools/ginkgo-allure --output-dir allure-results ... ``` -Then `simple-elf/allure-report-action` renders `allure-report`, and CI uploads -the generated HTML as the `e2e-allure-report-` artifact. - -This is intentionally a post-run report. During an active e2e run, GitHub Actions -logs remain the live view. After the run finishes, the Allure artifact provides a -visual timeline, status breakdown, labels, durations, and captured Ginkgo output. +During an active e2e run, GitHub Actions logs remain the live view. After the run +finishes, the downloaded Ginkgo JSON can be converted locally into an Allure +report with a visual timeline, status breakdown, labels, durations, and captured +Ginkgo output. The converter is deliberately small and one-way: @@ -27,8 +32,21 @@ The converter is deliberately small and one-way: - Captured `GinkgoWriter` and parallel stdout/stderr output are attached as plain-text test artifacts. -The existing raw Ginkgo JSON and timing-summary artifacts are still uploaded. -Those files remain the source of truth for local tooling and timing analysis. +## Allure labels + +The converter attaches labels so the report's grouping and timeline views are +useful for a parallel suite: + +- `thread` — `ginkgo-process-`, the Ginkgo parallel process that ran the + spec. This is what gives the Allure timeline its per-process lanes, so you can + see how specs were distributed across processes and spot a single overloaded + lane. Only emitted for parallel runs (`ParallelProcess > 0`). +- `suite` — the Ginkgo suite description. +- `package` — the base name of the suite's source path. +- `parentSuite` / `subSuite` — the spec's container hierarchy (outermost + `Describe` as `parentSuite`, the remaining nested containers as `subSuite`). +- `tag` — one per Ginkgo `Label(...)`, plus a `Serial` tag for specs that ran + serially. ## Local devcontainer use diff --git a/docs/ci/findings.md b/docs/ci/findings.md index 611b53923..93d571330 100644 --- a/docs/ci/findings.md +++ b/docs/ci/findings.md @@ -1,10 +1,10 @@ ## CI/Devcontainer Findings (Current Baseline) -Last updated: 2026-02-13 +Last updated: 2026-06-02 Recent investigation: [E2E full-suite shared-state flake](../design/e2e-full-suite-shared-state-investigation.md) -This folder documents why the repository uses its current devcontainer and CI behavior, especially around Go caches, workspace paths, and Kind access from inside the container. +This folder documents why the repository uses its current devcontainer and CI behavior, especially around Go caches, workspace paths, and k3d cluster access from inside the container. ### 1) Workspace path model @@ -46,7 +46,7 @@ Why: - Stable module/build caching independent of repo bind mount - Fewer permission regressions than putting caches in the workspace tree -### 4) Kind + kubectl access model inside devcontainer +### 4) k3d + kubectl access model inside devcontainer The current working model is: @@ -54,27 +54,25 @@ The current working model is: - Devcontainer run args include: - `--group-add=docker` - `--add-host=host.docker.internal:host-gateway` -- Kind cluster config sets: - - `networking.apiServerAddress: "0.0.0.0"` -- `test/e2e/cluster/start-cluster.sh` rewrites kubeconfig server endpoints from - `127.0.0.1|localhost|0.0.0.0` to `host.docker.internal:` and sets - `tls-server-name=localhost` +- `test/e2e/cluster/start-cluster.sh` lets k3d pick the API server port, then + rewrites kubeconfig server endpoints from `127.0.0.1|localhost|0.0.0.0` to + `host.docker.internal:` and sets `tls-server-name=localhost` Why this is required: -- If Docker publishes Kind API server on host loopback only (`127.0.0.1`), it is not reachable via `host.docker.internal` from the container. -- Binding on `0.0.0.0` plus kubeconfig rewrite makes in-container `kubectl` stable without host networking. +- If Docker publishes the k3d API server on host loopback only (`127.0.0.1`), it is not reachable via `host.docker.internal` from the container. +- The kubeconfig rewrite to `host.docker.internal` makes in-container `kubectl` stable without host networking. ### 5) Practical verification checklist After devcontainer rebuild/reopen: ```bash -# 1) Kind setup -make setup-cluster +# 1) Cluster setup (creates the k3d cluster via the e2e Taskfile) +bash test/e2e/cluster/start-cluster.sh # 2) Confirm API publish bind (expected 0.0.0.0 or ::) -docker inspect gitops-reverser-test-e2e-control-plane --format '{{json .NetworkSettings.Ports}}' +docker inspect k3d-gitops-reverser-test-e2e-server-0 --format '{{json .NetworkSettings.Ports}}' # 3) Confirm kubeconfig server rewrite kubectl config view --minify | sed -n '/server:/p;/tls-server-name:/p' diff --git a/docs/design/initial-seed-via-watch-list.md b/docs/design/initial-seed-via-watch-list.md index 92bb5cd5c..7f82fcae0 100644 --- a/docs/design/initial-seed-via-watch-list.md +++ b/docs/design/initial-seed-via-watch-list.md @@ -4,7 +4,7 @@ > Scope: `internal/watch` snapshot path, dynamic informer construction, the > client-go `WatchListClient` feature gate > Related: [`docs/upgrade-finding.md`](../upgrade-finding.md), -> [`docs/cozystack-bugreport.md`](../cozystack-bugreport.md) +> [`docs/serious-bug/cozystack-bugreport.md`](../serious-bug/cozystack-bugreport.md) ## Goal @@ -171,7 +171,7 @@ Standard kube-apiserver and resources backed by `genericregistry.Store` get this for free. Hand-written `rest.Storage` implementations in aggregated servers — exactly the kind that show up in projects like Cozystack — must implement it themselves. The Cozystack bug report -([`docs/cozystack-bugreport.md`](../cozystack-bugreport.md)) is a worked +([`docs/serious-bug/cozystack-bugreport.md`](../serious-bug/cozystack-bugreport.md)) is a worked example: one API group conformant, one not, in the same binary. Before relying on Phase B in any production cluster, we need to audit every diff --git a/docs/design/refactors-branch-review.md b/docs/design/refactors-branch-review.md index bf1fdb6a2..a25a35d53 100644 --- a/docs/design/refactors-branch-review.md +++ b/docs/design/refactors-branch-review.md @@ -1,7 +1,7 @@ # `refactors` branch — review & action checklist -> Review of `main...refactors` (57 files, ~5k added / ~2k deleted) plus the -> uncommitted Allure/Taskfile working-tree changes. Generated 2026-06-01. +> Review of `main...refactors` (57 files, ~7.5k added / ~2.1k deleted). +> Generated 2026-06-01 from a clean working tree. > Covers: (1) code-review findings, (2) docs triage, (3) refactor advice on the > two e2e docs and the stability-enabled follow-ups. @@ -10,13 +10,42 @@ clean: all 23 Manager specs are re-created, assertion counts match, and CRD/namespace cleanup was correctly redistributed. Findings concentrate in the new Phase-3 drain metrics and the new e2e helpers. +## Merge readiness + +### Block before merge + +- [x] Fix the queue-depth metric publication race (finding 1). +- [x] Fix discovery-lag / empty-plan handling for rule-set snapshots (finding 2). +- [x] Simplify CI Allure handling: keep uploading raw Ginkgo JSON and timing + summaries, but remove CI-side Allure result conversion, HTML rendering, + permission fixups, and Allure artifact upload from `.github/workflows/ci.yml`. + Local report generation via `task allure-e2e-report` is enough when a human + needs the visual timeline. + +### Good enough for follow-up + +- [x] Cleanup findings 5-9 (done); finding 10 CI half done, two Go micro-opts + deferred with rationale (see finding 10). +- [x] Docs archiving/relocation in section 2. +- [x] Local Allure docs polish in section 3. + +### Validation already run + +- [x] `git diff --check origin/main...HEAD` +- [x] `task lint-actions` +- [x] `go test ./test/e2e/tools/ginkgo-allure ./test/e2e/tools/spec-timings ./internal/git ./internal/watch` +- [x] `go test -run '^$' ./test/e2e` +- [x] `task lint` +- [x] `task test` +- [x] `task test-e2e` + --- ## 1. Code review ### Correctness (ranked) -- [ ] **1. Stale queue-depth gauge race — can hang the restart drain gate** +- [x] **1. Stale queue-depth gauge race — can hang the restart drain gate** — [branch_worker.go:315-331](../../internal/git/branch_worker.go#L315-L331). `recordQueueDepth()` runs from enqueue goroutines *and* the event loop; it does `Load(inflightItems)` → … → `Record()` non-atomically. An enqueue @@ -28,7 +57,7 @@ new Phase-3 drain metrics and the new e2e helpers. loop goroutine (drop `recordQueueDepth()` from the enqueue paths; the loop already republishes each iteration via `syncQueueDepthMetric`). -- [ ] **2. Discovery-lag causes spurious full re-snapshots** +- [x] **2. Discovery-lag causes spurious full re-snapshots** — [manager.go:1248](../../internal/watch/manager.go#L1248), [:1264](../../internal/watch/manager.go#L1264), [:1280](../../internal/watch/manager.go#L1280), @@ -42,8 +71,16 @@ new Phase-3 drain metrics and the new e2e helpers. `crd_lifecycle` Serial. **Fix:** log the resolve miss instead of discarding, and don't evict last-delivered state on an *empty* plan (distinguish "no rules" from "rules that didn't resolve yet"). + **Fixed (simple version):** `currentRuleSetSnapshots` now keeps targets that + still have rules even when the resolved entry set is empty, and + `snapshotTargetsNeedingDelivery` skips those empty plans without selecting or + pruning them. This preserves the delivered baseline across transient discovery + gaps. Known accepted tradeoff: `valid -> stable-empty -> valid` user edits do + not force a recovery snapshot until the richer settled-empty refinement lands. + Detailed proposal: + [rule-set-snapshot-discovery-lag-fix.md](rule-set-snapshot-discovery-lag-fix.md). -- [ ] **3. Reconcile counter skipped on transient error / fired on no-op** +- [x] **3. Reconcile counter skipped on transient error / fired on no-op** — [manager.go:1384-1392](../../internal/watch/manager.go#L1384-L1392). `recordTargetReconcileCompleted` fires only after **both** `RequestRepoState` and `RequestClusterState` succeed. A transient error on the second `continue`s @@ -51,8 +88,19 @@ new Phase-3 drain metrics and the new e2e helpers. `{pod=…} > 0` gate can time out at 90s on a one-off error. It also fires on a no-op snapshot, so `increase(target_reconcile_completed_total)` over-counts for non-test consumers. + **Fixed (transient-error half):** `emitSnapshotForRuleChange` now returns a + joined error for any target whose repo/cluster-state request failed; the target + is left pending (not marked delivered, counter not fired) and + `ReconcileForRuleChange` propagates the error so the controller requeues with + backoff and retries promptly instead of waiting out the 30s periodic tick. + The "fires on a no-op snapshot" half is **left intentionally**: the counter's + documented contract is to count snapshot passes that reached the git write path + (not commits produced), and a clean post-restart re-snapshot legitimately + produces an empty diff yet must still increment so the restart gate can observe + `{pod=} > 0`. Gating it on a non-empty diff would break that load-bearing + signal. -- [ ] **4. Shutdown leaves the depth gauge non-zero** +- [x] **4. Shutdown leaves the depth gauge non-zero** — [branch_worker.go:585](../../internal/git/branch_worker.go#L585), [:704](../../internal/git/branch_worker.go#L704). On `ctx.Done()` the loop finalizes + pushes and returns **without draining @@ -63,34 +111,55 @@ new Phase-3 drain metrics and the new e2e helpers. ### Cleanup / efficiency -- [ ] **5. ~13 inlined `git pull` blocks duplicate `pullLatestRepoState`** +- [x] **5. ~13 inlined `git pull` blocks duplicate `pullLatestRepoState`** — [watchrule_configmap_secret_e2e_test.go:174](../../test/e2e/watchrule_configmap_secret_e2e_test.go#L174) (+~6 more) and ~7 in `crd_lifecycle_e2e_test.go`. The canonical helper (fetch + checkout -B + reset --hard) is already used at [:707](../../test/e2e/watchrule_configmap_secret_e2e_test.go#L707) and is more robust than a bare `git pull` (which can flake on a diverged branch). -- [ ] **6. `gitPull` / `lastCommitMessageForPath` reinvent existing helpers** + **Fixed:** all 6 blocks in `watchrule_configmap_secret_e2e_test.go` and all 10 + in `crd_lifecycle_e2e_test.go` (both the error-checking and ignore-error + forms) now call `pullLatestRepoState(g, …)`. +- [x] **6. `gitPull` / `lastCommitMessageForPath` reinvent existing helpers** — [gittarget_isolation_e2e_test.go:187](../../test/e2e/gittarget_isolation_e2e_test.go#L187). Third pull implementation (no fetch/reset) + manual `git log` where `gitRun(dir, args...)` already exists in helpers.go. -- [ ] **7. Hand-rolled `minInt`** — [e2e_test.go:264](../../test/e2e/e2e_test.go#L264); - used once, Go 1.26 builtin `min` covers it. -- [ ] **8. Redundant "healthy GitProvider" spec** + **Fixed:** call site uses `pullLatestRepoState`; the local `gitPull` is removed + and `lastCommitMessageForPath` goes through `gitRun`. Unused `os/exec` import + dropped. +- [x] **7. Hand-rolled `minInt`** — [e2e_test.go:264](../../test/e2e/e2e_test.go#L264); + used once, Go 1.26 builtin `min` covers it. **Fixed:** `minInt` deleted, sole + caller uses builtin `min`. +- [x] **8. Redundant "healthy GitProvider" spec** — [watchrule_configmap_secret_e2e_test.go:80](../../test/e2e/watchrule_configmap_secret_e2e_test.go#L80) re-creates/re-verifies `gitprovider-normal` already built in `BeforeAll` ([:60-70](../../test/e2e/watchrule_configmap_secret_e2e_test.go#L60-L70)). -- [ ] **9. Duplicated endpoint→pod filtering** + **Fixed:** the spec now only re-asserts the BeforeAll-created provider stays + Ready; the redundant re-creation is gone. +- [x] **9. Duplicated endpoint→pod filtering** — [controller_basics_e2e_test.go:99](../../test/e2e/controller_basics_e2e_test.go#L99) vs [:135](../../test/e2e/controller_basics_e2e_test.go#L135); extract - `expectServiceRoutesToPod`. -- [ ] **10. Wasted work** — CI "Summarize E2E timing reports" `go run`s + `expectServiceRoutesToPod`. **Fixed:** both blocks call the new + `expectServiceRoutesToPod(g, service, pod)` helper in `helpers.go`; the + now-unused `strings` import is dropped. +- [x] **10. Wasted work** — CI "Summarize E2E timing reports" `go run`s spec-timings per report in a loop ([ci.yml:388](../../.github/workflows/ci.yml#L388)), recompiling N+1× (`go build -o` once instead); `currentRuleSetSnapshots` resolves GVRs twice per reconcile ([manager.go:1224](../../internal/watch/manager.go#L1224)); `syncQueueDepthMetric` records every loop iteration rather than on change. + **Fixed (CI half):** the summary step now `go build -o /tmp/spec-timings` once + and runs the binary per report instead of `go run` recompiling each time. + **Deferred intentionally (the two Go halves):** `syncQueueDepthMetric`'s + per-iteration `Record` is the load-bearing republish that finding #1's fix + relies on to keep the OTel gauge from latching a stale depth — switching it to + record-on-change would reintroduce that race. The "resolves GVRs twice" cost + spans two *different* methods (plan-hash in `currentRuleSetSnapshots` vs. + watch/list setup in the snapshot builder), so collapsing it needs a + per-reconcile resolution cache rather than a local tidy-up; left for a focused + follow-up. -> Uncommitted working tree (Allure env-var overrides + `specSkippedByFilter` + -> per-process thread label) looks good. Caveat: `specSkippedByFilter` +> The Allure env-var overrides + `specSkippedByFilter` + per-process thread +> label look good. Caveat: `specSkippedByFilter` > ([ginkgo-allure/main.go:263](../../test/e2e/tools/ginkgo-allure/main.go#L263)) > infers "filtered-out" from `LabelFilter!="" && Skipped && EndTime.IsZero() && > RunTime==0`; a real `Skip()` recording zero timing would also be dropped. Low @@ -105,36 +174,50 @@ new Phase-3 drain metrics and the new e2e helpers. ### Move to `docs/finished/` — shipped/resolved -- [ ] [design/gittarget-isolation-on-rule-change.md](gittarget-isolation-on-rule-change.md) — ✅ shipped (this branch) -- [ ] [design/commit-window-refactor.md](commit-window-refactor.md) — implemented -- [ ] [design/e2e-cleanup-scoping.md](e2e-cleanup-scoping.md) — implemented -- [ ] [design/partial-object-audit-event-handling.md](partial-object-audit-event-handling.md) — implemented -- [ ] [design/path-scoped-bootstrap-template-design.md](path-scoped-bootstrap-template-design.md) — implemented -- [ ] [design/remove-gittarget-admission-webhook.md](remove-gittarget-admission-webhook.md) — webhook file gone → done -- [ ] [design/watch-gvr-resolution-cleanup.md](watch-gvr-resolution-cleanup.md) — `APIResourceCatalog` cleanup landed -- [ ] [design/watchrule-gvr-resolution-plan.md](watchrule-gvr-resolution-plan.md) — implemented -- [ ] [design/aggregated-api-requestheader-ca-miswire-report.md](aggregated-api-requestheader-ca-miswire-report.md) — resolved report -- [ ] [design/audit-consumer-username-attribution-gap.md](audit-consumer-username-attribution-gap.md) — resolved -- [ ] [design/e2e-runtime-state-vs-stamps.md](e2e-runtime-state-vs-stamps.md) — resolved failure mode -- [ ] [design/shallow-audit-event-misclassification.md](shallow-audit-event-misclassification.md) — root cause confirmed -- [ ] [design/e2e-test-review.md](e2e-test-review.md) — point-in-time review -- [ ] [git-author.md](../git-author.md) — OIDC author plan, implemented -- [ ] [future/design-commit-request-api.md](../future/design-commit-request-api.md) — implemented -- [ ] [future/design-rule-change-snapshot-trigger.md](../future/design-rule-change-snapshot-trigger.md) — implemented -- [ ] [future/design-audit-ingestion-hardening.md](../future/design-audit-ingestion-hardening.md) — implemented -- [ ] [future/design-commit-context-api.md](../future/design-commit-context-api.md) — superseded -- [ ] [future/idea-audit-enrichment-side-channel.md](../future/idea-audit-enrichment-side-channel.md) — superseded +- [x] [finished/gittarget-isolation-on-rule-change.md](../finished/gittarget-isolation-on-rule-change.md) — ✅ shipped (this branch) +- [x] [finished/commit-window-refactor.md](../finished/commit-window-refactor.md) — implemented +- [x] [finished/e2e-cleanup-scoping.md](../finished/e2e-cleanup-scoping.md) — implemented +- [x] [finished/partial-object-audit-event-handling.md](../finished/partial-object-audit-event-handling.md) — implemented +- [x] [finished/path-scoped-bootstrap-template-design.md](../finished/path-scoped-bootstrap-template-design.md) — implemented +- [x] [finished/remove-gittarget-admission-webhook.md](../finished/remove-gittarget-admission-webhook.md) — webhook file gone → done +- [x] [finished/watch-gvr-resolution-cleanup.md](../finished/watch-gvr-resolution-cleanup.md) — `APIResourceCatalog` cleanup landed +- [x] [finished/watchrule-gvr-resolution-plan.md](../finished/watchrule-gvr-resolution-plan.md) — implemented +- [x] [finished/aggregated-api-requestheader-ca-miswire-report.md](../finished/aggregated-api-requestheader-ca-miswire-report.md) — resolved report +- [x] [finished/audit-consumer-username-attribution-gap.md](../finished/audit-consumer-username-attribution-gap.md) — resolved +- [x] [finished/e2e-runtime-state-vs-stamps.md](../finished/e2e-runtime-state-vs-stamps.md) — resolved failure mode +- [x] [finished/shallow-audit-event-misclassification.md](../finished/shallow-audit-event-misclassification.md) — root cause confirmed +- [x] [finished/e2e-test-review.md](../finished/e2e-test-review.md) — point-in-time review +- [x] [finished/git-author.md](../finished/git-author.md) — OIDC author plan, implemented +- [x] [finished/design-commit-request-api.md](../finished/design-commit-request-api.md) — implemented +- [x] [finished/design-rule-change-snapshot-trigger.md](../finished/design-rule-change-snapshot-trigger.md) — implemented +- [x] [finished/design-audit-ingestion-hardening.md](../finished/design-audit-ingestion-hardening.md) — implemented +- [x] [finished/design-commit-context-api.md](../finished/design-commit-context-api.md) — superseded +- [x] [finished/idea-audit-enrichment-side-channel.md](../finished/idea-audit-enrichment-side-channel.md) — superseded ### Consider removing / relocating -- [ ] [cozystack-bugreport.md](../cozystack-bugreport.md) — upstream bug report against +- [x] [serious-bug/cozystack-bugreport.md](../serious-bug/cozystack-bugreport.md) — upstream bug report against CozyStack, not our docs. If filed upstream, replace with a link. Referenced by `initial-seed-via-watch-list.md`, so update that link; `serious-bug/` is the - safe relocation. -- [ ] [ci/findings.md](../ci/findings.md) — stale (2026-02-13), overlaps + safe relocation. **Done:** moved to + [serious-bug/cozystack-bugreport.md](../serious-bug/cozystack-bugreport.md); + both references in `initial-seed-via-watch-list.md` updated. +- [x] [ci/findings.md](../ci/findings.md) — stale (2026-02-13), overlaps [e2e-ci-stability-report.md](e2e-ci-stability-report.md). Fold true bits in, drop it. -- [ ] Empty `docs/finished/`, `docs/serious-bug/`, `docs/contributing/` — populate - (per above) or remove the empty scaffolding. + **Deviated from the recommendation:** on inspection the two docs do *not* + overlap — `ci/findings.md` is devcontainer/CI infrastructure rationale + (workspace mounts, Go cache volumes, in-container cluster networking), while + `e2e-ci-stability-report.md` is about test stability and the e2e job's disk + pressure. Deleting it would lose useful rationale. It *was* stale on specifics + (referenced Kind + `make setup-cluster`; the project now uses k3d + `task`), so + it was refreshed in place (k3d node naming, `start-cluster.sh` port handling, + updated date) and kept. +- [x] Empty `docs/finished/`, `docs/serious-bug/`, `docs/contributing/` — populate + (per above) or remove the empty scaffolding. **Done:** `finished/` was already + populated by the section-2 moves and is now documented in + [docs/README.md](../README.md); `serious-bug/` is populated by the cozystack + relocation and documented in the README; the empty `docs/contributing/` was + removed (the contributor guide lives at top-level `CONTRIBUTING.md`). ### Keep on the active design surface @@ -155,11 +238,16 @@ explainers + `demo/*` + `audit-setup/*` stay. ## 3. The two e2e docs + stability-enabled refactors -**[e2e-allure-reporting.md](../ci/e2e-allure-reporting.md)** — accurate and -current. Only gap: +**[e2e-allure-reporting.md](../ci/e2e-allure-reporting.md)** — accurate for the +current branch, but should be updated if CI stops rendering Allure: -- [ ] Document the new per-process **`thread` Allure label** the converter now - emits (the point of the parallel-insight work). +- [x] If `.github/workflows/ci.yml` is simplified, rewrite the CI section to say + CI uploads only raw Ginkgo JSON/timing artifacts; developers can download + those reports and run `task allure-e2e-report` locally. +- [x] Document the new per-process **`thread` Allure label** the converter now + emits (the point of the parallel-insight work). **Done:** added an "Allure + labels" section to `e2e-allure-reporting.md` documenting `thread` plus the + `suite`/`package`/`parentSuite`/`subSuite`/`tag` labels. **[e2e-serial-registry.md](e2e-serial-registry.md)** — the Serial entries are **correctness-driven (shared state), not flakiness-driven**, so improved @@ -181,6 +269,8 @@ refactors the registry is implicitly waiting for: (singleton controller) are genuinely serial — leave them. - [ ] **Confidence-window pass (the speedup plan's open thread).** Now that the - race fixes landed (`03b95ab`, `0787b1b`), decide whether CI can move back off - `K3D_AGENT_COUNT=0` / `procs=1` toward parallelism, using the uploaded timing - summaries + Allure reports. + race fixes landed (`03b95ab`, `0787b1b`), decide whether CI should keep + `K3D_AGENT_COUNT=0` and the current per-matrix Ginkgo parallelism, or move + back toward agent-backed clusters. Use the uploaded timing summaries and raw + Ginkgo JSON as the evidence source; render Allure locally only when the visual + report is needed. diff --git a/docs/cozystack-bugreport.md b/docs/serious-bug/cozystack-bugreport.md similarity index 100% rename from docs/cozystack-bugreport.md rename to docs/serious-bug/cozystack-bugreport.md diff --git a/test/e2e/controller_basics_e2e_test.go b/test/e2e/controller_basics_e2e_test.go index 6ac98a377..70ac73a02 100644 --- a/test/e2e/controller_basics_e2e_test.go +++ b/test/e2e/controller_basics_e2e_test.go @@ -20,7 +20,6 @@ package e2e import ( "fmt" - "strings" "time" . "github.com/onsi/ginkgo/v2" @@ -86,32 +85,7 @@ var _ = Describe("Manager Controller Basics", Label("manager"), Ordered, func() By("verifying controller service routes to the controller pod") Eventually(func(g Gomega) { - output, endpointsErr := kubectlRunInNamespace( - namespace, - "get", - "endpoints", - controllerServiceName, - "-o", - "jsonpath={.subsets[*].addresses[*].targetRef.name}", - ) - g.Expect(endpointsErr).NotTo(HaveOccurred(), "Failed to get controller service endpoints") - - lines := utils.GetNonEmptyLines(output) - podSet := map[string]struct{}{} - for _, line := range lines { - if !strings.HasPrefix(line, "Warning:") && - !strings.Contains(line, "deprecated") && - strings.Contains(line, "gitops-reverser") { - podSet[line] = struct{}{} - } - } - var podNames []string - for podName := range podSet { - podNames = append(podNames, podName) - } - - g.Expect(podNames).To(HaveLen(1), "controller service should route to exactly 1 pod") - g.Expect(podNames[0]).To(Equal(controllerPodName), "controller service should route to controller pod") + expectServiceRoutesToPod(g, controllerServiceName, controllerPodName) }, 30*time.Second).Should(Succeed()) }) @@ -122,27 +96,7 @@ var _ = Describe("Manager Controller Basics", Label("manager"), Ordered, func() By("verifying audit service routes to the controller pod") Eventually(func(g Gomega) { - output, endpointsErr := kubectlRunInNamespace( - namespace, - "get", - "endpoints", - auditServiceName, - "-o", - "jsonpath={.subsets[*].addresses[*].targetRef.name}", - ) - g.Expect(endpointsErr).NotTo(HaveOccurred(), "Failed to get audit service endpoints") - - lines := utils.GetNonEmptyLines(output) - var podNames []string - for _, line := range lines { - if strings.HasPrefix(line, "Warning:") || strings.Contains(line, "deprecated") { - continue - } - podNames = append(podNames, line) - } - - g.Expect(podNames).To(HaveLen(1), "audit service should route to exactly 1 pod") - g.Expect(podNames[0]).To(Equal(controllerPodName), "audit service should route to controller pod") + expectServiceRoutesToPod(g, auditServiceName, controllerPodName) }, 30*time.Second).Should(Succeed()) }) diff --git a/test/e2e/crd_lifecycle_e2e_test.go b/test/e2e/crd_lifecycle_e2e_test.go index 5653bc176..777d08c78 100644 --- a/test/e2e/crd_lifecycle_e2e_test.go +++ b/test/e2e/crd_lifecycle_e2e_test.go @@ -140,13 +140,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Serial, Ordered, fun By("verifying CRD YAML file exists in Git repository (NO namespace in path - cluster resource)") verifyGitCommit := func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = crdLifecycleRepo.CheckoutDir - pullOutput, pullErr := pullCmd.CombinedOutput() - if pullErr != nil { - g.Expect(pullErr).NotTo(HaveOccurred(), - fmt.Sprintf("Should successfully pull latest changes. Output: %s", string(pullOutput))) - } + pullLatestRepoState(g, crdLifecycleRepo.CheckoutDir) // CRDs are cluster-scoped, so path should NOT include namespace expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, @@ -296,13 +290,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Serial, Ordered, fun By("verifying CRD instance YAML file exists in Gitea repository") verifyGitCommit := func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = crdLifecycleRepo.CheckoutDir - pullOutput, pullErr := pullCmd.CombinedOutput() - if pullErr != nil { - g.Expect(pullErr).NotTo(HaveOccurred(), - fmt.Sprintf("Should successfully pull latest changes. Output: %s", string(pullOutput))) - } + pullLatestRepoState(g, crdLifecycleRepo.CheckoutDir) expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, "e2e/icecream-test", @@ -383,9 +371,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Serial, Ordered, fun By("verifying no new commit was created and status is not in Git") verifyStatusNotCommitted := func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = crdLifecycleRepo.CheckoutDir - _, _ = pullCmd.CombinedOutput() + pullLatestRepoState(g, crdLifecycleRepo.CheckoutDir) // Check that commit hash hasn't changed gitRevCmd := exec.Command("git", "rev-parse", "HEAD") @@ -463,9 +449,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Serial, Ordered, fun By("waiting for initial CRD instance file to appear in Git") verifyInitialFile := func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = crdLifecycleRepo.CheckoutDir - _, _ = pullCmd.CombinedOutput() + pullLatestRepoState(g, crdLifecycleRepo.CheckoutDir) expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, "e2e/icecream-test", @@ -514,9 +498,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Serial, Ordered, fun By("verifying updated CRD instance content in Git") verifyUpdatedFile := func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = crdLifecycleRepo.CheckoutDir - _, _ = pullCmd.CombinedOutput() + pullLatestRepoState(g, crdLifecycleRepo.CheckoutDir) expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, "e2e/icecream-test", @@ -582,9 +564,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Serial, Ordered, fun By("waiting for CR file to appear in Git repository") verifyFileCreated := func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = crdLifecycleRepo.CheckoutDir - _, _ = pullCmd.CombinedOutput() + pullLatestRepoState(g, crdLifecycleRepo.CheckoutDir) expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, "e2e/icecream-test", @@ -604,9 +584,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Serial, Ordered, fun By("verifying CRD instance file is deleted from Git repository") verifyFileDeleted := func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = crdLifecycleRepo.CheckoutDir - _, _ = pullCmd.CombinedOutput() + pullLatestRepoState(g, crdLifecycleRepo.CheckoutDir) expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, "e2e/icecream-test", @@ -660,9 +638,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Serial, Ordered, fun By("verifying CRD file exists in Git before deletion") verifyFileExists := func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = crdLifecycleRepo.CheckoutDir - _, _ = pullCmd.CombinedOutput() + pullLatestRepoState(g, crdLifecycleRepo.CheckoutDir) expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, "e2e/crd-delete-test", @@ -677,9 +653,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Serial, Ordered, fun Expect(deleteErr).NotTo(HaveOccurred(), "CRD deletion should succeed") By("verifying CRD file is deleted from Git repository") verifyFileDeleted := func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = crdLifecycleRepo.CheckoutDir - _, _ = pullCmd.CombinedOutput() + pullLatestRepoState(g, crdLifecycleRepo.CheckoutDir) expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, "e2e/crd-delete-test", @@ -700,9 +674,7 @@ var _ = Describe("Manager CRD Lifecycle", Label("manager"), Serial, Ordered, fun By("verifying the deleted CRD file does not reappear after terminating updates") Consistently(func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = crdLifecycleRepo.CheckoutDir - _, _ = pullCmd.CombinedOutput() + pullLatestRepoState(g, crdLifecycleRepo.CheckoutDir) expectedFile := filepath.Join(crdLifecycleRepo.CheckoutDir, "e2e/crd-delete-test", diff --git a/test/e2e/e2e_test.go b/test/e2e/e2e_test.go index a28bd9302..141f03142 100644 --- a/test/e2e/e2e_test.go +++ b/test/e2e/e2e_test.go @@ -260,11 +260,3 @@ func showControllerLogs(context string) { fmt.Printf("%s\n", output) fmt.Printf("----------------------------------------\n") } - -// minInt returns the minimum of two integers. -func minInt(a, b int) int { - if a < b { - return a - } - return b -} diff --git a/test/e2e/gitprovider_validation_e2e_test.go b/test/e2e/gitprovider_validation_e2e_test.go index 2431c4635..e9bcdc114 100644 --- a/test/e2e/gitprovider_validation_e2e_test.go +++ b/test/e2e/gitprovider_validation_e2e_test.go @@ -132,7 +132,7 @@ var _ = Describe("Manager GitProvider Validation", Label("manager"), Ordered, fu if err != nil { fmt.Printf("❌ SSH secret not found: %v\n", err) } else { - previewLen := minInt(300, len(secretOutput)) + previewLen := min(300, len(secretOutput)) fmt.Printf( "✅ SSH secret exists - showing first %d chars:\n%s...\n", previewLen, diff --git a/test/e2e/gittarget_isolation_e2e_test.go b/test/e2e/gittarget_isolation_e2e_test.go index d9b1ff68d..49a2959e7 100644 --- a/test/e2e/gittarget_isolation_e2e_test.go +++ b/test/e2e/gittarget_isolation_e2e_test.go @@ -20,7 +20,6 @@ package e2e import ( "fmt" - "os/exec" "path" "time" @@ -130,7 +129,7 @@ var _ = Describe("Manager GitTarget Isolation", Label("manager"), Label("smoke") By("asserting target A commits it as an event commit, not a snapshot") relPath := path.Join(pathA, fmt.Sprintf("v1/configmaps/%s/%s.yaml", testNs, cmName)) assertEventCommit := func(g Gomega) { - gitPull(g, isoRepo.CheckoutDir) + pullLatestRepoState(g, isoRepo.CheckoutDir) msg := lastCommitMessageForPath(g, isoRepo.CheckoutDir, relPath) g.Expect(msg).To(ContainSubstring("[CREATE]"), @@ -182,22 +181,11 @@ func applyIsolationConfigMap(name, namespace string) { )).To(Succeed(), "failed to apply isolation ConfigMap %q", name) } -// gitPull pulls the latest changes into a checkout. It does not use utils.Run, -// which would override cmd.Dir with the project directory. -func gitPull(g Gomega, checkoutDir string) { - cmd := exec.Command("git", "pull") - cmd.Dir = checkoutDir - out, err := cmd.CombinedOutput() - g.Expect(err).NotTo(HaveOccurred(), "git pull failed: %s", string(out)) -} - // lastCommitMessageForPath returns the body of the most recent commit that // touched the given repo-relative path. Scoping by path keeps each GitTarget's // history unambiguous even when several targets share one repo. func lastCommitMessageForPath(g Gomega, checkoutDir, relPath string) string { - cmd := exec.Command("git", "log", "-1", "--pretty=%B", "--", relPath) - cmd.Dir = checkoutDir - out, err := cmd.CombinedOutput() - g.Expect(err).NotTo(HaveOccurred(), "git log failed: %s", string(out)) - return string(out) + out, err := gitRun(checkoutDir, "log", "-1", "--pretty=%B", "--", relPath) + g.Expect(err).NotTo(HaveOccurred(), "git log failed: %s", out) + return out } diff --git a/test/e2e/helpers.go b/test/e2e/helpers.go index d657488d9..7505f0898 100644 --- a/test/e2e/helpers.go +++ b/test/e2e/helpers.go @@ -336,6 +336,38 @@ func gitRun(dir string, args ...string) (string, error) { return string(out), err } +// expectServiceRoutesToPod asserts that the given Service's endpoints resolve +// to exactly the expected controller pod. +func expectServiceRoutesToPod(g Gomega, serviceName, expectedPod string) { + GinkgoHelper() + + output, err := kubectlRunInNamespace( + namespace, + "get", + "endpoints", + serviceName, + "-o", + "jsonpath={.subsets[*].addresses[*].targetRef.name}", + ) + g.Expect(err).NotTo(HaveOccurred(), "Failed to get %s endpoints", serviceName) + + podSet := map[string]struct{}{} + for _, line := range utils.GetNonEmptyLines(output) { + if strings.HasPrefix(line, "Warning:") || strings.Contains(line, "deprecated") { + continue + } + podSet[line] = struct{}{} + } + + podNames := make([]string, 0, len(podSet)) + for podName := range podSet { + podNames = append(podNames, podName) + } + + g.Expect(podNames).To(HaveLen(1), "%s should route to exactly 1 pod", serviceName) + g.Expect(podNames[0]).To(Equal(expectedPod), "%s should route to the controller pod", serviceName) +} + func dumpFailureDiagnostics() { if !CurrentSpecReport().Failed() { return diff --git a/test/e2e/watchrule_configmap_secret_e2e_test.go b/test/e2e/watchrule_configmap_secret_e2e_test.go index bfacd1778..920d730d0 100644 --- a/test/e2e/watchrule_configmap_secret_e2e_test.go +++ b/test/e2e/watchrule_configmap_secret_e2e_test.go @@ -78,15 +78,10 @@ var _ = Describe("Manager WatchRule ConfigMap and Secret", Label("manager"), Ord SetDefaultEventuallyPollingInterval(time.Second) It("should handle a normal and healthy GitProvider", Label("smoke"), func() { - gitProviderName := "gitprovider-normal" - createGitProviderWithURLInNamespace( - gitProviderName, - testNs, - watchRuleRepo.GitSecretHTTP, - watchRuleRepo.RepoURLHTTP, - ) + // gitprovider-normal is created and verified in BeforeAll; this spec + // re-asserts it stays Ready without re-creating it. verifyResourceStatus( - "gitprovider", gitProviderName, testNs, "True", "Ready", "Repository connectivity validated", + "gitprovider", "gitprovider-normal", testNs, "True", "Ready", "Repository connectivity validated", ) }) @@ -171,13 +166,7 @@ var _ = Describe("Manager WatchRule ConfigMap and Secret", Label("manager"), Ord By("verifying Secret file is committed and does not contain plaintext data") verifyEncryptedSecretCommitted := func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = watchRuleRepo.CheckoutDir - pullOutput, pullErr := pullCmd.CombinedOutput() - if pullErr != nil { - g.Expect(pullErr).NotTo(HaveOccurred(), - fmt.Sprintf("Should successfully pull latest changes. Output: %s", string(pullOutput))) - } + pullLatestRepoState(g, watchRuleRepo.CheckoutDir) expectedFile := filepath.Join(watchRuleRepo.CheckoutDir, "e2e/secret-encryption-test", @@ -297,13 +286,7 @@ var _ = Describe("Manager WatchRule ConfigMap and Secret", Label("manager"), Ord By("waiting for auto-generated target bootstrap file to be present") Eventually(func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = watchRuleRepo.CheckoutDir - pullOutput, pullErr := pullCmd.CombinedOutput() - if pullErr != nil { - g.Expect(pullErr).NotTo(HaveOccurred(), - fmt.Sprintf("Should successfully pull latest changes. Output: %s", string(pullOutput))) - } + pullLatestRepoState(g, watchRuleRepo.CheckoutDir) bootstrapSOPSFile := filepath.Join(watchRuleRepo.CheckoutDir, "e2e/secret-autogen-test", ".sops.yaml") bootstrapContent, bootstrapErr := os.ReadFile(bootstrapSOPSFile) @@ -338,13 +321,7 @@ var _ = Describe("Manager WatchRule ConfigMap and Secret", Label("manager"), Ord By("verifying committed secret is encrypted and decryptable with generated key") verifyEncryptedSecretCommitted := func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = watchRuleRepo.CheckoutDir - pullOutput, pullErr := pullCmd.CombinedOutput() - if pullErr != nil { - g.Expect(pullErr).NotTo(HaveOccurred(), - fmt.Sprintf("Should successfully pull latest changes. Output: %s", string(pullOutput))) - } + pullLatestRepoState(g, watchRuleRepo.CheckoutDir) expectedFile := filepath.Join(watchRuleRepo.CheckoutDir, "e2e/secret-autogen-test", @@ -447,16 +424,8 @@ var _ = Describe("Manager WatchRule ConfigMap and Secret", Label("manager"), Ord // Use the pre-checked out repository directory By("using pre-checked out repository for verification") - // Pull latest changes from the remote repository By("pulling latest changes from remote repository") - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = watchRuleRepo.CheckoutDir - // Don't use utils.Run() here because it overwrites cmd.Dir with the project directory - pullOutput, pullErr := pullCmd.CombinedOutput() - if pullErr != nil { - g.Expect(pullErr).NotTo(HaveOccurred(), - fmt.Sprintf("Should successfully pull latest changes. Output: %s", string(pullOutput))) - } + pullLatestRepoState(g, watchRuleRepo.CheckoutDir) // Check for the expected ConfigMap file (new API-aligned path) expectedFile := filepath.Join(watchRuleRepo.CheckoutDir, @@ -599,13 +568,7 @@ var _ = Describe("Manager WatchRule ConfigMap and Secret", Label("manager"), Ord By("waiting for the pre-existing ConfigMap to be backfilled into git") verifyBackfill := func(g Gomega) { - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = watchRuleRepo.CheckoutDir - pullOutput, pullErr := pullCmd.CombinedOutput() - if pullErr != nil { - g.Expect(pullErr).NotTo(HaveOccurred(), - fmt.Sprintf("Should successfully pull latest changes. Output: %s", string(pullOutput))) - } + pullLatestRepoState(g, watchRuleRepo.CheckoutDir) fileInfo, statErr := os.Stat(expectedFile) g.Expect(statErr).NotTo(HaveOccurred(), @@ -678,13 +641,7 @@ var _ = Describe("Manager WatchRule ConfigMap and Secret", Label("manager"), Ord By("waiting for ConfigMap file to appear in Git repository") verifyFileCreated := func(g Gomega) { // Pull latest changes from the remote repository - pullCmd := exec.Command("git", "pull") - pullCmd.Dir = watchRuleRepo.CheckoutDir - pullOutput, pullErr := pullCmd.CombinedOutput() - if pullErr != nil { - g.Expect(pullErr).NotTo(HaveOccurred(), - fmt.Sprintf("Should successfully pull latest changes. Output: %s", string(pullOutput))) - } + pullLatestRepoState(g, watchRuleRepo.CheckoutDir) // Check for the expected ConfigMap file (new API-aligned path) expectedFile := filepath.Join(watchRuleRepo.CheckoutDir, From 255b64c0d7b328db8819a4de322691bbee174dbd Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Tue, 2 Jun 2026 09:42:22 +0000 Subject: [PATCH 32/40] ci: Keep that upload simple --- .github/workflows/ci.yml | 41 ++-------------------------------------- 1 file changed, 2 insertions(+), 39 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1f2c48dad..49dbbf289 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -394,51 +394,14 @@ jobs: echo "" } | tee -a "$GITHUB_STEP_SUMMARY" - - name: Summarize E2E timing reports - if: always() - run: | - summary_file=".stamps/e2e-timing-summary-${{ matrix.name }}.md" - docker run --rm \ - -v "${GITHUB_WORKSPACE}:${{ env.CI_WORKDIR }}" \ - -w "${{ env.CI_WORKDIR }}" \ - -e SUMMARY_FILE="${summary_file}" \ - ${{ env.CI_CONTAINER }} \ - bash -c ' - set -euo pipefail - mkdir -p .stamps - reports="$(find .stamps -path "*/ginkgo-report-*.json" -type f | sort)" - { - echo "### E2E timing reports (${{ matrix.name }})" - if [ -z "${reports}" ]; then - echo - echo "No Ginkgo JSON reports were produced." - else - # Compile the tool once instead of recompiling per report. - go build -o /tmp/spec-timings ./test/e2e/tools/spec-timings - while IFS= read -r report; do - [ -n "${report}" ] || continue - echo - echo "#### ${report}" - echo - echo "```" - /tmp/spec-timings "${report}" || true - echo "```" - done <<<"${reports}" - fi - } > "${SUMMARY_FILE}" - ' - cat "${summary_file}" >> "${GITHUB_STEP_SUMMARY}" - - name: Upload E2E raw Ginkgo JSON reports if: always() uses: actions/upload-artifact@v7 with: name: e2e-ginkgo-reports-${{ matrix.name }} - path: | - .stamps/**/ginkgo-report-*.json - .stamps/e2e-timing-summary-${{ matrix.name }}.md + path: .stamps/**/ginkgo-report-*.json if-no-files-found: ignore - retention-days: 7 + retention-days: 60 # Release job only runs on push to main after tests pass release-please: From bd33f8ff3de369ba42cdda3239877e03f1aaaa36 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Tue, 2 Jun 2026 09:42:59 +0000 Subject: [PATCH 33/40] chore: let's put it to the test --- test/e2e/crd_lifecycle_e2e_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/e2e/crd_lifecycle_e2e_test.go b/test/e2e/crd_lifecycle_e2e_test.go index 777d08c78..0fbea3768 100644 --- a/test/e2e/crd_lifecycle_e2e_test.go +++ b/test/e2e/crd_lifecycle_e2e_test.go @@ -29,7 +29,7 @@ import ( . "github.com/onsi/gomega" ) -var _ = Describe("Manager CRD Lifecycle", Label("manager"), Serial, Ordered, func() { +var _ = Describe("Manager CRD Lifecycle", Label("manager"), Ordered, func() { var ( testNs string crdLifecycleRepo *RepoArtifacts From a637ab4dc1d8afcf0a341e6abb58c719116890e2 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Tue, 2 Jun 2026 10:40:19 +0000 Subject: [PATCH 34/40] ci: resolving another race during cluster build up (HelmRelease can't wait on Kustomize). And cleaning up further. --- Tiltfile | 1 - .../design/e2e-serial-deserialization-plan.md | 215 ++++++++++ docs/design/e2e-serial-registry.md | 50 ++- docs/design/e2e-speedup-plan.md | 25 +- docs/design/e2e-speedup-skipped.md | 2 +- docs/design/e2e-test-design.md | 26 +- docs/design/refactors-branch-review.md | 40 +- .../watch-audit-rule-matching-improvement.md | 3 +- test/e2e/Taskfile.yml | 21 +- test/e2e/aggregated_apiserver_e2e_test.go | 195 ++------- test/e2e/audit_helpers_test.go | 233 ++--------- test/e2e/audit_redis_e2e_test.go | 381 ------------------ .../e2e/commit_author_attribution_e2e_test.go | 177 ++++++++ test/e2e/commit_request_e2e_test.go | 33 +- test/e2e/commit_window_batching_e2e_test.go | 14 +- test/e2e/repo_assertions_test.go | 11 - .../releases => demo-only/flux}/ingress.yaml | 0 .../releases => demo-only/flux}/kro.yaml | 0 .../setup/demo-only/flux/kustomization.yaml | 12 + test/e2e/setup/flux/kustomization.yaml | 2 - .../flux/releases/prometheus-operator.yaml | 7 + 21 files changed, 588 insertions(+), 860 deletions(-) create mode 100644 docs/design/e2e-serial-deserialization-plan.md delete mode 100644 test/e2e/audit_redis_e2e_test.go create mode 100644 test/e2e/commit_author_attribution_e2e_test.go rename test/e2e/setup/{flux/releases => demo-only/flux}/ingress.yaml (100%) rename test/e2e/setup/{flux/releases => demo-only/flux}/kro.yaml (100%) create mode 100644 test/e2e/setup/demo-only/flux/kustomization.yaml diff --git a/Tiltfile b/Tiltfile index dd1fc5594..3a876666f 100644 --- a/Tiltfile +++ b/Tiltfile @@ -91,7 +91,6 @@ test_targets = [ 'test-e2e-full', 'test-e2e-signing', 'test-e2e-manager', - 'test-e2e-audit-redis', 'test-image-refresh', ] for name in test_targets: diff --git a/docs/design/e2e-serial-deserialization-plan.md b/docs/design/e2e-serial-deserialization-plan.md new file mode 100644 index 000000000..0f8305c80 --- /dev/null +++ b/docs/design/e2e-serial-deserialization-plan.md @@ -0,0 +1,215 @@ +# E2E Serial de-serialization plan + +> Status: **partly executed** (2026-06-02). +> Generated 2026-06-02 as the follow-up to section 3 of +> [refactors-branch-review.md](refactors-branch-review.md). +> Authoritative Serial list: [e2e-serial-registry.md](e2e-serial-registry.md). +> +> **Outcome so far:** +> - **Part A (`crd_lifecycle`)** — **done**: de-serialized and verified green +> under `--procs=4`. See the registry's "De-serialized" section. +> - **Part B (audit pipeline)** — the heavy B1 refactor was **declined** as +> planned. Instead the redundant `Audit Redis Queue`/`Consumer` specs were +> retired and their one unique assertion (OIDC author attribution) moved to the +> new **parallel** `commit_author_attribution_e2e_test.go`. `Commit Window +> Batching` and `Commit Request` remain `Serial`. + +## Why this is not "just remove the `Serial` labels" + +Every entry in the Serial registry is **correctness-driven (shared, cluster-wide +state), not flakiness-driven**. The race fixes that landed (`03b95ab`, +`0787b1b`) made the suite *stable*; they did not make the shared state +disappear. Removing a `Serial` label without first isolating the underlying +state re-introduces cross-spec bleed. So de-serialization is an isolation +*refactor*, gated on verification — not a cleanup. + +Of the 8 Serial containers, 4 are genuinely serial and stay: + +- `Restart Snapshot Safety`, `image refresh dependency chain` — restart/reimage + the singleton controller. +- `Aggregated API server` — installs/removes a cluster `APIService`; perturbs + apiserver discovery for every client. +- `Bi Directional` — asserts exact remote commit counts; any concurrent + controller activity breaks the loop assertions. + +The two candidates below are the only realistic de-serialization targets. They +are very different in size and risk, so they are planned independently. + +--- + +## Part A — `crd_lifecycle` (smaller; likely already unblocked) + +### Current Serial cause + +[crd_lifecycle_e2e_test.go:32](../../test/e2e/crd_lifecycle_e2e_test.go#L32) is +`Serial` because it installs/deletes a CRD and a (cluster-scoped) +`ClusterWatchRule`. Per the registry, that changes cluster-wide discovery / +the GVR catalog, which historically forced *unrelated* GitTargets to resnapshot; +the resulting `reconcile: sync …` commits hide other WatchRule specs' exact +`[CREATE]`/`[DELETE]` event-commit assertions. + +### What already changed + +1. **Finding #2 fix** ([manager.go:1191](../../internal/watch/manager.go#L1191)): + a target is only re-snapshotted when its *resolved* plan hash changes, and a + target whose rules currently resolve to nothing is kept as an empty plan + rather than evicted. A transient discovery gap no longer makes a target + vanish and resnapshot on recovery. +2. **Per-file CRD groups** ([icecream.go:30](../../test/e2e/icecream.go#L30)): + `crd_lifecycle` owns the API group `crd-lifecycle.e2e.example.com`, so its + `IceCreamOrder` CRD cannot collide by name with any other spec's CRD. + +### Why it is probably safe now (to be verified, not assumed) + +When `crd_lifecycle` installs its CRD, a *new* GVR becomes resolvable. A +concurrently-running spec would only be dragged into a resnapshot if it has a +rule that newly matches that GVR — i.e. a wildcard `ClusterWatchRule`. A scan of +the suite shows the only wildcard `ClusterWatchRule` +([restart/clusterwatchrule-wildcard.tmpl](../../test/e2e/templates/restart/clusterwatchrule-wildcard.tmpl)) +belongs to `restart_snapshot`, which is itself `Serial` and therefore never runs +concurrently. `demo_e2e_test.go` also uses a `ClusterWatchRule` and must be +checked (step 1). With no concurrent wildcard matcher, the icecream CRD only +changes `crd_lifecycle`'s own (name-isolated) targets, and finding #2 keeps +every other target's plan hash stable across the catalog refresh. + +### Plan + +1. **Confirm no concurrent wildcard matcher.** Audit `demo_e2e_test.go`'s + `ClusterWatchRule` scope (and any future broad rules). If it is scoped to + specific groups/resources that exclude the icecream group, it is safe; if it + is wildcard, either scope it down or keep `crd_lifecycle` Serial. +2. **Flip the label.** Drop `Serial` from + [crd_lifecycle_e2e_test.go:32](../../test/e2e/crd_lifecycle_e2e_test.go#L32) + (keep `Ordered`). Update the registry row. +3. **Verify under parallelism.** Run a `procs=2` (ideally higher) loop of the + `manager`-labelled specs together, e.g. several repetitions of + `task test-e2e` with `E2E_GINKGO_PROCS>=2`, and grep the WatchRule specs' + commit messages for unexpected `reconcile: sync` commits interleaved with + their `[CREATE]`/`[DELETE]` assertions. The gittarget-isolation spec already + encodes exactly this assertion + ([gittarget_isolation_e2e_test.go:140](../../test/e2e/gittarget_isolation_e2e_test.go#L140)) + and is the canary. +4. **If it bleeds**, do not force it. Capture the residual mechanism (likely a + `RefreshAPIResourceCatalog` churn path that perturbs targets even without a + matching rule) and fold it into the "settled-empty refinement" follow-up that + finding #2 already defers. + +### Effort / risk + +**Small–medium, low blast radius.** Mostly a label flip plus a verification +loop; the supporting code fix already shipped. The realistic failure mode is +"verification shows residual bleed," in which case we learn the exact remaining +mechanism and stop — no production code is touched. + +--- + +## Part B — the audit pipeline (historically 4 specs; reduced) + +### Scope + +This originally covered `Audit Redis Queue`, `Audit Redis Consumer`, +`Commit Window Batching`, and `Commit Request` — historically all labelled +`audit-redis`. +The queue/consumer containers have since been retired; the remaining +audit-consumer containers are `Commit Window Batching` +([commit_window_batching_e2e_test.go](../../test/e2e/commit_window_batching_e2e_test.go)) +and `Commit Request` +([commit_request_e2e_test.go](../../test/e2e/commit_request_e2e_test.go)). +The unique author-attribution assertion moved to the parallel +`Commit Author Attribution` +([commit_author_attribution_e2e_test.go](../../test/e2e/commit_author_attribution_e2e_test.go)) +container. + +### Current Serial cause (and the real shape of the problem) + +The audit pipeline is a **cluster-wide singleton firehose**: + +``` +apiserver audit policy → ONE audit webhook (internal/webhook/audit_handler.go:496) + → ONE Redis stream (cfg.auditRedisStream, default) + → ONE consumer group "gitopsreverser-consumer" (redis_audit_consumer.go:53) + → routes each event to GitTargets by matching WatchRules → commits +``` + +These specs assert commit **exclusivity** ("my commit touched only my file"; +commit-window batching asserts a batch contains only its own events). The +pollution source is *not just the other 3 audit specs* — it is **any** parallel +spec that writes a watched resource, because every such write produces an audit +event that flows through the same stream/consumer and can land in the same +batched commit. That is why they are fully `Serial`, not merely serial among +themselves. + +### Why the cheap approaches do not work + +- **Per-test stream name via config flip:** the apiserver audit policy targets a + single webhook endpoint on the singleton controller; tests cannot make the + apiserver route their events to a different stream. The stream name is a + *deploy-time* config, not a per-request property. +- **Per-test Redis consumer groups:** Redis consumer groups *fan out* — every + group receives every message. A new group sees all tests' events, so it + isolates nothing. +- **Namespace-scoped assertions only:** scoping each spec's "touched only my + file" assertion to its own namespace helps the simple cases, but + commit-window batching deliberately coalesces multiple events into one commit; + concurrent events from other specs still share that batch window. Assertion + scoping alone cannot fix the batching specs. + +### Options + +**B1 — Producer-side per-namespace (or per-target) stream routing.** Make the +webhook handler write to a stream keyed by the event's namespace/target instead +of one global stream, and have the consumer manage a group per active stream. +*Pro:* true isolation; the 4 specs go fully parallel. *Con:* significant change +to load-bearing production code (`audit_handler.go`, `redis_audit_queue.go`, +`redis_audit_consumer.go`), adds routing/lifecycle complexity that production +does **not** want (production is happy with one stream), and risks regressing +the very ordering/exclusivity guarantees the specs protect. Highest payoff, +highest risk. + +**B2 — Test-only routing mode.** Same as B1 but gated behind a test-only flag so +production keeps the single stream. *Pro:* shields production semantics. +*Con:* a test-only code path through the audit hot path is a maintenance smell +and can drift from the real path it is meant to validate — partially defeating +the point of an e2e test. + +**B3 — A dedicated "audit" lane: concurrent with the rest, serial among +themselves.** Keep the 4 specs mutually exclusive but let them run *alongside* +non-audit specs, with the non-audit specs taught not to pollute the audit +window. Concretely: (a) give the 4 audit specs a shared ordering/semaphore so +only one is "live" at a time, and (b) ensure no *other* parallel spec writes a +watched resource into a GitTarget the audit specs assert on — which in practice +means the audit specs must assert on a path/namespace no other spec writes, and +the consumer's commit for an audit spec must be scoped to that path. +*Pro:* no production change. *Con:* only works if the exclusivity assertions can +be made path-isolated despite shared batching — which is exactly the open +question; if batching coalesces a foreign event, the assertion still breaks. + +### Recommendation + +Do **not** attempt B1/B2 as part of routine speedup work. The audit pipeline's +singleton design is intentional, and bending it for test parallelism trades a +real production-correctness surface for a few minutes of CI time. If the audit +specs' runtime becomes a genuine bottleneck, pursue **B3 first** as a spike: +prove (or disprove) that the exclusivity assertions can be made path-isolated +under the shared consumer/batcher. Only if B3 is proven impossible *and* the +speedup is worth it should B1 be scheduled as its own design + review cycle. + +### Effort / risk + +**Large, high blast radius** for B1/B2 (touches the audit producer + consumer, +the most correctness-sensitive path in the system). **Medium** for the B3 spike, +but with a real chance the answer is "can't be isolated, leave Serial." + +--- + +## Sequencing recommendation + +1. **`crd_lifecycle` (Part A)** — do this first; it is mostly verification on top + of a fix that already shipped, and it removes one Serial entry with low risk. +2. **Audit B3 spike (Part B)** — only if audit-spec runtime is a measured + bottleneck; timebox it and accept "stays Serial" as a valid outcome. +3. **Audit B1** — only as a deliberately scheduled design cycle, never as a + drive-by. + +The `e2e-serial-registry.md` table is the source of truth; update the relevant +row in the same PR as any label change. diff --git a/docs/design/e2e-serial-registry.md b/docs/design/e2e-serial-registry.md index cf341808f..3de5db63e 100644 --- a/docs/design/e2e-serial-registry.md +++ b/docs/design/e2e-serial-registry.md @@ -5,6 +5,10 @@ containers that must run `Serial` (i.e. never concurrently with any other spec) under bounded Ginkgo parallelism (Phase 2.5 of [e2e-speedup-plan.md](e2e-speedup-plan.md)). +> De-serializing any row below is an isolation refactor, not a label flip. The +> two realistic candidates (`crd_lifecycle` and the four audit pipeline specs) are +> planned in [e2e-serial-deserialization-plan.md](e2e-serial-deserialization-plan.md). + ## Rule A test container is `Serial` only when it mutates shared, **process-level or @@ -25,17 +29,45 @@ row below. | [test/e2e/restart_snapshot_e2e_test.go](../../test/e2e/restart_snapshot_e2e_test.go) | `Restart Snapshot Safety` | Rollout-restarts the controller deployment. | The controller is a singleton; restarting it disrupts in-flight reconciles/commits for every other spec. | | [test/e2e/image_refresh_test.go](../../test/e2e/image_refresh_test.go) | `image refresh dependency chain` | Changes the controller image / redeploys the controller. | Same singleton controller; an image swap perturbs all concurrent specs. | | [test/e2e/aggregated_apiserver_e2e_test.go](../../test/e2e/aggregated_apiserver_e2e_test.go) | `Aggregated API server` | Installs/removes a cluster-scoped `APIService` (aggregation layer). | Registering/removing an aggregated API briefly disrupts apiserver discovery for **every** client, including unrelated kubectl calls in other specs. | -| [test/e2e/audit_redis_e2e_test.go](../../test/e2e/audit_redis_e2e_test.go) | `Audit Redis Queue`, `Audit Redis Consumer` | The single global audit pipeline: audit webhook → shared Redis stream → consumer. | The consumer is a cluster-wide singleton processing one shared stream; these specs assert on commit *exclusivity*, which any concurrent audit traffic violates. | -| [test/e2e/commit_window_batching_e2e_test.go](../../test/e2e/commit_window_batching_e2e_test.go) | `Commit Window Batching` | Same global audit pipeline (labelled `audit-redis`). | Bursts of ConfigMap events flow through the shared stream/consumer and leak into other audit specs' commits. | -| [test/e2e/commit_request_e2e_test.go](../../test/e2e/commit_request_e2e_test.go) | `Commit Request` | Same global audit pipeline (labelled `audit-redis`). | Shares the stream/consumer; commit-window/queue semantics interleave under concurrency. | +| [test/e2e/commit_window_batching_e2e_test.go](../../test/e2e/commit_window_batching_e2e_test.go) | `Commit Window Batching` | Same global audit pipeline (labelled `audit-consumer`). | Bursts of ConfigMap events flow through the shared audit consumer and leak into other audit specs' commits. | +| [test/e2e/commit_request_e2e_test.go](../../test/e2e/commit_request_e2e_test.go) | `Commit Request` | Same global audit pipeline (labelled `audit-consumer`). | Shares the audit consumer; commit-window/queue semantics interleave under concurrency. | | [test/e2e/bi_directional_e2e_test.go](../../test/e2e/bi_directional_e2e_test.go) | `Bi Directional` | Whole-cluster Flux↔gitops-reverser round-trip; asserts on **exact** remote commit counts to prove no commit loop. | Any concurrent controller activity adds/reorders commits and breaks the exact-count loop assertions. Passes sequentially; +2 commits only under parallelism. | -| [test/e2e/crd_lifecycle_e2e_test.go](../../test/e2e/crd_lifecycle_e2e_test.go) | `Manager CRD Lifecycle` | Installs/deletes a CRD and a wildcard `ClusterWatchRule`, changing cluster-wide discovery/GVR catalog state. | Per-file CRD groups prevent name collisions, but discovery changes still force unrelated GitTargets to resnapshot. That can hide exact `[CREATE]`/`[DELETE]` event-commit assertions in other WatchRule specs behind later `reconcile: sync ...` commits. | -All four `audit-redis`-labelled containers above were moved to `Serial` after the -first procs=2 smoke run: `Audit Redis Consumer` asserted its commit touched only -its own file but picked up `Commit Window Batching`'s files from the shared -consumer. Because Ginkgo runs `Serial` specs alone (after all parallel specs), -the audit pipeline now only ever has one client at a time. +### De-serialized + +`Manager CRD Lifecycle` ([crd_lifecycle_e2e_test.go](../../test/e2e/crd_lifecycle_e2e_test.go)) +**was** `Serial` because installing/deleting its CRD changes cluster-wide +discovery and historically forced unrelated GitTargets to resnapshot (leaking +`reconcile: sync …` commits into other specs' exact `[CREATE]`/`[DELETE]` +assertions). It now runs parallel (`Ordered`, not `Serial`) because: + +- Finding #2's fix ([manager.go:1191](../../internal/watch/manager.go#L1191)) + only resnapshots a target when its *resolved* plan hash changes, so a catalog + refresh no longer perturbs targets that do not match the new GVR. +- Per-file CRD groups ([icecream.go:30](../../test/e2e/icecream.go#L30)) keep the + `IceCreamOrder` CRD name-isolated, and the only other wildcard-ish + `ClusterWatchRule` (`restart_snapshot`) is itself `Serial`, so no concurrent + spec has a rule matching the icecream group. + +See [e2e-serial-deserialization-plan.md](e2e-serial-deserialization-plan.md) for +the full rationale. + +The former `Audit Redis Queue` / `Audit Redis Consumer` containers were retired: +the producer-path and basic consumer-commit assertions duplicated coverage in +the WatchRule suite, so they were dropped. The one genuinely unique assertion — +that OIDC name/email claims in `user.extra` end up as the Git commit author — +moved to `Commit Author Attribution` +([commit_author_attribution_e2e_test.go](../../test/e2e/commit_author_attribution_e2e_test.go)), +which runs **parallel** (not `Serial`): its dedicated GitProvider uses a 0s +commit window so each event is its own commit, and it reads the author scoped to +each ConfigMap's unique path, so concurrent audit traffic cannot change the +commit it asserts on. + +The remaining `audit-consumer`-labelled containers (`Commit Window Batching`, +`Commit Request`) stay `Serial`: they assert on commit-window/batching semantics +over the shared audit consumer, which any concurrent audit traffic violates. +Because Ginkgo runs `Serial` specs alone (after all parallel specs), the audit +pipeline only ever has one client at a time for those. ## Triage list (watch during stability runs) diff --git a/docs/design/e2e-speedup-plan.md b/docs/design/e2e-speedup-plan.md index 876ecd7f5..d01071a65 100644 --- a/docs/design/e2e-speedup-plan.md +++ b/docs/design/e2e-speedup-plan.md @@ -24,7 +24,7 @@ with rationale. | | | |---|---| | Warm `task prepare-e2e` | **1.4 s** (everything stamp-cached) | -| Smoke suite (`task test-e2e`, 20 of 47 specs) | **418.7 s** (~7 min) | +| Smoke suite (`task test-e2e`, 20 of 46 specs) | **418.7 s** (~7 min) | | BeforeSuite (Go re-invoking warm prepare) | 1.4 s | ### Per-spec ranking (smoke filter, warm) @@ -36,11 +36,11 @@ with rationale. 34.1 s 8.2 Aggregated API server should install and serve flunders through the aggregation layer 28.9 s 6.9 Manager should create Git commit when IceCreamOrder CRD is installed via ClusterWatchRule 25.7 s 6.2 Commit Signing should produce per-event commits verifiable locally and by Gitea - 25.2 s 6.0 Audit Redis Queue should enqueue incoming audit webhook events into a Redis stream - 24.4 s 5.9 Audit Redis Consumer should attribute the commit to the OIDC display name and email + 25.2 s 6.0 Audit Redis Queue should enqueue incoming audit webhook events into a Redis stream (retired) + 24.4 s 5.9 Audit Redis Consumer should attribute the commit to the OIDC display name and email (moved to Commit Author Attribution) 22.3 s 5.3 Commit Request finalizes a CommitRequest created with metadata.generateName 21.9 s 5.2 Commit Request finalizes the open commit window on demand and reports the resulting SHA - 16.4 s 3.9 Audit Redis Consumer should produce a Git commit from an audit stream event + 16.4 s 3.9 Audit Redis Consumer should produce a Git commit from an audit stream event (retired) 15.5 s 3.7 Manager should delete Git file when ConfigMap is deleted via WatchRule 9.9 s 2.4 Manager should create Git commit when ConfigMap is added via WatchRule 9.1 s 2.2 Manager should commit encrypted Secret manifests when WatchRule includes secrets @@ -439,15 +439,14 @@ Landed on branch `refactors` (PR #159). `gh` works directly now — no kubectl reference is qualified `icecreamorders.`. - **Serial registry** ([e2e-serial-registry.md](e2e-serial-registry.md)) is larger than the initial audit: `restart_snapshot`, `image_refresh`, - `aggregated_api` (controller/APIService level) **plus** the four - `audit-redis`-labelled containers (`Audit Redis Queue/Consumer`, - `Commit Window Batching`, `Commit Request`) — they share one global - audit pipeline (webhook → Redis → consumer) and cross-contaminated each - other's commits — **plus** `bi_directional` (asserts exact commit counts - to prove no commit loop; broken by any concurrent controller activity) - and `crd_lifecycle` (CRD/ClusterWatchRule changes are name-isolated, but - still perturb cluster discovery and can trigger unrelated GitTarget - resnapshots). + `aggregated_api` (controller/APIService level) **plus** the remaining + `audit-consumer`-labelled containers (`Commit Window Batching`, + `Commit Request`) — they share one global audit pipeline and + cross-contaminated each other's commits — **plus** + `bi_directional` (asserts exact commit counts to prove no commit loop; + broken by any concurrent controller activity). `crd_lifecycle` and the + former `Audit Redis Queue/Consumer` containers have since been retired from + the Serial set; see [e2e-serial-registry.md](e2e-serial-registry.md). - **Timeouts widened for parallel/slow load:** `verifyResourceStatus` readiness 30s → 90s (post-CRD-install the controller's discovery cache lags before serving the new GVR, so a dependent WatchRule can't reach diff --git a/docs/design/e2e-speedup-skipped.md b/docs/design/e2e-speedup-skipped.md index f17fa0cd3..fbf5b8b3b 100644 --- a/docs/design/e2e-speedup-skipped.md +++ b/docs/design/e2e-speedup-skipped.md @@ -30,7 +30,7 @@ that Phase 1's DAG refactor can't fix. **Proposal.** [.github/workflows/ci.yml:261-313](../../.github/workflows/ci.yml#L261-L313) runs the e2e suite as a 2-entry matrix `{quickstart, full}`. Splitting -`full` into 4–5 label-scoped shards (`smoke`, `signing`, `audit-redis`, +`full` into 4–5 label-scoped shards (`smoke`, `signing`, `audit-consumer`, `bi-directional`, `image-refresh`, `aggregated-api`) would parallelize across GitHub runners and drop wallclock to `max(shard)` instead of `sum(shard)`. diff --git a/docs/design/e2e-test-design.md b/docs/design/e2e-test-design.md index a68238354..23850c2f5 100644 --- a/docs/design/e2e-test-design.md +++ b/docs/design/e2e-test-design.md @@ -108,7 +108,6 @@ These are the canonical Task entry points on the current worktree: | `task test-e2e-full` | Entire e2e package | | `task test-e2e-manager` | Manager-focused scenarios | | `task test-e2e-signing` | Commit-signing scenarios | -| `task test-e2e-audit-redis` | Audit webhook to Redis scenarios | | `task test-image-refresh` | Image rebuild / reload invalidation chain | | `task test-e2e-quickstart-helm` | Quickstart framework with Helm install | | `task test-e2e-quickstart-manifest` | Quickstart framework with manifest install | @@ -120,7 +119,7 @@ installing the system under test. The intended interpretation of these entry points is: -- `task test-e2e`, `task test-e2e-manager`, `task test-e2e-signing`, `task test-e2e-audit-redis`, and `task test-image-refresh` are standard controller behavior checks and should normally run on the default `config-dir` install +- `task test-e2e`, `task test-e2e-manager`, `task test-e2e-signing` and `task test-image-refresh` are standard controller behavior checks and should normally run on the default `config-dir` install - `task test-e2e-quickstart-helm` is the important validation for the Helm install path - `task test-e2e-quickstart-manifest` is the important validation for the single-file manifest install path - `task test-e2e-full` is a full spec run that includes the bi-directional scenario, but it is still not the primary way to validate all install modes @@ -200,7 +199,7 @@ These namespaces are typically named: - `-test-manager` - `-test-signing` -- `-test-audit-redis` +- `-test-audit-consumer` - and so on Those namespaces are created in `BeforeAll` and removed in `AfterAll` for the suite. @@ -234,8 +233,9 @@ The only cluster-level variable still consumed across files is: Repo state (names, checkout paths, Secret names, optional receiver webhook info) flows through a typed `RepoArtifacts` struct returned by `SetupRepo`, not through package-global env vars. -One important current detail is that `audit_redis_e2e_test.go` uses file-local lazy repo -initialization so either top-level `Describe` can run first without hidden ordering assumptions. +One important current detail is that the remaining audit-consumer specs share a lazy repo +helper in `audit_helpers_test.go`, so `Commit Window Batching` and `Commit Request` can +run first without hidden ordering assumptions. ## Repo Model: One Repo Per E2E Test File @@ -245,7 +245,8 @@ Each repo-using e2e file owns its own repo: |---|---|---| | `e2e_test.go` | `testNamespaceFor("manager")` | `e2e-manager-` | | `signing_e2e_test.go` | `testNamespaceFor("signing")` | `e2e-signing-` | -| `audit_redis_e2e_test.go` | `testNamespaceFor("audit-consumer")` | `e2e-audit-redis-` | +| `commit_window_batching_e2e_test.go` / `commit_request_e2e_test.go` | `testNamespaceFor("audit-consumer")` via shared helper; per-spec producer namespaces | `e2e-audit-consumer-` | +| `commit_author_attribution_e2e_test.go` | `testNamespaceFor("commit-author")` | `e2e-commit-author-` | | `quickstart_framework_e2e_test.go` | `testNamespaceFor("quickstart-framework")` | `e2e-quickstart-framework-` | | `bi_directional_e2e_test.go` | `testNamespaceFor("bi-directional")` | `e2e-bi-directional-` | | `demo_e2e_test.go` | fixed `vote` | fixed `demo` | @@ -260,9 +261,9 @@ Repo stamps live under: not under the install namespace. -The `audit_redis_e2e_test.go` file additionally uses a `sync.Once`-backed `ensureAuditRedisRepo()` -helper so either of its two top-level `Describe` blocks can initialize the shared file-local repo -first without cross-container ordering coupling. +The `audit_helpers_test.go` file additionally uses a `sync.Once`-backed +`ensureAuditConsumerRepo()` helper so either audit-consumer container can initialize the shared +repo first without cross-container ordering coupling. ## Why CI Could Pass Even If This Was Fragile @@ -362,7 +363,9 @@ The e2e package currently contains: |---|---|---| | `e2e_test.go` | `manager` | Main reconciliation and GitOps behavior | | `signing_e2e_test.go` | `signing` | Commit signing behavior | -| `audit_redis_e2e_test.go` | `audit-redis` | Audit webhook to Redis producer/consumer | +| `commit_window_batching_e2e_test.go` | `audit-consumer` | Commit-window batching through the audit pipeline | +| `commit_request_e2e_test.go` | `audit-consumer` | CommitRequest finalization through the audit pipeline | +| `commit_author_attribution_e2e_test.go` | `manager`, `smoke` | OIDC claim to Git author attribution | | `image_refresh_test.go` | `image-refresh` | Build/load/restart invalidation chain | | `quickstart_framework_e2e_test.go` | `quickstart-framework` | Installer-level quickstart flow | | `bi_directional_e2e_test.go` | `bi-directional` | Flux plus gitops-reverser shared ownership | @@ -384,7 +387,7 @@ Currently that includes: - secret encryption path - ConfigMap create/delete Git commits - cluster-scoped CRD install Git commit -- audit-to-Redis producer/consumer path +- audit pipeline batching, commit-request, and author-attribution paths - one signing verification path That keeps the default run closer to a product smoke test than to a full infra regression sweep. @@ -416,7 +419,6 @@ Use: ```bash task test-e2e-manager task test-e2e-signing -task test-e2e-audit-redis task test-image-refresh task test-e2e-full ``` diff --git a/docs/design/refactors-branch-review.md b/docs/design/refactors-branch-review.md index a25a35d53..360c717f4 100644 --- a/docs/design/refactors-branch-review.md +++ b/docs/design/refactors-branch-review.md @@ -252,21 +252,33 @@ current branch, but should be updated if CI stops rendering Allure: **[e2e-serial-registry.md](e2e-serial-registry.md)** — the Serial entries are **correctness-driven (shared state), not flakiness-driven**, so improved stability doesn't justify de-serializing directly. It *does* unlock the deeper -refactors the registry is implicitly waiting for: - -- [ ] **Isolate the audit pipeline → de-serialize 4 of 8 entries.** `Audit Redis - Queue/Consumer`, `Commit Window Batching`, `Commit Request` are Serial only - because the webhook→Redis-stream→consumer pipeline is a global singleton. - Per-test stream keys / consumer groups (same "isolate by name" move as the - icecream CRD groups) lets all four run parallel. **Highest-leverage remaining - speedup.** -- [ ] **Fix the discovery-lag re-snapshot (code finding #2) → de-serialize - `crd_lifecycle`.** That spec is Serial because CRD install/delete forces - unrelated GitTargets to resnapshot — the `manager.go` empty-plan / swallowed - resolve-miss behavior. Fix at the source; name-isolated CRD groups then make - the spec parallel-safe. +refactors the registry is implicitly waiting for. Both items below were planned +in [e2e-serial-deserialization-plan.md](e2e-serial-deserialization-plan.md) and +then partly executed: + +- [~] **Audit pipeline.** The planned heavy refactor (per-test stream/consumer + isolation, plan option B1) was **not** done — it bends the intentionally + singleton audit pipeline for test parallelism and isn't worth the + production-correctness risk. Instead the audit-consumer Serial set was *reduced*: + `Audit Redis Queue` and `Audit Redis Consumer` + (`audit_redis_e2e_test.go`, now deleted) were retired — their producer-path and + basic consumer-commit assertions duplicated the WatchRule suite. The one unique + assertion (OIDC `user.extra` → commit author) moved to a new **parallel** + [commit_author_attribution_e2e_test.go](../../test/e2e/commit_author_attribution_e2e_test.go): + a dedicated 0s-commit-window GitProvider + per-path author read makes it + concurrency-safe, so it needs no `Serial`. `Commit Window Batching` and + `Commit Request` **stay Serial** — they assert batching/commit-window semantics + over the shared stream, which concurrent audit traffic genuinely violates. +- [x] **De-serialize `crd_lifecycle` (was gated on code finding #2).** **Done:** + with finding #2's fix shipped (a target only resnapshots when its *resolved* + plan hash changes) and per-file CRD groups keeping the `IceCreamOrder` CRD + name-isolated, `Manager CRD Lifecycle` dropped `Serial` and now runs `Ordered` + in the parallel pool. Verified green under `--procs=4` (the only other + wildcard `ClusterWatchRule`, in `restart_snapshot`, is itself Serial, so no + concurrent spec matches the icecream group). Registry updated. - `bi_directional` (exact commit-count) and `restart_snapshot` / `image_refresh` - (singleton controller) are genuinely serial — leave them. + (singleton controller) + `aggregated_apiserver` (cluster `APIService`) are + genuinely serial — leave them. - [ ] **Confidence-window pass (the speedup plan's open thread).** Now that the race fixes landed (`03b95ab`, `0787b1b`), decide whether CI should keep diff --git a/docs/design/watch-audit-rule-matching-improvement.md b/docs/design/watch-audit-rule-matching-improvement.md index 28d96f176..bcb2ef3a8 100644 --- a/docs/design/watch-audit-rule-matching-improvement.md +++ b/docs/design/watch-audit-rule-matching-improvement.md @@ -481,7 +481,8 @@ E2E tests (existing, to be strengthened per the E2E section below): - [test/e2e/e2e_test.go](/workspaces/gitops-reverser/test/e2e/e2e_test.go:717) - [test/e2e/tilt_playground_e2e_test.go](/workspaces/gitops-reverser/test/e2e/tilt_playground_e2e_test.go:55) -- [test/e2e/audit_redis_e2e_test.go](/workspaces/gitops-reverser/test/e2e/audit_redis_e2e_test.go:54) +- [test/e2e/commit_window_batching_e2e_test.go](/workspaces/gitops-reverser/test/e2e/commit_window_batching_e2e_test.go:54) +- [test/e2e/commit_request_e2e_test.go](/workspaces/gitops-reverser/test/e2e/commit_request_e2e_test.go:54) ## E2E Evidence From A Real Repo History diff --git a/test/e2e/Taskfile.yml b/test/e2e/Taskfile.yml index 8b0bee84e..720d5a5e8 100644 --- a/test/e2e/Taskfile.yml +++ b/test/e2e/Taskfile.yml @@ -198,7 +198,7 @@ tasks: test-e2e-full: desc: Run the full e2e suite, including slow and specialized scenarios vars: - # 47 specs at ~22s avg + ~6m cold prepare ≈ 25m; the smoke-tuned 15m default + # 46 specs at ~22s avg + ~6m cold prepare ≈ 25m; the smoke-tuned 15m default # times out mid-run and gives a goroutine dump instead of a useful report. E2E_FULL_TIMEOUT: '{{.E2E_FULL_TIMEOUT | default "30m"}}' cmds: @@ -521,17 +521,6 @@ tasks: fi "{{.KUBECTL}}" --context "${ctx}" -n "${ns}" delete secret "${name}" --ignore-not-found=true - test-e2e-audit-redis: - desc: Run dedicated audit to Redis enqueue scenario - cmds: - - | - export CTX="{{.CTX}}" - export INSTALL_MODE="{{.INSTALL_MODE}}" - export NAMESPACE="{{.NAMESPACE}}" - export E2E_AGE_KEY_FILE="{{.CS}}/age-key.txt" - go test -timeout "{{.E2E_GO_TEST_TIMEOUT}}" ./test/e2e/ -v -ginkgo.v -ginkgo.label-filter=audit-redis \ - -ginkgo.json-report="{{.CS}}/{{.NAMESPACE}}/ginkgo-report-audit-redis.json" - test-e2e-aggregated-api: desc: Run aggregated API server e2e scenarios cmds: @@ -1189,6 +1178,14 @@ tasks: crd/quizsessions.examples.configbutler.ai \ crd/quizsubmissions.examples.configbutler.ai \ --timeout=180s + # Traefik and kro live in the demo path (not the core bring-up): the + # demo podinfo flows below need kro's ResourceGraphDefinition CRD and + # traefik's IngressRoutes. Install them here and wait before applying + # anything that depends on them. + kubectl --context "{{.CTX}}" apply -k "{{.DEMO_MANIFESTS_DIR}}/flux" + kubectl --context "{{.CTX}}" -n flux-system wait \ + helmrelease/traefik helmrelease/kro \ + --for=condition=Ready --timeout=300s kubectl --context "{{.CTX}}" apply -f "{{.DEMO_MANIFESTS_DIR}}/podinfo-kro-template.yaml" kubectl --context "{{.CTX}}" wait --for=condition=Ready \ resourcegraphdefinition/podinfo-app --timeout=120s diff --git a/test/e2e/aggregated_apiserver_e2e_test.go b/test/e2e/aggregated_apiserver_e2e_test.go index 7cbf1d0b9..2188b7a21 100644 --- a/test/e2e/aggregated_apiserver_e2e_test.go +++ b/test/e2e/aggregated_apiserver_e2e_test.go @@ -19,7 +19,6 @@ limitations under the License. package e2e import ( - "context" "encoding/json" "fmt" "os" @@ -168,58 +167,31 @@ spec: }, 30*time.Second, time.Second).Should(Succeed()) }) - It("should recover Flunder audit request and response bodies through the proxy", func() { - streamName := defaultAuditRedisStream - configMapName := fmt.Sprintf("aggregated-audit-compare-cm-%d", GinkgoRandomSeed()) - flunderName := fmt.Sprintf("aggregated-audit-compare-flunder-%d", GinkgoRandomSeed()) - - By("connecting to Valkey through the e2e port-forward") - client := newE2EValkeyClient() - defer func() { - _ = client.Close() - }() - - Eventually(func(g Gomega) { - g.Expect(client.Ping(context.Background()).Err()).NotTo(HaveOccurred()) - }, 30*time.Second, 2*time.Second).Should(Succeed()) - - By("recording the latest audit stream entry before creating comparison resources") - baselineID, err := latestAuditStreamID(context.Background(), client, streamName) - Expect(err).NotTo(HaveOccurred()) - - configMapManifest := fmt.Sprintf(`apiVersion: v1 -kind: ConfigMap -metadata: - name: %s - namespace: %s -data: - compare: configmap -`, configMapName, testNs) + It("should never drop aggregated audit events as shallow", func() { + // A "shallow" audit event is one whose request/response body never + // arrived, so it cannot drive a high-quality Git write and is dropped, + // incrementing gitopsreverser_audit_shallow_dropped_total. In a correctly + // configured environment every audit event is paired with its body, so + // that counter must stay at zero — for this scenario and for every other + // test sharing the cluster. Asserting it here is the externally + // observable equivalent of confirming the proxy recovers full bodies. + ensureFlunderWatchRuleReady() + flunderName := fmt.Sprintf("aggregated-shallow-flunder-%d", GinkgoRandomSeed()) + repoPath := path.Join( + aggregatedPath, + fmt.Sprintf("wardle.example.com/v1alpha1/flunders/%s/%s.yaml", testNs, flunderName), + ) + repoFile := filepath.Join(repo.CheckoutDir, repoPath) flunderManifest := fmt.Sprintf(`apiVersion: wardle.example.com/v1alpha1 kind: Flunder metadata: name: %s namespace: %s spec: - reference: compare-reference + reference: shallow-check-reference `, flunderName, testNs) - By("creating a ConfigMap and Flunder via the same kubectl apply path") - _, err = kubectlRunWithStdin(testNs, configMapManifest, "apply", "-f", "-") - Expect(err).NotTo(HaveOccurred()) - _, err = kubectlRunWithStdin(testNs, flunderManifest, "apply", "-f", "-") - Expect(err).NotTo(HaveOccurred()) - - DeferCleanup(func() { - if skipCleanupBecauseResourcesArePreserved( - fmt.Sprintf("ConfigMap %s/%s", testNs, configMapName), - testNs, - ) { - return - } - _, _ = kubectlRunInNamespace(testNs, "delete", "configmap", configMapName, "--ignore-not-found=true") - }) DeferCleanup(func() { if skipCleanupBecauseResourcesArePreserved(fmt.Sprintf("Flunder %s/%s", testNs, flunderName), testNs) { return @@ -227,62 +199,26 @@ spec: _, _ = kubectlRunInNamespace(testNs, "delete", "flunder", flunderName, "--ignore-not-found=true") }) - By("capturing the corresponding raw audit payloads from the Valkey stream") + By("creating a Flunder through the aggregated API to exercise the audit body-join path") + _, err := kubectlRunWithStdin(testNs, flunderManifest, "apply", "-f", "-") + Expect(err).NotTo(HaveOccurred()) + + By("waiting for the create commit, confirming the audit body was recovered (not shallow)") Eventually(func(g Gomega) { - configMapAudit, findErr := findAuditPayloadSince( - context.Background(), - client, - streamName, - baselineID, - 300, - func(payload map[string]interface{}) bool { - return auditPayloadMatches(payload, "", "configmaps", testNs, configMapName, "create") - }, - ) - g.Expect(findErr).NotTo(HaveOccurred()) - - flunderAudit, findErr := findAuditPayloadSince( - context.Background(), - client, - streamName, - baselineID, - 300, - func(payload map[string]interface{}) bool { - return auditPayloadMatches(payload, "wardle.example.com", "flunders", testNs, flunderName, "create") - }, - ) - g.Expect(findErr).NotTo(HaveOccurred()) - - g.Expect(auditObjectRefName(configMapAudit.Payload)).To(Equal(configMapName)) - g.Expect(auditPayloadHasObject(configMapAudit.Payload, "requestObject")).To(BeTrue()) - g.Expect(auditPayloadHasObject(configMapAudit.Payload, "responseObject")).To(BeTrue()) - g.Expect(auditObjectRefName(flunderAudit.Payload)).To(Equal(flunderName)) - g.Expect(auditPayloadHasObject(flunderAudit.Payload, "requestObject")).To(BeTrue()) - g.Expect(auditPayloadHasObject(flunderAudit.Payload, "responseObject")).To(BeTrue()) - flunderAuditID := auditPayloadID(flunderAudit.Payload) - g.Expect(flunderAuditID).NotTo(BeEmpty()) - flunderCount, countErr := countAuditPayloadsByAuditIDSince( - context.Background(), - client, - streamName, - baselineID, - flunderAuditID, - 300, - ) - g.Expect(countErr).NotTo(HaveOccurred()) - g.Expect(flunderCount).To(Equal(1), "expected one canonical stream entry for the Flunder auditID") + pullLatestRepoState(g, repo.CheckoutDir) - _, _ = fmt.Fprintf( - GinkgoWriter, - "ConfigMap create audit payload for comparison:\n%s\n", - prettyAuditPayload(configMapAudit.Payload), - ) - _, _ = fmt.Fprintf( - GinkgoWriter, - "Flunder create audit payload for comparison:\n%s\n", - prettyAuditPayload(flunderAudit.Payload), - ) - }, 30*time.Second, 2*time.Second).Should(Succeed()) + content, readErr := os.ReadFile(repoFile) + g.Expect(readErr).NotTo(HaveOccurred(), "expected Flunder file to exist after create") + g.Expect(string(content)).To(ContainSubstring("reference: shallow-check-reference")) + }, 2*time.Minute, 2*time.Second).Should(Succeed()) + + By("verifying no audit events were dropped as shallow, cluster-wide") + Consistently(func(g Gomega) { + dropped, queryErr := queryPrometheus("sum(gitopsreverser_audit_shallow_dropped_total) or vector(0)") + g.Expect(queryErr).NotTo(HaveOccurred()) + g.Expect(dropped).To(BeZero(), + "gitopsreverser_audit_shallow_dropped_total must stay at zero in a healthy e2e environment") + }, 10*time.Second, 2*time.Second).Should(Succeed()) }) It("should accept an aggregated flunder WatchRule and mark it ready", func() { @@ -292,7 +228,6 @@ spec: It("should attribute Flunder commits to the impersonated user", func() { ensureFlunderWatchRuleReady() - streamName := defaultAuditRedisStream flunderName := fmt.Sprintf("aggregated-impersonation-flunder-%d", GinkgoRandomSeed()) repoPath := path.Join( aggregatedPath, @@ -308,20 +243,6 @@ spec: reference: impersonated-reference `, flunderName, testNs) - By("connecting to Valkey through the e2e port-forward") - client := newE2EValkeyClient() - defer func() { - _ = client.Close() - }() - - Eventually(func(g Gomega) { - g.Expect(client.Ping(context.Background()).Err()).NotTo(HaveOccurred()) - }, 30*time.Second, 2*time.Second).Should(Succeed()) - - By("recording the latest audit stream entry before the impersonated create") - baselineID, err := latestAuditStreamID(context.Background(), client, streamName) - Expect(err).NotTo(HaveOccurred()) - DeferCleanup(func() { if skipCleanupBecauseResourcesArePreserved(fmt.Sprintf("Flunder %s/%s", testNs, flunderName), testNs) { return @@ -330,26 +251,9 @@ spec: }) By("creating a Flunder through the aggregated API as an impersonated user") - _, err = kubectlRunWithStdin(testNs, flunderManifest, "--as=jane@acme.com", "apply", "-f", "-") + _, err := kubectlRunWithStdin(testNs, flunderManifest, "--as=jane@acme.com", "apply", "-f", "-") Expect(err).NotTo(HaveOccurred()) - By("observing the impersonated user in the audit payload") - Eventually(func(g Gomega) { - flunderAudit, findErr := findAuditPayloadSince( - context.Background(), - client, - streamName, - baselineID, - 300, - func(payload map[string]interface{}) bool { - return auditPayloadMatches(payload, "wardle.example.com", "flunders", testNs, flunderName, "create") - }, - ) - g.Expect(findErr).NotTo(HaveOccurred()) - g.Expect(auditObjectRefName(flunderAudit.Payload)).To(Equal(flunderName)) - g.Expect(auditEffectiveUsername(flunderAudit.Payload)).To(Equal("jane@acme.com")) - }, 30*time.Second, 2*time.Second).Should(Succeed()) - By("verifying the resulting git commit uses the impersonated user as author") Eventually(func(g Gomega) { pullLatestRepoState(g, repo.CheckoutDir) @@ -468,32 +372,3 @@ spec: }, 2*time.Minute, 2*time.Second).Should(Succeed()) }) }) - -func auditEffectiveUsername(payload map[string]interface{}) string { - if impersonated := nestedAuditString(payload, "impersonatedUser", "username"); impersonated != "" { - return impersonated - } - - return nestedAuditString(payload, "user", "username") -} - -func nestedAuditString(payload map[string]interface{}, fields ...string) string { - current := any(payload) - for _, field := range fields { - nextMap, ok := current.(map[string]interface{}) - if !ok { - return "" - } - current, ok = nextMap[field] - if !ok { - return "" - } - } - - value, ok := current.(string) - if !ok { - return "" - } - - return strings.TrimSpace(value) -} diff --git a/test/e2e/audit_helpers_test.go b/test/e2e/audit_helpers_test.go index 9264d350d..d083115c8 100644 --- a/test/e2e/audit_helpers_test.go +++ b/test/e2e/audit_helpers_test.go @@ -19,223 +19,38 @@ limitations under the License. package e2e import ( - "context" - "encoding/json" "fmt" - "strings" + "sync" - "github.com/redis/go-redis/v9" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" ) -type auditPayloadEntry struct { - ID string - Payload map[string]interface{} - RawJSON string -} - -func newE2EValkeyClient() *redis.Client { - return redis.NewClient(&redis.Options{ - Addr: valkeyPortForwardAddr(), - Password: valkeyPortForwardPassword(), - }) -} - -func latestAuditStreamID(ctx context.Context, client *redis.Client, stream string) (string, error) { - entries, err := client.XRevRangeN(ctx, stream, "+", "-", 1).Result() - if err != nil { - return "", err - } - if len(entries) == 0 { - return "", nil - } - - return entries[0].ID, nil -} - -func findAuditPayloadSince( - ctx context.Context, - client *redis.Client, - stream, afterID string, - limit int64, - match func(map[string]interface{}) bool, -) (auditPayloadEntry, error) { - start := "-" - if afterID != "" { - start = "(" + afterID - } - - entries, err := client.XRangeN(ctx, stream, start, "+", limit).Result() - if err != nil { - return auditPayloadEntry{}, err - } - - for _, entry := range entries { - rawJSON, ok := entry.Values["payload_json"].(string) - if !ok || rawJSON == "" { - continue - } - - var payload map[string]interface{} - if err := json.Unmarshal([]byte(rawJSON), &payload); err != nil { - return auditPayloadEntry{}, fmt.Errorf("decode payload_json for stream entry %s: %w", entry.ID, err) - } - if match(payload) { - return auditPayloadEntry{ - ID: entry.ID, - Payload: payload, - RawJSON: rawJSON, - }, nil - } - } - - return auditPayloadEntry{}, fmt.Errorf( - "no matching audit payload found after scanning %d entries from %q: %s", - len(entries), - start, - summarizeAuditPayloadEntries(entries, 12), - ) -} - -func countAuditPayloadsByAuditIDSince( - ctx context.Context, - client *redis.Client, - stream, afterID, auditID string, - limit int64, -) (int, error) { - start := "-" - if afterID != "" { - start = "(" + afterID - } - - entries, err := client.XRangeN(ctx, stream, start, "+", limit).Result() - if err != nil { - return 0, err - } - - count := 0 - for _, entry := range entries { - value, _ := entry.Values["audit_id"].(string) - if value == auditID { - count++ - } - } - return count, nil -} - -func auditPayloadID(payload map[string]interface{}) string { - value, _ := payload["auditID"].(string) - return value -} - -func auditPayloadMatches( - payload map[string]interface{}, - apiGroup, resource, namespace, name, verb string, -) bool { - objectRef, _ := payload["objectRef"].(map[string]interface{}) - if objectRef == nil { - return false - } - if value, _ := payload["verb"].(string); verb != "" && value != verb { - return false - } - if value, _ := objectRef["apiGroup"].(string); apiGroup != value { - return false - } - if value, _ := objectRef["resource"].(string); resource != value { - return false - } - if value, _ := objectRef["namespace"].(string); namespace != value { - return false - } - if name == "" { - return true - } - - value, _ := objectRef["name"].(string) - return name == value -} - -func auditObjectRefName(payload map[string]interface{}) string { - objectRef, _ := payload["objectRef"].(map[string]interface{}) - if objectRef == nil { - return "" - } - - value, _ := objectRef["name"].(string) - return value -} +var ( + // auditConsumerRepo holds the repo fixtures shared by the audit-consumer specs. + auditConsumerRepo *RepoArtifacts + auditConsumerRepoOnce sync.Once +) -func auditPayloadHasObject(payload map[string]interface{}, field string) bool { - object, _ := payload[field].(map[string]interface{}) - return len(object) > 0 -} +// ensureAuditConsumerRepo lazily provisions the shared Gitea repo used by the +// audit-consumer specs (commit-window batching, commit request). +func ensureAuditConsumerRepo() *RepoArtifacts { + GinkgoHelper() -func summarizeAuditPayloadEntries(entries []redis.XMessage, maxEntries int) string { - if len(entries) == 0 { - return "" - } - if maxEntries <= 0 { - maxEntries = len(entries) - } - start := 0 - if len(entries) > maxEntries { - start = len(entries) - maxEntries - } + auditConsumerRepoOnce.Do(func() { + consumerNs := testNamespaceFor("audit-consumer") - var builder strings.Builder - if start > 0 { - _, _ = fmt.Fprintf(&builder, "... %d earlier entries omitted; ", start) - } - for i, entry := range entries[start:] { - if i > 0 { - builder.WriteString("; ") - } - rawJSON, _ := entry.Values["payload_json"].(string) - if rawJSON == "" { - _, _ = fmt.Fprintf(&builder, "%s ", entry.ID) - continue - } + By("creating the audit consumer namespace for shared repo fixtures") + _, _ = kubectlRun("create", "namespace", consumerNs) - var payload map[string]interface{} - if err := json.Unmarshal([]byte(rawJSON), &payload); err != nil { - _, _ = fmt.Fprintf(&builder, "%s ", entry.ID, err) - continue - } - _, _ = fmt.Fprintf( - &builder, - "%s auditID=%s verb=%s ref=%s/%s ns=%s name=%s requestObject=%t responseObject=%t", - entry.ID, - auditPayloadID(payload), - auditPayloadString(payload, "verb"), - auditPayloadObjectRefString(payload, "apiGroup"), - auditPayloadObjectRefString(payload, "resource"), - auditPayloadObjectRefString(payload, "namespace"), - auditPayloadObjectRefString(payload, "name"), - auditPayloadHasObject(payload, "requestObject"), - auditPayloadHasObject(payload, "responseObject"), + By("setting up the shared Gitea repo for audit-consumer tests") + auditConsumerRepo = SetupRepo( + resolveE2EContext(), + consumerNs, + fmt.Sprintf("e2e-audit-consumer-%d", GinkgoRandomSeed()), ) - } - return builder.String() -} - -func auditPayloadString(payload map[string]interface{}, field string) string { - value, _ := payload[field].(string) - return value -} - -func auditPayloadObjectRefString(payload map[string]interface{}, field string) string { - objectRef, _ := payload["objectRef"].(map[string]interface{}) - if objectRef == nil { - return "" - } - value, _ := objectRef[field].(string) - return value -} + }) -func prettyAuditPayload(payload map[string]interface{}) string { - formatted, err := json.MarshalIndent(payload, "", " ") - if err != nil { - return fmt.Sprintf("", err) - } - return string(formatted) + Expect(auditConsumerRepo).NotTo(BeNil(), "expected audit consumer repo fixtures to be initialised") + return auditConsumerRepo } diff --git a/test/e2e/audit_redis_e2e_test.go b/test/e2e/audit_redis_e2e_test.go deleted file mode 100644 index e89cac519..000000000 --- a/test/e2e/audit_redis_e2e_test.go +++ /dev/null @@ -1,381 +0,0 @@ -/* -SPDX-License-Identifier: Apache-2.0 - -Copyright 2025 ConfigButler - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package e2e - -import ( - "context" - "fmt" - "os" - "os/exec" - "path" - "path/filepath" - "strings" - "sync" - "time" - - . "github.com/onsi/ginkgo/v2" - . "github.com/onsi/gomega" - "github.com/redis/go-redis/v9" -) - -const ( - defaultAuditRedisStream = "gitopsreverser.audit.events.v1" - defaultE2EValkeyPassword = "e2e-valkey-password" -) - -var ( - // auditRedisRepo holds the file-local repo fixtures for both audit-redis Describe blocks. - auditRedisRepo *RepoArtifacts - auditRedisRepoOnce sync.Once -) - -func ensureAuditRedisRepo() *RepoArtifacts { - GinkgoHelper() - - auditRedisRepoOnce.Do(func() { - consumerNs := testNamespaceFor("audit-consumer") - - By("creating the audit consumer namespace for shared repo fixtures") - _, _ = kubectlRun("create", "namespace", consumerNs) - - By("setting up the shared Gitea repo for audit-redis tests") - auditRedisRepo = SetupRepo( - resolveE2EContext(), - consumerNs, - fmt.Sprintf("e2e-audit-redis-%d", GinkgoRandomSeed()), - ) - }) - - Expect(auditRedisRepo).NotTo(BeNil(), "expected audit Redis repo fixtures to be initialised") - return auditRedisRepo -} - -// Serial: shares the single global audit pipeline (audit webhook → Redis stream -// → consumer) with every other audit-redis spec; concurrent audit traffic -// pollutes its exclusivity assertions. See docs/design/e2e-serial-registry.md. -var _ = Describe("Audit Redis Queue", Label("audit-redis", "smoke"), Serial, Ordered, func() { - var testNs string - - BeforeAll(func() { - By("creating producer test namespace") - testNs = testNamespaceFor("audit-redis") - _, _ = kubectlRun("create", "namespace", testNs) // idempotent; ignore AlreadyExists - - repo := ensureAuditRedisRepo() - - By("applying git secrets to producer test namespace") - _, err := kubectlRunInNamespace(testNs, "apply", "-f", repo.SecretsYAML) - Expect(err).NotTo(HaveOccurred(), "failed to apply git secrets to producer test namespace") - }) - - AfterAll(func() { - cleanupNamespace(testNs) - }) - - It("should enqueue incoming audit webhook events into a Redis stream", func() { - streamName := defaultAuditRedisStream - testConfigMapName := fmt.Sprintf("audit-redis-test-cm-%d", GinkgoRandomSeed()) - - By("connecting to Valkey through the e2e port-forward") - client := redis.NewClient(&redis.Options{ - Addr: valkeyPortForwardAddr(), - Password: valkeyPortForwardPassword(), - }) - Eventually(func(g Gomega) { - g.Expect(client.Ping(context.Background()).Err()).NotTo(HaveOccurred()) - }, 30*time.Second, 2*time.Second).Should(Succeed()) - - By("triggering kube-apiserver audit events with a ConfigMap create/update") - _, err := kubectlRunInNamespace( - testNs, - "delete", - "configmap", - testConfigMapName, - "--ignore-not-found=true", - ) - Expect(err).NotTo(HaveOccurred()) - - _, err = kubectlRunInNamespace( - testNs, - "create", - "configmap", - testConfigMapName, - "--from-literal=test=audit-redis", - ) - Expect(err).NotTo(HaveOccurred()) - - _, err = kubectlRunInNamespace( - testNs, - "patch", - "configmap", - testConfigMapName, - "--type=merge", - "--patch", - `{"data":{"test":"audit-redis-updated"}}`, - ) - Expect(err).NotTo(HaveOccurred()) - - By("verifying Redis stream receives an entry for the ConfigMap audit event") - Eventually(func(g Gomega) { - entries, readErr := client.XRange(context.Background(), streamName, "-", "+").Result() - g.Expect(readErr).NotTo(HaveOccurred()) - g.Expect(entries).NotTo(BeEmpty()) - - found := false - for _, entry := range entries { - resource, _ := entry.Values["resource"].(string) - name, _ := entry.Values["name"].(string) - payload, _ := entry.Values["payload_json"].(string) - if resource == "configmaps" && name == testConfigMapName && payload != "" { - found = true - break - } - } - g.Expect(found).To(BeTrue(), "expected ConfigMap audit event to be enqueued in Redis stream") - }, 2*time.Minute, 2*time.Second).Should(Succeed()) - - By("cleaning up test ConfigMap") - _, _ = kubectlRunInNamespace( - testNs, - "delete", - "configmap", - testConfigMapName, - "--ignore-not-found=true", - ) - }) -}) - -// auditConsumerDescribe holds the consumer-path e2e test. -// Label: audit-redis (runs together with the producer test via task test-e2e-audit-redis). -// Serial: shares the single global audit pipeline with every other audit-redis -// spec. See docs/design/e2e-serial-registry.md. -var _ = Describe("Audit Redis Consumer", Label("audit-redis", "smoke"), Serial, Ordered, func() { - var ( - testNs string - valkeyClient *redis.Client - gitProvName string - gitTargetName string - watchRuleName string - cmName string - ) - - BeforeAll(func() { - By("creating consumer test namespace and applying git secrets") - testNs = testNamespaceFor("audit-consumer") - repo := ensureAuditRedisRepo() - _, err := kubectlRunInNamespace(testNs, "apply", "-f", repo.SecretsYAML) - Expect(err).NotTo(HaveOccurred(), "failed to apply git secrets to consumer test namespace") - applySOPSAgeKeyToNamespace(testNs) - - By("connecting to Valkey through the e2e port-forward") - valkeyClient = redis.NewClient(&redis.Options{ - Addr: valkeyPortForwardAddr(), - Password: valkeyPortForwardPassword(), - }) - Eventually(func(g Gomega) { - g.Expect(valkeyClient.Ping(context.Background()).Err()).NotTo(HaveOccurred()) - }, 30*time.Second, 2*time.Second).Should(Succeed()) - - By("setting up GitProvider, GitTarget and WatchRule for consumer test") - seed := GinkgoRandomSeed() - gitProvName = fmt.Sprintf("audit-consumer-gitprovider-%d", seed) - gitTargetName = fmt.Sprintf("audit-consumer-gittarget-%d", seed) - watchRuleName = fmt.Sprintf("audit-consumer-watchrule-%d", seed) - cmName = fmt.Sprintf("audit-consumer-cm-%d", seed) - - createGitProviderWithURLInNamespace( - gitProvName, - testNs, - auditRedisRepo.GitSecretHTTP, - auditRedisRepo.RepoURLHTTP, - ) - verifyResourceStatus("gitprovider", gitProvName, testNs, "True", "Ready", "") - - createGitTarget(gitTargetName, testNs, gitProvName, "e2e/audit-consumer-test", "main") - verifyResourceStatus("gittarget", gitTargetName, testNs, "True", "Ready", "") - - data := struct { - Name string - Namespace string - DestinationName string - }{ - Name: watchRuleName, - Namespace: testNs, - DestinationName: gitTargetName, - } - err = applyFromTemplate("test/e2e/templates/manager/watchrule-configmap.tmpl", data, testNs) - Expect(err).NotTo(HaveOccurred(), "failed to apply WatchRule") - verifyResourceStatus("watchrule", watchRuleName, testNs, "True", "Ready", "") - }) - - AfterAll(func() { - cleanupWatchRule(watchRuleName, testNs) - cleanupGitTarget(gitTargetName, testNs) - cleanupNamespacedResource(testNs, "gitprovider", gitProvName) - cleanupNamespace(testNs) - if valkeyClient != nil { - _ = valkeyClient.Close() - } - }) - - It("should produce a Git commit from an audit stream event consumed by the consumer", func() { - streamName := defaultAuditRedisStream - - By("creating a ConfigMap to produce an audit event") - cmData := struct { - Name string - Namespace string - }{Name: cmName, Namespace: testNs} - err := applyFromTemplate("test/e2e/templates/manager/configmap.tmpl", cmData, testNs) - Expect(err).NotTo(HaveOccurred(), "failed to create test ConfigMap") - - By("waiting for the audit stream entry to appear (producer path)") - // cmName includes GinkgoRandomSeed so it uniquely identifies this test run's entry. - Eventually(func(g Gomega) { - entries, readErr := valkeyClient.XRange(context.Background(), streamName, "-", "+").Result() - g.Expect(readErr).NotTo(HaveOccurred()) - found := false - for _, entry := range entries { - resource, _ := entry.Values["resource"].(string) - name, _ := entry.Values["name"].(string) - if resource == "configmaps" && name == cmName { - found = true - break - } - } - g.Expect(found).To(BeTrue(), "audit stream entry for ConfigMap should appear") - }, 2*time.Minute, 2*time.Second).Should(Succeed()) - - By("waiting for the Git commit to appear (consumer path)") - expectedFile := filepath.Join(auditRedisRepo.CheckoutDir, - "e2e/audit-consumer-test", - fmt.Sprintf("v1/configmaps/%s/%s.yaml", testNs, cmName)) - - Eventually(func(g Gomega) { - pullLatestRepoState(g, auditRedisRepo.CheckoutDir) - - info, statErr := os.Stat(expectedFile) - g.Expect(statErr).NotTo(HaveOccurred(), - fmt.Sprintf("ConfigMap file should exist at %s", expectedFile)) - g.Expect(info.Size()).To(BeNumerically(">", 0)) - - expectedRepoPath := path.Join( - "e2e/audit-consumer-test", - fmt.Sprintf("v1/configmaps/%s/%s.yaml", testNs, cmName), - ) - assertLatestCommitTouchesOnly(g, auditRedisRepo.CheckoutDir, []string{expectedRepoPath}) - assertLatestCommitTouchesNoNamespaces( - g, - auditRedisRepo.CheckoutDir, - "e2e/audit-consumer-test/v1/configmaps", - []string{ - "gitops-reverser", - "flux-system", - "kube-system", - "tilt-playground", - }, - ) - - authorCmd := exec.Command("git", "log", "-1", "--pretty=%an") - authorCmd.Dir = auditRedisRepo.CheckoutDir - authorOut, authorErr := authorCmd.CombinedOutput() - g.Expect(authorErr).NotTo(HaveOccurred(), - fmt.Sprintf("git log author failed: %s", string(authorOut))) - g.Expect(string(authorOut)).To(ContainSubstring("system:admin")) - }, 3*time.Minute, 3*time.Second).Should(Succeed()) - - By("cleaning up test ConfigMap") - _, _ = kubectlRunInNamespace(testNs, "delete", "configmap", cmName, "--ignore-not-found=true") - }) - - It("should attribute the commit to the OIDC display name and email from user.extra", func() { - const ( - oidcDisplayName = "Simon Koudijs" - oidcEmail = "something@configbutler.ai" - ) - oidcCMName := fmt.Sprintf("audit-oidc-cm-%d", GinkgoRandomSeed()) - - By("creating a ConfigMap while impersonating an OIDC user carrying name/email extras") - // Impersonation makes the real kube-apiserver emit an audit event whose - // impersonatedUser.Extra carries these keys, mimicking the claims a - // structured authentication config maps into user.extra for an OIDC - // login. system:masters keeps the impersonated create authorized - // without provisioning per-user RBAC. - err := createConfigMapAsImpersonatedUser( - testNs, - oidcCMName, - "oidc-simon", - []string{"system:masters"}, - map[string][]string{ - "configbutler.ai/claims/display-name": {oidcDisplayName}, - "configbutler.ai/claims/email": {oidcEmail}, - }, - ) - Expect(err).NotTo(HaveOccurred(), "failed to create impersonated ConfigMap") - - repoPath := path.Join( - "e2e/audit-consumer-test", - fmt.Sprintf("v1/configmaps/%s/%s.yaml", testNs, oidcCMName), - ) - expectedFile := filepath.Join(auditRedisRepo.CheckoutDir, repoPath) - - By("waiting for the commit and asserting its author is the OIDC identity") - Eventually(func(g Gomega) { - pullLatestRepoState(g, auditRedisRepo.CheckoutDir) - - info, statErr := os.Stat(expectedFile) - g.Expect(statErr).NotTo(HaveOccurred(), - fmt.Sprintf("ConfigMap file should exist at %s", expectedFile)) - g.Expect(info.Size()).To(BeNumerically(">", 0)) - - // Scope the log to this run's unique file path so the author read - // back is unambiguously the commit produced by the impersonated - // create, regardless of commit-window coalescing. - authorCmd := exec.Command( - "git", "log", "-1", "--pretty=%an <%ae>", "--", repoPath, - ) - authorCmd.Dir = auditRedisRepo.CheckoutDir - authorOut, authorErr := authorCmd.CombinedOutput() - g.Expect(authorErr).NotTo(HaveOccurred(), - fmt.Sprintf("git log author failed: %s", string(authorOut))) - g.Expect(strings.TrimSpace(string(authorOut))).To( - Equal(fmt.Sprintf("%s <%s>", oidcDisplayName, oidcEmail)), - "commit author should be the OIDC display name and email from user.extra") - }, 3*time.Minute, 3*time.Second).Should(Succeed()) - - By("cleaning up the test ConfigMap") - _, _ = kubectlRunInNamespace(testNs, "delete", "configmap", oidcCMName, "--ignore-not-found=true") - }) -}) - -func valkeyPortForwardAddr() string { - port := strings.TrimSpace(os.Getenv("VALKEY_PORT")) - if port == "" { - port = "16379" - } - return "127.0.0.1:" + port -} - -func valkeyPortForwardPassword() string { - password := strings.TrimSpace(os.Getenv("E2E_VALKEY_PASSWORD")) - if password == "" { - return defaultE2EValkeyPassword - } - return password -} diff --git a/test/e2e/commit_author_attribution_e2e_test.go b/test/e2e/commit_author_attribution_e2e_test.go new file mode 100644 index 000000000..cac8bb352 --- /dev/null +++ b/test/e2e/commit_author_attribution_e2e_test.go @@ -0,0 +1,177 @@ +/* +SPDX-License-Identifier: Apache-2.0 + +Copyright 2025 ConfigButler + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "fmt" + "os" + "path" + "path/filepath" + "strings" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +// Commit Author Attribution proves the end-to-end identity chain: a write made +// by an impersonated user whose audit event carries OIDC name/email claims in +// user.extra ends up as the Git commit author. The kube-apiserver emits the +// real audit event, the controller's audit pipeline consumes it, and the +// resulting commit must be attributed to that identity. The author-mapping +// *logic* is unit-tested (internal/queue redis_audit_consumer_test.go +// TestResolveUserInfo); this spec is the only place that proves the full +// apiserver → audit webhook → consumer → git-author wiring carries the keys +// through. +// +// Not Serial: the dedicated GitProvider uses a 0s commit window, so every +// watched event is committed immediately as its own commit (no batching). Each +// assertion reads the author scoped to its own unique file path, so concurrent +// audit traffic from other specs lands in separate commits and cannot change +// the author of this spec's commit. +var _ = Describe("Commit Author Attribution", Label("manager", "smoke"), Ordered, func() { + var ( + testNs string + repo *RepoArtifacts + gitProvName string + gitTargetName string + watchRuleName string + ) + + const basePath = "e2e/commit-author-test" + + BeforeAll(func() { + By("creating commit-author test namespace") + testNs = testNamespaceFor("commit-author") + _, _ = kubectlRun("create", "namespace", testNs) // idempotent; ignore AlreadyExists + + By("setting up the Gitea repo and credentials") + repo = SetupRepo( + resolveE2EContext(), + testNs, + fmt.Sprintf("e2e-commit-author-%d", GinkgoRandomSeed()), + ) + _, err := kubectlRunInNamespace(testNs, "apply", "-f", repo.SecretsYAML) + Expect(err).NotTo(HaveOccurred(), "failed to apply git secrets to test namespace") + applySOPSAgeKeyToNamespace(testNs) + + By("setting up GitProvider (0s commit window), GitTarget and WatchRule") + seed := GinkgoRandomSeed() + gitProvName = fmt.Sprintf("commit-author-gitprovider-%d", seed) + gitTargetName = fmt.Sprintf("commit-author-gittarget-%d", seed) + watchRuleName = fmt.Sprintf("commit-author-watchrule-%d", seed) + + // createGitProviderWithURLInNamespace uses a 0s commit window, so each + // event commits immediately and never coalesces with another's. + createGitProviderWithURLInNamespace(gitProvName, testNs, repo.GitSecretHTTP, repo.RepoURLHTTP) + verifyResourceStatus("gitprovider", gitProvName, testNs, "True", "Ready", "") + + createGitTarget(gitTargetName, testNs, gitProvName, basePath, "main") + verifyResourceStatus("gittarget", gitTargetName, testNs, "True", "Ready", "") + + data := struct { + Name string + Namespace string + DestinationName string + }{ + Name: watchRuleName, + Namespace: testNs, + DestinationName: gitTargetName, + } + err = applyFromTemplate("test/e2e/templates/manager/watchrule-configmap.tmpl", data, testNs) + Expect(err).NotTo(HaveOccurred(), "failed to apply WatchRule") + verifyResourceStatus("watchrule", watchRuleName, testNs, "True", "Ready", "") + }) + + AfterAll(func() { + cleanupWatchRule(watchRuleName, testNs) + cleanupGitTarget(gitTargetName, testNs) + cleanupNamespacedResource(testNs, "gitprovider", gitProvName) + cleanupNamespace(testNs) + }) + + // assertCommitAuthorFromOIDCClaims creates a ConfigMap while impersonating a + // user that carries the given OIDC display-name/email claims in user.extra, + // then asserts the commit for that ConfigMap's unique path is authored by + // " ". + assertCommitAuthorFromOIDCClaims := func(cmName, asUser, displayName, email string) { + GinkgoHelper() + + By(fmt.Sprintf("creating ConfigMap %q while impersonating %q with OIDC name/email extras", cmName, asUser)) + // Impersonation makes the real kube-apiserver emit an audit event whose + // impersonatedUser.Extra carries these keys, mimicking the claims a + // structured authentication config maps into user.extra for an OIDC + // login. system:masters keeps the impersonated create authorized + // without provisioning per-user RBAC. + err := createConfigMapAsImpersonatedUser( + testNs, + cmName, + asUser, + []string{"system:masters"}, + map[string][]string{ + "configbutler.ai/claims/display-name": {displayName}, + "configbutler.ai/claims/email": {email}, + }, + ) + Expect(err).NotTo(HaveOccurred(), "failed to create impersonated ConfigMap") + + repoPath := path.Join(basePath, fmt.Sprintf("v1/configmaps/%s/%s.yaml", testNs, cmName)) + expectedFile := filepath.Join(repo.CheckoutDir, repoPath) + + By("waiting for the commit and asserting its author is the OIDC identity") + Eventually(func(g Gomega) { + pullLatestRepoState(g, repo.CheckoutDir) + + info, statErr := os.Stat(expectedFile) + g.Expect(statErr).NotTo(HaveOccurred(), + fmt.Sprintf("ConfigMap file should exist at %s", expectedFile)) + g.Expect(info.Size()).To(BeNumerically(">", 0)) + + // Scope the log to this ConfigMap's unique path so the author read + // back is unambiguously the commit produced by this impersonated + // create, independent of any concurrent audit traffic. + out, logErr := gitRun(repo.CheckoutDir, "log", "-1", "--pretty=%an <%ae>", "--", repoPath) + g.Expect(logErr).NotTo(HaveOccurred(), fmt.Sprintf("git log author failed: %s", out)) + g.Expect(strings.TrimSpace(out)).To( + Equal(fmt.Sprintf("%s <%s>", displayName, email)), + "commit author should be the OIDC display name and email from user.extra") + }, 3*time.Minute, 3*time.Second).Should(Succeed()) + + By("cleaning up the test ConfigMap") + _, _ = kubectlRunInNamespace(testNs, "delete", "configmap", cmName, "--ignore-not-found=true") + } + + It("attributes a commit to the OIDC display name and email from user.extra", func() { + assertCommitAuthorFromOIDCClaims( + fmt.Sprintf("commit-author-oidc-cm-%d", GinkgoRandomSeed()), + "oidc-simon", + "Simon Koudijs", + "something@configbutler.ai", + ) + }) + + It("attributes a second commit to a different OIDC identity (claims are data-driven)", func() { + assertCommitAuthorFromOIDCClaims( + fmt.Sprintf("commit-author-oidc-cm2-%d", GinkgoRandomSeed()), + "oidc-ada", + "Ada Lovelace", + "ada@configbutler.ai", + ) + }) +}) diff --git a/test/e2e/commit_request_e2e_test.go b/test/e2e/commit_request_e2e_test.go index d5db3cdc6..4355328f9 100644 --- a/test/e2e/commit_request_e2e_test.go +++ b/test/e2e/commit_request_e2e_test.go @@ -19,7 +19,6 @@ limitations under the License. package e2e import ( - "context" "fmt" "os" "path/filepath" @@ -36,10 +35,10 @@ import ( // a deliberately long commitWindow so that, without the CommitRequest, the // edit would not be committed for minutes — observing the commit promptly is // what proves the commit-request path works. -// Serial: shares the single global audit pipeline (audit webhook → Redis stream -// → consumer) with every other audit-redis spec. See +// Serial: shares the single global audit pipeline with every other +// audit-consumer spec. See // docs/design/e2e-serial-registry.md. -var _ = Describe("Commit Request", Label("commit-request", "audit-redis", "smoke"), Serial, Ordered, func() { +var _ = Describe("Commit Request", Label("commit-request", "audit-consumer", "smoke"), Serial, Ordered, func() { var ( testNs string gitProvName string @@ -55,7 +54,7 @@ var _ = Describe("Commit Request", Label("commit-request", "audit-redis", "smoke By("creating commit-request test namespace and applying git secrets") testNs = testNamespaceFor("commit-request") _, _ = kubectlRun("create", "namespace", testNs) - repo := ensureAuditRedisRepo() + repo := ensureAuditConsumerRepo() _, err := kubectlRunInNamespace(testNs, "apply", "-f", repo.SecretsYAML) Expect(err).NotTo(HaveOccurred(), "failed to apply git secrets to namespace") applySOPSAgeKeyToNamespace(testNs) @@ -94,7 +93,7 @@ var _ = Describe("Commit Request", Label("commit-request", "audit-redis", "smoke }) It("finalizes the open commit window on demand and reports the resulting SHA", func() { - repo := auditRedisRepo + repo := auditConsumerRepo basePath := "e2e/commit-request-test" seed := GinkgoRandomSeed() cmName := fmt.Sprintf("commit-request-cm-%d", seed) @@ -104,26 +103,6 @@ var _ = Describe("Commit Request", Label("commit-request", "audit-redis", "smoke By("editing a ConfigMap to open a commit window") applyConfigMap(testNs, cmName) - By("waiting for the ConfigMap audit event to reach the Redis stream") - // Confirming the producer side guarantees the edit is observable by the - // consumer before the CommitRequest's own (later) audit event. - valkeyClient := newE2EValkeyClient() - defer func() { _ = valkeyClient.Close() }() - Eventually(func(g Gomega) { - entries, readErr := valkeyClient.XRange(context.Background(), defaultAuditRedisStream, "-", "+").Result() - g.Expect(readErr).NotTo(HaveOccurred()) - found := false - for _, entry := range entries { - resource, _ := entry.Values["resource"].(string) - name, _ := entry.Values["name"].(string) - if resource == "configmaps" && name == cmName { - found = true - break - } - } - g.Expect(found).To(BeTrue(), "ConfigMap audit event should appear in the stream") - }, 2*time.Minute, 2*time.Second).Should(Succeed()) - By("confirming the edit has NOT been committed yet (long commitWindow)") Consistently(func(g Gomega) { pullLatestRepoState(g, repo.CheckoutDir) @@ -180,7 +159,7 @@ var _ = Describe("Commit Request", Label("commit-request", "audit-redis", "smoke // from responseObject. Without the fix the CommitRequest stays stuck in // WaitingForAuditEvent forever. It("finalizes a CommitRequest created with metadata.generateName", func() { - repo := auditRedisRepo + repo := auditConsumerRepo basePath := "e2e/commit-request-test" seed := GinkgoRandomSeed() cmName := fmt.Sprintf("commit-request-gen-cm-%d", seed) diff --git a/test/e2e/commit_window_batching_e2e_test.go b/test/e2e/commit_window_batching_e2e_test.go index e011dea89..4be6d5330 100644 --- a/test/e2e/commit_window_batching_e2e_test.go +++ b/test/e2e/commit_window_batching_e2e_test.go @@ -32,14 +32,14 @@ import ( // Commit-window batching exercises the grouped-commit path: multiple events // arriving within the rolling silence window collapse into one grouped commit -// and one push. The audit-redis consumer pipeline is the real path under test -// — events flow kubectl → audit webhook → Valkey stream → consumer → +// and one push. The audit consumer pipeline is the real path under test: +// events flow kubectl → audit webhook → consumer → // BranchWorker.commitGroups → BranchWorker.pushPendingCommits. -// Serial: shares the single global audit pipeline (audit webhook → Redis stream -// → consumer) with every other audit-redis spec. See +// Serial: shares the single global audit pipeline with every other +// audit-consumer spec. See // docs/design/e2e-serial-registry.md. var _ = Describe("Commit Window Batching", - Label("commit-window-batching", "audit-redis", "smoke"), Serial, Ordered, func() { + Label("commit-window-batching", "audit-consumer", "smoke"), Serial, Ordered, func() { var ( testNs string gitProvName string @@ -53,7 +53,7 @@ var _ = Describe("Commit Window Batching", By("creating commit-window-batching test namespace and applying git secrets") testNs = testNamespaceFor("commit-window-batching") _, _ = kubectlRun("create", "namespace", testNs) - repo := ensureAuditRedisRepo() + repo := ensureAuditConsumerRepo() _, err := kubectlRunInNamespace(testNs, "apply", "-f", repo.SecretsYAML) Expect(err).NotTo(HaveOccurred(), "failed to apply git secrets to namespace") applySOPSAgeKeyToNamespace(testNs) @@ -100,7 +100,7 @@ var _ = Describe("Commit Window Batching", It("collapses a burst of events into one grouped commit and one push", func() { const burstSize = 4 - repo := auditRedisRepo + repo := auditConsumerRepo basePath := "e2e/commit-window-test" seed := GinkgoRandomSeed() burstPrefix := fmt.Sprintf("commit-window-burst-%d", seed) diff --git a/test/e2e/repo_assertions_test.go b/test/e2e/repo_assertions_test.go index 1fbf81dae..ec53cfeba 100644 --- a/test/e2e/repo_assertions_test.go +++ b/test/e2e/repo_assertions_test.go @@ -50,17 +50,6 @@ func pullLatestRepoState(g Gomega, checkoutDir string) { } } -func assertLatestCommitTouchesOnly(g Gomega, checkoutDir string, expectedPaths []string) { - GinkgoHelper() - - actualPaths := commitTouchedPaths(g, checkoutDir, "HEAD") - expected := append([]string(nil), expectedPaths...) - sort.Strings(expected) - - g.Expect(actualPaths).To(Equal(expected), - fmt.Sprintf("Latest commit should touch only expected paths. Actual: %v", actualPaths)) -} - func assertLatestCommitForPathTouchesOnlyWithOptional( g Gomega, checkoutDir string, diff --git a/test/e2e/setup/flux/releases/ingress.yaml b/test/e2e/setup/demo-only/flux/ingress.yaml similarity index 100% rename from test/e2e/setup/flux/releases/ingress.yaml rename to test/e2e/setup/demo-only/flux/ingress.yaml diff --git a/test/e2e/setup/flux/releases/kro.yaml b/test/e2e/setup/demo-only/flux/kro.yaml similarity index 100% rename from test/e2e/setup/flux/releases/kro.yaml rename to test/e2e/setup/demo-only/flux/kro.yaml diff --git a/test/e2e/setup/demo-only/flux/kustomization.yaml b/test/e2e/setup/demo-only/flux/kustomization.yaml new file mode 100644 index 000000000..da52112be --- /dev/null +++ b/test/e2e/setup/demo-only/flux/kustomization.yaml @@ -0,0 +1,12 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +# Traefik and kro are only needed by the demo-only flows (podinfo IngressRoutes +# and kro ResourceGraphDefinitions). They are applied and waited on by the +# _demo-ready task, kept out of the core bring-up (test/e2e/setup/flux) so a +# slow/failed install never blocks the measured e2e suites. Both charts default +# to rendering a ServiceMonitor; keeping them here also means the core path no +# longer races the prometheus-operator CRD install. +resources: + - ingress.yaml + - kro.yaml diff --git a/test/e2e/setup/flux/kustomization.yaml b/test/e2e/setup/flux/kustomization.yaml index 72d222841..4ee832b86 100644 --- a/test/e2e/setup/flux/kustomization.yaml +++ b/test/e2e/setup/flux/kustomization.yaml @@ -12,12 +12,10 @@ resources: - repositories/jetstack.yaml - repositories/gitea-charts.yaml - repositories/valkey.yaml - - releases/ingress.yaml - releases/prometheus-operator.yaml - releases/cert-manager.yaml - releases/gitea.yaml - releases/valkey.yaml - - releases/kro.yaml - releases/reflector.yaml generatorOptions: diff --git a/test/e2e/setup/flux/releases/prometheus-operator.yaml b/test/e2e/setup/flux/releases/prometheus-operator.yaml index 880c732df..dd393f382 100644 --- a/test/e2e/setup/flux/releases/prometheus-operator.yaml +++ b/test/e2e/setup/flux/releases/prometheus-operator.yaml @@ -5,6 +5,13 @@ metadata: namespace: flux-system spec: interval: 30m + # The monitoring.coreos.com CRDs have very large schemas and are slow to reach + # Established; with wait:true the first reconcile can time out finding them + # 'InProgress'. Without retryInterval the next attempt is a full interval (30m) + # away — longer than the bring-up budget — so a transient establishment lag + # becomes a hard failure (wait-flux-services.sh times out on this Kustomization). + # Retry quickly so the CRDs settle within the bring-up window instead. + retryInterval: 20s timeout: 5m path: ./ prune: true From 903d7920162ecd56b9d863666959fc7f7d77e3e1 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Tue, 2 Jun 2026 11:50:47 +0000 Subject: [PATCH 35/40] fix: upgrade to latest a apiservice-audit-proxy should resolve our non 0 shallow events, also included tests to verify that we don't misread deletecollection events as shallow --- .devcontainer/Dockerfile | 2 +- .../audit_joiner_deletecollection_test.go | 250 ++++++++++++++++++ ...-deletecollection-official-raw-hollow.json | 46 ++++ ...tecollection-official-teardown-hollow.json | 48 ++++ ...t-deletecollection-proxy-raw-listbody.json | 129 +++++++++ ...llection-proxy-teardown-deleteoptions.json | 49 ++++ test/e2e/aggregated_apiserver_e2e_test.go | 9 + .../setup/flux/releases/aggregated-api.yaml | 2 +- 8 files changed, 533 insertions(+), 2 deletions(-) create mode 100644 internal/webhook/audit_joiner_deletecollection_test.go create mode 100644 internal/webhook/testdata/audit-events/audit-deletecollection-official-raw-hollow.json create mode 100644 internal/webhook/testdata/audit-events/audit-deletecollection-official-teardown-hollow.json create mode 100644 internal/webhook/testdata/audit-events/audit-deletecollection-proxy-raw-listbody.json create mode 100644 internal/webhook/testdata/audit-events/audit-deletecollection-proxy-teardown-deleteoptions.json diff --git a/.devcontainer/Dockerfile b/.devcontainer/Dockerfile index 14a5087f2..ab551b59a 100644 --- a/.devcontainer/Dockerfile +++ b/.devcontainer/Dockerfile @@ -44,7 +44,7 @@ RUN apt-get update \ # KUSTOMIZE_VERSION -> https://github.com/kubernetes-sigs/kustomize/releases # KUBEBUILDER_VERSION -> https://github.com/kubernetes-sigs/kubebuilder # GOLANGCI_LINT_VERSION -> https://github.com/golangci/golangci-lint/releases -# K3D_VERSION -> https://github.com/k3d-io/k3d (seems a bit stsale?) +# K3D_VERSION -> https://github.com/k3d-io/k3d # FLUX_VERSION -> https://github.com/fluxcd/flux2 # FLUX_OPERATOR_VERSION -> https://github.com/controlplaneio-fluxcd/flux-operator # TASK_VERSION -> https://github.com/go-task/task/releases diff --git a/internal/webhook/audit_joiner_deletecollection_test.go b/internal/webhook/audit_joiner_deletecollection_test.go new file mode 100644 index 000000000..90bf912dc --- /dev/null +++ b/internal/webhook/audit_joiner_deletecollection_test.go @@ -0,0 +1,250 @@ +/* +SPDX-License-Identifier: Apache-2.0 + +Copyright 2025 ConfigButler + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package webhook + +// These tests pin the shallow-detector and joiner behaviour for aggregated-API +// `deletecollection`, using exact copies of real audit events captured by +// apiservice-audit-proxy (see its docs/examples). deletecollection is the +// awkward verb: the kube-apiserver's own (Lane A) event for an aggregated +// resource is hollow — it has the objectRef but no request/response body — and +// the only body comes from the proxy (Lane B). Two flavours matter: +// +// - "raw" — `kubectl delete flunder --all`; the proxy captures a real +// FlunderList responseObject. +// - "teardown" — namespace deletion; the namespace-controller issues the +// deletecollection and the proxy only ever sees a DeleteOptions +// requestObject (no list body at all). +// +// The teardown flavour is the trap: its only proxy body is DeleteOptions. These +// tests assert that such an event is still NOT dropped as shallow (the proxy +// body is enough to take the merge/emit path), while a deletecollection with no +// proxy body at all IS correctly dropped. + +import ( + "context" + "os" + "testing" + + "github.com/alicebob/miniredis/v2" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "k8s.io/apimachinery/pkg/runtime" + auditv1 "k8s.io/apiserver/pkg/apis/audit/v1" + "sigs.k8s.io/yaml" +) + +const ( + deletecollectionOfficialTeardownFixture = "testdata/audit-events/audit-deletecollection-official-teardown-hollow.json" + deletecollectionOfficialRawFixture = "testdata/audit-events/audit-deletecollection-official-raw-hollow.json" + deletecollectionProxyTeardownFixture = "testdata/audit-events/audit-deletecollection-proxy-teardown-deleteoptions.json" + deletecollectionProxyRawFixture = "testdata/audit-events/audit-deletecollection-proxy-raw-listbody.json" +) + +// loadAuditEventListItem reads an audit.k8s.io/v1 EventList fixture and returns +// its single item. yaml.Unmarshal handles JSON input and populates the +// runtime.Unknown Raw bodies the same way the live decoder does. +func loadAuditEventListItem(t *testing.T, path string) auditv1.Event { + t.Helper() + raw, err := os.ReadFile(path) + require.NoError(t, err) + + var list auditv1.EventList + require.NoError(t, yaml.Unmarshal(raw, &list)) + require.Len(t, list.Items, 1, "fixture %s must contain exactly one audit event", path) + return list.Items[0] +} + +// TestClassifyAuditEventQuality_DeleteCollectionFixtures documents how the +// shallow detector classifies each real deletecollection event in isolation. +func TestClassifyAuditEventQuality_DeleteCollectionFixtures(t *testing.T) { + tests := []struct { + name string + fixture string + source AuditSource + want AuditEventQuality + }{ + { + // Lane A, namespace teardown: hollow. Alone it has no body, so it is + // correctly identity-shallow — it must rely on a parked proxy body. + name: "official teardown (hollow) is identity-shallow", + fixture: deletecollectionOfficialTeardownFixture, + source: AuditSourceOfficial, + want: AuditEventQualityIdentityShallow, + }, + { + name: "official raw (hollow) is identity-shallow", + fixture: deletecollectionOfficialRawFixture, + source: AuditSourceOfficial, + want: AuditEventQualityIdentityShallow, + }, + { + // Lane B, namespace teardown: only a DeleteOptions requestObject. It + // must still register as a real body (Collection), not Malformed — + // otherwise the proxy contribution is discarded and the official + // event has nothing to merge and gets dropped. + name: "proxy teardown (DeleteOptions body) is collection, not malformed", + fixture: deletecollectionProxyTeardownFixture, + source: AuditSourceAdditional, + want: AuditEventQualityCollection, + }, + { + // Lane B, --all: a real FlunderList responseObject. + name: "proxy raw (FlunderList body) is collection", + fixture: deletecollectionProxyRawFixture, + source: AuditSourceAdditional, + want: AuditEventQualityCollection, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + event := loadAuditEventListItem(t, tt.fixture) + assert.Equal(t, tt.want, classifyAuditEventQuality(tt.source, &event)) + }) + } +} + +// TestRedisAuditEventJoiner_DeleteCollectionTeardownNotDroppedAsShallow is the +// core namespace-deletion check: the hollow official deletecollection plus the +// proxy's DeleteOptions-only body must be emitted, not dropped as shallow. +func TestRedisAuditEventJoiner_DeleteCollectionTeardownNotDroppedAsShallow(t *testing.T) { + mr := miniredis.RunT(t) + joiner := newTestJoiner(t, mr) + ctx := context.Background() + + proxy := loadAuditEventListItem(t, deletecollectionProxyTeardownFixture) + parked, err := decide(ctx, t, joiner, AuditSourceAdditional, &proxy) + require.NoError(t, err) + require.Equal(t, AuditJoinActionParked, parked.Action, "proxy teardown body must park") + + official := loadAuditEventListItem(t, deletecollectionOfficialTeardownFixture) + require.Equal(t, proxy.AuditID, official.AuditID, "fixtures must share an auditID to join") + + decision, err := decide(ctx, t, joiner, AuditSourceOfficial, &official) + require.NoError(t, err) + + require.Equal(t, AuditJoinActionEmit, decision.Action, + "a deletecollection with a parked proxy body must be emitted, not dropped as shallow") + // The objectRef identifies the collection to remove (resource + namespace, + // no name). The DeleteOptions body is intentionally not merged onto a + // deletecollection, so the emitted event stays bodyless and the consumer + // acts on the objectRef. + require.NotNil(t, decision.Event.ObjectRef) + assert.Equal(t, "flunders", decision.Event.ObjectRef.Resource) + assert.Equal(t, "example-teardown", decision.Event.ObjectRef.Namespace) + assert.Empty(t, decision.Event.ObjectRef.Name) + assert.Nil(t, decision.Event.RequestObject, "DeleteOptions must not be merged onto a deletecollection") + assert.Nil(t, decision.Event.ResponseObject) +} + +// TestRedisAuditEventJoiner_DeleteCollectionRawNotDroppedAsShallow covers the +// `kubectl delete flunder --all` flavour, where the proxy supplies a real +// FlunderList responseObject. +func TestRedisAuditEventJoiner_DeleteCollectionRawNotDroppedAsShallow(t *testing.T) { + mr := miniredis.RunT(t) + joiner := newTestJoiner(t, mr) + ctx := context.Background() + + proxy := loadAuditEventListItem(t, deletecollectionProxyRawFixture) + parked, err := decide(ctx, t, joiner, AuditSourceAdditional, &proxy) + require.NoError(t, err) + require.Equal(t, AuditJoinActionParked, parked.Action, "proxy raw body must park") + + official := loadAuditEventListItem(t, deletecollectionOfficialRawFixture) + require.Equal(t, proxy.AuditID, official.AuditID, "fixtures must share an auditID to join") + + decision, err := decide(ctx, t, joiner, AuditSourceOfficial, &official) + require.NoError(t, err) + + require.Equal(t, AuditJoinActionEmit, decision.Action, + "a deletecollection with a parked proxy body must be emitted, not dropped as shallow") + require.NotNil(t, decision.Event.ObjectRef) + assert.Equal(t, "flunders", decision.Event.ObjectRef.Resource) + assert.Equal(t, "example-raw", decision.Event.ObjectRef.Namespace) + assert.Empty(t, decision.Event.ObjectRef.Name) +} + +// TestRedisAuditEventJoiner_DeleteCollectionWithoutProxyBodyIsShallow is the +// negative control: with no proxy (Lane B) body parked, the hollow official +// deletecollection has nothing to act on and is correctly dropped as shallow. +// This is the pre-0.6.0 / proxy-missing behaviour and the boundary the proxy +// fix moves the teardown case across. +func TestRedisAuditEventJoiner_DeleteCollectionWithoutProxyBodyIsShallow(t *testing.T) { + mr := miniredis.RunT(t) + joiner := newTestJoiner(t, mr) + + official := loadAuditEventListItem(t, deletecollectionOfficialTeardownFixture) + decision, err := decide(context.Background(), t, joiner, AuditSourceOfficial, &official) + require.NoError(t, err) + + assert.Equal(t, AuditJoinActionDrop, decision.Action, + "a hollow deletecollection with no proxy body must be dropped as shallow") + assert.False(t, mr.Exists(bodyKey(string(official.AuditID))), "shallow official must not park") +} + +// TestClassifyAuditEventQuality_DeleteCollectionBodyShapeIndependent guards the +// fact that the responseObject a backend returns for a deletecollection is +// aggregated-API-specific: wardle returns a FlunderList, but another apiserver +// may return a metav1.Status, an empty object, or only echo a DeleteOptions +// requestObject. The detector keys off "is any body present" (hasAuditV1ObjectBody) +// and the objectRef, never the body shape — so every non-empty variant must +// classify as Collection, and only a truly bodyless event is shallow. The base +// event is the real hollow official teardown fixture with its body swapped, so +// these stay synthetic-shape variants of a real event rather than invented data. +func TestClassifyAuditEventQuality_DeleteCollectionBodyShapeIndependent(t *testing.T) { + statusBody := `{"kind":"Status","apiVersion":"v1","status":"Success","code":200}` + listBody := `{"kind":"FlunderList","apiVersion":"wardle.example.com/v1alpha1","items":[]}` + deleteOptionsBody := `{"kind":"DeleteOptions","apiVersion":"v1","propagationPolicy":"Background"}` + + tests := []struct { + name string + requestRaw string + responseRaw string + want AuditEventQuality + }{ + {name: "metav1.Status response body", responseRaw: statusBody, want: AuditEventQualityCollection}, + {name: "List response body", responseRaw: listBody, want: AuditEventQualityCollection}, + {name: "empty-object response body", responseRaw: `{}`, want: AuditEventQualityCollection}, + {name: "DeleteOptions request body only", requestRaw: deleteOptionsBody, want: AuditEventQualityCollection}, + // A bodyless proxy (additional-source) contribution is useless and is + // discarded as Malformed before it can park, so the hollow official has + // nothing to merge and is later dropped as shallow. Either way: not + // actionable. (For the official source the same empty event is + // IdentityShallow; the label differs, the "dropped" outcome does not.) + {name: "no body at all is not actionable", want: AuditEventQualityMalformed}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + // Start from a real hollow official deletecollection (objectRef set, + // no body) and attach only the body shape under test. + event := loadAuditEventListItem(t, deletecollectionOfficialTeardownFixture) + require.Nil(t, event.RequestObject) + require.Nil(t, event.ResponseObject) + if tt.requestRaw != "" { + event.RequestObject = &runtime.Unknown{Raw: []byte(tt.requestRaw)} + } + if tt.responseRaw != "" { + event.ResponseObject = &runtime.Unknown{Raw: []byte(tt.responseRaw)} + } + + assert.Equal(t, tt.want, classifyAuditEventQuality(AuditSourceAdditional, &event)) + }) + } +} diff --git a/internal/webhook/testdata/audit-events/audit-deletecollection-official-raw-hollow.json b/internal/webhook/testdata/audit-events/audit-deletecollection-official-raw-hollow.json new file mode 100644 index 000000000..6870b4365 --- /dev/null +++ b/internal/webhook/testdata/audit-events/audit-deletecollection-official-raw-hollow.json @@ -0,0 +1,46 @@ +{ + "apiVersion": "audit.k8s.io/v1", + "kind": "EventList", + "metadata": {}, + "items": [ + { + "level": "RequestResponse", + "auditID": "5b41e1ef-d0a5-4df0-b0bd-d5c317cbad8b", + "stage": "ResponseComplete", + "requestURI": "/apis/wardle.example.com/v1alpha1/namespaces/example-raw/flunders", + "verb": "deletecollection", + "user": { + "username": "system:admin", + "groups": [ + "system:masters", + "system:authenticated" + ], + "extra": { + "authentication.kubernetes.io/credential-id": [ + "X509SHA256=fb5afb7b73ca3bc9fac7563a91d609a917d56708dd1cb3bbc32abaab6cd48fd4" + ] + } + }, + "sourceIPs": [ + "172.20.0.6" + ], + "userAgent": "kubectl/v1.36.1 (linux/amd64) kubernetes/7569396", + "objectRef": { + "resource": "flunders", + "namespace": "example-raw", + "apiGroup": "wardle.example.com", + "apiVersion": "v1alpha1" + }, + "responseStatus": { + "metadata": {}, + "code": 200 + }, + "requestReceivedTimestamp": "2026-06-02T11:16:29.161836Z", + "stageTimestamp": "2026-06-02T11:16:29.174537Z", + "annotations": { + "authorization.k8s.io/decision": "allow", + "authorization.k8s.io/reason": "" + } + } + ] +} diff --git a/internal/webhook/testdata/audit-events/audit-deletecollection-official-teardown-hollow.json b/internal/webhook/testdata/audit-events/audit-deletecollection-official-teardown-hollow.json new file mode 100644 index 000000000..8bfbbcc5b --- /dev/null +++ b/internal/webhook/testdata/audit-events/audit-deletecollection-official-teardown-hollow.json @@ -0,0 +1,48 @@ +{ + "apiVersion": "audit.k8s.io/v1", + "kind": "EventList", + "metadata": {}, + "items": [ + { + "level": "RequestResponse", + "auditID": "097dcb8c-c4d1-4e72-94f9-794cc2681e24", + "stage": "ResponseComplete", + "requestURI": "/apis/wardle.example.com/v1alpha1/namespaces/example-teardown/flunders", + "verb": "deletecollection", + "user": { + "username": "system:serviceaccount:kube-system:namespace-controller", + "uid": "9766626a-31eb-44ee-9dfe-59242cbf2a75", + "groups": [ + "system:serviceaccounts", + "system:serviceaccounts:kube-system", + "system:authenticated" + ], + "extra": { + "authentication.kubernetes.io/credential-id": [ + "JTI=a124b540-01b8-444a-8f37-791b30052ae4" + ] + } + }, + "sourceIPs": [ + "127.0.0.1" + ], + "userAgent": "k3s/v1.35.2+k3s1 (linux/amd64) kubernetes/13563fe/system:serviceaccount:kube-system:namespace-controller", + "objectRef": { + "resource": "flunders", + "namespace": "example-teardown", + "apiGroup": "wardle.example.com", + "apiVersion": "v1alpha1" + }, + "responseStatus": { + "metadata": {}, + "code": 200 + }, + "requestReceivedTimestamp": "2026-06-02T11:16:44.602001Z", + "stageTimestamp": "2026-06-02T11:16:44.605469Z", + "annotations": { + "authorization.k8s.io/decision": "allow", + "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"system:controller:namespace-controller\" of ClusterRole \"system:controller:namespace-controller\" to ServiceAccount \"namespace-controller/kube-system\"" + } + } + ] +} diff --git a/internal/webhook/testdata/audit-events/audit-deletecollection-proxy-raw-listbody.json b/internal/webhook/testdata/audit-events/audit-deletecollection-proxy-raw-listbody.json new file mode 100644 index 000000000..0a0bc22fb --- /dev/null +++ b/internal/webhook/testdata/audit-events/audit-deletecollection-proxy-raw-listbody.json @@ -0,0 +1,129 @@ +{ + "apiVersion": "audit.k8s.io/v1", + "kind": "EventList", + "metadata": {}, + "items": [ + { + "level": "RequestResponse", + "auditID": "5b41e1ef-d0a5-4df0-b0bd-d5c317cbad8b", + "stage": "ResponseComplete", + "requestURI": "/apis/wardle.example.com/v1alpha1/namespaces/example-raw/flunders", + "verb": "deletecollection", + "user": { + "username": "system:admin", + "groups": [ + "system:masters", + "system:authenticated" + ], + "extra": { + "authentication.kubernetes.io/credential-id": [ + "X509SHA256=fb5afb7b73ca3bc9fac7563a91d609a917d56708dd1cb3bbc32abaab6cd48fd4" + ] + } + }, + "sourceIPs": [ + "172.20.0.6", + "10.42.0.0" + ], + "userAgent": "kubectl/v1.36.1 (linux/amd64) kubernetes/7569396", + "objectRef": { + "resource": "flunders", + "namespace": "example-raw", + "apiGroup": "wardle.example.com", + "apiVersion": "v1alpha1", + "resourceVersion": "50" + }, + "responseStatus": { + "metadata": {}, + "code": 200 + }, + "responseObject": { + "kind": "FlunderList", + "apiVersion": "wardle.example.com/v1alpha1", + "metadata": { + "resourceVersion": "50" + }, + "items": [ + { + "metadata": { + "name": "member-0", + "namespace": "example-raw", + "uid": "e1b076ff-f430-4b82-ad7b-c170c4095fe3", + "resourceVersion": "49", + "creationTimestamp": "2026-06-02T11:16:29Z", + "annotations": { + "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"wardle.example.com/v1alpha1\",\"kind\":\"Flunder\",\"metadata\":{\"annotations\":{},\"name\":\"member-0\",\"namespace\":\"example-raw\"},\"spec\":{\"reference\":\"example\"}}\n" + }, + "managedFields": [ + { + "manager": "kubectl-client-side-apply", + "operation": "Update", + "apiVersion": "wardle.example.com/v1alpha1", + "time": "2026-06-02T11:16:29Z", + "fieldsType": "FieldsV1", + "fieldsV1": { + "f:metadata": { + "f:annotations": { + ".": {}, + "f:kubectl.kubernetes.io/last-applied-configuration": {} + } + }, + "f:spec": { + "f:reference": {}, + "f:referenceType": {} + } + } + } + ] + }, + "spec": { + "reference": "example", + "referenceType": "Flunder" + }, + "status": {} + }, + { + "metadata": { + "name": "member-1", + "namespace": "example-raw", + "uid": "5ce03312-bab3-4e72-af37-e0ff893a7b76", + "resourceVersion": "50", + "creationTimestamp": "2026-06-02T11:16:29Z", + "annotations": { + "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"wardle.example.com/v1alpha1\",\"kind\":\"Flunder\",\"metadata\":{\"annotations\":{},\"name\":\"member-1\",\"namespace\":\"example-raw\"},\"spec\":{\"reference\":\"example\"}}\n" + }, + "managedFields": [ + { + "manager": "kubectl-client-side-apply", + "operation": "Update", + "apiVersion": "wardle.example.com/v1alpha1", + "time": "2026-06-02T11:16:29Z", + "fieldsType": "FieldsV1", + "fieldsV1": { + "f:metadata": { + "f:annotations": { + ".": {}, + "f:kubectl.kubernetes.io/last-applied-configuration": {} + } + }, + "f:spec": { + "f:reference": {}, + "f:referenceType": {} + } + } + } + ] + }, + "spec": { + "reference": "example", + "referenceType": "Flunder" + }, + "status": {} + } + ] + }, + "requestReceivedTimestamp": "2026-06-02T11:16:29.162706Z", + "stageTimestamp": "2026-06-02T11:16:29.174126Z" + } + ] +} diff --git a/internal/webhook/testdata/audit-events/audit-deletecollection-proxy-teardown-deleteoptions.json b/internal/webhook/testdata/audit-events/audit-deletecollection-proxy-teardown-deleteoptions.json new file mode 100644 index 000000000..8c2ff24a1 --- /dev/null +++ b/internal/webhook/testdata/audit-events/audit-deletecollection-proxy-teardown-deleteoptions.json @@ -0,0 +1,49 @@ +{ + "apiVersion": "audit.k8s.io/v1", + "kind": "EventList", + "metadata": {}, + "items": [ + { + "level": "RequestResponse", + "auditID": "097dcb8c-c4d1-4e72-94f9-794cc2681e24", + "stage": "ResponseComplete", + "requestURI": "/apis/wardle.example.com/v1alpha1/namespaces/example-teardown/flunders", + "verb": "deletecollection", + "user": { + "username": "system:serviceaccount:kube-system:namespace-controller", + "groups": [ + "system:serviceaccounts", + "system:serviceaccounts:kube-system", + "system:authenticated" + ], + "extra": { + "authentication.kubernetes.io/credential-id": [ + "JTI=a124b540-01b8-444a-8f37-791b30052ae4" + ] + } + }, + "sourceIPs": [ + "127.0.0.1", + "10.42.0.0" + ], + "userAgent": "k3s/v1.35.2+k3s1 (linux/amd64) kubernetes/13563fe/system:serviceaccount:kube-system:namespace-controller", + "objectRef": { + "resource": "flunders", + "namespace": "example-teardown", + "apiGroup": "wardle.example.com", + "apiVersion": "v1alpha1" + }, + "responseStatus": { + "metadata": {}, + "code": 200 + }, + "requestObject": { + "kind": "DeleteOptions", + "apiVersion": "v1", + "propagationPolicy": "Background" + }, + "requestReceivedTimestamp": "2026-06-02T11:16:44.602749Z", + "stageTimestamp": "2026-06-02T11:16:44.605108Z" + } + ] +} diff --git a/test/e2e/aggregated_apiserver_e2e_test.go b/test/e2e/aggregated_apiserver_e2e_test.go index 2188b7a21..5f6b6b1b7 100644 --- a/test/e2e/aggregated_apiserver_e2e_test.go +++ b/test/e2e/aggregated_apiserver_e2e_test.go @@ -54,6 +54,10 @@ var _ = Describe("Aggregated API server", Label("aggregated-api"), Serial, Order _, _ = kubectlRun("create", "namespace", testNs) + By("setting up the Prometheus client for the shallow-drop metric check") + setupPrometheusClient() + verifyPrometheusAvailable() + repo = SetupRepo( resolveE2EContext(), testNs, @@ -213,6 +217,11 @@ spec: }, 2*time.Minute, 2*time.Second).Should(Succeed()) By("verifying no audit events were dropped as shallow, cluster-wide") + // Zero is the only acceptable value here. We never want a shallow drop: + // not for this Flunder, and not as a side effect of any other test + // sharing this cluster. Any non-zero count means a request/response body + // failed to join its official event, so this assertion stays absolute + // (not a baseline delta) on purpose. Consistently(func(g Gomega) { dropped, queryErr := queryPrometheus("sum(gitopsreverser_audit_shallow_dropped_total) or vector(0)") g.Expect(queryErr).NotTo(HaveOccurred()) diff --git a/test/e2e/setup/flux/releases/aggregated-api.yaml b/test/e2e/setup/flux/releases/aggregated-api.yaml index 0dce37166..6494add62 100644 --- a/test/e2e/setup/flux/releases/aggregated-api.yaml +++ b/test/e2e/setup/flux/releases/aggregated-api.yaml @@ -10,7 +10,7 @@ spec: mediaType: application/vnd.cncf.helm.chart.content.v1.tar+gzip operation: copy ref: - semver: 0.4.0 + semver: 0.6.0 --- apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease From ddbcbb399fea816e6d63614330b3fbba3ce5f0c0 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Tue, 2 Jun 2026 12:10:20 +0000 Subject: [PATCH 36/40] fix: prevent race in creating gitea org --- test/e2e/e2e_suite_test.go | 22 ++++++++++++++++++++++ test/e2e/repo_setup.go | 8 ++++---- 2 files changed, 26 insertions(+), 4 deletions(-) diff --git a/test/e2e/e2e_suite_test.go b/test/e2e/e2e_suite_test.go index 8fa61dcb9..dc64f1ea2 100644 --- a/test/e2e/e2e_suite_test.go +++ b/test/e2e/e2e_suite_test.go @@ -121,6 +121,28 @@ func prepareE2EClusterOnce() { "failed to delete IceCreamOrder CRD %q before tests", iceCreamCRDName(group)) _, _ = fmt.Fprintf(GinkgoWriter, "%s", output) } + + // The Gitea org is a singleton every repo fixture lives under. Create it + // once here — this runs on parallel process #1 only and blocks the other + // processes until it returns — so no two per-spec BeforeAlls ever race a + // concurrent POST /orgs. An in-process lock cannot solve this: Ginkgo + // --procs runs specs in separate OS processes, so the race is cross-process. + By("ensuring the shared Gitea organization exists before parallel specs run") + ensureSharedGiteaOrgOnce() +} + +// ensureSharedGiteaOrgOnce creates the shared test org exactly once, from the +// SynchronizedBeforeSuite process-#1 function. Per-spec bootstrap then only +// creates its own uniquely-named repo under an org that already exists. +func ensureSharedGiteaOrgOnce() { + gitea := giteaTestInstance() + Expect(waitForGiteaAPI(gitea.Client())).To(Succeed(), + "Gitea API must be reachable before creating the shared org") + + ctx, cancel := gitea.Context() + defer cancel() + _, err := gitea.Client().EnsureOrg(ctx, gitea.Org, "Test Organization", "E2E Test Organization") + Expect(err).NotTo(HaveOccurred(), "failed to ensure shared Gitea org %q", gitea.Org) } // configureE2EProcess runs in every parallel process. The Task prepare flow diff --git a/test/e2e/repo_setup.go b/test/e2e/repo_setup.go index 2e31fabe5..032c8f9b2 100644 --- a/test/e2e/repo_setup.go +++ b/test/e2e/repo_setup.go @@ -70,10 +70,10 @@ func bootstrapRepoArtifacts(namespace, repoName string) (*RepoArtifacts, error) ctx, cancel := gitea.Context() defer cancel() - if _, err := gitea.Client().EnsureOrg(ctx, gitea.Org, "Test Organization", "E2E Test Organization"); err != nil { - return nil, fmt.Errorf("ensure Gitea org %q: %w", gitea.Org, err) - } - + // The shared org is created once in SynchronizedBeforeSuite (process #1) + // before any spec runs, so it always exists here. Creating it per-spec used + // to race a concurrent POST /orgs across the parallel Ginkgo processes. Each + // repo name is unique, so the repo create below cannot collide the same way. if _, err := gitea.Client().EnsureOrgRepo( ctx, gitea.Org, From f819e9a5ed445b78388108510df5afed216d9486 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Tue, 2 Jun 2026 12:10:39 +0000 Subject: [PATCH 37/40] feat: count loc --- .devcontainer/Dockerfile | 3 ++- Taskfile-build.yml | 6 ++++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/.devcontainer/Dockerfile b/.devcontainer/Dockerfile index ab551b59a..1057b3ac6 100644 --- a/.devcontainer/Dockerfile +++ b/.devcontainer/Dockerfile @@ -153,7 +153,8 @@ RUN chgrp -R godev /go && \ RUN go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.19.0 \ && go install github.com/onsi/ginkgo/v2/ginkgo@v2.28.1 \ && go install sigs.k8s.io/controller-runtime/tools/setup-envtest@v0.0.0-20260318145839-6c9615a2a166 \ - && go install golang.org/x/tools/cmd/goimports@v0.43.0 + && go install golang.org/x/tools/cmd/goimports@v0.43.0 \ + && go install github.com/boyter/scc/v3@v3.7.0 # Initialize golangci-lint cache by running it once on an empty directory # This downloads linter dependencies without needing source code diff --git a/Taskfile-build.yml b/Taskfile-build.yml index 022dfff04..f00c912ae 100644 --- a/Taskfile-build.yml +++ b/Taskfile-build.yml @@ -8,6 +8,7 @@ vars: CONTROLLER_GEN: '{{.CONTROLLER_GEN | default "controller-gen"}}' ENVTEST: '{{.ENVTEST | default "setup-envtest"}}' GOLANGCI_LINT: '{{.GOLANGCI_LINT | default "golangci-lint"}}' + SCC: '{{.SCC | default "scc"}}' INSTALL_NAME: '{{.INSTALL_NAME | default "gitops-reverser"}}' NAMESPACE: '{{.NAMESPACE | default "gitops-reverser"}}' DEFAULT_AUDIT_REDIS_ADDR: '{{.DEFAULT_AUDIT_REDIS_ADDR | default "valkey:6379"}}' @@ -143,6 +144,11 @@ tasks: cmds: - '{{.GOLANGCI_LINT}} config verify' + loc: + desc: Count lines of project code with scc (excludes vendored external-sources) + cmds: + - '{{.SCC}} --exclude-dir external-sources,vendor,bin,dist {{.CLI_ARGS}}' + build: desc: Build manager binary deps: From 8e2e2a86723c3411cc773cf85d29f538bc3ea52f Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Tue, 2 Jun 2026 12:42:31 +0000 Subject: [PATCH 38/40] chore: Let's try to remove more "serial" --- .../design/e2e-serial-deserialization-plan.md | 34 +++++++---- docs/design/e2e-serial-registry.md | 39 +++++++++---- test/e2e/audit_helpers_test.go | 56 ------------------- test/e2e/bi_directional_e2e_test.go | 17 ++++-- test/e2e/commit_request_e2e_test.go | 21 ++++--- test/e2e/commit_window_batching_e2e_test.go | 19 +++++-- test/e2e/demo-todo.md | 5 -- 7 files changed, 91 insertions(+), 100 deletions(-) delete mode 100644 test/e2e/audit_helpers_test.go delete mode 100644 test/e2e/demo-todo.md diff --git a/docs/design/e2e-serial-deserialization-plan.md b/docs/design/e2e-serial-deserialization-plan.md index 0f8305c80..f1882f757 100644 --- a/docs/design/e2e-serial-deserialization-plan.md +++ b/docs/design/e2e-serial-deserialization-plan.md @@ -8,11 +8,22 @@ > **Outcome so far:** > - **Part A (`crd_lifecycle`)** — **done**: de-serialized and verified green > under `--procs=4`. See the registry's "De-serialized" section. -> - **Part B (audit pipeline)** — the heavy B1 refactor was **declined** as -> planned. Instead the redundant `Audit Redis Queue`/`Consumer` specs were -> retired and their one unique assertion (OIDC author attribution) moved to the -> new **parallel** `commit_author_attribution_e2e_test.go`. `Commit Window -> Batching` and `Commit Request` remain `Serial`. +> - **Part B (audit pipeline)** — the heavy B1 producer-side refactor was +> **declined** as planned, but turned out to be unnecessary. Re-reading the +> specs showed their real serial cause was a shared *repo* (one `sync.Once` +> Gitea repo + whole-branch `HEAD`/count assertions), not the shared audit +> *pipeline* — namespace-scoped WatchRules already route each spec's events to +> its own GitTarget. Giving each spec its **own** repo via `SetupRepo` +> de-serialized both `Commit Window Batching` and `Commit Request` with no +> production change. The redundant `Audit Redis Queue`/`Consumer` specs were +> also retired and their unique OIDC-author assertion moved to the **parallel** +> `commit_author_attribution_e2e_test.go`. +> - **`bi_directional`** — also de-serialized: it already owned its repo, and +> finding #2's plan-hash-gated resnapshot fix removed the cluster-wide catalog +> churn that produced its "+2 commits under parallelism". +> +> All de-serializations need a stability run under `E2E_GINKGO_PROCS=2` before the +> labels are trusted (see Part A step 3). ## Why this is not "just remove the `Serial` labels" @@ -23,17 +34,20 @@ disappear. Removing a `Serial` label without first isolating the underlying state re-introduces cross-spec bleed. So de-serialization is an isolation *refactor*, gated on verification — not a cleanup. -Of the 8 Serial containers, 4 are genuinely serial and stay: +Only 3 Serial containers are genuinely serial and stay: - `Restart Snapshot Safety`, `image refresh dependency chain` — restart/reimage the singleton controller. - `Aggregated API server` — installs/removes a cluster `APIService`; perturbs apiserver discovery for every client. -- `Bi Directional` — asserts exact remote commit counts; any concurrent - controller activity breaks the loop assertions. -The two candidates below are the only realistic de-serialization targets. They -are very different in size and risk, so they are planned independently. +`Bi Directional` was previously listed here as "genuinely serial" on the theory +that any concurrent controller activity breaks its exact-count loop assertions. +That turned out to be too pessimistic: with its own repo and finding #2's +plan-hash-gated resnapshot fix, no concurrent spec writes to its `main` and no +catalog refresh perturbs its target, so it has been de-serialized. The two +audit-pipeline candidates below were also de-serialized — by per-spec repo +isolation, not the B1 refactor (see the outcome note at the top of this doc). --- diff --git a/docs/design/e2e-serial-registry.md b/docs/design/e2e-serial-registry.md index 3de5db63e..2f2242e42 100644 --- a/docs/design/e2e-serial-registry.md +++ b/docs/design/e2e-serial-registry.md @@ -5,9 +5,10 @@ containers that must run `Serial` (i.e. never concurrently with any other spec) under bounded Ginkgo parallelism (Phase 2.5 of [e2e-speedup-plan.md](e2e-speedup-plan.md)). -> De-serializing any row below is an isolation refactor, not a label flip. The -> two realistic candidates (`crd_lifecycle` and the four audit pipeline specs) are -> planned in [e2e-serial-deserialization-plan.md](e2e-serial-deserialization-plan.md). +> De-serializing any row below is an isolation refactor, not a label flip. See +> [e2e-serial-deserialization-plan.md](e2e-serial-deserialization-plan.md) for the +> history; `crd_lifecycle`, the two audit-consumer specs, and `bi_directional` +> have all been de-serialized (see the "De-serialized" section). ## Rule @@ -29,9 +30,6 @@ row below. | [test/e2e/restart_snapshot_e2e_test.go](../../test/e2e/restart_snapshot_e2e_test.go) | `Restart Snapshot Safety` | Rollout-restarts the controller deployment. | The controller is a singleton; restarting it disrupts in-flight reconciles/commits for every other spec. | | [test/e2e/image_refresh_test.go](../../test/e2e/image_refresh_test.go) | `image refresh dependency chain` | Changes the controller image / redeploys the controller. | Same singleton controller; an image swap perturbs all concurrent specs. | | [test/e2e/aggregated_apiserver_e2e_test.go](../../test/e2e/aggregated_apiserver_e2e_test.go) | `Aggregated API server` | Installs/removes a cluster-scoped `APIService` (aggregation layer). | Registering/removing an aggregated API briefly disrupts apiserver discovery for **every** client, including unrelated kubectl calls in other specs. | -| [test/e2e/commit_window_batching_e2e_test.go](../../test/e2e/commit_window_batching_e2e_test.go) | `Commit Window Batching` | Same global audit pipeline (labelled `audit-consumer`). | Bursts of ConfigMap events flow through the shared audit consumer and leak into other audit specs' commits. | -| [test/e2e/commit_request_e2e_test.go](../../test/e2e/commit_request_e2e_test.go) | `Commit Request` | Same global audit pipeline (labelled `audit-consumer`). | Shares the audit consumer; commit-window/queue semantics interleave under concurrency. | -| [test/e2e/bi_directional_e2e_test.go](../../test/e2e/bi_directional_e2e_test.go) | `Bi Directional` | Whole-cluster Flux↔gitops-reverser round-trip; asserts on **exact** remote commit counts to prove no commit loop. | Any concurrent controller activity adds/reorders commits and breaks the exact-count loop assertions. Passes sequentially; +2 commits only under parallelism. | ### De-serialized @@ -63,11 +61,30 @@ commit window so each event is its own commit, and it reads the author scoped to each ConfigMap's unique path, so concurrent audit traffic cannot change the commit it asserts on. -The remaining `audit-consumer`-labelled containers (`Commit Window Batching`, -`Commit Request`) stay `Serial`: they assert on commit-window/batching semantics -over the shared audit consumer, which any concurrent audit traffic violates. -Because Ginkgo runs `Serial` specs alone (after all parallel specs), the audit -pipeline only ever has one client at a time for those. +The two `audit-consumer`-labelled containers (`Commit Window Batching` +([commit_window_batching_e2e_test.go](../../test/e2e/commit_window_batching_e2e_test.go)), +`Commit Request` ([commit_request_e2e_test.go](../../test/e2e/commit_request_e2e_test.go))) +were also de-serialized. Their real serial cause was **not** the shared audit +*pipeline* — namespace-scoped WatchRules already route each spec's events only to +its own GitTarget — but a shared *repo*: both borrowed one Gitea repo from a +`sync.Once` helper and asserted on whole-branch state (`Commit Request` reads +`HEAD`/`status.sha`; `Commit Window Batching` reads `git rev-list --count main`), +so the *other* audit spec's commits to the same `main` corrupted those reads. +Each now provisions its **own** repo via `SetupRepo`, so the only writer to its +`main` is its own GitTarget, fed exclusively by audit events from its own +namespace. The batching property is a per-GitTarget branch-worker behaviour (each +target runs its own commit window), so events for other GitTargets cannot enter +this spec's grouped commit. The retired `sync.Once` helper (`audit_helpers_test.go`) +was deleted. + +`Bi Directional` ([bi_directional_e2e_test.go](../../test/e2e/bi_directional_e2e_test.go)) +was de-serialized too. It already owned a dedicated repo and per-file CRD group; +its historical "+2 commits only under parallelism" came from cluster-wide GVR +catalog churn (another spec installing/deleting a CRD) dragging unrelated targets +into a resnapshot whose `reconcile: sync …` commits inflated its exact count. +Finding #2's fix ([manager.go:1191](../../internal/watch/manager.go#L1191)) — +a target only resnapshots when its *resolved* plan hash changes — removed that +mechanism, the same fix that unblocked `crd_lifecycle`. ## Triage list (watch during stability runs) diff --git a/test/e2e/audit_helpers_test.go b/test/e2e/audit_helpers_test.go deleted file mode 100644 index d083115c8..000000000 --- a/test/e2e/audit_helpers_test.go +++ /dev/null @@ -1,56 +0,0 @@ -/* -SPDX-License-Identifier: Apache-2.0 - -Copyright 2025 ConfigButler - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package e2e - -import ( - "fmt" - "sync" - - . "github.com/onsi/ginkgo/v2" - . "github.com/onsi/gomega" -) - -var ( - // auditConsumerRepo holds the repo fixtures shared by the audit-consumer specs. - auditConsumerRepo *RepoArtifacts - auditConsumerRepoOnce sync.Once -) - -// ensureAuditConsumerRepo lazily provisions the shared Gitea repo used by the -// audit-consumer specs (commit-window batching, commit request). -func ensureAuditConsumerRepo() *RepoArtifacts { - GinkgoHelper() - - auditConsumerRepoOnce.Do(func() { - consumerNs := testNamespaceFor("audit-consumer") - - By("creating the audit consumer namespace for shared repo fixtures") - _, _ = kubectlRun("create", "namespace", consumerNs) - - By("setting up the shared Gitea repo for audit-consumer tests") - auditConsumerRepo = SetupRepo( - resolveE2EContext(), - consumerNs, - fmt.Sprintf("e2e-audit-consumer-%d", GinkgoRandomSeed()), - ) - }) - - Expect(auditConsumerRepo).NotTo(BeNil(), "expected audit consumer repo fixtures to be initialised") - return auditConsumerRepo -} diff --git a/test/e2e/bi_directional_e2e_test.go b/test/e2e/bi_directional_e2e_test.go index ca857e9ef..20d9a4aef 100644 --- a/test/e2e/bi_directional_e2e_test.go +++ b/test/e2e/bi_directional_e2e_test.go @@ -95,12 +95,17 @@ type iceCreamScoop struct { // biDirectionalRepo holds the file-local repo fixtures for the Bi Directional describe block. var biDirectionalRepo *RepoArtifacts -// Serial: a whole-cluster bidirectional behavioral test (Flux applies from git -// while gitops-reverser mirrors live state back) that asserts on *exact* commit -// counts to prove no commit loop. Concurrent controller activity from other -// processes perturbs those counts. Passes sequentially; +2 commits only under -// parallelism. See docs/design/e2e-serial-registry.md. -var _ = Describe("Bi Directional", Label("bi-directional"), Serial, Ordered, func() { +// Not Serial: a whole-cluster bidirectional behavioral test (Flux applies from +// git while gitops-reverser mirrors live state back) that asserts on *exact* +// commit counts to prove no commit loop. It owns a dedicated Gitea repo and its +// own per-file CRD group, so the only writer to its main branch is its own +// GitTarget. The historical +2-commits-under-parallelism flake came from +// cluster-wide GVR catalog churn (another spec installing/deleting a CRD) +// dragging unrelated targets into a resnapshot; that is fixed — a target now +// only resnapshots when its *resolved* plan hash changes (manager.go) — so a +// foreign catalog refresh no longer perturbs this repo's commit count. See +// docs/design/e2e-serial-registry.md. +var _ = Describe("Bi Directional", Label("bi-directional"), Ordered, func() { var run biDirectionalRun var testNs string diff --git a/test/e2e/commit_request_e2e_test.go b/test/e2e/commit_request_e2e_test.go index 4355328f9..f667ad6ad 100644 --- a/test/e2e/commit_request_e2e_test.go +++ b/test/e2e/commit_request_e2e_test.go @@ -35,12 +35,17 @@ import ( // a deliberately long commitWindow so that, without the CommitRequest, the // edit would not be committed for minutes — observing the commit promptly is // what proves the commit-request path works. -// Serial: shares the single global audit pipeline with every other -// audit-consumer spec. See -// docs/design/e2e-serial-registry.md. -var _ = Describe("Commit Request", Label("commit-request", "audit-consumer", "smoke"), Serial, Ordered, func() { +// +// Not Serial: this spec owns a dedicated Gitea repo (its own GitProvider → +// GitTarget → namespace-scoped WatchRule), so the only writer to its main +// branch is its own GitTarget, fed exclusively by audit events from its own +// namespace. The HEAD/SHA assertions below therefore read back only this spec's +// own commit; concurrent audit traffic for other GitTargets lands in other +// repos and cannot move this repo's HEAD. See docs/design/e2e-serial-registry.md. +var _ = Describe("Commit Request", Label("commit-request", "audit-consumer", "smoke"), Ordered, func() { var ( testNs string + repo *RepoArtifacts gitProvName string gitTargetName string watchRuleName string @@ -54,7 +59,11 @@ var _ = Describe("Commit Request", Label("commit-request", "audit-consumer", "sm By("creating commit-request test namespace and applying git secrets") testNs = testNamespaceFor("commit-request") _, _ = kubectlRun("create", "namespace", testNs) - repo := ensureAuditConsumerRepo() + repo = SetupRepo( + resolveE2EContext(), + testNs, + fmt.Sprintf("e2e-commit-request-%d", GinkgoRandomSeed()), + ) _, err := kubectlRunInNamespace(testNs, "apply", "-f", repo.SecretsYAML) Expect(err).NotTo(HaveOccurred(), "failed to apply git secrets to namespace") applySOPSAgeKeyToNamespace(testNs) @@ -93,7 +102,6 @@ var _ = Describe("Commit Request", Label("commit-request", "audit-consumer", "sm }) It("finalizes the open commit window on demand and reports the resulting SHA", func() { - repo := auditConsumerRepo basePath := "e2e/commit-request-test" seed := GinkgoRandomSeed() cmName := fmt.Sprintf("commit-request-cm-%d", seed) @@ -159,7 +167,6 @@ var _ = Describe("Commit Request", Label("commit-request", "audit-consumer", "sm // from responseObject. Without the fix the CommitRequest stays stuck in // WaitingForAuditEvent forever. It("finalizes a CommitRequest created with metadata.generateName", func() { - repo := auditConsumerRepo basePath := "e2e/commit-request-test" seed := GinkgoRandomSeed() cmName := fmt.Sprintf("commit-request-gen-cm-%d", seed) diff --git a/test/e2e/commit_window_batching_e2e_test.go b/test/e2e/commit_window_batching_e2e_test.go index 4be6d5330..3aa466e24 100644 --- a/test/e2e/commit_window_batching_e2e_test.go +++ b/test/e2e/commit_window_batching_e2e_test.go @@ -35,13 +35,19 @@ import ( // and one push. The audit consumer pipeline is the real path under test: // events flow kubectl → audit webhook → consumer → // BranchWorker.commitGroups → BranchWorker.pushPendingCommits. -// Serial: shares the single global audit pipeline with every other -// audit-consumer spec. See +// +// Not Serial: this spec owns a dedicated Gitea repo, so the commit-count delta +// it asserts on counts only its own GitTarget's commits — fed exclusively by +// audit events from its own namespace. The batching property (a same-author +// burst inside one commit window collapses into one commit) is a per-GitTarget +// branch-worker behaviour: each GitTarget runs its own commit window, so events +// routed to other GitTargets cannot land in this spec's grouped commit. See // docs/design/e2e-serial-registry.md. var _ = Describe("Commit Window Batching", - Label("commit-window-batching", "audit-consumer", "smoke"), Serial, Ordered, func() { + Label("commit-window-batching", "audit-consumer", "smoke"), Ordered, func() { var ( testNs string + repo *RepoArtifacts gitProvName string gitTargetName string watchRuleName string @@ -53,7 +59,11 @@ var _ = Describe("Commit Window Batching", By("creating commit-window-batching test namespace and applying git secrets") testNs = testNamespaceFor("commit-window-batching") _, _ = kubectlRun("create", "namespace", testNs) - repo := ensureAuditConsumerRepo() + repo = SetupRepo( + resolveE2EContext(), + testNs, + fmt.Sprintf("e2e-commit-window-%d", GinkgoRandomSeed()), + ) _, err := kubectlRunInNamespace(testNs, "apply", "-f", repo.SecretsYAML) Expect(err).NotTo(HaveOccurred(), "failed to apply git secrets to namespace") applySOPSAgeKeyToNamespace(testNs) @@ -100,7 +110,6 @@ var _ = Describe("Commit Window Batching", It("collapses a burst of events into one grouped commit and one push", func() { const burstSize = 4 - repo := auditConsumerRepo basePath := "e2e/commit-window-test" seed := GinkgoRandomSeed() burstPrefix := fmt.Sprintf("commit-window-burst-%d", seed) diff --git a/test/e2e/demo-todo.md b/test/e2e/demo-todo.md deleted file mode 100644 index 1df51b18f..000000000 --- a/test/e2e/demo-todo.md +++ /dev/null @@ -1,5 +0,0 @@ - - -* Get a 100 votes at the same time! -* move to a voter that is released by GH: https://github.com/sunib/voter -* https within cluster \ No newline at end of file From 9987259dba3e393125bb2ea51c0c3d390bfc02d5 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Tue, 2 Jun 2026 13:02:13 +0000 Subject: [PATCH 39/40] chore: one less serial, and really upload these ginkgo results --- .github/workflows/ci.yml | 6 +++- .../design/e2e-serial-deserialization-plan.md | 29 +++++++++++-------- docs/design/e2e-serial-registry.md | 14 ++++++++- test/e2e/aggregated_apiserver_e2e_test.go | 8 ++--- 4 files changed, 39 insertions(+), 18 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 49dbbf289..71d6ba316 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -400,7 +400,11 @@ jobs: with: name: e2e-ginkgo-reports-${{ matrix.name }} path: .stamps/**/ginkgo-report-*.json - if-no-files-found: ignore + # Reports live under the dot-dir .stamps/, which upload-artifact treats + # as hidden and excludes by default — without this the upload silently + # finds no files. See the "No files were found" gotcha in upload-artifact v4+. + include-hidden-files: true + if-no-files-found: error retention-days: 60 # Release job only runs on push to main after tests pass diff --git a/docs/design/e2e-serial-deserialization-plan.md b/docs/design/e2e-serial-deserialization-plan.md index f1882f757..f58f3777b 100644 --- a/docs/design/e2e-serial-deserialization-plan.md +++ b/docs/design/e2e-serial-deserialization-plan.md @@ -34,20 +34,25 @@ disappear. Removing a `Serial` label without first isolating the underlying state re-introduces cross-spec bleed. So de-serialization is an isolation *refactor*, gated on verification — not a cleanup. -Only 3 Serial containers are genuinely serial and stay: +Only 2 Serial containers are genuinely serial and stay: - `Restart Snapshot Safety`, `image refresh dependency chain` — restart/reimage - the singleton controller. -- `Aggregated API server` — installs/removes a cluster `APIService`; perturbs - apiserver discovery for every client. - -`Bi Directional` was previously listed here as "genuinely serial" on the theory -that any concurrent controller activity breaks its exact-count loop assertions. -That turned out to be too pessimistic: with its own repo and finding #2's -plan-hash-gated resnapshot fix, no concurrent spec writes to its `main` and no -catalog refresh perturbs its target, so it has been de-serialized. The two -audit-pipeline candidates below were also de-serialized — by per-spec repo -isolation, not the B1 refactor (see the outcome note at the top of this doc). + the singleton controller, which every other spec depends on. No name isolation + can fix a shared singleton. + +Three other containers previously assumed immovable have been de-serialized: + +- `Bi Directional` was listed as "genuinely serial" on the theory that any + concurrent controller activity breaks its exact-count loop assertions. Too + pessimistic: with its own repo and finding #2's plan-hash-gated resnapshot fix, + no concurrent spec writes to its `main` and no catalog refresh perturbs its + target. +- `Aggregated API server` was marked Serial for "installs/removes a cluster + `APIService` in flight" — but it never installs one; Flux does, once, at + cluster setup, and it stays Available the whole run. The spec only reads it. +- The two audit-pipeline candidates (`Commit Window Batching`, `Commit Request`) + were de-serialized by per-spec repo isolation, not the B1 refactor (see the + outcome note at the top of this doc). --- diff --git a/docs/design/e2e-serial-registry.md b/docs/design/e2e-serial-registry.md index 2f2242e42..4f12adad7 100644 --- a/docs/design/e2e-serial-registry.md +++ b/docs/design/e2e-serial-registry.md @@ -29,7 +29,6 @@ row below. |---|---|---|---| | [test/e2e/restart_snapshot_e2e_test.go](../../test/e2e/restart_snapshot_e2e_test.go) | `Restart Snapshot Safety` | Rollout-restarts the controller deployment. | The controller is a singleton; restarting it disrupts in-flight reconciles/commits for every other spec. | | [test/e2e/image_refresh_test.go](../../test/e2e/image_refresh_test.go) | `image refresh dependency chain` | Changes the controller image / redeploys the controller. | Same singleton controller; an image swap perturbs all concurrent specs. | -| [test/e2e/aggregated_apiserver_e2e_test.go](../../test/e2e/aggregated_apiserver_e2e_test.go) | `Aggregated API server` | Installs/removes a cluster-scoped `APIService` (aggregation layer). | Registering/removing an aggregated API briefly disrupts apiserver discovery for **every** client, including unrelated kubectl calls in other specs. | ### De-serialized @@ -86,6 +85,19 @@ Finding #2's fix ([manager.go:1191](../../internal/watch/manager.go#L1191)) — a target only resnapshots when its *resolved* plan hash changes — removed that mechanism, the same fix that unblocked `crd_lifecycle`. +`Aggregated API server` ([aggregated_apiserver_e2e_test.go](../../test/e2e/aggregated_apiserver_e2e_test.go)) +was de-serialized after its `Serial` justification proved **stale**. The note +claimed it "installs/removes a cluster-scoped `APIService` while it is in flight", +but the wardle APIService (`v1alpha1.wardle.example.com`) is installed once at +cluster setup by the apiservice-audit-proxy HelmRelease +([setup/flux/releases/aggregated-api.yaml](../../test/e2e/setup/flux/releases/aggregated-api.yaml)) +and stays Available for the whole suite. The spec only *reads* it — it never +registers or removes it — so there is no in-flight discovery perturbation to +serialize against. Everything the spec mutates is name-isolated (own namespace, +own repo, namespace-scoped flunder WatchRule, namespaced Flunders), and its one +cluster-wide assertion (`gitopsreverser_audit_shallow_dropped_total == 0`) is a +global zero-invariant that is parallel-safe by design. + ## Triage list (watch during stability runs) `controller_basics_e2e_test.go` spec #5 ("should receive audit webhook events") diff --git a/test/e2e/aggregated_apiserver_e2e_test.go b/test/e2e/aggregated_apiserver_e2e_test.go index 5f6b6b1b7..38bae82fd 100644 --- a/test/e2e/aggregated_apiserver_e2e_test.go +++ b/test/e2e/aggregated_apiserver_e2e_test.go @@ -32,10 +32,10 @@ import ( "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" ) -// Serial: installs/removes a cluster-scoped APIService (aggregation layer), -// which perturbs apiserver discovery for every other client while it is in -// flight. See docs/design/e2e-serial-registry.md. -var _ = Describe("Aggregated API server", Label("aggregated-api"), Serial, Ordered, func() { +// Not Serial: the wardle APIService is installed once at cluster setup (Flux) +// and only read here; everything this spec mutates is name-isolated. See +// docs/design/e2e-serial-registry.md. +var _ = Describe("Aggregated API server", Label("aggregated-api"), Ordered, func() { var ( testNs string repo *RepoArtifacts From 7de7a65aa16a2925eed8a7e0539ff7a057bdb893 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Tue, 2 Jun 2026 13:19:13 +0000 Subject: [PATCH 40/40] ci: fixing broken build and cleaning docs --- .github/workflows/ci.yml | 10 + docs/design/e2e-ci-stability-report.md | 377 --------- .../design/e2e-serial-deserialization-plan.md | 234 ------ docs/design/e2e-serial-registry.md | 11 +- docs/design/e2e-speedup-plan.md | 787 +++--------------- docs/design/e2e-speedup-skipped.md | 122 --- docs/design/refactors-branch-review.md | 288 ------- docs/{ => design}/upgrade-finding.md | 0 docs/design/watch-and-catalog-architecture.md | 4 +- ...hrule-wildcard-and-resolution-semantics.md | 6 +- .../rule-set-snapshot-discovery-lag-fix.md | 6 +- .../initial-seed-via-watch-list.md | 6 +- .../watchrule-wildcard-support-plan.md | 6 +- 13 files changed, 152 insertions(+), 1705 deletions(-) delete mode 100644 docs/design/e2e-ci-stability-report.md delete mode 100644 docs/design/e2e-serial-deserialization-plan.md delete mode 100644 docs/design/e2e-speedup-skipped.md delete mode 100644 docs/design/refactors-branch-review.md rename docs/{ => design}/upgrade-finding.md (100%) rename docs/{design => finished}/rule-set-snapshot-discovery-lag-fix.md (98%) rename docs/{design => future}/initial-seed-via-watch-list.md (97%) rename docs/{design => future}/watchrule-wildcard-support-plan.md (97%) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 71d6ba316..08b75515d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -394,6 +394,16 @@ jobs: echo "" } | tee -a "$GITHUB_STEP_SUMMARY" + - name: Make E2E artifacts readable by the runner + if: always() + # The E2E suite runs in a root container that mounts the workspace, so + # everything it writes under .stamps/ (e.g. e2e-repo-artifacts/) is + # root-owned with restrictive perms. The upload step below runs on the + # host as the non-root runner user and fails with EACCES when its glob + # tries to scandir those dirs. Relax perms so the runner can traverse + # and read them. a+rX adds read for files and traverse for dirs only. + run: sudo chmod -R a+rX .stamps || true + - name: Upload E2E raw Ginkgo JSON reports if: always() uses: actions/upload-artifact@v7 diff --git a/docs/design/e2e-ci-stability-report.md b/docs/design/e2e-ci-stability-report.md deleted file mode 100644 index ba1ed1ff8..000000000 --- a/docs/design/e2e-ci-stability-report.md +++ /dev/null @@ -1,377 +0,0 @@ -# E2E CI stability — how the setup works & where it hurts - -A walkthrough of how the e2e job in [`.github/workflows/ci.yml`](../../.github/workflows/ci.yml) -runs today, a check of your assumptions, and concrete answers to the two -questions (run `prepare-e2e` up front; observe disk usage). - ---- - -## 1. How a run is wired today - -### 1.1 The `e2e` job and its matrix - -``` -e2e (runs-on: ubuntu-latest) - ├─ matrix leg "quickstart" → E2E_GINKGO_PROCS=2, K3D_AGENT_COUNT=0, needs_artifact=true - │ script: INSTALL_MODE=helm task test-e2e-quickstart-helm - │ → task test-image-refresh - │ → INSTALL_MODE=plain-manifests-file task test-e2e-quickstart-manifest - └─ matrix leg "full" → E2E_GINKGO_PROCS=1, K3D_AGENT_COUNT=0, needs_artifact=false - script: task test-e2e-full -``` - -`needs: [build-ci-container, docker-build, lint-helm]` — so by the time e2e -starts, the CI base image, the project image, and the release bundle -(`install.yaml` + `gitops-reverser.tgz`) already exist. - -Each matrix leg runs the actual suite inside the CI container via -**Docker-outside-of-Docker**: - -```yaml -docker run --rm --network host \ - -v "${GITHUB_WORKSPACE}:/workspaces/gitops-reverser" \ - -v /var/run/docker.sock:/var/run/docker.sock \ - ... ${CI_CONTAINER} bash -c "" -``` - -The container talks to the **runner's** Docker daemon (the mounted socket), and -that same daemon is what k3d uses to create the cluster nodes. So everything — -the CI container, k3d node containers, the project image, and every Flux-managed -service image — lands on the **one** runner VM's disk. - -### 1.2 What `task test-e2e-*` actually does - -The Task targets ([`test/e2e/Taskfile.yml`](../../test/e2e/Taskfile.yml)) still -invoke Ginkgo: - -``` -go run .../ginkgo --procs=$E2E_GINKGO_PROCS --timeout=$E2E_GO_TEST_TIMEOUT ... ./test/e2e/ -``` - -For local runs, cluster prep happens **inside** the suite. `SynchronizedBeforeSuite` -(process #1 only) calls `prepareE2EClusterOnce()` -([`e2e_suite_test.go:82`](../../test/e2e/e2e_suite_test.go#L82)), which shells out -to `task prepare-e2e` with the leg's `CTX / INSTALL_MODE / NAMESPACE / -INSTALL_NAME`. - -In CI, the workflow now runs the install-mode-independent `_services-ready` -slice first, outside Ginkgo, so cluster + Flux + Flux-managed service logs -stream in their own step. The suite still calls `prepare-e2e`; because the -cluster/service stamps already exist, it skips that expensive subtree and -continues with the install-mode-specific controller/image tail. - -### 1.3 The `prepare-e2e` dependency graph - -`prepare-e2e` is a go-task DAG, every node gated by a stamp file under -`.stamps/cluster//`. Stamps make it **idempotent**: a node whose stamp -already exists and whose sources are unchanged is skipped. The chain -(leaf → root): - -``` -prepare-e2e - ├─ _prepare-e2e-ready - │ ├─ _controller-deployed ── _image-loaded ── _project-image-ready (load-image.sh; pull mode in CI) - │ │ └─ _cluster-ready ── start-cluster.sh (k3d create) - │ ├─ _webhook-tls-ready ─┐ - │ ├─ _sops-secret-applied ┤ - │ └─ _aggregated-api-ready┘ - │ └─ _services-ready ── _flux-setup-ready ── _flux-installed ── _cluster-ready - └─ portforward-ensure (runs after the ready stamp, never in parallel with it) -``` - -The genuinely slow / hang-prone nodes: - -| Node | What it waits on | Timeout | -|------|------------------|---------| -| `_cluster-ready` | `start-cluster.sh`: k3d create, all nodes `Ready` | 60×2s health loop | -| `_flux-installed` | flux-operator install + FluxInstance + Flux deployments `Available` | `FLUX_WAIT_TIMEOUT=500s` | -| `_flux-setup-ready` | every HelmRelease/Kustomization (gitea, valkey, prometheus, traefik…) `Ready` | `FLUX_SERVICES_WAIT_TIMEOUT=600s` per resource | - -For local runs, the whole Ginkgo run — prepare **plus** the specs — is bounded by -one timeout: `E2E_GO_TEST_TIMEOUT=15m` (smoke/quickstart) or -`E2E_FULL_TIMEOUT=30m` (full). In CI, `_services-ready` is outside that Ginkgo -timeout; the timeout now covers the stamped/no-op cluster prep, the -install-mode-specific tail, and the specs. - -### 1.4 Why it "goes silent for ~15 min, then green on retry" - -This is the most important mechanical detail. `prepare-e2e` runs **inside -Ginkgo's `SynchronizedBeforeSuite`**, and its output is written to -`GinkgoWriter`. In `-v` (non-streaming) mode Ginkgo **buffers** that writer and -only flushes it when the node finishes or fails. So while the cluster + Flux + -services are coming up, the CI log shows nothing. - -If any readiness wait stalls (a HelmRelease that never reconciles, a node that -never goes `Ready`, a slow/stalled image pull), prepare sits there until the -outer **15m Ginkgo timeout** fires — at which point you get a goroutine dump -instead of a useful "service X never became ready" message. A re-run starts from -a clean VM and usually wins the race, which is exactly the signature of a -**flaky readiness/resource-contention problem**, not a hard deterministic -failure. - ---- - -## 2. Your assumptions, checked - -> **Every job (incl. every matrix leg) gets a new runner.** ✅ Correct. -GitHub provisions a fresh VM per job, and each matrix combination is its own job. -The two e2e legs never share disk, Docker daemon, or `.stamps`. - -> **Public-repo runners: max 20 concurrent, 4 vCPU / 16 GB / 14 GB SSD.** ✅ -Matches the current standard free Linux runner (4 vCPU, 16 GB RAM, 14 GB SSD) -and the 20-concurrent-job ceiling for the free tier. The **14 GB SSD is the -squeeze** — the runner image already pre-consumes a chunk of it, and *everything* -(CI container, k3d nodes, project image, all Flux service images, buildx/Go -caches) shares that one disk. - -> **Biggest chance is running out of disk.** ⚠️ Plausible but **not yet proven — -and the symptom argues against it.** A full disk almost always fails *loudly and -fast* ("no space left on device"), not with a 15-minute silent hang that -disappears on retry. The silent-hang-then-green pattern points more at a -**readiness wait stalling** (Flux service contention, image-pull stall, node not -Ready). Disk pressure can *cause* those stalls (e.g. kubelet -`DiskPressure` taint → pods won't schedule → HelmRelease never Ready), so it's -worth ruling in or out — but **measure before you commit to that theory.** That's -exactly what §4 adds. - -> **Set `K3D_AGENT_COUNT=1`.** Superseded. Both matrix legs now pass -`k3d_agent_count: "0"` (single-node k3d; k3s schedules workloads on the server). -This reduces node-container RAM/disk/inotify footprint, but leaves less -scheduling headroom for the quickstart leg's `--procs=2`. Keep watching the -quickstart timing and resource samples before treating `0` agents as permanent. - -> **Spin up 4 e2e legs and run more in parallel.** ◑ Mixed. More legs = more -*isolated* runners, so wall-clock can drop **if you split specs across them**. -But it does **not** reduce per-leg disk/RAM pressure — each new leg re-pulls the -full image set onto its own 14 GB disk. If disk/resource pressure is the real -root cause, adding legs spreads load but each leg is just as likely to hit it. -Fix the per-leg footprint first (measure → single-node/0-agent if it keeps -passing), *then* fan out for speed. - ---- - -## 3. Question 1 — bring the cluster up explicitly first - -**Recommended: yes — and target `_services-ready`, not the full `prepare-e2e`.** -Add an explicit cluster-bring-up step *before* the test `docker run`, in the same -container + workspace volume, with logs streaming live. Keep the in-Ginkgo -`prepareE2EClusterOnce()` call as-is — because the DAG is stamp-gated, it then -**no-ops** the parts already done. - -### Why `_services-ready` is the right target (not `prepare-e2e`) - -The hang is in *waiting for the Flux install + Flux-managed services to become -ready*. `_services-ready` is precisely that subtree and nothing more: - -``` -_services-ready - └─ _flux-setup-ready ── _flux-installed ── _cluster-ready (k3d create) -``` - -It deliberately stops short of `_image-loaded`, `_controller-deployed`, and the -whole install-mode tail. That gives two concrete advantages over running -`prepare-e2e`: - -- **No install-mode variables to decide.** The entire `_services-ready` chain is - independent of `INSTALL_MODE` / `NAMESPACE` / `INSTALL_NAME`; it needs only - `CTX`, which already defaults to `k3d-gitops-reverser-test-e2e`. So the step is - the same for *both* matrix legs — no `helm` vs `plain-manifests-file` vs - `config-dir` branching, and no risk of priming the wrong install tail. -- **It's exactly the hang surface.** Cluster create + Flux operator/instance + - every HelmRelease/Kustomization `Ready` wait — the three slow nodes from §1.3 - — and nothing you don't care about debugging right now. - -The image load and controller deploy stay inside Ginkgo's `prepare-e2e`, which is -fine: those aren't where it hangs, and `_cluster-ready`/`_flux-*`/`_services-ready` -are already stamped by the time Ginkgo runs, so it skips straight to them. - -Why this kills the symptom either way: run **outside** Ginkgo, the output -**streams** instead of being buffered to `GinkgoWriter`, so you see *which* node -hangs in real time instead of a 15-min silence + goroutine dump — and a genuine -bring-up failure fails a short, clearly-named CI step instead of masquerading as -a test timeout. - -### Two ways to invoke it - -`task _services-ready` works **today** — no task in -[`test/e2e/Taskfile.yml`](../../test/e2e/Taskfile.yml) is marked -`internal: true`, so the leading `_` is only a naming convention, not a runtime -guard. But depending on a `_`-prefixed target from CI is fragile (a future -`internal: true` or rename breaks CI silently). Cheapest hardening is a one-line -public wrapper: - -```yaml -# test/e2e/Taskfile.yml - prepare-cluster: - desc: Bring up the e2e cluster + Flux + Flux-managed services (install-mode independent) - cmds: - - task: _services-ready -``` - -CI step (new, before "Run E2E tests in CI container"), identical for both legs: - -```yaml -- name: Bring up cluster + Flux services (explicit, streaming logs) - run: | - docker run --rm --network host \ - -v "${GITHUB_WORKSPACE}:${{ env.CI_WORKDIR }}" \ - -v /var/run/docker.sock:/var/run/docker.sock \ - -w "${{ env.CI_WORKDIR }}" \ - -e IMAGE_DELIVERY_MODE=${{ env.IMAGE_DELIVERY_MODE }} \ - -e K3D_AGENT_COUNT=${{ matrix.k3d_agent_count }} \ - -e HOST_PROJECT_PATH=${{ github.workspace }} \ - ${{ env.CI_CONTAINER }} \ - bash -c " - git config --global --add safe.directory ${{ env.CI_WORKDIR }} - task prepare-cluster # or: task _services-ready - " -``` - -Note this step needs **no `PROJECT_IMAGE`, no `INSTALL_MODE`** — that's the whole -point of stopping at `_services-ready`. Keep `CTX` at its default (or pass the -same value the suite uses) so the stamps match and Ginkgo skips the work. - -### 3.1 Why the boundary stays at `_services-ready` (and not `_aggregated-api-ready`) - -A natural question is whether to push the explicit pre-step one node further, to -`_aggregated-api-ready`. **No — that node is not install-mode- or -image-independent.** Its full subtree: - -``` -_aggregated-api-ready - └─ _aggregated-api-webhook-kubeconfig-ready - ├─ _services-ready ← cluster + Flux + all flux services - └─ _webhook-tls-ready - └─ _controller-deployed ← install-mode-specific + project image - ├─ install (INSTALL_MODE: helm / plain-manifests-file / config-dir) - └─ _image-loaded (needs PROJECT_IMAGE) -``` - -Folding `_aggregated-api-ready` into the pre-step would re-introduce exactly the -two things stopping at `_services-ready` lets us avoid — choosing `INSTALL_MODE` -and passing `PROJECT_IMAGE` — and would tie the step to one matrix leg's mode -(the quickstart leg switches modes mid-run). So `_services-ready` is the last -node that is both **install-mode-agnostic** and **image-agnostic**; one step -further and that property is gone. The aggregated-api bring-up keeps running -inside the in-Ginkgo `prepare-e2e`, where it belongs. - -### 3.2 Observation: `_services-ready` is an all-or-nothing barrier (future refactor) - -`_aggregated-api-ready` *functionally* needs only **cert-manager** and the -**aggregated-api** HelmRelease — its commands only wait on the two cert-manager -`Certificate`s, the `aggregated-api` namespace deployments, and the `wardle` -APIService/discovery. It does **not** need gitea, valkey, prometheus-operator, -kro, reflector, or ingress. But it depends on `_services-ready`, and -[`hack/e2e/wait-flux-services.sh`](../../hack/e2e/wait-flux-services.sh) holds -that single gate until **every** HelmRelease/Kustomization is `Ready`. - -Consequence for stability: `_services-ready` is a coarse barrier. If *any one* -Flux service is slow or flaky to reconcile, it blocks aggregated-api, the -controller install, and effectively the whole suite — even specs that never -touch that service. A finer-grained dependency (e.g. aggregated-api waiting only -on cert-manager + its own release) would isolate such failures. - -**For now we are not refactoring this** — start with the immediate -`_services-ready` pre-step and let the streaming logs + disk metrics tell us -whether a specific service is the culprit. The barrier-narrowing is a candidate -follow-up *if* the data shows a single slow service is gating everything. - ---- - -## 4. Question 2 — observe disk usage (after the run, on failure, and the peak) - -Three layers, cheap to add, all `if: always()` so they fire on failure too. - -### 4.1 Snapshot after the run (and on failure) - -```yaml -- name: Disk & Docker usage (post-run) - if: always() - run: | - echo "## Disk after ${{ matrix.name }}" >> "$GITHUB_STEP_SUMMARY" - { echo '```'; df -h; echo; docker system df -v; echo '```'; } >> "$GITHUB_STEP_SUMMARY" - echo "### Biggest consumers" - sudo du -xh --max-depth=1 /var/lib/docker 2>/dev/null | sort -rh | head - du -sh .stamps 2>/dev/null || true -``` - -`df -h` tells you headroom; `docker system df -v` breaks it down by -images / containers / build cache / **volumes** (k3d node data lives here). - -### 4.2 Capture the *peak* with a background sampler - -A post-run snapshot misses the peak (k3d image imports + Flux pulls spike -mid-prepare, then settle). Sample throughout instead: - -```yaml -- name: Start disk sampler - run: | - ( while true; do - printf '%s %s\n' "$(date -u +%H:%M:%S)" \ - "$(df --output=avail -BM / | tail -1 | tr -d ' ')" - sleep 10 - done ) > df-samples-${{ matrix.name }}.log & - echo "SAMPLER_PID=$!" >> "$GITHUB_ENV" - -# ... prepare + test steps ... - -- name: Stop sampler & report peak - if: always() - run: | - kill "${SAMPLER_PID}" 2>/dev/null || true - echo "### Min free space on / during ${{ matrix.name }}" >> "$GITHUB_STEP_SUMMARY" - sort -k2 -n df-samples-${{ matrix.name }}.log | head -1 >> "$GITHUB_STEP_SUMMARY" - { echo '```'; cat df-samples-${{ matrix.name }}.log; echo '```'; } >> "$GITHUB_STEP_SUMMARY" - -- name: Upload disk samples - if: always() - uses: actions/upload-artifact@v7 - with: - name: e2e-disk-samples-${{ matrix.name }} - path: df-samples-${{ matrix.name }}.log - if-no-files-found: ignore -``` - -Lowest "avail" line = your true low-water mark. If it never drops below a few -hundred MB, disk is **not** your problem and you should look at Flux/service -readiness instead. - -### 4.3 If §4 confirms disk pressure — reclaim space first - -GitHub runners ship with large unused toolchains. A one-liner up front buys -~10–30 GB before the cluster is even created: - -```yaml -- name: Free disk space - run: | - sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc \ - /opt/hostedtoolcache/CodeQL /usr/local/.ghcup - sudo docker image prune -af - df -h / -``` - -(or use `jlumbroso/free-disk-space` / `easimon/maximize-build-space`.) - ---- - -## 5. Recommended order of attack - -1. **Add the instrumentation (§4.1–4.2) — do this first.** One run tells you - whether disk is actually the bottleneck. Cheap, non-invasive, settles the - open question instead of guessing. -2. **Add the explicit streaming `task _services-ready` / `prepare-cluster` step - (§3).** Independently valuable: turns the silent 15-min hang into a readable, - correctly-attributed failure, needs no install-mode variables, and the - in-Ginkgo `prepare-e2e` call still no-ops the cluster/Flux/services part. -3. **Then, based on the data:** - - Disk-bound → reclaim space (§4.3), consider `K3D_AGENT_COUNT=0`. - - Readiness-bound → look at *which* HelmRelease/Kustomization stalls (now - visible thanks to step 2) and tune that service or its wait. -4. **Only after the per-leg run is stable**, consider fanning out into more - matrix legs for wall-clock speed (§2) — splitting specs, not duplicating the - whole suite. - -The throughline: the "green on retry" behaviour is a **flaky readiness race**, -and the current design **hides the evidence** by burying prepare inside a -buffered Ginkgo writer under a single coarse 15-minute timeout. Steps 1 and 2 -make the evidence visible; everything else is tuning once you can see it. diff --git a/docs/design/e2e-serial-deserialization-plan.md b/docs/design/e2e-serial-deserialization-plan.md deleted file mode 100644 index f58f3777b..000000000 --- a/docs/design/e2e-serial-deserialization-plan.md +++ /dev/null @@ -1,234 +0,0 @@ -# E2E Serial de-serialization plan - -> Status: **partly executed** (2026-06-02). -> Generated 2026-06-02 as the follow-up to section 3 of -> [refactors-branch-review.md](refactors-branch-review.md). -> Authoritative Serial list: [e2e-serial-registry.md](e2e-serial-registry.md). -> -> **Outcome so far:** -> - **Part A (`crd_lifecycle`)** — **done**: de-serialized and verified green -> under `--procs=4`. See the registry's "De-serialized" section. -> - **Part B (audit pipeline)** — the heavy B1 producer-side refactor was -> **declined** as planned, but turned out to be unnecessary. Re-reading the -> specs showed their real serial cause was a shared *repo* (one `sync.Once` -> Gitea repo + whole-branch `HEAD`/count assertions), not the shared audit -> *pipeline* — namespace-scoped WatchRules already route each spec's events to -> its own GitTarget. Giving each spec its **own** repo via `SetupRepo` -> de-serialized both `Commit Window Batching` and `Commit Request` with no -> production change. The redundant `Audit Redis Queue`/`Consumer` specs were -> also retired and their unique OIDC-author assertion moved to the **parallel** -> `commit_author_attribution_e2e_test.go`. -> - **`bi_directional`** — also de-serialized: it already owned its repo, and -> finding #2's plan-hash-gated resnapshot fix removed the cluster-wide catalog -> churn that produced its "+2 commits under parallelism". -> -> All de-serializations need a stability run under `E2E_GINKGO_PROCS=2` before the -> labels are trusted (see Part A step 3). - -## Why this is not "just remove the `Serial` labels" - -Every entry in the Serial registry is **correctness-driven (shared, cluster-wide -state), not flakiness-driven**. The race fixes that landed (`03b95ab`, -`0787b1b`) made the suite *stable*; they did not make the shared state -disappear. Removing a `Serial` label without first isolating the underlying -state re-introduces cross-spec bleed. So de-serialization is an isolation -*refactor*, gated on verification — not a cleanup. - -Only 2 Serial containers are genuinely serial and stay: - -- `Restart Snapshot Safety`, `image refresh dependency chain` — restart/reimage - the singleton controller, which every other spec depends on. No name isolation - can fix a shared singleton. - -Three other containers previously assumed immovable have been de-serialized: - -- `Bi Directional` was listed as "genuinely serial" on the theory that any - concurrent controller activity breaks its exact-count loop assertions. Too - pessimistic: with its own repo and finding #2's plan-hash-gated resnapshot fix, - no concurrent spec writes to its `main` and no catalog refresh perturbs its - target. -- `Aggregated API server` was marked Serial for "installs/removes a cluster - `APIService` in flight" — but it never installs one; Flux does, once, at - cluster setup, and it stays Available the whole run. The spec only reads it. -- The two audit-pipeline candidates (`Commit Window Batching`, `Commit Request`) - were de-serialized by per-spec repo isolation, not the B1 refactor (see the - outcome note at the top of this doc). - ---- - -## Part A — `crd_lifecycle` (smaller; likely already unblocked) - -### Current Serial cause - -[crd_lifecycle_e2e_test.go:32](../../test/e2e/crd_lifecycle_e2e_test.go#L32) is -`Serial` because it installs/deletes a CRD and a (cluster-scoped) -`ClusterWatchRule`. Per the registry, that changes cluster-wide discovery / -the GVR catalog, which historically forced *unrelated* GitTargets to resnapshot; -the resulting `reconcile: sync …` commits hide other WatchRule specs' exact -`[CREATE]`/`[DELETE]` event-commit assertions. - -### What already changed - -1. **Finding #2 fix** ([manager.go:1191](../../internal/watch/manager.go#L1191)): - a target is only re-snapshotted when its *resolved* plan hash changes, and a - target whose rules currently resolve to nothing is kept as an empty plan - rather than evicted. A transient discovery gap no longer makes a target - vanish and resnapshot on recovery. -2. **Per-file CRD groups** ([icecream.go:30](../../test/e2e/icecream.go#L30)): - `crd_lifecycle` owns the API group `crd-lifecycle.e2e.example.com`, so its - `IceCreamOrder` CRD cannot collide by name with any other spec's CRD. - -### Why it is probably safe now (to be verified, not assumed) - -When `crd_lifecycle` installs its CRD, a *new* GVR becomes resolvable. A -concurrently-running spec would only be dragged into a resnapshot if it has a -rule that newly matches that GVR — i.e. a wildcard `ClusterWatchRule`. A scan of -the suite shows the only wildcard `ClusterWatchRule` -([restart/clusterwatchrule-wildcard.tmpl](../../test/e2e/templates/restart/clusterwatchrule-wildcard.tmpl)) -belongs to `restart_snapshot`, which is itself `Serial` and therefore never runs -concurrently. `demo_e2e_test.go` also uses a `ClusterWatchRule` and must be -checked (step 1). With no concurrent wildcard matcher, the icecream CRD only -changes `crd_lifecycle`'s own (name-isolated) targets, and finding #2 keeps -every other target's plan hash stable across the catalog refresh. - -### Plan - -1. **Confirm no concurrent wildcard matcher.** Audit `demo_e2e_test.go`'s - `ClusterWatchRule` scope (and any future broad rules). If it is scoped to - specific groups/resources that exclude the icecream group, it is safe; if it - is wildcard, either scope it down or keep `crd_lifecycle` Serial. -2. **Flip the label.** Drop `Serial` from - [crd_lifecycle_e2e_test.go:32](../../test/e2e/crd_lifecycle_e2e_test.go#L32) - (keep `Ordered`). Update the registry row. -3. **Verify under parallelism.** Run a `procs=2` (ideally higher) loop of the - `manager`-labelled specs together, e.g. several repetitions of - `task test-e2e` with `E2E_GINKGO_PROCS>=2`, and grep the WatchRule specs' - commit messages for unexpected `reconcile: sync` commits interleaved with - their `[CREATE]`/`[DELETE]` assertions. The gittarget-isolation spec already - encodes exactly this assertion - ([gittarget_isolation_e2e_test.go:140](../../test/e2e/gittarget_isolation_e2e_test.go#L140)) - and is the canary. -4. **If it bleeds**, do not force it. Capture the residual mechanism (likely a - `RefreshAPIResourceCatalog` churn path that perturbs targets even without a - matching rule) and fold it into the "settled-empty refinement" follow-up that - finding #2 already defers. - -### Effort / risk - -**Small–medium, low blast radius.** Mostly a label flip plus a verification -loop; the supporting code fix already shipped. The realistic failure mode is -"verification shows residual bleed," in which case we learn the exact remaining -mechanism and stop — no production code is touched. - ---- - -## Part B — the audit pipeline (historically 4 specs; reduced) - -### Scope - -This originally covered `Audit Redis Queue`, `Audit Redis Consumer`, -`Commit Window Batching`, and `Commit Request` — historically all labelled -`audit-redis`. -The queue/consumer containers have since been retired; the remaining -audit-consumer containers are `Commit Window Batching` -([commit_window_batching_e2e_test.go](../../test/e2e/commit_window_batching_e2e_test.go)) -and `Commit Request` -([commit_request_e2e_test.go](../../test/e2e/commit_request_e2e_test.go)). -The unique author-attribution assertion moved to the parallel -`Commit Author Attribution` -([commit_author_attribution_e2e_test.go](../../test/e2e/commit_author_attribution_e2e_test.go)) -container. - -### Current Serial cause (and the real shape of the problem) - -The audit pipeline is a **cluster-wide singleton firehose**: - -``` -apiserver audit policy → ONE audit webhook (internal/webhook/audit_handler.go:496) - → ONE Redis stream (cfg.auditRedisStream, default) - → ONE consumer group "gitopsreverser-consumer" (redis_audit_consumer.go:53) - → routes each event to GitTargets by matching WatchRules → commits -``` - -These specs assert commit **exclusivity** ("my commit touched only my file"; -commit-window batching asserts a batch contains only its own events). The -pollution source is *not just the other 3 audit specs* — it is **any** parallel -spec that writes a watched resource, because every such write produces an audit -event that flows through the same stream/consumer and can land in the same -batched commit. That is why they are fully `Serial`, not merely serial among -themselves. - -### Why the cheap approaches do not work - -- **Per-test stream name via config flip:** the apiserver audit policy targets a - single webhook endpoint on the singleton controller; tests cannot make the - apiserver route their events to a different stream. The stream name is a - *deploy-time* config, not a per-request property. -- **Per-test Redis consumer groups:** Redis consumer groups *fan out* — every - group receives every message. A new group sees all tests' events, so it - isolates nothing. -- **Namespace-scoped assertions only:** scoping each spec's "touched only my - file" assertion to its own namespace helps the simple cases, but - commit-window batching deliberately coalesces multiple events into one commit; - concurrent events from other specs still share that batch window. Assertion - scoping alone cannot fix the batching specs. - -### Options - -**B1 — Producer-side per-namespace (or per-target) stream routing.** Make the -webhook handler write to a stream keyed by the event's namespace/target instead -of one global stream, and have the consumer manage a group per active stream. -*Pro:* true isolation; the 4 specs go fully parallel. *Con:* significant change -to load-bearing production code (`audit_handler.go`, `redis_audit_queue.go`, -`redis_audit_consumer.go`), adds routing/lifecycle complexity that production -does **not** want (production is happy with one stream), and risks regressing -the very ordering/exclusivity guarantees the specs protect. Highest payoff, -highest risk. - -**B2 — Test-only routing mode.** Same as B1 but gated behind a test-only flag so -production keeps the single stream. *Pro:* shields production semantics. -*Con:* a test-only code path through the audit hot path is a maintenance smell -and can drift from the real path it is meant to validate — partially defeating -the point of an e2e test. - -**B3 — A dedicated "audit" lane: concurrent with the rest, serial among -themselves.** Keep the 4 specs mutually exclusive but let them run *alongside* -non-audit specs, with the non-audit specs taught not to pollute the audit -window. Concretely: (a) give the 4 audit specs a shared ordering/semaphore so -only one is "live" at a time, and (b) ensure no *other* parallel spec writes a -watched resource into a GitTarget the audit specs assert on — which in practice -means the audit specs must assert on a path/namespace no other spec writes, and -the consumer's commit for an audit spec must be scoped to that path. -*Pro:* no production change. *Con:* only works if the exclusivity assertions can -be made path-isolated despite shared batching — which is exactly the open -question; if batching coalesces a foreign event, the assertion still breaks. - -### Recommendation - -Do **not** attempt B1/B2 as part of routine speedup work. The audit pipeline's -singleton design is intentional, and bending it for test parallelism trades a -real production-correctness surface for a few minutes of CI time. If the audit -specs' runtime becomes a genuine bottleneck, pursue **B3 first** as a spike: -prove (or disprove) that the exclusivity assertions can be made path-isolated -under the shared consumer/batcher. Only if B3 is proven impossible *and* the -speedup is worth it should B1 be scheduled as its own design + review cycle. - -### Effort / risk - -**Large, high blast radius** for B1/B2 (touches the audit producer + consumer, -the most correctness-sensitive path in the system). **Medium** for the B3 spike, -but with a real chance the answer is "can't be isolated, leave Serial." - ---- - -## Sequencing recommendation - -1. **`crd_lifecycle` (Part A)** — do this first; it is mostly verification on top - of a fix that already shipped, and it removes one Serial entry with low risk. -2. **Audit B3 spike (Part B)** — only if audit-spec runtime is a measured - bottleneck; timebox it and accept "stays Serial" as a valid outcome. -3. **Audit B1** — only as a deliberately scheduled design cycle, never as a - drive-by. - -The `e2e-serial-registry.md` table is the source of truth; update the relevant -row in the same PR as any label change. diff --git a/docs/design/e2e-serial-registry.md b/docs/design/e2e-serial-registry.md index 4f12adad7..85f6106ba 100644 --- a/docs/design/e2e-serial-registry.md +++ b/docs/design/e2e-serial-registry.md @@ -5,10 +5,10 @@ containers that must run `Serial` (i.e. never concurrently with any other spec) under bounded Ginkgo parallelism (Phase 2.5 of [e2e-speedup-plan.md](e2e-speedup-plan.md)). -> De-serializing any row below is an isolation refactor, not a label flip. See -> [e2e-serial-deserialization-plan.md](e2e-serial-deserialization-plan.md) for the -> history; `crd_lifecycle`, the two audit-consumer specs, and `bi_directional` -> have all been de-serialized (see the "De-serialized" section). +> De-serializing any row below is an isolation refactor, not a label flip. +> `crd_lifecycle`, the two audit-consumer specs, `bi_directional`, and +> `aggregated_apiserver` have all been de-serialized (see the "De-serialized" +> section) — only the two singleton-controller specs remain. ## Rule @@ -46,9 +46,6 @@ assertions). It now runs parallel (`Ordered`, not `Serial`) because: `ClusterWatchRule` (`restart_snapshot`) is itself `Serial`, so no concurrent spec has a rule matching the icecream group. -See [e2e-serial-deserialization-plan.md](e2e-serial-deserialization-plan.md) for -the full rationale. - The former `Audit Redis Queue` / `Audit Redis Consumer` containers were retired: the producer-path and basic consumer-commit assertions duplicated coverage in the WatchRule suite, so they were dropped. The one genuinely unique assertion — diff --git a/docs/design/e2e-speedup-plan.md b/docs/design/e2e-speedup-plan.md index d01071a65..ac2630f2a 100644 --- a/docs/design/e2e-speedup-plan.md +++ b/docs/design/e2e-speedup-plan.md @@ -1,23 +1,17 @@ # E2E Speedup Plan -## Goal +Make `task test-e2e*` faster for the three audiences that pay the cost: local +devs, AI agents, and CI. Phases 0–3 below have **shipped**; this doc keeps the +baseline, the outcomes, and the rationale for what was deliberately *not* done. -Make `task test-e2e*` faster for the three audiences that pay the cost: -local devs, AI agents, and CI. - -Phased plan with explicit exit criteria. **Phase 0** ships the visibility -tooling that lets every later phase be measured against the same baseline. - -Items considered and excluded (BeforeSuite short-circuit, CI matrix -sharding, cluster reuse, Tekton/Shipwright, dropping `Ordered` inside the -new Manager files) are in [e2e-speedup-skipped.md](e2e-speedup-skipped.md) -with rationale. +The authoritative list of which specs still run `Serial` (and why) is the living +[e2e-serial-registry.md](e2e-serial-registry.md). ## Non-goals - Rewriting the e2e harness from scratch. -- Reducing the number of e2e tests (this plan does not merge or delete - specs; Phase 2 is a mechanical split that preserves spec count). +- Reducing the number of e2e specs (no spec was merged or deleted; the Phase 2 + split preserved spec count). ## Baseline (2026-05-28, warm devcontainer, k3d reused) @@ -25,7 +19,6 @@ with rationale. |---|---| | Warm `task prepare-e2e` | **1.4 s** (everything stamp-cached) | | Smoke suite (`task test-e2e`, 20 of 46 specs) | **418.7 s** (~7 min) | -| BeforeSuite (Go re-invoking warm prepare) | 1.4 s | ### Per-spec ranking (smoke filter, warm) @@ -36,11 +29,8 @@ with rationale. 34.1 s 8.2 Aggregated API server should install and serve flunders through the aggregation layer 28.9 s 6.9 Manager should create Git commit when IceCreamOrder CRD is installed via ClusterWatchRule 25.7 s 6.2 Commit Signing should produce per-event commits verifiable locally and by Gitea - 25.2 s 6.0 Audit Redis Queue should enqueue incoming audit webhook events into a Redis stream (retired) - 24.4 s 5.9 Audit Redis Consumer should attribute the commit to the OIDC display name and email (moved to Commit Author Attribution) 22.3 s 5.3 Commit Request finalizes a CommitRequest created with metadata.generateName 21.9 s 5.2 Commit Request finalizes the open commit window on demand and reports the resulting SHA - 16.4 s 3.9 Audit Redis Consumer should produce a Git commit from an audit stream event (retired) 15.5 s 3.7 Manager should delete Git file when ConfigMap is deleted via WatchRule 9.9 s 2.4 Manager should create Git commit when ConfigMap is added via WatchRule 9.1 s 2.2 Manager should commit encrypted Secret manifests when WatchRule includes secrets @@ -51,657 +41,128 @@ with rationale. 1.7 s 0.4 Manager should validate GitProvider with real Gitea repository 0.1 s 0.0 Manager should ensure the metrics endpoint is serving metrics 0.1 s 0.0 Manager should expose the controller service - 417.3 s total ``` -### Findings from the data - -1. **Top 5 specs = 62 %** of smoke wallclock. -2. **`Restart Snapshot Safety` is 31 % alone.** 75 s of its 128 s is a - passive negative-assertion wait. -3. **Namespace teardown is consistently 20–25 s** per spec (finalizer - waits). Dominant cleanup cost — relevant to Phase 2's risk profile. -4. **Per-spec setup ~5–10 s.** Specs already use per-spec namespaces + - repos; the structural blockers to parallelism are the shared - `Manager` Describe and cluster-scoped resources (CRDs). -5. **`prepare-e2e` is a script, not a DAG.** Eight sub-tasks run - sequentially via `cmds: - task: X` - ([test/e2e/Taskfile.yml:106-119](../../test/e2e/Taskfile.yml#L106-L119)) - even though several stamps have no real interdependency. - -## Phases - -### Phase 0 — Visibility via Ginkgo's built-in reporting - -**Why first:** every later phase claims a speedup. We need a structured, -machine-readable record before and after. - -**Primary mechanism:** add `-ginkgo.json-report=...` to the e2e `go test` -invocations. Ginkgo writes a per-spec JSON report with names, durations, -labels, and pass/fail without any log scraping. Parse it with a small Go -tool (`test/e2e/tools/spec-timings/`) that takes the JSON path and prints -a duration-sorted table plus totals — same shape as the baseline above, -but driven by structured data, not regex-on-logs. - -**Secondary mechanism (optional):** a tiny line-buffered timestamper at -`test/e2e/tools/ts/` (~20 lines of Go, `bufio.Scanner` over stdin) so -free-form logs can be decorated with wallclock when debugging cold runs -where structured JSON isn't enough. Not on the critical path. - -**Why not a `ReportAfterEach` reporter:** Ginkgo's built-in -`--json-report` already covers this. No new reporter code needed. - -**Exit criterion:** `go run ./test/e2e/tools/spec-timings ` -reproduces the per-spec table in this doc from a fresh `task test-e2e` -run. - -### Phase 1 — Express `prepare-e2e` as a real DAG - -**Change:** replace flat `cmds: - task: X` chains in `prepare-e2e` and -the upstream `install` flow with proper `deps:` so `task` schedules the -graph. - -**Concrete parallel branches** (verified against current stamp wiring): - -- `_age-key` (instant) is independent of `_cluster-ready` and - `_project-image-ready`. -- `_cluster-ready` (~30–60 s cold) and `_project-image-ready` (~10–40 s - on Go change) can run concurrently. -- `_image-loaded` joins both. -- `_flux-installed` → `_flux-setup-ready` → `_services-ready` chain - depends only on `_cluster-ready`. -- `_sops-secret-yaml` only needs `_age-key`. -- `_aggregated-api-ready` joins `_services-ready` ∧ `_webhook-tls-ready`. - -**Risk:** very low. Stamps make tasks idempotent; concurrent execution -can't corrupt. Side effect: interleaved logs. If that becomes a debugging -pain, keep a `prepare-e2e-verbose` alias that forces sequential -execution. - -**Exit criterion:** - -- Cold `task prepare-e2e` ≥ 25 % faster than the sequential baseline. -- Warm `task prepare-e2e` still ≈ 1.4 s. - -**DAG (as implemented).** Two regressions surfaced during validation and -shaped the final shape: - -- `portforward-ensure` had to stay *strictly sequential after* - `_prepare-e2e-ready` (not a `deps:` peer): the k3d server-node restart - inside `_webhook-tls-ready` kills any kubectl port-forwards started - concurrently with it. -- `cleanup-installs.sh` (which wipes `.stamps/cluster///`) - needed to be its own DAG node, `_install-cleanup`, so - `_sops-secret-yaml` can depend on it. Previously the cleanup ran inside - `_install-{mode}` cmds, and in parallel mode `_sops-secret-yaml`'s - status was evaluated *before* the cleanup, reported "up to date" from a - prior run's stamp, and then the cleanup deleted the file out from - under `_sops-secret-applied`. `_install-cleanup`'s status check - (`test -f /install.yaml`) lets it stay idempotent across - warm runs while still firing on a mode switch or a fresh CTX. - -```mermaid -flowchart TD - cluster([_cluster-ready]):::root - age([_age-key]):::root - imageid([_controller-image-id]):::root - - imageid --> imageready["_project-image-ready-local
+ _project-image-ready"] - - cluster --> cleanup[_install-cleanup] - cluster --> flux[_flux-installed] - cluster --> imageloaded[_image-loaded] - imageready --> imageloaded - - flux --> fluxsetup[_flux-setup-ready] - fluxsetup --> services[_services-ready] - - age --> sopsyaml[_sops-secret-yaml] - cleanup --> sopsyaml - - services --> install["install dispatcher
+ _install-{mode}"] - cleanup --> install - - install --> ctrl[_controller-deployed] - imageloaded --> ctrl - ctrl --> webhook[_webhook-tls-ready] - - services --> aggkube["_aggregated-api-
webhook-kubeconfig-ready"] - webhook --> aggkube - aggkube --> aggapi[_aggregated-api-ready] - - sopsyaml --> sopsapplied[_sops-secret-applied] - install --> sopsapplied - - ctrl --> ready[_prepare-e2e-ready] - webhook --> ready - sopsapplied --> ready - aggapi --> ready - - services --> pf[portforward-ensure] - ready ==> prepare([prepare-e2e]):::top - pf ==> prepare - - classDef root fill:#e8f4f8,stroke:#0066cc,stroke-width:2px - classDef top fill:#f0e8f8,stroke:#6600cc,stroke-width:2px -``` +**Findings that shaped the plan:** the top 5 specs are ~62 % of smoke wallclock; +`Restart Snapshot Safety` alone is 31 %, of which ~75 s was a passive +negative-assertion wait; namespace teardown is a consistent 20–25 s per spec +(finalizer waits); per-spec setup is ~5–10 s. The structural blockers to +parallelism were the shared `Manager` Describe and cluster-scoped resources. + +## What shipped + +### Phase 0 — Visibility -Legend: - -- Thin arrows are go-task `deps:` — parallel-safe; a task starts as soon - as all incoming arrows have completed. -- Thick arrows are sequential `cmds:` ordering inside `prepare-e2e` - itself (the only place where ordering is not expressed via `deps:`). -- Blue nodes (`_cluster-ready`, `_age-key`, `_controller-image-id`) have - no deps. Purple is the user-facing entry point. -- The `install` node collapses three steps that act as a single unit: - the templated dispatcher `install` (`task: 'install-{{.INSTALL_MODE}}'`), - the per-mode wrapper `install-{helm,config-dir,plain-manifests-file}`, - and the leaf `_install-{mode}` it depends on. Each `_install-{mode}` - carries its own build-side dep from `Taskfile-build.yml` - (`helm-sync`, `manifests`, or `dist-install`); those are elided here - to keep the diagram readable. - -The critical path on a cold run is -`_cluster-ready → _flux-installed → _flux-setup-ready → _services-ready -→ install → _controller-deployed → _webhook-tls-ready -→ _aggregated-api-webhook-kubeconfig-ready → _aggregated-api-ready -→ _prepare-e2e-ready → portforward-ensure`. Everything else folds onto -that path before its outputs are needed. - -**Measured impact:** cold `task prepare-e2e` 230 s → 213 s (7.4 %), well -short of the 25 % target. Honest accounting: the Flux ramp-up is a -strictly serial ~150 s of that budget and the docker image was cached in -this run, so the parallel branches (`_age-key`/`_sops-secret-yaml`, -`_image-loaded`, `_install-cleanup`, `manifests`) only saved the ~17 s -of work they would have spent on the critical path sequentially. With a -fresh Go build (the scenario the original plan estimate assumed), -`_project-image-ready` would absorb tens of seconds of additional cluster -ramp-up and the gain should approach the original estimate. Warm -`task prepare-e2e` stays at 1 s — no regression. +Added `-ginkgo.json-report` to the e2e `go test` invocations plus a small parser +(`test/e2e/tools/spec-timings/`) that prints the duration-sorted table above from +structured data. No custom reporter needed. + +### Phase 1 — `prepare-e2e` as a real `task` DAG + +Replaced the flat `cmds:` chains in `prepare-e2e` with `deps:` so independent +stamps (`_age-key`/`_sops-secret-yaml`, `_image-loaded`, `_install-cleanup`, +`manifests`) run concurrently. The critical path is the strictly-serial Flux +ramp-up (`_cluster-ready → _flux-installed → _flux-setup-ready → +_services-ready → install → _controller-deployed → _webhook-tls-ready → +_aggregated-api-ready`); everything else folds onto it. + +Two ordering constraints had to stay sequential: `portforward-ensure` runs +*after* `_webhook-tls-ready` (the k3d server-node restart inside it kills +concurrent port-forwards), and `_install-cleanup` is its own node so +`_sops-secret-yaml` can depend on it rather than racing a cleanup that wipes its +output. + +**Measured:** cold `task prepare-e2e` 230 s → 213 s (Flux ramp dominates a cached +image build; with a fresh Go build the parallel branches save more). Warm stays +~1 s. ### Phase 2 — Mechanical split of the `Manager` Describe -The `Manager` Describe in -[test/e2e/e2e_test.go](../../test/e2e/e2e_test.go) is a single `Ordered` -block of 21 specs sharing a package-global `managerRepo` and a -`gitprovider-normal` GitProvider implicitly created by spec #10. Two -specs hold `time.Sleep(5*time.Second)` admitting cross-spec state leak. - -**Scope.** This phase is a *mechanical* refactor: - -- Keep all 21 specs as they are; preserve spec count and observable - behavior. -- Move them into four files (see below), each its own top-level Describe - with its own per-file fixtures. -- Replace the package-global `var managerRepo *RepoArtifacts` with a - per-Describe local; each file's BeforeAll calls `SetupRepo` for itself. -- Move the cluster-scoped CRD ownership into the file that owns it - (see Phase 2.5 / CRD isolation). - -**Out of scope for this phase:** - -- No spec merging (e.g. old #18+#19+#20 stay three separate specs; - collapsing them is a follow-up). -- No `Ordered` removed *within* the new files. -- No Ginkgo parallelism enabled yet (Phase 2.5). -- No fix for the two `time.Sleep(5s)` hacks (separate root cause). - -**Spec inventory** (read 2026-05-28): - -| # | Spec | What it needs | Cleanup | -|---|------|---------------|---------| -| 1 | should run successfully | controller pod up | sets `controllerPodName` | -| 2 | should expose the controller service | `controllerPodName` from #1 | — | -| 3 | should expose the audit service separately | `controllerPodName` from #1 | — | -| 4 | should ensure the metrics endpoint is serving metrics | `controllerPodName` from #1 + log scrape | — | -| 5 | should receive audit webhook events from kube-apiserver | global Prometheus audit counter (delta-based) | deletes its CM | -| 6 | should validate GitProvider with real Gitea repository | `managerRepo` HTTP creds | deletes its GitProvider | -| 7 | should handle GitProvider with invalid credentials | `managerRepo` invalid creds | deletes its GitProvider | -| 8 | should handle GitTarget with nonexistent branch pattern | `managerRepo` HTTP creds | deletes its GitProvider + GitTarget | -| 9 | should validate GitProvider with SSH authentication | `managerRepo` SSH creds | deletes its GitProvider | -| 10 | should handle a normal and healthy GitProvider | `managerRepo` HTTP creds | **leaves `gitprovider-normal`** | -| 11 | should reconcile a WatchRule CR | `gitprovider-normal` | deletes own WR + GT | -| 12 | should commit encrypted Secret manifests when WatchRule includes secrets | `gitprovider-normal` | deletes own resources | -| 13 | should generate missing SOPS age secret when … generateWhenMissing | `gitprovider-normal` | deletes own resources | -| 14 | should create Git commit when ConfigMap is added via WatchRule | `gitprovider-normal` + **5 s sleep** | deletes own resources | -| 15 | should backfill pre-existing ConfigMap when WatchRule is added afterwards | `gitprovider-normal` | deletes own resources | -| 16 | should delete Git file when ConfigMap is deleted via WatchRule | `gitprovider-normal` + **5 s sleep** | deletes own resources | -| 17 | should create Git commit when IceCreamOrder CRD is installed via ClusterWatchRule | `gitprovider-normal` | **leaves CRD installed** | -| 18 | should create Git commit when IceCreamOrder is added via WatchRule | CRD from #17 (also re-applies) | **leaves WR + GT + CRD** | -| 19 | should update Git file when IceCreamOrder is modified via WatchRule | WR + GT from #18 | leaves them | -| 20 | should delete Git file when IceCreamOrder is deleted via WatchRule | WR + GT from #18 | leaves them | -| 21 | should delete Git file when IceCreamOrder CRD is deleted via ClusterWatchRule | CRD still installed | deletes CRD globally | - -The shared `AfterAll` (L1712) then cleans up `watchrule-icecream-orders`, -its GitTarget, `gitprovider-normal`, and the CRD. - -**Proposed file layout** (label `manager` kept on all four so -`task test-e2e-manager` keeps working unchanged): - -### Characteristics of the new files - -| File | Specs | Needs Gitea repo | Controller intervention | Ordered by design | Touches cluster-scoped objects | Touches global metrics | Leaves resources behind across specs | Parallel-safe with other groups | Smoke time today | -|---|---|---|---|---|---|---|---|---|---| -| `controller_basics_e2e_test.go` (Group A: specs 1–5) | 5 | **no** | no | partial — specs 2–4 reuse `controllerPodName` from #1 | no | yes — Prometheus audit counters (delta read) | no | yes | ~6 s | -| `gitprovider_validation_e2e_test.go` (Group B: specs 6–9) | 4 | yes — `managerRepo` HTTP/SSH/invalid creds | no | no — each spec creates+deletes its own GitProvider | no | no | no | yes | ~2 s (only #6 in smoke) | -| `watchrule_configmap_secret_e2e_test.go` (Group C: specs 10–16) | 7 | yes — `managerRepo` + shared `gitprovider-normal` in BeforeAll | no | no — once `gitprovider-normal` is in BeforeAll, specs 11–16 are independent (the two `time.Sleep(5s)` hacks aside) | no | no | yes — own `managerRepo` accumulates commits across specs in distinct subpaths | yes (own namespace) | ~41 s | -| `crd_lifecycle_e2e_test.go` (Group D: specs 17–21) | 5 | yes — `managerRepo` + `gitprovider-normal` in BeforeAll | no | **yes** — #18 hands WR+GT to #19+#20; #21 destroys the CRD others rely on | **yes — owns its own per-file CRD** (see CRD isolation, Phase 2.5) | no | yes within the file (intentional) | yes (own CRD name → no collision with other groups) | ~29 s (only #17 in smoke) | - -**Column glossary:** - -- **Needs Gitea repo** — does the file need `SetupRepo()` in BeforeAll? - A "no" is a meaningful setup saving. -- **Controller intervention** — does any spec restart, scale, or - perturb the controller deployment? None of these do; kept as a column - because it's the property that flags Phase 2.5's Serial registry. -- **Ordered by design** — would removing `Ordered` change the test's - observable contract? "Partial" / "no" means we keep it for now but - it could be dropped without semantic change. -- **Touches cluster-scoped objects** — flags CRD, ClusterWatchRule, - cluster RBAC. Main parallel-safety risk. -- **Touches global metrics** — Prometheus counters shared across - specs. Delta-based reads make this safe within one run. -- **Leaves resources behind across specs** — depends on a resource - created by a previous spec in the same file (Ordered intent), or - leaks past `AfterAll`? -- **Parallel-safe with other groups** — could this file run on a - Ginkgo worker in parallel with the others? After Phase 2.5's CRD - isolation, yes for all four. -- **Smoke time today** — sum of per-spec durations from the - 2026-05-28 baseline for specs labelled `smoke`. Lower bound, not a - total. - -**Per-file design notes:** - -1. **`controller_basics_e2e_test.go`** — no `managerRepo` setup at all. - BeforeAll only verifies the controller pod exists. Specs 1–5. Saves - the repo/secrets/SOPS setup this group never used. -2. **`gitprovider_validation_e2e_test.go`** — BeforeAll creates a - per-file namespace + `managerRepo`. Specs 6–9. -3. **`watchrule_configmap_secret_e2e_test.go`** — BeforeAll creates a - per-file namespace + `managerRepo` + `gitprovider-normal`. Old #10 - becomes the BeforeAll. Each remaining spec keeps its own WR+GT - create/delete. The two `time.Sleep(5s)` hacks stay. -4. **`crd_lifecycle_e2e_test.go`** — BeforeAll creates a per-file - namespace + `managerRepo` + `gitprovider-normal`. Internal `Ordered` - carries specs 17 → 18 → 19 → 20 → 21. **Owns its own per-file CRD**; - AfterAll deletes it. - -**Risks:** - -- **Sequential runs may get slower** before Phase 2.5 lands. Each new - file adds a namespace teardown (`cleanupNamespace` in - [test/e2e/helpers.go:254](../../test/e2e/helpers.go#L254) hits - finalizer waits of 20–25 s). Three extra namespaces ≈ 60–75 s of - added wallclock if Phase 2.5 is not yet on. Phase 2 and Phase 2.5 - should land back-to-back; if shipping just Phase 2, advertise as a - prerequisite, not a speedup. -- Group A's audit-events spec (#5) reads global Prometheus counters - with a delta pattern. Preserve baseline-before-act ordering. -- CRD cleanup ownership moves to `crd_lifecycle_e2e_test.go` — verify no - other file still expects it to be present or absent at startup. -- N Gitea repos created per run instead of 1. Each ~1 s. Acceptable. - -**Exit criterion:** - -- 21 specs from the old `Manager` Describe run as 21 specs across 4 - files. No spec merged or removed. Suite green. -- Per-spec timings (from Phase 0 JSON report) unchanged within noise. -- No new shared state between the four Describes. - -### Phase 2.5 — Enable bounded Ginkgo parallelism + Serial registry - -Without this phase, Phase 2's wallclock is *worse* than today on -sequential runs. This phase is what realises the speedup. - -**Change:** - -- Add `E2E_GINKGO_PROCS` (default e.g. `2`) wired into - [test/e2e/Taskfile.yml:194](../../test/e2e/Taskfile.yml#L194) so - `go test ... -ginkgo.parallel -ginkgo.procs=$E2E_GINKGO_PROCS`. Local - default sane for a devcontainer; CI may pass a higher value when - Phase 6 (separately, not in active plan) is reconsidered. -- Establish a **suite-wide Serial registry** — a short list of test - containers marked `Serial` because they touch shared cluster-scoped - state that cannot be isolated by name alone. Created as a living - document `docs/design/e2e-serial-registry.md` in this phase. Initial - entries from the 2026-05-28 audit: - - `restart_snapshot_e2e_test.go` — restarts the controller deployment; - interferes with any other test reading or writing through the - controller while restarted. - - Anything else that performs `kubectl delete crd` cluster-wide or - rolls the controller. Phase 2.5 PR populates the initial list from - a grep audit; afterward, adding cluster-scoped writes requires - updating the registry in the same PR. - -**CRD isolation (the easier solution).** The shared -`icecreamorders.shop.example.com` CRD currently appears in: - -- [test/e2e/e2e_test.go](../../test/e2e/e2e_test.go) (Manager block; moves to - `crd_lifecycle_e2e_test.go` in Phase 2) -- [test/e2e/restart_snapshot_e2e_test.go:94-96](../../test/e2e/restart_snapshot_e2e_test.go#L94-L96) -- [test/e2e/bi_directional_e2e_test.go:130](../../test/e2e/bi_directional_e2e_test.go#L130) -- [test/e2e/e2e_suite_test.go](../../test/e2e/e2e_suite_test.go) (pre-run cleanup) - -Rather than serialise these three files against each other, **give each -its own CRD** in its own API group: - -| File | CRD group | Kind | -|---|---|---| -| `crd_lifecycle_e2e_test.go` | `crd-lifecycle.e2e.example.com` | IceCreamOrder | -| `restart_snapshot_e2e_test.go` | `restart-snapshot.e2e.example.com` | IceCreamOrder | -| `bi_directional_e2e_test.go` | `bi-directional.e2e.example.com` | IceCreamOrder | - -Implementation: take the existing -[test/e2e/templates/icecreamorder-crd.yaml](../../test/e2e/templates/icecreamorder-crd.yaml) -fixture and templatize the group + plural. Each test renders its own. -The pre-cleanup in `e2e_suite_test.go` deletes all three by name. - -This removes the cross-file shared cluster-scoped state and lets the -three files run in parallel without any Serial marking. The Serial -registry then only needs to capture genuinely process-level conflicts -(restart_snapshot at the controller-deployment level, future ones). - -**Risks:** - -- Hidden ordering assumptions only surface under `-procs > 1`. Run for - a stability window (say 10 consecutive clean runs) before flipping - the default. -- Some helpers may write to package-global state. Phase 2 already - removes the only obvious one (`managerRepo`); audit during this PR. - -**Exit criterion:** - -- `task test-e2e` with `E2E_GINKGO_PROCS=2` runs green across N - consecutive runs (N decided when starting; suggested 10). -- Smoke wallclock drops by at least the parallel-safe Group A+B+C share. -- Serial registry exists and is non-empty. - -**Implementation status & where we are (2026-05-29 — read this first on resume).** -Landed on branch `refactors` (PR #159). `gh` works directly now — no -`unset GH_TOKEN` needed. - -**What is implemented (and green locally at `procs=2`):** - -- **Parallelism is driven by the `ginkgo` CLI, not `go test`.** `go test` - has no `-ginkgo.procs` flag (it only accepts the `-ginkgo.parallel.*` - flags the CLI sets on its workers). `test-e2e`, `test-e2e-full` and - `test-e2e-manager` now run - `go run github.com/onsi/ginkgo/v2/ginkgo --procs={{.E2E_GINKGO_PROCS}}` - (pinned to the module version to avoid a CLI/library mismatch). - `E2E_GINKGO_PROCS` defaults to `2`. -- **`SynchronizedBeforeSuite`** runs `prepare-e2e` + CRD pre-cleanup once - on process #1; the `E2E_AGE_KEY_FILE` fallback runs per-process. -- **Per-file CRD group isolation** via `test/e2e/icecream.go`: - `crd_lifecycle`/`restart_snapshot`/`bi_directional` each own a group - (`*.e2e.example.com`); the shared `icecreamorders` plural means every - kubectl reference is qualified `icecreamorders.`. -- **Serial registry** ([e2e-serial-registry.md](e2e-serial-registry.md)) - is larger than the initial audit: `restart_snapshot`, `image_refresh`, - `aggregated_api` (controller/APIService level) **plus** the remaining - `audit-consumer`-labelled containers (`Commit Window Batching`, - `Commit Request`) — they share one global audit pipeline and - cross-contaminated each other's commits — **plus** - `bi_directional` (asserts exact commit counts to prove no commit loop; - broken by any concurrent controller activity). `crd_lifecycle` and the - former `Audit Redis Queue/Consumer` containers have since been retired from - the Serial set; see [e2e-serial-registry.md](e2e-serial-registry.md). -- **Timeouts widened for parallel/slow load:** `verifyResourceStatus` - readiness 30s → 90s (post-CRD-install the controller's discovery cache - lags before serving the new GVR, so a dependent WatchRule can't reach - Ready in 30s); manager ConfigMap-delete `Eventually` 30s → 60s; signing - commit-waits 60s → 90s. - -**Commits on `refactors` since the `1205369` squash:** `cad9903` (Serial: -bi-directional + audit pipeline), `9942e0c` (readiness 90s), `dba83ac` -(this doc), `1294814` (k3d agents 1→2), `452961c` (signing 90s), `b7a00b4` -(k3d agents 2→3). - -**Measured (local devcontainer, `procs=2`):** smoke **352 s** vs 418.7 s -sequential baseline (~16 %); cold full **45/45** in ~20 min; warm full -**45/45** in ~17 min. Rock solid locally across smoke + 2 full runs. The -full-suite gain is modest *by design*: the heavy specs are `Serial` for -correctness and the single biggest spec, `Restart Snapshot Safety` -(~128 s), is Serial and untouched until Phase 3. Smoke's ~16 % is the -honest gain today; the bigger win is Phase 3. - -**Resolution for merge — CI full is sequential, local remains parallel -(2026-05-30).** On the stock `ubuntu-latest` GitHub runners the single -controller is **CPU-starved** under two concurrent test streams plus its -own CPU-heavy SSH-signing/git work. Symptom: a *different* spec times out -each run (crd_lifecycle WatchRule readiness; watchrule "create"; signing — -the last failed even at a 90 s wait, i.e. genuine starvation, not a tight -timeout). The follow-up run after 3 agents also failed during setup while -waiting for `helmrelease/kro`, so the blocking CI signal is broader than -spec logic: runner/resource pressure plus external chart/registry -readiness. - -GitHub API throttling was checked with `gh api rate_limit` and was not the -limiting factor (4967/5000 core requests remaining at review time). The -workflow now uploads any Ginkgo JSON reports and writes a per-spec timing -summary to the GitHub job summary so build-server timings are visible. - -Applied fallback: - -- CI **E2E (full)** sets `E2E_GINKGO_PROCS=1`. -- Local/default `task test-e2e*` remains `E2E_GINKGO_PROCS=2`. -- k3d local/default agents remain 3 for `procs=2`; CI currently passes - `K3D_AGENT_COUNT=0` for both e2e legs to reduce per-leg runner footprint. -- Warm k3d cluster reuse now verifies the expected node count and Ready - condition instead of trusting `kubectl get ns` plus stamps. -- CI pre-runs `_services-ready` outside Ginkgo so cluster + Flux + service - readiness logs stream live; the in-suite `prepare-e2e` call then reuses the - stamps and only performs the install-mode-specific tail. - -History that led here: -Progression tried: -- 1 agent → red. -- **2 agents** (`1294814`) → E2E (full) went **green once**, then **red** - on the next run (`452961c`, signing at 90 s). Reduced but didn't - eliminate. -- **3 agents** (`b7a00b4`) → E2E (full) went **green** (run - `26660720283`). But this is **one** green run, and 2 agents also went - green-once-then-red, so stability is **not yet proven** — needs a few - consecutive green reruns before we trust it. The 3→1 drop was originally - due to `fs.inotify.max_user_instances` (default 128) exhaustion; that is - now mitigated to 512 by `ensure_inotify_limits`, so 1 server + 3 agents - is possible but no longer the CI default. -- **0 agents** (`82715f2`/`de3dc09` era) → the branch has produced multiple - green CI runs with the explicit `_services-ready` pre-step and Allure/timing - artifacts, but this is still below the preferred stability window. Treat it - as promising, not proven, until there are 5-10 consecutive green full e2e - runs on the same shape. - -Confirmed the suite uses a **single** cluster (the `audit-pass-through-e2e` -cluster seen locally is an unrelated leftover). - -**Current best next work (2026-06-01):** do not raise the full-suite timeout -again yet. `test-e2e-full` already has a 30 minute timeout, and recent green CI -full legs complete well below it. The next useful work is a confidence pass: -rerun the current CI shape enough times to build a 5-10 green-run window, then -use the uploaded timing summaries, disk samples, and Allure reports to decide -whether `K3D_AGENT_COUNT=0` is stable enough to merge. If it is stable, the next -speed target is the residual fixed setup in `Restart Snapshot Safety` and the -coarse `_services-ready` barrier, not more timeout. - -### Phase 3 — Target-scoped reconcile-complete + drain signal for `Restart Snapshot Safety` - -**Why a pod-level gauge alone is insufficient** (per reviewer): a -pod-scoped "initial reconcile complete" can be set after -`ReconcileForRuleChange` returns -([internal/watch/manager.go:679](../../internal/watch/manager.go#L679)) -while snapshot writes for a specific GitTarget are still pending — they -flow through `MaybeReplaySnapshot` -([internal/watch/manager.go:1216](../../internal/watch/manager.go#L1216)) -and enqueue via the branch worker -([internal/watch/git_target_event_stream.go:139](../../internal/watch/git_target_event_stream.go#L139)) -which then pushes asynchronously. The restart test's negative assertion -("the mirror is not wiped after this restart") requires evidence that -*this GitTarget's* post-restart processing has fully drained, not just -that the pod started. - -**Change.** Pair two signals: - -1. **`gitopsreverser_target_reconcile_completed_total{gittarget_namespace,gittarget_name,trigger}`** - — **counter** per GitTarget, incremented once each rule-set snapshot - reconcile pass completes end-to-end (snapshot decision made, queue - submission completed). `trigger` is `rule_change` (the GVR/rule reconcile - path) or `startup_replay` (a snapshot replayed once a FolderReconciler is - created). A counter, **not a latched gauge**: see the stale-latch note - below. -2. **`gitopsreverser_branch_worker_queue_depth{provider_namespace,provider_name,branch}`** - — gauge of pending work items for this branch worker. Wait for `0` - to confirm the queue has drained. - -> **Why a counter, not a latched gauge, and why gate per-pod.** An earlier -> draft used a gauge `gitopsreverser_target_initial_reconcile_complete` that -> latched to `1`. During a rollout Prometheus can still hold the **old** -> pod's series (distinct `pod` target label) alongside the new pod's, so a -> `max(...) == 1` check is satisfied by the dying pod's stale `1` and proves -> nothing about the new pod — re-weakening the safety gate toward the old -> blind wait. A counter is the right shape, but a *cross-pod* -> `sum(...) > baseline` delta still has a failure mode: counters reset to `0` -> on the fresh pod, so once Prometheus marks the old pod's series stale, the -> new pod's first increment can bring the cross-pod sum back to the old total -> (e.g. old=1 → stale → new=1 → sum stays 1) and `> baseline` never passes. -> The robust gate is therefore **per-pod**: discover the new pod's name after -> the rollout completes and assert `…{pod=""} > 0`. A counter on a -> freshly started pod starts at `0`, so any value `> 0` proves *that pod* -> completed its own post-restart reconcile. The counter also has long-term -> value — `increase(...[5m])` surfaces excessive/looping reconciles. The name -> reflects what it actually instruments: rule-set snapshot delivery/replay, -> not a literal "initial" controller path. - -The metric package is `internal/telemetry/` (not `internal/metrics/` — -see `internal/telemetry/exporter.go`); add the new metrics alongside -the existing ones. - -> **Label naming — do not use `namespace`/`name`.** The e2e Prometheus -> scrapes the controller as a pod target with `honor_labels: false` -> ([test/e2e/setup/manifests/prometheus/servicemonitor.yaml](../../test/e2e/setup/manifests/prometheus/servicemonitor.yaml)). -> A pod scrape attaches the scraped pod's own `namespace`/`pod`/`job` -> target labels; with honor disabled, any metric attribute literally named -> `namespace` is overwritten and the original is moved to -> `exported_namespace`. A `{namespace=...}` selector would then silently -> match nothing. Hence the prefixed keys above -> (`gittarget_namespace`, `provider_namespace`, …), matching the existing -> convention where every gitopsreverser metric avoids bare `namespace`. - -**Restart test reframe:** - -1. Rollout-restart the controller deployment (`restartControllerDeployment` - blocks until the rollout completes). -2. Discover the new controller pod — after the rollout there is exactly one - non-terminating pod matching `control-plane=gitops-reverser`. -3. Wait for `sum(target_reconcile_completed_total{gittarget_*,pod=""}) - > 0` for the restart test's GitTarget. -4. Wait for `sum(branch_worker_queue_depth{provider_*,branch,pod=""}) - == 0`. -5. Apply a short stability window (a few seconds). -6. Assert no destructive commit happened. - -Both gates pin the `pod` target label to the freshly started pod, so neither -the old pod's lingering samples nor a counter reset can produce a false -pass/timeout during the rollout's brief two-series overlap. - -**Risks:** - -- The new metrics' contracts must be exact enough to be load-bearing in - a test. Document the lifecycle of each in code comments. Treat as a - public observability surface from day one — renaming costs cluster - rollouts. -- Worker-queue-depth must include in-flight items, not just queued ones, - or step 4 races with the worker. Verify against the actual - implementation when writing the metric. **Implemented:** depth = - `inflightItems + (hasUnpushedWork ? 1 : 0)`, where `inflightItems` - (`atomic.Int64`) is incremented *before* each `eventQueue` send and - decremented only *after* `handleQueueItem` returns, and `hasUnpushedWork` - (`atomic.Bool`, refreshed each loop iteration) tracks an open commit - window or retained pending writes. Using `len(eventQueue)` would read 0 - the instant the loop receives an item — before it is handled — so a scrape - in that window could satisfy the drain gate mid-commit; counting accepted - items instead closes that race. The gauge reaches 0 only once every - accepted item is fully handled and a successful push has cleared - `pendingWrites`. See the in-flight follow-up below. - -**Exit criterion:** - -- Spec runs in < 30 s (down from 128 s). -- Both metrics are scrapable via the existing Prometheus pipeline. -- Suite still green; no regressions in other restart-related specs. - -**Implementation status (2026-05-31).** Landed on branch `refactors`. - -- New metrics in [internal/telemetry/exporter.go](../../internal/telemetry/exporter.go): - counter `gitopsreverser_target_reconcile_completed_total` (incremented in - `emitSnapshotForRuleChange`'s per-target success branch, - [internal/watch/manager.go](../../internal/watch/manager.go), with the - `trigger` passed by each caller) and gauge - `gitopsreverser_branch_worker_queue_depth` (recorded from the branch - worker event loop + enqueue paths, - [internal/git/branch_worker.go](../../internal/git/branch_worker.go)). -- Unit tests: - [internal/git/branch_worker_metrics_test.go](../../internal/git/branch_worker_metrics_test.go), - [internal/watch/target_reconcile_metric_test.go](../../internal/watch/target_reconcile_metric_test.go). -- **First e2e run failed** — the query used a bare `namespace` label and - matched nothing (the honor_labels collision above). Renaming to the - prefixed keys and adding aggregation fixed it. The controller-side - delivery was correct on the first run (7 snapshot emits, zero "No - reconciler registered"); the bug was purely in the metric's label - contract. -- **Review follow-up 1:** the reconcile signal started as a latched gauge - read with `max(...) == 1`; review flagged that the old pod's stale `1` - could satisfy that during a rollout. Replaced with the counter. -- **Review follow-up 2:** a cross-pod `sum(...) > baseline` delta was then - flagged as still race-prone — counters reset on the new pod, so once the - old series goes stale the new pod's first increment can return the sum to - the baseline rather than above it, causing a false timeout. Replaced with - a **per-pod** gate: discover the new pod after the rollout and assert - `…{pod=""} > 0` (and `branch_worker_queue_depth{…,pod=""} == 0`). -- **Review follow-up 3:** the queue-depth gauge originally derived its - pending count from `len(eventQueue)` plus an `atomic.Bool`. Review flagged - that the channel length drops to 0 the moment the loop *receives* an item, - before `handleQueueItem` finishes, so a scrape in that window could satisfy - the drain gate while a commit/push was still in flight. Replaced the - channel-length read with an `inflightItems atomic.Int64` incremented before - the send and decremented after handling, so an accepted-but-unhandled item - keeps depth `> 0`. Covered by `TestRecordQueueDepth_InflightItemDequeuedButUnhandled` - and verified under `go test -race`. -- **Measured (per-pod gate, 2026-05-31):** restart spec `Restart Snapshot - Safety` ~**67 s** total wallclock (down from ~128 s; earlier gauge/baseline - runs measured 57–67 s — the spread is fixed-setup variance, not the gates), - full smoke suite green (20/20). Both gates resolved within the first poll - after the rollout settled — the remaining time is BeforeAll setup - (Prometheus client, Gitea repo, CRD install, building the mirror, the - rollout restart itself) plus the 15 s stability window, **not** the old - 75 s blind negative-assertion wait. The signal-driven wait does not hit the - <30 s exit-criterion target on its own because the per-spec figure is - dominated by fixed setup, not the drain wait that Phase 3 removed; the - drain wait itself is now ~0 s. Tightening the residual setup/stability cost - is follow-up work. - -## Decisions - -- **File naming.** Files use the form `_e2e_test.go`. The CRD - lifecycle file is `crd_lifecycle_e2e_test.go` (consistent throughout - this doc; earlier drafts used `manager_icecream_lifecycle_test.go` - inconsistently — fixed). -- **Label.** All four new Describes keep `Label("manager")` so - `task test-e2e-manager` works unchanged. Renaming the label is a - separate decision not in this plan. -- **No spec merges in Phase 2.** The earlier draft proposed collapsing - old #18+#19+#20 into one create→update→delete spec; deferred to a - follow-up so the Phase 2 PR is purely mechanical and easy to review. -- **Metric package is `internal/telemetry`.** Confirmed against - `internal/telemetry/exporter.go`. No `internal/metrics` directory - exists. -- **CRD isolation before Serial marking.** Where multiple e2e files - install the same cluster-scoped CRD, the cheap first fix is to give - each file its own CRD group (Phase 2.5). Serial marking is still - required when the remaining cluster-wide side effect cannot be - isolated by name, such as controller restarts or CRD discovery/GVR - catalog churn. +Split the 21-spec `Manager` Describe into four files, each its own top-level +Describe with per-file fixtures, replacing the package-global `managerRepo` with +per-Describe locals: + +- `controller_basics_e2e_test.go` (no Gitea repo) +- `gitprovider_validation_e2e_test.go` +- `watchrule_configmap_secret_e2e_test.go` +- `crd_lifecycle_e2e_test.go` (owns its own per-file CRD) + +Spec count and observable behavior preserved. This phase alone is *slower* +sequentially (more namespace teardowns); it's the prerequisite for Phase 2.5. + +### Phase 2.5 — Bounded Ginkgo parallelism + Serial registry + +Parallelism is driven by the `ginkgo` CLI (`--procs=$E2E_GINKGO_PROCS`, default +`2`), not `go test`. `SynchronizedBeforeSuite` runs `prepare-e2e` + CRD +pre-cleanup once on process #1. Per-file CRD-group isolation +([test/e2e/icecream.go](../../test/e2e/icecream.go)) gives `crd_lifecycle`, +`restart_snapshot`, and `bi_directional` their own API groups so they never +collide by name. The [e2e-serial-registry.md](e2e-serial-registry.md) captures +the specs that still need `Serial`. + +**Measured (local, procs=2):** smoke **352 s** vs 418.7 s sequential (~16 %). +**CI runs the `full` leg at `E2E_GINKGO_PROCS=1`** — on stock `ubuntu-latest` +runners the singleton controller is CPU-starved under two concurrent streams plus +its own signing/git work, so a different spec timed out each run. Local/default +stays `procs=2`. + +### Phase 3 — Drain-signal metrics for `Restart Snapshot Safety` + +Replaced the ~75 s blind negative-assertion wait with two metrics, both gated on +the *new* controller pod (`pod=""`) so a rollout's brief two-series overlap +can't produce a false pass: + +- `gitopsreverser_target_reconcile_completed_total{gittarget_namespace,gittarget_name,trigger}` + — a **counter** (not a latched gauge): on a fresh pod it starts at 0, so + `> 0` proves *that pod* completed its own post-restart snapshot reconcile. +- `gitopsreverser_branch_worker_queue_depth{provider_namespace,provider_name,branch}` + — a gauge that reaches 0 only once every accepted item is fully handled and a + successful push has cleared `pendingWrites` (depth counts in-flight items via + an `atomic.Int64`, not `len(eventQueue)`, to avoid a mid-commit false drain). + +Metrics live in [internal/telemetry/exporter.go](../../internal/telemetry/exporter.go). +Label keys are prefixed (`gittarget_namespace`, not bare `namespace`) because the +pod scrape with `honor_labels: false` would otherwise overwrite a bare +`namespace` and move it to `exported_namespace`. + +**Measured:** restart spec ~128 s → ~67 s. Both gates resolve on the first poll +after the rollout settles; the residual time is fixed setup (Prometheus client, +Gitea repo, CRD install, building the mirror, the rollout itself) plus the 15 s +stability window — not the drain wait, which is now ~0 s. + +## Considered and skipped + +Recorded so revisiting is cheap. + +1. **Short-circuit `BeforeSuite` to skip warm `prepare-e2e`.** Re-implementing + the stamp freshness check in Go duplicates logic `task` already owns and gives + two sources of truth. The ~1.4 s warm cost is small; Phase 1's DAG is the + better lever. *Revisit if* warm prepare grows back past a few seconds. + +2. **CI matrix sharding by Ginkgo label.** GitHub Actions throttles this repo at + 3+ concurrent runners, so shards queue instead of running in parallel — + defeating the wallclock benefit while burning more CPU minutes. *Revisit if* we + get real concurrent runner capacity, a different CI provider, or self-hosted + runners. + +3. **CI cluster reuse (self-hosted runner or Tekton).** Both require durable + operational ownership (a VM/cluster to secure, upgrade, monitor, pay for) plus + multi-tenant isolation between concurrent pipelines. Phases 1–3 attack the + cheaper per-run cost first. *Revisit if* CI wallclock is still dominated by + cluster boot after Phases 1–3 *and* the team will take on the ops burden. + +4. **Removing `Ordered` *within* the new Manager files.** Phase 2 already lets the + four files run on four workers; intra-file parallelism only helps a file with + many slow specs, which the post-split layout doesn't have, and adds real + ordering-correctness risk. *Revisit if* one file grows a slow tail of + independent specs. + +5. **Tekton / Shipwright "cloud-native CI".** Iteration speed is dominated by + cluster boot and test execution, not the build step; pushing images through a + registry every build makes per-iteration cost go *up*. The one legitimate + angle (warm infra on a persistent cluster) is item 3 with extra moving parts. + *Revisit if* we want pipeline-as-CRD for a separate reason (preview + environments, on-demand test infra). ## Reproducing the baseline @@ -713,5 +174,3 @@ go test -timeout 15m ./test/e2e/ \ -ginkgo.json-report=/tmp/e2e-baseline/smoke.json go run ./test/e2e/tools/spec-timings /tmp/e2e-baseline/smoke.json ``` - -(`test/e2e/tools/spec-timings/` is a Phase 0 deliverable.) diff --git a/docs/design/e2e-speedup-skipped.md b/docs/design/e2e-speedup-skipped.md deleted file mode 100644 index fbf5b8b3b..000000000 --- a/docs/design/e2e-speedup-skipped.md +++ /dev/null @@ -1,122 +0,0 @@ -# E2E Speedup — Items Considered and Skipped - -Companion to [e2e-speedup-plan.md](e2e-speedup-plan.md). This document records -proposals that were evaluated during the 2026-05-28 planning round and -*excluded* from the active plan, with the reasoning. The point is to make -future revisiting cheap: when someone asks "did we look at X?", the answer -and the *why* are here, not lost in chat history. - -## 1. Short-circuit `BeforeSuite` to skip warm `task prepare-e2e` - -**Proposal.** [test/e2e/e2e_suite_test.go:64](../../test/e2e/e2e_suite_test.go#L64) -unconditionally re-invokes `task prepare-e2e` on every `go test` run. Warm -cost is ~1.4 s. The idea was to `stat` the `prepare-e2e.ready` stamp from Go -and skip the subshell if it's newer than the relevant inputs. - -**Why skipped.** It duplicates logic that `task` already owns. The whole -point of the stamp files is that `task` itself decides whether to do work or -no-op. Re-implementing the freshness check in Go gives us two sources of -truth for the same thing, with the obvious failure mode that they drift. - -The 1–2 s warm cost is small enough that the better fix — if it ever -matters — is to make `task` cheaper to invoke for an all-no-op DAG (Phase 1 -of the active plan already moves in that direction by converting chains into -parallel `deps:`). - -**Revisit if:** warm prepare ever grows back past a few seconds in a way -that Phase 1's DAG refactor can't fix. - -## 2. CI matrix sharding (split the `full` e2e job by Ginkgo label) - -**Proposal.** [.github/workflows/ci.yml:261-313](../../.github/workflows/ci.yml#L261-L313) -runs the e2e suite as a 2-entry matrix `{quickstart, full}`. Splitting -`full` into 4–5 label-scoped shards (`smoke`, `signing`, `audit-consumer`, -`bi-directional`, `image-refresh`, `aggregated-api`) would parallelize -across GitHub runners and drop wallclock to `max(shard)` instead of -`sum(shard)`. - -**Why skipped.** Empirically GitHub Actions chokes when this repo asks for -3+ concurrent runners — there are throttles in practice that make the -"free parallelism" assumption false. The shards would queue rather than -actually running in parallel, defeating the wallclock benefit and burning -more CPU minutes per run. - -**Revisit if:** - -- We move to a GitHub plan / runner pool with real concurrent capacity, or -- We move to a different CI provider where concurrent runners are not - throttled, or -- We adopt self-hosted runners (see item 3) where we control the - parallelism budget. - -## 3. CI cluster reuse via self-hosted runner or Tekton - -**Proposal.** Today every CI job creates a fresh k3d cluster; that boot is -the dominant CI cost. Two flavours of reuse were considered: - -- **Self-hosted GitHub runner** holding a long-lived k3d. PRs run against - namespace-scoped slices. -- **Tekton on a persistent Kubernetes cluster.** Pipelines run as - CRD-defined Tasks; the host cluster (or warm test fixtures within it) - survives across pipeline runs. - -**Why skipped (for now).** Both require taking on durable operational -responsibility: a VM or cluster we own, secure, upgrade, monitor, and pay -for. Multi-tenant isolation between concurrent PR pipelines (namespaces, -nested k3d, or vCluster) is its own design problem. Tekton additionally -requires running an image registry for CI-built images (`k3d image import` -no longer works once builds live in pods) and maintaining pipeline-as-CRD -definitions alongside GitHub Actions. - -The active plan's Phases 1–3 attack the *per-run* cost of the e2e harness. -That's the cheaper lever to pull first. We can re-measure CI wallclock -after they land, and only then decide whether the remaining cold-boot -cost justifies the ops investment. - -**Revisit if:** Phases 1–3 of the active plan are complete and CI wallclock -is *still* dominated by cluster boot, *and* the team is willing to take on -the ops burden of a persistent runner or cluster. - -## 4. Removing `Ordered` *within* the new Manager files (Ginkgo intra-file parallelism) - -**Proposal.** After Phase 2 of the active plan splits the `Manager` -Describe into four files, also drop the `Ordered` decorator inside each -new file so Ginkgo can parallelize specs within a single file. - -**Why skipped (for now).** Phase 2 already buys the parallelism that -matters: Ginkgo runs *across* top-level Describes in parallel, so the four -new files can run on four workers. Dropping `Ordered` inside each file -would only help if a single file has many slow specs — which the -post-split layout does not. Group D (IceCreamOrder lifecycle) genuinely -needs `Ordered`. Groups A/B/C have cheap specs (most < 5 s); the gain -from intra-file parallelism is small and the correctness risk (implicit -ordering not caught yet) is real. - -**Revisit after:** Phase 2 lands and we have stability data. If one of the -new files has a slow tail of independent specs, drop `Ordered` there -specifically. - -## 5. Move build to Tekton / Shipwright for cloud-native CI - -**Proposal.** Replace (or supplement) GitHub Actions with Tekton pipelines -and/or Shipwright build CRDs, on the theory that "cloud-native CI" would -naturally make iteration faster. - -**Why skipped.** The premise doesn't survive contact with the actual cost -model: - -- Iteration speed is dominated by *cluster boot* and *test execution*, not - by the *build* step. Shipwright only addresses image builds; Tekton is a - pipeline orchestrator, not a test accelerator. -- Pushing/pulling images through a registry on every build (Tekton can't do - `k3d image import` the way local `task` can) makes per-iteration cost go - *up*, not down. -- The legitimate angle — Tekton on a persistent cluster keeping test - infrastructure warm across pipeline runs — is the same problem as item 3 - with extra moving parts. Self-hosted GitHub runner gets ~80 % of the - benefit at ~10 % of the complexity. - -**Revisit if:** we have a separate reason to want pipeline-as-CRD (preview -environments, dynamic on-demand test infrastructure, multi-product -delivery) — in that case the iteration-speed benefit becomes a bonus rather -than the justification. diff --git a/docs/design/refactors-branch-review.md b/docs/design/refactors-branch-review.md deleted file mode 100644 index 360c717f4..000000000 --- a/docs/design/refactors-branch-review.md +++ /dev/null @@ -1,288 +0,0 @@ -# `refactors` branch — review & action checklist - -> Review of `main...refactors` (57 files, ~7.5k added / ~2.1k deleted). -> Generated 2026-06-01 from a clean working tree. -> Covers: (1) code-review findings, (2) docs triage, (3) refactor advice on the -> two e2e docs and the stability-enabled follow-ups. - -The big test-file split (`e2e_test.go` → four `*_e2e_test.go` files) verified -clean: all 23 Manager specs are re-created, assertion counts match, and -CRD/namespace cleanup was correctly redistributed. Findings concentrate in the -new Phase-3 drain metrics and the new e2e helpers. - -## Merge readiness - -### Block before merge - -- [x] Fix the queue-depth metric publication race (finding 1). -- [x] Fix discovery-lag / empty-plan handling for rule-set snapshots (finding 2). -- [x] Simplify CI Allure handling: keep uploading raw Ginkgo JSON and timing - summaries, but remove CI-side Allure result conversion, HTML rendering, - permission fixups, and Allure artifact upload from `.github/workflows/ci.yml`. - Local report generation via `task allure-e2e-report` is enough when a human - needs the visual timeline. - -### Good enough for follow-up - -- [x] Cleanup findings 5-9 (done); finding 10 CI half done, two Go micro-opts - deferred with rationale (see finding 10). -- [x] Docs archiving/relocation in section 2. -- [x] Local Allure docs polish in section 3. - -### Validation already run - -- [x] `git diff --check origin/main...HEAD` -- [x] `task lint-actions` -- [x] `go test ./test/e2e/tools/ginkgo-allure ./test/e2e/tools/spec-timings ./internal/git ./internal/watch` -- [x] `go test -run '^$' ./test/e2e` -- [x] `task lint` -- [x] `task test` -- [x] `task test-e2e` - ---- - -## 1. Code review - -### Correctness (ranked) - -- [x] **1. Stale queue-depth gauge race — can hang the restart drain gate** - — [branch_worker.go:315-331](../../internal/git/branch_worker.go#L315-L331). - `recordQueueDepth()` runs from enqueue goroutines *and* the event loop; it - does `Load(inflightItems)` → … → `Record()` non-atomically. An enqueue - goroutine can load `depth=1`, get preempted while the loop drains the item and - `Record(0)` then idles, and then publish the stale `1`. OTel gauge is - last-writer-wins and the idle loop never republishes → gauge latches at `1`, - so the restart-snapshot gate `branch_worker_queue_depth … == 0` times out at - 90s though the worker is drained. **Fix:** publish the gauge *only* from the - loop goroutine (drop `recordQueueDepth()` from the enqueue paths; the loop - already republishes each iteration via `syncQueueDepthMetric`). - -- [x] **2. Discovery-lag causes spurious full re-snapshots** - — [manager.go:1248](../../internal/watch/manager.go#L1248), - [:1264](../../internal/watch/manager.go#L1264), - [:1280](../../internal/watch/manager.go#L1280), - [:1189-1191](../../internal/watch/manager.go#L1189-L1191). - `currentRuleSetSnapshots` hashes the *resolved* GVR plan and swallows the - resolve error (`gvrs, _ := resolver.Resolve(...)`); an empty-plan target is - dropped and `snapshotTargetsNeedingDelivery` then deletes its - `lastDeliveredRuleSetHash`. A transient API-discovery gap makes a target - vanish and, on recovery, get re-snapshotted (full repo-state diff). New - regression vs the old rule-text hash; it's the same mechanism that forces - `crd_lifecycle` Serial. **Fix:** log the resolve miss instead of discarding, - and don't evict last-delivered state on an *empty* plan (distinguish "no - rules" from "rules that didn't resolve yet"). - **Fixed (simple version):** `currentRuleSetSnapshots` now keeps targets that - still have rules even when the resolved entry set is empty, and - `snapshotTargetsNeedingDelivery` skips those empty plans without selecting or - pruning them. This preserves the delivered baseline across transient discovery - gaps. Known accepted tradeoff: `valid -> stable-empty -> valid` user edits do - not force a recovery snapshot until the richer settled-empty refinement lands. - Detailed proposal: - [rule-set-snapshot-discovery-lag-fix.md](rule-set-snapshot-discovery-lag-fix.md). - -- [x] **3. Reconcile counter skipped on transient error / fired on no-op** - — [manager.go:1384-1392](../../internal/watch/manager.go#L1384-L1392). - `recordTargetReconcileCompleted` fires only after **both** `RequestRepoState` - and `RequestClusterState` succeed. A transient error on the second `continue`s - (counter never increments though partial state was enqueued) → restart - `{pod=…} > 0` gate can time out at 90s on a one-off error. It also fires on a - no-op snapshot, so `increase(target_reconcile_completed_total)` over-counts - for non-test consumers. - **Fixed (transient-error half):** `emitSnapshotForRuleChange` now returns a - joined error for any target whose repo/cluster-state request failed; the target - is left pending (not marked delivered, counter not fired) and - `ReconcileForRuleChange` propagates the error so the controller requeues with - backoff and retries promptly instead of waiting out the 30s periodic tick. - The "fires on a no-op snapshot" half is **left intentionally**: the counter's - documented contract is to count snapshot passes that reached the git write path - (not commits produced), and a clean post-restart re-snapshot legitimately - produces an empty diff yet must still increment so the restart gate can observe - `{pod=} > 0`. Gating it on a non-empty diff would break that load-bearing - signal. - -- [x] **4. Shutdown leaves the depth gauge non-zero** - — [branch_worker.go:585](../../internal/git/branch_worker.go#L585), - [:704](../../internal/git/branch_worker.go#L704). - On `ctx.Done()` the loop finalizes + pushes and returns **without draining - items still buffered in `eventQueue`**; each was counted into `inflightItems` - at enqueue, so the final `recordQueueDepth` publishes a non-zero depth for the - exiting pod that never clears. Shielded today by the per-pod gate; latent leak - for any cross-pod use. - -### Cleanup / efficiency - -- [x] **5. ~13 inlined `git pull` blocks duplicate `pullLatestRepoState`** - — [watchrule_configmap_secret_e2e_test.go:174](../../test/e2e/watchrule_configmap_secret_e2e_test.go#L174) - (+~6 more) and ~7 in `crd_lifecycle_e2e_test.go`. The canonical helper - (fetch + checkout -B + reset --hard) is already used at - [:707](../../test/e2e/watchrule_configmap_secret_e2e_test.go#L707) and is more - robust than a bare `git pull` (which can flake on a diverged branch). - **Fixed:** all 6 blocks in `watchrule_configmap_secret_e2e_test.go` and all 10 - in `crd_lifecycle_e2e_test.go` (both the error-checking and ignore-error - forms) now call `pullLatestRepoState(g, …)`. -- [x] **6. `gitPull` / `lastCommitMessageForPath` reinvent existing helpers** - — [gittarget_isolation_e2e_test.go:187](../../test/e2e/gittarget_isolation_e2e_test.go#L187). - Third pull implementation (no fetch/reset) + manual `git log` where - `gitRun(dir, args...)` already exists in helpers.go. - **Fixed:** call site uses `pullLatestRepoState`; the local `gitPull` is removed - and `lastCommitMessageForPath` goes through `gitRun`. Unused `os/exec` import - dropped. -- [x] **7. Hand-rolled `minInt`** — [e2e_test.go:264](../../test/e2e/e2e_test.go#L264); - used once, Go 1.26 builtin `min` covers it. **Fixed:** `minInt` deleted, sole - caller uses builtin `min`. -- [x] **8. Redundant "healthy GitProvider" spec** - — [watchrule_configmap_secret_e2e_test.go:80](../../test/e2e/watchrule_configmap_secret_e2e_test.go#L80) - re-creates/re-verifies `gitprovider-normal` already built in `BeforeAll` - ([:60-70](../../test/e2e/watchrule_configmap_secret_e2e_test.go#L60-L70)). - **Fixed:** the spec now only re-asserts the BeforeAll-created provider stays - Ready; the redundant re-creation is gone. -- [x] **9. Duplicated endpoint→pod filtering** - — [controller_basics_e2e_test.go:99](../../test/e2e/controller_basics_e2e_test.go#L99) - vs [:135](../../test/e2e/controller_basics_e2e_test.go#L135); extract - `expectServiceRoutesToPod`. **Fixed:** both blocks call the new - `expectServiceRoutesToPod(g, service, pod)` helper in `helpers.go`; the - now-unused `strings` import is dropped. -- [x] **10. Wasted work** — CI "Summarize E2E timing reports" `go run`s - spec-timings per report in a loop ([ci.yml:388](../../.github/workflows/ci.yml#L388)), - recompiling N+1× (`go build -o` once instead); `currentRuleSetSnapshots` - resolves GVRs twice per reconcile ([manager.go:1224](../../internal/watch/manager.go#L1224)); - `syncQueueDepthMetric` records every loop iteration rather than on change. - **Fixed (CI half):** the summary step now `go build -o /tmp/spec-timings` once - and runs the binary per report instead of `go run` recompiling each time. - **Deferred intentionally (the two Go halves):** `syncQueueDepthMetric`'s - per-iteration `Record` is the load-bearing republish that finding #1's fix - relies on to keep the OTel gauge from latching a stale depth — switching it to - record-on-change would reintroduce that race. The "resolves GVRs twice" cost - spans two *different* methods (plan-hash in `currentRuleSetSnapshots` vs. - watch/list setup in the snapshot builder), so collapsing it needs a - per-reconcile resolution cache rather than a local tidy-up; left for a focused - follow-up. - -> The Allure env-var overrides + `specSkippedByFilter` + per-process thread -> label look good. Caveat: `specSkippedByFilter` -> ([ginkgo-allure/main.go:263](../../test/e2e/tools/ginkgo-allure/main.go#L263)) -> infers "filtered-out" from `LabelFilter!="" && Skipped && EndTime.IsZero() && -> RunTime==0`; a real `Skip()` recording zero timing would also be dropped. Low -> risk — a heuristic, not a guarantee. - ---- - -## 2. Docs triage - -> `docs/finished/` already exists (empty) but isn't in the README — adopt it as -> the "done" folder and document it in [docs/README.md](../README.md). - -### Move to `docs/finished/` — shipped/resolved - -- [x] [finished/gittarget-isolation-on-rule-change.md](../finished/gittarget-isolation-on-rule-change.md) — ✅ shipped (this branch) -- [x] [finished/commit-window-refactor.md](../finished/commit-window-refactor.md) — implemented -- [x] [finished/e2e-cleanup-scoping.md](../finished/e2e-cleanup-scoping.md) — implemented -- [x] [finished/partial-object-audit-event-handling.md](../finished/partial-object-audit-event-handling.md) — implemented -- [x] [finished/path-scoped-bootstrap-template-design.md](../finished/path-scoped-bootstrap-template-design.md) — implemented -- [x] [finished/remove-gittarget-admission-webhook.md](../finished/remove-gittarget-admission-webhook.md) — webhook file gone → done -- [x] [finished/watch-gvr-resolution-cleanup.md](../finished/watch-gvr-resolution-cleanup.md) — `APIResourceCatalog` cleanup landed -- [x] [finished/watchrule-gvr-resolution-plan.md](../finished/watchrule-gvr-resolution-plan.md) — implemented -- [x] [finished/aggregated-api-requestheader-ca-miswire-report.md](../finished/aggregated-api-requestheader-ca-miswire-report.md) — resolved report -- [x] [finished/audit-consumer-username-attribution-gap.md](../finished/audit-consumer-username-attribution-gap.md) — resolved -- [x] [finished/e2e-runtime-state-vs-stamps.md](../finished/e2e-runtime-state-vs-stamps.md) — resolved failure mode -- [x] [finished/shallow-audit-event-misclassification.md](../finished/shallow-audit-event-misclassification.md) — root cause confirmed -- [x] [finished/e2e-test-review.md](../finished/e2e-test-review.md) — point-in-time review -- [x] [finished/git-author.md](../finished/git-author.md) — OIDC author plan, implemented -- [x] [finished/design-commit-request-api.md](../finished/design-commit-request-api.md) — implemented -- [x] [finished/design-rule-change-snapshot-trigger.md](../finished/design-rule-change-snapshot-trigger.md) — implemented -- [x] [finished/design-audit-ingestion-hardening.md](../finished/design-audit-ingestion-hardening.md) — implemented -- [x] [finished/design-commit-context-api.md](../finished/design-commit-context-api.md) — superseded -- [x] [finished/idea-audit-enrichment-side-channel.md](../finished/idea-audit-enrichment-side-channel.md) — superseded - -### Consider removing / relocating - -- [x] [serious-bug/cozystack-bugreport.md](../serious-bug/cozystack-bugreport.md) — upstream bug report against - CozyStack, not our docs. If filed upstream, replace with a link. Referenced by - `initial-seed-via-watch-list.md`, so update that link; `serious-bug/` is the - safe relocation. **Done:** moved to - [serious-bug/cozystack-bugreport.md](../serious-bug/cozystack-bugreport.md); - both references in `initial-seed-via-watch-list.md` updated. -- [x] [ci/findings.md](../ci/findings.md) — stale (2026-02-13), overlaps - [e2e-ci-stability-report.md](e2e-ci-stability-report.md). Fold true bits in, drop it. - **Deviated from the recommendation:** on inspection the two docs do *not* - overlap — `ci/findings.md` is devcontainer/CI infrastructure rationale - (workspace mounts, Go cache volumes, in-container cluster networking), while - `e2e-ci-stability-report.md` is about test stability and the e2e job's disk - pressure. Deleting it would lose useful rationale. It *was* stale on specifics - (referenced Kind + `make setup-cluster`; the project now uses k3d + `task`), so - it was refreshed in place (k3d node naming, `start-cluster.sh` port handling, - updated date) and kept. -- [x] Empty `docs/finished/`, `docs/serious-bug/`, `docs/contributing/` — populate - (per above) or remove the empty scaffolding. **Done:** `finished/` was already - populated by the section-2 moves and is now documented in - [docs/README.md](../README.md); `serious-bug/` is populated by the cozystack - relocation and documented in the README; the empty `docs/contributing/` was - removed (the contributor guide lives at top-level `CONTRIBUTING.md`). - -### Keep on the active design surface - -Still-relevant plans/proposals: [e2e-speedup-plan.md](e2e-speedup-plan.md), -[e2e-serial-registry.md](e2e-serial-registry.md), -[e2e-ci-stability-report.md](e2e-ci-stability-report.md), -[e2e-speedup-skipped.md](e2e-speedup-skipped.md), -[audit-metrics-overhaul-plan.md](audit-metrics-overhaul-plan.md) (proposed), -[initial-seed-via-watch-list.md](initial-seed-via-watch-list.md) (proposed) + -[upgrade-finding.md](../upgrade-finding.md), the sensitive-resource plan + -follow-up, and [e2e-watchrule-cross-spec-interference.md](e2e-watchrule-cross-spec-interference.md) -(status: **open**). All `future/idea-*.md` stay. Reference guides -(`kubernetes-*`, `crd-relationships`, `status-conditions-guide`, audit-webhook -designs, lifecycle/architecture) stay. All top-level user guides + `ci/*` -explainers + `demo/*` + `audit-setup/*` stay. - ---- - -## 3. The two e2e docs + stability-enabled refactors - -**[e2e-allure-reporting.md](../ci/e2e-allure-reporting.md)** — accurate for the -current branch, but should be updated if CI stops rendering Allure: - -- [x] If `.github/workflows/ci.yml` is simplified, rewrite the CI section to say - CI uploads only raw Ginkgo JSON/timing artifacts; developers can download - those reports and run `task allure-e2e-report` locally. -- [x] Document the new per-process **`thread` Allure label** the converter now - emits (the point of the parallel-insight work). **Done:** added an "Allure - labels" section to `e2e-allure-reporting.md` documenting `thread` plus the - `suite`/`package`/`parentSuite`/`subSuite`/`tag` labels. - -**[e2e-serial-registry.md](e2e-serial-registry.md)** — the Serial entries are -**correctness-driven (shared state), not flakiness-driven**, so improved -stability doesn't justify de-serializing directly. It *does* unlock the deeper -refactors the registry is implicitly waiting for. Both items below were planned -in [e2e-serial-deserialization-plan.md](e2e-serial-deserialization-plan.md) and -then partly executed: - -- [~] **Audit pipeline.** The planned heavy refactor (per-test stream/consumer - isolation, plan option B1) was **not** done — it bends the intentionally - singleton audit pipeline for test parallelism and isn't worth the - production-correctness risk. Instead the audit-consumer Serial set was *reduced*: - `Audit Redis Queue` and `Audit Redis Consumer` - (`audit_redis_e2e_test.go`, now deleted) were retired — their producer-path and - basic consumer-commit assertions duplicated the WatchRule suite. The one unique - assertion (OIDC `user.extra` → commit author) moved to a new **parallel** - [commit_author_attribution_e2e_test.go](../../test/e2e/commit_author_attribution_e2e_test.go): - a dedicated 0s-commit-window GitProvider + per-path author read makes it - concurrency-safe, so it needs no `Serial`. `Commit Window Batching` and - `Commit Request` **stay Serial** — they assert batching/commit-window semantics - over the shared stream, which concurrent audit traffic genuinely violates. -- [x] **De-serialize `crd_lifecycle` (was gated on code finding #2).** **Done:** - with finding #2's fix shipped (a target only resnapshots when its *resolved* - plan hash changes) and per-file CRD groups keeping the `IceCreamOrder` CRD - name-isolated, `Manager CRD Lifecycle` dropped `Serial` and now runs `Ordered` - in the parallel pool. Verified green under `--procs=4` (the only other - wildcard `ClusterWatchRule`, in `restart_snapshot`, is itself Serial, so no - concurrent spec matches the icecream group). Registry updated. -- `bi_directional` (exact commit-count) and `restart_snapshot` / `image_refresh` - (singleton controller) + `aggregated_apiserver` (cluster `APIService`) are - genuinely serial — leave them. - -- [ ] **Confidence-window pass (the speedup plan's open thread).** Now that the - race fixes landed (`03b95ab`, `0787b1b`), decide whether CI should keep - `K3D_AGENT_COUNT=0` and the current per-matrix Ginkgo parallelism, or move - back toward agent-backed clusters. Use the uploaded timing summaries and raw - Ginkgo JSON as the evidence source; render Allure locally only when the visual - report is needed. diff --git a/docs/upgrade-finding.md b/docs/design/upgrade-finding.md similarity index 100% rename from docs/upgrade-finding.md rename to docs/design/upgrade-finding.md diff --git a/docs/design/watch-and-catalog-architecture.md b/docs/design/watch-and-catalog-architecture.md index 22451ebb4..760fc7e6f 100644 --- a/docs/design/watch-and-catalog-architecture.md +++ b/docs/design/watch-and-catalog-architecture.md @@ -1,9 +1,9 @@ # Watch & Catalog Architecture — Requirements, Current State, Target Design Status: **vision / proposed** — supersedes the framing in -[watchrule-wildcard-support-plan.md](watchrule-wildcard-support-plan.md), +[watchrule-wildcard-support-plan.md](../future/watchrule-wildcard-support-plan.md), [watchrule-wildcard-and-resolution-semantics.md](watchrule-wildcard-and-resolution-semantics.md), -and [rule-set-snapshot-discovery-lag-fix.md](rule-set-snapshot-discovery-lag-fix.md); +and [rule-set-snapshot-discovery-lag-fix.md](../finished/rule-set-snapshot-discovery-lag-fix.md); those become implementation details of the model described here. This document is in three parts: diff --git a/docs/design/watchrule-wildcard-and-resolution-semantics.md b/docs/design/watchrule-wildcard-and-resolution-semantics.md index c0e818335..6b4fe8a22 100644 --- a/docs/design/watchrule-wildcard-and-resolution-semantics.md +++ b/docs/design/watchrule-wildcard-and-resolution-semantics.md @@ -12,7 +12,7 @@ Source of truth: `RuleGVRResolver` in > **Note:** this documents *current* behavior. The CRD field docs promise > admission-webhook-style wildcard semantics that this resolver does not yet > implement — see -> [watchrule-wildcard-support-plan.md](watchrule-wildcard-support-plan.md) for +> [watchrule-wildcard-support-plan.md](../future/watchrule-wildcard-support-plan.md) for > that finding and the plan to resolve it. ## Two directions, treated differently @@ -101,7 +101,7 @@ expressible. CRD is installed or removed, churning the per-GitTarget effective-watch-plan hash on unrelated cluster activity — the same spurious re-snapshot problem described in - [rule-set-snapshot-discovery-lag-fix.md](rule-set-snapshot-discovery-lag-fix.md), + [rule-set-snapshot-discovery-lag-fix.md](../finished/rule-set-snapshot-discovery-lag-fix.md), but permanent and amplified. Refusing wildcards keeps watch plans deterministic. 3. **Snapshot safety.** Snapshotting `*` would commit the entire cluster (minus @@ -126,6 +126,6 @@ two families differ in nature: error to surface to the user. This distinction matters for the state model in -[rule-set-snapshot-discovery-lag-fix.md](rule-set-snapshot-discovery-lag-fix.md): +[rule-set-snapshot-discovery-lag-fix.md](../finished/rule-set-snapshot-discovery-lag-fix.md): only the transient family should drive "retry next cycle"; the wildcard family will not resolve on its own. diff --git a/docs/design/rule-set-snapshot-discovery-lag-fix.md b/docs/finished/rule-set-snapshot-discovery-lag-fix.md similarity index 98% rename from docs/design/rule-set-snapshot-discovery-lag-fix.md rename to docs/finished/rule-set-snapshot-discovery-lag-fix.md index 0c1b1d259..95923ef7a 100644 --- a/docs/design/rule-set-snapshot-discovery-lag-fix.md +++ b/docs/finished/rule-set-snapshot-discovery-lag-fix.md @@ -1,9 +1,11 @@ # Rule-Set Snapshot Discovery-Lag Fix +> Status: **shipped**. Drives the per-GitTarget effective-watch-plan hash in +> `internal/watch/manager.go`. + ## Problem -Finding 2 in [refactors-branch-review.md](refactors-branch-review.md) is about -the new per-GitTarget effective-watch-plan hash: +This fix introduced the per-GitTarget effective-watch-plan hash: - `currentRuleSetSnapshots()` resolves rules through the API resource catalog. - It ignores `ResolveMiss` values from `resolver.Resolve(...)` diff --git a/docs/design/initial-seed-via-watch-list.md b/docs/future/initial-seed-via-watch-list.md similarity index 97% rename from docs/design/initial-seed-via-watch-list.md rename to docs/future/initial-seed-via-watch-list.md index 7f82fcae0..7cf831e3e 100644 --- a/docs/design/initial-seed-via-watch-list.md +++ b/docs/future/initial-seed-via-watch-list.md @@ -3,7 +3,7 @@ > Status: proposed — not yet implemented > Scope: `internal/watch` snapshot path, dynamic informer construction, the > client-go `WatchListClient` feature gate -> Related: [`docs/upgrade-finding.md`](../upgrade-finding.md), +> Related: [`docs/design/upgrade-finding.md`](../design/upgrade-finding.md), > [`docs/serious-bug/cozystack-bugreport.md`](../serious-bug/cozystack-bugreport.md) ## Goal @@ -16,7 +16,7 @@ and stop issuing a separate hand-rolled `LIST` per GVR per namespace whenever a Use the client-go `WatchListClient` feature (streaming-list, on by default since client-go v1.35 / v0.36.x) as the *transport* that hydrates that cache, rather than something to fight against. The bookmark-warning we observed in -production ([`docs/upgrade-finding.md`](../upgrade-finding.md)) was a hint that +production ([`docs/design/upgrade-finding.md`](../design/upgrade-finding.md)) was a hint that we had silently opted into streaming-list without redesigning the seed path around it; the right response is to redesign, not to disable the gate. @@ -220,7 +220,7 @@ instead of hanging the reflector. change the filter contract. - The audit-ingestion pipeline. Snapshot path and audit path are independent; this plan does not touch - [`docs/design/audit-metrics-overhaul-plan.md`](audit-metrics-overhaul-plan.md). + [`docs/design/audit-metrics-overhaul-plan.md`](../design/audit-metrics-overhaul-plan.md). - A binary-level "disable WatchListClient" flag. Explicitly rejected by the decision above. Operators who need to disable the gate in an emergency can still set `KUBE_FEATURE_WatchListClient=false` via the chart's `.Values.env` diff --git a/docs/design/watchrule-wildcard-support-plan.md b/docs/future/watchrule-wildcard-support-plan.md similarity index 97% rename from docs/design/watchrule-wildcard-support-plan.md rename to docs/future/watchrule-wildcard-support-plan.md index 6d984474b..94d67c6e2 100644 --- a/docs/design/watchrule-wildcard-support-plan.md +++ b/docs/future/watchrule-wildcard-support-plan.md @@ -4,7 +4,7 @@ Status: **proposed** (not started) > **Superseded framing:** the finding below stands, but the recommended way to > resolve it is the unified model in -> [watch-and-catalog-architecture.md](watch-and-catalog-architecture.md). Treat +> [watch-and-catalog-architecture.md](../design/watch-and-catalog-architecture.md). Treat > this document as the focused finding + a tactical (Phase-0-first) fallback if > the larger refactor is deferred. @@ -40,7 +40,7 @@ does the opposite: There is **no admission validation** rejecting these (only `Operations` has an enum), so a user can `kubectl apply` a `resources: ["*"]` rule, get no error, and **silently watch nothing** — only a `V(1)` resolve-miss log. See -[watchrule-wildcard-and-resolution-semantics.md](watchrule-wildcard-and-resolution-semantics.md) +[watchrule-wildcard-and-resolution-semantics.md](../design/watchrule-wildcard-and-resolution-semantics.md) for the full current semantics. ### Why it matters @@ -80,7 +80,7 @@ is **already built and running**: - Per-GitTarget snapshot isolation already scopes which targets re-snapshot on a plan change ([gittarget-isolation-on-rule-change.md](../finished/gittarget-isolation-on-rule-change.md), - [rule-set-snapshot-discovery-lag-fix.md](rule-set-snapshot-discovery-lag-fix.md)). + [rule-set-snapshot-discovery-lag-fix.md](../finished/rule-set-snapshot-discovery-lag-fix.md)). So "we'd have to redo reconciles when a GVR is added" is **already the steady state** for explicit rules. Wildcard support mostly means letting the resolver