Repository navigation
feat(server): GitOps onboarding — configure everything via external CUE repo - #59
Merged
Merged
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| .record_error(error.clone(), now_seconds()); | ||
| tracing::warn!(%error, "config repo sync failed"); | ||
| } | ||
| Err(join) => tracing::warn!(%join, "config sync task panicked"), |
Foundation for GitOps config sync: an instance's configuration becomes a real CUE module evaluated into the typed InstanceConfig, and the pure-Rust gitsync engine is pinned for the upcoming sync wiring. - core: evaluate_instance_config(dir) runs cuengine over a `package comtrya` module read-only (never dirtying the source worktree), deserializes the unified root value through serde wire structs, converts to the domain InstanceConfig, and validates once. Untyped JSON stays inside this module. - core: extend InstanceConfig with admins, repositories, and labels — the GitOps source of truth for OIDC-claim admins, declared repos, and global inheritable labels — each with validation (admin needs a matcher; repo paths are slugs within the visibility ceiling; labels need #rrggbb). - server: pin gitsync to git `main` (v2.0.0 pending). It is gix-based, so no system libgit2, and re-exports gitsync::Oid. Wired into startup next.
Add the GitOps config-repo source: when COMTRYA_CONFIG_REPO_URL is set, the server clones it (gitsync) into data_dir/config-repo on startup, fast-forwards to the latest commit, and evaluates the checked-out CUE module into the typed InstanceConfig. A synced repo is the source of truth, so it declares the full extension set. - config_sync::ConfigRepo::from_env builds a gitsync::GitSync from the environment (URL, ref, HTTPS token/username or SSH key + passphrase) and returns None when no repo URL is set. - bootstrap_and_load clones-if-absent, fast-forwards, logs the synced commit, and runs comtrya_core::evaluate_instance_config over the checkout. - Runtime::start resolves config in precedence order: config repo, then the local --config file, then the unconfigured dev default. The periodic 15-min sync loop, the unconfigured read-only UI state, and deleting the legacy string-parser config loader follow in subsequent steps.
Remove the local config-file string parser and make the synced CUE config repo the sole source of instance config. Resolution is now the config repo (gitsync) or an unconfigured dev default — no --config / COMTRYA_CONFIG file path. - Delete the ~470-line stringly-typed config loader (load_config_file*, LoadedConfig, the cue_*/quoted_* helpers) and its 6 tests; config is parsed only through comtrya_core::evaluate_instance_config now. - Split Runtime::start into start() (resolves the config source) and start_with_config() so tests inject a config directly instead of via a file. Drop StartupOptions.config_path, COMTRYA_CONFIG, and --config. - A config repo that declares extensions is authoritative; an omitted/empty set falls back to first-party discovery (preserves current behaviour). - fixtures/config-repo: bundled CUE config module. start.sh materialises it into a local git repo and points COMTRYA_CONFIG_REPO_URL at it via file://; override the env var to use your own repo. config/*.cue deleted. - Tests: clone a file:// repo via gitsync and evaluate it; evaluate the bundled fixture. Update ARCHITECTURE / PRODUCTION_TESTBED / RUNBOOK / CONTAINER docs. The e2e ./start.sh smoke still needs a real run to confirm the shell wiring; the gitsync clone + evaluate path itself is covered by the new tests.
Add a background task that fast-forwards the config repo on an interval (COMTRYA_CONFIG_SYNC_INTERVAL_SECONDS, default 900s) and re-evaluates on change so validation errors surface. Applying the new config live is the reconciler's job (next phase); unconfigured instances no-op. - config_sync: SyncStatus (configured, repoURL, lastCommit, lastSyncedUnix, lastError, intervalSeconds) + SyncTick. bootstrap_and_load now returns the synced commit; sync_and_validate fast-forwards and re-evaluates on change. - Runtime carries config_sync_status; start() seeds it from the initial sync and the loop updates it; admin_telemetry surfaces it as `configSync`. - main() spawns the loop beside the eviction task (spawn_blocking for the gitsync I/O) and aborts it on shutdown. The unconfigured signal (configSync.configured = false) is surfaced now; the read-only "not configured" UI lands with the admin surface in Phase 7.
…oncile Introduce admin authorization (admins come from OIDC claims, never local auth) and apply admin/OIDC config changes live on each sync. - Add PrincipalStatus/StoredPrincipal AdminCredential. On OIDC login, claims matching a configured admin mint an AdminCredential session; the three authenticated-credential authz checks include it. - Runtime holds a live admin allow-list (RwLock) seeded from config; is_admin_claims matches OIDC issuer/subject/email. adminTelemetry is gated — non-admins receive null. - reconcile::reconcile_live applies the admin list + OIDC issuers (AuthService::set_issuers keeps provisioned users) on each background sync; config_sync::poll returns the changed config to drive it. - Tests: admin identity matching + live reconcile of the admin list. The manual "Sync now" mutation and the read-only unconfigured UI land with the admin surface in Phase 7.
On each config sync the reconciler brings repositories in line with the config repo: declared-but-missing repos are created (with their declared visibility and description), and repos absent from config are deleted — document and on-disk bare repo — since the config repo is the source of truth. - Runtime::repository_docs (canonical path + id diff source), create_declared_repository (create + stamp visibility/description), delete_repository (remove doc + bare repo dir, emits repository.deleted). - reconcile::reconcile_repositories diffs desired vs actual; invalid declared paths are skipped with a warning, not fatal. Wired into reconcile_live so the background sync applies it. - Test: a declared repo is created and an undeclared one is deleted.
Add a `labels` core storage collection and reconcile the instance-declared label set on each config sync: create missing, update color/description, and delete labels absent from config (the config repo is authoritative). Labels are keyed by (scope, name) so global (scope=None) and scoped labels coexist; exposed on the GraphQL `labels` field. - core_storage_collections registers `labels` (by_scope_name index). - Runtime: label_docs / create_label / update_label / delete_label. - reconcile::reconcile_labels (keyed diff) wired into reconcile_live. - Test: create, recolor, and delete labels across two reconciles. Per-repo scoped labels declared in a repo's own CUE (effective set = global union repo, via evaluate_repo_config) build on this storage + scope field next.
Surface when the synced config's enabled-extension set differs from the set loaded at process start, via SyncStatus.pendingExtensionReload. OIDC, admins, repos, and labels apply live; the extension component set is applied on the next restart rather than swapped in-process — a consistent in-process swap must atomically replace both the compiled-component registry and the served record map (UI manifests, dispatch metadata) while no request is mid-flight, tracked as a follow-up. - SyncStatus.pending_extension_reload, surfaced in admin telemetry. - reconcile::reconcile_extensions compares enabled ids against the startup set and records the signal; wired into reconcile_live. - Test: drift flips the flag; an unchanged set does not.
Add the syncConfig GraphQL mutation backing the /admin "Sync now" button: admin-gated, fast-forwards the config repo, reconciles live on change, and returns the current sync status. Sync failures are recorded in the status.
…erification) Surface GitOps config-repo sync on the /admin Access page: configured state, repo URL, last commit/time, interval, pending-extension-reload, and last error, plus an admin-only "Sync now" button calling the syncConfig mutation. configSync is typed in admin-telemetry and exposed through the useAdminTelemetry composable. Typecheck, build, and the 98 frontend tests pass. NOT yet verified in a real browser per the UI Verification Hard Rule — that needs a running ./start.sh + Chrome MCP + an admin session, and is flagged as the remaining step before this UI is considered done.
…fault The frontend defaults OIDC sign-in to provider `google` (DEFAULT_OIDC_LOGIN_PATH = "/auth/oidc/google/login"), but the bundled config fixture declared issuer id `testbed`, so any OIDC path in the testbed failed with "unknown OIDC provider: google". Rename the fixture issuer to `google` (and its redirectURL) so the provider resolves.
Anonymous (zero-auth) callers can now read the workspace and public repositories. The main GraphQL query path uses a read guard that permits anonymous (rejecting only Invalid credentials) and PUBLIC-filters repositories for anonymous principals; mutations and admin surfaces keep the stricter authenticated/admin gates. - graphql_read_guard (permits anonymous, rejects Invalid) on graphql_response. - repositories_value is PUBLIC-filtered when the principal is Anonymous; every downstream read field (repositoryByPath, repositories, repository) derives from it, so one filter covers the whole read surface. - Tests: anonymous workspace read returns 200; anonymous sees public repos but not private ones. (comments/relations read handlers stay auth-gated for now.)
Realize the frontend half of the access model: the shell no longer force-redirects to OIDC sign-in when there is no operator-code. With no operator-code it proceeds anonymously (read-only public access); when a PUBLIC_COMTRYA_OPERATOR_CODE is present it still exchanges it for a testbed session. Sign-in becomes an explicit user action rather than an automatic on-load redirect. - session bootstrap returns undefined (anonymous) when no operator-code, instead of redirecting; removes the now-dead DEFAULT_OIDC_LOGIN_PATH, defaultRedirect, and the redirectTo/oidcLoginPath options. - Tests updated: anonymous-default (no token, no exchange, no redirect). Pairs with the server's zero-auth public read. Explicit OIDC sign-in UI is a follow-up. Verified by unit tests; full in-browser verification still blocked on the pre-existing repo-code smoke (see report).
Add opt/kubernetes with a base + production overlay to deploy comtrya-server: namespace, service account, config + template secret, ReadWriteOnce PVC, a hardened Deployment, and a ClusterIP Service; the production overlay adds a TLS-terminating Ingress and points the config-repo URL via a patch. - Single replica / Recreate / ReadWriteOnce — state is SQLite + bare git repos on /app/data, so the server is not horizontally scalable. - Pure GitOps: config repo URL + sync interval via ConfigMap; operator code and git token via Secret (a placeholder template; swap for a secret manager). - Hardened pod: non-root uid 10001, fsGroup 10001, read-only root filesystem with a writable emptyDir at /tmp (CUE worktree materialisation), all caps dropped, RuntimeDefault seccomp. - Probes: startup + liveness on /healthz, readiness on /readyz. TLS terminates at the Ingress (COMTRYA_TLS_TERMINATED=true). README documents the separate frontend and operator setup steps. Both kustomizations validated with `kustomize build`.
rawkode
force-pushed
the
feat-gitops-config-sync
branch
from
May 27, 2026 21:09
b16766b to
8fc4453
Compare
There was a problem hiding this comment.
Pull request overview
This PR moves Comtrya toward GitOps-driven instance configuration via external CUE repositories, while replacing seeded/demo surfaces with runtime-backed data and adding admin telemetry/Kubernetes deployment scaffolding.
Changes:
- Adds CUE instance-config evaluation, Git config-repo sync, live reconciliation for admins/OIDC/repos/labels/extensions, and related server/core models.
- Replaces mock/demo frontend/admin screens with telemetry-backed or empty runtime views and anonymous read session behavior.
- Updates docs, fixtures, container/Kubernetes assets, and removes legacy demo seed configuration.
Reviewed changes
Copilot reviewed 33 out of 34 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
TODO.md |
Updates product truth and roadmap wording for runtime-created repositories. |
README.md |
Refreshes runbook description. |
PRODUCTION_TESTBED.md |
Documents GitOps config and no boot-time demo seed behavior. |
opt/kubernetes/README.md |
Adds Kubernetes deployment guide. |
opt/kubernetes/overlays/production/kustomization.yaml |
Adds production Kustomize overlay. |
opt/kubernetes/overlays/production/ingress.yaml |
Adds production Ingress template. |
opt/kubernetes/base/serviceaccount.yaml |
Adds dedicated ServiceAccount. |
opt/kubernetes/base/service.yaml |
Adds ClusterIP service. |
opt/kubernetes/base/secret.yaml |
Adds secret template for operator/config repo credentials. |
opt/kubernetes/base/pvc.yaml |
Adds persistent data PVC. |
opt/kubernetes/base/namespace.yaml |
Adds namespace manifest. |
opt/kubernetes/base/kustomization.yaml |
Adds Kustomize base. |
opt/kubernetes/base/deployment.yaml |
Adds server Deployment. |
opt/kubernetes/base/configmap.yaml |
Adds runtime ConfigMap. |
frontend/src/user-layout.ts |
Updates offline layout comment. |
frontend/src/styles.css |
Adjusts repo code layout and mobile styles. |
frontend/src/routes/RepoHome.vue |
Embeds code/sidebar slots in repo code view. |
frontend/src/routes/RepoCode.vue |
Removes old static repo code screen. |
frontend/src/routes/Releases.vue |
Replaces mock releases screen with empty runtime state. |
frontend/src/routes/Pipelines.vue |
Replaces mock pipelines screen with empty runtime state. |
frontend/src/routes/IssueTracker.vue |
Routes issue board through extension surface. |
frontend/src/routes/AdminStorage.vue |
Replaces static storage mock with admin telemetry view. |
frontend/src/routes/AdminOverview.vue |
Replaces static overview mock with admin telemetry view. |
frontend/src/routes/AdminAccess.vue |
Replaces static access mock with telemetry and config sync UI. |
frontend/src/router.ts |
Rewires repo code/pipeline/release routes and removes RepoCode. |
frontend/src/components/ProjectsPanel.vue |
Uses workspace.repositoryByPath only. |
frontend/src/admin-telemetry.ts |
Adds shared admin telemetry composable and formatters. |
frontend/packages/sdk-vue/src/comtrya-config.ts |
Uses repository-by-path config lookup only. |
frontend/packages/sdk-core/src/session.ts |
Allows anonymous zero-auth reads instead of forced OIDC redirect. |
frontend/packages/sdk-core/src/session.test.ts |
Updates session bootstrap tests for anonymous behavior. |
fixtures/demo/conference.json |
Removes seeded demo fixture. |
fixtures/config-repo/cue.mod/module.cue |
Adds bundled GitOps fixture module. |
fixtures/config-repo/config.cue |
Updates bundled GitOps config fixture. |
extensions/first-party/ext_pull_requests/ui/src/PullsDetail.vue |
Removes seeded diff fallback and loads repo CUE config by path. |
extensions/first-party/ext_pull_requests/ui/manifest.json |
Updates UI asset integrity. |
extensions/first-party/ext_pull_requests/manifest.json |
Removes demo seed metadata. |
extensions/first-party/ext_issues/ui/src/policy.ts |
Uses repository-by-path config lookup only. |
extensions/first-party/ext_issues/ui/manifest.json |
Updates UI asset integrity. |
extensions/first-party/ext_docs/ui/src/DocsPanel.vue |
Uses repository-by-path config lookup only. |
extensions/first-party/ext_docs/ui/manifest.json |
Updates UI asset integrity. |
extensions/first-party/ext_checks/manifest.json |
Removes demo seed metadata. |
docs/v3-overhaul.md |
Updates manifest storage description. |
docs/RUNBOOK.md |
Updates operational guidance for GitOps config and empty backup store. |
docs/PANIC_AUDIT.md |
Removes obsolete demo repo panic entry. |
docs/manifest.schema.json |
Removes demoSeed schema support. |
docs/extensions.md |
Updates collection docs. |
docs/CONTAINER.md |
Documents config repo env vars. |
docs/architecture.d2 |
Updates storage diagram metadata files. |
Dockerfile |
Removes fixture copy and updates healthcheck notes. |
DEMO_RUNBOOK.md |
Reframes local runtime smoke flow without seeded demo data. |
crates/server/src/reconcile.rs |
Adds GitOps live reconciliation logic. |
crates/server/src/persistence.rs |
Adds admin credential principal and telemetry counts. |
crates/server/src/config_sync.rs |
Adds gitsync-backed config repo sync subsystem. |
crates/server/Cargo.toml |
Adds gitsync dependency. |
crates/core/src/lib.rs |
Exports instance config evaluator. |
crates/core/src/instance_config_eval.rs |
Adds CUE-to-typed-instance-config evaluator. |
crates/core/src/config.rs |
Adds admins/repositories/labels config models and validation. |
crates/core/src/backup.rs |
Replaces demo backup store with empty store. |
crates/core/src/auth.rs |
Adds live OIDC issuer replacement. |
crates/core/Cargo.toml |
Adds serde dependencies. |
crates/cli/src/main.rs |
Updates backup/restore commands to empty store. |
config/config.cue |
Removes old local config file. |
ARCHITECTURE.md |
Updates startup and runtime storage description. |
AGENTS.md |
Removes fixtures from repository layout summary. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| -v "$(pwd)/data:/app/data" \ | ||
| -v "$(pwd)/config/production-testbed.cue:/app/config.cue:ro" \ | ||
| -e COMTRYA_CONFIG=/app/config.cue \ | ||
| -e COMTRYA_CONFIG_REPO_URL="https://github.com/your-org/comtrya-config.git" \ |
Comment on lines
+94
to
+99
| pub struct AdminConfig { | ||
| pub issuer: Option<String>, | ||
| pub subject: Option<String>, | ||
| pub email: Option<String>, | ||
| pub handle: Option<String>, | ||
| } |
Comment on lines
+18
to
+25
| if let Ok(mut auth) = runtime.auth_service.lock() { | ||
| auth.set_issuers(&config.oidc_issuers); | ||
| } | ||
| reconcile_repositories(runtime, config); | ||
| reconcile_labels(runtime, config); | ||
| reconcile_extensions(runtime, config); | ||
| } | ||
|
|
Comment on lines
+495
to
+499
| fn parse_environment(value: Option<&str>) -> Environment { | ||
| match value { | ||
| Some("production") => Environment::Production, | ||
| _ => Environment::Development, | ||
| } |
| name: comtrya-server-secrets | ||
| type: Opaque | ||
| stringData: | ||
| COMTRYA_OPERATOR_CODE: "REPLACE_WITH_A_STRONG_OPERATOR_CODE" |
| fsGroup: 10001 | ||
| seccompProfile: | ||
| type: RuntimeDefault | ||
| # Graceful shutdown drains in-flight requests for up to 30s. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal
Make onboarding stellar by configuring everything via CUE in an external
git repo, synced on startup and every 15 minutes (with a manual Sync now in
/admin): OIDC config, admin identities (from OIDC claims — never local auth),repositories, labels, and the enabled extension set. The config repo is the
absolute source of truth (strict GitOps).
Confirmed decisions
(
COMTRYA_CONFIG_REPO_URL+ ref + creds). Unset → read-only "not configured"state. No local auth.
repositories).
repos / labels live and hot-loads/unloads WASM components.
each repo's own in-repo CUE adds scoped labels.
Engine: gitsync
Sync uses
rawkode/gitsync(issue#6 → v2.0.0). It is pure-Rust
(gix, no system libgit2), honors
branch, returnsSyncOutcome { changed, previous, current }for cheap no-op ticks, and supports HTTPS token auth.Pinned to git
mainuntil the tag lands.Phases (each a reviewable step)
parsing with real cuengine evaluation into
InstanceConfig; extend it withadmins/repositories/labels. ✅
GitOps (string parser deleted,
fixtures/config-repo+start.sh) ✅;periodic 15-min sync loop +
configSynctelemetry ✅ (unconfigured read-onlyUI folds into Phase 7).
AdminCredentialfrom OIDC claims;adminTelemetrygated;reconcile_liveapplies admins + OIDC issuers on each sync.
undeclared repos deleted (doc + on-disk bare repo).
labelscore collection + GraphQLlabels✅ (per-repo CUE-declared scopedlabels build on this next).
pendingExtensionReloadsignal ✅. OIDC/admins/repos/labels apply live; thein-process WASM component swap (consistent registry+records swap) is a
tracked follow-up, so extension-set changes apply on restart.
Done in this PR
comtrya_core::evaluate_instance_config(dir): cuengine over apackage comtryamodule (read-only), serde wire structs → typedInstanceConfig,validate once.
InstanceConfiggainsadmins/repositories/labelswith validation(admin matcher required; repo path segments are slugs within the visibility
ceiling; labels require
#rrggbb).AdminConfig::matchesresolves admin fromOIDC claims.
main) in the server crate.config_sync::ConfigRepo: whenCOMTRYA_CONFIG_REPO_URLis set, clone viagitsync into
data_dir/config-repo, fast-forward, and evaluate the checkout.tests;
Runtime::startsplit intostart()/start_with_config(); dropped--config/COMTRYA_CONFIG.fixtures/config-repois materialised into alocal git repo by
start.sh(overrideCOMTRYA_CONFIG_REPO_URLfor your own).file://repo via gitsync + evaluate; evaluate the bundledfixture. Docs updated (ARCHITECTURE/PRODUCTION_TESTBED/RUNBOOK/CONTAINER).
Env:
COMTRYA_CONFIG_REPO_URL,COMTRYA_CONFIG_REPO_REF,COMTRYA_CONFIG_REPO_TOKEN(+_USERNAME),COMTRYA_CONFIG_REPO_SSH_KEY(+
_SSH_PASSPHRASE).Access model (zero-auth read / OIDC admin)
guard; repos PUBLIC-filtered for anonymous). OIDC admin / operator-code →
full read + write + admin. Mutations and admin surfaces stay gated.
signs in explicitly. Issuer id aligned to the frontend default (
google).e2e status
./start.shboots the GitOps stack end-to-end (config repo cloned, synced,evaluated; first-party extensions served
200). The bundled headless repo-codebrowser smoke fails on a pre-existing issue: the page receives a GraphQL
OIDC discovery failed for https://issuer.example.testerror (the testbed'sdeliberately-unreachable issuer). Root-caused to the server
oidc_loginhandlerbeing reached at runtime; the trigger isn't traceable statically and needs an
interactive network trace. Not caused by this work (old config failed the same
smoke). Tracked for a focused follow-up.
Verification
cargo test -p comtrya-core(139 pass, incl. 6 new),cargo clippy -p comtrya-core -- -D warningsclean,cargo check --workspaceclean../start.sh+ Chrome MCP verification lands with the admin UI inPhase 7.
This PR was created with the assistance of a LLM.