Skip to content

feat: lessons from Tinkercaster: coordinated k3s release, public site and new gates - #482

Merged
agjs merged 25 commits into
mainfrom
feat/tinkercaster-lessons
Oct 9, 2026
Merged

agjs merged 25 commits into
mainfrom
feat/tinkercaster-lessons

Conversation

@agjs

@agjs agjs commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR feeds back what the first product built on BoringStack (Tinkercaster, live on k3s with billing, a public site and an LLM assistant) learned between 2026-09-14 and 2026-10-08. The template stopped receiving feedback after the first report on 2026-09-14. This change records the second report in docs/maintenance/product-feedback.md (entries 84–101) and fixes what it found.

Production path (k3s)

  • Coordinated release: production-release.yml builds api, migrations, ui and site as candidate-<sha> images once CI passes at that commit, then pins all four digests in one commit. argocd-image-updater is removed, and sync waves order the first sync. The UI can no longer ship before its API.
  • The migration Job is migrate-only and runs from its own image. It no longer re-seeds on every sync.
  • One Postgres TLS policy for the runtime client and drizzle-kit, both fed by DATABASE_SSL_CA.
  • API and migrations run on a read-only root (bun --no-install, /tmp emptyDir).
  • Fixes the VSO _raw secret leak, admits migration pods through the Postgres network policy, and adds API PDB, Prometheus scrape and Valkey AOF fixes.
  • Business metrics: CNPG aggregate queries over the template's own tables, a PodMonitor and a Grafana dashboard. Migration 0002 grants pg_monitor only the counted columns, so no PII.

Public marketing site: apps/site

  • Static Astro site at the apex domain, with the app at app.<domain>.
  • Landing, pricing, about, contact, legal stubs and a Markdown blog.
  • Canonical URLs, sitemap, JSON-LD and build-time social cards.
  • Consent-gated analytics, off by default.
  • nginx with CSP on a read-only root.
  • Config is validated by the build, never at container start (the product's site crash-looped on that).
  • Wired into Compose (WITH_SITE), k3s, the release, CI, security scans and Dependabot.

Gates that replace prose

  • Conventional Commits: a commit-msg hook and a pr-title workflow share scripts/ci/check-commit-title.sh. The product went from 98% to 13% conventional subjects after its agent harness changed.
  • Bundle budget raises need a reason line in apps/ui/budgets.md (check:size-budget). The product raised budgets 30 times without one.
  • packages/* workspaces are validated by root check, pre-push and Dependabot. Pre-push discovers apps/*, and dependency scans skip apps that are missing.
  • Test inventories: feature verification records additions automatically. Removals and empty runs still block, with one acknowledgement command. Before this, the "accept" step was a third of the product's early commits and was then abandoned.
  • CI_RUNNER moves jobs to self-hosted runners. Fork PRs always stay on ubuntu-24.04, enforced by the github-actions-runner-pinned lint-meta rule.

Docs

  • The docs site gains a Public site section.
  • Release, k3s, observability, layout, quickstart, commands and changelog pages are updated.
  • AGENTS.md and the public agents.md record the new rules.

Activation and follow-ups

  • Existing k3s installs must remove the image-updater annotations from the live Application, then set the repo variable PRODUCTION_RELEASE_MODE=coordinated. Until then the release workflow fails before building.
  • Ledger 100 (open): in Compose production the API enables TLS for Postgres, but the Compose Postgres offers no TLS. Migrations now follow the same policy. This needs a reproduction with the prod profile and a decision.
  • Still open: CI does not compare observed test cases with the committed inventories; lint-meta does not scan packages/; no restore drill.

Test plan

  • apps/api: bun run check && bun run test (1289 pass, 1 skip)
  • apps/ui: bun run validate, including check:size-budget. Exercised end to end: a raise without a reason fails, and with a reason it passes.
  • apps/site: bun run validate (25 unit and 34 output tests, build). Image built and run --read-only, headers and routes curled.
  • apps/docs: bun run build:ci plus the offline lychee link check
  • bun test infra/k3s/tests scripts/release (30 pass); kubectl kustomize base and prod; no :latest in the render
  • bun run agent:quality, plus every tools/agent/*.test.ts file
  • api and ui lint-meta, shellcheck, yamllint and actionlint (no new findings); Compose dev/prod renders and guardrails
  • drizzle-kit check. Business-metrics queries were run as a pg_monitor member against a scratch Postgres, and reading emails was denied.
  • Not verified: a live Argo or CNPG cluster, and the TLS handshake against CNPG.
  • Local pre-push: security gates passed. The Playwright smoke reused an unrelated stack already bound to :7330/:7331 and failed against it, so it was skipped for the push. CI's infra-compose-playwright-e2e runs the same specs on a fresh stack.

agjs added 23 commits October 8, 2026 23:33
The first product built on the template went from 98% conventional subjects
to 13% once the agent harness changed and the prose guidance stopped being
read. A commit-msg hook and a pr-title workflow now share one checker;
squash merges take the PR title. AGENTS.md records the harness-neutral rule
and keeps generated content out of hand-written PRs.
Tinkercaster added packages/wiring-engine (36k lines) and apps/site and both
fell outside the gates; template tooling also hard-coded the app list and
failed when an app was removed. Root check and pre-push now validate every
packages/<name> with a package.json, pre-push discovers apps/*, the per-app
dependency scans skip apps that are no longer in the checkout, and
Dependabot covers packages/*. The shared-packages guide documents the
convention.
Tinkercaster raised .size-limit.json in 30 commits (CSS 12 KB to 26.5 KB)
although the documented policy reviews every raise. check:size-budget now
fails when a limit goes up, or an entry is added, without a new reason line
in apps/ui/budgets.md compared with the merge-base. It runs in validate,
pre-push and the apps-ui-validate workflow.
In Tinkercaster 20 of the first 62 commits only accepted test inventories,
and after day three the step was skipped and the protection lapsed. Feature
verification now writes additions-only changes to the committed inventory
after the checkout-stability check, so the diff rides along with the tests.
Removals, renames and substitutions still block and print one
acknowledgement command with the removed identities; zero observed cases
block with test_inventory_empty and cannot be acknowledged. The workflow
and README record that CI does not yet compare observed cases.
drizzle-kit ignored DATABASE_SSL_CA, so migrations could pass against a
trust store the runtime client then refused. Both paths now build ssl from
one helper. drizzle-kit only accepts ssl with host-based credentials, so the
migration config splits DATABASE_URL when TLS is on.
…artup

drizzle-kit is a devDependency and was missing from the runtime image. A
migrations target now carries it. The API CMD passes --no-install so bun
never tries to write packages to a read-only root filesystem (EROFS).
The migrations stage sits before production so a plain docker build still
produces the API.
Per-image Argo Image Updater bumped api and ui independently, so the UI
could ship before the API image it needed existed. production-release.yml
is now the only publisher for Kubernetes. It builds api, migrations and ui
as candidates for one commit, then pins all three digests in one GitOps
commit. The image-updater annotations are removed. The prod overlay has no
mutable tags, and Argo waves order infrastructure, migrations, API, UI.
Compose/WUD keeps apps-api-release.yml and apps-ui-release.yml.

Adds release-script tests and k3s invariant tests, run by infra-k3s-validate.
The migration Job ran db:seed on every sync and reverted admin edits. It is
now a Sync hook at wave -1 that runs db:migrate only, from the migrations
image. It mounts no app secrets, runs on a read-only root with /tmp, and
verifies the CNPG CA the same way the API does. A PreSync hook also ran
before the database existed on a first sync.

The API runs on a read-only root with /tmp and Bun's transpiler cache
there. Its CMD passes --no-install. DATABASE_SSL_CA is injected from the
CNPG CA secret. Release images are amd64, so pods select amd64 nodes.
- postgres policy admits the migration pods, which are not app: api.
- api policy admits Prometheus scrapes of /metrics.
- api PDB uses maxUnavailable: 1 so a node drain can complete on a small cluster.
- VSO secrets use excludeRaw. Without it, every secret also lands in one
  JSON env var in api and migration pods. Rollouts follow Vault changes.
- Valkey runs AOF with everysec fsync so queued jobs survive a restart.
- CNPG CPU is written as "1", the form it stores, so there is no drift.
- Certificate comment records the zone-scoped DNS-01 issuer pitfall.
Production migrations are migrate-only, so the first admin needs an explicit
run. The runtime image has no source tree, so the migrations image now
carries it, and db:seed works there. The seed stays idempotent. Env and
script comments no longer claim the migrate job creates the admin.
Replaces the image-updater runbook with a production releases runbook and
updates the k3s overview and provisioning pages. The k3s README covers
activation, digest rollback, the zone-scoped DNS-01 issuer pitfall, the
migrate-only Job and the one-off superuser bootstrap.
… dashboard

Ported from Tinkercaster and rewritten over the template's own auth, audit
and billing tables: sign-ups per day, active users, accounts by plan and
status, cancellations and Stripe event counts. Migration 0002 grants
pg_monitor column-level SELECT on only the counted columns, so the exporter
cannot read emails, names, IPs or Stripe identifiers. A contract test checks
every query column against the drizzle snapshot and the grants.
The authenticated React app cannot carry SEO, so Tinkercaster added an Astro
site at the apex domain with the app at app.<domain>. Every product needs
that, so the template now ships it: landing, pricing, about, contact, legal
stubs and a Markdown blog; canonical URLs, sitemap, robots, JSON-LD and
build-time social cards; config validated by the build, never at container
start; consent-gated, provider-neutral analytics that is off by default; and
nginx with CSP and cache headers on a read-only root. Root check, CI, security
scans, Dependabot and the scaffold manifest (WITH_SITE) cover it.
…mages

The site is the fourth candidate in the coordinated release, pinned by digest
in the same promotion commit. app.<domain> routes the UI and API. The site
rolls out in wave 1 with the UI, on a read-only root. Contract tests cover
the image set, the wave and the no-build-at-start rule.
…for forks

Tinkercaster moved its CI onto its own cluster runners. Every job now runs on
vars.CI_RUNNER when it is set; pull requests from forks always stay on
ubuntu-24.04 so outside code never reaches the operator's machines, and the
production release stays GitHub-hosted. Unset, nothing changes. The
infra-compose workflow no longer triggers on apps/site, whose image is built
and smoke-tested by apps-site-validate.
… and new gates

Adds a Public site section, updates layout, quickstart, infra, k3s and release
pages for the four-image coordinated release and the apex/app domain split,
documents business metrics, the Conventional Commits gate, packages/*, budget
reasons and the guarded CI_RUNNER, and records the release in the changelog.
@github-advanced-security

Copy link
Copy Markdown

You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool.

What Enabling Code Scanning Means:

  • The 'Security' tab will display more code scanning analysis results (e.g., for the default branch).
  • Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results.
  • You will be able to see the analysis results for the pull request's branch on this overview once the scans have completed and the checks have passed.

For more information about GitHub Code Scanning, check out the documentation.

Comment thread apps/site/tests/output/output.test.ts Fixed
agjs added 2 commits October 9, 2026 00:51
Exact overrides and bumps, each published at least seven days ago, clear
osv-scanner and bun audit in api, ui, docs and site. Ignores with an expiry
cover only braces and http-cache-semantics (no fixed release; build tooling
only) and handlebars until 4.7.10 clears the release-age window on
2026-10-12.
@agjs
agjs merged commit 61c84ae into main Oct 9, 2026
39 checks passed
@agjs
agjs deleted the feat/tinkercaster-lessons branch October 9, 2026 05:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants