From 15b189357ed76374d4066ec11dee04cad65f2055 Mon Sep 17 00:00:00 2001 From: Bernd Rodler Date: Wed, 5 Aug 2026 13:44:07 +0200 Subject: [PATCH] docs: architecture overview, sandbox dev manual, production scale-out plan Written from direct SSH inspection of both real clusters (node1-3 prod HA, dev-k8s-1-3 dev) done while building the GitLab CI + ArgoCD pipeline (MR !1) - not re-derived from the aspirational docs/manifests that predated that inspection. ARCHITECTURE.md: system diagram (clients, both clusters, Stalwart, EJBCA CA, the CI+ArgoCD flow) plus the storage-coupling fact that everything else hinges on - 4 RWO PVCs + strategy:Recreate is why the app is single-replica today. SANDBOX-DEV-MANUAL.md: day-to-day branch/MR/CI/ArgoCD flow, one-time bootstrap, troubleshooting, and what's explicitly out of scope for normal dev work (the CA, the still-inert prod overlay). PRODUCTION-SCALE-OUT-PLAN.md: phased path to a 100k+-user production deployment on node1-3 - breaking the storage coupling first (rook-ceph CephFS RWX as the fast path, migrating mutable state into the already-installed-but-unused CNPG Postgres as the correct one), then autoscaling, Stalwart's own scaling track, networking/edge, the observability gap (none found on either cluster), security hardening, load testing, DR, and the go-live sequence. Includes a "scale at any time" manual lever, not just HPA. --- docs/ARCHITECTURE.md | 113 ++++++++++++++++++++ docs/PRODUCTION-SCALE-OUT-PLAN.md | 168 ++++++++++++++++++++++++++++++ docs/SANDBOX-DEV-MANUAL.md | 131 +++++++++++++++++++++++ 3 files changed, 412 insertions(+) create mode 100644 docs/ARCHITECTURE.md create mode 100644 docs/PRODUCTION-SCALE-OUT-PLAN.md create mode 100644 docs/SANDBOX-DEV-MANUAL.md diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 00000000..682b5993 --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,113 @@ +# VNCmail+ — Architecture + +VNCmail+ is VNC's fork of [Bulwark](https://github.com/bulwarkmail/webmail), a +Next.js (App Router) webmail client that speaks JMAP to **Stalwart** (the mail +server — SMTP/IMAP/JMAP, source of truth for all mail/calendar/contacts/files). +VNCmail+ holds no mail data itself; it's a UI + a thin server-side JMAP proxy. + +This doc is the map. For day-to-day sandbox work see +[SANDBOX-DEV-MANUAL.md](SANDBOX-DEV-MANUAL.md); for going live at scale see +[PRODUCTION-SCALE-OUT-PLAN.md](PRODUCTION-SCALE-OUT-PLAN.md). + +## System diagram + +```mermaid +flowchart TB + subgraph Clients + Browser["Web browser"] + Electron["Electron desktop\n(+ local SQLite/FTS5 search index)"] + Mobile["vncmail-native (React Native)\n+ vncmail-relay (push)"] + end + + subgraph "dev-k8s-1/2/3 — dev cluster" + direction TB + TraefikDev["Traefik ingress"] + AppDev["VNCmail+ pod(s)\nnamespace: vncmail"] + ArgoCD["ArgoCD\n(GitOps controller)"] + TraefikDev --> AppDev + end + + subgraph "node1/2/3 — prod HA cluster" + direction TB + TraefikProd["Traefik ingress"] + AppProd["VNCmail+ pod(s)\nnamespace: vncmail-prod\n(not live yet)"] + Ceph["rook-ceph\n(RWX storage, once wired)"] + TraefikProd --> AppProd + AppProd -.-> Ceph + end + + subgraph "Mail backend (per environment)" + Stalwart["Stalwart\nSMTP/IMAP/JMAP server\n(source of truth)"] + end + + subgraph "S/MIME internal CA — namespace vnc-ca, isolated" + EJBCA["EJBCA\n(cert issuance/enrolment)"] + end + + subgraph "GitLab (gitlab.vnc.biz) — canonical repo" + MR["MR into dev\n(verify: typecheck/lint/test/build)"] + Registry["Container registry\nregistry.gitlab.vnc.biz/.../vncmail-plus"] + end + + Browser --> TraefikDev + Electron --> TraefikDev + Mobile --> TraefikDev + Browser -.->|"later, once real"| TraefikProd + + AppDev -->|"JMAP over HTTPS\n(proxy.ts, server-side only)"| Stalwart + AppProd -.->|JMAP| Stalwart + AppDev -.->|"S/MIME enrolment\n(RA client cert, port 8443)"| EJBCA + + MR -->|merge to dev| Registry + Registry -->|"bump-dev job pins the tag"| ArgoCD + ArgoCD -->|"sync (auto)"| AppDev + Registry -.->|"bump-prod pins the tag\n(no rebuild)"| ArgoCD + ArgoCD -.->|"sync — MANUAL, permanent gate"| AppProd +``` + +## Components + +| Component | What it is | Where | +|---|---|---| +| **VNCmail+** (this repo) | Next.js 16 App Router webmail UI + server-side JMAP proxy (`proxy.ts`, `app/api/*`). Stateful: writes settings/admin/telemetry to `/app/data/*` — see storage note below. | Container, `vncmail` (dev) / `vncmail-prod` (prod, not live) namespaces | +| **Stalwart** | External JMAP/SMTP/IMAP mail server. Owns all mail/calendar/contact/file data. VNCmail+ never touches a database directly — every read/write goes over JMAP. | `stalwart.sandbox.vnc.de` (dev; prod instance doesn't exist yet) | +| **EJBCA** (`deploy/k8s/ca/`) | Internal CA issuing S/MIME certs for the S/MIME plugin. Deliberately isolated: own namespace `vnc-ca`, own MariaDB, `NetworkPolicy` allows only the `vncmail` namespace to call its REST API. Root-key ceremony is a manual, human-only runbook — never automated. | `vnc-ca` namespace | +| **Electron desktop client** | Same Next.js app, packaged with `electron-builder`, standalone server spawned as a child process. Adds a local encrypted SQLite/FTS5 search index (`lib/mail-index/`) — event-driven, refreshed off the same JMAP push connection, for AI/RAG-style "search your mail" queries. Unsigned builds today (no Apple/Windows code-signing cert yet). | Desktop, not cluster-hosted | +| **vncmail-native** (separate repo) | React Native/Expo mobile app, forked from upstream `bulwarkmail/native`. Full JMAP delta-sync engine + SQLCipher-encrypted local mail replica (unlike Electron's search-index-only scope). | Mobile (Android verified on emulator; iOS pending) | +| **vncmail-relay** (separate repo) | Push notification relay for the mobile app (forked from `bulwarkmail/relay`). | — | +| **GitLab CI** (`.gitlab-ci.yml`) | Builds+pushes container images, bumps a git-tracked image tag. **Never touches any cluster** — no cluster credentials in CI at all. | Runs on a GitLab Runner | +| **ArgoCD** | GitOps controller, already installed on `dev-k8s` (found idle with zero Applications when this pipeline was built — more idiomatic than having CI run `kubectl` directly). Watches this repo, applies `deploy/k8s/overlays/{dev,prod}`. `vncmail-dev` = automated sync (once bootstrapped); `vncmail-prod` = **permanently manual sync** — that's the Vercel-style "promote to production" gate. | `argocd` namespace on `dev-k8s`; UI at `https://argo.devcluster.vnc.de` | + +## The two clusters + +| | `dev-k8s-1/2/3` | `node1/node2/node3` | +|---|---|---| +| Role | dev / sandbox | production (HA) | +| Storage | `microk8s-hostpath` only (node-local, single-replica-only) | `rook-ceph`: `ceph-rbd` (RWO, default) **and `ceph-cephfs` (RWX, distributed)** | +| Ingress | Traefik | Traefik | +| cert-manager issuer | `letsencrypt-staging` | **none configured yet** | +| ArgoCD | yes, installed | no — not registered as an ArgoCD-managed cluster yet | +| Live workloads today | none (fresh) | none (fresh) | + +Both were confirmed empty when this was written — no `vncmail`, `vnc-ca`, or +`stalwart` anything on either cluster. Any reference elsewhere in this repo's +history to a "live sandbox at vncmail.sandbox.vnc.de" was aspirational +(manifests + docs existed, nothing was ever actually applied). + +## The storage coupling — the one fact that shapes the scale-out plan + +`base/deployment.yaml` mounts 4 PVCs, all `ReadWriteOnce`, `strategy: +Recreate`: + +| Dir | Contents | Write pattern | +|---|---|---| +| `settings` | Per-user encrypted settings (AES-256-GCM, keyed by `hash(username:serverUrl)`) — `lib/settings-sync.ts` | Read+write, per-user | +| `admin` (config) | Operator-authored: `config.json`, `policy.json`, admin password hash, plugins, themes, branding uploads | Write-once-ish — can be mounted `:ro` after initial setup (`ADMIN_CONFIG_READONLY=true`, already a supported mode — `lib/admin/paths.ts`) | +| `admin-state` | Runtime mutations: login timestamps, audit log, setup token | Always read-write, low volume | +| `telemetry` | Version-check / usage state | Read+write, low volume | + +**This is why the app is single-replica today.** RWO + `Recreate` means one +pod, one node, ever. It's not a bug — it's the correct choice for a +single-sandbox deployment — but it's the first thing that has to change to +run more than one replica, which is why it's the opening move in +[PRODUCTION-SCALE-OUT-PLAN.md](PRODUCTION-SCALE-OUT-PLAN.md). diff --git a/docs/PRODUCTION-SCALE-OUT-PLAN.md b/docs/PRODUCTION-SCALE-OUT-PLAN.md new file mode 100644 index 00000000..b0930d71 --- /dev/null +++ b/docs/PRODUCTION-SCALE-OUT-PLAN.md @@ -0,0 +1,168 @@ +# VNCmail+ — Production Scale-Out Plan (target: 100k+ users, scalable on demand) + +Goal: take VNCmail+ from "doesn't exist on `node1-3`" to a production +deployment that can grow past 100k users and be scaled **at any time** — +both automatically (load-driven) and on a single manual command (ahead of an +expected spike), not just reactively. + +Read [ARCHITECTURE.md](ARCHITECTURE.md) first, especially "The storage +coupling" section — it's the reason this is phased the way it is. + +## Where things stand today (verified by direct inspection, not assumed) + +- `node1-3` is a healthy 3-node HA microk8s cluster (rook-ceph, traefik, + metallb, cert-manager) with **zero application workloads and zero + ClusterIssuers**. It's a clean slate, not a half-finished deployment. +- `rook-ceph` is already there, with both `ceph-rbd` (RWO) and **`ceph-cephfs` + (RWX, distributed)** StorageClasses available — the key piece that makes + multi-replica VNCmail+ possible without inventing new infrastructure. +- `cnpg-system` (CloudNativePG, a Postgres operator) is **already installed + on both clusters** and currently unused by anything. This is the natural + home for the app's mutable state once it moves off local files (Phase 1). +- No Prometheus/Grafana/logging stack was found on either cluster — this is + a real gap, not a "probably fine," and it's a prerequisite for safe + autoscaling (HPA needs a metrics pipeline) and for running anything at + 100k-user scale with any visibility into it. +- Stalwart's own scaling story is **not covered here** — it's a separate + system owned by the backend/infra side of this decision. It's called out + explicitly at each phase below because VNCmail+ scaling is moot if + Stalwart can't handle the same load; plan the two together, not + sequentially. + +## Phase 1 — Break the storage coupling (blocking; do this first) + +Today: 4 RWO PVCs, `strategy: Recreate`, one pod max, ever. Two ways to fix, +pick based on how much time you have before you need >1 replica: + +**Tactical (fast, days)**: switch the 4 PVCs to the `ceph-cephfs` StorageClass +(RWX) and the Deployment `strategy` to `RollingUpdate`. This alone unblocks +multiple replicas with no code changes. Real risk: `admin-state`/`telemetry` +are multi-writer files on a shared filesystem — fine at low write volume +(login timestamps, audit log, version-check state), but it's a shortcut, not +the target architecture. `settings` (per-user, keyed by `hash(username:serverUrl)`) +has no cross-writer conflict risk since each user only ever writes their own +file — this one is safe on RWX indefinitely. + +**Structural (correct, weeks)**: migrate `admin-state` and `telemetry` into +CNPG Postgres (already installed, unused) — proper multi-writer semantics, +no filesystem-locking edge cases, and it's the natural place for this kind +of low-volume operational state anyway. Keep `admin` (config) as a +**read-only mount** after setup — `ADMIN_CONFIG_READONLY=true` is already a +supported mode (`lib/admin/paths.ts`), so this can be baked into the image +or a ConfigMap at deploy time instead of a writable volume at all. `settings` +can either stay on CephFS RWX (it's genuinely safe there) or also move to +Postgres if you want zero PVCs in the final state. + +Either way: this is the one item that has to happen before Phase 2 means +anything. Everything downstream assumes replicas > 1 is possible. + +## Phase 2 — Autoscaling & headroom + +- Install a metrics pipeline (`metrics-server` at minimum for HPA; + Prometheus+Grafana for real visibility — see Phase 5, do it once, not twice). +- `HorizontalPodAutoscaler` on CPU/memory to start; revisit with a custom + metric (JMAP request rate, active WebSocket/SSE connections) once you have + real traffic shape. +- `PodDisruptionBudget` so rolling updates and node maintenance don't drop + below your minimum replica count. +- Re-size `resources.requests/limits` from real load-test numbers (Phase 7) + — the sandbox's `100m/256Mi` requests are sandbox-appropriate, not + production-appropriate; don't carry them forward by default. +- **The "scale at any time" requirement**: HPA covers load-driven scaling, + but also document (and rehearse once) a single manual command to add + capacity ahead of a known event, before HPA would react: + `kubectl -n vncmail-prod scale deploy/vncmail-plus --replicas=N` or + bumping the HPA's `minReplicas`. This should be a one-line runbook entry, + not something someone has to figure out under pressure. + +## Phase 3 — Stalwart scaling (parallel track, not this repo's code) + +VNCmail+ has no database and does no caching of its own — every request is +a live JMAP call to Stalwart. At 100k users, Stalwart's own architecture +decision matters as much as anything in this repo: + +- Storage backend: Stalwart supports RocksDB (single-node) or FoundationDB + (distributed, HA) — FoundationDB is the one that scales past a single + node. +- Blob storage: point Stalwart's message-blob storage at an S3-compatible + backend — rook-ceph's object gateway (RGW), if enabled, is already + sitting on the same cluster. +- Confirm Stalwart's own capacity plan (connections, IOPS, memory) against + the same 100k-user target this doc is aiming for, ideally before Phase 7's + load test, not after it fails. + +## Phase 4 — Networking & edge + +- Create a real `ClusterIssuer` on `node1-3` — **none exists today**. Decide + ACME account + DNS-01 or HTTP-01 solver before anything else in this phase. +- Decide the real production hostname (still an open decision — see + `deploy/k8s/overlays/prod/patch-ingress.yaml`'s placeholder). +- Rate limiting at the Traefik ingress (a `Middleware` CRD) before opening + up publicly at this scale — nothing enforces this today. +- Consider a CDN in front of `_next/static` and other cacheable assets to + keep origin load down as user count grows. + +## Phase 5 — Observability + +Stand up Prometheus + Grafana (or point at existing org tooling if one +already covers this cluster — worth checking before installing a second +stack) **before** Phase 2's HPA and **before** Phase 7's load test — you +need to see what's happening in both. At minimum: request rate/latency/error +rate per pod, JMAP call latency to Stalwart, PVC/CephFS I/O if Phase 1 went +the tactical route, and alerting on pod restarts / ImagePullBackOff / cert +expiry. + +## Phase 6 — Security hardening + +- `NetworkPolicy` for `vncmail-prod`, mirroring the `vnc-ca` namespace's + existing default-deny-plus-narrow-allow pattern — nothing enforces + network isolation for `vncmail-prod` today. +- Confirm the microk8s CNI on `node1-3` actually enforces `NetworkPolicy` + (Calico does, flannel-without-a-policy-plugin silently doesn't — the + `vnc-ca` README already flags this exact trap, re-verify for this + namespace too rather than assuming). +- Image scanning in the CI build stage. +- S/MIME CA promotion to prod is its own separate, human-only runbook + (`deploy/k8s/ca/README.md` §9) — sequence it, don't bundle it into this + plan's steps. + +## Phase 7 — Load testing & capacity planning + +Model the actual target before guessing replica counts: concurrent users, +JMAP poll/push connection count, expected sync volume per user, attachment +upload size/frequency. Run a load test against a **prod-shaped** deployment +(real storage backend from Phase 1, real Stalwart capacity from Phase 3, HPA +from Phase 2) before the real cutover — a load test against the sandbox's +single-hostPath-replica setup would tell you nothing useful about 100k users. + +Recommend a staged ramp for the actual rollout (soft-launch a cohort → +watch Phase 5's dashboards → widen) rather than a single cutover to the full +100k target on day one. + +## Phase 8 — Backup & DR + +- rook-ceph snapshot policy for whatever PVCs remain after Phase 1. +- Stalwart's own backup strategy (backend-owned, but confirm it exists and + is tested — a mail server's data loss is a much worse incident than this + app's). +- A written, rehearsed restore runbook — not just "backups exist." + +## Go-live sequence (once Phases 1–6 are actually done, not just planned) + +1. Register `node1-3` as an ArgoCD-managed cluster (`argocd cluster add`, or + an equivalent ServiceAccount+kubeconfig secret) — not done yet, and + deliberately not done before this point. +2. Fill in the real values in `deploy/k8s/overlays/prod/` (hostname, prod + Stalwart's `JMAP_SERVER_URL`) and apply `deploy/argocd/vncmail-prod-app.yaml`. +3. Create the real `vncmail-env` secret + registry pull secret in + `vncmail-prod`, by hand, same as dev's one-time bootstrap. +4. Merge `dev` → `main` (fast-forward only — `git log dev..main` must be + empty first). +5. Click **Sync** on `vncmail-prod` in the ArgoCD UI. This stays a + permanent manual gate — there is no plan to automate this step, ever. +6. Smoke test against the real hostname, watch Phase 5's dashboards, then + proceed with Phase 7's staged ramp. + +Nothing in Phases 1–8 requires the go-live sequence to happen first — build +and verify the scaling story in isolation (e.g. on `dev-k8s` at smaller +scale, or in a throwaway prod-shaped namespace) before the actual cutover. diff --git a/docs/SANDBOX-DEV-MANUAL.md b/docs/SANDBOX-DEV-MANUAL.md new file mode 100644 index 00000000..90c21230 --- /dev/null +++ b/docs/SANDBOX-DEV-MANUAL.md @@ -0,0 +1,131 @@ +# VNCmail+ — Sandbox / Dev Manual + +Practical, day-to-day guide for developing VNCmail+ and getting changes into +the sandbox (`dev-k8s-1/2/3` cluster). For the big picture see +[ARCHITECTURE.md](ARCHITECTURE.md); for how to eventually go live see +[PRODUCTION-SCALE-OUT-PLAN.md](PRODUCTION-SCALE-OUT-PLAN.md). + +## 1. Repo & branches + +- **Canonical remote**: `gitlab.vnc.biz/gitlab-instance-b9b5cf2f/vncmail-plus` + (GitHub `origin` is a passive mirror — never push feature work there). +- `main` = production (protected, fast-forward-only from `dev`, no direct pushes). +- `dev` = integration/default branch (protected, MR-required). +- `vnc/*` or `feature/*` = your working branches → MR into `dev`. + +```bash +git clone git@gitlab.vnc.biz:gitlab-instance-b9b5cf2f/vncmail-plus.git +cd vncmail-plus +git checkout -b vnc/my-change dev +``` + +## 2. Local development + +```bash +npm ci +cp .env.dev.example .env.local # built-in mock JMAP server, DEV_MOCK_JMAP=true +npm run dev # http://localhost:3000, log in with any username/password +``` + +The mock JMAP server (`/api/dev-jmap`) means you don't need a real Stalwart +instance for UI work. Useful scripts: + +```bash +npm run typecheck # tsc --noEmit +npm run lint # eslint . +npm run test:translations # vitest, fast +npm run test:integration # bash integration/run-tests.sh — spins up a REAL + # Stalwart via docker-compose (integration/), slower +``` + +For Electron: + +```bash +npm run electron:dev # build:standalone + build:electron + launch +npm run test:electron # Playwright, no OS permissions needed (Electron CDP) +``` + +## 3. Opening a change + +1. Push your branch, open a Merge Request into `dev` on GitLab. +2. The `verify` CI job runs automatically: typecheck, lint, unit tests, build. + **This is a required check** — it never pushes an image or touches any + cluster, just proves the branch builds. +3. Get it reviewed, merge. + +## 4. What happens after merge — the pipeline + +``` +merge to dev + → CI `build`: docker build, push registry.gitlab.vnc.biz/.../vncmail-plus:sha- + → CI `bump-dev`: commits that tag into + deploy/k8s/overlays/dev/image-tag/kustomization.yaml (a small file CI + owns — don't hand-edit it, your edit will be overwritten on the next push) + → ArgoCD's `vncmail-dev` Application notices the git change and syncs +``` + +CI never runs `kubectl` and holds no cluster credentials — it only talks to +the registry and to this git repo. ArgoCD (already running on `dev-k8s`, +found idle when this pipeline was built) does the actual applying. + +**Until the one-time bootstrap below is done**, `vncmail-dev`'s sync policy +is manual on purpose — check its status: + +```bash +ssh dev-k8s-1 # or dev-k8s-2 / dev-k8s-3 +export PATH=/snap/bin:$PATH +microk8s kubectl -n argocd get application vncmail-dev +``` + +Or the UI: `https://argo.devcluster.vnc.de` (`admin` / see +`kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d` — +rotate after first login). + +## 5. One-time bootstrap (already done or being done — see MR !1 / VNCMAIL-SETUP.md) + +Secrets are **never** managed by CI or ArgoCD — created once, by hand: + +```bash +kubectl create secret docker-registry ghcr-pull -n vncmail ... # or make the registry package public +cp deploy/k8s/overlays/dev/secret.example.yaml secret.yaml # edit SESSION_SECRET +kubectl apply -f secret.yaml +``` + +Then a first manual Sync in the ArgoCD UI. Once that's clean, flip +`deploy/argocd/vncmail-dev-app.yaml`'s `automated:` block on and re-apply — +from then on, every merge to `dev` deploys itself. + +## 6. Checking on the running sandbox + +```bash +ssh dev-k8s-1 +export PATH=/snap/bin:$PATH +microk8s kubectl -n vncmail get pods,pvc,ingress +microk8s kubectl -n vncmail logs deploy/vncmail-plus --tail=100 -f +microk8s kubectl -n vncmail rollout status deploy/vncmail-plus +``` + +No local kubeconfig is assumed — everything above is run over `ssh` directly +on a cluster node (`node1/2/3` for prod, `dev-k8s-1/2/3` for dev), using the +`microk8s.kubectl` binaries installed there (put `/snap/bin` on `PATH`). + +## 7. Troubleshooting + +| Symptom | Likely cause | +|---|---| +| ArgoCD shows `vncmail-dev` as `ComparisonError` / SSH handshake failed | The ArgoCD deploy key hasn't been added to GitLab yet (Project → Settings → Repository → Deploy keys) | +| `bump-dev`/`bump-prod` CI job fails to push | `CI_JOB_TOKEN` self-push isn't enabled (Settings → CI/CD → Job token permissions), and no `GITLAB_PUSH_TOKEN` variable is set as a fallback | +| Pod `ImagePullBackOff` | Registry pull secret missing/expired, or package still private | +| Pod `CrashLoopBackOff`, `EACCES` on `/app/data` | `securityContext.fsGroup: 1001` must stay set — some storage drivers also need it on the PVC itself | +| Ingress has no address / no cert | Wrong `ingressClassName` (must be `traefik` on both real clusters) or a missing `ClusterIssuer` — `node1-3` (prod) has **none** configured today | +| "Ein Fehler ist aufgetreten" on login | Use the full email address (`user@sandbox.vnc.de`), not a bare username — Stalwart auths on the full address | + +## 8. Don't touch (out of scope for day-to-day dev) + +- `deploy/k8s/ca/` (EJBCA internal CA) — separate namespace `vnc-ca`, own + README, root-key ceremony is a manual human-only runbook. Never wire CI or + ArgoCD automation into it. +- `overlays/prod/` and `deploy/argocd/vncmail-prod-app.yaml` — scaffolded, + deliberately inert (placeholder hostname, no prod Stalwart, `node1-3` not + yet registered with ArgoCD). See [PRODUCTION-SCALE-OUT-PLAN.md](PRODUCTION-SCALE-OUT-PLAN.md) + for what has to happen before any of that becomes real.