2026-08-05 17:58:04 +02:00
parent ab79288be8
commit 15b189357e
3 changed files with 412 additions and 0 deletions
@@ -0,0 +1,113 @@
# VNCmail+ — Architecture
VNCmail+ is VNC's fork of [Bulwark](https://github.com/bulwarkmail/webmail), a
Next.js (App Router) webmail client that speaks JMAP to **Stalwart** (the mail
server — SMTP/IMAP/JMAP, source of truth for all mail/calendar/contacts/files).
VNCmail+ holds no mail data itself; it's a UI + a thin server-side JMAP proxy.
This doc is the map. For day-to-day sandbox work see
[SANDBOX-DEV-MANUAL.md](SANDBOX-DEV-MANUAL.md); for going live at scale see
[PRODUCTION-SCALE-OUT-PLAN.md](PRODUCTION-SCALE-OUT-PLAN.md).
## System diagram
```mermaid
flowchart TB
    subgraph Clients
        Browser["Web browser"]
        Electron["Electron desktop\n(+ local SQLite/FTS5 search index)"]
        Mobile["vncmail-native (React Native)\n+ vncmail-relay (push)"]
    end
    subgraph "dev-k8s-1/2/3 — dev cluster"
        direction TB
        TraefikDev["Traefik ingress"]
        AppDev["VNCmail+ pod(s)\nnamespace: vncmail"]
        ArgoCD["ArgoCD\n(GitOps controller)"]
        TraefikDev --> AppDev
    end
    subgraph "node1/2/3 — prod HA cluster"
        direction TB
        TraefikProd["Traefik ingress"]
        AppProd["VNCmail+ pod(s)\nnamespace: vncmail-prod\n(not live yet)"]
        Ceph["rook-ceph\n(RWX storage, once wired)"]
        TraefikProd --> AppProd
        AppProd -.-> Ceph
    end
    subgraph "Mail backend (per environment)"
        Stalwart["Stalwart\nSMTP/IMAP/JMAP server\n(source of truth)"]
    end
    subgraph "S/MIME internal CA — namespace vnc-ca, isolated"
        EJBCA["EJBCA\n(cert issuance/enrolment)"]
    end
    subgraph "GitLab (gitlab.vnc.biz) — canonical repo"
        MR["MR into dev\n(verify: typecheck/lint/test/build)"]
        Registry["Container registry\nregistry.gitlab.vnc.biz/.../vncmail-plus"]
    end
    Browser --> TraefikDev
    Electron --> TraefikDev
    Mobile --> TraefikDev
    Browser -.->|"later, once real"| TraefikProd
    AppDev -->|"JMAP over HTTPS\n(proxy.ts, server-side only)"| Stalwart
    AppProd -.->|JMAP| Stalwart
    AppDev -.->|"S/MIME enrolment\n(RA client cert, port 8443)"| EJBCA
    MR -->|merge to dev| Registry
    Registry -->|"bump-dev job pins the tag"| ArgoCD
    ArgoCD -->|"sync (auto)"| AppDev
    Registry -.->|"bump-prod pins the tag\n(no rebuild)"| ArgoCD
    ArgoCD -.->|"sync — MANUAL, permanent gate"| AppProd
```
## Components
| Component | What it is | Where |
|---|---|---|
| **VNCmail+** (this repo) | Next.js 16 App Router webmail UI + server-side JMAP proxy (`proxy.ts`, `app/api/*`). Stateful: writes settings/admin/telemetry to `/app/data/*` — see storage note below. | Container, `vncmail` (dev) / `vncmail-prod` (prod, not live) namespaces |
| **Stalwart** | External JMAP/SMTP/IMAP mail server. Owns all mail/calendar/contact/file data. VNCmail+ never touches a database directly — every read/write goes over JMAP. | `stalwart.sandbox.vnc.de` (dev; prod instance doesn't exist yet) |
| **EJBCA** (`deploy/k8s/ca/`) | Internal CA issuing S/MIME certs for the S/MIME plugin. Deliberately isolated: own namespace `vnc-ca`, own MariaDB, `NetworkPolicy` allows only the `vncmail` namespace to call its REST API. Root-key ceremony is a manual, human-only runbook — never automated. | `vnc-ca` namespace |
| **Electron desktop client** | Same Next.js app, packaged with `electron-builder`, standalone server spawned as a child process. Adds a local encrypted SQLite/FTS5 search index (`lib/mail-index/`) — event-driven, refreshed off the same JMAP push connection, for AI/RAG-style "search your mail" queries. Unsigned builds today (no Apple/Windows code-signing cert yet). | Desktop, not cluster-hosted |
| **vncmail-native** (separate repo) | React Native/Expo mobile app, forked from upstream `bulwarkmail/native`. Full JMAP delta-sync engine + SQLCipher-encrypted local mail replica (unlike Electron's search-index-only scope). | Mobile (Android verified on emulator; iOS pending) |
| **vncmail-relay** (separate repo) | Push notification relay for the mobile app (forked from `bulwarkmail/relay`). | — |
| **GitLab CI** (`.gitlab-ci.yml`) | Builds+pushes container images, bumps a git-tracked image tag. **Never touches any cluster** — no cluster credentials in CI at all. | Runs on a GitLab Runner |
| **ArgoCD** | GitOps controller, already installed on `dev-k8s` (found idle with zero Applications when this pipeline was built — more idiomatic than having CI run `kubectl` directly). Watches this repo, applies `deploy/k8s/overlays/{dev,prod}`. `vncmail-dev` = automated sync (once bootstrapped); `vncmail-prod` = **permanently manual sync** — that's the Vercel-style "promote to production" gate. | `argocd` namespace on `dev-k8s`; UI at `https://argo.devcluster.vnc.de` |
## The two clusters
| | `dev-k8s-1/2/3` | `node1/node2/node3` |
|---|---|---|
| Role | dev / sandbox | production (HA) |
| Storage | `microk8s-hostpath` only (node-local, single-replica-only) | `rook-ceph`: `ceph-rbd` (RWO, default) **and `ceph-cephfs` (RWX, distributed)** |
| Ingress | Traefik | Traefik |
| cert-manager issuer | `letsencrypt-staging` | **none configured yet** |
| ArgoCD | yes, installed | no — not registered as an ArgoCD-managed cluster yet |
| Live workloads today | none (fresh) | none (fresh) |
Both were confirmed empty when this was written — no `vncmail`, `vnc-ca`, or
`stalwart` anything on either cluster. Any reference elsewhere in this repo's
history to a "live sandbox at vncmail.sandbox.vnc.de" was aspirational
(manifests + docs existed, nothing was ever actually applied).
## The storage coupling — the one fact that shapes the scale-out plan
`base/deployment.yaml` mounts 4 PVCs, all `ReadWriteOnce`, `strategy:
Recreate`:
| Dir | Contents | Write pattern |
|---|---|---|
| `settings` | Per-user encrypted settings (AES-256-GCM, keyed by `hash(username:serverUrl)`) — `lib/settings-sync.ts` | Read+write, per-user |
| `admin` (config) | Operator-authored: `config.json`, `policy.json`, admin password hash, plugins, themes, branding uploads | Write-once-ish — can be mounted `:ro` after initial setup (`ADMIN_CONFIG_READONLY=true`, already a supported mode — `lib/admin/paths.ts`) |
| `admin-state` | Runtime mutations: login timestamps, audit log, setup token | Always read-write, low volume |
| `telemetry` | Version-check / usage state | Read+write, low volume |
**This is why the app is single-replica today.** RWO + `Recreate` means one
pod, one node, ever. It's not a bug — it's the correct choice for a
single-sandbox deployment — but it's the first thing that has to change to
run more than one replica, which is why it's the opening move in
[PRODUCTION-SCALE-OUT-PLAN.md](PRODUCTION-SCALE-OUT-PLAN.md).
@@ -0,0 +1,168 @@
# VNCmail+ — Production Scale-Out Plan (target: 100k+ users, scalable on demand)
Goal: take VNCmail+ from "doesn't exist on `node1-3`" to a production
deployment that can grow past 100k users and be scaled **at any time** —
both automatically (load-driven) and on a single manual command (ahead of an
expected spike), not just reactively.
Read [ARCHITECTURE.md](ARCHITECTURE.md) first, especially "The storage
coupling" section — it's the reason this is phased the way it is.
## Where things stand today (verified by direct inspection, not assumed)
- `node1-3` is a healthy 3-node HA microk8s cluster (rook-ceph, traefik,
  metallb, cert-manager) with **zero application workloads and zero
  ClusterIssuers**. It's a clean slate, not a half-finished deployment.
- `rook-ceph` is already there, with both `ceph-rbd` (RWO) and **`ceph-cephfs`
  (RWX, distributed)** StorageClasses available — the key piece that makes
  multi-replica VNCmail+ possible without inventing new infrastructure.
- `cnpg-system` (CloudNativePG, a Postgres operator) is **already installed
  on both clusters** and currently unused by anything. This is the natural
  home for the app's mutable state once it moves off local files (Phase 1).
- No Prometheus/Grafana/logging stack was found on either cluster — this is
  a real gap, not a "probably fine," and it's a prerequisite for safe
  autoscaling (HPA needs a metrics pipeline) and for running anything at
  100k-user scale with any visibility into it.
- Stalwart's own scaling story is **not covered here** — it's a separate
  system owned by the backend/infra side of this decision. It's called out
  explicitly at each phase below because VNCmail+ scaling is moot if
  Stalwart can't handle the same load; plan the two together, not
  sequentially.
## Phase 1 — Break the storage coupling (blocking; do this first)
Today: 4 RWO PVCs, `strategy: Recreate`, one pod max, ever. Two ways to fix,
pick based on how much time you have before you need >1 replica:
**Tactical (fast, days)**: switch the 4 PVCs to the `ceph-cephfs` StorageClass
(RWX) and the Deployment `strategy` to `RollingUpdate`. This alone unblocks
multiple replicas with no code changes. Real risk: `admin-state`/`telemetry`
are multi-writer files on a shared filesystem — fine at low write volume
(login timestamps, audit log, version-check state), but it's a shortcut, not
the target architecture. `settings` (per-user, keyed by `hash(username:serverUrl)`)
has no cross-writer conflict risk since each user only ever writes their own
file — this one is safe on RWX indefinitely.
**Structural (correct, weeks)**: migrate `admin-state` and `telemetry` into
CNPG Postgres (already installed, unused) — proper multi-writer semantics,
no filesystem-locking edge cases, and it's the natural place for this kind
of low-volume operational state anyway. Keep `admin` (config) as a
**read-only mount** after setup — `ADMIN_CONFIG_READONLY=true` is already a
supported mode (`lib/admin/paths.ts`), so this can be baked into the image
or a ConfigMap at deploy time instead of a writable volume at all. `settings`
can either stay on CephFS RWX (it's genuinely safe there) or also move to
Postgres if you want zero PVCs in the final state.
Either way: this is the one item that has to happen before Phase 2 means
anything. Everything downstream assumes replicas > 1 is possible.
## Phase 2 — Autoscaling & headroom
- Install a metrics pipeline (`metrics-server` at minimum for HPA;
  Prometheus+Grafana for real visibility — see Phase 5, do it once, not twice).
- `HorizontalPodAutoscaler` on CPU/memory to start; revisit with a custom
  metric (JMAP request rate, active WebSocket/SSE connections) once you have
  real traffic shape.
- `PodDisruptionBudget` so rolling updates and node maintenance don't drop
  below your minimum replica count.
- Re-size `resources.requests/limits` from real load-test numbers (Phase 7)
  — the sandbox's `100m/256Mi` requests are sandbox-appropriate, not
  production-appropriate; don't carry them forward by default.
- **The "scale at any time" requirement**: HPA covers load-driven scaling,
  but also document (and rehearse once) a single manual command to add
  capacity ahead of a known event, before HPA would react:
  `kubectl -n vncmail-prod scale deploy/vncmail-plus --replicas=N` or
  bumping the HPA's `minReplicas`. This should be a one-line runbook entry,
  not something someone has to figure out under pressure.
## Phase 3 — Stalwart scaling (parallel track, not this repo's code)
VNCmail+ has no database and does no caching of its own — every request is
a live JMAP call to Stalwart. At 100k users, Stalwart's own architecture
decision matters as much as anything in this repo:
- Storage backend: Stalwart supports RocksDB (single-node) or FoundationDB
  (distributed, HA) — FoundationDB is the one that scales past a single
  node.
- Blob storage: point Stalwart's message-blob storage at an S3-compatible
  backend — rook-ceph's object gateway (RGW), if enabled, is already
  sitting on the same cluster.
- Confirm Stalwart's own capacity plan (connections, IOPS, memory) against
  the same 100k-user target this doc is aiming for, ideally before Phase 7's
  load test, not after it fails.
## Phase 4 — Networking & edge
- Create a real `ClusterIssuer` on `node1-3` — **none exists today**. Decide
  ACME account + DNS-01 or HTTP-01 solver before anything else in this phase.
- Decide the real production hostname (still an open decision — see
  `deploy/k8s/overlays/prod/patch-ingress.yaml`'s placeholder).
- Rate limiting at the Traefik ingress (a `Middleware` CRD) before opening
  up publicly at this scale — nothing enforces this today.
- Consider a CDN in front of `_next/static` and other cacheable assets to
  keep origin load down as user count grows.
## Phase 5 — Observability
Stand up Prometheus + Grafana (or point at existing org tooling if one
already covers this cluster — worth checking before installing a second
stack) **before** Phase 2's HPA and **before** Phase 7's load test — you
need to see what's happening in both. At minimum: request rate/latency/error
rate per pod, JMAP call latency to Stalwart, PVC/CephFS I/O if Phase 1 went
the tactical route, and alerting on pod restarts / ImagePullBackOff / cert
expiry.
## Phase 6 — Security hardening
- `NetworkPolicy` for `vncmail-prod`, mirroring the `vnc-ca` namespace's
  existing default-deny-plus-narrow-allow pattern — nothing enforces
  network isolation for `vncmail-prod` today.
- Confirm the microk8s CNI on `node1-3` actually enforces `NetworkPolicy`
  (Calico does, flannel-without-a-policy-plugin silently doesn't — the
  `vnc-ca` README already flags this exact trap, re-verify for this
  namespace too rather than assuming).
- Image scanning in the CI build stage.
- S/MIME CA promotion to prod is its own separate, human-only runbook
  (`deploy/k8s/ca/README.md` §9) — sequence it, don't bundle it into this
  plan's steps.
## Phase 7 — Load testing & capacity planning
Model the actual target before guessing replica counts: concurrent users,
JMAP poll/push connection count, expected sync volume per user, attachment
upload size/frequency. Run a load test against a **prod-shaped** deployment
(real storage backend from Phase 1, real Stalwart capacity from Phase 3, HPA
from Phase 2) before the real cutover — a load test against the sandbox's
single-hostPath-replica setup would tell you nothing useful about 100k users.
Recommend a staged ramp for the actual rollout (soft-launch a cohort →
watch Phase 5's dashboards → widen) rather than a single cutover to the full
100k target on day one.
## Phase 8 — Backup & DR
- rook-ceph snapshot policy for whatever PVCs remain after Phase 1.
- Stalwart's own backup strategy (backend-owned, but confirm it exists and
  is tested — a mail server's data loss is a much worse incident than this
  app's).
- A written, rehearsed restore runbook — not just "backups exist."
## Go-live sequence (once Phases 1–6 are actually done, not just planned)
1. Register `node1-3` as an ArgoCD-managed cluster (`argocd cluster add`, or
   an equivalent ServiceAccount+kubeconfig secret) — not done yet, and
   deliberately not done before this point.
2. Fill in the real values in `deploy/k8s/overlays/prod/` (hostname, prod
   Stalwart's `JMAP_SERVER_URL`) and apply `deploy/argocd/vncmail-prod-app.yaml`.
3. Create the real `vncmail-env` secret + registry pull secret in
   `vncmail-prod`, by hand, same as dev's one-time bootstrap.
4. Merge `dev` → `main` (fast-forward only — `git log dev..main` must be
   empty first).
5. Click **Sync** on `vncmail-prod` in the ArgoCD UI. This stays a
   permanent manual gate — there is no plan to automate this step, ever.
6. Smoke test against the real hostname, watch Phase 5's dashboards, then
   proceed with Phase 7's staged ramp.
Nothing in Phases 1–8 requires the go-live sequence to happen first — build
and verify the scaling story in isolation (e.g. on `dev-k8s` at smaller
scale, or in a throwaway prod-shaped namespace) before the actual cutover.
@@ -0,0 +1,131 @@
# VNCmail+ — Sandbox / Dev Manual
Practical, day-to-day guide for developing VNCmail+ and getting changes into
the sandbox (`dev-k8s-1/2/3` cluster). For the big picture see
[ARCHITECTURE.md](ARCHITECTURE.md); for how to eventually go live see
[PRODUCTION-SCALE-OUT-PLAN.md](PRODUCTION-SCALE-OUT-PLAN.md).
## 1. Repo & branches
- **Canonical remote**: `gitlab.vnc.biz/gitlab-instance-b9b5cf2f/vncmail-plus`
  (GitHub `origin` is a passive mirror — never push feature work there).
- `main` = production (protected, fast-forward-only from `dev`, no direct pushes).
- `dev` = integration/default branch (protected, MR-required).
- `vnc/*` or `feature/*` = your working branches → MR into `dev`.
```bash
git clone git@gitlab.vnc.biz:gitlab-instance-b9b5cf2f/vncmail-plus.git
cd vncmail-plus
git checkout -b vnc/my-change dev
```
## 2. Local development
```bash
npm ci
cp .env.dev.example .env.local   # built-in mock JMAP server, DEV_MOCK_JMAP=true
npm run dev                       # http://localhost:3000, log in with any username/password
```
The mock JMAP server (`/api/dev-jmap`) means you don't need a real Stalwart
instance for UI work. Useful scripts:
```bash
npm run typecheck     # tsc --noEmit
npm run lint           # eslint .
npm run test:translations   # vitest, fast
npm run test:integration    # bash integration/run-tests.sh — spins up a REAL
                             # Stalwart via docker-compose (integration/), slower
```
For Electron:
```bash
npm run electron:dev    # build:standalone + build:electron + launch
npm run test:electron    # Playwright, no OS permissions needed (Electron CDP)
```
## 3. Opening a change
1. Push your branch, open a Merge Request into `dev` on GitLab.
2. The `verify` CI job runs automatically: typecheck, lint, unit tests, build.
   **This is a required check** — it never pushes an image or touches any
   cluster, just proves the branch builds.
3. Get it reviewed, merge.
## 4. What happens after merge — the pipeline
```
merge to dev
  → CI `build`: docker build, push registry.gitlab.vnc.biz/.../vncmail-plus:sha-<sha>
  → CI `bump-dev`: commits that tag into
    deploy/k8s/overlays/dev/image-tag/kustomization.yaml (a small file CI
    owns — don't hand-edit it, your edit will be overwritten on the next push)
  → ArgoCD's `vncmail-dev` Application notices the git change and syncs
```
CI never runs `kubectl` and holds no cluster credentials — it only talks to
the registry and to this git repo. ArgoCD (already running on `dev-k8s`,
found idle when this pipeline was built) does the actual applying.
**Until the one-time bootstrap below is done**, `vncmail-dev`'s sync policy
is manual on purpose — check its status:
```bash
ssh dev-k8s-1   # or dev-k8s-2 / dev-k8s-3
export PATH=/snap/bin:$PATH
microk8s kubectl -n argocd get application vncmail-dev
```
Or the UI: `https://argo.devcluster.vnc.de` (`admin` / see
`kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d` —
rotate after first login).
## 5. One-time bootstrap (already done or being done — see MR !1 / VNCMAIL-SETUP.md)
Secrets are **never** managed by CI or ArgoCD — created once, by hand:
```bash
kubectl create secret docker-registry ghcr-pull -n vncmail ...   # or make the registry package public
cp deploy/k8s/overlays/dev/secret.example.yaml secret.yaml       # edit SESSION_SECRET
kubectl apply -f secret.yaml
```
Then a first manual Sync in the ArgoCD UI. Once that's clean, flip
`deploy/argocd/vncmail-dev-app.yaml`'s `automated:` block on and re-apply —
from then on, every merge to `dev` deploys itself.
## 6. Checking on the running sandbox
```bash
ssh dev-k8s-1
export PATH=/snap/bin:$PATH
microk8s kubectl -n vncmail get pods,pvc,ingress
microk8s kubectl -n vncmail logs deploy/vncmail-plus --tail=100 -f
microk8s kubectl -n vncmail rollout status deploy/vncmail-plus
```
No local kubeconfig is assumed — everything above is run over `ssh` directly
on a cluster node (`node1/2/3` for prod, `dev-k8s-1/2/3` for dev), using the
`microk8s.kubectl` binaries installed there (put `/snap/bin` on `PATH`).
## 7. Troubleshooting
| Symptom | Likely cause |
|---|---|
| ArgoCD shows `vncmail-dev` as `ComparisonError` / SSH handshake failed | The ArgoCD deploy key hasn't been added to GitLab yet (Project → Settings → Repository → Deploy keys) |
| `bump-dev`/`bump-prod` CI job fails to push | `CI_JOB_TOKEN` self-push isn't enabled (Settings → CI/CD → Job token permissions), and no `GITLAB_PUSH_TOKEN` variable is set as a fallback |
| Pod `ImagePullBackOff` | Registry pull secret missing/expired, or package still private |
| Pod `CrashLoopBackOff`, `EACCES` on `/app/data` | `securityContext.fsGroup: 1001` must stay set — some storage drivers also need it on the PVC itself |
| Ingress has no address / no cert | Wrong `ingressClassName` (must be `traefik` on both real clusters) or a missing `ClusterIssuer` — `node1-3` (prod) has **none** configured today |
| "Ein Fehler ist aufgetreten" on login | Use the full email address (`user@sandbox.vnc.de`), not a bare username — Stalwart auths on the full address |
## 8. Don't touch (out of scope for day-to-day dev)
- `deploy/k8s/ca/` (EJBCA internal CA) — separate namespace `vnc-ca`, own
  README, root-key ceremony is a manual human-only runbook. Never wire CI or
  ArgoCD automation into it.
- `overlays/prod/` and `deploy/argocd/vncmail-prod-app.yaml` — scaffolded,
  deliberately inert (placeholder hostname, no prod Stalwart, `node1-3` not
  yet registered with ArgoCD). See [PRODUCTION-SCALE-OUT-PLAN.md](PRODUCTION-SCALE-OUT-PLAN.md)
  for what has to happen before any of that becomes real.