Bernd Rodler
15b189357e
Written from direct SSH inspection of both real clusters (node1-3 prod HA, dev-k8s-1-3 dev) done while building the GitLab CI + ArgoCD pipeline (MR !1) - not re-derived from the aspirational docs/manifests that predated that inspection. ARCHITECTURE.md: system diagram (clients, both clusters, Stalwart, EJBCA CA, the CI+ArgoCD flow) plus the storage-coupling fact that everything else hinges on - 4 RWO PVCs + strategy:Recreate is why the app is single-replica today. SANDBOX-DEV-MANUAL.md: day-to-day branch/MR/CI/ArgoCD flow, one-time bootstrap, troubleshooting, and what's explicitly out of scope for normal dev work (the CA, the still-inert prod overlay). PRODUCTION-SCALE-OUT-PLAN.md: phased path to a 100k+-user production deployment on node1-3 - breaking the storage coupling first (rook-ceph CephFS RWX as the fast path, migrating mutable state into the already-installed-but-unused CNPG Postgres as the correct one), then autoscaling, Stalwart's own scaling track, networking/edge, the observability gap (none found on either cluster), security hardening, load testing, DR, and the go-live sequence. Includes a "scale at any time" manual lever, not just HPA.
114 lines
6.6 KiB
Markdown
114 lines
6.6 KiB
Markdown
# VNCmail+ — Architecture
|
|
|
|
VNCmail+ is VNC's fork of [Bulwark](https://github.com/bulwarkmail/webmail), a
|
|
Next.js (App Router) webmail client that speaks JMAP to **Stalwart** (the mail
|
|
server — SMTP/IMAP/JMAP, source of truth for all mail/calendar/contacts/files).
|
|
VNCmail+ holds no mail data itself; it's a UI + a thin server-side JMAP proxy.
|
|
|
|
This doc is the map. For day-to-day sandbox work see
|
|
[SANDBOX-DEV-MANUAL.md](SANDBOX-DEV-MANUAL.md); for going live at scale see
|
|
[PRODUCTION-SCALE-OUT-PLAN.md](PRODUCTION-SCALE-OUT-PLAN.md).
|
|
|
|
## System diagram
|
|
|
|
```mermaid
|
|
flowchart TB
|
|
subgraph Clients
|
|
Browser["Web browser"]
|
|
Electron["Electron desktop\n(+ local SQLite/FTS5 search index)"]
|
|
Mobile["vncmail-native (React Native)\n+ vncmail-relay (push)"]
|
|
end
|
|
|
|
subgraph "dev-k8s-1/2/3 — dev cluster"
|
|
direction TB
|
|
TraefikDev["Traefik ingress"]
|
|
AppDev["VNCmail+ pod(s)\nnamespace: vncmail"]
|
|
ArgoCD["ArgoCD\n(GitOps controller)"]
|
|
TraefikDev --> AppDev
|
|
end
|
|
|
|
subgraph "node1/2/3 — prod HA cluster"
|
|
direction TB
|
|
TraefikProd["Traefik ingress"]
|
|
AppProd["VNCmail+ pod(s)\nnamespace: vncmail-prod\n(not live yet)"]
|
|
Ceph["rook-ceph\n(RWX storage, once wired)"]
|
|
TraefikProd --> AppProd
|
|
AppProd -.-> Ceph
|
|
end
|
|
|
|
subgraph "Mail backend (per environment)"
|
|
Stalwart["Stalwart\nSMTP/IMAP/JMAP server\n(source of truth)"]
|
|
end
|
|
|
|
subgraph "S/MIME internal CA — namespace vnc-ca, isolated"
|
|
EJBCA["EJBCA\n(cert issuance/enrolment)"]
|
|
end
|
|
|
|
subgraph "GitLab (gitlab.vnc.biz) — canonical repo"
|
|
MR["MR into dev\n(verify: typecheck/lint/test/build)"]
|
|
Registry["Container registry\nregistry.gitlab.vnc.biz/.../vncmail-plus"]
|
|
end
|
|
|
|
Browser --> TraefikDev
|
|
Electron --> TraefikDev
|
|
Mobile --> TraefikDev
|
|
Browser -.->|"later, once real"| TraefikProd
|
|
|
|
AppDev -->|"JMAP over HTTPS\n(proxy.ts, server-side only)"| Stalwart
|
|
AppProd -.->|JMAP| Stalwart
|
|
AppDev -.->|"S/MIME enrolment\n(RA client cert, port 8443)"| EJBCA
|
|
|
|
MR -->|merge to dev| Registry
|
|
Registry -->|"bump-dev job pins the tag"| ArgoCD
|
|
ArgoCD -->|"sync (auto)"| AppDev
|
|
Registry -.->|"bump-prod pins the tag\n(no rebuild)"| ArgoCD
|
|
ArgoCD -.->|"sync — MANUAL, permanent gate"| AppProd
|
|
```
|
|
|
|
## Components
|
|
|
|
| Component | What it is | Where |
|
|
|---|---|---|
|
|
| **VNCmail+** (this repo) | Next.js 16 App Router webmail UI + server-side JMAP proxy (`proxy.ts`, `app/api/*`). Stateful: writes settings/admin/telemetry to `/app/data/*` — see storage note below. | Container, `vncmail` (dev) / `vncmail-prod` (prod, not live) namespaces |
|
|
| **Stalwart** | External JMAP/SMTP/IMAP mail server. Owns all mail/calendar/contact/file data. VNCmail+ never touches a database directly — every read/write goes over JMAP. | `stalwart.sandbox.vnc.de` (dev; prod instance doesn't exist yet) |
|
|
| **EJBCA** (`deploy/k8s/ca/`) | Internal CA issuing S/MIME certs for the S/MIME plugin. Deliberately isolated: own namespace `vnc-ca`, own MariaDB, `NetworkPolicy` allows only the `vncmail` namespace to call its REST API. Root-key ceremony is a manual, human-only runbook — never automated. | `vnc-ca` namespace |
|
|
| **Electron desktop client** | Same Next.js app, packaged with `electron-builder`, standalone server spawned as a child process. Adds a local encrypted SQLite/FTS5 search index (`lib/mail-index/`) — event-driven, refreshed off the same JMAP push connection, for AI/RAG-style "search your mail" queries. Unsigned builds today (no Apple/Windows code-signing cert yet). | Desktop, not cluster-hosted |
|
|
| **vncmail-native** (separate repo) | React Native/Expo mobile app, forked from upstream `bulwarkmail/native`. Full JMAP delta-sync engine + SQLCipher-encrypted local mail replica (unlike Electron's search-index-only scope). | Mobile (Android verified on emulator; iOS pending) |
|
|
| **vncmail-relay** (separate repo) | Push notification relay for the mobile app (forked from `bulwarkmail/relay`). | — |
|
|
| **GitLab CI** (`.gitlab-ci.yml`) | Builds+pushes container images, bumps a git-tracked image tag. **Never touches any cluster** — no cluster credentials in CI at all. | Runs on a GitLab Runner |
|
|
| **ArgoCD** | GitOps controller, already installed on `dev-k8s` (found idle with zero Applications when this pipeline was built — more idiomatic than having CI run `kubectl` directly). Watches this repo, applies `deploy/k8s/overlays/{dev,prod}`. `vncmail-dev` = automated sync (once bootstrapped); `vncmail-prod` = **permanently manual sync** — that's the Vercel-style "promote to production" gate. | `argocd` namespace on `dev-k8s`; UI at `https://argo.devcluster.vnc.de` |
|
|
|
|
## The two clusters
|
|
|
|
| | `dev-k8s-1/2/3` | `node1/node2/node3` |
|
|
|---|---|---|
|
|
| Role | dev / sandbox | production (HA) |
|
|
| Storage | `microk8s-hostpath` only (node-local, single-replica-only) | `rook-ceph`: `ceph-rbd` (RWO, default) **and `ceph-cephfs` (RWX, distributed)** |
|
|
| Ingress | Traefik | Traefik |
|
|
| cert-manager issuer | `letsencrypt-staging` | **none configured yet** |
|
|
| ArgoCD | yes, installed | no — not registered as an ArgoCD-managed cluster yet |
|
|
| Live workloads today | none (fresh) | none (fresh) |
|
|
|
|
Both were confirmed empty when this was written — no `vncmail`, `vnc-ca`, or
|
|
`stalwart` anything on either cluster. Any reference elsewhere in this repo's
|
|
history to a "live sandbox at vncmail.sandbox.vnc.de" was aspirational
|
|
(manifests + docs existed, nothing was ever actually applied).
|
|
|
|
## The storage coupling — the one fact that shapes the scale-out plan
|
|
|
|
`base/deployment.yaml` mounts 4 PVCs, all `ReadWriteOnce`, `strategy:
|
|
Recreate`:
|
|
|
|
| Dir | Contents | Write pattern |
|
|
|---|---|---|
|
|
| `settings` | Per-user encrypted settings (AES-256-GCM, keyed by `hash(username:serverUrl)`) — `lib/settings-sync.ts` | Read+write, per-user |
|
|
| `admin` (config) | Operator-authored: `config.json`, `policy.json`, admin password hash, plugins, themes, branding uploads | Write-once-ish — can be mounted `:ro` after initial setup (`ADMIN_CONFIG_READONLY=true`, already a supported mode — `lib/admin/paths.ts`) |
|
|
| `admin-state` | Runtime mutations: login timestamps, audit log, setup token | Always read-write, low volume |
|
|
| `telemetry` | Version-check / usage state | Read+write, low volume |
|
|
|
|
**This is why the app is single-replica today.** RWO + `Recreate` means one
|
|
pod, one node, ever. It's not a bug — it's the correct choice for a
|
|
single-sandbox deployment — but it's the first thing that has to change to
|
|
run more than one replica, which is why it's the opening move in
|
|
[PRODUCTION-SCALE-OUT-PLAN.md](PRODUCTION-SCALE-OUT-PLAN.md).
|