SRCmail/docs/PRODUCTION-SCALE-OUT-PLAN.md

VNCmail+ — Production Scale-Out Plan (target: 100k+ users, scalable on demand)

Goal: take VNCmail+ from "doesn't exist on node1-3" to a production deployment that can grow past 100k users and be scaled at any time — both automatically (load-driven) and on a single manual command (ahead of an expected spike), not just reactively.

Read ARCHITECTURE.md first, especially "The storage coupling" section — it's the reason this is phased the way it is.

Where things stand today (verified by direct inspection, not assumed)

• node1-3 is a healthy 3-node HA microk8s cluster (rook-ceph, traefik, metallb, cert-manager) with zero application workloads and zero ClusterIssuers. It's a clean slate, not a half-finished deployment.
• rook-ceph is already there, with both ceph-rbd (RWO) and ceph-cephfs (RWX, distributed) StorageClasses available — the key piece that makes multi-replica VNCmail+ possible without inventing new infrastructure.
• cnpg-system (CloudNativePG, a Postgres operator) is already installed on both clusters and currently unused by anything. This is the natural home for the app's mutable state once it moves off local files (Phase 1).
• No Prometheus/Grafana/logging stack was found on either cluster — this is a real gap, not a "probably fine," and it's a prerequisite for safe autoscaling (HPA needs a metrics pipeline) and for running anything at 100k-user scale with any visibility into it.
• Stalwart's own scaling story is not covered here — it's a separate system owned by the backend/infra side of this decision. It's called out explicitly at each phase below because VNCmail+ scaling is moot if Stalwart can't handle the same load; plan the two together, not sequentially.

Phase 1 — Break the storage coupling (blocking; do this first)

Today: 4 RWO PVCs, strategy: Recreate, one pod max, ever. Two ways to fix, pick based on how much time you have before you need >1 replica:

Tactical (fast, days): switch the 4 PVCs to the ceph-cephfs StorageClass (RWX) and the Deployment strategy to RollingUpdate. This alone unblocks multiple replicas with no code changes. Real risk: admin-state/telemetry are multi-writer files on a shared filesystem — fine at low write volume (login timestamps, audit log, version-check state), but it's a shortcut, not the target architecture. settings (per-user, keyed by hash(username:serverUrl)) has no cross-writer conflict risk since each user only ever writes their own file — this one is safe on RWX indefinitely.

Structural (correct, weeks): migrate admin-state and telemetry into CNPG Postgres (already installed, unused) — proper multi-writer semantics, no filesystem-locking edge cases, and it's the natural place for this kind of low-volume operational state anyway. Keep admin (config) as a read-only mount after setup — ADMIN_CONFIG_READONLY=true is already a supported mode (lib/admin/paths.ts), so this can be baked into the image or a ConfigMap at deploy time instead of a writable volume at all. settings can either stay on CephFS RWX (it's genuinely safe there) or also move to Postgres if you want zero PVCs in the final state.

Either way: this is the one item that has to happen before Phase 2 means anything. Everything downstream assumes replicas > 1 is possible.

Phase 2 — Autoscaling & headroom

• Install a metrics pipeline (metrics-server at minimum for HPA; Prometheus+Grafana for real visibility — see Phase 5, do it once, not twice).
• HorizontalPodAutoscaler on CPU/memory to start; revisit with a custom metric (JMAP request rate, active WebSocket/SSE connections) once you have real traffic shape.
• PodDisruptionBudget so rolling updates and node maintenance don't drop below your minimum replica count.
• Re-size resources.requests/limits from real load-test numbers (Phase 7) — the sandbox's 100m/256Mi requests are sandbox-appropriate, not production-appropriate; don't carry them forward by default.
• The "scale at any time" requirement: HPA covers load-driven scaling, but also document (and rehearse once) a single manual command to add capacity ahead of a known event, before HPA would react: kubectl -n vncmail-prod scale deploy/vncmail-plus --replicas=N or bumping the HPA's minReplicas. This should be a one-line runbook entry, not something someone has to figure out under pressure.

Phase 3 — Stalwart scaling (parallel track, not this repo's code)

VNCmail+ has no database and does no caching of its own — every request is a live JMAP call to Stalwart. At 100k users, Stalwart's own architecture decision matters as much as anything in this repo:

• Storage backend: Stalwart supports RocksDB (single-node) or FoundationDB (distributed, HA) — FoundationDB is the one that scales past a single node.
• Blob storage: point Stalwart's message-blob storage at an S3-compatible backend — rook-ceph's object gateway (RGW), if enabled, is already sitting on the same cluster.
• Confirm Stalwart's own capacity plan (connections, IOPS, memory) against the same 100k-user target this doc is aiming for, ideally before Phase 7's load test, not after it fails.

Phase 4 — Networking & edge

• Create a real ClusterIssuer on node1-3 — none exists today. Decide ACME account + DNS-01 or HTTP-01 solver before anything else in this phase.
• Decide the real production hostname (still an open decision — see deploy/k8s/overlays/prod/patch-ingress.yaml's placeholder).
• Rate limiting at the Traefik ingress (a Middleware CRD) before opening up publicly at this scale — nothing enforces this today.
• Consider a CDN in front of _next/static and other cacheable assets to keep origin load down as user count grows.

Phase 5 — Observability

Stand up Prometheus + Grafana (or point at existing org tooling if one already covers this cluster — worth checking before installing a second stack) before Phase 2's HPA and before Phase 7's load test — you need to see what's happening in both. At minimum: request rate/latency/error rate per pod, JMAP call latency to Stalwart, PVC/CephFS I/O if Phase 1 went the tactical route, and alerting on pod restarts / ImagePullBackOff / cert expiry.

Phase 6 — Security hardening

• NetworkPolicy for vncmail-prod, mirroring the vnc-ca namespace's existing default-deny-plus-narrow-allow pattern — nothing enforces network isolation for vncmail-prod today.
• Confirm the microk8s CNI on node1-3 actually enforces NetworkPolicy (Calico does, flannel-without-a-policy-plugin silently doesn't — the vnc-ca README already flags this exact trap, re-verify for this namespace too rather than assuming).
• Image scanning in the CI build stage.
• S/MIME CA promotion to prod is its own separate, human-only runbook (deploy/k8s/ca/README.md §9) — sequence it, don't bundle it into this plan's steps.

Phase 7 — Load testing & capacity planning

Model the actual target before guessing replica counts: concurrent users, JMAP poll/push connection count, expected sync volume per user, attachment upload size/frequency. Run a load test against a prod-shaped deployment (real storage backend from Phase 1, real Stalwart capacity from Phase 3, HPA from Phase 2) before the real cutover — a load test against the sandbox's single-hostPath-replica setup would tell you nothing useful about 100k users.

Recommend a staged ramp for the actual rollout (soft-launch a cohort → watch Phase 5's dashboards → widen) rather than a single cutover to the full 100k target on day one.

Phase 8 — Backup & DR

• rook-ceph snapshot policy for whatever PVCs remain after Phase 1.
• Stalwart's own backup strategy (backend-owned, but confirm it exists and is tested — a mail server's data loss is a much worse incident than this app's).
• A written, rehearsed restore runbook — not just "backups exist."

Go-live sequence (once Phases 1–6 are actually done, not just planned)

1. Register node1-3 as an ArgoCD-managed cluster (argocd cluster add, or an equivalent ServiceAccount+kubeconfig secret) — not done yet, and deliberately not done before this point.
2. Fill in the real values in deploy/k8s/overlays/prod/ (hostname, prod Stalwart's JMAP_SERVER_URL) and apply deploy/argocd/vncmail-prod-app.yaml.
3. Create the real vncmail-env secret + registry pull secret in vncmail-prod, by hand, same as dev's one-time bootstrap.
4. Merge dev → main (fast-forward only — git log dev..main must be empty first).
5. Click Sync on vncmail-prod in the ArgoCD UI. This stays a permanent manual gate — there is no plan to automate this step, ever.
6. Smoke test against the real hostname, watch Phase 5's dashboards, then proceed with Phase 7's staged ramp.

Nothing in Phases 1–8 requires the go-live sequence to happen first — build and verify the scaling story in isolation (e.g. on dev-k8s at smaller scale, or in a throwaway prod-shaped namespace) before the actual cutover.