Files
SRCmail/docs/SANDBOX-DEV-MANUAL.md
Bernd Rodler 15b189357e docs: architecture overview, sandbox dev manual, production scale-out plan
Written from direct SSH inspection of both real clusters (node1-3 prod HA,
dev-k8s-1-3 dev) done while building the GitLab CI + ArgoCD pipeline (MR
!1) - not re-derived from the aspirational docs/manifests that predated
that inspection.

ARCHITECTURE.md: system diagram (clients, both clusters, Stalwart, EJBCA
CA, the CI+ArgoCD flow) plus the storage-coupling fact that everything
else hinges on - 4 RWO PVCs + strategy:Recreate is why the app is
single-replica today.

SANDBOX-DEV-MANUAL.md: day-to-day branch/MR/CI/ArgoCD flow, one-time
bootstrap, troubleshooting, and what's explicitly out of scope for normal
dev work (the CA, the still-inert prod overlay).

PRODUCTION-SCALE-OUT-PLAN.md: phased path to a 100k+-user production
deployment on node1-3 - breaking the storage coupling first (rook-ceph
CephFS RWX as the fast path, migrating mutable state into the
already-installed-but-unused CNPG Postgres as the correct one), then
autoscaling, Stalwart's own scaling track, networking/edge, the
observability gap (none found on either cluster), security hardening,
load testing, DR, and the go-live sequence. Includes a "scale at any
time" manual lever, not just HPA.
2026-08-05 17:58:04 +02:00

5.6 KiB

VNCmail+ — Sandbox / Dev Manual

Practical, day-to-day guide for developing VNCmail+ and getting changes into the sandbox (dev-k8s-1/2/3 cluster). For the big picture see ARCHITECTURE.md; for how to eventually go live see PRODUCTION-SCALE-OUT-PLAN.md.

1. Repo & branches

  • Canonical remote: gitlab.vnc.biz/gitlab-instance-b9b5cf2f/vncmail-plus (GitHub origin is a passive mirror — never push feature work there).
  • main = production (protected, fast-forward-only from dev, no direct pushes).
  • dev = integration/default branch (protected, MR-required).
  • vnc/* or feature/* = your working branches → MR into dev.
git clone git@gitlab.vnc.biz:gitlab-instance-b9b5cf2f/vncmail-plus.git
cd vncmail-plus
git checkout -b vnc/my-change dev

2. Local development

npm ci
cp .env.dev.example .env.local   # built-in mock JMAP server, DEV_MOCK_JMAP=true
npm run dev                       # http://localhost:3000, log in with any username/password

The mock JMAP server (/api/dev-jmap) means you don't need a real Stalwart instance for UI work. Useful scripts:

npm run typecheck     # tsc --noEmit
npm run lint           # eslint .
npm run test:translations   # vitest, fast
npm run test:integration    # bash integration/run-tests.sh — spins up a REAL
                             # Stalwart via docker-compose (integration/), slower

For Electron:

npm run electron:dev    # build:standalone + build:electron + launch
npm run test:electron    # Playwright, no OS permissions needed (Electron CDP)

3. Opening a change

  1. Push your branch, open a Merge Request into dev on GitLab.
  2. The verify CI job runs automatically: typecheck, lint, unit tests, build. This is a required check — it never pushes an image or touches any cluster, just proves the branch builds.
  3. Get it reviewed, merge.

4. What happens after merge — the pipeline

merge to dev
  → CI `build`: docker build, push registry.gitlab.vnc.biz/.../vncmail-plus:sha-<sha>
  → CI `bump-dev`: commits that tag into
    deploy/k8s/overlays/dev/image-tag/kustomization.yaml (a small file CI
    owns — don't hand-edit it, your edit will be overwritten on the next push)
  → ArgoCD's `vncmail-dev` Application notices the git change and syncs

CI never runs kubectl and holds no cluster credentials — it only talks to the registry and to this git repo. ArgoCD (already running on dev-k8s, found idle when this pipeline was built) does the actual applying.

Until the one-time bootstrap below is done, vncmail-dev's sync policy is manual on purpose — check its status:

ssh dev-k8s-1   # or dev-k8s-2 / dev-k8s-3
export PATH=/snap/bin:$PATH
microk8s kubectl -n argocd get application vncmail-dev

Or the UI: https://argo.devcluster.vnc.de (admin / see kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d — rotate after first login).

5. One-time bootstrap (already done or being done — see MR !1 / VNCMAIL-SETUP.md)

Secrets are never managed by CI or ArgoCD — created once, by hand:

kubectl create secret docker-registry ghcr-pull -n vncmail ...   # or make the registry package public
cp deploy/k8s/overlays/dev/secret.example.yaml secret.yaml       # edit SESSION_SECRET
kubectl apply -f secret.yaml

Then a first manual Sync in the ArgoCD UI. Once that's clean, flip deploy/argocd/vncmail-dev-app.yaml's automated: block on and re-apply — from then on, every merge to dev deploys itself.

6. Checking on the running sandbox

ssh dev-k8s-1
export PATH=/snap/bin:$PATH
microk8s kubectl -n vncmail get pods,pvc,ingress
microk8s kubectl -n vncmail logs deploy/vncmail-plus --tail=100 -f
microk8s kubectl -n vncmail rollout status deploy/vncmail-plus

No local kubeconfig is assumed — everything above is run over ssh directly on a cluster node (node1/2/3 for prod, dev-k8s-1/2/3 for dev), using the microk8s.kubectl binaries installed there (put /snap/bin on PATH).

7. Troubleshooting

Symptom Likely cause
ArgoCD shows vncmail-dev as ComparisonError / SSH handshake failed The ArgoCD deploy key hasn't been added to GitLab yet (Project → Settings → Repository → Deploy keys)
bump-dev/bump-prod CI job fails to push CI_JOB_TOKEN self-push isn't enabled (Settings → CI/CD → Job token permissions), and no GITLAB_PUSH_TOKEN variable is set as a fallback
Pod ImagePullBackOff Registry pull secret missing/expired, or package still private
Pod CrashLoopBackOff, EACCES on /app/data securityContext.fsGroup: 1001 must stay set — some storage drivers also need it on the PVC itself
Ingress has no address / no cert Wrong ingressClassName (must be traefik on both real clusters) or a missing ClusterIssuernode1-3 (prod) has none configured today
"Ein Fehler ist aufgetreten" on login Use the full email address (user@sandbox.vnc.de), not a bare username — Stalwart auths on the full address

8. Don't touch (out of scope for day-to-day dev)

  • deploy/k8s/ca/ (EJBCA internal CA) — separate namespace vnc-ca, own README, root-key ceremony is a manual human-only runbook. Never wire CI or ArgoCD automation into it.
  • overlays/prod/ and deploy/argocd/vncmail-prod-app.yaml — scaffolded, deliberately inert (placeholder hostname, no prod Stalwart, node1-3 not yet registered with ArgoCD). See PRODUCTION-SCALE-OUT-PLAN.md for what has to happen before any of that becomes real.