Written from direct SSH inspection of both real clusters (node1-3 prod HA, dev-k8s-1-3 dev) done while building the GitLab CI + ArgoCD pipeline (MR !1) - not re-derived from the aspirational docs/manifests that predated that inspection. ARCHITECTURE.md: system diagram (clients, both clusters, Stalwart, EJBCA CA, the CI+ArgoCD flow) plus the storage-coupling fact that everything else hinges on - 4 RWO PVCs + strategy:Recreate is why the app is single-replica today. SANDBOX-DEV-MANUAL.md: day-to-day branch/MR/CI/ArgoCD flow, one-time bootstrap, troubleshooting, and what's explicitly out of scope for normal dev work (the CA, the still-inert prod overlay). PRODUCTION-SCALE-OUT-PLAN.md: phased path to a 100k+-user production deployment on node1-3 - breaking the storage coupling first (rook-ceph CephFS RWX as the fast path, migrating mutable state into the already-installed-but-unused CNPG Postgres as the correct one), then autoscaling, Stalwart's own scaling track, networking/edge, the observability gap (none found on either cluster), security hardening, load testing, DR, and the go-live sequence. Includes a "scale at any time" manual lever, not just HPA.
5.6 KiB
VNCmail+ — Sandbox / Dev Manual
Practical, day-to-day guide for developing VNCmail+ and getting changes into
the sandbox (dev-k8s-1/2/3 cluster). For the big picture see
ARCHITECTURE.md; for how to eventually go live see
PRODUCTION-SCALE-OUT-PLAN.md.
1. Repo & branches
- Canonical remote:
gitlab.vnc.biz/gitlab-instance-b9b5cf2f/vncmail-plus(GitHuboriginis a passive mirror — never push feature work there). main= production (protected, fast-forward-only fromdev, no direct pushes).dev= integration/default branch (protected, MR-required).vnc/*orfeature/*= your working branches → MR intodev.
git clone git@gitlab.vnc.biz:gitlab-instance-b9b5cf2f/vncmail-plus.git
cd vncmail-plus
git checkout -b vnc/my-change dev
2. Local development
npm ci
cp .env.dev.example .env.local # built-in mock JMAP server, DEV_MOCK_JMAP=true
npm run dev # http://localhost:3000, log in with any username/password
The mock JMAP server (/api/dev-jmap) means you don't need a real Stalwart
instance for UI work. Useful scripts:
npm run typecheck # tsc --noEmit
npm run lint # eslint .
npm run test:translations # vitest, fast
npm run test:integration # bash integration/run-tests.sh — spins up a REAL
# Stalwart via docker-compose (integration/), slower
For Electron:
npm run electron:dev # build:standalone + build:electron + launch
npm run test:electron # Playwright, no OS permissions needed (Electron CDP)
3. Opening a change
- Push your branch, open a Merge Request into
devon GitLab. - The
verifyCI job runs automatically: typecheck, lint, unit tests, build. This is a required check — it never pushes an image or touches any cluster, just proves the branch builds. - Get it reviewed, merge.
4. What happens after merge — the pipeline
merge to dev
→ CI `build`: docker build, push registry.gitlab.vnc.biz/.../vncmail-plus:sha-<sha>
→ CI `bump-dev`: commits that tag into
deploy/k8s/overlays/dev/image-tag/kustomization.yaml (a small file CI
owns — don't hand-edit it, your edit will be overwritten on the next push)
→ ArgoCD's `vncmail-dev` Application notices the git change and syncs
CI never runs kubectl and holds no cluster credentials — it only talks to
the registry and to this git repo. ArgoCD (already running on dev-k8s,
found idle when this pipeline was built) does the actual applying.
Until the one-time bootstrap below is done, vncmail-dev's sync policy
is manual on purpose — check its status:
ssh dev-k8s-1 # or dev-k8s-2 / dev-k8s-3
export PATH=/snap/bin:$PATH
microk8s kubectl -n argocd get application vncmail-dev
Or the UI: https://argo.devcluster.vnc.de (admin / see
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d —
rotate after first login).
5. One-time bootstrap (already done or being done — see MR !1 / VNCMAIL-SETUP.md)
Secrets are never managed by CI or ArgoCD — created once, by hand:
kubectl create secret docker-registry ghcr-pull -n vncmail ... # or make the registry package public
cp deploy/k8s/overlays/dev/secret.example.yaml secret.yaml # edit SESSION_SECRET
kubectl apply -f secret.yaml
Then a first manual Sync in the ArgoCD UI. Once that's clean, flip
deploy/argocd/vncmail-dev-app.yaml's automated: block on and re-apply —
from then on, every merge to dev deploys itself.
6. Checking on the running sandbox
ssh dev-k8s-1
export PATH=/snap/bin:$PATH
microk8s kubectl -n vncmail get pods,pvc,ingress
microk8s kubectl -n vncmail logs deploy/vncmail-plus --tail=100 -f
microk8s kubectl -n vncmail rollout status deploy/vncmail-plus
No local kubeconfig is assumed — everything above is run over ssh directly
on a cluster node (node1/2/3 for prod, dev-k8s-1/2/3 for dev), using the
microk8s.kubectl binaries installed there (put /snap/bin on PATH).
7. Troubleshooting
| Symptom | Likely cause |
|---|---|
ArgoCD shows vncmail-dev as ComparisonError / SSH handshake failed |
The ArgoCD deploy key hasn't been added to GitLab yet (Project → Settings → Repository → Deploy keys) |
bump-dev/bump-prod CI job fails to push |
CI_JOB_TOKEN self-push isn't enabled (Settings → CI/CD → Job token permissions), and no GITLAB_PUSH_TOKEN variable is set as a fallback |
Pod ImagePullBackOff |
Registry pull secret missing/expired, or package still private |
Pod CrashLoopBackOff, EACCES on /app/data |
securityContext.fsGroup: 1001 must stay set — some storage drivers also need it on the PVC itself |
| Ingress has no address / no cert | Wrong ingressClassName (must be traefik on both real clusters) or a missing ClusterIssuer — node1-3 (prod) has none configured today |
| "Ein Fehler ist aufgetreten" on login | Use the full email address (user@sandbox.vnc.de), not a bare username — Stalwart auths on the full address |
8. Don't touch (out of scope for day-to-day dev)
deploy/k8s/ca/(EJBCA internal CA) — separate namespacevnc-ca, own README, root-key ceremony is a manual human-only runbook. Never wire CI or ArgoCD automation into it.overlays/prod/anddeploy/argocd/vncmail-prod-app.yaml— scaffolded, deliberately inert (placeholder hostname, no prod Stalwart,node1-3not yet registered with ArgoCD). See PRODUCTION-SCALE-OUT-PLAN.md for what has to happen before any of that becomes real.