Direct SSH access to the actual clusters (node1-3 "prod" HA, dev-k8s-1-3
"dev") revealed two things that made the previous design wrong:
1. Neither cluster has vncmail/vnc-ca namespaces or a bulwark ingress at
all - the "live sandbox" referenced in this repo's docs/manifests was
never actually applied anywhere. Both ingress.yaml's ingressClassName
(public) and cert-manager issuer (letsencrypt-prod) were also wrong:
both clusters run Traefik (class is literally named `traefik`), and
only dev-k8s has any ClusterIssuer at all (`letsencrypt-staging`).
node1-3 has zero ClusterIssuers configured.
2. dev-k8s already has ArgoCD installed, idle, zero Applications - more
idiomatic to use it than have GitLab Runner execute kubectl directly.
Pivots .gitlab-ci.yml: build+push image, then commit the tag into a small
per-overlay Component (overlays/{dev,prod}/image-tag/) that ArgoCD's
Application watches - CI never touches the cluster, only the registry and
this repo. dev's Application (vncmail-dev) is registered and applied
already (manual sync for now, until the one-time namespace secret
bootstrap is done - see VNCMAIL-SETUP.md). prod's Application is
scaffolded in deploy/argocd/ but deliberately not applied - it targets a
different cluster (node1-3) that isn't registered with ArgoCD yet, and
there's still no real prod hostname/Stalwart/ClusterIssuer.
Fixes base/ingress.yaml to the real ingressClassName: traefik (was the
nginx-style `public`, which doesn't exist on either cluster) and gives
each overlay its own cert-manager issuer patch instead of one hardcoded
value, since dev and prod need different (or, for prod, nonexistent)
issuers.
8.8 KiB
VNCmail+ — setup & deploy runbook
VNCmail+ is VNC's fork of Bulwark, a
Next.js (App Router) JMAP webmail client for Stalwart. Stalwart is the source
of truth; VNCmail+ is the UI. It deploys as a container on Kubernetes
(microk8s) at vncmail.sandbox.vnc.de — see deploy/k8s/.
License: AGPL-3.0. Serving a modified VNCmail+ to users over the network obligates VNC to offer those users the corresponding source. Keeping this fork public (with a "Source" link in the imprint/UI) satisfies that. Loop in legal before a public/customer-facing launch if a closed fork is ever desired.
Architecture — why a container, not Vercel
- Bulwark is a stateful, long-lived server: it persists settings-sync, admin
config/state, and telemetry to a local data directory (
/app/data/*). - Vercel serverless was tried and dropped — its filesystem is read-only
except
/tmp, so Bulwark'smkdir ./datacrashes (ENOENT /var/task/data). You cannot point its data dirs at a remote host either (they're POSIX paths, not URLs). Bulwark's native model is a container + persistent volumes. - So VNCmail+ runs as a Docker image with 4 persistent volumes, exactly
like the existing
bulwark.sandbox.vnc.de. - JMAP calls go through server-side
/api/*routes (proxy.ts) → server-to- server to Stalwart, no browser CORS. Config is runtime-read.
Branches (dev-first)
| Branch | Role |
|---|---|
main |
Production. Only updated by git merge --ff-only dev, then an explicit manual promote in CI. No prod environment exists yet — see "CI/CD" below. |
dev |
Integration + QA — default working branch. Every push auto-builds and auto-deploys to the sandbox (vncmail.sandbox.vnc.de). |
vnc/* |
Feature branches for UI work (branch off dev, MR into dev — required, gated by CI). |
All VNC customization lives under vnc/ (see vnc/VNC-CHANGES.md).
CI/CD — GitLab (canonical) + ArgoCD GitOps, Vercel-style dev→prod
Multiple developers work on this repo now. .gitlab-ci.yml on
gitlab.vnc.biz
(the canonical remote — GitHub origin is a passive mirror, not where CI or
deploys happen) builds images and bumps a tag in git; ArgoCD does the
actual deploying — already installed and idle on the dev-k8s-1/2/3
cluster, discovered when standing this up. GitLab CI needs zero cluster
credentials as a result.
Two real clusters, confirmed by direct inspection:
| Cluster | Role | Notes |
|---|---|---|
dev-k8s-1/2/3 |
dev/sandbox | ~hours old when set up here. Traefik, metallb, cert-manager (letsencrypt-staging issuer only), ArgoCD already running. |
node1/node2/node3 |
prod (HA) | Older, rook-ceph+traefik+metallb+cert-manager, but zero apps and zero ClusterIssuers — genuinely a clean slate. |
Neither cluster had a vncmail namespace, vnc-ca namespace, or bulwark
ingress — the "live sandbox at vncmail.sandbox.vnc.de" referenced earlier in
this doc's history was aspirational (manifests + docs existed, nothing was
ever actually applied). The ingress manifests also assumed nginx (class: public, an nginx body-size annotation) — fixed to Traefik's real
ingressClassName: traefik (Traefik has no default body-size cap, so no
replacement annotation is needed).
Flow:
- MR into
dev→verifystage (typecheck/lint/unit test/build). Required check — no push, no deploy. - Merge to
dev→buildpushes one image,registry.gitlab.vnc.biz/.../vncmail-plus:sha-<sha>, thenbump-devcommits that tag intodeploy/k8s/overlays/dev/image-tag/kustomization.yaml([skip ci]). ArgoCD'svncmail-devApplication picks up the git change. - Merge to
main(fast-forward only, see below) →bump-prodpointsoverlays/prod/image-tag/at that same tag — no rebuild. The actual promotion gate is a human clicking Sync on thevncmail-prodArgoCD Application, which is permanently manual-sync (never automated) — that's the Vercel-style "Promote to Production" button, just living in ArgoCD's UI instead of GitLab's.
What's left to wire up (one-time, human steps)
- Add the ArgoCD deploy key to GitLab — Project → Settings → Repository
→ Deploy keys → add (read-only is enough):
Until this is added,
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOURjX/Y9zfB785DyLEF1GUq4HhWujrqeXag8oxdMciq argocd@dev-k8s (vncmail-plus read-only)vncmail-dev's ArgoCD Application (already created,kubectl -n argocd get application vncmail-dev) shows a benignComparisonError(SSH handshake failing) — expected, not a bug. - Let CI push tag-bumps back to this repo — either enable "this project
can be accessed by CI/CD job tokens from other projects" → actually
simpler: Settings → CI/CD → Job token permissions → allow this project's
own job token to push to itself, OR create a Project Access Token
(
write_repositoryscope) and add it as a masked CI/CD variableGITLAB_PUSH_TOKEN(the pipeline tries that first, falls back toCI_JOB_TOKEN). - One-time namespace bootstrap (CI/ArgoCD deliberately never manage
secret contents — see
deploy/k8s/README.md§3):# against dev-k8s (ArgoCD's CreateNamespace=true will make `vncmail` on # first sync, or create it yourself first — either order works) kubectl create secret docker-registry ghcr-pull -n vncmail ... # or make the GHCR package public cp deploy/k8s/overlays/dev/secret.example.yaml secret.yaml # edit SESSION_SECRET kubectl apply -f secret.yaml - First sync — ArgoCD UI at
https://argo.devcluster.vnc.de(usernameadmin, password:kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d— rotate it after logging in once) →vncmail-dev→ Sync. Once that's clean, flipdeploy/argocd/vncmail-dev-app.yaml's commented-outautomated:block on and re-apply, so dev auto-syncs on every push from then on. - Production (later, deliberately not wired yet): decide a real
hostname, stand up prod Stalwart, register
node1-3as an ArgoCD-managed cluster, applydeploy/argocd/vncmail-prod-app.yaml, fill in realoverlays/prodvalues, create a real ClusterIssuer onnode1-3(there isn't one today), then click Sync once — deliberately not before.
Historical note: the old -dev/-beta GHCR image-name split
(.github/workflows/docker-publish.yml) is retired by this — one image name
now, environment lives only in the tag.
Deploy (Kubernetes / microk8s)
Full runbook: deploy/k8s/README.md. In short:
- CI (above) builds and pushes the image, one name/many tags, to GitLab's registry.
kubectl apply -k deploy/k8s/overlays/dev(oroverlays/prod, once real) — base manifests (namespace, 4 PVCs, deployment, service, ingress) live indeploy/k8s/base/, environment differences (namespace, hostname, replica count) are overlay patches.- DNS + a
secret.yaml(from the overlay'ssecret.example.yaml, gitignored, created once by hand — CI never manages secret contents) + an image-pull secret are the remaining manual, human, one-time steps per environment.
Runs alongside the existing bulwark.sandbox.vnc.de. Match your cluster's
StorageClass / IngressClass / cert issuer to bulwark's (see the runbook).
Deploy workflow (dev-first — ALWAYS)
Same flow as every other VNC/SRC repo, now enforced structurally by CI rather than by convention:
- Work on
dev(orvnc/*→ MR intodev, CI-gated). Merge → auto-builds and auto-deploys tovncmail.sandbox.vnc.de. QA there. - Promote to production only on explicit go-live — merge
dev→main:Then clickgit log dev..main # MUST be empty — main must have nothing dev lacks (else prod would revert) git checkout main && git merge --ff-only dev git push gitlab main # never GitHub — opens the manual `promote` job, does not run it git checkout devpromotein the GitLab pipeline UI (protectedproductionenvironment — requires the right role) once prod actually exists (see "CI/CD" above). Never push straight tomain. Never let a dev→main merge silently revert prod.
Syncing upstream (Bulwark releases)
Bring upstream into dev (NOT main), integrate + QA on the dev image, then promote as above:
git fetch upstream
git checkout dev && git merge upstream/main # resolve conflicts via vnc/VNC-CHANGES.md; QA on preview
Auth
Basic auth via Stalwart is the default — users sign in with their
@sandbox.vnc.de address + password; VNCmail+ authenticates them over JMAP. No
extra config. (SSO via vncdirectory/OIDC is a later option — see
vnc/vercel.env.template.)