Bernd Rodler
177b2aca57
Direct SSH access to the actual clusters (node1-3 "prod" HA, dev-k8s-1-3
"dev") revealed two things that made the previous design wrong:
1. Neither cluster has vncmail/vnc-ca namespaces or a bulwark ingress at
all - the "live sandbox" referenced in this repo's docs/manifests was
never actually applied anywhere. Both ingress.yaml's ingressClassName
(public) and cert-manager issuer (letsencrypt-prod) were also wrong:
both clusters run Traefik (class is literally named `traefik`), and
only dev-k8s has any ClusterIssuer at all (`letsencrypt-staging`).
node1-3 has zero ClusterIssuers configured.
2. dev-k8s already has ArgoCD installed, idle, zero Applications - more
idiomatic to use it than have GitLab Runner execute kubectl directly.
Pivots .gitlab-ci.yml: build+push image, then commit the tag into a small
per-overlay Component (overlays/{dev,prod}/image-tag/) that ArgoCD's
Application watches - CI never touches the cluster, only the registry and
this repo. dev's Application (vncmail-dev) is registered and applied
already (manual sync for now, until the one-time namespace secret
bootstrap is done - see VNCMAIL-SETUP.md). prod's Application is
scaffolded in deploy/argocd/ but deliberately not applied - it targets a
different cluster (node1-3) that isn't registered with ArgoCD yet, and
there's still no real prod hostname/Stalwart/ClusterIssuer.
Fixes base/ingress.yaml to the real ingressClassName: traefik (was the
nginx-style `public`, which doesn't exist on either cluster) and gives
each overlay its own cert-manager issuer patch instead of one hardcoded
value, since dev and prod need different (or, for prod, nonexistent)
issuers.
170 lines
8.8 KiB
Markdown
170 lines
8.8 KiB
Markdown
# VNCmail+ — setup & deploy runbook
|
|
|
|
VNCmail+ is VNC's fork of [Bulwark](https://github.com/bulwarkmail/webmail), a
|
|
Next.js (App Router) JMAP webmail client for **Stalwart**. Stalwart is the source
|
|
of truth; VNCmail+ is the UI. It deploys as a **container on Kubernetes
|
|
(microk8s)** at `vncmail.sandbox.vnc.de` — see **[deploy/k8s/](deploy/k8s/README.md)**.
|
|
|
|
> **License:** AGPL-3.0. Serving a modified VNCmail+ to users over the network
|
|
> obligates VNC to offer those users the corresponding source. Keeping this fork
|
|
> public (with a "Source" link in the imprint/UI) satisfies that. Loop in legal
|
|
> before a public/customer-facing launch if a closed fork is ever desired.
|
|
|
|
## Architecture — why a container, not Vercel
|
|
|
|
- Bulwark is a **stateful, long-lived server**: it persists settings-sync, admin
|
|
config/state, and telemetry to a **local data directory** (`/app/data/*`).
|
|
- **Vercel serverless was tried and dropped** — its filesystem is read-only
|
|
except `/tmp`, so Bulwark's `mkdir ./data` crashes (`ENOENT /var/task/data`).
|
|
You cannot point its data dirs at a remote host either (they're POSIX paths,
|
|
not URLs). Bulwark's native model is a container + persistent volumes.
|
|
- So VNCmail+ runs as a Docker image with **4 persistent volumes**, exactly
|
|
like the existing `bulwark.sandbox.vnc.de`.
|
|
- JMAP calls go through **server-side `/api/*` routes** (`proxy.ts`) → server-to-
|
|
server to Stalwart, **no browser CORS**. Config is **runtime-read**.
|
|
|
|
## Branches (dev-first)
|
|
|
|
| Branch | Role |
|
|
|--------|------|
|
|
| `main` | **Production.** Only updated by `git merge --ff-only dev`, then an explicit manual promote in CI. No prod environment exists yet — see "CI/CD" below. |
|
|
| `dev` | Integration + QA — default working branch. Every push auto-builds and auto-deploys to the sandbox (`vncmail.sandbox.vnc.de`). |
|
|
| `vnc/*`| Feature branches for UI work (branch off `dev`, MR into `dev` — required, gated by CI). |
|
|
|
|
All VNC customization lives under `vnc/` (see `vnc/VNC-CHANGES.md`).
|
|
|
|
## CI/CD — GitLab (canonical) + ArgoCD GitOps, Vercel-style dev→prod
|
|
|
|
Multiple developers work on this repo now. `.gitlab-ci.yml` on
|
|
[gitlab.vnc.biz](https://gitlab.vnc.biz/gitlab-instance-b9b5cf2f/vncmail-plus)
|
|
(the canonical remote — GitHub `origin` is a passive mirror, not where CI or
|
|
deploys happen) builds images and bumps a tag in git; **ArgoCD does the
|
|
actual deploying** — already installed and idle on the `dev-k8s-1/2/3`
|
|
cluster, discovered when standing this up. GitLab CI needs zero cluster
|
|
credentials as a result.
|
|
|
|
Two real clusters, confirmed by direct inspection:
|
|
|
|
| Cluster | Role | Notes |
|
|
|---|---|---|
|
|
| `dev-k8s-1/2/3` | dev/sandbox | ~hours old when set up here. Traefik, metallb, cert-manager (`letsencrypt-staging` issuer only), **ArgoCD already running**. |
|
|
| `node1/node2/node3` | prod (HA) | Older, rook-ceph+traefik+metallb+cert-manager, but **zero apps and zero ClusterIssuers** — genuinely a clean slate. |
|
|
|
|
Neither cluster had a `vncmail` namespace, `vnc-ca` namespace, or `bulwark`
|
|
ingress — the "live sandbox at vncmail.sandbox.vnc.de" referenced earlier in
|
|
this doc's history was aspirational (manifests + docs existed, nothing was
|
|
ever actually applied). The ingress manifests also assumed nginx (`class:
|
|
public`, an nginx body-size annotation) — fixed to Traefik's real
|
|
`ingressClassName: traefik` (Traefik has no default body-size cap, so no
|
|
replacement annotation is needed).
|
|
|
|
Flow:
|
|
|
|
1. **MR into `dev`** → `verify` stage (typecheck/lint/unit test/build).
|
|
Required check — no push, no deploy.
|
|
2. **Merge to `dev`** → `build` pushes one image,
|
|
`registry.gitlab.vnc.biz/.../vncmail-plus:sha-<sha>`, then `bump-dev`
|
|
commits that tag into `deploy/k8s/overlays/dev/image-tag/kustomization.yaml`
|
|
(`[skip ci]`). ArgoCD's `vncmail-dev` Application picks up the git change.
|
|
3. **Merge to `main`** (fast-forward only, see below) → `bump-prod` points
|
|
`overlays/prod/image-tag/` at that same tag — **no rebuild**. The actual
|
|
promotion gate is a **human clicking Sync** on the `vncmail-prod` ArgoCD
|
|
Application, which is permanently manual-sync (never automated) — that's
|
|
the Vercel-style "Promote to Production" button, just living in ArgoCD's
|
|
UI instead of GitLab's.
|
|
|
|
### What's left to wire up (one-time, human steps)
|
|
|
|
1. **Add the ArgoCD deploy key to GitLab** — Project → Settings → Repository
|
|
→ Deploy keys → add (read-only is enough):
|
|
```
|
|
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOURjX/Y9zfB785DyLEF1GUq4HhWujrqeXag8oxdMciq argocd@dev-k8s (vncmail-plus read-only)
|
|
```
|
|
Until this is added, `vncmail-dev`'s ArgoCD Application (already created,
|
|
`kubectl -n argocd get application vncmail-dev`) shows a benign
|
|
`ComparisonError` (SSH handshake failing) — expected, not a bug.
|
|
2. **Let CI push tag-bumps back to this repo** — either enable "this project
|
|
can be accessed by CI/CD job tokens from other projects" → actually
|
|
simpler: Settings → CI/CD → Job token permissions → allow this project's
|
|
own job token to push to itself, OR create a Project Access Token
|
|
(`write_repository` scope) and add it as a masked CI/CD variable
|
|
`GITLAB_PUSH_TOKEN` (the pipeline tries that first, falls back to
|
|
`CI_JOB_TOKEN`).
|
|
3. **One-time namespace bootstrap** (CI/ArgoCD deliberately never manage
|
|
secret contents — see `deploy/k8s/README.md` §3):
|
|
```bash
|
|
# against dev-k8s (ArgoCD's CreateNamespace=true will make `vncmail` on
|
|
# first sync, or create it yourself first — either order works)
|
|
kubectl create secret docker-registry ghcr-pull -n vncmail ... # or make the GHCR package public
|
|
cp deploy/k8s/overlays/dev/secret.example.yaml secret.yaml # edit SESSION_SECRET
|
|
kubectl apply -f secret.yaml
|
|
```
|
|
4. **First sync** — ArgoCD UI at `https://argo.devcluster.vnc.de`
|
|
(username `admin`, password: `kubectl -n argocd get secret
|
|
argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d`
|
|
— rotate it after logging in once) → `vncmail-dev` → Sync. Once that's
|
|
clean, flip `deploy/argocd/vncmail-dev-app.yaml`'s commented-out
|
|
`automated:` block on and re-apply, so dev auto-syncs on every push from
|
|
then on.
|
|
5. **Production** (later, deliberately not wired yet): decide a real
|
|
hostname, stand up prod Stalwart, register `node1-3` as an ArgoCD-managed
|
|
cluster, apply `deploy/argocd/vncmail-prod-app.yaml`, fill in real
|
|
`overlays/prod` values, create a real ClusterIssuer on `node1-3` (there
|
|
isn't one today), then click Sync once — deliberately not before.
|
|
|
|
Historical note: the old `-dev`/`-beta` GHCR image-name split
|
|
(`.github/workflows/docker-publish.yml`) is retired by this — one image name
|
|
now, environment lives only in the tag.
|
|
|
|
## Deploy (Kubernetes / microk8s)
|
|
|
|
Full runbook: **[deploy/k8s/README.md](deploy/k8s/README.md)**. In short:
|
|
|
|
1. CI (above) builds and pushes the image, one name/many tags, to GitLab's
|
|
registry.
|
|
2. `kubectl apply -k deploy/k8s/overlays/dev` (or `overlays/prod`, once real)
|
|
— base manifests (namespace, 4 PVCs, deployment, service, ingress) live in
|
|
`deploy/k8s/base/`, environment differences (namespace, hostname, replica
|
|
count) are overlay patches.
|
|
3. DNS + a `secret.yaml` (from the overlay's `secret.example.yaml`, gitignored,
|
|
created once by hand — CI never manages secret contents) + an image-pull
|
|
secret are the remaining manual, human, one-time steps per environment.
|
|
|
|
Runs alongside the existing `bulwark.sandbox.vnc.de`. Match your cluster's
|
|
StorageClass / IngressClass / cert issuer to bulwark's (see the runbook).
|
|
|
|
## Deploy workflow (dev-first — ALWAYS)
|
|
|
|
Same flow as every other VNC/SRC repo, now enforced structurally by CI rather
|
|
than by convention:
|
|
|
|
1. Work on `dev` (or `vnc/*` → MR into `dev`, CI-gated). Merge → auto-builds
|
|
and auto-deploys to `vncmail.sandbox.vnc.de`. QA there.
|
|
2. **Promote to production only on explicit go-live** — merge `dev` → `main`:
|
|
```bash
|
|
git log dev..main # MUST be empty — main must have nothing dev lacks (else prod would revert)
|
|
git checkout main && git merge --ff-only dev
|
|
git push gitlab main # never GitHub — opens the manual `promote` job, does not run it
|
|
git checkout dev
|
|
```
|
|
Then click `promote` in the GitLab pipeline UI (protected `production`
|
|
environment — requires the right role) once prod actually exists (see
|
|
"CI/CD" above). Never push straight to `main`. Never let a dev→main merge
|
|
silently revert prod.
|
|
|
|
## Syncing upstream (Bulwark releases)
|
|
|
|
Bring upstream into `dev` (NOT main), integrate + QA on the dev image, then promote as above:
|
|
|
|
```bash
|
|
git fetch upstream
|
|
git checkout dev && git merge upstream/main # resolve conflicts via vnc/VNC-CHANGES.md; QA on preview
|
|
```
|
|
|
|
## Auth
|
|
|
|
Basic auth via Stalwart is the default — users sign in with their
|
|
`@sandbox.vnc.de` address + password; VNCmail+ authenticates them over JMAP. No
|
|
extra config. (SSO via vncdirectory/OIDC is a later option — see
|
|
`vnc/vercel.env.template`.)
|