Bernd Rodler
177b2aca57
Direct SSH access to the actual clusters (node1-3 "prod" HA, dev-k8s-1-3
"dev") revealed two things that made the previous design wrong:
1. Neither cluster has vncmail/vnc-ca namespaces or a bulwark ingress at
all - the "live sandbox" referenced in this repo's docs/manifests was
never actually applied anywhere. Both ingress.yaml's ingressClassName
(public) and cert-manager issuer (letsencrypt-prod) were also wrong:
both clusters run Traefik (class is literally named `traefik`), and
only dev-k8s has any ClusterIssuer at all (`letsencrypt-staging`).
node1-3 has zero ClusterIssuers configured.
2. dev-k8s already has ArgoCD installed, idle, zero Applications - more
idiomatic to use it than have GitLab Runner execute kubectl directly.
Pivots .gitlab-ci.yml: build+push image, then commit the tag into a small
per-overlay Component (overlays/{dev,prod}/image-tag/) that ArgoCD's
Application watches - CI never touches the cluster, only the registry and
this repo. dev's Application (vncmail-dev) is registered and applied
already (manual sync for now, until the one-time namespace secret
bootstrap is done - see VNCMAIL-SETUP.md). prod's Application is
scaffolded in deploy/argocd/ but deliberately not applied - it targets a
different cluster (node1-3) that isn't registered with ArgoCD yet, and
there's still no real prod hostname/Stalwart/ClusterIssuer.
Fixes base/ingress.yaml to the real ingressClassName: traefik (was the
nginx-style `public`, which doesn't exist on either cluster) and gives
each overlay its own cert-manager issuer patch instead of one hardcoded
value, since dev and prod need different (or, for prod, nonexistent)
issuers.
179 lines
7.8 KiB
Markdown
179 lines
7.8 KiB
Markdown
# VNCmail+ — Admin Deployment Guide (microk8s)
|
||
|
||
Deploy VNCmail+ (VNC's Bulwark fork) as a container at **`vncmail.sandbox.vnc.de`**,
|
||
**alongside** the existing `bulwark.sandbox.vnc.de`.
|
||
|
||
> Why a container (not Vercel): Bulwark is stateful — it writes settings/admin/
|
||
> telemetry to `/app/data`, which needs persistent volumes.
|
||
|
||
## Structure — base + overlays
|
||
|
||
```
|
||
deploy/k8s/
|
||
base/ # shared manifest shapes (namespace-agnostic)
|
||
overlays/
|
||
dev/ # the live sandbox — vncmail.sandbox.vnc.de, namespace vncmail
|
||
prod/ # scaffolded, NOT YET LIVE — see "Production status" below
|
||
ca/ # separate, isolated EJBCA internal CA — see ca/README.md.
|
||
# Never composed with base/ or either overlay above.
|
||
```
|
||
|
||
`kubectl apply -k overlays/dev` (or `overlays/prod`, once real) instead of
|
||
applying `base/` directly — `base/` alone has no namespace and won't apply
|
||
meaningfully on its own.
|
||
|
||
## Routine deploys go through CI + ArgoCD now
|
||
|
||
As of the GitLab CI/CD pipeline (`.gitlab-ci.yml`, see `../../VNCMAIL-SETUP.md`
|
||
§ CI/CD), **pushing to `dev` auto-builds and bumps the deploy tag; ArgoCD's
|
||
`vncmail-dev` Application applies it** — you should not normally need to run
|
||
`kubectl apply` for the sandbox by hand anymore, and CI never touches the
|
||
cluster directly (it only ever talks to the registry and to this git repo).
|
||
This guide's manual steps below are for first-time setup, the one-time
|
||
secret creation CI/ArgoCD deliberately never automate, and troubleshooting.
|
||
|
||
## Production status
|
||
|
||
**There is no production VNCmail+ deployment yet.** `overlays/prod/` exists
|
||
in the repo but is inert: its ingress hostname and its secret's
|
||
`JMAP_SERVER_URL` are both obvious placeholders (`vncmail.CHANGEME.invalid` /
|
||
`https://REPLACE-ME-prod-stalwart-not-yet-deployed.invalid`) that will fail
|
||
loudly rather than silently deploy against the wrong backend. Applying it
|
||
requires, in order: a real prod Stalwart instance to exist, a real hostname
|
||
decision, DNS, a real `secret.yaml`, and the `.gitlab-ci.yml` `promote` job's
|
||
`kubectl apply` step (currently a TODO placeholder) filled in. None of that
|
||
is CI's job to decide — it's an explicit, human-triggered event.
|
||
|
||
---
|
||
|
||
## 1. What you are deploying (per overlay)
|
||
|
||
| # | Object | File | Purpose |
|
||
|---|--------|------|---------|
|
||
| 1 | Namespace | `overlays/<env>/namespace.yaml` | Isolates the app (`vncmail` for dev, `vncmail-prod` for prod) |
|
||
| 2 | 4× PersistentVolumeClaim | `base/pvc.yaml` | `/app/data/{settings,admin,admin-state,telemetry}` |
|
||
| 3 | Secret `vncmail-env` | `overlays/<env>/secret.yaml` *(you create it)* | App config (JMAP URL, session secret, branding) |
|
||
| 4 | Image-pull secret | *(you create it — command below)* | Pull the (currently private) image |
|
||
| 5 | Deployment `vncmail-plus` | `base/deployment.yaml` (+ overlay patches) | The app pod |
|
||
| 6 | Service `vncmail-plus` | `base/service.yaml` | ClusterIP :80 → pod :3000 |
|
||
| 7 | Ingress `vncmail-plus` | `base/ingress.yaml` (+ overlay patches for prod) | TLS host |
|
||
|
||
**Image:** CI builds and pushes to `registry.gitlab.vnc.biz/gitlab-instance-b9b5cf2f/vncmail-plus`
|
||
(tag `sha-<sha>` per deploy, moving pointers `dev-latest`/`prod-latest`). The
|
||
`ghcr.io/brvncde-dotcom/vncmail-plus-dev` image referenced in `base/deployment.yaml`
|
||
is a legacy default only — CI overrides it per-deploy via `kubectl set image`,
|
||
so what's committed there never needs to track what's actually running.
|
||
|
||
---
|
||
|
||
## 2. Pre-flight — confirm 3 cluster values (2 min)
|
||
|
||
The manifests use microk8s defaults. **Copy the exact values the existing
|
||
Bulwark uses** so VNCmail+ matches your cluster:
|
||
|
||
```bash
|
||
# Find bulwark's ingress and read off its class + cert-manager annotations:
|
||
kubectl get ingress -A | grep -i bulwark
|
||
kubectl get ingress <bulwark-ingress-name> -n <bulwark-ns> -o yaml
|
||
|
||
# List available storage classes and ingress classes:
|
||
kubectl get sc
|
||
kubectl get ingressclass
|
||
kubectl get clusterissuer # cert-manager issuers (if used)
|
||
```
|
||
|
||
Then edit if they differ from the defaults below (in `base/`, so both overlays
|
||
pick up the fix):
|
||
|
||
| Value | Default in manifests | File to edit |
|
||
|-------|----------------------|--------------|
|
||
| StorageClass | `microk8s-hostpath` | `base/pvc.yaml` (all 4) |
|
||
| IngressClass | `public` | `base/ingress.yaml` |
|
||
| cert-manager issuer | `letsencrypt-prod` | `base/ingress.yaml` |
|
||
|
||
---
|
||
|
||
## 3. First-time setup (one-time, per environment — CI never does this)
|
||
|
||
```bash
|
||
cd deploy/k8s/overlays/dev # or overlays/prod, once real
|
||
|
||
# a) Image-pull secret — the registry package is private.
|
||
kubectl create secret docker-registry ghcr-pull \
|
||
--namespace vncmail \
|
||
--docker-server=ghcr.io \
|
||
--docker-username=brvncde-dotcom \
|
||
--docker-password='<GITHUB_PAT_read:packages>' \
|
||
--docker-email=br@vnc.biz
|
||
# Once CI has cut over to registry.gitlab.vnc.biz, this becomes a
|
||
# docker-registry secret for that registry instead — see VNCMAIL-SETUP.md.
|
||
|
||
# b) App config secret — copy the template, set a real SESSION_SECRET, apply.
|
||
cp secret.example.yaml secret.yaml
|
||
# edit secret.yaml: SESSION_SECRET: "$(openssl rand -base64 32)"
|
||
kubectl apply -f secret.yaml
|
||
|
||
# c) Everything else (namespace, PVCs, Deployment, Service, Ingress)
|
||
kubectl apply -k .
|
||
```
|
||
|
||
> Alternative to (a): make the registry package public, then delete the
|
||
> `imagePullSecrets:` block from `base/deployment.yaml`.
|
||
|
||
After this one-time setup, routine deploys to `dev` happen automatically via
|
||
CI on every push — see "Routine deploys go through CI now" above. This
|
||
section is for first-time bring-up (or `overlays/prod`, once it's real) and
|
||
troubleshooting, not the everyday path.
|
||
|
||
---
|
||
|
||
## 4. Verify
|
||
|
||
```bash
|
||
kubectl -n vncmail rollout status deploy/vncmail-plus # -> successfully rolled out
|
||
kubectl -n vncmail get pods,pvc,ingress
|
||
|
||
# DNS: point vncmail.sandbox.vnc.de at the same ingress IP as bulwark.sandbox.vnc.de.
|
||
# cert-manager issues TLS once DNS resolves. Then:
|
||
curl -sI https://vncmail.sandbox.vnc.de/api/health # -> HTTP/2 200
|
||
```
|
||
|
||
Open `https://vncmail.sandbox.vnc.de` and log in with a **full** email address
|
||
(e.g. `bernd.rodler@sandbox.vnc.de`) — Stalwart authenticates the full email, not
|
||
a bare username.
|
||
|
||
---
|
||
|
||
## 5. Update to a new build
|
||
|
||
Normally you don't — CI's `bump-dev` job + ArgoCD's automated sync do this
|
||
on every push to `dev`. To do it by hand (e.g. troubleshooting, before
|
||
automated sync is turned on):
|
||
|
||
```bash
|
||
kubectl -n vncmail set image deploy/vncmail-plus \
|
||
vncmail-plus=registry.gitlab.vnc.biz/gitlab-instance-b9b5cf2f/vncmail-plus:sha-<sha>
|
||
```
|
||
|
||
ArgoCD will overwrite this on its next sync unless you also update
|
||
`deploy/k8s/overlays/dev/image-tag/kustomization.yaml` to match — that file
|
||
is CI-owned (see its header comment), so a by-hand `set image` is only ever
|
||
a temporary override, not a real fix.
|
||
|
||
Rollback (bypassing ArgoCD temporarily): `kubectl -n vncmail rollout undo deploy/vncmail-plus`.
|
||
The real rollback is reverting the commit that bumped the tag and letting
|
||
ArgoCD re-sync.
|
||
|
||
---
|
||
|
||
## 6. Troubleshooting
|
||
|
||
| Symptom | Cause / fix |
|
||
|---------|-------------|
|
||
| Pod `ImagePullBackOff` | `ghcr-pull` secret missing/expired, or package still private. Recreate the secret (§3a) or make the package public. |
|
||
| Pod `CrashLoopBackOff`, logs show `EACCES`/permission on `/app/data` | Volume not writable by uid 1001. `securityContext.fsGroup: 1001` is set in `base/deployment.yaml` — keep it; some storage drivers also need it on the PVC. |
|
||
| PVC stuck `Pending` | Wrong `storageClassName` in `base/pvc.yaml`. Set it to one from `kubectl get sc`. |
|
||
| Ingress has no address / no cert | Wrong `ingressClassName` or cert issuer. Match bulwark's (§2). Check `kubectl -n vncmail describe ingress vncmail-plus`. |
|
||
| Login shows "Ein Fehler ist aufgetreten" | Use the **full** email (`user@sandbox.vnc.de`), not a bare username. |
|
||
| Can't reach Stalwart | Check `JMAP_SERVER_URL` in the secret = `https://stalwart.sandbox.vnc.de`. |
|