feat(ci): pivot to ArgoCD GitOps, fix Traefik ingress after real-cluster check

Direct SSH access to the actual clusters (node1-3 "prod" HA, dev-k8s-1-3
"dev") revealed two things that made the previous design wrong:

1. Neither cluster has vncmail/vnc-ca namespaces or a bulwark ingress at
   all - the "live sandbox" referenced in this repo's docs/manifests was
   never actually applied anywhere. Both ingress.yaml's ingressClassName
   (public) and cert-manager issuer (letsencrypt-prod) were also wrong:
   both clusters run Traefik (class is literally named `traefik`), and
   only dev-k8s has any ClusterIssuer at all (`letsencrypt-staging`).
   node1-3 has zero ClusterIssuers configured.

2. dev-k8s already has ArgoCD installed, idle, zero Applications - more
   idiomatic to use it than have GitLab Runner execute kubectl directly.

Pivots .gitlab-ci.yml: build+push image, then commit the tag into a small
per-overlay Component (overlays/{dev,prod}/image-tag/) that ArgoCD's
Application watches - CI never touches the cluster, only the registry and
this repo. dev's Application (vncmail-dev) is registered and applied
already (manual sync for now, until the one-time namespace secret
bootstrap is done - see VNCMAIL-SETUP.md). prod's Application is
scaffolded in deploy/argocd/ but deliberately not applied - it targets a
different cluster (node1-3) that isn't registered with ArgoCD yet, and
there's still no real prod hostname/Stalwart/ClusterIssuer.

Fixes base/ingress.yaml to the real ingressClassName: traefik (was the
nginx-style `public`, which doesn't exist on either cluster) and gives
each overlay its own cert-manager issuer patch instead of one hardcoded
value, since dev and prod need different (or, for prod, nonexistent)
issuers.

.gitlab-ci.yml

@@ -1,51 +1,61 @@
-# GitLab-CI dev→prod pipeline for VNCmail+.
+# GitLab-CI dev→prod pipeline for VNCmail+ — GitOps via ArgoCD.
 #
-# Design (see the approved plan for full rationale):
+# Revised after direct inspection of the real infrastructure found ArgoCD
+# already installed (idle, zero Applications) on the dev-k8s-1/2/3 cluster.
+# That's more idiomatic than a runner-executes-kubectl design, and it means
+# this pipeline needs ZERO cluster credentials — CI only ever talks to the
+# container registry and to this git repo. ArgoCD (which already has
+# whatever cluster access it needs, set up once when its Applications were
+# registered — see deploy/argocd/) is what actually applies anything.
+#
+# Design:
 #   - One image name, environment lives only in the tag. No more -dev/-beta
-#     name confusion.
+#     name confusion from the old GitHub Actions workflow.
 #   - MR into `dev`: verify only (typecheck/lint/unit test/build check). No
 #     push, no deploy — this is the multi-developer merge gate.
-#   - Push to `dev`: build+push an immutable `sha-<sha>` tag, auto-deploy it
-#     to the vncmail (sandbox) namespace. No approval needed — dev always
-#     deploys.
+#   - Push to `dev`: build+push an immutable `sha-<sha>` tag, then commit a
+#     one-line tag-bump into overlays/dev/image-tag/kustomization.yaml
+#     (`[skip ci]`, so this doesn't retrigger itself). ArgoCD's `vncmail-dev`
+#     Application has automated sync — it notices the git change and applies
+#     it. No approval needed, dev always deploys, and this job never touches
+#     the cluster directly.
 #   - Push to `main`: NEVER rebuilds. `main` only ever advances via
 #     `git merge --ff-only dev`, so main's HEAD commit already has a built
-#     image. The `promote` job retags that exact digest (registry-side copy,
-#     same primitive the old docker-publish.yml GHA workflow already used for
-#     its multi-arch manifest-list merge) and applies it to prod. `when:
-#     manual` + a protected `production` GitLab environment is the approval
-#     gate — nobody but an authorized user can click it, and nothing here
-#     runs automatically on main.
+#     image (the same sha- tag dev already deployed). This job just bumps
+#     overlays/prod/image-tag/kustomization.yaml to point at that same tag.
+#     The actual promotion gate is a HUMAN clicking Sync on the `vncmail-prod` ArgoCD
+#     Application (deliberately NOT automated sync) — not a GitLab manual
+#     job, since ArgoCD already provides that exact gate more directly.
+#     Until prod Stalwart/hostname/secrets are real (see VNCMAIL-SETUP.md),
+#     nobody should click that Sync button — but nothing here does it for
+#     you either way.
 #
-# Deliberately single-platform (linux/amd64) for the cluster build — this
-# pipeline's job is deploying to a known amd64 microk8s cluster, not public
-# multi-arch distribution (that's what the GHCR release workflows are for,
-# and they're untouched by this file).
+# Deliberately single-platform (linux/amd64) — this pipeline serves two
+# known amd64 microk8s clusters, not public multi-arch distribution (that's
+# what the GHCR release workflows are for, untouched by this file).
 #
-# Prerequisites this pipeline assumes are already in place (see the plan's
-# "Split of responsibility" — these are admin/infra actions, not something
-# this file can set up):
-#   - GitLab Container Registry enabled for this project (CI_REGISTRY_* vars
-#     are then provided automatically — no manual credential setup needed).
-#   - A GitLab Runner with the Kubernetes executor, whose deploy-stage jobs
-#     run as a `gitlab-deployer` ServiceAccount scoped (namespaced Role, not
-#     cluster-admin) to the `vncmail` namespace (and later `vncmail-prod`).
-#     kubectl auto-detects in-cluster config from that ServiceAccount's
-#     mounted token — no KUBECONFIG variable required.
+# Prerequisite this file assumes (documented in VNCMAIL-SETUP.md, not
+# something this file can set up itself):
+#   - GitLab Container Registry enabled for this project (confirmed done).
+#   - A GitLab Runner (any kind — no cluster access needed at all now).
+#   - Either "allow this job token to push to this project" enabled
+#     (Settings → CI/CD → Job token permissions), OR a project access token
+#     with `write_repository` scope in $GITLAB_PUSH_TOKEN. The job below
+#     tries CI_JOB_TOKEN first (see the script).
 #
-# deploy/k8s/ca/ (the EJBCA internal CA) is never referenced anywhere below —
-# that stays a fully manual, human-only runbook (see deploy/k8s/ca/README.md).
+# deploy/k8s/ca/ (the EJBCA internal CA) is never referenced anywhere below,
+# and neither ArgoCD Application in deploy/argocd/ points at it — that stays
+# a fully manual, human-only runbook (see deploy/k8s/ca/README.md).
 
 stages:
   - verify
   - build
-  - deploy-dev
-  - promote
+  - bump-dev
+  - bump-prod
 
 variables:
   IMAGE: $CI_REGISTRY_IMAGE/vncmail-plus
-  DEV_NAMESPACE: vncmail
-  PROD_NAMESPACE: vncmail-prod
+  GIT_STRATEGY: clone
 
 # ---------------------------------------------------------------------------
 # verify — required check on every MR into dev. No registry, no cluster.
@@ -62,9 +72,8 @@ verify:
     - npm run test:translations
     - npm run build
   # test:integration is deliberately NOT here — it spins up a real Stalwart
-  # fixture via docker-compose (Docker-in-Docker), which is heavier than a
-  # fast MR gate should be. Candidate for a separate scheduled/optional job
-  # later, not a blocker for this pipeline's first cut.
+  # fixture via docker-compose (Docker-in-Docker), heavier than a fast MR
+  # gate should be. Candidate for a separate scheduled job, not a blocker.
 
 # ---------------------------------------------------------------------------
 # build — push to dev only. Builds once; main never rebuilds (see header).
@@ -84,61 +93,73 @@ build:
     - docker push "$IMAGE:dev-latest"
 
 # ---------------------------------------------------------------------------
-# deploy-dev — automatic, no approval. Deploys the immutable sha tag, never
-# the moving dev-latest pointer, so what's running always matches one commit.
+# bump-dev — no cluster access. Commits the just-built tag into the overlay
+# ArgoCD watches; ArgoCD's automated sync does the actual apply.
 # ---------------------------------------------------------------------------
-deploy-dev:
-  stage: deploy-dev
-  image: bitnami/kubectl:1.31
-  environment:
-    name: dev
-    url: https://vncmail.sandbox.vnc.de
+bump-dev:
+  stage: bump-dev
+  image: alpine/git:2.47.0
   rules:
     - if: '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == "dev"'
   script:
-    # Apply the manifests first (structure/config), then set the exact image
-    # this pipeline just built — imperative `set image`, not a kustomize-file
-    # edit, so overlays/dev never needs a commit to change what's deployed.
-    - kubectl apply -k deploy/k8s/overlays/dev
-    - kubectl -n $DEV_NAMESPACE set image deployment/vncmail-plus vncmail-plus="$IMAGE:sha-$CI_COMMIT_SHORT_SHA"
-    - kubectl -n $DEV_NAMESPACE rollout status deploy/vncmail-plus --timeout=120s
+    - TAG="sha-$CI_COMMIT_SHORT_SHA"
+    - |
+      cat > deploy/k8s/overlays/dev/image-tag/kustomization.yaml <<EOF
+      # Owned by CI (bump-dev job in .gitlab-ci.yml) - regenerated every
+      # push to dev. Do not hand-edit; edits here get overwritten.
+      apiVersion: kustomize.config.k8s.io/v1alpha1
+      kind: Component
+      images:
+        - name: ghcr.io/brvncde-dotcom/vncmail-plus-dev
+          newName: $IMAGE
+          newTag: $TAG
+      EOF
+    - git config user.name "vncmail-ci"
+    - git config user.email "ci@vnc.biz"
+    - git add deploy/k8s/overlays/dev/image-tag/kustomization.yaml
+    - |
+      if git diff --cached --quiet; then
+        echo "No change (tag already pinned) - nothing to commit"
+      else
+        git commit -m "chore(deploy): pin dev to $TAG [skip ci]"
+        git push "https://gitlab-ci-token:${GITLAB_PUSH_TOKEN:-$CI_JOB_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git" HEAD:dev
+      fi
 
 # ---------------------------------------------------------------------------
-# promote — manual, protected `production` environment. No docker build here
-# — retags the exact digest already deployed to dev, then applies prod
-# pinned to that digest (never a mutable tag).
+# bump-prod — no cluster access, no rebuild. Points overlays/prod at the
+# exact tag already running on dev. Does NOT deploy anything: vncmail-prod's
+# ArgoCD Application has manual sync, so this only prepares what a human
+# would be syncing, it doesn't sync it.
 # ---------------------------------------------------------------------------
-promote:
-  stage: promote
-  image: docker:27-cli
-  services:
-    - docker:27-dind
-  environment:
-    name: production
-    url: https://vncmail.CHANGEME.invalid # placeholder until the real prod host is decided
+bump-prod:
+  stage: bump-prod
+  image: alpine/git:2.47.0
   rules:
-    # `when: manual` lives inside the rule (not as a top-level job key) —
-    # required syntax once `rules:` is used at all.
     - if: '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == "main"'
-      when: manual
-  before_script:
-    - echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" "$CI_REGISTRY" --password-stdin
   script:
-    - echo "Retagging the image already built+deployed for dev commit $CI_COMMIT_SHA — no rebuild."
-    - docker buildx imagetools create --tag "$IMAGE:prod-latest" "$IMAGE:sha-$CI_COMMIT_SHORT_SHA"
-    - DIGEST=$(docker buildx imagetools inspect "$IMAGE:sha-$CI_COMMIT_SHORT_SHA" | awk '/^Digest:/{print $2}')
-    - echo "Resolved digest for prod = $IMAGE@$DIGEST"
-    - >
-      echo "STOPPING HERE ON PURPOSE: deploy/k8s/overlays/prod is still
-      scaffolded/inactive (placeholder hostname, placeholder JMAP_SERVER_URL
-      — no prod Stalwart exists yet). Once both are real (Phase D in the
-      pipeline plan / VNCMAIL-SETUP.md), replace this echo with the same
-      pattern deploy-dev uses, against a bitnami/kubectl image and
-      \$PROD_NAMESPACE: kubectl apply -k deploy/k8s/overlays/prod &&
-      kubectl -n \$PROD_NAMESPACE set image deployment/vncmail-plus
-      vncmail-plus=$IMAGE@$DIGEST"
-  # Deliberately does NOT run `kubectl apply -k overlays/prod` yet — prod
-  # namespace/hostname/Stalwart don't exist (Phase C/D in the plan). Once
-  # they do, replace the placeholder echo above with the same
-  # `kubectl apply -k .` + `set image ...@$DIGEST` pattern deploy-dev uses,
-  # against $PROD_NAMESPACE, using the bitnami/kubectl image.
+    - TAG="sha-$CI_COMMIT_SHORT_SHA"
+    - echo "main advanced to $CI_COMMIT_SHA (must be a dev commit, ff-only) - that image already exists as $IMAGE:$TAG"
+    - |
+      cat > deploy/k8s/overlays/prod/image-tag/kustomization.yaml <<EOF
+      # Owned by CI (bump-prod job in .gitlab-ci.yml) - regenerated every
+      # push to main. Do not hand-edit; edits here get overwritten. Bumping
+      # this is NOT the same as deploying it - vncmail-prod's ArgoCD
+      # Application has manual sync, see the note in the parent
+      # kustomization.yaml.
+      apiVersion: kustomize.config.k8s.io/v1alpha1
+      kind: Component
+      images:
+        - name: ghcr.io/brvncde-dotcom/vncmail-plus-dev
+          newName: $IMAGE
+          newTag: $TAG
+      EOF
+    - git config user.name "vncmail-ci"
+    - git config user.email "ci@vnc.biz"
+    - git add deploy/k8s/overlays/prod/image-tag/kustomization.yaml
+    - |
+      if git diff --cached --quiet; then
+        echo "No change (tag already pinned) - nothing to commit"
+      else
+        git commit -m "chore(deploy): point prod overlay at $TAG (not synced - manual gate in ArgoCD) [skip ci]"
+        git push "https://gitlab-ci-token:${GITLAB_PUSH_TOKEN:-$CI_JOB_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git" HEAD:main
+      fi

VNCMAIL-SETUP.md

@@ -33,37 +33,89 @@ of truth; VNCmail+ is the UI. It deploys as a **container on Kubernetes
 
 All VNC customization lives under `vnc/` (see `vnc/VNC-CHANGES.md`).
 
-## CI/CD — GitLab (canonical), Vercel-style dev→prod
+## CI/CD — GitLab (canonical) + ArgoCD GitOps, Vercel-style dev→prod
 
-Multiple developers work on this repo now, so `.gitlab-ci.yml` on
+Multiple developers work on this repo now. `.gitlab-ci.yml` on
 [gitlab.vnc.biz](https://gitlab.vnc.biz/gitlab-instance-b9b5cf2f/vncmail-plus)
 (the canonical remote — GitHub `origin` is a passive mirror, not where CI or
-deploys happen) drives the whole flow:
+deploys happen) builds images and bumps a tag in git; **ArgoCD does the
+actual deploying** — already installed and idle on the `dev-k8s-1/2/3`
+cluster, discovered when standing this up. GitLab CI needs zero cluster
+credentials as a result.
 
-1. **MR into `dev`** → `verify` stage runs (typecheck/lint/unit test/build).
-   Required check — no push, no deploy. This is the multi-developer gate.
+Two real clusters, confirmed by direct inspection:
+
+| Cluster | Role | Notes |
+|---|---|---|
+| `dev-k8s-1/2/3` | dev/sandbox | ~hours old when set up here. Traefik, metallb, cert-manager (`letsencrypt-staging` issuer only), **ArgoCD already running**. |
+| `node1/node2/node3` | prod (HA) | Older, rook-ceph+traefik+metallb+cert-manager, but **zero apps and zero ClusterIssuers** — genuinely a clean slate. |
+
+Neither cluster had a `vncmail` namespace, `vnc-ca` namespace, or `bulwark`
+ingress — the "live sandbox at vncmail.sandbox.vnc.de" referenced earlier in
+this doc's history was aspirational (manifests + docs existed, nothing was
+ever actually applied). The ingress manifests also assumed nginx (`class:
+public`, an nginx body-size annotation) — fixed to Traefik's real
+`ingressClassName: traefik` (Traefik has no default body-size cap, so no
+replacement annotation is needed).
+
+Flow:
+
+1. **MR into `dev`** → `verify` stage (typecheck/lint/unit test/build).
+   Required check — no push, no deploy.
 2. **Merge to `dev`** → `build` pushes one image,
-   `registry.gitlab.vnc.biz/.../vncmail-plus:sha-<sha>`, then `deploy-dev`
-   applies it to the sandbox automatically. No approval needed — dev always
-   deploys first.
-3. **Merge to `main`** (fast-forward only, see below) → a `promote` job
-   appears, `when: manual`, gated behind a protected `production`
-   GitLab environment. It **never rebuilds** — it retags the exact image
-   already running on dev (registry-side copy, same digest) and would apply
-   it to a `vncmail-prod` namespace pinned to that digest.
+   `registry.gitlab.vnc.biz/.../vncmail-plus:sha-<sha>`, then `bump-dev`
+   commits that tag into `deploy/k8s/overlays/dev/image-tag/kustomization.yaml`
+   (`[skip ci]`). ArgoCD's `vncmail-dev` Application picks up the git change.
+3. **Merge to `main`** (fast-forward only, see below) → `bump-prod` points
+   `overlays/prod/image-tag/` at that same tag — **no rebuild**. The actual
+   promotion gate is a **human clicking Sync** on the `vncmail-prod` ArgoCD
+   Application, which is permanently manual-sync (never automated) — that's
+   the Vercel-style "Promote to Production" button, just living in ArgoCD's
+   UI instead of GitLab's.
+
+### What's left to wire up (one-time, human steps)
+
+1. **Add the ArgoCD deploy key to GitLab** — Project → Settings → Repository
+   → Deploy keys → add (read-only is enough):
+   ```
+   ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOURjX/Y9zfB785DyLEF1GUq4HhWujrqeXag8oxdMciq argocd@dev-k8s (vncmail-plus read-only)
+   ```
+   Until this is added, `vncmail-dev`'s ArgoCD Application (already created,
+   `kubectl -n argocd get application vncmail-dev`) shows a benign
+   `ComparisonError` (SSH handshake failing) — expected, not a bug.
+2. **Let CI push tag-bumps back to this repo** — either enable "this project
+   can be accessed by CI/CD job tokens from other projects" → actually
+   simpler: Settings → CI/CD → Job token permissions → allow this project's
+   own job token to push to itself, OR create a Project Access Token
+   (`write_repository` scope) and add it as a masked CI/CD variable
+   `GITLAB_PUSH_TOKEN` (the pipeline tries that first, falls back to
+   `CI_JOB_TOKEN`).
+3. **One-time namespace bootstrap** (CI/ArgoCD deliberately never manage
+   secret contents — see `deploy/k8s/README.md` §3):
+   ```bash
+   # against dev-k8s (ArgoCD's CreateNamespace=true will make `vncmail` on
+   # first sync, or create it yourself first — either order works)
+   kubectl create secret docker-registry ghcr-pull -n vncmail ...   # or make the GHCR package public
+   cp deploy/k8s/overlays/dev/secret.example.yaml secret.yaml   # edit SESSION_SECRET
+   kubectl apply -f secret.yaml
+   ```
+4. **First sync** — ArgoCD UI at `https://argo.devcluster.vnc.de`
+   (username `admin`, password: `kubectl -n argocd get secret
+   argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d`
+   — rotate it after logging in once) → `vncmail-dev` → Sync. Once that's
+   clean, flip `deploy/argocd/vncmail-dev-app.yaml`'s commented-out
+   `automated:` block on and re-apply, so dev auto-syncs on every push from
+   then on.
+5. **Production** (later, deliberately not wired yet): decide a real
+   hostname, stand up prod Stalwart, register `node1-3` as an ArgoCD-managed
+   cluster, apply `deploy/argocd/vncmail-prod-app.yaml`, fill in real
+   `overlays/prod` values, create a real ClusterIssuer on `node1-3` (there
+   isn't one today), then click Sync once — deliberately not before.
 
 Historical note: the old `-dev`/`-beta` GHCR image-name split
 (`.github/workflows/docker-publish.yml`) is retired by this — one image name
 now, environment lives only in the tag.
 
-**Production doesn't exist yet.** `deploy/k8s/overlays/prod/` is scaffolded
-(placeholder hostname, placeholder `JMAP_SERVER_URL` — there's no prod
-Stalwart instance to point it at either) but inert: the `promote` job's real
-`kubectl apply` step is deliberately left as a TODO in `.gitlab-ci.yml` until
-a real hostname is decided and prod Stalwart exists. Standing up the runner/
-RBAC/registry this pipeline needs is an infra prerequisite, not something CI
-itself does — see the pipeline design doc referenced in `deploy/k8s/README.md`.
-
 ## Deploy (Kubernetes / microk8s)
 
 Full runbook: **[deploy/k8s/README.md](deploy/k8s/README.md)**. In short:

deploy/argocd/vncmail-dev-app.yaml

@@ -0,0 +1,29 @@
+# Registered on the dev-k8s-1/2/3 cluster (where ArgoCD already lives) via
+# `kubectl apply` directly to the argocd namespace — this file is the
+# version-controlled record of that, not something ArgoCD itself syncs
+# (no app-of-apps here, deliberately kept simple for two Applications).
+#
+# syncPolicy starts WITHOUT automated — manual sync until the one-time
+# per-namespace bootstrap (vncmail-env secret, image-pull secret — see
+# deploy/k8s/README.md §3) is done by hand once. Flip to automated (see
+# commented block below) only after a first manual sync succeeds cleanly.
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: vncmail-dev
+  namespace: argocd
+spec:
+  project: default
+  source:
+    repoURL: git@gitlab.vnc.biz:gitlab-instance-b9b5cf2f/vncmail-plus.git
+    targetRevision: dev
+    path: deploy/k8s/overlays/dev
+  destination:
+    server: https://kubernetes.default.svc # in-cluster — ArgoCD and vncmail-dev share this cluster
+    namespace: vncmail
+  syncPolicy:
+    syncOptions:
+      - CreateNamespace=true
+    # automated:
+    #   prune: true
+    #   selfHeal: true

deploy/argocd/vncmail-prod-app.yaml

@@ -0,0 +1,30 @@
+# NOT YET APPLIED to any cluster. Scaffolding only, matching
+# deploy/k8s/overlays/prod's own "inert until Phase D" status.
+#
+# Unlike vncmail-dev-app.yaml, this targets a DIFFERENT cluster (node1-3,
+# the HA "prod" cluster) than the one ArgoCD itself runs on (dev-k8s).
+# That means before this can be applied, node1-3 needs to be registered as
+# an ArgoCD-managed cluster (`argocd cluster add`, or an equivalent
+# ServiceAccount+kubeconfig secret) — deliberately not done yet: there's no
+# reason to wire cross-cluster RBAC into the prod HA cluster before prod
+# hostname/Stalwart/secrets are real and someone's actually promoting.
+#
+# syncPolicy has no automated block at all, and won't get one even later —
+# prod stays manual-sync-only permanently. That's the promotion gate.
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: vncmail-prod
+  namespace: argocd
+spec:
+  project: default
+  source:
+    repoURL: git@gitlab.vnc.biz:gitlab-instance-b9b5cf2f/vncmail-plus.git
+    targetRevision: main
+    path: deploy/k8s/overlays/prod
+  destination:
+    server: CHANGEME # the node1-3 cluster's registered ArgoCD server URL, once added
+    namespace: vncmail-prod
+  syncPolicy:
+    syncOptions:
+      - CreateNamespace=true

deploy/k8s/README.md

@@ -22,13 +22,15 @@ deploy/k8s/
 applying `base/` directly — `base/` alone has no namespace and won't apply
 meaningfully on its own.
 
-## Routine deploys go through CI now
+## Routine deploys go through CI + ArgoCD now
 
 As of the GitLab CI/CD pipeline (`.gitlab-ci.yml`, see `../../VNCMAIL-SETUP.md`
-§ CI/CD), **pushing to `dev` auto-builds and auto-deploys** — you should not
-normally need to run `kubectl apply` for the sandbox by hand anymore. This
-guide's manual steps below are for first-time setup, the one-time secret
-creation CI deliberately never automates, and troubleshooting.
+§ CI/CD), **pushing to `dev` auto-builds and bumps the deploy tag; ArgoCD's
+`vncmail-dev` Application applies it** — you should not normally need to run
+`kubectl apply` for the sandbox by hand anymore, and CI never touches the
+cluster directly (it only ever talks to the registry and to this git repo).
+This guide's manual steps below are for first-time setup, the one-time
+secret creation CI/ArgoCD deliberately never automate, and troubleshooting.
 
 ## Production status
 
@@ -144,15 +146,23 @@ a bare username.
 
 ## 5. Update to a new build
 
-Normally you don't — CI's `deploy-dev` job does this automatically on every
-push to `dev`. To do it by hand (e.g. troubleshooting):
+Normally you don't — CI's `bump-dev` job + ArgoCD's automated sync do this
+on every push to `dev`. To do it by hand (e.g. troubleshooting, before
+automated sync is turned on):
 
 ```bash
 kubectl -n vncmail set image deploy/vncmail-plus \
   vncmail-plus=registry.gitlab.vnc.biz/gitlab-instance-b9b5cf2f/vncmail-plus:sha-<sha>
 ```
 
-Rollback: `kubectl -n vncmail rollout undo deploy/vncmail-plus`
+ArgoCD will overwrite this on its next sync unless you also update
+`deploy/k8s/overlays/dev/image-tag/kustomization.yaml` to match — that file
+is CI-owned (see its header comment), so a by-hand `set image` is only ever
+a temporary override, not a real fix.
+
+Rollback (bypassing ArgoCD temporarily): `kubectl -n vncmail rollout undo deploy/vncmail-plus`.
+The real rollback is reverting the commit that bumped the tag and letting
+ArgoCD re-sync.
 
 ---
 

deploy/k8s/base/ingress.yaml

@@ -1,27 +1,28 @@
-# Exposes VNCmail+ at vncmail.sandbox.vnc.de, alongside bulwark.sandbox.vnc.de.
-# MATCH YOUR CLUSTER — inspect the existing Bulwark ingress and copy its
-# ingressClassName + TLS/cert-manager annotations:
-#   kubectl get ingress -A | grep bulwark
-#   kubectl get ingress <bulwark-ingress> -n <ns> -o yaml
+# Both real clusters (node1-3 "prod", dev-k8s-1-3 "dev") run Traefik, not
+# nginx — confirmed via `kubectl get ingressclass` (class is literally named
+# `traefik`). Unlike nginx's restrictive 1MB default, Traefik has no default
+# request-body-size cap, so there's no equivalent needed for mail attachment
+# uploads (the old nginx.ingress.kubernetes.io/proxy-body-size annotation
+# this file used to carry is simply not applicable here).
+#
+# Host, TLS secretName, and cert-manager issuer are ALL overlay-specific now
+# (dev-k8s only has a `letsencrypt-staging` issuer; node1-3/prod has none
+# configured yet) — every overlay's patch-ingress.yaml must override the
+# CHANGEME placeholders below.
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
   name: vncmail-plus
   annotations:
-    # cert-manager issuer — set to whatever bulwark.sandbox.vnc.de uses.
-    cert-manager.io/cluster-issuer: letsencrypt-prod
-    # Mail attachments can be large; raise the nginx body limit.
-    nginx.ingress.kubernetes.io/proxy-body-size: "100m"
+    cert-manager.io/cluster-issuer: CHANGEME
 spec:
-  # microk8s ingress addon class is usually "public" (nginx). Confirm with
-  # `kubectl get ingressclass` and match bulwark's.
-  ingressClassName: public
+  ingressClassName: traefik
   tls:
     - hosts:
-        - vncmail.sandbox.vnc.de
+        - CHANGEME.invalid
       secretName: vncmail-plus-tls
   rules:
-    - host: vncmail.sandbox.vnc.de
+    - host: CHANGEME.invalid
       http:
         paths:
           - path: /

deploy/k8s/overlays/dev/image-tag/kustomization.yaml

@@ -0,0 +1,11 @@
+# Owned by CI (the bump-dev job in .gitlab-ci.yml), not by hand. Kept as its
+# own Component so CI only ever rewrites this 6-line file, never the parent
+# overlays/dev/kustomization.yaml (structure/patches there stay under normal
+# code review — CI regenerating a whole hand-maintained file on every push
+# would silently revert any change made there between deploys).
+apiVersion: kustomize.config.k8s.io/v1alpha1
+kind: Component
+images:
+  - name: ghcr.io/brvncde-dotcom/vncmail-plus-dev
+    newName: ghcr.io/brvncde-dotcom/vncmail-plus-dev
+    newTag: latest

deploy/k8s/overlays/dev/kustomization.yaml

@@ -6,8 +6,13 @@ resources:
   - namespace.yaml
   # - secret.yaml   # create from secret.example.yaml; not committed
 
-# This is the live sandbox (vncmail.sandbox.vnc.de) — deliberately zero patches
-# beyond namespace/resource wiring, so `kubectl kustomize .` renders identical
-# to the pre-restructure flat deploy/k8s/. The image is left at base's default
-# and overridden per-deploy by CI (`kubectl set image`, see .gitlab-ci.yml's
-# deploy-dev job) rather than pinned here, so this file never goes stale.
+patches:
+  - path: patch-ingress.yaml
+
+components:
+  - image-tag
+
+# Targets the dev-k8s-1/2/3 cluster (confirmed via direct access: this is
+# where ArgoCD already lives). The image tag lives in image-tag/ (a separate
+# Component CI owns — see .gitlab-ci.yml's bump-dev job) rather than here, so
+# CI never needs to touch this file.

deploy/k8s/overlays/dev/patch-ingress.yaml

@@ -0,0 +1,27 @@
+# dev-k8s cluster confirmed to have a `letsencrypt-staging` ClusterIssuer
+# already (no `letsencrypt-prod` exists there) - staging avoids burning
+# Let's Encrypt's real rate limits while this is still being stood up.
+# vncmail.sandbox.vnc.de DNS does not point here yet either - this is the
+# intended host, not a live one (see VNCMAIL-SETUP.md for what's still open).
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: vncmail-plus
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-staging
+spec:
+  tls:
+    - hosts:
+        - vncmail.sandbox.vnc.de
+      secretName: vncmail-plus-tls
+  rules:
+    - host: vncmail.sandbox.vnc.de
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: vncmail-plus
+                port:
+                  number: 80

deploy/k8s/overlays/prod/image-tag/kustomization.yaml

@@ -0,0 +1,12 @@
+# Owned by CI (the bump-prod job in .gitlab-ci.yml), not by hand — same
+# reasoning as overlays/dev/image-tag/. Starts pointed at an obviously-fake
+# tag on purpose: nothing has been promoted yet, and vncmail-prod's ArgoCD
+# Application has manual sync anyway, so this being "wrong" doesn't deploy
+# anything wrong — it just means there's nothing to sync until a real
+# `git push` to main updates it.
+apiVersion: kustomize.config.k8s.io/v1alpha1
+kind: Component
+images:
+  - name: ghcr.io/brvncde-dotcom/vncmail-plus-dev
+    newName: registry.gitlab.vnc.biz/gitlab-instance-b9b5cf2f/vncmail-plus
+    newTag: not-yet-promoted

deploy/k8s/overlays/prod/kustomization.yaml

@@ -10,12 +10,12 @@ patches:
   - path: patch-ingress.yaml
   - path: patch-deployment.yaml
 
-# NOT MEANT TO BE APPLIED AS COMMITTED. Scaffolding only (see the pipeline
-# plan's Phase C/D) — the tag below is an obviously-invalid placeholder;
-# the real promote job (.gitlab-ci.yml) resolves and pins an actual digest at
-# deploy time via `kubectl set image`, it never trusts whatever is checked
-# in here.
-images:
-  - name: ghcr.io/brvncde-dotcom/vncmail-plus-dev
-    newName: registry.gitlab.vnc.biz/gitlab-instance-b9b5cf2f/vncmail-plus
-    newTag: not-yet-promoted
+components:
+  - image-tag
+
+# NOT MEANT TO BE SYNCED AS COMMITTED. Scaffolding only (see the pipeline
+# plan's Phase C/D) — image-tag/'s placeholder tag is obviously-invalid on
+# purpose. The bump-prod job in .gitlab-ci.yml keeps that tag pointed at
+# whatever's already on dev once main advances, but vncmail-prod's ArgoCD
+# Application has manual sync — a human still has to click Sync (or
+# `argocd app sync vncmail-prod`) for any of this to actually apply.

deploy/k8s/overlays/prod/patch-ingress.yaml

@@ -3,6 +3,12 @@
 # deliberately unresolvable: applying this overlay as committed will not
 # issue a cert or route traffic anywhere. Replace both occurrences below,
 # and the matching TLS secretName, before Phase D (first real prod deploy).
+#
+# Targets node1-3 (the HA "prod" cluster). Deliberately does NOT override
+# base's `cert-manager.io/cluster-issuer: CHANGEME` — node1-3 has ZERO
+# ClusterIssuers configured today (confirmed via direct access). A human
+# needs to create a real one there (ACME account, DNS-01 or HTTP-01 solver)
+# before this can be anything but a placeholder.
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata: