feat(ci): pivot to ArgoCD GitOps, fix Traefik ingress after real-cluster check

Direct SSH access to the actual clusters (node1-3 "prod" HA, dev-k8s-1-3 "dev") revealed two things that made the previous design wrong: 1. Neither cluster has vncmail/vnc-ca namespaces or a bulwark ingress at all - the "live sandbox" referenced in this repo's docs/manifests was never actually applied anywhere. Both ingress.yaml's ingressClassName (public) and cert-manager issuer (letsencrypt-prod) were also wrong: both clusters run Traefik (class is literally named `traefik`), and only dev-k8s has any ClusterIssuer at all (`letsencrypt-staging`). node1-3 has zero ClusterIssuers configured. 2. dev-k8s already has ArgoCD installed, idle, zero Applications - more idiomatic to use it than have GitLab Runner execute kubectl directly. Pivots .gitlab-ci.yml: build+push image, then commit the tag into a small per-overlay Component (overlays/{dev,prod}/image-tag/) that ArgoCD's Application watches - CI never touches the cluster, only the registry and this repo. dev's Application (vncmail-dev) is registered and applied already (manual sync for now, until the one-time namespace secret bootstrap is done - see VNCMAIL-SETUP.md). prod's Application is scaffolded in deploy/argocd/ but deliberately not applied - it targets a different cluster (node1-3) that isn't registered with ArgoCD yet, and there's still no real prod hostname/Stalwart/ClusterIssuer. Fixes base/ingress.yaml to the real ingressClassName: traefik (was the nginx-style `public`, which doesn't exist on either cluster) and gives each overlay its own cert-manager issuer patch instead of one hardcoded value, since dev and prod need different (or, for prod, nonexistent) issuers.
2026-08-05 13:06:22 +02:00
parent 3512f935d1
commit 177b2aca57
12 changed files with 344 additions and 140 deletions
@@ -1,51 +1,61 @@
-# GitLab-CI dev→prod pipeline for VNCmail+.
+# GitLab-CI dev→prod pipeline for VNCmail+ — GitOps via ArgoCD.
 #
-# Design (see the approved plan for full rationale):
+# Revised after direct inspection of the real infrastructure found ArgoCD
 # already installed (idle, zero Applications) on the dev-k8s-1/2/3 cluster.
 # That's more idiomatic than a runner-executes-kubectl design, and it means
 # this pipeline needs ZERO cluster credentials — CI only ever talks to the
 # container registry and to this git repo. ArgoCD (which already has
 # whatever cluster access it needs, set up once when its Applications were
 # registered — see deploy/argocd/) is what actually applies anything.
 #
 # Design:
 #   - One image name, environment lives only in the tag. No more -dev/-beta
-#     name confusion.
+#     name confusion from the old GitHub Actions workflow.
 #   - MR into `dev`: verify only (typecheck/lint/unit test/build check). No
 #     push, no deploy — this is the multi-developer merge gate.
-#   - Push to `dev`: build+push an immutable `sha-<sha>` tag, auto-deploy it
+#   - Push to `dev`: build+push an immutable `sha-<sha>` tag, then commit a
-#     to the vncmail (sandbox) namespace. No approval needed — dev always
+#     one-line tag-bump into overlays/dev/image-tag/kustomization.yaml
-#     deploys.
+#     (`[skip ci]`, so this doesn't retrigger itself). ArgoCD's `vncmail-dev`
 #     Application has automated sync — it notices the git change and applies
 #     it. No approval needed, dev always deploys, and this job never touches
 #     the cluster directly.
 #   - Push to `main`: NEVER rebuilds. `main` only ever advances via
 #     `git merge --ff-only dev`, so main's HEAD commit already has a built
-#     image. The `promote` job retags that exact digest (registry-side copy,
+#     image (the same sha- tag dev already deployed). This job just bumps
-#     same primitive the old docker-publish.yml GHA workflow already used for
+#     overlays/prod/image-tag/kustomization.yaml to point at that same tag.
-#     its multi-arch manifest-list merge) and applies it to prod. `when:
+#     The actual promotion gate is a HUMAN clicking Sync on the `vncmail-prod` ArgoCD
-#     manual` + a protected `production` GitLab environment is the approval
+#     Application (deliberately NOT automated sync) — not a GitLab manual
-#     gate — nobody but an authorized user can click it, and nothing here
+#     job, since ArgoCD already provides that exact gate more directly.
-#     runs automatically on main.
+#     Until prod Stalwart/hostname/secrets are real (see VNCMAIL-SETUP.md),
 #     nobody should click that Sync button — but nothing here does it for
 #     you either way.
 #
-# Deliberately single-platform (linux/amd64) for the cluster build — this
+# Deliberately single-platform (linux/amd64) — this pipeline serves two
-# pipeline's job is deploying to a known amd64 microk8s cluster, not public
+# known amd64 microk8s clusters, not public multi-arch distribution (that's
-# multi-arch distribution (that's what the GHCR release workflows are for,
+# what the GHCR release workflows are for, untouched by this file).
 # and they're untouched by this file).
 #
-# Prerequisites this pipeline assumes are already in place (see the plan's
+# Prerequisite this file assumes (documented in VNCMAIL-SETUP.md, not
-# "Split of responsibility" — these are admin/infra actions, not something
+# something this file can set up itself):
-# this file can set up):
+#   - GitLab Container Registry enabled for this project (confirmed done).
-#   - GitLab Container Registry enabled for this project (CI_REGISTRY_* vars
+#   - A GitLab Runner (any kind — no cluster access needed at all now).
-#     are then provided automatically — no manual credential setup needed).
+#   - Either "allow this job token to push to this project" enabled
-#   - A GitLab Runner with the Kubernetes executor, whose deploy-stage jobs
+#     (Settings → CI/CD → Job token permissions), OR a project access token
-#     run as a `gitlab-deployer` ServiceAccount scoped (namespaced Role, not
+#     with `write_repository` scope in $GITLAB_PUSH_TOKEN. The job below
-#     cluster-admin) to the `vncmail` namespace (and later `vncmail-prod`).
+#     tries CI_JOB_TOKEN first (see the script).
 #     kubectl auto-detects in-cluster config from that ServiceAccount's
 #     mounted token — no KUBECONFIG variable required.
 #
-# deploy/k8s/ca/ (the EJBCA internal CA) is never referenced anywhere below —
+# deploy/k8s/ca/ (the EJBCA internal CA) is never referenced anywhere below,
-# that stays a fully manual, human-only runbook (see deploy/k8s/ca/README.md).
+# and neither ArgoCD Application in deploy/argocd/ points at it — that stays
 # a fully manual, human-only runbook (see deploy/k8s/ca/README.md).
 stages:
  - verify
  - build
-  - deploy-dev
+  - bump-dev
-  - promote
+  - bump-prod
 variables:
  IMAGE: $CI_REGISTRY_IMAGE/vncmail-plus
-  DEV_NAMESPACE: vncmail
+  GIT_STRATEGY: clone
  PROD_NAMESPACE: vncmail-prod
 # ---------------------------------------------------------------------------
 # verify — required check on every MR into dev. No registry, no cluster.
@@ -62,9 +72,8 @@ verify:
    - npm run test:translations
    - npm run build
  # test:integration is deliberately NOT here — it spins up a real Stalwart
-  # fixture via docker-compose (Docker-in-Docker), which is heavier than a
+  # fixture via docker-compose (Docker-in-Docker), heavier than a fast MR
-  # fast MR gate should be. Candidate for a separate scheduled/optional job
+  # gate should be. Candidate for a separate scheduled job, not a blocker.
  # later, not a blocker for this pipeline's first cut.
 # ---------------------------------------------------------------------------
 # build — push to dev only. Builds once; main never rebuilds (see header).
@@ -84,61 +93,73 @@ build:
    - docker push "$IMAGE:dev-latest"
 # ---------------------------------------------------------------------------
-# deploy-dev — automatic, no approval. Deploys the immutable sha tag, never
+# bump-dev — no cluster access. Commits the just-built tag into the overlay
-# the moving dev-latest pointer, so what's running always matches one commit.
+# ArgoCD watches; ArgoCD's automated sync does the actual apply.
 # ---------------------------------------------------------------------------
-deploy-dev:
+bump-dev:
-  stage: deploy-dev
+  stage: bump-dev
-  image: bitnami/kubectl:1.31
+  image: alpine/git:2.47.0
  environment:
    name: dev
    url: https://vncmail.sandbox.vnc.de
  rules:
    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == "dev"'
  script:
-    # Apply the manifests first (structure/config), then set the exact image
+    - TAG="sha-$CI_COMMIT_SHORT_SHA"
-    # this pipeline just built — imperative `set image`, not a kustomize-file
+    - |
-    # edit, so overlays/dev never needs a commit to change what's deployed.
+      cat > deploy/k8s/overlays/dev/image-tag/kustomization.yaml <<EOF
-    - kubectl apply -k deploy/k8s/overlays/dev
+      # Owned by CI (bump-dev job in .gitlab-ci.yml) - regenerated every
-    - kubectl -n $DEV_NAMESPACE set image deployment/vncmail-plus vncmail-plus="$IMAGE:sha-$CI_COMMIT_SHORT_SHA"
+      # push to dev. Do not hand-edit; edits here get overwritten.
-    - kubectl -n $DEV_NAMESPACE rollout status deploy/vncmail-plus --timeout=120s
+      apiVersion: kustomize.config.k8s.io/v1alpha1
      kind: Component
      images:
        - name: ghcr.io/brvncde-dotcom/vncmail-plus-dev
          newName: $IMAGE
          newTag: $TAG
      EOF
    - git config user.name "vncmail-ci"
    - git config user.email "ci@vnc.biz"
    - git add deploy/k8s/overlays/dev/image-tag/kustomization.yaml
    - |
      if git diff --cached --quiet; then
        echo "No change (tag already pinned) - nothing to commit"
      else
        git commit -m "chore(deploy): pin dev to $TAG [skip ci]"
        git push "https://gitlab-ci-token:${GITLAB_PUSH_TOKEN:-$CI_JOB_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git" HEAD:dev
      fi
 # ---------------------------------------------------------------------------
-# promote — manual, protected `production` environment. No docker build here
+# bump-prod — no cluster access, no rebuild. Points overlays/prod at the
-# — retags the exact digest already deployed to dev, then applies prod
+# exact tag already running on dev. Does NOT deploy anything: vncmail-prod's
-# pinned to that digest (never a mutable tag).
+# ArgoCD Application has manual sync, so this only prepares what a human
 # would be syncing, it doesn't sync it.
 # ---------------------------------------------------------------------------
-promote:
+bump-prod:
-  stage: promote
+  stage: bump-prod
-  image: docker:27-cli
+  image: alpine/git:2.47.0
  services:
    - docker:27-dind
  environment:
    name: production
    url: https://vncmail.CHANGEME.invalid # placeholder until the real prod host is decided
  rules:
    # `when: manual` lives inside the rule (not as a top-level job key) —
    # required syntax once `rules:` is used at all.
    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == "main"'
      when: manual
  before_script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" "$CI_REGISTRY" --password-stdin
  script:
-    - echo "Retagging the image already built+deployed for dev commit $CI_COMMIT_SHA — no rebuild."
+    - TAG="sha-$CI_COMMIT_SHORT_SHA"
-    - docker buildx imagetools create --tag "$IMAGE:prod-latest" "$IMAGE:sha-$CI_COMMIT_SHORT_SHA"
+    - echo "main advanced to $CI_COMMIT_SHA (must be a dev commit, ff-only) - that image already exists as $IMAGE:$TAG"
-    - DIGEST=$(docker buildx imagetools inspect "$IMAGE:sha-$CI_COMMIT_SHORT_SHA" | awk '/^Digest:/{print $2}')
+    - |
-    - echo "Resolved digest for prod = $IMAGE@$DIGEST"
+      cat > deploy/k8s/overlays/prod/image-tag/kustomization.yaml <<EOF
-    - >
+      # Owned by CI (bump-prod job in .gitlab-ci.yml) - regenerated every
-      echo "STOPPING HERE ON PURPOSE: deploy/k8s/overlays/prod is still
+      # push to main. Do not hand-edit; edits here get overwritten. Bumping
-      scaffolded/inactive (placeholder hostname, placeholder JMAP_SERVER_URL
+      # this is NOT the same as deploying it - vncmail-prod's ArgoCD
-      — no prod Stalwart exists yet). Once both are real (Phase D in the
+      # Application has manual sync, see the note in the parent
-      pipeline plan / VNCMAIL-SETUP.md), replace this echo with the same
+      # kustomization.yaml.
-      pattern deploy-dev uses, against a bitnami/kubectl image and
+      apiVersion: kustomize.config.k8s.io/v1alpha1
-      \$PROD_NAMESPACE: kubectl apply -k deploy/k8s/overlays/prod &&
+      kind: Component
-      kubectl -n \$PROD_NAMESPACE set image deployment/vncmail-plus
+      images:
-      vncmail-plus=$IMAGE@$DIGEST"
+        - name: ghcr.io/brvncde-dotcom/vncmail-plus-dev
-  # Deliberately does NOT run `kubectl apply -k overlays/prod` yet — prod
+          newName: $IMAGE
-  # namespace/hostname/Stalwart don't exist (Phase C/D in the plan). Once
+          newTag: $TAG
-  # they do, replace the placeholder echo above with the same
+      EOF
-  # `kubectl apply -k .` + `set image ...@$DIGEST` pattern deploy-dev uses,
+    - git config user.name "vncmail-ci"
-  # against $PROD_NAMESPACE, using the bitnami/kubectl image.
+    - git config user.email "ci@vnc.biz"
    - git add deploy/k8s/overlays/prod/image-tag/kustomization.yaml
    - |
      if git diff --cached --quiet; then
        echo "No change (tag already pinned) - nothing to commit"
      else
        git commit -m "chore(deploy): point prod overlay at $TAG (not synced - manual gate in ArgoCD) [skip ci]"
        git push "https://gitlab-ci-token:${GITLAB_PUSH_TOKEN:-$CI_JOB_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git" HEAD:main
      fi
@@ -33,37 +33,89 @@ of truth; VNCmail+ is the UI. It deploys as a **container on Kubernetes
 All VNC customization lives under `vnc/` (see `vnc/VNC-CHANGES.md`).
-## CI/CD — GitLab (canonical), Vercel-style dev→prod
+## CI/CD — GitLab (canonical) + ArgoCD GitOps, Vercel-style dev→prod
-Multiple developers work on this repo now, so `.gitlab-ci.yml` on
+Multiple developers work on this repo now. `.gitlab-ci.yml` on
 [gitlab.vnc.biz](https://gitlab.vnc.biz/gitlab-instance-b9b5cf2f/vncmail-plus)
 (the canonical remote — GitHub `origin` is a passive mirror, not where CI or
-deploys happen) drives the whole flow:
+deploys happen) builds images and bumps a tag in git; **ArgoCD does the
 actual deploying** — already installed and idle on the `dev-k8s-1/2/3`
 cluster, discovered when standing this up. GitLab CI needs zero cluster
 credentials as a result.
-1. **MR into `dev`** → `verify` stage runs (typecheck/lint/unit test/build).
+Two real clusters, confirmed by direct inspection:
-   Required check — no push, no deploy. This is the multi-developer gate.
+
 | Cluster | Role | Notes |
 |---|---|---|
 | `dev-k8s-1/2/3` | dev/sandbox | ~hours old when set up here. Traefik, metallb, cert-manager (`letsencrypt-staging` issuer only), **ArgoCD already running**. |
 | `node1/node2/node3` | prod (HA) | Older, rook-ceph+traefik+metallb+cert-manager, but **zero apps and zero ClusterIssuers** — genuinely a clean slate. |
 Neither cluster had a `vncmail` namespace, `vnc-ca` namespace, or `bulwark`
 ingress — the "live sandbox at vncmail.sandbox.vnc.de" referenced earlier in
 this doc's history was aspirational (manifests + docs existed, nothing was
 ever actually applied). The ingress manifests also assumed nginx (`class:
 public`, an nginx body-size annotation) — fixed to Traefik's real
 `ingressClassName: traefik` (Traefik has no default body-size cap, so no
 replacement annotation is needed).
 Flow:
 1. **MR into `dev`** → `verify` stage (typecheck/lint/unit test/build).
   Required check — no push, no deploy.
 2. **Merge to `dev`** → `build` pushes one image,
-   `registry.gitlab.vnc.biz/.../vncmail-plus:sha-<sha>`, then `deploy-dev`
+   `registry.gitlab.vnc.biz/.../vncmail-plus:sha-<sha>`, then `bump-dev`
-   applies it to the sandbox automatically. No approval needed — dev always
+   commits that tag into `deploy/k8s/overlays/dev/image-tag/kustomization.yaml`
-   deploys first.
+   (`[skip ci]`). ArgoCD's `vncmail-dev` Application picks up the git change.
-3. **Merge to `main`** (fast-forward only, see below) → a `promote` job
+3. **Merge to `main`** (fast-forward only, see below) → `bump-prod` points
-   appears, `when: manual`, gated behind a protected `production`
+   `overlays/prod/image-tag/` at that same tag — **no rebuild**. The actual
-   GitLab environment. It **never rebuilds** — it retags the exact image
+   promotion gate is a **human clicking Sync** on the `vncmail-prod` ArgoCD
-   already running on dev (registry-side copy, same digest) and would apply
+   Application, which is permanently manual-sync (never automated) — that's
-   it to a `vncmail-prod` namespace pinned to that digest.
+   the Vercel-style "Promote to Production" button, just living in ArgoCD's
   UI instead of GitLab's.
 ### What's left to wire up (one-time, human steps)
 1. **Add the ArgoCD deploy key to GitLab** — Project → Settings → Repository
   → Deploy keys → add (read-only is enough):
   ```
   ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOURjX/Y9zfB785DyLEF1GUq4HhWujrqeXag8oxdMciq argocd@dev-k8s (vncmail-plus read-only)
   ```
   Until this is added, `vncmail-dev`'s ArgoCD Application (already created,
   `kubectl -n argocd get application vncmail-dev`) shows a benign
   `ComparisonError` (SSH handshake failing) — expected, not a bug.
 2. **Let CI push tag-bumps back to this repo** — either enable "this project
   can be accessed by CI/CD job tokens from other projects" → actually
   simpler: Settings → CI/CD → Job token permissions → allow this project's
   own job token to push to itself, OR create a Project Access Token
   (`write_repository` scope) and add it as a masked CI/CD variable
   `GITLAB_PUSH_TOKEN` (the pipeline tries that first, falls back to
   `CI_JOB_TOKEN`).
 3. **One-time namespace bootstrap** (CI/ArgoCD deliberately never manage
   secret contents — see `deploy/k8s/README.md` §3):
   ```bash
   # against dev-k8s (ArgoCD's CreateNamespace=true will make `vncmail` on
   # first sync, or create it yourself first — either order works)
   kubectl create secret docker-registry ghcr-pull -n vncmail ...   # or make the GHCR package public
   cp deploy/k8s/overlays/dev/secret.example.yaml secret.yaml   # edit SESSION_SECRET
   kubectl apply -f secret.yaml
   ```
 4. **First sync** — ArgoCD UI at `https://argo.devcluster.vnc.de`
   (username `admin`, password: `kubectl -n argocd get secret
   argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d`
   — rotate it after logging in once) → `vncmail-dev` → Sync. Once that's
   clean, flip `deploy/argocd/vncmail-dev-app.yaml`'s commented-out
   `automated:` block on and re-apply, so dev auto-syncs on every push from
   then on.
 5. **Production** (later, deliberately not wired yet): decide a real
   hostname, stand up prod Stalwart, register `node1-3` as an ArgoCD-managed
   cluster, apply `deploy/argocd/vncmail-prod-app.yaml`, fill in real
   `overlays/prod` values, create a real ClusterIssuer on `node1-3` (there
   isn't one today), then click Sync once — deliberately not before.
 Historical note: the old `-dev`/`-beta` GHCR image-name split
 (`.github/workflows/docker-publish.yml`) is retired by this — one image name
 now, environment lives only in the tag.
 **Production doesn't exist yet.** `deploy/k8s/overlays/prod/` is scaffolded
 (placeholder hostname, placeholder `JMAP_SERVER_URL` — there's no prod
 Stalwart instance to point it at either) but inert: the `promote` job's real
 `kubectl apply` step is deliberately left as a TODO in `.gitlab-ci.yml` until
 a real hostname is decided and prod Stalwart exists. Standing up the runner/
 RBAC/registry this pipeline needs is an infra prerequisite, not something CI
 itself does — see the pipeline design doc referenced in `deploy/k8s/README.md`.
 ## Deploy (Kubernetes / microk8s)
 Full runbook: **[deploy/k8s/README.md](deploy/k8s/README.md)**. In short:
@@ -0,0 +1,29 @@
 # Registered on the dev-k8s-1/2/3 cluster (where ArgoCD already lives) via
 # `kubectl apply` directly to the argocd namespace — this file is the
 # version-controlled record of that, not something ArgoCD itself syncs
 # (no app-of-apps here, deliberately kept simple for two Applications).
 #
 # syncPolicy starts WITHOUT automated — manual sync until the one-time
 # per-namespace bootstrap (vncmail-env secret, image-pull secret — see
 # deploy/k8s/README.md §3) is done by hand once. Flip to automated (see
 # commented block below) only after a first manual sync succeeds cleanly.
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: vncmail-dev
  namespace: argocd
 spec:
  project: default
  source:
    repoURL: git@gitlab.vnc.biz:gitlab-instance-b9b5cf2f/vncmail-plus.git
    targetRevision: dev
    path: deploy/k8s/overlays/dev
  destination:
    server: https://kubernetes.default.svc # in-cluster — ArgoCD and vncmail-dev share this cluster
    namespace: vncmail
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
    # automated:
    #   prune: true
    #   selfHeal: true
@@ -0,0 +1,30 @@
 # NOT YET APPLIED to any cluster. Scaffolding only, matching
 # deploy/k8s/overlays/prod's own "inert until Phase D" status.
 #
 # Unlike vncmail-dev-app.yaml, this targets a DIFFERENT cluster (node1-3,
 # the HA "prod" cluster) than the one ArgoCD itself runs on (dev-k8s).
 # That means before this can be applied, node1-3 needs to be registered as
 # an ArgoCD-managed cluster (`argocd cluster add`, or an equivalent
 # ServiceAccount+kubeconfig secret) — deliberately not done yet: there's no
 # reason to wire cross-cluster RBAC into the prod HA cluster before prod
 # hostname/Stalwart/secrets are real and someone's actually promoting.
 #
 # syncPolicy has no automated block at all, and won't get one even later —
 # prod stays manual-sync-only permanently. That's the promotion gate.
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: vncmail-prod
  namespace: argocd
 spec:
  project: default
  source:
    repoURL: git@gitlab.vnc.biz:gitlab-instance-b9b5cf2f/vncmail-plus.git
    targetRevision: main
    path: deploy/k8s/overlays/prod
  destination:
    server: CHANGEME # the node1-3 cluster's registered ArgoCD server URL, once added
    namespace: vncmail-prod
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
@@ -22,13 +22,15 @@ deploy/k8s/
 applying `base/` directly — `base/` alone has no namespace and won't apply
 meaningfully on its own.
-## Routine deploys go through CI now
+## Routine deploys go through CI + ArgoCD now
 As of the GitLab CI/CD pipeline (`.gitlab-ci.yml`, see `../../VNCMAIL-SETUP.md`
-§ CI/CD), **pushing to `dev` auto-builds and auto-deploys** — you should not
+§ CI/CD), **pushing to `dev` auto-builds and bumps the deploy tag; ArgoCD's
-normally need to run `kubectl apply` for the sandbox by hand anymore. This
+`vncmail-dev` Application applies it** — you should not normally need to run
-guide's manual steps below are for first-time setup, the one-time secret
+`kubectl apply` for the sandbox by hand anymore, and CI never touches the
-creation CI deliberately never automates, and troubleshooting.
+cluster directly (it only ever talks to the registry and to this git repo).
 This guide's manual steps below are for first-time setup, the one-time
 secret creation CI/ArgoCD deliberately never automate, and troubleshooting.
 ## Production status
@@ -144,15 +146,23 @@ a bare username.
 ## 5. Update to a new build
-Normally you don't — CI's `deploy-dev` job does this automatically on every
+Normally you don't — CI's `bump-dev` job + ArgoCD's automated sync do this
-push to `dev`. To do it by hand (e.g. troubleshooting):
+on every push to `dev`. To do it by hand (e.g. troubleshooting, before
 automated sync is turned on):
 ```bash
 kubectl -n vncmail set image deploy/vncmail-plus \
  vncmail-plus=registry.gitlab.vnc.biz/gitlab-instance-b9b5cf2f/vncmail-plus:sha-<sha>
 ```
-Rollback: `kubectl -n vncmail rollout undo deploy/vncmail-plus`
+ArgoCD will overwrite this on its next sync unless you also update
 `deploy/k8s/overlays/dev/image-tag/kustomization.yaml` to match — that file
 is CI-owned (see its header comment), so a by-hand `set image` is only ever
 a temporary override, not a real fix.
 Rollback (bypassing ArgoCD temporarily): `kubectl -n vncmail rollout undo deploy/vncmail-plus`.
 The real rollback is reverting the commit that bumped the tag and letting
 ArgoCD re-sync.
 ---
@@ -1,27 +1,28 @@
-# Exposes VNCmail+ at vncmail.sandbox.vnc.de, alongside bulwark.sandbox.vnc.de.
+# Both real clusters (node1-3 "prod", dev-k8s-1-3 "dev") run Traefik, not
-# MATCH YOUR CLUSTER — inspect the existing Bulwark ingress and copy its
+# nginx — confirmed via `kubectl get ingressclass` (class is literally named
-# ingressClassName + TLS/cert-manager annotations:
+# `traefik`). Unlike nginx's restrictive 1MB default, Traefik has no default
-#   kubectl get ingress -A | grep bulwark
+# request-body-size cap, so there's no equivalent needed for mail attachment
-#   kubectl get ingress <bulwark-ingress> -n <ns> -o yaml
+# uploads (the old nginx.ingress.kubernetes.io/proxy-body-size annotation
 # this file used to carry is simply not applicable here).
 #
 # Host, TLS secretName, and cert-manager issuer are ALL overlay-specific now
 # (dev-k8s only has a `letsencrypt-staging` issuer; node1-3/prod has none
 # configured yet) — every overlay's patch-ingress.yaml must override the
 # CHANGEME placeholders below.
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
  name: vncmail-plus
  annotations:
-    # cert-manager issuer — set to whatever bulwark.sandbox.vnc.de uses.
+    cert-manager.io/cluster-issuer: CHANGEME
    cert-manager.io/cluster-issuer: letsencrypt-prod
    # Mail attachments can be large; raise the nginx body limit.
    nginx.ingress.kubernetes.io/proxy-body-size: "100m"
 spec:
-  # microk8s ingress addon class is usually "public" (nginx). Confirm with
+  ingressClassName: traefik
  # `kubectl get ingressclass` and match bulwark's.
  ingressClassName: public
  tls:
    - hosts:
-        - vncmail.sandbox.vnc.de
+        - CHANGEME.invalid
      secretName: vncmail-plus-tls
  rules:
-    - host: vncmail.sandbox.vnc.de
+    - host: CHANGEME.invalid
      http:
        paths:
          - path: /
@@ -0,0 +1,11 @@
 # Owned by CI (the bump-dev job in .gitlab-ci.yml), not by hand. Kept as its
 # own Component so CI only ever rewrites this 6-line file, never the parent
 # overlays/dev/kustomization.yaml (structure/patches there stay under normal
 # code review — CI regenerating a whole hand-maintained file on every push
 # would silently revert any change made there between deploys).
 apiVersion: kustomize.config.k8s.io/v1alpha1
 kind: Component
 images:
  - name: ghcr.io/brvncde-dotcom/vncmail-plus-dev
    newName: ghcr.io/brvncde-dotcom/vncmail-plus-dev
    newTag: latest
@@ -6,8 +6,13 @@ resources:
  - namespace.yaml
  # - secret.yaml   # create from secret.example.yaml; not committed
-# This is the live sandbox (vncmail.sandbox.vnc.de) — deliberately zero patches
+patches:
-# beyond namespace/resource wiring, so `kubectl kustomize .` renders identical
+  - path: patch-ingress.yaml
-# to the pre-restructure flat deploy/k8s/. The image is left at base's default
+
-# and overridden per-deploy by CI (`kubectl set image`, see .gitlab-ci.yml's
+components:
-# deploy-dev job) rather than pinned here, so this file never goes stale.
+  - image-tag
 # Targets the dev-k8s-1/2/3 cluster (confirmed via direct access: this is
 # where ArgoCD already lives). The image tag lives in image-tag/ (a separate
 # Component CI owns — see .gitlab-ci.yml's bump-dev job) rather than here, so
 # CI never needs to touch this file.
@@ -0,0 +1,27 @@
 # dev-k8s cluster confirmed to have a `letsencrypt-staging` ClusterIssuer
 # already (no `letsencrypt-prod` exists there) - staging avoids burning
 # Let's Encrypt's real rate limits while this is still being stood up.
 # vncmail.sandbox.vnc.de DNS does not point here yet either - this is the
 # intended host, not a live one (see VNCMAIL-SETUP.md for what's still open).
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
  name: vncmail-plus
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-staging
 spec:
  tls:
    - hosts:
        - vncmail.sandbox.vnc.de
      secretName: vncmail-plus-tls
  rules:
    - host: vncmail.sandbox.vnc.de
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: vncmail-plus
                port:
                  number: 80
@@ -0,0 +1,12 @@
 # Owned by CI (the bump-prod job in .gitlab-ci.yml), not by hand — same
 # reasoning as overlays/dev/image-tag/. Starts pointed at an obviously-fake
 # tag on purpose: nothing has been promoted yet, and vncmail-prod's ArgoCD
 # Application has manual sync anyway, so this being "wrong" doesn't deploy
 # anything wrong — it just means there's nothing to sync until a real
 # `git push` to main updates it.
 apiVersion: kustomize.config.k8s.io/v1alpha1
 kind: Component
 images:
  - name: ghcr.io/brvncde-dotcom/vncmail-plus-dev
    newName: registry.gitlab.vnc.biz/gitlab-instance-b9b5cf2f/vncmail-plus
    newTag: not-yet-promoted
@@ -10,12 +10,12 @@ patches:
  - path: patch-ingress.yaml
  - path: patch-deployment.yaml
-# NOT MEANT TO BE APPLIED AS COMMITTED. Scaffolding only (see the pipeline
+components:
-# plan's Phase C/D) — the tag below is an obviously-invalid placeholder;
+  - image-tag
-# the real promote job (.gitlab-ci.yml) resolves and pins an actual digest at
+
-# deploy time via `kubectl set image`, it never trusts whatever is checked
+# NOT MEANT TO BE SYNCED AS COMMITTED. Scaffolding only (see the pipeline
-# in here.
+# plan's Phase C/D) — image-tag/'s placeholder tag is obviously-invalid on
-images:
+# purpose. The bump-prod job in .gitlab-ci.yml keeps that tag pointed at
-  - name: ghcr.io/brvncde-dotcom/vncmail-plus-dev
+# whatever's already on dev once main advances, but vncmail-prod's ArgoCD
-    newName: registry.gitlab.vnc.biz/gitlab-instance-b9b5cf2f/vncmail-plus
+# Application has manual sync — a human still has to click Sync (or
-    newTag: not-yet-promoted
+# `argocd app sync vncmail-prod`) for any of this to actually apply.
@@ -3,6 +3,12 @@
 # deliberately unresolvable: applying this overlay as committed will not
 # issue a cert or route traffic anywhere. Replace both occurrences below,
 # and the matching TLS secretName, before Phase D (first real prod deploy).
 #
 # Targets node1-3 (the HA "prod" cluster). Deliberately does NOT override
 # base's `cert-manager.io/cluster-issuer: CHANGEME` — node1-3 has ZERO
 # ClusterIssuers configured today (confirmed via direct access). A human
 # needs to create a real one there (ACME account, DNS-01 or HTTP-01 solver)
 # before this can be anything but a placeholder.
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata: