feat(ci): pivot to ArgoCD GitOps, fix Traefik ingress after real-cluster check

Direct SSH access to the actual clusters (node1-3 "prod" HA, dev-k8s-1-3 "dev") revealed two things that made the previous design wrong: 1. Neither cluster has vncmail/vnc-ca namespaces or a bulwark ingress at all - the "live sandbox" referenced in this repo's docs/manifests was never actually applied anywhere. Both ingress.yaml's ingressClassName (public) and cert-manager issuer (letsencrypt-prod) were also wrong: both clusters run Traefik (class is literally named `traefik`), and only dev-k8s has any ClusterIssuer at all (`letsencrypt-staging`). node1-3 has zero ClusterIssuers configured. 2. dev-k8s already has ArgoCD installed, idle, zero Applications - more idiomatic to use it than have GitLab Runner execute kubectl directly. Pivots .gitlab-ci.yml: build+push image, then commit the tag into a small per-overlay Component (overlays/{dev,prod}/image-tag/) that ArgoCD's Application watches - CI never touches the cluster, only the registry and this repo. dev's Application (vncmail-dev) is registered and applied already (manual sync for now, until the one-time namespace secret bootstrap is done - see VNCMAIL-SETUP.md). prod's Application is scaffolded in deploy/argocd/ but deliberately not applied - it targets a different cluster (node1-3) that isn't registered with ArgoCD yet, and there's still no real prod hostname/Stalwart/ClusterIssuer. Fixes base/ingress.yaml to the real ingressClassName: traefik (was the nginx-style `public`, which doesn't exist on either cluster) and gives each overlay its own cert-manager issuer patch instead of one hardcoded value, since dev and prod need different (or, for prod, nonexistent) issuers.
2026-08-05 13:06:22 +02:00
parent 3512f935d1
commit 177b2aca57
12 changed files with 344 additions and 140 deletions
@@ -0,0 +1,29 @@
+# Registered on the dev-k8s-1/2/3 cluster (where ArgoCD already lives) via
+# `kubectl apply` directly to the argocd namespace — this file is the
+# version-controlled record of that, not something ArgoCD itself syncs
+# (no app-of-apps here, deliberately kept simple for two Applications).
+#
+# syncPolicy starts WITHOUT automated — manual sync until the one-time
+# per-namespace bootstrap (vncmail-env secret, image-pull secret — see
+# deploy/k8s/README.md §3) is done by hand once. Flip to automated (see
+# commented block below) only after a first manual sync succeeds cleanly.
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: vncmail-dev
+  namespace: argocd
+spec:
+  project: default
+  source:
+    repoURL: git@gitlab.vnc.biz:gitlab-instance-b9b5cf2f/vncmail-plus.git
+    targetRevision: dev
+    path: deploy/k8s/overlays/dev
+  destination:
+    server: https://kubernetes.default.svc # in-cluster — ArgoCD and vncmail-dev share this cluster
+    namespace: vncmail
+  syncPolicy:
+    syncOptions:
+      - CreateNamespace=true
+    # automated:
+    #   prune: true
+    #   selfHeal: true
@@ -0,0 +1,30 @@
+# NOT YET APPLIED to any cluster. Scaffolding only, matching
+# deploy/k8s/overlays/prod's own "inert until Phase D" status.
+#
+# Unlike vncmail-dev-app.yaml, this targets a DIFFERENT cluster (node1-3,
+# the HA "prod" cluster) than the one ArgoCD itself runs on (dev-k8s).
+# That means before this can be applied, node1-3 needs to be registered as
+# an ArgoCD-managed cluster (`argocd cluster add`, or an equivalent
+# ServiceAccount+kubeconfig secret) — deliberately not done yet: there's no
+# reason to wire cross-cluster RBAC into the prod HA cluster before prod
+# hostname/Stalwart/secrets are real and someone's actually promoting.
+#
+# syncPolicy has no automated block at all, and won't get one even later —
+# prod stays manual-sync-only permanently. That's the promotion gate.
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: vncmail-prod
+  namespace: argocd
+spec:
+  project: default
+  source:
+    repoURL: git@gitlab.vnc.biz:gitlab-instance-b9b5cf2f/vncmail-plus.git
+    targetRevision: main
+    path: deploy/k8s/overlays/prod
+  destination:
+    server: CHANGEME # the node1-3 cluster's registered ArgoCD server URL, once added
+    namespace: vncmail-prod
+  syncPolicy:
+    syncOptions:
+      - CreateNamespace=true
@@ -22,13 +22,15 @@ deploy/k8s/
 applying `base/` directly — `base/` alone has no namespace and won't apply
 meaningfully on its own.

-## Routine deploys go through CI now
+## Routine deploys go through CI + ArgoCD now

 As of the GitLab CI/CD pipeline (`.gitlab-ci.yml`, see `../../VNCMAIL-SETUP.md`
-§ CI/CD), **pushing to `dev` auto-builds and auto-deploys** — you should not
-normally need to run `kubectl apply` for the sandbox by hand anymore. This
-guide's manual steps below are for first-time setup, the one-time secret
-creation CI deliberately never automates, and troubleshooting.
+§ CI/CD), **pushing to `dev` auto-builds and bumps the deploy tag; ArgoCD's
+`vncmail-dev` Application applies it** — you should not normally need to run
+`kubectl apply` for the sandbox by hand anymore, and CI never touches the
+cluster directly (it only ever talks to the registry and to this git repo).
+This guide's manual steps below are for first-time setup, the one-time
+secret creation CI/ArgoCD deliberately never automate, and troubleshooting.

 ## Production status

@@ -144,15 +146,23 @@ a bare username.

 ## 5. Update to a new build

-Normally you don't — CI's `deploy-dev` job does this automatically on every
-push to `dev`. To do it by hand (e.g. troubleshooting):
+Normally you don't — CI's `bump-dev` job + ArgoCD's automated sync do this
+on every push to `dev`. To do it by hand (e.g. troubleshooting, before
+automated sync is turned on):

 ```bash
 kubectl -n vncmail set image deploy/vncmail-plus \
  vncmail-plus=registry.gitlab.vnc.biz/gitlab-instance-b9b5cf2f/vncmail-plus:sha-<sha>
 ```

-Rollback: `kubectl -n vncmail rollout undo deploy/vncmail-plus`
+ArgoCD will overwrite this on its next sync unless you also update
+`deploy/k8s/overlays/dev/image-tag/kustomization.yaml` to match — that file
+is CI-owned (see its header comment), so a by-hand `set image` is only ever
+a temporary override, not a real fix.
+
+Rollback (bypassing ArgoCD temporarily): `kubectl -n vncmail rollout undo deploy/vncmail-plus`.
+The real rollback is reverting the commit that bumped the tag and letting
+ArgoCD re-sync.

 ---

@@ -1,27 +1,28 @@
-# Exposes VNCmail+ at vncmail.sandbox.vnc.de, alongside bulwark.sandbox.vnc.de.
-# MATCH YOUR CLUSTER — inspect the existing Bulwark ingress and copy its
-# ingressClassName + TLS/cert-manager annotations:
-#   kubectl get ingress -A | grep bulwark
-#   kubectl get ingress <bulwark-ingress> -n <ns> -o yaml
+# Both real clusters (node1-3 "prod", dev-k8s-1-3 "dev") run Traefik, not
+# nginx — confirmed via `kubectl get ingressclass` (class is literally named
+# `traefik`). Unlike nginx's restrictive 1MB default, Traefik has no default
+# request-body-size cap, so there's no equivalent needed for mail attachment
+# uploads (the old nginx.ingress.kubernetes.io/proxy-body-size annotation
+# this file used to carry is simply not applicable here).
+#
+# Host, TLS secretName, and cert-manager issuer are ALL overlay-specific now
+# (dev-k8s only has a `letsencrypt-staging` issuer; node1-3/prod has none
+# configured yet) — every overlay's patch-ingress.yaml must override the
+# CHANGEME placeholders below.
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
  name: vncmail-plus
  annotations:
-    # cert-manager issuer — set to whatever bulwark.sandbox.vnc.de uses.
-    cert-manager.io/cluster-issuer: letsencrypt-prod
-    # Mail attachments can be large; raise the nginx body limit.
-    nginx.ingress.kubernetes.io/proxy-body-size: "100m"
+    cert-manager.io/cluster-issuer: CHANGEME
 spec:
-  # microk8s ingress addon class is usually "public" (nginx). Confirm with
-  # `kubectl get ingressclass` and match bulwark's.
-  ingressClassName: public
+  ingressClassName: traefik
  tls:
    - hosts:
-        - vncmail.sandbox.vnc.de
+        - CHANGEME.invalid
      secretName: vncmail-plus-tls
  rules:
-    - host: vncmail.sandbox.vnc.de
+    - host: CHANGEME.invalid
      http:
        paths:
          - path: /
@@ -0,0 +1,11 @@
+# Owned by CI (the bump-dev job in .gitlab-ci.yml), not by hand. Kept as its
+# own Component so CI only ever rewrites this 6-line file, never the parent
+# overlays/dev/kustomization.yaml (structure/patches there stay under normal
+# code review — CI regenerating a whole hand-maintained file on every push
+# would silently revert any change made there between deploys).
+apiVersion: kustomize.config.k8s.io/v1alpha1
+kind: Component
+images:
+  - name: ghcr.io/brvncde-dotcom/vncmail-plus-dev
+    newName: ghcr.io/brvncde-dotcom/vncmail-plus-dev
+    newTag: latest
@@ -6,8 +6,13 @@ resources:
  - namespace.yaml
  # - secret.yaml   # create from secret.example.yaml; not committed

-# This is the live sandbox (vncmail.sandbox.vnc.de) — deliberately zero patches
-# beyond namespace/resource wiring, so `kubectl kustomize .` renders identical
-# to the pre-restructure flat deploy/k8s/. The image is left at base's default
-# and overridden per-deploy by CI (`kubectl set image`, see .gitlab-ci.yml's
-# deploy-dev job) rather than pinned here, so this file never goes stale.
+patches:
+  - path: patch-ingress.yaml
+
+components:
+  - image-tag
+
+# Targets the dev-k8s-1/2/3 cluster (confirmed via direct access: this is
+# where ArgoCD already lives). The image tag lives in image-tag/ (a separate
+# Component CI owns — see .gitlab-ci.yml's bump-dev job) rather than here, so
+# CI never needs to touch this file.
@@ -0,0 +1,27 @@
+# dev-k8s cluster confirmed to have a `letsencrypt-staging` ClusterIssuer
+# already (no `letsencrypt-prod` exists there) - staging avoids burning
+# Let's Encrypt's real rate limits while this is still being stood up.
+# vncmail.sandbox.vnc.de DNS does not point here yet either - this is the
+# intended host, not a live one (see VNCMAIL-SETUP.md for what's still open).
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: vncmail-plus
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-staging
+spec:
+  tls:
+    - hosts:
+        - vncmail.sandbox.vnc.de
+      secretName: vncmail-plus-tls
+  rules:
+    - host: vncmail.sandbox.vnc.de
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: vncmail-plus
+                port:
+                  number: 80
@@ -0,0 +1,12 @@
+# Owned by CI (the bump-prod job in .gitlab-ci.yml), not by hand — same
+# reasoning as overlays/dev/image-tag/. Starts pointed at an obviously-fake
+# tag on purpose: nothing has been promoted yet, and vncmail-prod's ArgoCD
+# Application has manual sync anyway, so this being "wrong" doesn't deploy
+# anything wrong — it just means there's nothing to sync until a real
+# `git push` to main updates it.
+apiVersion: kustomize.config.k8s.io/v1alpha1
+kind: Component
+images:
+  - name: ghcr.io/brvncde-dotcom/vncmail-plus-dev
+    newName: registry.gitlab.vnc.biz/gitlab-instance-b9b5cf2f/vncmail-plus
+    newTag: not-yet-promoted
@@ -10,12 +10,12 @@ patches:
  - path: patch-ingress.yaml
  - path: patch-deployment.yaml

-# NOT MEANT TO BE APPLIED AS COMMITTED. Scaffolding only (see the pipeline
-# plan's Phase C/D) — the tag below is an obviously-invalid placeholder;
-# the real promote job (.gitlab-ci.yml) resolves and pins an actual digest at
-# deploy time via `kubectl set image`, it never trusts whatever is checked
-# in here.
-images:
-  - name: ghcr.io/brvncde-dotcom/vncmail-plus-dev
-    newName: registry.gitlab.vnc.biz/gitlab-instance-b9b5cf2f/vncmail-plus
-    newTag: not-yet-promoted
+components:
+  - image-tag
+
+# NOT MEANT TO BE SYNCED AS COMMITTED. Scaffolding only (see the pipeline
+# plan's Phase C/D) — image-tag/'s placeholder tag is obviously-invalid on
+# purpose. The bump-prod job in .gitlab-ci.yml keeps that tag pointed at
+# whatever's already on dev once main advances, but vncmail-prod's ArgoCD
+# Application has manual sync — a human still has to click Sync (or
+# `argocd app sync vncmail-prod`) for any of this to actually apply.
@@ -3,6 +3,12 @@
 # deliberately unresolvable: applying this overlay as committed will not
 # issue a cert or route traffic anywhere. Replace both occurrences below,
 # and the matching TLS secretName, before Phase D (first real prod deploy).
+#
+# Targets node1-3 (the HA "prod" cluster). Deliberately does NOT override
+# base's `cert-manager.io/cluster-issuer: CHANGEME` — node1-3 has ZERO
+# ClusterIssuers configured today (confirmed via direct access). A human
+# needs to create a real one there (ACME account, DNS-01 or HTTP-01 solver)
+# before this can be anything but a placeholder.
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata: