feat(ci): pivot to ArgoCD GitOps, fix Traefik ingress after real-cluster check
Direct SSH access to the actual clusters (node1-3 "prod" HA, dev-k8s-1-3
"dev") revealed two things that made the previous design wrong:
1. Neither cluster has vncmail/vnc-ca namespaces or a bulwark ingress at
all - the "live sandbox" referenced in this repo's docs/manifests was
never actually applied anywhere. Both ingress.yaml's ingressClassName
(public) and cert-manager issuer (letsencrypt-prod) were also wrong:
both clusters run Traefik (class is literally named `traefik`), and
only dev-k8s has any ClusterIssuer at all (`letsencrypt-staging`).
node1-3 has zero ClusterIssuers configured.
2. dev-k8s already has ArgoCD installed, idle, zero Applications - more
idiomatic to use it than have GitLab Runner execute kubectl directly.
Pivots .gitlab-ci.yml: build+push image, then commit the tag into a small
per-overlay Component (overlays/{dev,prod}/image-tag/) that ArgoCD's
Application watches - CI never touches the cluster, only the registry and
this repo. dev's Application (vncmail-dev) is registered and applied
already (manual sync for now, until the one-time namespace secret
bootstrap is done - see VNCMAIL-SETUP.md). prod's Application is
scaffolded in deploy/argocd/ but deliberately not applied - it targets a
different cluster (node1-3) that isn't registered with ArgoCD yet, and
there's still no real prod hostname/Stalwart/ClusterIssuer.
Fixes base/ingress.yaml to the real ingressClassName: traefik (was the
nginx-style `public`, which doesn't exist on either cluster) and gives
each overlay its own cert-manager issuer patch instead of one hardcoded
value, since dev and prod need different (or, for prod, nonexistent)
issuers.
This commit is contained in:
Bernd Rodler
+104
-83
@@ -1,51 +1,61 @@
|
||||
# GitLab-CI dev→prod pipeline for VNCmail+.
|
||||
# GitLab-CI dev→prod pipeline for VNCmail+ — GitOps via ArgoCD.
|
||||
#
|
||||
# Design (see the approved plan for full rationale):
|
||||
# Revised after direct inspection of the real infrastructure found ArgoCD
|
||||
# already installed (idle, zero Applications) on the dev-k8s-1/2/3 cluster.
|
||||
# That's more idiomatic than a runner-executes-kubectl design, and it means
|
||||
# this pipeline needs ZERO cluster credentials — CI only ever talks to the
|
||||
# container registry and to this git repo. ArgoCD (which already has
|
||||
# whatever cluster access it needs, set up once when its Applications were
|
||||
# registered — see deploy/argocd/) is what actually applies anything.
|
||||
#
|
||||
# Design:
|
||||
# - One image name, environment lives only in the tag. No more -dev/-beta
|
||||
# name confusion.
|
||||
# name confusion from the old GitHub Actions workflow.
|
||||
# - MR into `dev`: verify only (typecheck/lint/unit test/build check). No
|
||||
# push, no deploy — this is the multi-developer merge gate.
|
||||
# - Push to `dev`: build+push an immutable `sha-<sha>` tag, auto-deploy it
|
||||
# to the vncmail (sandbox) namespace. No approval needed — dev always
|
||||
# deploys.
|
||||
# - Push to `dev`: build+push an immutable `sha-<sha>` tag, then commit a
|
||||
# one-line tag-bump into overlays/dev/image-tag/kustomization.yaml
|
||||
# (`[skip ci]`, so this doesn't retrigger itself). ArgoCD's `vncmail-dev`
|
||||
# Application has automated sync — it notices the git change and applies
|
||||
# it. No approval needed, dev always deploys, and this job never touches
|
||||
# the cluster directly.
|
||||
# - Push to `main`: NEVER rebuilds. `main` only ever advances via
|
||||
# `git merge --ff-only dev`, so main's HEAD commit already has a built
|
||||
# image. The `promote` job retags that exact digest (registry-side copy,
|
||||
# same primitive the old docker-publish.yml GHA workflow already used for
|
||||
# its multi-arch manifest-list merge) and applies it to prod. `when:
|
||||
# manual` + a protected `production` GitLab environment is the approval
|
||||
# gate — nobody but an authorized user can click it, and nothing here
|
||||
# runs automatically on main.
|
||||
# image (the same sha- tag dev already deployed). This job just bumps
|
||||
# overlays/prod/image-tag/kustomization.yaml to point at that same tag.
|
||||
# The actual promotion gate is a HUMAN clicking Sync on the `vncmail-prod` ArgoCD
|
||||
# Application (deliberately NOT automated sync) — not a GitLab manual
|
||||
# job, since ArgoCD already provides that exact gate more directly.
|
||||
# Until prod Stalwart/hostname/secrets are real (see VNCMAIL-SETUP.md),
|
||||
# nobody should click that Sync button — but nothing here does it for
|
||||
# you either way.
|
||||
#
|
||||
# Deliberately single-platform (linux/amd64) for the cluster build — this
|
||||
# pipeline's job is deploying to a known amd64 microk8s cluster, not public
|
||||
# multi-arch distribution (that's what the GHCR release workflows are for,
|
||||
# and they're untouched by this file).
|
||||
# Deliberately single-platform (linux/amd64) — this pipeline serves two
|
||||
# known amd64 microk8s clusters, not public multi-arch distribution (that's
|
||||
# what the GHCR release workflows are for, untouched by this file).
|
||||
#
|
||||
# Prerequisites this pipeline assumes are already in place (see the plan's
|
||||
# "Split of responsibility" — these are admin/infra actions, not something
|
||||
# this file can set up):
|
||||
# - GitLab Container Registry enabled for this project (CI_REGISTRY_* vars
|
||||
# are then provided automatically — no manual credential setup needed).
|
||||
# - A GitLab Runner with the Kubernetes executor, whose deploy-stage jobs
|
||||
# run as a `gitlab-deployer` ServiceAccount scoped (namespaced Role, not
|
||||
# cluster-admin) to the `vncmail` namespace (and later `vncmail-prod`).
|
||||
# kubectl auto-detects in-cluster config from that ServiceAccount's
|
||||
# mounted token — no KUBECONFIG variable required.
|
||||
# Prerequisite this file assumes (documented in VNCMAIL-SETUP.md, not
|
||||
# something this file can set up itself):
|
||||
# - GitLab Container Registry enabled for this project (confirmed done).
|
||||
# - A GitLab Runner (any kind — no cluster access needed at all now).
|
||||
# - Either "allow this job token to push to this project" enabled
|
||||
# (Settings → CI/CD → Job token permissions), OR a project access token
|
||||
# with `write_repository` scope in $GITLAB_PUSH_TOKEN. The job below
|
||||
# tries CI_JOB_TOKEN first (see the script).
|
||||
#
|
||||
# deploy/k8s/ca/ (the EJBCA internal CA) is never referenced anywhere below —
|
||||
# that stays a fully manual, human-only runbook (see deploy/k8s/ca/README.md).
|
||||
# deploy/k8s/ca/ (the EJBCA internal CA) is never referenced anywhere below,
|
||||
# and neither ArgoCD Application in deploy/argocd/ points at it — that stays
|
||||
# a fully manual, human-only runbook (see deploy/k8s/ca/README.md).
|
||||
|
||||
stages:
|
||||
- verify
|
||||
- build
|
||||
- deploy-dev
|
||||
- promote
|
||||
- bump-dev
|
||||
- bump-prod
|
||||
|
||||
variables:
|
||||
IMAGE: $CI_REGISTRY_IMAGE/vncmail-plus
|
||||
DEV_NAMESPACE: vncmail
|
||||
PROD_NAMESPACE: vncmail-prod
|
||||
GIT_STRATEGY: clone
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# verify — required check on every MR into dev. No registry, no cluster.
|
||||
@@ -62,9 +72,8 @@ verify:
|
||||
- npm run test:translations
|
||||
- npm run build
|
||||
# test:integration is deliberately NOT here — it spins up a real Stalwart
|
||||
# fixture via docker-compose (Docker-in-Docker), which is heavier than a
|
||||
# fast MR gate should be. Candidate for a separate scheduled/optional job
|
||||
# later, not a blocker for this pipeline's first cut.
|
||||
# fixture via docker-compose (Docker-in-Docker), heavier than a fast MR
|
||||
# gate should be. Candidate for a separate scheduled job, not a blocker.
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# build — push to dev only. Builds once; main never rebuilds (see header).
|
||||
@@ -84,61 +93,73 @@ build:
|
||||
- docker push "$IMAGE:dev-latest"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# deploy-dev — automatic, no approval. Deploys the immutable sha tag, never
|
||||
# the moving dev-latest pointer, so what's running always matches one commit.
|
||||
# bump-dev — no cluster access. Commits the just-built tag into the overlay
|
||||
# ArgoCD watches; ArgoCD's automated sync does the actual apply.
|
||||
# ---------------------------------------------------------------------------
|
||||
deploy-dev:
|
||||
stage: deploy-dev
|
||||
image: bitnami/kubectl:1.31
|
||||
environment:
|
||||
name: dev
|
||||
url: https://vncmail.sandbox.vnc.de
|
||||
bump-dev:
|
||||
stage: bump-dev
|
||||
image: alpine/git:2.47.0
|
||||
rules:
|
||||
- if: '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == "dev"'
|
||||
script:
|
||||
# Apply the manifests first (structure/config), then set the exact image
|
||||
# this pipeline just built — imperative `set image`, not a kustomize-file
|
||||
# edit, so overlays/dev never needs a commit to change what's deployed.
|
||||
- kubectl apply -k deploy/k8s/overlays/dev
|
||||
- kubectl -n $DEV_NAMESPACE set image deployment/vncmail-plus vncmail-plus="$IMAGE:sha-$CI_COMMIT_SHORT_SHA"
|
||||
- kubectl -n $DEV_NAMESPACE rollout status deploy/vncmail-plus --timeout=120s
|
||||
- TAG="sha-$CI_COMMIT_SHORT_SHA"
|
||||
- |
|
||||
cat > deploy/k8s/overlays/dev/image-tag/kustomization.yaml <<EOF
|
||||
# Owned by CI (bump-dev job in .gitlab-ci.yml) - regenerated every
|
||||
# push to dev. Do not hand-edit; edits here get overwritten.
|
||||
apiVersion: kustomize.config.k8s.io/v1alpha1
|
||||
kind: Component
|
||||
images:
|
||||
- name: ghcr.io/brvncde-dotcom/vncmail-plus-dev
|
||||
newName: $IMAGE
|
||||
newTag: $TAG
|
||||
EOF
|
||||
- git config user.name "vncmail-ci"
|
||||
- git config user.email "ci@vnc.biz"
|
||||
- git add deploy/k8s/overlays/dev/image-tag/kustomization.yaml
|
||||
- |
|
||||
if git diff --cached --quiet; then
|
||||
echo "No change (tag already pinned) - nothing to commit"
|
||||
else
|
||||
git commit -m "chore(deploy): pin dev to $TAG [skip ci]"
|
||||
git push "https://gitlab-ci-token:${GITLAB_PUSH_TOKEN:-$CI_JOB_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git" HEAD:dev
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# promote — manual, protected `production` environment. No docker build here
|
||||
# — retags the exact digest already deployed to dev, then applies prod
|
||||
# pinned to that digest (never a mutable tag).
|
||||
# bump-prod — no cluster access, no rebuild. Points overlays/prod at the
|
||||
# exact tag already running on dev. Does NOT deploy anything: vncmail-prod's
|
||||
# ArgoCD Application has manual sync, so this only prepares what a human
|
||||
# would be syncing, it doesn't sync it.
|
||||
# ---------------------------------------------------------------------------
|
||||
promote:
|
||||
stage: promote
|
||||
image: docker:27-cli
|
||||
services:
|
||||
- docker:27-dind
|
||||
environment:
|
||||
name: production
|
||||
url: https://vncmail.CHANGEME.invalid # placeholder until the real prod host is decided
|
||||
bump-prod:
|
||||
stage: bump-prod
|
||||
image: alpine/git:2.47.0
|
||||
rules:
|
||||
# `when: manual` lives inside the rule (not as a top-level job key) —
|
||||
# required syntax once `rules:` is used at all.
|
||||
- if: '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == "main"'
|
||||
when: manual
|
||||
before_script:
|
||||
- echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" "$CI_REGISTRY" --password-stdin
|
||||
script:
|
||||
- echo "Retagging the image already built+deployed for dev commit $CI_COMMIT_SHA — no rebuild."
|
||||
- docker buildx imagetools create --tag "$IMAGE:prod-latest" "$IMAGE:sha-$CI_COMMIT_SHORT_SHA"
|
||||
- DIGEST=$(docker buildx imagetools inspect "$IMAGE:sha-$CI_COMMIT_SHORT_SHA" | awk '/^Digest:/{print $2}')
|
||||
- echo "Resolved digest for prod = $IMAGE@$DIGEST"
|
||||
- >
|
||||
echo "STOPPING HERE ON PURPOSE: deploy/k8s/overlays/prod is still
|
||||
scaffolded/inactive (placeholder hostname, placeholder JMAP_SERVER_URL
|
||||
— no prod Stalwart exists yet). Once both are real (Phase D in the
|
||||
pipeline plan / VNCMAIL-SETUP.md), replace this echo with the same
|
||||
pattern deploy-dev uses, against a bitnami/kubectl image and
|
||||
\$PROD_NAMESPACE: kubectl apply -k deploy/k8s/overlays/prod &&
|
||||
kubectl -n \$PROD_NAMESPACE set image deployment/vncmail-plus
|
||||
vncmail-plus=$IMAGE@$DIGEST"
|
||||
# Deliberately does NOT run `kubectl apply -k overlays/prod` yet — prod
|
||||
# namespace/hostname/Stalwart don't exist (Phase C/D in the plan). Once
|
||||
# they do, replace the placeholder echo above with the same
|
||||
# `kubectl apply -k .` + `set image ...@$DIGEST` pattern deploy-dev uses,
|
||||
# against $PROD_NAMESPACE, using the bitnami/kubectl image.
|
||||
- TAG="sha-$CI_COMMIT_SHORT_SHA"
|
||||
- echo "main advanced to $CI_COMMIT_SHA (must be a dev commit, ff-only) - that image already exists as $IMAGE:$TAG"
|
||||
- |
|
||||
cat > deploy/k8s/overlays/prod/image-tag/kustomization.yaml <<EOF
|
||||
# Owned by CI (bump-prod job in .gitlab-ci.yml) - regenerated every
|
||||
# push to main. Do not hand-edit; edits here get overwritten. Bumping
|
||||
# this is NOT the same as deploying it - vncmail-prod's ArgoCD
|
||||
# Application has manual sync, see the note in the parent
|
||||
# kustomization.yaml.
|
||||
apiVersion: kustomize.config.k8s.io/v1alpha1
|
||||
kind: Component
|
||||
images:
|
||||
- name: ghcr.io/brvncde-dotcom/vncmail-plus-dev
|
||||
newName: $IMAGE
|
||||
newTag: $TAG
|
||||
EOF
|
||||
- git config user.name "vncmail-ci"
|
||||
- git config user.email "ci@vnc.biz"
|
||||
- git add deploy/k8s/overlays/prod/image-tag/kustomization.yaml
|
||||
- |
|
||||
if git diff --cached --quiet; then
|
||||
echo "No change (tag already pinned) - nothing to commit"
|
||||
else
|
||||
git commit -m "chore(deploy): point prod overlay at $TAG (not synced - manual gate in ArgoCD) [skip ci]"
|
||||
git push "https://gitlab-ci-token:${GITLAB_PUSH_TOKEN:-$CI_JOB_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git" HEAD:main
|
||||
fi
|
||||
|
||||
Reference in New Issue
Block a user