The bump-dev (and identically-configured bump-prod) job failed during
prepare_script with:
ERROR: Job failed: prepare environment: waiting for pod running:
pulling image "alpine/git:2.47.0": image pull failed: ... not found
Root cause: alpine/git:2.47.0 does not exist on Docker Hub. The alpine/git
2.47.x line starts at 2.47.1 — there is no 2.47.0 build. The runner's
image pull correctly fails with 'not found', and GitLab's Kubernetes
executor treats an image-pull failure during prepare_script as fatal, so
the job never reaches its script block.
Fix: pin both bump-dev and bump-prod to alpine/git:2.47.2 (latest 2.47.x).
Pinned rather than 'latest' so the job stays reproducible. bump-prod had
the same nonexistent tag and would have hit the identical failure on its
next run (whenever main advances), so both are fixed together.
The build job failed with:
Cannot connect to the Docker daemon at tcp://localhost:2375.
Is the docker daemon running?
Three things were wrong:
1. DOCKER_HOST was set to tcp://localhost:2375. The DinD daemon runs in
the service sidecar container, not in the build container, so
localhost was always going to refuse the connection. The correct host
is the service alias 'docker'.
2. The docker:28.4.0-dind service was declared without an explicit
alias. Without alias: docker, GitLab derives the hostname from the
image string 'docker:28.4.0-dind', and since ':' is invalid in DNS,
the 'docker' hostname never resolves. The explicit alias is required
for tcp://docker:2375 to work at all.
3. docker:28.4.0-dind enables TLS by default and listens on 2376, but
DOCKER_HOST points at 2375. Setting DOCKER_TLS_CERTDIR="" disables
TLS so the daemon listens on plaintext 2375, matching DOCKER_HOST.
This mirrors the known-working pattern in the vnc-localidp pipeline
(docker:20.10.17-dind + alias: docker + DOCKER_HOST=tcp://docker:2375
+ DOCKER_TLS_CERTDIR=""). The TLS-defaults behavior has been unchanged
since docker 19.03, so the same pattern applies on 28.4.0.
DOCKER_TLS_CERTDIR forced the docker:28.4.0-dind service to listen on
port 2376 with TLS, but DOCKER_HOST pointed to the non-TLS port 2375.
This caused the docker client to loop forever with:
Cannot connect to the Docker daemon at tcp://localhost:2375.
Removing DOCKER_TLS_CERTDIR lets the daemon listen on 2375 again,
matching DOCKER_HOST, restoring docker-in-docker connectivity.
Diagnosed definitively rather than by log-guessing this time:
$ curl -i https://registry.gitlab.vnc.biz/v2/
www-authenticate: Bearer realm="http://gitlab.vnc.biz/jwt/auth",
service="dependency_proxy"
x-runtime: 0.020470
x-gitlab-meta: {"correlation_id":...}
x-runtime/x-gitlab-meta are Rails headers and the service is
"dependency_proxy" - nginx routes that hostname to the GitLab Rails app,
which treats /v2/ as the Docker Hub pull-through cache, not as this
project's container registry. The registry service was never wired behind
the vhost, which is why an unscoped docker login succeeded while kaniko's
scoped :push request got 403 (the dependency proxy has no push concept).
Fixing that is server-side nginx/omnibus work. Keeping kaniko (it solved
the real dind-needs-privileged problem) and pointing it at GHCR, plus an
upfront credential check so a missing variable fails in seconds instead of
after a full Next.js build.
dind never actually came up on this runner regardless of how it was
addressed (unix socket, docker:2375, localhost:2375 all failed identically
after a successful registry login) — on GitLab's Kubernetes executor that
means the dind container needs `privileged: true` in the runner's own
config.toml, which is admin-side, not something this file can set.
Kaniko builds OCI images without any daemon, so it needs no privileged
pod and no dind service at all — GitLab's own recommended path for this
exact executor, and safer on a shared cluster besides.
This runner is GitLab's Kubernetes executor (pod names in the job log:
runner-uncqet63-project-499-concurrent-*), where all containers in a job
share one pod's network namespace. The docker: service-alias hostname is
a Docker-executor convention (bridge network + DNS alias) and doesn't
apply here — tcp://docker:2375 correctly read the variable but nothing
answered at that name. localhost is the right host for this executor.
registry login now succeeds (CI_REGISTRY populated correctly) but the
build step failed separately: docker:27-dind defaults to TLS on :2376,
which the docker:27-cli client image doesn't know to use without a
mounted cert dir. DOCKER_HOST=tcp://docker:2375 + DOCKER_TLS_CERTDIR=""
is the standard fix for GitLab's Kubernetes executor, where both
containers share the job's pod network namespace.
Confirmed 2026-08-05 the project's Container Registry is now enabled
server-side (visible in the left sidebar under Deploy). That's strictly
better than the GHCR detour: $CI_REGISTRY/$CI_REGISTRY_USER/$CI_REGISTRY_PASSWORD
are predefined GitLab CI variables scoped to this project, so this needs
zero manually-created credentials (no GitHub PAT to hold in CI/CD variables).
GitLab's Container Registry was enabled at the omnibus service level
(registry.gitlab.vnc.biz responds, confirmed with a real GitLab-shaped
401), but the pipeline's build job kept trying to auth against Docker
Hub instead - CI_REGISTRY was empty. Root cause: registry_external_url
only starts the registry SERVICE; gitlab_rails['registry_enabled'] = true
is a separate key that tells the Rails app the registry exists, and it
was never set. Symptom matched exactly: registry reachable, but no
Container Registry toggle anywhere in project settings OR admin settings,
and CI_REGISTRY empty in every job regardless of retry.
Reverting the pipeline to ghcr.io/brvncde-dotcom/vncmail-plus-dev - the
exact image the sandbox was already running before any of this session's
pipeline existed, confirmed public (no imagePullSecrets needed). This is
a revert to a known-working path, not a new risk.
Needs $GITLAB_CI_GHCR_TOKEN (GitHub PAT, write:packages) and
$GITLAB_CI_GHCR_USER as masked/protected CI/CD variables - a GitHub
credential has to come from GitHub, nothing on the GitLab side can
substitute for it.
Direct SSH access to the actual clusters (node1-3 "prod" HA, dev-k8s-1-3
"dev") revealed two things that made the previous design wrong:
1. Neither cluster has vncmail/vnc-ca namespaces or a bulwark ingress at
all - the "live sandbox" referenced in this repo's docs/manifests was
never actually applied anywhere. Both ingress.yaml's ingressClassName
(public) and cert-manager issuer (letsencrypt-prod) were also wrong:
both clusters run Traefik (class is literally named `traefik`), and
only dev-k8s has any ClusterIssuer at all (`letsencrypt-staging`).
node1-3 has zero ClusterIssuers configured.
2. dev-k8s already has ArgoCD installed, idle, zero Applications - more
idiomatic to use it than have GitLab Runner execute kubectl directly.
Pivots .gitlab-ci.yml: build+push image, then commit the tag into a small
per-overlay Component (overlays/{dev,prod}/image-tag/) that ArgoCD's
Application watches - CI never touches the cluster, only the registry and
this repo. dev's Application (vncmail-dev) is registered and applied
already (manual sync for now, until the one-time namespace secret
bootstrap is done - see VNCMAIL-SETUP.md). prod's Application is
scaffolded in deploy/argocd/ but deliberately not applied - it targets a
different cluster (node1-3) that isn't registered with ArgoCD yet, and
there's still no real prod hostname/Stalwart/ClusterIssuer.
Fixes base/ingress.yaml to the real ingressClassName: traefik (was the
nginx-style `public`, which doesn't exist on either cluster) and gives
each overlay its own cert-manager issuer patch instead of one hardcoded
value, since dev and prod need different (or, for prod, nonexistent)
issuers.
Multiple developers now work on this repo, and the only working deploy
trigger required pushing to GitHub - which contradicts the standing
GitLab-canonical policy for this repo - while every actual deploy was a
manual kubectl run against one environment (no prod exists at all).
Restructures deploy/k8s/ into base/ + overlays/{dev,prod}: overlays/dev
is a verified byte-for-byte no-op for the live sandbox (kubectl kustomize
diff against the old flat layout is empty), overlays/prod is scaffolded
but inert (placeholder hostname + JMAP_SERVER_URL, since neither a prod
hostname decision nor a prod Stalwart exist yet). deploy/k8s/ca/ (the
EJBCA internal CA) is untouched and never referenced by either overlay.
Adds .gitlab-ci.yml: verify (MR gate, no push/deploy) -> build+deploy-dev
(automatic on push to dev, one image name/tag-only environments, fixing
the old -dev/-beta naming split) -> promote (manual, protected
`production` environment, retags the exact dev digest via
`docker buildx imagetools create` - never rebuilds - and is left as a
documented TODO for the actual `kubectl apply` until prod is real).
Updates VNCMAIL-SETUP.md and deploy/k8s/README.md to describe the new
flow and correct the aspirational promotion description that assumed a
"production image" CI never actually built.
Also fixes a pre-existing lint error (no-control-regex false positive on
an intentional DN-sanitizing character class in lib/smime-ca/ejbca.ts)
that was blocking this commit's pre-commit hook - unrelated to this
change otherwise, confirmed already present on dev before this branch.
Runner/RBAC/registry setup is an infra prerequisite this commit cannot
provide - documented in the pipeline plan, not part of this diff.
Stefan-Sanger
Bernd Rodler