SRCmail/deploy/k8s/ca

VNC internal CA — EJBCA Community on microk8s

Runbook for A-01 / A-06. Issues 1-year S/MIME certificates for VNCmail+.

You run every command here. Claude wrote the manifests and cannot reach the cluster (no kubeconfig on the authoring machine), and the root-key ceremony in §3 must not be automated by an agent — the entire value of an offline root is that its private key never exists on a machine that runs services or tooling.

---

0. One decision to make before you type anything

Name the root for the organisation, not the environment.

You asked for sandbox first with the ability to promote to vncmail at any time. The way that stays cheap is a single root, generated once, with per-environment intermediates underneath it:

VNC Root CA R1                          offline · 15y · RSA 4096 · pathlen:1
├─ VNC S/MIME Issuing CA Sandbox R1     in-cluster · 5y · RSA 4096 · pathlen:0   → *@sandbox.vnc.de
└─ VNC S/MIME Issuing CA R1             in-cluster · 5y · RSA 4096 · pathlen:0   → *@vncmail.de   (later)

Promotion is then "issue a second intermediate from the same root" — a one-hour ceremony. The trust anchor you distribute to laptops, phones and partners does not change, and certificates already issued keep validating.

The alternative — a throwaway VNC Sandbox Root — means that on promotion you redistribute a new trust anchor to every device and every external party who ever verified one of your signatures. That is the expensive path, and it is only visible as expensive later.

So: generate the root at prod grade, once, now, even though the first intermediate only serves @sandbox.vnc.de. The extra cost today is choosing a better passphrase and a safe to keep the USB key in.

RSA 4096 rather than an elliptic curve throughout, deliberately. ECDSA S/MIME is still poorly handled by older Outlook and by several mobile clients, and S/MIME interop failures are silent — the recipient sees a broken signature, not an error you get told about. Pay the key-size cost for interop you can't test.

---

1. Install

kubectl apply -f deploy/k8s/ca/namespace.yaml

Fill in and apply the secret out-of-band (never commit real values):

cp deploy/k8s/ca/secret.example.yaml /tmp/ca-secret.yaml && $EDITOR /tmp/ca-secret.yaml

kubectl apply -f /tmp/ca-secret.yaml && shred -u /tmp/ca-secret.yaml

kubectl apply -k deploy/k8s/ca/

First boot builds the EJBCA schema and takes several minutes. Watch it rather than assuming it hung:

kubectl -n vnc-ca logs -f deploy/ejbca

kubectl -n vnc-ca get pods -w

Verify before going further

kubectl -n vnc-ca exec deploy/ejbca -- curl -sf http://localhost:8080/ejbca/publicweb/healthcheck/ejbcahealth && echo OK

If the manifests' env-var names have drifted from the image tag you pulled, this is where it shows up — EJBCA will start but fail to bind its datasource. Check the documented variables for your tag before editing anything else:

kubectl -n vnc-ca logs deploy/ejbca | grep -iE "datasource|jdbc|database"

---

2. Get administrative access

EJBCA's admin web requires a client certificate. On first boot the container enrols a SuperAdmin and writes a PKCS#12 inside the pod.

kubectl -n vnc-ca exec deploy/ejbca -- find / -name "*.p12" -newermt "-1 day" 2>/dev/null

Copy it out, import it into your browser, then reach the admin web by port-forward — it is not exposed through any ingress and must not be:

kubectl -n vnc-ca port-forward deploy/ejbca 8443:8443

Then open https://localhost:8443/ejbca/adminweb.

If the container did not create a SuperAdmin (behaviour differs by tag), use the CLI inside the pod instead: kubectl -n vnc-ca exec -it deploy/ejbca -- /opt/keyfactor/bin/ejbca.sh ra addendentity ... followed by setclearpwd and a browser enrolment against https://localhost:8443/ejbca/ra/.

---

3. Root ceremony — you, offline, once

Do this on a machine that is not this cluster and not your daily laptop if you can manage it. A live USB session on a machine with networking physically off is enough for a sandbox-grade start; the point is that the root key never touches a host that runs services.

Everything below happens in one directory that you will destroy at the end.

3.1 Prepare the config. Save as root.cnf:

[ req ]
default_md         = sha256
prompt             = no
distinguished_name = dn
x509_extensions    = root_ext

[ dn ]
C  = CH
O  = VNC AG
CN = VNC Root CA R1

[ root_ext ]
basicConstraints       = critical,CA:TRUE,pathlen:1
keyUsage               = critical,keyCertSign,cRLSign
subjectKeyIdentifier   = hash

# --- used in 3.4 to sign the intermediate CSR ---
[ ca ]
default_ca = CA_root

[ CA_root ]
new_certs_dir    = .
database         = index.txt
serial           = serial
private_key      = root.key
certificate      = root.crt
default_md       = sha256
policy           = policy_any
crl              = root.crl
default_crl_days = 365
unique_subject   = no

[ policy_any ]
countryName            = optional
organizationName       = optional
organizationalUnitName = optional
commonName             = supplied

[ int_ext ]
basicConstraints       = critical,CA:TRUE,pathlen:0
keyUsage               = critical,keyCertSign,cRLSign
subjectKeyIdentifier   = hash
authorityKeyIdentifier = keyid:always
# Revocation pointers for the INTERMEDIATE itself, served by the root's CRL.
crlDistributionPoints  = URI:http://ca.sandbox.vnc.de/ejbca/publicweb/crls/root.crl

pathlen:1 on the root and pathlen:0 on the intermediate together mean the intermediate can issue end-entity certificates and nothing else. It cannot mint a further CA even if its key is stolen — that limits a compromise to "revoke one intermediate" instead of "the whole hierarchy is untrustworthy".

3.2 Generate the root key. You will be asked for a passphrase. Generate it with a password manager, minimum 24 random characters, and record where it lives before you type it — a root key whose passphrase is lost is a hierarchy you have to rebuild.

openssl genrsa -aes256 -out root.key 4096

3.3 Self-sign the root. 15 years, so it outlives several intermediate rotations and you do the ceremony once:

openssl req -new -x509 -config root.cnf -key root.key -sha256 -days 5480 -out root.crt

openssl x509 -in root.crt -noout -text | sed -n '1,25p'

Confirm in that output: CA:TRUE, pathlen:1, Key Usage: Certificate Sign, CRL Sign, and a 15-year validity window. If basicConstraints is missing the root is useless — the config's x509_extensions did not apply.

3.4 Sign the intermediate. EJBCA generates the intermediate key inside the cluster and hands you a CSR; the intermediate's private key never leaves EJBCA and never appears in this directory.

In the admin web: CA Functions → Certificate Authorities → Create CA

• Name: VNC S/MIME Issuing CA Sandbox R1
• Subject DN: CN=VNC S/MIME Issuing CA Sandbox R1,O=VNC AG,C=CH
• Crypto Token: create a new soft token, PIN = EJBCA_CRYPTO_TOKEN_PIN from your secret
• Key: RSA 4096, signing algorithm SHA256WithRSA
• Signed By: External CA ← this is what makes it emit a CSR instead of self-signing
• Validity: 5y
• CRL Expire Period: 1d, CRL Overlap: 10m
• Default CRL Distribution Point: http://ca.sandbox.vnc.de/ejbca/publicweb/crls/search.cgi?iHash=... (EJBCA fills the hash — take what it offers)

Save, download the CSR, move it to the offline machine, then:

touch index.txt && echo 1000 > serial

openssl ca -config root.cnf -extensions int_ext -days 1825 -notext -in sandbox-issuing.csr -out sandbox-issuing.crt

3.5 Issue the root CRL. Do this now, in the same ceremony — not later. A root that has never published a CRL cannot revoke a compromised intermediate, and you will not want to bring the root key out under incident pressure just to discover the procedure doesn't work:

openssl ca -config root.cnf -gencrl -out root.crl

openssl crl -in root.crl -noout -text | head -12

3.6 Take the outputs off, then destroy the directory. Off the machine: root.crt, root.crl, sandbox-issuing.crt, and root.key (to encrypted storage, two copies, two physical locations).

shred -u root.key && rm -rf ./*

The root key comes out of the safe for exactly three reasons: signing a new intermediate (promotion to vncmail.de), refreshing the root CRL before it expires (annually — put it in a calendar now), or revoking an intermediate.

3.7 Import the chain back into EJBCA. Admin web → the CA you created → Import CA certificate, upload root.crt then sandbox-issuing.crt. The CA status must move to Active. Publish root.crl so the URL in the intermediate's CDP actually resolves.

---

4. Certificate profile — 1-year S/MIME

Certificate Profiles → Add → VNC S/MIME 1y, type End Entity.

Setting	Value	Why
Validity	1y	your decision
Key algorithms	RSA 2048, 3072, 4096	2048 floor for interop; no ECDSA yet (§0)
Key Usage	digitalSignature, keyEncipherment	signing and decryption need both
Extended Key Usage	emailProtection	critical — see below
Subject Alternative Name	rfc822Name, required	this is the authoritative address
Basic Constraints	CA:FALSE, critical
CRL Distribution Point	use CA default
OCSP Service Locator (AIA)	http://ca.sandbox.vnc.de/ejbca/publicweb/status/ocsp
Allow key recovery	on	see §7
Allow subject DN override by CSR	OFF	load-bearing, see below
Allow extension override by CSR	OFF	load-bearing, see below
Allow subject alt name override by CSR	OFF	load-bearing, see below

The three override settings must be OFF, and this is the single most important line in this document.

The enrolment route deliberately does not inspect the CSR to police what it asks for. It doesn't need to: the route supplies the subject and the rfc822Name SAN itself, from addresses Stalwart confirmed the account may send from, and the CSR contributes only a public key plus proof the requester holds the matching private key.

That reasoning is only sound while EJBCA ignores the CSR's own subject and extensions. Turn any of these overrides on and a hand-crafted CSR claiming rfc822Name=ceo@vnc.de gets exactly that certificate — no code change, no alert, and the enrolment route still looks correct in review. It is a one-checkbox path from "authenticated users get certificates for their own addresses" to "authenticated users get certificates for anyone's address".

Verify it rather than trusting the profile screen, once the route is live:

openssl req -new -key /tmp/t.key -subj "/CN=Impostor" -addext "subjectAltName=email:ceo@vnc.de" -out /tmp/t.csr

Submit that CSR through the enrolment route as an ordinary user. The certificate that comes back must carry your own address, not ceo@vnc.de.

Two of these carry real weight:

emailProtection EKU, and only that. A certificate with no EKU is treated by some clients as valid for anything — TLS server auth included. Constrain it.

rfc822Name SAN required. Modern clients bind the sender address from the SAN, not the emailAddress DN attribute. Our forked plugin's fix-1 check (signerEmailMatch, which refuses to auto-import a signer cert whose address doesn't match the From header) now reads the address the same way clients do — SAN first, and matched against every address the certificate carries. If EJBCA issues certificates without an rfc822Name SAN, that check fails closed and encryption silently never becomes available.

Populating the DN emailAddress attribute as well, for old Outlook, is safe — but only as of finding 11. Until then the plugin read the DN attribute in preference to the SAN and compared only the first address it found, so an EJBCA certificate with both fields populated would have reported every genuine signature as "signer ≠ From" and blocked the import. Covered now by vnc/plugins/smime/verify-address-binding.mjs.

End Entity Profiles → Add → VNC S/MIME User:

• Default Certificate Profile: VNC S/MIME 1y; available: the same only
• Subject DN: CN required + modifiable, O=VNC AG and C=CH fixed
• Subject Alt Name: rfc822Name required, and tick "Use entity email field"
• Default CA: VNC S/MIME Issuing CA Sandbox R1

---

5. RA credential for the enrolment route

The webmail server — not the browser — calls the REST API. It needs its own client certificate with only the authority to enrol end entities.

5.1 Create a certificate profile VNC RA Client (End Entity, EKU clientAuth, validity 1y) and enrol one entity CN=vncmail-ra-sandbox against it. Download as PKCS#12.

5.2 Restrict it. System Functions → Administrator Roles → Add → VNCmail RA (sandbox):

Rule	Access
/ca_functionality/create_certificate	Allow
/ca/VNC S/MIME Issuing CA Sandbox R1	Allow
/endentityprofilesrules/VNC S/MIME User/**	Allow
/ra_functionality/revoke_end_entity	Allow
everything else	not granted

Match by the certificate's serial + issuer DN, not by CN. Do not give this role /administrator or any /system_functionality rule: this credential lives on an internet-facing pod, and the blast radius of it leaking should be "issue and revoke S/MIME certs under one profile", not "reconfigure the CA".

5.3 Load it into the webmail namespace:

kubectl -n vncmail create secret generic smime-ra \
  --from-file=client.p12=./vncmail-ra-sandbox.p12 \
  --from-literal=client-password='<p12 passphrase>' \
  --from-file=ca-chain.pem=./chain.pem

chain.pem is sandbox-issuing.crt followed by root.crt. The enrolment route pins this chain when it connects to EJBCA on 8443 — it does not trust the public root store, so EJBCA's self-signed server certificate (TLS_SETUP_ENABLED=simple) is correct and expected here.

---

6. Verify the network policy actually enforces

Applying a NetworkPolicy on a CNI that doesn't implement it succeeds silently and protects nothing. Prove it:

kubectl -n default run np-probe --rm -it --image=curlimages/curl --restart=Never -- \
  curl -sS -m 5 -k https://ejbca.vnc-ca.svc.cluster.local:8443/ejbca/ejbca-rest-api/v1/ca

This must time out or be refused. If it returns anything HTTP-shaped — including a 401 — the policy is not being enforced and the REST API is exposed cluster-wide. Check your CNI before continuing:

kubectl -n kube-system get pods | grep -iE "calico|cilium|flannel"

---

7. Key recovery is not optional here

S/MIME differs from TLS in a way that has bitten every organisation that deployed it without thinking about this: if a user loses their private key, every message ever encrypted to them is permanently unreadable. Not inconvenient — gone. Re-issuing a certificate does not help, because the old messages were encrypted to the old key.

So Allow key recovery in §4 is deliberate, and it is a real trade-off:

• on — EJBCA escrows the decryption key. Lost laptop is recoverable. But the CA database now contains material that decrypts users' mail, so §8 backup handling and the §5 role restrictions become load-bearing, and the escrow is something you must be able to explain to a user asking whether their mail is end-to-end encrypted. It is, from the wire's perspective; it is not, from the CA operator's.
• off — nobody but the user can ever read their mail, and a lost device is permanent data loss with no recourse.

For a corporate deployment where mail is a business record, escrow on is the defensible choice, and it's what §4 sets. Decide this consciously — it is far cheaper to turn on now than to explain later why three years of mail is gone.

If you keep it on, use a separate key-recovery role with two-person approval rather than folding that authority into the RA credential.

---

8. Backup

ejbca-db-data contains the intermediate CA private key and — per §7 — escrowed user decryption keys. A dump of it is equivalent to the CA itself.

kubectl -n vnc-ca exec deploy/ejbca-db -- sh -c \
  'mariadb-dump -u root -p"$MARIADB_ROOT_PASSWORD" --single-transaction ejbca' \
  | gzip > ejbca-$(date +%F).sql.gz

Encrypt before it leaves your machine — an unencrypted CA dump in object storage is the whole hierarchy:

gpg --symmetric --cipher-algo AES256 ejbca-$(date +%F).sql.gz

Then to the shared R2 bucket (vnc-backups1) and delete the plaintext. Restore-test it once, now, against a scratch namespace — an untested CA backup is a belief, not a backup.

Not in this backup, by design and stored separately: the offline root key (§3.6), EJBCA_CRYPTO_TOKEN_PIN, and the RA PKCS#12 passphrase.

---

9. Promotion to production

Nothing here is thrown away. Same root, new intermediate:

1. Bring root.key out of the safe; repeat §3.4–3.6 for CN=VNC S/MIME Issuing CA R1 — sign it with the same root.
2. Duplicate the §4 profiles as VNC S/MIME 1y (prod) bound to the new CA.
3. Fresh RA credential and role for the prod webmail namespace (§5). Never share the sandbox one across environments.
4. Point the prod CDP/AIA at a stable production hostname. Those URLs are baked into every certificate for its full year, so get the hostname right before the first issuance.

The trust anchor on user devices does not change, and sandbox-issued certificates keep validating.

10. SwissSign (P7, deferred)

The point of the CaProvider interface on the application side is that this whole document becomes one implementation of it. Moving to SwissSign-issued certificates — for ZertES/eIDAS-qualified signatures that external parties validate without installing anything — is then a second implementation plus an identity-verification step, not a rewrite of the enrolment flow.

What survives unchanged: in-browser key generation, CSR construction, the enrolment route, storage, sign/encrypt/decrypt, the UI. What changes: who signs the CSR, and the fact that a human must prove their identity before a qualified certificate is issued — which is a process requirement, not a code one.