SRCmail/deploy/k8s/ca/README.md

# VNC internal CA — EJBCA Community on microk8s

Runbook for `A-01` / `A-06`. Issues 1-year S/MIME certificates for VNCmail+.

You run every command here. Claude wrote the manifests and cannot reach the
cluster (no kubeconfig on the authoring machine), and the root-key ceremony in
§3 **must not** be automated by an agent — the entire value of an offline root is
that its private key never exists on a machine that runs services or tooling.

---

## 0. One decision to make before you type anything

**Name the root for the organisation, not the environment.**

You asked for sandbox first with the ability to promote to `vncmail` at any time.
The way that stays cheap is a single root, generated once, with *per-environment
intermediates* underneath it:

```
VNC Root CA R1                          offline · 15y · RSA 4096 · pathlen:1
├─ VNC S/MIME Issuing CA Sandbox R1     in-cluster · 5y · RSA 4096 · pathlen:0   → *@sandbox.vnc.de
└─ VNC S/MIME Issuing CA R1             in-cluster · 5y · RSA 4096 · pathlen:0   → *@vncmail.de   (later)
```

Promotion is then "issue a second intermediate from the same root" — a one-hour
ceremony. The trust anchor you distribute to laptops, phones and partners does
not change, and certificates already issued keep validating.

The alternative — a throwaway `VNC Sandbox Root` — means that on promotion you
redistribute a new trust anchor to every device and every external party who
ever verified one of your signatures. That is the expensive path, and it is only
visible as expensive later.

So: **generate the root at prod grade, once, now**, even though the first
intermediate only serves `@sandbox.vnc.de`. The extra cost today is choosing a
better passphrase and a safe to keep the USB key in.

> RSA 4096 rather than an elliptic curve throughout, deliberately. ECDSA S/MIME
> is still poorly handled by older Outlook and by several mobile clients, and
> S/MIME interop failures are silent — the recipient sees a broken signature, not
> an error you get told about. Pay the key-size cost for interop you can't test.

---

## 1. Install

```bash
kubectl apply -f deploy/k8s/ca/namespace.yaml
```

Fill in and apply the secret out-of-band (never commit real values):

```bash
cp deploy/k8s/ca/secret.example.yaml /tmp/ca-secret.yaml && $EDITOR /tmp/ca-secret.yaml
```

```bash
kubectl apply -f /tmp/ca-secret.yaml && shred -u /tmp/ca-secret.yaml
```

```bash
kubectl apply -k deploy/k8s/ca/
```

First boot builds the EJBCA schema and takes several minutes. Watch it rather
than assuming it hung:

```bash
kubectl -n vnc-ca logs -f deploy/ejbca
```

```bash
kubectl -n vnc-ca get pods -w
```

### Verify before going further

```bash
kubectl -n vnc-ca exec deploy/ejbca -- curl -sf http://localhost:8080/ejbca/publicweb/healthcheck/ejbcahealth && echo OK
```

If the manifests' env-var names have drifted from the image tag you pulled, this
is where it shows up — EJBCA will start but fail to bind its datasource. Check
the documented variables for your tag before editing anything else:

```bash
kubectl -n vnc-ca logs deploy/ejbca | grep -iE "datasource|jdbc|database"
```

---

## 2. Get administrative access

EJBCA's admin web requires a client certificate. On first boot the container
enrols a `SuperAdmin` and writes a PKCS#12 inside the pod.

```bash
kubectl -n vnc-ca exec deploy/ejbca -- find / -name "*.p12" -newermt "-1 day" 2>/dev/null
```

Copy it out, import it into your browser, then reach the admin web by
port-forward — it is not exposed through any ingress and must not be:

```bash
kubectl -n vnc-ca port-forward deploy/ejbca 8443:8443
```

Then open `https://localhost:8443/ejbca/adminweb`.

> If the container did not create a SuperAdmin (behaviour differs by tag), use
> the CLI inside the pod instead:
> `kubectl -n vnc-ca exec -it deploy/ejbca -- /opt/keyfactor/bin/ejbca.sh ra addendentity ...`
> followed by `setclearpwd` and a browser enrolment against
> `https://localhost:8443/ejbca/ra/`.

---

## 3. Root ceremony — you, offline, once

Do this on a machine that is **not** this cluster and **not** your daily laptop
if you can manage it. A live USB session on a machine with networking physically
off is enough for a sandbox-grade start; the point is that the root key never
touches a host that runs services.

Everything below happens in one directory that you will destroy at the end.

**3.1 Prepare the config.** Save as `root.cnf`:

```ini
[ req ]
default_md         = sha256
prompt             = no
distinguished_name = dn
x509_extensions    = root_ext

[ dn ]
C  = CH
O  = VNC AG
CN = VNC Root CA R1

[ root_ext ]
basicConstraints       = critical,CA:TRUE,pathlen:1
keyUsage               = critical,keyCertSign,cRLSign
subjectKeyIdentifier   = hash

# --- used in 3.4 to sign the intermediate CSR ---
[ ca ]
default_ca = CA_root

[ CA_root ]
new_certs_dir    = .
database         = index.txt
serial           = serial
private_key      = root.key
certificate      = root.crt
default_md       = sha256
policy           = policy_any
crl              = root.crl
default_crl_days = 365
unique_subject   = no

[ policy_any ]
countryName            = optional
organizationName       = optional
organizationalUnitName = optional
commonName             = supplied

[ int_ext ]
basicConstraints       = critical,CA:TRUE,pathlen:0
keyUsage               = critical,keyCertSign,cRLSign
subjectKeyIdentifier   = hash
authorityKeyIdentifier = keyid:always
# Revocation pointers for the INTERMEDIATE itself, served by the root's CRL.
crlDistributionPoints  = URI:http://ca.sandbox.vnc.de/ejbca/publicweb/crls/root.crl
```

`pathlen:1` on the root and `pathlen:0` on the intermediate together mean the
intermediate can issue end-entity certificates and nothing else. It cannot mint
a further CA even if its key is stolen — that limits a compromise to "revoke one
intermediate" instead of "the whole hierarchy is untrustworthy".

**3.2 Generate the root key.** You will be asked for a passphrase. Generate it
with a password manager, minimum 24 random characters, and record where it lives
*before* you type it — a root key whose passphrase is lost is a hierarchy you
have to rebuild.

```bash
openssl genrsa -aes256 -out root.key 4096
```

**3.3 Self-sign the root.** 15 years, so it outlives several intermediate
rotations and you do the ceremony once:

```bash
openssl req -new -x509 -config root.cnf -key root.key -sha256 -days 5480 -out root.crt
```

```bash
openssl x509 -in root.crt -noout -text | sed -n '1,25p'
```

Confirm in that output: `CA:TRUE, pathlen:1`, `Key Usage: Certificate Sign, CRL Sign`,
and a 15-year validity window. If `basicConstraints` is missing the root is
useless — the config's `x509_extensions` did not apply.

**3.4 Sign the intermediate.** EJBCA generates the intermediate key *inside the
cluster* and hands you a CSR; the intermediate's private key never leaves EJBCA
and never appears in this directory.

In the admin web: **CA Functions → Certificate Authorities → Create CA**
- Name: `VNC S/MIME Issuing CA Sandbox R1`
- Subject DN: `CN=VNC S/MIME Issuing CA Sandbox R1,O=VNC AG,C=CH`
- Crypto Token: create a new soft token, PIN = `EJBCA_CRYPTO_TOKEN_PIN` from your secret
- Key: RSA 4096, signing algorithm SHA256WithRSA
- **Signed By: External CA** ← this is what makes it emit a CSR instead of self-signing
- Validity: `5y`
- CRL Expire Period: `1d`, CRL Overlap: `10m`
- Default CRL Distribution Point: `http://ca.sandbox.vnc.de/ejbca/publicweb/crls/search.cgi?iHash=...` (EJBCA fills the hash — take what it offers)

Save, download the CSR, move it to the offline machine, then:

```bash
touch index.txt && echo 1000 > serial
```

```bash
openssl ca -config root.cnf -extensions int_ext -days 1825 -notext -in sandbox-issuing.csr -out sandbox-issuing.crt
```

**3.5 Issue the root CRL.** Do this now, in the same ceremony — not later. A root
that has never published a CRL cannot revoke a compromised intermediate, and you
will not want to bring the root key out under incident pressure just to
discover the procedure doesn't work:

```bash
openssl ca -config root.cnf -gencrl -out root.crl
```

```bash
openssl crl -in root.crl -noout -text | head -12
```

**3.6 Take the outputs off, then destroy the directory.** Off the machine:
`root.crt`, `root.crl`, `sandbox-issuing.crt`, and `root.key` (to encrypted
storage, two copies, two physical locations).

```bash
shred -u root.key && rm -rf ./*
```

The root key comes out of the safe for exactly three reasons: signing a new
intermediate (promotion to `vncmail.de`), refreshing the root CRL before it
expires (annually — put it in a calendar now), or revoking an intermediate.

**3.7 Import the chain back into EJBCA.** Admin web → the CA you created →
**Import CA certificate**, upload `root.crt` then `sandbox-issuing.crt`. The CA
status must move to `Active`. Publish `root.crl` so the URL in the
intermediate's CDP actually resolves.

---

## 4. Certificate profile — 1-year S/MIME

**Certificate Profiles → Add** → `VNC S/MIME 1y`, type *End Entity*.

| Setting | Value | Why |
|---|---|---|
| Validity | `1y` | your decision |
| Key algorithms | RSA 2048, 3072, 4096 | 2048 floor for interop; no ECDSA yet (§0) |
| Key Usage | `digitalSignature`, `keyEncipherment` | signing **and** decryption need both |
| Extended Key Usage | `emailProtection` | critical — see below |
| Subject Alternative Name | `rfc822Name`, **required** | this is the authoritative address |
| Basic Constraints | CA:FALSE, critical | |
| CRL Distribution Point | use CA default | |
| OCSP Service Locator (AIA) | `http://ca.sandbox.vnc.de/ejbca/publicweb/status/ocsp` | |
| Allow key recovery | **on** | see §7 |
| Allow subject DN override by CSR | **OFF** | load-bearing, see below |
| Allow extension override by CSR | **OFF** | load-bearing, see below |
| Allow subject alt name override by CSR | **OFF** | load-bearing, see below |

**The three override settings must be OFF, and this is the single most important
line in this document.**

The enrolment route deliberately does *not* inspect the CSR to police what it
asks for. It doesn't need to: the route supplies the subject and the
`rfc822Name` SAN itself, from addresses Stalwart confirmed the account may send
from, and the CSR contributes only a public key plus proof the requester holds
the matching private key.

That reasoning is only sound while EJBCA ignores the CSR's own subject and
extensions. Turn any of these overrides on and a hand-crafted CSR claiming
`rfc822Name=ceo@vnc.de` gets exactly that certificate — no code change, no
alert, and the enrolment route still looks correct in review. It is a
one-checkbox path from "authenticated users get certificates for their own
addresses" to "authenticated users get certificates for anyone's address".

Verify it rather than trusting the profile screen, once the route is live:

```bash
openssl req -new -key /tmp/t.key -subj "/CN=Impostor" -addext "subjectAltName=email:ceo@vnc.de" -out /tmp/t.csr
```

Submit that CSR through the enrolment route as an ordinary user. The certificate
that comes back must carry **your own** address, not `ceo@vnc.de`.

Two of these carry real weight:

**`emailProtection` EKU, and only that.** A certificate with no EKU is treated by
some clients as valid for *anything* — TLS server auth included. Constrain it.

**`rfc822Name` SAN required.** Modern clients bind the sender address from the
SAN, not the `emailAddress` DN attribute. Our forked plugin's fix-1 check
(`signerEmailMatch`, which refuses to auto-import a signer cert whose address
doesn't match the `From` header) now reads the address the same way clients do —
SAN first, and matched against *every* address the certificate carries. If EJBCA
issues certificates without an `rfc822Name` SAN, that check fails closed and
encryption silently never becomes available.

Populating the DN `emailAddress` attribute as well, for old Outlook, is safe —
but only as of finding 11. Until then the plugin read the DN attribute *in
preference to* the SAN and compared only the first address it found, so an EJBCA
certificate with both fields populated would have reported every genuine
signature as "signer ≠ From" and blocked the import. Covered now by
`vnc/plugins/smime/verify-address-binding.mjs`.

**End Entity Profiles → Add** → `VNC S/MIME User`:
- Default Certificate Profile: `VNC S/MIME 1y`; available: the same only
- Subject DN: `CN` required + modifiable, `O=VNC AG` and `C=CH` fixed
- Subject Alt Name: `rfc822Name` required, **and tick "Use entity email field"**
- Default CA: `VNC S/MIME Issuing CA Sandbox R1`

---

## 5. RA credential for the enrolment route

The webmail server — not the browser — calls the REST API. It needs its own
client certificate with *only* the authority to enrol end entities.

**5.1** Create a certificate profile `VNC RA Client` (End Entity, EKU
`clientAuth`, validity `1y`) and enrol one entity `CN=vncmail-ra-sandbox`
against it. Download as PKCS#12.

**5.2** Restrict it. **System Functions → Administrator Roles → Add** →
`VNCmail RA (sandbox)`:

| Rule | Access |
|---|---|
| `/ca_functionality/create_certificate` | Allow |
| `/ca/VNC S/MIME Issuing CA Sandbox R1` | Allow |
| `/endentityprofilesrules/VNC S/MIME User/**` | Allow |
| `/ra_functionality/revoke_end_entity` | Allow |
| everything else | **not granted** |

Match by the certificate's serial + issuer DN, not by CN. Do **not** give this
role `/administrator` or any `/system_functionality` rule: this credential lives
on an internet-facing pod, and the blast radius of it leaking should be "issue
and revoke S/MIME certs under one profile", not "reconfigure the CA".

**5.3** Load it into the webmail namespace:

```bash
kubectl -n vncmail create secret generic smime-ra \
  --from-file=client.p12=./vncmail-ra-sandbox.p12 \
  --from-literal=client-password='<p12 passphrase>' \
  --from-file=ca-chain.pem=./chain.pem
```

`chain.pem` is `sandbox-issuing.crt` followed by `root.crt`. The enrolment route
pins this chain when it connects to EJBCA on 8443 — it does not trust the public
root store, so EJBCA's self-signed server certificate (`TLS_SETUP_ENABLED=simple`)
is correct and expected here.

---

## 6. Verify the network policy actually enforces

Applying a NetworkPolicy on a CNI that doesn't implement it succeeds silently
and protects nothing. Prove it:

```bash
kubectl -n default run np-probe --rm -it --image=curlimages/curl --restart=Never -- \
  curl -sS -m 5 -k https://ejbca.vnc-ca.svc.cluster.local:8443/ejbca/ejbca-rest-api/v1/ca
```

This **must** time out or be refused. If it returns anything HTTP-shaped —
including a `401` — the policy is not being enforced and the REST API is exposed
cluster-wide. Check your CNI before continuing:

```bash
kubectl -n kube-system get pods | grep -iE "calico|cilium|flannel"
```

---

## 7. Key recovery is not optional here

S/MIME differs from TLS in a way that has bitten every organisation that
deployed it without thinking about this: **if a user loses their private key,
every message ever encrypted to them is permanently unreadable.** Not
inconvenient — gone. Re-issuing a certificate does not help, because the old
messages were encrypted to the old key.

So `Allow key recovery` in §4 is deliberate, and it is a real trade-off:

- **on** — EJBCA escrows the decryption key. Lost laptop is recoverable. But the
  CA database now contains material that decrypts users' mail, so §8 backup
  handling and the §5 role restrictions become load-bearing, and the escrow is
  something you must be able to explain to a user asking whether their mail is
  end-to-end encrypted. It is, from the wire's perspective; it is not, from the
  CA operator's.
- **off** — nobody but the user can ever read their mail, and a lost device is
  permanent data loss with no recourse.

For a corporate deployment where mail is a business record, escrow on is the
defensible choice, and it's what §4 sets. Decide this consciously — it is far
cheaper to turn on now than to explain later why three years of mail is gone.

If you keep it on, use a separate key-recovery role with two-person approval
rather than folding that authority into the RA credential.

---

## 8. Backup

`ejbca-db-data` contains the intermediate CA private key and — per §7 — escrowed
user decryption keys. A dump of it is equivalent to the CA itself.

```bash
kubectl -n vnc-ca exec deploy/ejbca-db -- sh -c \
  'mariadb-dump -u root -p"$MARIADB_ROOT_PASSWORD" --single-transaction ejbca' \
  | gzip > ejbca-$(date +%F).sql.gz
```

Encrypt before it leaves your machine — an unencrypted CA dump in object storage
is the whole hierarchy:

```bash
gpg --symmetric --cipher-algo AES256 ejbca-$(date +%F).sql.gz
```

Then to the shared R2 bucket (`vnc-backups1`) and **delete the plaintext**.
Restore-test it once, now, against a scratch namespace — an untested CA backup is
a belief, not a backup.

Not in this backup, by design and stored separately: the offline root key
(§3.6), `EJBCA_CRYPTO_TOKEN_PIN`, and the RA PKCS#12 passphrase.

---

## 9. Promotion to production

Nothing here is thrown away. Same root, new intermediate:

1. Bring `root.key` out of the safe; repeat §3.4–3.6 for
   `CN=VNC S/MIME Issuing CA R1` — sign it with the **same root**.
2. Duplicate the §4 profiles as `VNC S/MIME 1y (prod)` bound to the new CA.
3. Fresh RA credential and role for the prod webmail namespace (§5). Never share
   the sandbox one across environments.
4. Point the prod CDP/AIA at a stable production hostname. Those URLs are baked
   into every certificate for its full year, so get the hostname right *before*
   the first issuance.

The trust anchor on user devices does not change, and sandbox-issued
certificates keep validating.

## 10. SwissSign (P7, deferred)

The point of the `CaProvider` interface on the application side is that this
whole document becomes one implementation of it. Moving to SwissSign-issued
certificates — for ZertES/eIDAS-qualified signatures that external parties
validate without installing anything — is then a second implementation plus an
identity-verification step, not a rewrite of the enrolment flow.

What survives unchanged: in-browser key generation, CSR construction, the
enrolment route, storage, sign/encrypt/decrypt, the UI.
What changes: who signs the CSR, and the fact that a human must prove their
identity before a qualified certificate is issued — which is a process
requirement, not a code one.