docs: tomorrow-morning test basis (AI, S/MIME web+mobile, Theme); fix JMAP_SERVER_URL landmine in dev env example
Concrete, runnable test steps per area with explicit known-gaps sections so nothing reads as more finished than it is. Also documents the mobile S/MIME merge (vncmail-native main 7b89839) done this session. .env.dev.example: the documented relative JMAP_SERVER_URL 400s /api/auth/stalwart-context (resolveTrustedJmapUrl rejects relative URLs), silently breaking the real session-cookie flow that S/MIME enrollment, offline sync, and the AI server/retrieval routes all depend on. Switched the example to an absolute URL with an explanatory comment.
This commit is contained in:
Bernd Rodler
@@ -0,0 +1,232 @@
|
||||
# Test basis — 2026-08-06 morning
|
||||
|
||||
Covers everything shipped overnight: AI Assistant (web), S/MIME (web + mobile),
|
||||
and the 2-theme rebrand. For each area: how to get into a testable state,
|
||||
concrete steps with expected results, and — importantly — what's still a
|
||||
known gap so nothing here gets mistaken for more finished than it is.
|
||||
|
||||
All web steps assume `vncmail-plus` running locally via the built-in mock
|
||||
JMAP server (fastest path, no real Stalwart needed):
|
||||
|
||||
```bash
|
||||
cp .env.dev.example .env.local
|
||||
# then set, in .env.local:
|
||||
# AI_SERVER_BASE_URL=http://127.0.0.1:11434 (real Ollama on this Mac)
|
||||
# SMIME_CA_DEV_LOCAL=true
|
||||
npm run dev
|
||||
```
|
||||
|
||||
Open `http://localhost:3000` and **log in via the "Anmelden" (dev-mode
|
||||
login) button, not "Demo starten"**. Demo mode is a pure client-side
|
||||
in-memory session with no server-side cookie — AI's `server` class, S/MIME
|
||||
enrollment, and offline routes all need the real cookie the login button
|
||||
sets, and will look broken (401s) under Demo mode for reasons that have
|
||||
nothing to do with the features themselves.
|
||||
|
||||
---
|
||||
|
||||
## 1. AI Assistant (web)
|
||||
|
||||
Settings → AI Assistant.
|
||||
|
||||
### 1.1 Local (Ollama on this Mac)
|
||||
|
||||
1. Select provider **Local**. Model list should populate from the running
|
||||
`ollama serve` (confirm with `ollama list` in a terminal first).
|
||||
2. Ask a question in "Try it" with no special mail content, e.g. "reply with
|
||||
exactly the words LOCAL AI WORKS".
|
||||
3. **Expect:** exact echo back, no seat/entitlement banner (local is free,
|
||||
unmetered, never reaches this app's own backend).
|
||||
|
||||
### 1.2 Server (this app's backend → your Ollama)
|
||||
|
||||
1. Select provider **Server**. Model list comes from `/api/ai/server/models`
|
||||
(same models as Local, proxied — confirms the same-origin proxy path
|
||||
works, not just direct-to-loopback).
|
||||
2. Ask a first question. **Expect:** a small "seat assigned" notice on the
|
||||
*first* successful call for this user, none on subsequent calls.
|
||||
3. Check `data/admin-state/ai-metering.jsonl` on disk — a new line should
|
||||
appear with real `promptTokens`/`completionTokens`/`latencyMs` from the
|
||||
actual Ollama call, not placeholders.
|
||||
4. To test the entitlement *ceiling*: `curl -X PUT localhost:3000/api/admin/ai/entitlement -H 'Content-Type: application/json' -d '{"seatsTotal":0}'`
|
||||
(needs an admin session cookie), then retry a Server-class question as a
|
||||
*different* username. **Expect:** HTTP 402, "no licensed seats configured".
|
||||
Set `seatsTotal` back up afterwards.
|
||||
|
||||
### 1.3 Public (BYOK)
|
||||
|
||||
1. Add a profile: name, model, base URL, API key (any real provider you
|
||||
have a key for, or a fake key just to test the UI — the request itself
|
||||
will fail at the provider, not in this app).
|
||||
2. Add a *second* profile with a different name/model.
|
||||
3. In "Try it", use the **"Answer with"** selector to choose between the two
|
||||
profiles for the same question. **Expect:** each answers using its own
|
||||
configured model, no cross-talk.
|
||||
4. Remove a profile from the list. **Expect:** it disappears immediately,
|
||||
and its key is gone from `localStorage` (`DevTools → Application →
|
||||
Local Storage`, key prefix `vncmail.ai.key.`).
|
||||
|
||||
### 1.4 Retrieval (grounded answers over your own mail)
|
||||
|
||||
The demo/mock inbox includes a "Villa sul Lago" booking confirmation email.
|
||||
|
||||
1. On **Server** or **Local**, ask: *"When is check-in for the Villa sul
|
||||
Lago booking?"*
|
||||
2. **Expect:** a specific date/time, with a cited source referencing the
|
||||
actual email (not a generic non-answer). This exercises the full
|
||||
pipeline: real JMAP `Email/query`+`Email/get`, real Ollama embeddings
|
||||
(`nomic-embed-text`), cosine similarity, RRF fusion with the local
|
||||
text-match leg.
|
||||
|
||||
### 1.5 Known gaps — don't be surprised by these
|
||||
|
||||
- **No admin UI yet for any of the above.** Seat totals, model allow-lists,
|
||||
per-class on/off, usage — all real on the backend (§1.2, §1.3) but only
|
||||
reachable via `curl`/direct API calls today. A full spec + visual mockup
|
||||
for the admin console exists at
|
||||
[`docs/ADMIN-AI-POLICY-CONSOLE-SPEC.md`](ADMIN-AI-POLICY-CONSOLE-SPEC.md)
|
||||
— presented for approval, **intentionally not built** pending sign-off.
|
||||
- **BYOK provider allow-listing does not exist at all yet** — a user can
|
||||
point a Public profile at literally any URL. Not a regression, just never
|
||||
built; see the spec's §6.1 for the proposed (client-side-only) approach.
|
||||
- Tier (base/standard/pro) is stored but doesn't change behavior yet — only
|
||||
`seatsTotal` gates anything.
|
||||
|
||||
---
|
||||
|
||||
## 2. S/MIME — web
|
||||
|
||||
### 2.1 What to test today
|
||||
|
||||
With `SMIME_CA_DEV_LOCAL=true` set, the CA-issuance backend is real and
|
||||
independently verified (4 passing tests: CSR signature verification, full
|
||||
chain verification via `leaf.verify(caCert)`, correct SAN/rfc822Name
|
||||
addresses, `emailProtection` EKU present — see
|
||||
`lib/smime-ca/__tests__/local-dev-provider.test.ts`).
|
||||
|
||||
There is **no user-facing "Enroll" button wired up yet** — that was a
|
||||
deliberate scope decision made overnight (see the commit message on
|
||||
`c2c07293`), not an oversight: this plugin has known, still-open
|
||||
security-audit findings, and generating brand-new CSR/key-import code
|
||||
against it at 1am risked introducing a new one rather than closing an old
|
||||
one. So today, test the backend directly:
|
||||
|
||||
```bash
|
||||
# Generate a throwaway CSR (any tool - openssl shown here):
|
||||
openssl req -new -newkey rsa:2048 -nodes -keyout /tmp/test.key \
|
||||
-out /tmp/test.csr -subj "/CN=Test User"
|
||||
|
||||
# Call the enrollment route directly (needs an authenticated session cookie
|
||||
# from the "Anmelden" login above - copy it from DevTools → Application →
|
||||
# Cookies, or use a logged-in curl session):
|
||||
curl -X POST localhost:3000/api/smime/enroll \
|
||||
-H 'Content-Type: application/json' \
|
||||
--cookie "<your session cookie>" \
|
||||
-d "{\"csrPem\": $(python3 -c 'import json,sys; print(json.dumps(open("/tmp/test.csr").read()))'), \"addresses\": [\"you@example.com\"]}"
|
||||
```
|
||||
|
||||
**Expect:** a JSON response with `certificatePem`, `chainPem`, `serialNumber`,
|
||||
`notAfter`. Verify it's a real cert: `openssl x509 -in <(echo "$certificatePem") -noout -text`
|
||||
should show issuer `VNCmail+ LOCAL DEV S/MIME CA`, your `you@example.com` in
|
||||
Subject Alternative Name, and Extended Key Usage including `E-mail
|
||||
Protection`.
|
||||
|
||||
### 2.2 Known gap — the important one
|
||||
|
||||
**Nobody can actually enroll from the UI today.** The plugin's Settings
|
||||
section still has no "Get a certificate" button, no client-side CSR
|
||||
generation, and no wiring to import an issued cert into its existing
|
||||
encrypted-at-rest key storage (`vnc/plugins/smime/src/pkcs12.js`'s
|
||||
AES-GCM+PBKDF2 convention). The plugin itself is confirmed *mounted*
|
||||
(privileged iframe, same-origin fetch works without a bridge) with 0 certs
|
||||
imported — that's expected, not a bug. This is real, well-scoped follow-up
|
||||
work, not blocked on anything — just deliberately not rushed overnight.
|
||||
|
||||
---
|
||||
|
||||
## 3. S/MIME — mobile (vncmail-native)
|
||||
|
||||
Unlike web, mobile got the **full stack**, merged to `main` this session
|
||||
(`7b89839`, combining the S/MIME work with the AI/offline-sync work that
|
||||
landed on `main` in parallel): real CMS sign/verify/encrypt/decrypt on
|
||||
`node-forge` primitives (wire-compatible with the audited webmail plugin,
|
||||
verified byte-for-byte against OpenSSL), PKCS#12 import, keys wrapped
|
||||
AES-256-GCM/PBKDF2 in `expo-secure-store` (Keychain/Keystore — never
|
||||
AsyncStorage in the clear).
|
||||
|
||||
### 3.1 Test steps
|
||||
|
||||
1. Build and run on a simulator/device (`npm run ios` / `npm run android`
|
||||
from `vncmail-native`).
|
||||
2. Settings → S/MIME → **Import certificate**. Use a real `.p12` file (an
|
||||
OpenSSL-generated one works fine for testing) and its passphrase.
|
||||
3. **Expect:** cert imported, persists across an app restart, a wrong
|
||||
passphrase is rejected, the right one unlocks it.
|
||||
4. Compose a new email to yourself → enable **Sign** (and/or **Encrypt**) →
|
||||
send.
|
||||
5. Open the received message. **Expect:** a signature banner showing
|
||||
verified/signed-by, and if encrypted, the body decrypts and displays
|
||||
normally.
|
||||
6. Try receiving a deliberately malformed/unauthenticated-cipher message
|
||||
(or just trust the 259-line hardening test suite —
|
||||
`src/lib/__tests__/smime-hardening.test.ts` — covers this without
|
||||
needing to hand-craft one). **Expect:** unauthenticated content renders
|
||||
as inert plaintext, never live HTML.
|
||||
|
||||
### 3.2 Known-good, already verified
|
||||
|
||||
479+ unit tests (834 after the merge with the AI/sync work) pass, including
|
||||
a full round-trip test. `typecheck` is clean. Also verified live on an
|
||||
Android emulator during that session: real OpenSSL-3-generated `.p12`
|
||||
imported through the actual device file picker.
|
||||
|
||||
### 3.3 Known gap
|
||||
|
||||
No cross-checking was done *this session* between the mobile
|
||||
implementation and the CA now issuing certs on web (§2) — they were built
|
||||
independently and haven't been tested importing a *web-CA-issued*
|
||||
certificate into the mobile app. Worth doing once web's Enroll UI (§2.2)
|
||||
exists, not before.
|
||||
|
||||
---
|
||||
|
||||
## 4. Theme
|
||||
|
||||
Settings → Themes. Exactly 2 built-in themes now: **SRC** (default, Swiss
|
||||
red `#D52B1E`) and **VNClagoon** (navy/cyan). The other 6 generic built-ins
|
||||
(Nord, Catppuccin, Solarized, Roundcube Elastic, Aurora Glass, plus the old
|
||||
default) are hidden via admin theme policy, not deleted — an admin could
|
||||
re-enable them, a normal user can't see them.
|
||||
|
||||
### 4.1 Test steps
|
||||
|
||||
1. Confirm SRC is active on first load (no theme ever selected before).
|
||||
2. Switch to VNClagoon. **Expect:** a "Theme activated" toast, and the
|
||||
chrome genuinely re-skins (navy/cyan replaces red throughout — sidebar,
|
||||
buttons, unread markers, not just an accent color here and there).
|
||||
3. Switch back to SRC. **Expect:** same, in reverse.
|
||||
4. Search the UI (empty-state illustrations, manifest/PWA name, login page,
|
||||
locale strings in a couple of languages) for the string "Bulwark" — all
|
||||
24 locale files were rebranded to "VNCmail+" this session, plus
|
||||
`app/manifest.ts`, demo fixtures, and the inbox empty-state logo, which
|
||||
now resolves through the same theme-logo system every other themed
|
||||
surface uses instead of a hardcoded path.
|
||||
|
||||
### 4.2 Known gap
|
||||
|
||||
Not exhaustively re-checked for every locale/every screen — the 24-file
|
||||
find-and-replace was mechanical (script-driven) and spot-checked, not
|
||||
manually walked screen-by-screen in each language. If a stray "Bulwark"
|
||||
turns up somewhere, it's most likely a screen that wasn't in scope of the
|
||||
locale-string sweep (e.g. a hardcoded string in a component rather than a
|
||||
translation key) — same category as the logo fix above, worth a quick grep
|
||||
(`grep -ril bulwark`) if one surfaces.
|
||||
|
||||
---
|
||||
|
||||
## 5. Offline / mail-index
|
||||
|
||||
Already QA'd and fixed this session (not re-listed as a to-do): a leaked
|
||||
SQLite handle on `PRAGMA` failure, and contacts/files that weren't being
|
||||
removed from the local index on delete. Both fixed and covered by tests —
|
||||
no separate manual test needed tomorrow unless something else turns up.
|
||||
Reference in New Issue
Block a user