Bernd Rodler
eda3302298
Both were "known gaps" in the original doc; rewrite those sections to reflect the actual shipped, live-verified state and note what's newly open instead (BYOK allow-list is advisory-only, consent text has no client-side reader yet).
253 lines
12 KiB
Markdown
253 lines
12 KiB
Markdown
# Test basis — 2026-08-06 morning
|
||
|
||
Covers everything shipped overnight: AI Assistant (web), S/MIME (web + mobile),
|
||
and the 2-theme rebrand. For each area: how to get into a testable state,
|
||
concrete steps with expected results, and — importantly — what's still a
|
||
known gap so nothing here gets mistaken for more finished than it is.
|
||
|
||
All web steps assume `vncmail-plus` running locally via the built-in mock
|
||
JMAP server (fastest path, no real Stalwart needed):
|
||
|
||
```bash
|
||
cp .env.dev.example .env.local
|
||
# then set, in .env.local:
|
||
# AI_SERVER_BASE_URL=http://127.0.0.1:11434 (real Ollama on this Mac)
|
||
# SMIME_CA_DEV_LOCAL=true
|
||
npm run dev
|
||
```
|
||
|
||
Open `http://localhost:3000` and **log in via the "Anmelden" (dev-mode
|
||
login) button, not "Demo starten"**. Demo mode is a pure client-side
|
||
in-memory session with no server-side cookie — AI's `server` class, S/MIME
|
||
enrollment, and offline routes all need the real cookie the login button
|
||
sets, and will look broken (401s) under Demo mode for reasons that have
|
||
nothing to do with the features themselves.
|
||
|
||
---
|
||
|
||
## 1. AI Assistant (web)
|
||
|
||
Settings → AI Assistant.
|
||
|
||
### 1.1 Local (Ollama on this Mac)
|
||
|
||
1. Select provider **Local**. Model list should populate from the running
|
||
`ollama serve` (confirm with `ollama list` in a terminal first).
|
||
2. Ask a question in "Try it" with no special mail content, e.g. "reply with
|
||
exactly the words LOCAL AI WORKS".
|
||
3. **Expect:** exact echo back, no seat/entitlement banner (local is free,
|
||
unmetered, never reaches this app's own backend).
|
||
|
||
### 1.2 Server (this app's backend → your Ollama)
|
||
|
||
1. Select provider **Server**. Model list comes from `/api/ai/server/models`
|
||
(same models as Local, proxied — confirms the same-origin proxy path
|
||
works, not just direct-to-loopback).
|
||
2. Ask a first question. **Expect:** a small "seat assigned" notice on the
|
||
*first* successful call for this user, none on subsequent calls.
|
||
3. Check `data/admin-state/ai-metering.jsonl` on disk — a new line should
|
||
appear with real `promptTokens`/`completionTokens`/`latencyMs` from the
|
||
actual Ollama call, not placeholders.
|
||
4. To test the entitlement *ceiling*: `curl -X PUT localhost:3000/api/admin/ai/entitlement -H 'Content-Type: application/json' -d '{"seatsTotal":0}'`
|
||
(needs an admin session cookie), then retry a Server-class question as a
|
||
*different* username. **Expect:** HTTP 402, "no licensed seats configured".
|
||
Set `seatsTotal` back up afterwards.
|
||
|
||
### 1.3 Public (BYOK)
|
||
|
||
1. Add a profile: name, model, base URL, API key (any real provider you
|
||
have a key for, or a fake key just to test the UI — the request itself
|
||
will fail at the provider, not in this app).
|
||
2. Add a *second* profile with a different name/model.
|
||
3. In "Try it", use the **"Answer with"** selector to choose between the two
|
||
profiles for the same question. **Expect:** each answers using its own
|
||
configured model, no cross-talk.
|
||
4. Remove a profile from the list. **Expect:** it disappears immediately,
|
||
and its key is gone from `localStorage` (`DevTools → Application →
|
||
Local Storage`, key prefix `vncmail.ai.key.`).
|
||
|
||
### 1.4 Retrieval (grounded answers over your own mail)
|
||
|
||
The demo/mock inbox includes a "Villa sul Lago" booking confirmation email.
|
||
|
||
1. On **Server** or **Local**, ask: *"When is check-in for the Villa sul
|
||
Lago booking?"*
|
||
2. **Expect:** a specific date/time, with a cited source referencing the
|
||
actual email (not a generic non-answer). This exercises the full
|
||
pipeline: real JMAP `Email/query`+`Email/get`, real Ollama embeddings
|
||
(`nomic-embed-text`), cosine similarity, RRF fusion with the local
|
||
text-match leg.
|
||
|
||
### 1.5 Known gaps — don't be surprised by these
|
||
|
||
- Tier (base/standard/pro) is stored but doesn't change behavior yet — only
|
||
`seatsTotal` gates anything. The admin console (§1.6) labels the picker
|
||
as cosmetic for exactly this reason.
|
||
|
||
### 1.6 Admin AI Policy console — now built, test as an admin
|
||
|
||
Approved and implemented after the spec+mockup review. Admin panel → **AI**
|
||
(new tab, next to Policy).
|
||
|
||
1. **Provider classes** — toggle Server off, then try the Server class as a
|
||
user. **Expect:** 403 "disabled by admin policy" instead of a normal
|
||
answer (real enforcement in `/api/ai/server/chat`, not just a hidden
|
||
radio button). Toggle back on.
|
||
2. **Server model allow-list** — switch to "Restrict to selected", add one
|
||
model name, save. **Expect:** the model picker in Settings → AI
|
||
Assistant now shows only that model, *and* a direct API call with a
|
||
different model name gets 403 (checked at the chat chokepoint too).
|
||
3. **Seats** — change "Seats licensed", watch the "N of M assigned" count
|
||
update; **Revoke** a seat and confirm that user gets 402 again on next
|
||
use (frees the seat for someone else, matches `checkAndAssignSeat`'s
|
||
first-use semantics).
|
||
4. **Usage** — real rows from `ai-metering.jsonl`, not placeholders — should
|
||
match whatever calls you made in §1.2.
|
||
5. **Retrieval & consent** — toggle retrieval off, then ask a mail-grounded
|
||
question. **Expect:** 403 from `/api/ai/retrieve`, plain chat still
|
||
works. Edit the consent text, bump the version — no user-facing consent
|
||
*prompt* is wired up to read this yet (that's the one open item: the
|
||
text/version are stored and served over `/api/ai/policy`, but no UI
|
||
currently shows it to end users before first BYOK use — worth flagging
|
||
if that matters before wider rollout).
|
||
|
||
### 1.7 Known gaps, updated
|
||
|
||
- BYOK provider allow-listing is real now but **client-side/advisory
|
||
only** — a user's own browser checks it at profile-save time
|
||
(`ai-assistant-settings.tsx`), there's no server-side network boundary
|
||
enforcing it. Matches the spec's §6.1 recommendation; revisit if that
|
||
needs to become a hard boundary.
|
||
- The consent text/version admins set has no reader yet on the client side
|
||
(see §1.6 point 5) — stored and served, not yet displayed.
|
||
|
||
---
|
||
|
||
## 2. S/MIME — web
|
||
|
||
### 2.1 What to test today — now the full UI flow
|
||
|
||
With `SMIME_CA_DEV_LOCAL=true` set, enrollment is fully wired end to end and
|
||
live-verified (not just unit-tested): Settings → S/MIME → **Get a
|
||
certificate** → enter a storage passphrase → a real certificate comes back
|
||
(RSA-2048, correct validity window, real fingerprint) and appears in "Your
|
||
keys". **Unlock** with the same passphrase round-trips correctly through
|
||
the identical encrypted-at-rest path a PKCS#12 import would use.
|
||
|
||
1. Log in via "Anmelden" (real session-cookie flow, not "Demo starten" —
|
||
see the caveat at the top of this doc).
|
||
2. Settings → S/MIME → **Get a certificate**.
|
||
3. Enter any passphrase, confirm.
|
||
4. **Expect:** a new key card appears — `RSA-2048 · valid <today> – <+397
|
||
days>`, a real SHA-256 fingerprint, Unlock/Delete buttons.
|
||
5. Click **Unlock**, enter the same passphrase. **Expect:** button flips to
|
||
**Lock** — the private key decrypted correctly.
|
||
6. Compose a message to yourself, toggle Sign/Encrypt, send, then open it —
|
||
this exercises the CA-issued cert through the plugin's existing
|
||
sign/encrypt/decrypt/verify code paths (unchanged by tonight's work,
|
||
since the key record shape is identical to a PKCS#12 import's).
|
||
|
||
The CA-issuance backend itself is additionally covered by 4 independent
|
||
unit tests (CSR signature verification, full chain verification via
|
||
`leaf.verify(caCert)`, correct SAN/rfc822Name addresses, `emailProtection`
|
||
EKU present) — see `lib/smime-ca/__tests__/local-dev-provider.test.ts`.
|
||
|
||
### 2.2 Known gap — real but narrow
|
||
|
||
No cross-testing yet between a **web-CA-issued** certificate and the
|
||
**mobile** S/MIME implementation (§3) — worth doing once both have been
|
||
used interactively at least once, not before. Also unchanged from
|
||
yesterday: `SMIME_CA_DEV_LOCAL` is explicitly a stand-in for the real
|
||
EJBCA (needs a client mTLS credential this environment doesn't have) —
|
||
switching to production CA is a config change (`SMIME_CA_URL` +
|
||
credentials), not a code change.
|
||
|
||
---
|
||
|
||
## 3. S/MIME — mobile (vncmail-native)
|
||
|
||
Unlike web, mobile got the **full stack**, merged to `main` this session
|
||
(`7b89839`, combining the S/MIME work with the AI/offline-sync work that
|
||
landed on `main` in parallel): real CMS sign/verify/encrypt/decrypt on
|
||
`node-forge` primitives (wire-compatible with the audited webmail plugin,
|
||
verified byte-for-byte against OpenSSL), PKCS#12 import, keys wrapped
|
||
AES-256-GCM/PBKDF2 in `expo-secure-store` (Keychain/Keystore — never
|
||
AsyncStorage in the clear).
|
||
|
||
### 3.1 Test steps
|
||
|
||
1. Build and run on a simulator/device (`npm run ios` / `npm run android`
|
||
from `vncmail-native`).
|
||
2. Settings → S/MIME → **Import certificate**. Use a real `.p12` file (an
|
||
OpenSSL-generated one works fine for testing) and its passphrase.
|
||
3. **Expect:** cert imported, persists across an app restart, a wrong
|
||
passphrase is rejected, the right one unlocks it.
|
||
4. Compose a new email to yourself → enable **Sign** (and/or **Encrypt**) →
|
||
send.
|
||
5. Open the received message. **Expect:** a signature banner showing
|
||
verified/signed-by, and if encrypted, the body decrypts and displays
|
||
normally.
|
||
6. Try receiving a deliberately malformed/unauthenticated-cipher message
|
||
(or just trust the 259-line hardening test suite —
|
||
`src/lib/__tests__/smime-hardening.test.ts` — covers this without
|
||
needing to hand-craft one). **Expect:** unauthenticated content renders
|
||
as inert plaintext, never live HTML.
|
||
|
||
### 3.2 Known-good, already verified
|
||
|
||
479+ unit tests (834 after the merge with the AI/sync work) pass, including
|
||
a full round-trip test. `typecheck` is clean. Also verified live on an
|
||
Android emulator during that session: real OpenSSL-3-generated `.p12`
|
||
imported through the actual device file picker.
|
||
|
||
### 3.3 Known gap
|
||
|
||
No cross-checking was done *this session* between the mobile
|
||
implementation and the CA now issuing certs on web (§2) — they were built
|
||
independently and haven't been tested importing a *web-CA-issued*
|
||
certificate into the mobile app. Worth doing once web's Enroll UI (§2.2)
|
||
exists, not before.
|
||
|
||
---
|
||
|
||
## 4. Theme
|
||
|
||
Settings → Themes. Exactly 2 built-in themes now: **SRC** (default, Swiss
|
||
red `#D52B1E`) and **VNClagoon** (navy/cyan). The other 6 generic built-ins
|
||
(Nord, Catppuccin, Solarized, Roundcube Elastic, Aurora Glass, plus the old
|
||
default) are hidden via admin theme policy, not deleted — an admin could
|
||
re-enable them, a normal user can't see them.
|
||
|
||
### 4.1 Test steps
|
||
|
||
1. Confirm SRC is active on first load (no theme ever selected before).
|
||
2. Switch to VNClagoon. **Expect:** a "Theme activated" toast, and the
|
||
chrome genuinely re-skins (navy/cyan replaces red throughout — sidebar,
|
||
buttons, unread markers, not just an accent color here and there).
|
||
3. Switch back to SRC. **Expect:** same, in reverse.
|
||
4. Search the UI (empty-state illustrations, manifest/PWA name, login page,
|
||
locale strings in a couple of languages) for the string "Bulwark" — all
|
||
24 locale files were rebranded to "VNCmail+" this session, plus
|
||
`app/manifest.ts`, demo fixtures, and the inbox empty-state logo, which
|
||
now resolves through the same theme-logo system every other themed
|
||
surface uses instead of a hardcoded path.
|
||
|
||
### 4.2 Known gap
|
||
|
||
Not exhaustively re-checked for every locale/every screen — the 24-file
|
||
find-and-replace was mechanical (script-driven) and spot-checked, not
|
||
manually walked screen-by-screen in each language. If a stray "Bulwark"
|
||
turns up somewhere, it's most likely a screen that wasn't in scope of the
|
||
locale-string sweep (e.g. a hardcoded string in a component rather than a
|
||
translation key) — same category as the logo fix above, worth a quick grep
|
||
(`grep -ril bulwark`) if one surfaces.
|
||
|
||
---
|
||
|
||
## 5. Offline / mail-index
|
||
|
||
Already QA'd and fixed this session (not re-listed as a to-do): a leaked
|
||
SQLite handle on `PRAGMA` failure, and contacts/files that weren't being
|
||
removed from the local index on delete. Both fixed and covered by tests —
|
||
no separate manual test needed tomorrow unless something else turns up.
|