diff --git a/docs/VNCMAIL-NATIVE-BUILD-MANUAL.md b/docs/VNCMAIL-NATIVE-BUILD-MANUAL.md new file mode 100644 index 00000000..49d6a649 --- /dev/null +++ b/docs/VNCMAIL-NATIVE-BUILD-MANUAL.md @@ -0,0 +1,280 @@ +# VNCmail+ Native & Desktop Client — Build Manual + +Status: living document, last updated 2026-08-04. This is the canonical reference for the +program that takes VNCmail+ (Bulwark) beyond the hosted webmail: an Electron desktop client, a +React Native mobile client, and a self-hosted push relay, working toward true offline mail with +an encrypted local index. It consolidates everything decided and built so far across three +repositories, so nothing lives only in chat history or a session's memory. + +Companion documents: +- `docs/OFFLINE-CLIENT-ARCHITECTURE.md` (in `~/vncmail-plus`) — the original gap analysis this + program is based on. +- `~/.claude/skills/VNCprodbuild/SKILL.md` — the step-by-step build plan this manual reports + progress against. That file is the operational checklist; this file is the narrative reference. + +--- + +## 1. Why this program exists + +Bulwark/VNCmail+ is a Next.js JMAP webmail app. As shipped, it has zero offline capability: the +service worker caches nothing by design, there's no local mail store, no local search index, and +no mobile or desktop native client. The goal of this program is to change that — ship a desktop +app, a mobile app, real push notifications, and (eventually) a true offline-first local data +layer with an encrypted search index — without re-deriving work that already exists upstream or +duplicating effort across repos. + +The single most important strategic fact discovered along the way: **an upstream React Native +mobile client already exists and already solves most of what looked like the hardest problems** +(auth, multi-account, device pairing, Android push). Building a second mobile client from +scratch (e.g. wrapping the webmail in Capacitor) would have thrown that away for no reason. The +whole shape of this program reflects that discovery — see §4. + +## 2. Repository map + +All three repos are AGPL-3.0 forks of the upstream Bulwark project (`bulwarkmail` on GitHub), +owned by `brvncde-dotcom`: + +| Repo | Forked from | Purpose | Local path | +|---|---|---|---| +| `vncmail-plus` | `bulwarkmail/webmail` | The Next.js webmail app itself — mail, calendar, contacts, files, admin, plugins. Deploys as a container on microk8s at `vncmail.sandbox.vnc.de`. | `~/vncmail-plus` (⚠️ shared checkout — see §8) | +| `vncmail-native` | `bulwarkmail/native` | React Native/Expo mobile client (Android + iOS). Beta/WIP upstream. | `~/vncmail-native` | +| `vncmail-relay` | `bulwarkmail/relay` | Push notification relay — terminates JMAP `PushSubscription` pushes, forwards to FCM (mobile) or Web Push (PWA/desktop). Self-hosted per the decision in §4. | `~/vncmail-relay` | + +The webmail's Electron desktop work happens in a **dedicated worktree**, not the shared +checkout directly: `~/worktrees/vncmail-electron`, branch `claude/electron-desktop`, based off +`vncmail-plus`'s `dev` branch. This will eventually become a PR into `dev`. + +Backend: Stalwart Mail Server (`stalwartlabs/mail-server`), sandbox instance at +`stalwart.sandbox.vnc.de`, speaking JMAP (mail/calendar/contacts), SMTP, IMAP, and ManageSieve. + +## 3. Architecture recap + +**The core blocker for "true offline":** `lib/jmap/client.ts` in the webmail is pure `fetch()`, +zero Node dependencies — it's portable into any WebView, Electron renderer, or React Native +context unchanged. But everything *around* it in the webmail — auth-cookie encryption +(`lib/auth/crypto.ts`, uses Node's `node:crypto`), the push relay wiring, and every `app/api/**` +route — is server-dependent. A native shell that just points a WebView at a bundled static +export of the webmail won't work without either: + +- **Option A — remote shell.** The native wrapper loads the *hosted* URL. Fast, gets native push + and an installable binary, but requires connectivity for every screen — not offline. +- **Option B — true offline-first.** The client authenticates and syncs JMAP data directly against + Stalwart, keeps a local encrypted store, and only needs connectivity to *sync*, not to *read*. + +For **desktop**, this fork-in-the-road barely matters: Electron can bundle the webmail's own +standalone Next.js server (the same artifact the `Dockerfile` already produces for the Docker +image) inside its Node runtime and point a `BrowserWindow` at `localhost`. That's Option A and B +at the same time, practically for free — see §5. + +For **mobile**, the fork-in-the-road is real, which is why §4's discovery mattered so much: it +meant Option A was already mostly done upstream, letting the plan skip straight to figuring out +what Option B (the real offline engine) needs — instead of re-building Option A from scratch in +Capacitor first. + +## 4. Decision log + +Every entry here was an explicit `[DECISION]` gate in the `VNCprodbuild` skill — resolved either +by direct research/verification or by explicit user sign-off. Dates are when each was resolved. + +| Date | Decision | Resolution | Why | +|---|---|---|---| +| 2026-08-04 | Does an offline cache need to support multiple accounts per device? | **Yes** | The webmail already has an `account-registry` store; the mobile offline cache must isolate per-account, including per-account SQLCipher keys later. | +| 2026-08-04 | Does an upstream React Native app already solve push/pairing? | **Yes — `bulwarkmail/native`** has multi-account JMAP auth, full QR cross-device pairing, and Android FCM push via `bulwarkmail/relay`. Forked to `vncmail-native`. | Duplicating working auth/pairing/push code in a fresh Capacitor wrapper has no upside. **This flipped the entire mobile strategy** — see §6. | +| 2026-08-04 | Self-host the push relay, or depend on upstream's shared hosted instance? | **Self-host.** Forked `bulwarkmail/relay` → `vncmail-relay`. | Keeps push metadata (FCM tokens, timing) on VNC infrastructure rather than a third party's. | +| 2026-08-04 | Which SQLite library for the eventual SQLCipher-backed local index, and what Expo workflow? | **`expo-sqlite`'s official `useSQLCipher` config-plugin option** (verified via web search — Android/iOS/macOS support, [docs](https://docs.expo.dev/versions/latest/sdk/sqlite/)), not a third-party binding. **Stay Continuous Native Generation** (don't commit `ios`/`android`, let `expo prebuild` regenerate them) rather than going fully bare. | The official plugin already covers this. SQLCipher is unusable in Expo Go — day-to-day development necessarily moves to a custom dev client, which the user explicitly accepted. Going bare would turn every future merge from upstream `bulwarkmail/native` into a native-project merge conflict, for no offsetting benefit. | +| 2026-08-04 | Electron's background/foreground notification strategy: JMAP WebSocket push, polling, or quit-to-tray? | **JMAP WebSocket push.** Confirmed live and available: the Stalwart sandbox's JMAP session resource advertises `urn:ietf:params:jmap:websocket` with `supportsPush: true` (`wss://stalwart.sandbox.vnc.de/jmap/ws`), independently confirmed by two separate build agents. | Lower latency, no polling/backoff logic to write, and it's not hypothetical — it's live on the server this build already targets. | +| 2026-08-04 | Code-signing: Apple Developer ID? Windows cert? | **Apple: yes** (also unblocks Phase 2's iOS push) — **human must actually enroll**, no agent can create the account or pay the ~$99/yr fee. **Windows: yes, eventually** — but ship unsigned for now during this build phase. | Signing is a purchase/account-creation action, categorically outside what an agent can do. | + +## 5. Phase 1 — Electron desktop client + +**Location:** `~/worktrees/vncmail-electron`, branch `claude/electron-desktop`, 7+ commits ahead +of `dev` as of this writing. Not pushed, no PR yet — review the worktree directly first. + +### What exists + +- `electron/main.ts` — boots the exact standalone Next.js server artifact the `Dockerfile` + already produces, as a child process on a random localhost port; opens a `BrowserWindow` + pointed at it. No parallel server-bundling approach was invented. +- `electron/preload.ts` — `contextBridge` exposing `window.vnc.isElectron` and + `window.vnc.showNotification(title, options)`, wired to Electron's native `Notification` API in + the main process. +- `e2e/electron-smoke.spec.ts` + `playwright.electron.config.ts` — the regression gate, using + Playwright's `_electron.launch()`. Run via `npm run test:electron`. Verified green, including + against a real packaged (`--dir`) build, not just the dev skeleton. +- `electron-builder.config.js` + `scripts/assemble-standalone.mjs` + `scripts/build-electron.mjs` + — packaging for macOS (`dmg`/`zip`, x64+arm64), Windows (`nsis`), Linux (`AppImage`/`deb`). + Currently unsigned. +- `electron-updater` wired against GitHub Releases (`brvncde-dotcom/vncmail-plus`), defensively + wrapped so a failed update check never crashes the app. +- `.github/workflows/electron-build.yml` — CI matrix (mac/win/linux) with `npm run test:electron` + as a required gate before packaging/upload. + +### How to build and run it locally + +```bash +cd ~/worktrees/vncmail-electron +npm install +npm run electron:dev # dev loop against the local Next dev server +npm run build:standalone # produces the standalone server artifact (same as Docker uses) +npm run build:electron # packages via electron-builder (unsigned) +npm run test:electron # the smoke-test regression gate +``` + +### Two real bugs found and fixed while building this (worth knowing about) + +1. **Repo-wide eslint gap.** `vnc/plugins/smime` (an independent sub-package) was missing from + the eslint ignore list, so `npm run lint` / the husky pre-commit hook failed for *any* commit + touching that path on `dev`, regardless of what changed. Fixed alongside `repos/**`/ + `examples/**`. **Flag this for whoever reviews the eventual PR** — it's a shared-config fix + unrelated to Electron, worth landing on `dev` on its own merits. +2. **electron-builder `extraResources` footgun.** electron-builder's resource-copy step + unconditionally drops any directory literally named `node_modules` when copying + `extraResources` — it was silently stripping the bundled standalone server's dependencies and + crashing on launch with `Cannot find module 'next'`. Caught only because the build was + actually launched and tested, not just configured. Worth remembering for any future + electron-builder work generally, not just this project. + +### Still open + +- **JMAP WebSocket push implementation** (skill steps 6-7) — decision resolved (§4), build not + yet done as of this manual's last update; check the skill's status log or task tracker for + current state. +- **Code signing** — blocked on the human actually enrolling in the Apple Developer Program + (§4). Once done, wiring the signing identity + notarization into `electron-builder` and CI + secrets is a config change, not a rewrite — the current config is structured for it. +- **App icon** — using the 512×512 PWA icon as a stand-in. + `public/branding/Bulwark_Icon_App.svg` should be rasterized at 1024×1024+ for a proper icon; no + SVG rasterization tooling was available in-agent. +- **Internal dogfood gate** — a human should install an unsigned build locally and sign off on + UX before this goes any further (wider rollout, PR, etc.). + +## 6. Phase 2 — Native mobile client + push relay + +### 6.1 `vncmail-native` — what it already had vs. what this program added + +Upstream `bulwarkmail/native` (forked as-is, no rewrite) already ships: + +- Multi-account JMAP sign-in against any server. +- Full QR-code cross-device pairing (`src/screens/LoginScreen.tsx`, `QrScanModal`, + `redeemPairingCode`/OAuth handoff in `src/lib/oauth.ts`). +- Android push notifications via FCM, dispatched through `bulwarkmail/relay`. +- A basic offline mail cache (`src/lib/offline-sync.ts`, `src/stores/offline-cache-store.ts`) — + bulk-downloads the last N days of mail via `Email/query`+`Email/get` into AsyncStorage, with a + size cap and eviction. **Not** the delta-sync/SQLCipher/FTS engine §7 describes — a periodic + bulk re-download, not incremental sync, plain JSON not an encrypted database. +- Android + iOS release pipelines already working (`release-android.yml` sideloads an APK from + GitHub Releases; `release-ios.yml` + `docs/ios-release.md` ship to TestFlight) — iOS *builds* + already work, just without push (Android-only so far per its own README). + +This program's first pass (2026-08-04) added, without touching any of the above: +- Verified `npm install`, typecheck, and the existing test suite all pass cleanly (429/430 tests; + one pre-existing, unrelated transform failure in `src/stores/__tests__/auth-store.test.ts`, not + introduced by this work — worth a look eventually, not urgent). +- Confirmed live reachability to `stalwart.sandbox.vnc.de` (HTTP 307 → `/jmap/session`, valid + JMAP session JSON returned) — and incidentally re-confirmed the WebSocket push capability from + §4/§5. +- Added `.github/workflows/android-emulator-smoke.yml` — builds the debug APK, boots a cached + AVD via `reactivecircus/android-emulator-runner`, installs, launches the app, fails on process + death or a `FATAL EXCEPTION` in logcat within a settle window. + +### How to build and run it locally + +```bash +cd ~/vncmail-native +npm install +npx expo start # Expo Go works fine UNTIL SQLCipher is added (§4) — see note below +``` + +**Once SQLCipher work starts (§7):** switch to a custom dev client — `npx expo prebuild` + +`npx expo run:android` / `npx expo run:ios`, or an EAS development build. Expo Go cannot run an +app with `useSQLCipher` enabled. Do not commit the generated `ios`/`android` directories — +Continuous Native Generation regenerates them from config plugins on each build (§4). + +### 6.2 `vncmail-relay` — self-hosted push relay + +Forked as-is from `bulwarkmail/relay`. This program added: + +- `deploy/k8s/{namespace,pvc,secret.example,deployment,service,ingress,kustomization}.yaml` + + `deploy/k8s/README.md` — mirrors the conventions already used to deploy `vncmail-plus` on + microk8s (same namespace, same Recreate-strategy/PVC pattern). **One deliberately unresolved + item:** the relay's own Dockerfile creates its runtime user via unpinned `adduser -S` (unlike + `vncmail-plus`'s documented uid 1001) — `runAsUser`/`fsGroup` are left unset in the manifest + with instructions to verify against the real built image before first deploy, rather than + guessing a UID. +- `.github/workflows/docker-publish.yml` — publishes to `ghcr.io/brvncde-dotcom/vncmail-relay`, + same multi-arch buildx/digest-merge structure as `vncmail-plus`'s own publish workflow. +- `SETUP-VNC.md` — documents a generated VAPID keypair (values are in that file only, referenced + by name — not the actual secret — in `secret.example.yaml`'s placeholders) and flags what's + still human-owned before this can go live: a dedicated Firebase project + its FCM + service-account JSON. + +**Not done, and deliberately so:** no `kubectl apply` was run — there is no kubeconfig available +in the build environment; deploying is a human-only action. The manifests and a full runbook are +ready in `deploy/k8s/README.md`, waiting on: + +1. Create a dedicated Firebase project (not reusing `vncmail-plus`'s or `src-website`'s) and + generate its service-account JSON. +2. `kubectl apply` the manifests (with real secrets substituted for `secret.example.yaml`'s + placeholders) against the microk8s cluster. +3. Once the relay is live and reachable (e.g. `vncmail-relay.sandbox.vnc.de`), repoint both + `vncmail-plus`'s `DEFAULT_RELAY_BASE_URL` and `vncmail-native`'s equivalent relay base URL + (check `src/api/push.ts`/`src/lib/push-notifications.ts`) at it instead of upstream's shared + instance. Re-run the webmail's existing Web Push smoke path end-to-end against the new relay + before treating it as the default. + +## 7. Remaining roadmap (not yet started) + +In rough order, per the `VNCprodbuild` skill: + +1. **iOS push (`vncmail-native`)** — blocked on the human's Apple Developer Program enrollment + (§4/§5). `vncmail-native` already builds for iOS and ships via TestFlight; only push and + client certs are missing. +2. **JMAP delta-sync engine** — replace `offline-sync.ts`'s bulk AsyncStorage download with a + real `Email/changes`/`Mailbox/changes` cursor-based incremental sync. **This is the + highest-stakes step in the entire program** — the skill calls for high/xhigh reasoning effort + plus an independent, fresh-context agent adversarially reviewing the design before any + implementation starts. Not yet begun. +3. **SQLCipher local store** — swap AsyncStorage for `expo-sqlite` with `useSQLCipher: true` + (§4), one isolated database/key per account (multi-account confirmed required, §4). Needs an + explicit, security-sign-off decision on key derivation/lifecycle (from-password vs. + device-random-key wrapped by biometric; wipe-on-logout) before implementation — do not let an + agent default this silently. +4. **FTS5 search index** — SQLite FTS5 population job tied to the sync engine above. +5. **Offline compose/outbox** — queue composed messages while offline, replay via JMAP + `Email/set` on reconnect, handle conflicts. +6. **Platform hardening** — background refresh scheduling (`BGTaskScheduler`/`WorkManager`), + Apple export-compliance declaration (`ITSAppUsesNonExemptEncryption`, triggered once SQLCipher + ships in the iOS binary — an agent can draft the text, only a human can file it), Google Play + Console account/signing key, final store submissions. +7. **Fix the webmail's own no-op service worker** — `public/sw.js` intentionally caches nothing + today; adding Workbox-style precaching of the app shell is a cheap, independent improvement to + the PWA's offline-shell behavior, unrelated to the native-client work above. + +## 8. Known landmines + +- **`~/vncmail-plus` is a shared, actively-used checkout.** Other sessions commit and switch + branches there concurrently. An untracked file written directly into that checkout was lost + mid-session to a concurrent branch switch — confirmed incident, 2026-08-04. **Any work meant + to persist must go into a dedicated worktree (like `~/worktrees/vncmail-electron`) or be + committed immediately** — never leave meaningful uncommitted/untracked work sitting in the + shared checkout. +- **`~/vncmail-native` and `~/vncmail-relay` are fresh clones** (created 2026-08-04) with no + confirmed concurrent-session activity yet — lower risk today, but don't assume that stays true + as more work lands there. +- **electron-builder + `node_modules`** — see §5's bug writeup; a general electron-builder + landmine, not specific to this codebase. +- **Expo Go + SQLCipher are mutually exclusive** — see §4/§6.1. The moment `useSQLCipher` is + enabled, Expo Go can no longer run the app at all; this is a hard platform constraint, not a + configuration bug to work around. + +## 9. Before merging any of this + +None of the three repos' branches described here have been pushed or opened as a PR. Before +that happens: + +- Run the full existing test/lint suites in each repo, not just the new smoke tests added here. +- `vncmail-plus` has its own `VERSION`/`CHANGELOG.md` convention (currently `1.7.8`) — a version + bump belongs at actual release/merge time, not mid-feature-branch; this manual deliberately + did not touch either file. +- Cross-check the eslint-ignore fix (§5) lands even if the rest of the Electron work is split out + or delayed — it's an independent, valuable fix on its own.