diff --git a/docs/ELECTRON-OFFLINE-ENGINE-DESIGN.md b/docs/ELECTRON-OFFLINE-ENGINE-DESIGN.md new file mode 100644 index 00000000..215300fd --- /dev/null +++ b/docs/ELECTRON-OFFLINE-ENGINE-DESIGN.md @@ -0,0 +1,1330 @@ +# Electron Offline Engine — Design + +Status: **design only, not implemented.** Nothing outside this file has been changed on this +branch. `electron/main.ts`, `electron/preload.ts` and `lib/jmap/client.ts` are untouched. + +Repo: `brvncde-dotcom/vncmail-plus`, branch `claude/electron-offline-design`, worktree +`~/worktrees/vncmail-electron-sqlite`. Based on `claude/electron-desktop` (the working desktop +shell + RFC 8887 WebSocket push), HEAD `b15098a6`. + +Companion documents: + +- **`~/worktrees/vncmail-native-sync-impl/docs/DELTA-SYNC-ENGINE-DESIGN.md`** (revision 3, 2012 + lines) — the finalized, twice-adversarially-reviewed, implemented and real-device-verified JMAP + delta-sync design for this program's React Native client. **This document is an adaptation of + that one, not a replacement for it.** Read it first; it is the normative source for everything + marked *[reused]* below. Cited as **M§n** throughout. +- `~/worktrees/vncmail-native-sync-impl/docs/DELTA-SYNC-DESIGN-REVIEW.md` — the adversarial review + that produced M's revision 2 (findings S1–S16). Cited as **MR**. +- `docs/VNCMAIL-NATIVE-BUILD-MANUAL.md` — program narrative; §4 decision log, §5 (what the Electron + shell already has), §8 known landmines. +- `~/.claude/skills/VNCprodbuild/SKILL.md` — the build plan this belongs to. + +Normative references, cited by section: **RFC 8620** (JMAP core) and **RFC 8621** (JMAP Mail), +exactly as enumerated in M's preamble. This document does not re-derive any RFC reading; every +protocol-level claim is M's, verified there. + +--- + +## 0. How to read this document + +The JMAP delta-sync problem is platform-independent. M solved it, was attacked twice over it, and +shipped it. Re-deriving it here would produce a second, subtly different set of invariants for the +same protocol — which is how one client silently loses mail the other doesn't. + +So every section is tagged: + +| Tag | Meaning | +|---|---| +| **[reused]** | Adopted from M unchanged. The cited M section is normative; this document only records *that* it applies and any Electron-specific naming. Do not re-litigate. | +| **[adapted]** | M's decision holds but its mechanism doesn't, because Electron's runtime differs. The difference is stated explicitly. | +| **[new]** | No M counterpart, or M's answer is actively wrong here. Designed from scratch in this document. | + +The genuinely new work is §2 (which process hosts the engine), §3 (SQLCipher from day one), §6 (key +storage), and the parts of §5/§7/§8 that follow from those. Everything else is M. + +### 0.1 Scope + +**In scope:** where the engine and its SQLite file live; which SQLite binding; whether encryption +ships on day one and how its key is stored; how the store keys against *this* repo's account model; +what the existing renderer-side push pipeline must and must not do once the engine exists; the +schema; triggering; the staged rollout with its verify-first gate. + +**Out of scope, deliberately:** + +- FTS5 index population (VNCprodbuild step 9). §7.5 reserves the hook. Note §3.4: FTS5 is + empirically present in every candidate binding, so this is not a binding-selection input. +- Offline **compose/outbox**. Unlike the mobile app, **this repo has no outbox and no optimistic + mutation layer at all** (§1.6) — so M§5.6's read-time overlay has nothing to overlay. v1 desktop + offline is **read-only**. This is a scope decision, recorded in §5.4, not an oversight. +- Attachment blob storage (M§9.4's second bullet applies verbatim when it lands). +- Calendar / Contacts / Files delta sync. +- Shared/group ("delegated") mail. Account-scoped primary keys are in place from day one (M§9.3, + MR S3) so adding it later is inserting rows. +- Code signing (VNCprodbuild step 9). It is *referenced* in §6.3 because it interacts with + macOS key storage, but it is not resolved here. + +**Non-goal:** compatibility with anything on disk today. There is nothing on disk today (§1.6). + +--- + +## 1. What exists today, verified + +File:line references are to this worktree at `b15098a6`. Everything in this section was read, and +every runtime claim in §3/§6 was executed against the Electron binary actually pinned by +`package.json` — see §3.1 for the transcript summary. + +### 1.1 The desktop shell + +`electron/main.ts` (225 lines) boots the **same** Next.js `output: "standalone"` artifact the +`Dockerfile` ships (`next.config.ts`'s `output: "standalone"`), as a **child process**: + +- `getStandaloneServerEntry()` (`:26-31`) — `process.resourcesPath/standalone/server.js` when + packaged, `.next/standalone/server.js` in dev. +- `startStandaloneServer()` (`:70-105`) — allocates a random free localhost port (`:33-48`), spawns + `process.execPath` with `ELECTRON_RUN_AS_NODE: "1"` (`:85-94`) so no system Node is required, then + polls until reachable (`:50-68`). +- `createMainWindow()` (`:114-143`) — `BrowserWindow` with `contextIsolation: true`, + `nodeIntegration: false`, **`sandbox: true`**, preload at `dist-electron/preload.js`, loading + `http://127.0.0.1:`. +- The notification bridge: `ipcMain.handle("vnc:show-notification", …)` (`:155-176`), reached from + `electron/preload.ts:16-27`'s `contextBridge.exposeInMainWorld("vnc", …)`, wrapped by + `lib/electron-bridge.ts`'s `isElectronShell()` / `showElectronNotification()`, and called from + `app/(main)/[locale]/page.tsx:1198-1222`. **This is the existing IPC pattern** — one + `ipcMain.handle` + one `contextBridge` method, no channel registry, no streaming. + +Packaging (`electron-builder.config.js`): the standalone server ships as `extraResources` copied +`from: ".next"` with a `standalone/**/*` filter — deliberately, to dodge app-builder-lib +unconditionally dropping a copy-root directory literally named `node_modules` (documented in that +file, and in the manual §5 as a bug found by actually launching a `--dir` build). Targets: macOS +dmg+zip **x64 and arm64**, Windows nsis x64, Linux AppImage+deb x64. Unsigned. +`scripts/build-electron.mjs` bundles `electron/*.ts` with esbuild, CJS, `external: ["electron", +"electron-updater"]`. + +CI: `.github/workflows/electron-build.yml`, matrix macos/windows/ubuntu, **Node 22** on the runner +(note: not Electron's Node — see §3.3), `npm run test:electron` as a required gate before packaging. + +### 1.2 The JMAP client, and where the credentials actually are + +`lib/jmap/client.ts` (7413 lines) is a **renderer-side** class. `JMAPClient` (`:542`) holds +`serverUrl`, `username`, `password` and an `authHeader` built in the constructor as +`Basic ${btoa(username:password)}` (`:579-585`), or `Bearer …` via `static withBearer` (`:586-598`). +`authenticatedFetch` (`:672`) is plain browser `fetch()` straight to the mail server. There is no +JMAP proxy route in front of it for normal traffic. + +**But the credentials are also recoverable server-side, and that is the load-bearing fact for §2.** +`app/api/auth/session/route.ts`: + +- `POST` stores `encryptSession(serverUrl, username, password)` — AES-256-GCM under + `SESSION_SECRET` (`lib/auth/crypto.ts:23-33`) — in an **httpOnly** cookie + `jmap_session[_]` (`lib/auth/session-cookie.ts`), one per account slot, + `MAX_ACCOUNT_SLOTS = 50` (`lib/account-utils.ts`). +- `GET` returns only `{serverUrl, username}`; `PUT` returns the **full credentials** for session + restoration, gated on `Sec-Fetch-*` headers proving a same-origin browser `fetch()`. +- OAuth/TOTP accounts instead park a **refresh token** in an httpOnly cookie + (`app/api/auth/token/route.ts` POST), and `PUT` on that route mints a fresh access token from it — + **rotating the stored refresh token whenever the server returns a new one** (`:104-106`). Remember + that; §2.5 has to keep two independent refreshers from existing. + +So: any code running in the standalone server process can, for any account slot, obtain either a +Basic auth header (decrypt the session cookie) or a bearer token (refresh-token grant) **without a +single new credential path, IPC message, or storage location.** This is not true of Electron's main +process, which sees none of those cookies. + +### 1.3 The push pipeline as shipped, and the wall it hit + +`setupPushNotifications()` (`:6130`) prefers RFC 8887 JMAP-over-WebSocket +(`getWebSocketUrl()`, `:3803-3809`, reading the `urn:ietf:params:jmap:websocket` capability off the +session — never hardcoded), falling back to SSE, then polling. Tight reconnect ladder (200 ms base / +5 s cap), 30 s heartbeat, and a 3-consecutive-handshake-failure circuit breaker +(`wsPermanentlyDisabled`, `:6060`ff). + +The committed comment at `:6038-6059` records the empirical outcome, and it is the single most +important existing finding for this design: + +> `stalwart.sandbox.vnc.de`'s `/jmap/ws` requires the same HTTP `Authorization` header as every +> other JMAP endpoint **on the WebSocket UPGRADE request itself**. The browser `WebSocket` +> constructor cannot attach custom headers (a WHATWG restriction; credentials-in-URL are rejected +> too). So from the renderer, every attempt fails the handshake and the circuit breaker correctly +> falls back to SSE. + +And the alternative it explicitly declined (`:6055-6059`): + +> opening it from Electron's main process via a header-capable client like the `ws` package "would +> mean piping raw credentials from the renderer to the main process over IPC, which is a materially +> bigger security-sensitive change than what was scoped here." + +That objection is **correct for the main process and inapplicable to the standalone server** — which +already holds the credentials (§1.2) and would pipe nothing. §2.5 acts on this. + +The transport-agnostic "genuine new mail" signal is `email-store.newEmailNotification` +(`stores/email-store.ts:3082-3086`, set by `refreshCurrentMailbox`), consumed once in +`app/(main)/[locale]/page.tsx:1198-1222`. It already fires identically over WS, SSE and polling. +**The engine must not add a second notification path.** §2.5 states the rule. + +### 1.4 Multi-account model (differs from mobile — check, don't assume) + +`stores/account-store.ts` — a Zustand `persist` store named `account-registry`, holding +`AccountEntry[]` with: + +- `id`: `` `${username}@${new URL(serverUrl).hostname}` `` via + `lib/account-utils.ts generateAccountId()`. **Same shape as mobile's `LocalAccountId`** — a + genuine coincidence worth stating, because it means M§3.1's `LocalAccountId` type carries over + verbatim. +- `cookieSlot: number` — **new relative to mobile.** The index into the per-slot cookie namespace of + §1.2, assigned by `getNextCookieSlot()` (first free integer, reused after removal). +- `serverIdentifiers?: string[]` — server-confirmed account-id forms captured at login, used by the + account-switch guard so a short login name canonicalized by the server is still recognized. +- `activeAccountId`, `defaultAccountId`; caps `MAX_ACCOUNTS_HTTP1 = 5` (HTTP/1.1 SSE-connection + budget) lifting to `MAX_ACCOUNT_SLOTS = 50` once h2/h3 is observed. + +Two consequences for the schema (§7): + +1. The durable key is `accountId` (`username@host`), **never `cookieSlot`** — slots are recycled by + `getNextCookieSlot()`, so a slot number is a transport detail with a shorter lifetime than the + data. A `slot → accountId` confusion is a cross-account data-mixing bug of exactly M's D6 shape. +2. Any API surface addressed by slot (as §1.2's routes are) must **resolve slot → accountId and + re-verify** against the session's confirmed username before touching the store. §5.3. + +### 1.5 CSP — a hard constraint on option C + +`proxy.ts:88-141` builds the CSP. In production: + +``` +script-src 'self' 'nonce-' # no 'unsafe-eval', no 'wasm-unsafe-eval' +connect-src 'self' https: wss: # 'wss:' was added for §1.3's WS push +``` + +`'unsafe-eval'` exists **only** for `isDev` and the plugin-sandbox path. WebAssembly compilation +requires `'wasm-unsafe-eval'` or `'unsafe-eval'` under CSP3. §2.3. + +### 1.6 What does *not* exist here (and does in the mobile repo) + +This is the inverse of M§1.1/§1.2, and it is mostly good news: + +| | mobile (`vncmail-native`) | here | +|---|---|---| +| Existing offline cache | `offline-sync.ts` + `offline-cache-store.ts`, carrying defects D1–D8 | **nothing.** No IndexedDB mail cache, no offline list, no offline read path. `lib/plugin-storage.ts` uses IndexedDB but only for plugin assets. | +| `Email/changes` / `Mailbox/changes` wrappers | already present, already driving an incremental list path | **none.** The only occurrence in the repo is a mock in `app/api/dev-jmap/[...path]/route.ts:1949`. Greenfield. | +| Outbox / optimistic mutations | `outbox-store.ts`, full-state idempotent queue | **none.** Mutations go straight to the server. | +| Push transport | SSE + FCM relay | WS (blocked, §1.3) → SSE → polling, all renderer-side | +| Stalwart integration fixture | in a *sibling* repo — MR S16 costed this as real cross-repo CI work | **in this repo**: `integration/docker-compose.yml` + 11 specs incl. `11-electron-notification.spec.ts`, which logs in against real Stalwart, injects mail over real SMTP, and asserts the native notification. Free to extend. | + +**Therefore M§1.3's defect list D1–D8 does not apply here.** There is no legacy cache to inherit +bugs from, no `patchCache()` write-through to delete, no D4 cursor fast-forward in shipped code, and +M§14.1's "discard, don't migrate" is vacuous. What *does* carry over is the *class* of each defect +as a thing not to introduce — which is what M's invariants I1–I13 are for (§4.3). + +One inherited defect *shape* is worth naming, because this repo has it too: `stores/file-store.ts` +and others use `try { localStorage.setItem(...) } catch { /* ignore */ }` in a dozen places — M's D2 +pattern. **Banned in the sync path** (M I4). §7.2. + +--- + +## 2. Decision 1 — which process hosts the engine **[new]** + +M has no counterpart: React Native has one JS context. Electron has three candidate homes, and this +codebase makes the choice non-obvious in both directions. + +### 2.1 Candidate A — engine + SQLite inside the standalone Next.js server process + +The renderer reaches cached data through new `app/api/**` routes, exactly as it reaches everything +else server-side today. + +**For:** + +1. **The credentials are already there, encrypted, per account** (§1.2). No new credential path, no + IPC carrying secrets, no second copy of the TOTP/refresh state machine. Every other candidate has + to solve this, and B can only solve it by doing the thing `client.ts:6055-6059` explicitly + declined. +2. **It unlocks real WS push, which the renderer structurally cannot have** (§1.3). A Node process + can set `Authorization` on a WebSocket upgrade (`ws` package). This is not a side benefit: it + converts a documented dead end into a working transport, on the server this program actually + targets, with no Stalwart-side change. §2.5. +3. **No new IPC surface at all.** `window.vnc` stays a one-method bridge. Nothing about + `contextIsolation: true` / `sandbox: true` has to be relaxed or extended. +4. **Native module packaging is already solved for this process.** The standalone server ships as + `extraResources` **outside `app.asar`**, with its own traced `node_modules` — the exact copy path + whose one footgun is already found, fixed and documented (`electron-builder.config.js`). A `.node` + binary in an unpacked directory needs no `asarUnpack` reasoning at all. +5. **Blocking is cheapest here.** `better-sqlite3` and `@signalapp/sqlcipher` are synchronous + (§3.2). Blocking this event loop delays local-cache HTTP responses; it does not block the + renderer's paint (React runs in the renderer) and does not block window/menu/IPC handling (that's + the main process). It is the *least* latency-critical of the three loops. +6. Reads are trivially observable — the existing integration suite drives the app over HTTP and can + assert on new routes without any Electron-specific harness. + +**Against:** + +1. **This process is also what a hosted, multi-user Docker deployment runs.** Unconditional offline + routes would have a shared server start caching *every user's* mail into a server-side SQLite + file. This is the strongest argument against A and it must be closed by construction, not by + convention — §2.4. +2. **Next.js output-file-tracing vs. a native module.** `serverExternalPackages` (already used for + `esbuild`, `next.config.ts`) plus NFT must actually carry `prebuilds/**/*.node` into + `.next/standalone/node_modules`. `node-gyp-build`'s resolution is directory-scan-based, which NFT + handles specially but not infallibly. **Verify-first, §12 Stage A.** +3. The DB path must be handed in: `app.getPath('userData')` is a main-process API, so `main.ts` must + pass it as an env var on spawn (`:85-94` already builds the env). One line, but it is a coupling. +4. Server-process lifetime is `window-all-closed` / `before-quit` (`main.ts:210-219`), so an + in-flight cycle is killed by process death rather than by a cooperative abort. M's crash-recovery + design (I1, M§6.3) already makes that safe — cost is one page — but a graceful-shutdown IPC is + worth adding later. + +### 2.2 Candidate B — engine + SQLite in `electron/main.ts`, over `contextBridge`/IPC + +**For:** + +1. **The standalone server's code stays byte-identical to a hosted deployment's.** A's §2.1-against-1 + simply does not arise: there is no Electron-only server code to accidentally ship in Docker. +2. Mirrors the existing notification bridge, so the pattern is familiar. +3. `electron-builder` already auto-unpacks `**/*.node` from the asar, and production `node_modules` + are collected regardless of the narrow `files: ["dist-electron/**/*", "package.json"]` (that's why + `electron-updater` is `external` in `build-electron.mjs` and still ships). Low packaging friction. +4. `safeStorage` (§6) lives in the main process natively — no bridging for the key. + +**Against:** + +1. **It requires exactly the thing `lib/jmap/client.ts:6055-6059` refused.** The engine needs + credentials. The main process has none: the `jmap_session` / refresh-token cookies belong to the + renderer's origin. So either the renderer ships the Basic header / bearer token over IPC (the + declined "materially bigger security-sensitive change"), or the main process learns to read + Electron's cookie jar (`session.defaultSession.cookies`) and re-implement `decryptSession` — which + means shipping `SESSION_SECRET` into the main process too. Both are net-new secret handling to + reach a place that currently, deliberately, holds no secrets. +2. **A large new IPC surface.** The offline read path is not one fire-and-forget notification; it is + list queries, single-message reads, per-account status, settings changes and abort signals — each + an `ipcMain.handle` returning structured data across `contextIsolation`. Every one is a new trust + boundary in a window that is currently `sandbox: true` with a 12-line preload. +3. **Blocking hurts most here.** Synchronous SQLite on the main process's loop is jank in window + dragging, menu response and IPC dispatch. Mitigable with `worker_threads`, but then B is A's + complexity plus IPC. +4. The renderer's offline read path becomes Electron-only by construction, so the web/PWA deployment + can never share it. That may be acceptable — but it is a fork, and A's routes would work in both. + +### 2.3 Candidate C — engine in the renderer, WASM SQLite over OPFS + +**Investigated, not assumed. Findings:** + +- The official WASM build is `@sqlite.org/sqlite-wasm` (3.53.0-build1), which `sqlocal` (0.18.0) + wraps for OPFS. **Neither has any encryption** — SQLCipher is a *fork* of SQLite's source, not a + loadable extension, so an official build cannot have it. +- Encrypted WASM builds **do** exist on npm: `@7mind.io/sqlcipher-wasm` (1.2.0, "production-ready + WebAssembly build of SQLCipher with real OpenSSL-based encryption") and `@aztec/sqlite3mc-wasm` + (5.1.0, SQLite3MultipleCiphers 2.3.5 as WASM). So the honest answer to "does a WASM SQLCipher + genuinely exist?" is **yes, but only from small third-party publishers** — not from the SQLite + project, not from a vendor with a desktop-mail-scale user base. For a component whose failure mode + is "the user's whole mailbox is readable on a stolen laptop", that provenance is the finding. +- **The CSP problem is decisive independently of encryption.** §1.5: production `script-src` is + `'self' 'nonce-…'`. WASM compilation needs `'wasm-unsafe-eval'`. Adding it in `proxy.ts` widens the + CSP for **every deployment of this product, including the hosted web one**, to buy a desktop-only + feature. That is a security regression with the wrong blast radius. +- Even granted both, the encryption key would live in the renderer's JS heap — the same context that + renders untrusted HTML mail bodies and hosts the plugin sandbox. A/B keep it in a Node process the + renderer cannot address. +- Secondary, verify-first if C is ever revisited: the official `opfs` VFS uses `SharedArrayBuffer` + + `Atomics.wait` and therefore needs COOP/COEP headers; `opfs-sahpool` does not. Neither is + configured in `proxy.ts` today. + +**Against, summarised:** requires a product-wide CSP widening, puts the key in the most exposed +context, and its only encrypted backends are unvetted third-party WASM builds. **For:** no IPC, no +native module, no packaging story, works identically in the browser PWA — a real benefit, and the +reason to keep C on record rather than dismiss it. If the offline store were *unencrypted* and +*browser-first*, C would be the right answer. It isn't either. + +### 2.4 Decision: **A**, with the hosted-deployment gate as part of the design + +The engine and the SQLite file live in the **standalone Next.js server process**. Rationale in +priority order: it is the only candidate where credentials are already present and correctly scoped +(§2.1-for-1); it is the only candidate that makes RFC 8887 push actually work (§2.1-for-2); it adds +no IPC and no preload surface; and its native-module packaging path is the one already exercised and +debugged in this repo. + +A's one serious objection — the same process serves hosted multi-user deployments — is closed +structurally, not by convention. Three layers, all required: + +1. **A desktop marker env var.** `main.ts`'s spawn env (`:85-94`) gains + `VNCMAIL_DESKTOP_STORE_DIR=/offline`. Absent or empty ⇒ the engine + module is never constructed, and this also supplies §2.1-against-3's path. One variable does both + jobs, so they cannot drift apart. +2. **Every new route refuses to run without it.** `app/api/offline/**` returns `404` (not 403 — + nothing should learn the routes exist) when the marker is unset. This mirrors the existing + "routes 503-on-misconfig" habit elsewhere in the program. +3. **A single-user assertion.** With the marker set, the engine asserts at open time that the store + directory is per-OS-user (it is, being under `userData`) and records the resolved + `serverUrl`+`username` of every account it materialises. A store whose recorded account set + doesn't match the requesting session's is a purge trigger (§5.5), not a merge. + +Additionally, and non-negotiably: **the engine runs on a `worker_threads` Worker inside the server +process, never on the request event loop.** Synchronous SQLite plus JMAP page application is exactly +the workload that turns a shared event loop into a latency problem, and M's own I11 (jobs strictly +sequential within an account, M§3.4) is naturally expressed as "one worker per account, one job at a +time" rather than as a hand-rolled mutex. API routes talk to the worker via `postMessage` and never +touch the database handle. This also localises §2.1-against-4: the worker gets an explicit +`terminate` path. + +Rejected explicitly, for the record: B, because it can only be built by moving credentials into a +process that today holds none — the change `client.ts` already declined on its merits, and nothing +about an offline store makes that trade better. C, because it needs a product-wide CSP widening and +its only encrypted backends are unvetted. + +### 2.5 Consequence for the already-working WS-push renderer code + +This is the question that must not be answered by accident. + +**The renderer's push pipeline stays exactly as it is. No line of `lib/jmap/client.ts` changes for +v1.** Its SSE/polling path is the renderer's own liveness for the *visible* list, it is working, and +it is what feeds `newEmailNotification` → `showElectronNotification` (§1.3). The engine does not +replace it and does not read from it. + +Three rules, in order of how easy they are to get wrong: + +1. **The engine gets its own push connection, and it is the header-capable one.** In the server + process the engine opens `wss://…/jmap/ws` with an `Authorization` header (via `ws`), which is the + connection the renderer cannot open (§1.3). It subscribes with `WebSocketPushEnable`, and treats + the resulting `StateChange` exactly as M§10.4 specifies: **a wake signal, never a cursor.** M's + two load-bearing rules (a pushed `newState` is never written as a cursor; state-equality against + our cursor is a cheap safe dedupe) apply verbatim. *[reused: M§10.4]* +2. **The engine never fires a notification.** `newEmailNotification` remains the single source of + the OS notification, in the renderer, via the existing bridge. An engine-side notification would + double-notify on the common path (both connections see the same delivery) and diverge on the + uncommon one. What the engine *may* do is expose "account X changed" on its status channel; the + renderer decides whether to refresh, exactly as M§5.7 specifies the engine→email-store direction + (and only that direction). +3. **Duplicate work is bounded and acceptable; duplicate *state* is not.** Yes, two connections to + the same server per account, and both wake on the same delivery. That is deliberate, and it is + M§5.7's argument transplanted: the renderer's list cursor and the engine's `/changes` cursors + page differently, invalidate differently, and *one being wrong must not corrupt the other*. The + engine must never read the renderer's `lastStates` (`client.ts:571`) and the renderer must never + read the engine's cursors. The cost is one extra socket per account; the cap is + `MAX_ACCOUNTS_HTTP1 = 5` today, and the engine's socket is server→server, so it does not consume + the browser's per-origin HTTP/1.1 connection budget that cap exists to protect. + +**Deferred, and worth stating so it isn't done silently:** once the engine's WS connection is proven, +the renderer's transport could be retired in favour of the engine pushing "account changed" down an +SSE/`EventSource` from the local server — one server-side socket per account instead of two, and the +renderer's circuit-breaker-into-SSE path becomes dead code. That is a *follow-up*, gated on the +engine's connection being verified against real Stalwart, not part of v1. Doing it in v1 would make +a working notification path depend on an unproven one. + +--- + +## 3. Decision 2 — SQLite binding, and SQLCipher on day one **[new]** + +M deferred `useSQLCipher` for one specific reason: Expo Go cannot load it, so a plaintext-first phase +was the only way to keep the day-to-day dev workflow (M§9.2, M§14.3 step 3.1, MR S4/V4). **Electron +has no Expo Go.** The deferral's entire justification is absent, so the question is genuinely open +here and has to be answered on the evidence. + +### 3.1 What was measured, and how + +All of the following was executed against the Electron binary this repo pins +(`electron@43.2.0`, resolved from `~/worktrees/vncmail-electron/node_modules` — this worktree has no +`node_modules` installed), both as the main process and under `ELECTRON_RUN_AS_NODE=1` (the mode the +standalone server actually runs in, `main.ts:88`): + +| Fact | Result | How | +|---|---|---| +| Electron 43.2.0's bundled Node | **24.18.0**, ABI `modules=148`, `napi=10` | `process.versions` | +| Its bundled SQLite | **3.53.1, with `ENABLE_FTS5`** | `pragma compile_options` | +| `node:sqlite` present and working | yes; exports `DatabaseSync, StatementSync, Session, constants, backup`; no `ExperimentalWarning` observed | `require('node:sqlite')` | +| `node:sqlite` encryption | **none.** `compile_options` has no codec. `PRAGMA key='…'` is **silently accepted and does nothing** — the file was written with a `SQLite format 3` header and a plaintext canary string recoverable with `grep` | wrote a real file, read the bytes back | +| `node:sqlite` stability (Node 24) | **1.2 — Release Candidate** (RC since v24.15.0; no longer behind `--experimental-sqlite`), not stability-2 stable | Node 24 docs | +| `better-sqlite3@13.0.2` | installs with **zero build step**; ships in-tarball N-API prebuilds for `darwin-{arm64,x64}`, `linux-{arm64,x64}`, `linuxmusl-{arm64,x64}`, `win32-{arm64,x64}`; **loads in Electron 43 both as main process and under `ELECTRON_RUN_AS_NODE`**; FTS5 available; `PRAGMA key` silently a no-op | `npm install` + load in Electron | +| `better-sqlite3` 12.x vs 13.x | 12.x used `install: prebuild-install \|\| node-gyp rebuild` (per-ABI downloads). **13.0.0 dropped that** for `gypfile: false` + in-tarball prebuilds — i.e. moved to ABI-stable N-API. This is why no `electron-rebuild` is needed | npm metadata for 13.0.2 vs 12.11.1 | +| `better-sqlite3-multiple-ciphers` | latest is **12.11.1** (2026-06-18) — on the *old* 12.x prebuild-install model. Its GitHub release carries 98 Electron prebuilds, ABIs **121…146**. **Electron 43 needs ABI 148 — absent.** So it would fall through to `node-gyp rebuild`: a C++ toolchain + Python + Electron headers on every contributor machine and every CI runner | npm metadata + GitHub releases API | +| **`@signalapp/sqlcipher@4.0.3`** | **N-API** (`prebuildify --strip --napi`, `node-gyp-build`), in-tarball prebuilds for `darwin-{arm64,x64}`, `linux-{arm64,x64}`, `win32-{arm64,x64}`. **Loads in Electron 43 with no rebuild, both process modes.** Real **SQLCipher 4.10.0 community**; `PRAGMA cipher_version` reports it; **file header is ciphertext, canary absent from the bytes, wrong key rejected with `SQLITE_NOTADB`, right key reads the row back**; FTS5 available; `better-sqlite3`-shaped synchronous API (`db.exec`, `db.prepare().run()/all()`, `db.pragma()`); **AGPL-3.0-only**, matching this repo's own licence | `npm install` + full round-trip in Electron main process | + +Two of those deserve to be called out as landmines rather than table rows: + +- **`PRAGMA key` failing silently is the worst possible ergonomics.** On both `node:sqlite` and plain + `better-sqlite3`, setting a key "works", the database works, and the mail is on disk in cleartext. + There is no error to notice. Whatever binding ships, the store's open path must **assert + encryption positively** — read `PRAGMA cipher_version` and refuse to proceed if it is empty — and + a test must assert a canary string is *absent* from the raw file bytes. Both are in §11. +- **`better-sqlite3-multiple-ciphers` lags Electron by roughly one to two majors** (ABI 146 vs 148, + and its 12.x base trails better-sqlite3's 13.x). That lag is structural, not a one-off: with + per-ABI prebuilds, every Electron major bump re-opens the question. `@signalapp/sqlcipher`'s N-API + prebuilds are immune to Electron majors by construction. + +### 3.2 Recommendation: **ship SQLCipher on day one, via `@signalapp/sqlcipher`** + +The friction the mobile design was avoiding **does not exist here**, and the evidence is unusually +clean: a package built and maintained specifically to run SQLCipher inside an Electron desktop +application, N-API so it needs no rebuild against Electron 43, prebuilds covering every platform this +repo actually packages (§1.1: darwin x64+arm64, win32 x64, linux x64 — all present), FTS5 already +compiled in, an API close enough to `better-sqlite3` that the backend is the same code either way, +and a licence identical to this repo's. + +Verified working, in this environment, against this Electron version. Not inferred. + +So: **no plaintext-first phase.** M§14.3's plain-then-encrypted staging exists to protect a dev +workflow that has no analogue here; importing it would mean deliberately shipping a plaintext +mailbox on disk for a phase, plus building and then discarding the store-format-migration machinery +of M§8.4.1 to get out of it. Both costs, no benefit. + +Corollaries: + +- The abstraction boundary of M§9.1 (`SyncStore` / `SyncTxn`) stays **exactly** as M specifies. It is + what makes this reversible: if `@signalapp/sqlcipher` ever becomes untenable, `store-sqlite.ts` is + the only file that changes. Do not skip it on the grounds that the binding question is now settled + — the boundary is also what makes `store-memory.ts` possible, and M§13's contract-tests-against-two-backends + is most of the test plan's value. +- M§8.4.1's **out-of-band store-format marker is still required**, for the reason V4 gives, minus one: + `schemaVersion` still lives inside a file that a future format change could make unreadable, so it + must be mirrored outside. What day-one encryption removes is only the *plain→cipher* transition + that would otherwise be the marker's first customer. Keep the marker; it costs a JSON file. +- `node:sqlite` is **rejected**, on two independent grounds: no encryption at any price, and + stability 1.2 (RC) for a component holding the user's mail. Its one advantage — zero dependencies — + is worth nothing once encryption is a requirement. Worth re-evaluating only if a decision is ever + taken to ship unencrypted. +- Plain `better-sqlite3@13.0.2` is the **fallback**, not the plan: adopt it only if Stage A (§12) + finds `@signalapp/sqlcipher` cannot be packaged, and in that case the human decides between + "unencrypted desktop store" and "no desktop store yet" (§13, open question 1). + +### 3.3 Friction that does exist, stated plainly + +Not zero, just small — and the human should see it rather than have it smoothed over: + +1. **A ~6 MB native dependency with 6 platform prebuilds in the tarball**, entering + `dependencies`. Install size and `npm ci` time grow for everyone, including web-only contributors + who will never run Electron. +2. **NFT tracing is the real unknown** (§2.1-against-2). `serverExternalPackages` plus a verification + that `prebuilds/**/*.node` reached `.next/standalone/node_modules` is Stage A's first task. If NFT + won't carry it, the fallback is a copy step in `scripts/assemble-standalone.mjs` — which already + exists precisely to patch up what standalone output omits, so this is a known-shaped fix. +3. **CI runs Node 22 while Electron bundles Node 24** (§1.1). Harmless for an N-API module (the + prebuild is selected by platform+arch, not ABI) — but it *would* have been fatal for a + per-ABI package, which is worth recording as another reason the N-API choice matters. Any + `npm test` that loads the binding under the runner's own Node exercises a different Node than + production; the binding load assertion must therefore run **inside Electron** (`npm run + test:electron`), not only in vitest. +4. **Cross-arch macOS packaging.** CI builds both x64 and arm64 dmg/zip on one macOS runner. + In-tarball prebuilds ship *all* platforms, so this works — whereas `prebuild-install` downloads + only the host's, and would have broken the cross-arch target. Verify in Stage A that the arm64 + `.node` is what ends up in the arm64 build and vice versa. +5. **`linuxmusl` is not covered** by `@signalapp/sqlcipher` (`better-sqlite3` does cover it). Irrelevant + for AppImage/deb (glibc); relevant if an Alpine-based container ever wants the engine — which, + per §2.4, it must not. + +### 3.4 What is *not* an input to this decision + +FTS5 (VNCprodbuild step 9) is present in all three candidates — Electron's own bundled SQLite, +`better-sqlite3` 13, and `@signalapp/sqlcipher` (which additionally ships Signal's FTS5 segmenting +extension and an `initTokenizer()`). So the search step cannot be used to argue for a binding. +Recorded so a later session doesn't relitigate the choice on those grounds. + +--- + +## 4. The sync engine itself — mostly M, verbatim + +Everything in this section is **[reused]** unless marked otherwise. The M section cited is +normative; what follows is a map, not a restatement, so that a reader can tell reuse from +re-derivation at a glance. + +### 4.1 Architecture: three state machines *[reused: M§2, M§3.4 I11]* + +Per account: **A** delta (`Mailbox/changes` then `Email/changes`, one cursor each), **B** coverage +(the envelope-window enumeration; `/changes` structurally cannot deliver pre-existing mail, so +coverage owns history and is also the bootstrap), **C1** body-queue drain, **C2** body backfill +(MR S9 — without it, widening body retention silently does nothing for already-covered envelopes). + +Logically independent state, **operationally serialised** (I11). Here that serialisation is +structural rather than disciplinary: one worker thread per account, one job at a time (§2.4). M's +warning stands regardless — "run bodies in parallel, it's separate state" is forbidden, and F48 +(a body landing for an envelope destroyed in the same cycle) is what it costs. + +Module layout: M§2.2 verbatim, relocated. `src/sync/**` in the mobile repo becomes **`lib/sync/**`** +here (this repo has no `src/`): `engine.ts`, `cursor.ts`, `apply.ts`, `coverage.ts`, `bodies.ts`, +`retention.ts`, `errors.ts`, `states.ts`, `store.ts`, `store-sqlite.ts`, `store-memory.ts`. M's hard +requirement that `apply.ts` be **pure** (no network, no storage, no store access) carries over +unchanged and is the single highest-leverage constraint in the document: it is what turns M§11's +failure-mode table into a vitest suite. `overlay.ts` is **not** ported — see §5.4. + +### 4.2 Two record tiers, two retention windows *[reused: M§2.1]* + +Envelope tier = `EMAIL_LIST_PROPERTIES` — which in *this* repo is +`lib/jmap/client.ts:139-154`: `id, threadId, mailboxIds, keywords, size, receivedAt, from, to, cc, +subject, preview, hasAttachment, blobId`. (Note the extra `blobId`, present so list rows can serve +drag-out to the filesystem as `.eml`; it belongs in the envelope tier here.) Body tier = +`bodyStructure, textBody, htmlBody, bodyValues, attachments, bcc, replyTo, sentAt`. + +Independent retention: `offlineEnvelopeDays` ≫ `offlineBodyDays`, MB cap on **bodies only**. The +human decision M records — widen envelopes well beyond bodies, so a message never falls out of the +offline *list* over a body-size cap — is a program-level decision and applies here identically. +Concrete numbers remain open (§13). + +### 4.3 Cursors, provenance and invariants *[reused: M§3]* + +Adopted without modification: + +- `SyncCursor` / `CoverageState` / `BodyQueueEntry` / `AccountSyncState` as M§3.1 defines them, + including per-cursor failure counters (MR S6), `sweepFloor` + `deferredTargetFrom` (MR S2), and + `gapMarkers`. +- **Branded state types** (M§3.2): `ChangesState` vs `SnapshotState`, `advanceCursor(key, next: + ChangesState)` as the delta path's only cursor write, `seedCursor(key, commitment: + EnumerationCommitment)` for bootstrap/reconcile, and `EnumerationCommitment` made genuinely + unforgeable by an **unexported real `Symbol()`** tag — including M's implementation note that + `declare const … : unique symbol` emits no runtime value and throws `ReferenceError` as a computed + key. That note came out of M's actual build; it would have been re-discovered here otherwise. +- The ordering rule as I2, not the false "only a changes state, ever" of M's revision 1. +- **I1–I13 in full** (M§3.4). Cursor-last; provenance-as-ordering; monotonic-or-invalidated; no + silent write loss; idempotent application; account containment; deletion provenance; no + clock-dependent cursors; bounded work; no wedge; sequential execution; field-level state writes; + corrupt-state-blob ⇒ resync. +- The "not a cursor" list (M§3.3): `Email/query`'s `queryState`, a pushed `StateChange.newState`, + `sessionState`, Thread state, `EmailDelivery`. + +Two Electron notes on I12/I4, both simplifications rather than changes: with a real SQLite +`BEGIN…COMMIT` in place from day one (§3.2), M's per-account mutex and read-merge-write discipline +become belt-and-braces rather than load-bearing (M§9.2 anticipated exactly this), and I4's "every +write either succeeds or raises" is the binding's default behaviour rather than something to enforce +against a fire-and-forget storage API. + +Cursor keying: `(LocalAccountId, JmapAccountId, CursorType)`, all three required, for M§3.1's +reasons. `LocalAccountId` is this repo's `AccountEntry.id` (§1.4) — **never `cookieSlot`**. + +### 4.4 Bootstrap *[reused: M§4]* + +Replace-the-code-keep-the-shape does not apply (there is no `runOfflineSync` here), but M§4.1's +**mandatory order** does, and it is the one thing in this document most likely to be "optimised" into +a permanent data hole: + +1. Capture both cursors **first**, in one JMAP request (`Mailbox/get {ids: []}` + `Email/get + {ids: []}`), and `seedCursor` them inside one `EnumerationCommitment` that in the same transaction + writes `coverage {phase:'scanning', targetFrom, sweepFloor: targetFrom}`. +2. Full `Mailbox/get` → upsert every mailbox row. +3. The seeded cursors are **live from here**: each cycle runs A1, A2, then B. +4. Scan reaches `targetFrom` ⇒ `coveredFrom = sweepFloor`, `phase = 'complete'`. Bootstrap has no + delete sweep; only reconcile sweeps. + +The cursor is deliberately *older* than the data, so the first delta cycle re-delivers some changes +we already have. That is I5 working. The cheaper opposite order silently loses mail. + +### 4.5 Change application *[reused: M§5.1–§5.5]* + +Order within a cycle (A1 → A2 → B → C1 → C2) and M's timeline argument for why delta-before-coverage +is safe *given I11* — the resurrection hazard, and the fact that the unsafe configuration is +concurrency, not ordering. `Mailbox/changes` `updatedProperties` count-only optimisation (RFC 8621 +§2.2). `Email/changes`: `created` ⇒ envelope fetch + conditional body enqueue; `updated` **present** +⇒ a **3-property** `Email/get {id, keywords, mailboxIds}` and never a body (RFC 8621 §4.1 — the two +mutable properties); `updated` **absent** ⇒ unconditional no-op with the ids filtered out *before* +the fetch is issued (MR S16); `destroyed` ⇒ delete envelope + body + membership + queue row. +Create-then-update-then-destroy ordering within a page. `notFound` is normal, not an error. +Mailbox/Email transient inconsistency tolerated, never repaired (I7: no deletion by inference). + +### 4.6 Pagination *[reused: M§6]* + +Ascending keyset walk on `receivedAt` with `calculateTotal: false`; `after` is **spec-inclusive** +(RFC 8621 §4.4.1, MR S14) so boundary re-delivery is normal and deduped by id; forward progress needs +strictly-greater `max(receivedAt)`; the no-progress guard is `anchor`/`anchorOffset` first and only +then, on `anchorNotFound`, a +1 ms advance with a `WARN` and a durable gap marker. Position-based +paging is rejected for M§6.2's reason. Budgets per M§6.4 — but see §8.2 for the desktop numbers. + +### 4.7 Errors, retry, reconcile, anti-wedge *[reused: M§7]* + +The seven-class taxonomy (Transport / RateLimit / ServerTransient / RequestLimit / Auth / Fatal / +StateInvalid), with **exactly one class moving a cursor** and unrecognised method errors defaulting +to ServerTransient. Full-jitter backoff. "Offline is not an error." Partial-failure semantics inside +a page (records may commit partially; the cursor may not). The eight cursor-advance rules of M§7.5. +`cannotCalculateChanges` handled as RFC-mandated **without blanking the UI**, with the **pinned +`sweepFloor`** (MR S2) and with the freshly-seeded cursor **live immediately** so a wide-window +rebuild doesn't stall incoming mail (MR S9). `oldState` mismatch re-issued once before escalating, +plus the ≤4-reconciles-per-24 h ceiling (MR S10). The monotonically **shrinking** `maxChanges` ladder +with every rung clamped to rung 0 (MR S7 + V2), and per-cursor counters with "any job failed ⇒ cycle +failed for escalation purposes" (MR S6). + +One Electron-specific input: this repo's client already has `RateLimitError` with `Retry-After` +parsing (`client.ts:54-62`, `authenticatedFetch`'s 429 branch) and a client-wide rate-limit gate. +The engine's own JMAP layer (§10.1) must reproduce that behaviour rather than inherit it, since it +will not be using `JMAPClient`. + +--- + +## 5. Multi-account isolation, account identity, lifecycle **[adapted]** + +Requirement confirmed at program level (manual §4: an offline cache must isolate per account, +including per-account keys). M§8 is the design; what changes is the identity plumbing, because this +repo's account model differs (§1.4). + +### 5.1 Namespacing *[reused: M§8.1, adapted paths]* + +``` +/ + registry.json # ONLY: account ids present, purge tombstones, + # monotonic epochs, store-format markers (M§8.4.1) + accounts/.db # one SQLCipher file per account: mailbox, envelope, + # email_mailbox, body, body_queue, sync_state +``` + +Filenames are hashed, not `username@host`, so the directory listing is not a plaintext account +inventory on disk. **No cursor, coverage row, record or resync flag lives outside an account's own +file** — M§8.1's forward-compatibility requirement, which here is load-bearing on day one rather than +later, because §5.5's purge deletes the key and the file together and a cursor surviving that would +be advanced against a freshly-empty store. + +`registry.json` is deliberately **plaintext** and M§8.1's accepted-limitation argument transfers with +one improvement: mobile's justification was that `account-store` already persists usernames to plain +AsyncStorage. Here, `account-store.ts`'s `persist` (`:219-227`, name `account-registry`) already puts +`username` and `email` for every account into renderer `localStorage`, so the registry adds no new +exposure — and hashing the filenames means the registry is the *only* place the account list appears +in the store directory. It must be readable before any key exists (that is the whole point of +M§8.4.1's format marker), so it cannot itself be encrypted. + +`epoch` lives in the registry, outside the per-account namespace, because it must be monotonic +**across** a purge (M§8.3). Owner: `SyncStoreFactory`. Not writable from `SyncTxn`; a transaction +reads it to validate itself and rejects with `EpochMismatchError`. + +### 5.2 JMAP-level accounts within one login *[reused: M§8.2, M§9.3]* + +Cursors and **every SQL primary key** carry `jmap_account_id` (MR S3: JMAP ids are unique only within +an account). v1 syncs the **primary mail account only**; delegated/shared accounts stay online-only, +as they effectively are today. + +Note this repo already carries the same evidence mobile did: `client.ts:388`'s +`namespaceMailboxIds()` prefixes ids when returning emails for a non-active account (five call sites: +`:632`, `:1271`, `:2138`, `:2185`, `:2323`). Same collision, same workaround, same conclusion — +account-scoped keys from day one. + +### 5.3 Slot → account resolution **[new]** + +The one piece of identity plumbing with no mobile counterpart, and the one most likely to produce a +cross-account write. + +`app/api/offline/**` routes are addressed the way every other authenticated route here is: by +`?slot=N` (§1.2). The resolution rule, in order, all steps required: + +1. Read `jmap_session[_slot]`; `decryptSession` ⇒ `{serverUrl, username}`. +2. `accountId = generateAccountId(username, serverUrl)` — the *server-confirmed* username from the + cookie, not a client-supplied one. +3. Open the store for `accountId`. **Never** derive a path from `slot`. Slots are recycled by + `getNextCookieSlot()`, so a stale slot number pointing at a re-added different account is an + ordinary occurrence, not an edge case. +4. Cross-check against the JMAP session's own `username` (`client.ts:3822`'s `getSessionUsername()`, + which exists precisely because a short login name may be canonicalized server-side — and which + `AccountEntry.serverIdentifiers` was added to handle). A mismatch is a **hard error**, not a + best-effort match. +5. Every commit re-validates `(accountId, epoch)` (I6), and every network call re-verifies that the + engine's JMAP session still serves that account — **not only at cycle start**, because a cycle is + long-lived. This is M§8.3's generalisation of `jmapClientServesActiveAccount`, and it is what + makes M's D6 (persisted cross-account contamination) unreachable here rather than merely unlikely. + +### 5.4 Local mutations: not applicable in v1, and why that is a decision **[new]** + +M§5.6 makes the outbox the sole durable record of local intent and composes it into reads via a pure +`overlay.ts`; M§5.6.1 then requires fixing the outbox's fire-and-forget persistence, because that +promotion made its durability load-bearing (V1). + +**None of that machinery exists here** (§1.6): no outbox, no optimistic mutation queue, no +`patchCache()`. Mutations go straight to the server and fail when offline. + +**Decision: v1 desktop offline is read-only.** The durable store holds server-derived state only — +which is M§5.6's core property, reached by having no write path at all rather than by removing one. +Consequences, stated so they are chosen rather than discovered: + +- Marking a message read while offline does not work at all (rather than working locally and + syncing later). That is today's behaviour; the engine does not regress it. +- M§5.6.2's two explicit non-coverages (unread badge counts read a server-maintained + `Mailbox.unreadEmails` scalar and cannot be overlaid; SQL/FTS predicates see server truth) are + moot in v1 and become live the moment an outbox is added. +- **When offline mutations are added later, M§5.6 and §5.6.1 are the design** — including the + durability requirement. Do not invent a write-through into `envelope`/`body`; that is the failure + mode M removed rather than guarded (MR S11). + +### 5.5 Logout, account removal, disable, purge *[reused: M§8.4]* + +``` +purgeAccount(accountId, reason: 'logout' | 'removed' | 'feature-disabled' | 'store-format-change'): + 1. registry: { accountId, purgePending: true } # durable intent, crash-safe + 2. epoch++ # in-flight commits now rejected + 3. delete the SQLCipher key from safeStorage-protected key file -- FIRST + 4. delete accounts/.db (+ -wal, -shm) + 5. registry: remove the entry, KEEP the epoch +``` + +Ordering 3-before-4 is the security property: an interrupted purge must leave **unreadable** data. +Crash between 1 and 5 ⇒ the next launch completes the purge **before any cycle starts**. Triggers: +`account-store.removeAccount`, logout (single or all), the offline-cache setting being turned off +(MR S13 — purge, with a confirming Settings copy, since re-enabling costs a full bootstrap), and a +store-format/schema marker mismatch (M§8.4.1). **`AuthenticationError` during a cycle is not a purge +signal** — a server hiccup returning 401 must never delete a user's offline mail. + +**Lazy materialisation** (M§9.5, MR S13) is if anything more important here than on mobile: read +paths check the setting for that account **before** calling `open()`, and `open()` on a +non-materialised account returns an empty read-only store and creates **no file and no key**. A user +who never enables offline mail must not end up with an encrypted database and a keychain entry +materialised by a read path. + +--- + +## 6. Where the encryption key lives **[new]** + +Mobile used `expo-secure-store` (OS-keychain backed). Electron's equivalent is `safeStorage`. + +### 6.1 What `safeStorage` actually is, verified + +Measured in Electron 43.2.0 on macOS (main process, after `app.whenReady()`): +`isEncryptionAvailable() === true`, `encryptString`/`decryptString` round-trip correct, ciphertext +prefixed `v10` (Chromium's OSCrypt format). No Keychain prompt appeared. + +Per Electron's documented behaviour (`docs/latest/api/safe-storage`): + +- macOS: Keychain-backed. "Access to the system Keychain is required and these calls can block the + current thread to collect user input." +- Windows: DPAPI; requires the `ready` event. +- **Linux: `isEncryptionAvailable()` returns true even when no secret store exists**, in which case + items are "encrypted via hardcoded plaintext password" and `getSelectedStorageBackend()` returns + **`basic_text`**. Real backends are `gnome_libsecret`, `kwallet` / `kwallet5` / `kwallet6`; + `unknown` means it was called before `ready`. `setUsePlainTextEncryption()` forces an in-memory + password on Linux and is a no-op elsewhere. + +### 6.2 Decision: `safeStorage`, in the main process, with an explicit Linux gate + +`safeStorage` (built in, no dependency) over `keytar` (unmaintained). Per-account key, generated +once as 32 random bytes, wrapped with `safeStorage.encryptString()` and written to +`/keys/.bin`. + +The awkward part, stated rather than hidden: **`safeStorage` is a main-process API, and §2.4 put the +engine in the server process.** Options, and the choice: + +- ~~Give the server process its own key wrapping (e.g. a file with 0600 perms)~~ — rejected: that is + a key protected by nothing but filesystem permissions, i.e. materially weaker than the OS keychain + the rest of the desktop ecosystem uses, and it silently discards the one thing `safeStorage` buys. +- **Chosen: the key crosses the existing IPC bridge, in one direction, once per account per app + launch.** `main.ts` gains a single `ipcMain.handle("vnc:offline-key", …)`-shaped path that unwraps + the per-account key and hands it to the **server process** — *not* to the renderer. Mechanically + this means main.ts fetches/creates+wraps the key and passes it to the standalone server over a + small local channel established at spawn time (a `stdio` extra fd, or a one-shot loopback request + authenticated by a nonce also passed in the spawn env). The renderer is never in the path and + `window.vnc` gains nothing. + +This is a real cost of choosing A over B — B would have had the key and the database in the same +process — and it is the one place where B is genuinely simpler. It is outweighed by §2.1-for-1/2: +moving the *engine* to main to co-locate the key would drag the *credentials* there too, which is a +much larger secret-handling change (§2.2-against-1). Moving 32 bytes once per launch is the smaller +of the two. + +**Mechanism is deliberately left open** as an implementation choice between the extra-fd and +nonce-authenticated-loopback variants; both are small, and Stage A should pick whichever proves +cleaner against the packaged build. What is *not* open: the renderer must never see the key, and the +key must never be written unwrapped. + +### 6.3 Caveats to design around, not discover + +1. **Linux `basic_text` is the important one.** On a Linux desktop with no keyring daemon — an + AppImage on a minimal WM, a container, a headless CI box — `isEncryptionAvailable()` returns + **true** while the key is protected by a hardcoded password that is public knowledge. That is + *worse than an honest failure*, because it looks like it worked. **Rule: at key-creation time, + `getSelectedStorageBackend()` must be consulted, and `basic_text` must not silently proceed.** + Recommended behaviour: refuse to materialise a store, surface "offline mail can't be stored + securely on this system (no OS keyring available)", and offer an explicit opt-in that records the + downgrade. The decision on whether that opt-in exists at all is a human one (§13, open question + 2). +2. **`ready` ordering.** `safeStorage` must not be touched before `app.whenReady()`, and + `getSelectedStorageBackend()` returns `unknown` if it is. `main.ts:205-208` already does its work + inside `whenReady().then(...)`, so the key path must sit there — and, since the server spawn + happens inside `createMainWindow()`, the key must be resolved **before or as part of** the spawn. +3. **macOS Keychain vs. unsigned builds — flagged, not resolved.** Keychain ACLs are tied to app + identity. Builds are currently **unsigned** (`electron-builder.config.js`, `hardenedRuntime: + false`, VNCprodbuild step 9 open). Whether an ad-hoc-signed Electron app retains Keychain access + across an `electron-updater` upgrade, or prompts, or silently loses the item — **could not be + verified in this environment** and is not documented by Electron either way. The failure mode if + it does lose access is not data loss but "offline mail must re-bootstrap after every update", + which §5.5's purge-on-unreadable path handles gracefully. **Stage A must test this on a real + packaged build across a simulated update.** It is also an argument for step 9 (signing) being a + soft prerequisite for shipping the encrypted store to users, not merely a nice-to-have. +4. **A lost key is a purge, never a prompt.** If the wrapped key cannot be unwrapped, or the database + opens but `PRAGMA cipher_version` is empty, or the key fails (`SQLITE_NOTADB`), the response is + `purgeAccount(..., 'store-format-change')` and a fresh bootstrap. Never a "enter your password to + recover" flow — the key was never derived from a user secret, so there is nothing to enter. + +--- + +## 7. Storage interface and schema + +### 7.1 Interface *[reused: M§9.1]* + +`SyncStore` / `SyncTxn` / `SyncStoreFactory` exactly as M§9.1 defines them, including: +field-level state patches only and **no whole-struct `AccountSyncState` write** (I12, MR S1); +`advanceCursor(key, next: ChangesState)` and `seedCursor(key, commitment)`; +`putBodyIfEnvelopeExists` (F48); `enqueueBodies` insert-or-ignore that **never resets `attempts`** +(MR S12, F41); `listBodiesForEviction` reading `body.received_at` from the body table alone; +`listOrphanBodies`; `clearRecords()` clearing records **and the body queue** while *not* nulling +cursors; `loadAccountState()` throwing `CorruptStateError` so the caller applies I13; the +`StoreFormatMarker` read/write pair and `completePendingPurges()` running once at launch before any +cycle. + +The engine imports `SyncStore` and nothing else about persistence — no SQL, no binding import, no +path strings outside `store*.ts`. Two backends: `store-sqlite.ts` (`@signalapp/sqlcipher`) and +`store-memory.ts` (unit tests, and the second implementation that proves the boundary). + +**One Electron addition:** `SyncStoreFactory.open()` must assert encryption positively — +`PRAGMA cipher_version` non-empty — and throw otherwise. §3.1's silent-`PRAGMA key` landmine makes +this the difference between an encrypted store and a plaintext one. + +### 7.2 Backend notes *[adapted: M§9.2]* + +M§9.2's staging question (AsyncStorage vs `expo-sqlite`, plain vs cipher) is **closed here by §3.2**: +one backend, encrypted, from the first commit. M's contingency section does not apply — there is no +key-value fallback worth building in a process that has a filesystem. + +Concrete choices for this binding: + +- `PRAGMA journal_mode = WAL` and `synchronous = NORMAL`. WAL means the `-wal`/`-shm` siblings must + be included in every delete path (§5.5 step 4) — a classic leak. +- `PRAGMA key` is set as the **first statement after open**, before any other statement, then + `cipher_version` is asserted (§7.1). +- Synchronous API on a worker thread (§2.4), so a long `BEGIN…COMMIT` cannot stall an HTTP response. +- `transaction()` is a real `BEGIN…COMMIT`, so **cursor-last (I1) is enforced by the database** rather + than by write ordering — the payoff M§9.2 predicted for shipping SQLite before the engine. +- `void setItem(...).catch(warn)` and `try { … } catch { /* ignore */ }` around a store write are + **banned** in the sync path (I4; §1.6's note about `file-store.ts`). + +### 7.3 Schema *[reused: M§9.3]* + +M§9.3 verbatim: `mailbox`, `envelope`, `email_mailbox`, `body`, `body_queue`, `sync_state`, all +primary keys `(jmap_account_id, id)` per MR S3; `envelope_received` and `envelope_nobody` indexes +(the latter being job C2's driver); `email_mailbox_by_mailbox`; `body.received_at` present so +eviction is a single-table ordered scan (MR S12); **deliberately no foreign keys and no cascades** +(M§5.5's transient inconsistency is normal; a cascade on mailbox delete would delete mail, violating +I7). + +One field to add for this repo: `envelope.blob_id`, since `blobId` is in this codebase's +`EMAIL_LIST_PROPERTIES` (§4.2). + +`sync_state` living in the same file as the records is what makes §5.5's atomic wipe work. + +### 7.4 What the renderer reads, and how + +New routes under `app/api/offline/`, all gated per §2.4 and resolved per §5.3: + +| Route | Backs | +|---|---| +| `GET /api/offline/emails?slot&mailboxId&limit&before` | the offline mailbox list (indexed `queryEnvelopes`) | +| `GET /api/offline/email/:id?slot` | a single cached message incl. body | +| `GET /api/offline/status?slot` | phase/progress/coverage/error for the UI | +| `POST /api/offline/sync?slot` | user-initiated "sync now" (coalesces, never aborts — M§10.3, D7) | +| `DELETE /api/offline/store?slot` | clear cache / purge (§5.5) | + +The engine→UI direction only, per M§5.7: the engine notifies "account X changed"; the renderer +decides whether to refresh. The engine never reads renderer state. + +### 7.5 Reserved hooks *[reused: M§9.4]* + +FTS5 (step 9) hangs off `upsertEnvelopes` / `putBodyIfEnvelopeExists` as the only write paths for +indexable content — no engine change. §3.4: FTS5 is compiled in. Attachment blobs are out of scope; +when added, their deletion belongs in `deleteEmails` and `purge` so they cannot leak past an account +wipe. + +--- + +## 8. Triggering **[adapted: M§10]** + +The trigger *model* is M's; the trigger *set* is not, because a desktop app has different lifecycle +events than a mobile one (no `AppState` backgrounding, no OS-governed background budget, but real +window minimise/hide, system sleep/wake, and a process that can outlive its window on macOS). + +### 8.1 Triggers + +| # | Trigger | Jobs | Throttle | vs. M | +|---|---|---|---|---| +| T1 | Server process ready + an account's credentials resolvable | A, B, C | 2 s delay | M T1 | +| T2 | Window shown / focused (`BrowserWindow` `focus`, via a small IPC ping) | A, C | min 30 s since last cycle | M T2 (`AppState` → active) | +| T3 | User "sync now" (`POST /api/offline/sync`) | A, B, C | none; **coalesces into a running cycle, never aborts it** | M T3, closes M's D7 | +| T4 | Network regained | A, C | 3 s debounce + per-account jitter | M T4 | +| T5 | `StateChange` on the **engine's own** WS/SSE connection (§2.5) | A, C | 2 s debounce + M§10.4's state-equality check | M T5 | +| T6 | Retention setting changed | B (envelope widen), C2 (body widen), eviction only (narrow) | none | M T6 | +| T9 | **Unfinished work:** previous cycle `partial`, or any cursor `drainPending`, or `coverage.phase ∈ {scanning, reconciling}`, or a non-empty body queue | the unfinished job(s) | 5 s, subject to §8.3's chaining rule | M T9 (MR S8) | +| T10 | Offline caching disabled for an account | abort + purge (§5.5) | none | M T10 (MR S13) | +| **T11** | **System resume from sleep** (`powerMonitor` `resume`), and `unlock-screen` | A, C | 5 s debounce; treat as network-uncertain, so T4's logic applies | **[new]** — no mobile analogue; a laptop lid closed for a day is the single most common way a desktop cursor gets far behind | +| **T12** | **App quit requested** | none — *cooperative stop* | n/a | **[new]** — see §8.4 | + +Explicitly **not** triggers: a periodic timer; opening a mailbox; opening a message; scrolling. The +engine must never be on the critical path of a UI interaction (M§10.2) — if it is, its budgets and +backoff become user-visible latency. + +M's T8 (OS background refresh) has no counterpart: on desktop the process simply keeps running, so +`partial`+T9 covers it. + +### 8.2 Budgets **[adapted: M§6.4]** + +M's foreground/background split is replaced by a **window-visible / window-hidden** split. A hidden +window on a plugged-in laptop is not the constrained environment a backgrounded phone is, so the +hidden column is *lower for politeness to the server and the user's battery*, not because an OS will +kill us: + +| Bound | Window visible | Window hidden / minimised | +|---|---|---| +| Pages per cycle, per cursor | 40 | 20 | +| Wall clock per cycle | 90 s soft deadline, checked between pages | 60 s | +| Body queue items per cycle (C1+C2) | 200 | 100 | +| Coverage pages per cycle | 25 | 15 | + +Exceeding a budget is a **normal** outcome (`partial`), not an error (M§6.4): the cursor stands at +the last committed page, `drainPending` stays true, T9 resumes. This is also the answer to a server +whose `hasMoreChanges` never goes false (F14). + +### 8.3 Single-flight, coalescing, chaining *[reused: M§10.3]* + +Per `LocalAccountId`: a second trigger during a cycle sets `wakePending` and awaits the same promise +— it never aborts (M's D7). Chained cycles continue **only while `madeProgress` is true**, so fixing +M's stall (MR S8) does not create a hot loop. + +Abort triggers here: logout/purge, offline caching disabled (T10), the account being removed, network +loss, a budget deadline, **and app quit (T12)**. All leave a committed cursor and resumable state. + +**Cross-account:** M is limited to the active account because `jmapClient` is a renderer singleton. +That constraint does **not** exist here — the engine constructs its own per-account JMAP layer from +per-slot credentials (§5.3), so it can sync **all logged-in accounts**, active or not, with a worker +per account. This is a genuine capability gain from choosing A, and one of the few places this design +is *more* capable than M. It is also a new load consideration: up to 5 accounts × (1 WS + delta +traffic) against one Stalwart. M§7.2's jitter is what keeps T4/T11 from producing a synchronised +stampede, and it becomes more important here than there. + +### 8.4 Process lifetime **[new]** + +M§10.5's headless-callability constraint holds trivially — the engine has no React, no store, no +component dependency by construction (§2.4). Two Electron-specific rules: + +- **`main.ts` currently kills the server on `window-all-closed` and `before-quit` (`:210-219`) with + `serverProcess.kill()`** — SIGTERM, no coordination. A cycle dies mid-page. That is *safe* (I1: the + cursor is the last fully-applied page; cost is one page's refetch) but wasteful, and it is worth + T12: a `before-quit` that asks the engine to stop at the next page boundary, with a short timeout + before falling through to the existing kill. Small change, and it must not be allowed to delay quit + perceptibly. +- **macOS keeps the app alive with no windows.** `window-all-closed` does not `app.quit()` on darwin + (`:210-215`) yet *does* stop the server. So on macOS today, closing the window stops sync and + reopening restarts it. Acceptable for v1; worth revisiting if "sync while closed" is ever wanted, + because that is the only configuration where a desktop mail client can usefully sync with no UI. + +--- + +## 9. Failure modes **[reused + new rows]** + +**M§11's table (F1–F49) applies in full and is not reproduced here.** Every row is a JMAP-protocol or +engine-state scenario, and none of them changes because the host process changed. The ones most worth +re-reading before implementing: F1 (kill mid-drain), F3 (kill mid-bootstrap), F4 (kill mid-purge), +F9 (`cannotCalculateChanges`), F26 (`updated` for an id we don't hold), F37 (concurrent write vs. +commit), F38 (retention widened during reconcile — M's worst potential data-loss bug), F44 +(clock jump), F47 (one cursor healthy, one wedged), F48 (body for a destroyed envelope). + +Electron-specific additions: + +| # | Scenario | Rule | +|---|---|---| +| **E1** | Server child process SIGTERM'd on window close / quit (`main.ts:210-219`) | Same class as M's F1: the cursor is the last fully-applied page (I1), `drainPending` survives, T1+T9 resume on next launch. Cost ≤1 page. T12 (§8.4) reduces it to ~0 but is not required for correctness. | +| **E2** | Native module fails to load in the packaged build (NFT dropped `prebuilds/`, asar, wrong arch) | Engine never constructs; `/api/offline/**` returns 404 exactly as in a hosted deployment; the app is fully functional online-only. **Must never be a launch failure.** This is also why Stage A verifies packaging before any engine code exists. | +| **E3** | `safeStorage` reports `basic_text` (Linux, no keyring) | Do **not** materialise a store. Surface "offline mail can't be stored securely here". Optional recorded opt-in (§6.3.1, §13 q2). Never silently encrypt with the public hardcoded password. | +| **E4** | Wrapped key unwraps but the DB rejects it (`SQLITE_NOTADB`), or `cipher_version` is empty | `purgeAccount(..., 'store-format-change')` + fresh bootstrap. Never a user-facing recovery prompt (§6.3.4). | +| **E5** | Keychain item lost across an `electron-updater` upgrade of an unsigned build | Same as E4 — re-bootstrap, one full sync. Cost is bandwidth, not data. Verify empirically (§12 Stage A); it is an argument for code signing. | +| **E6** | The desktop marker env var is absent (hosted Docker deployment, or a dev `next dev` run) | Engine module never constructed; routes 404. **No SQLite file is created anywhere.** The single most important non-failure in the document (§2.4). | +| **E7** | Two app instances launched against the same `userData` | Second instance's SQLite open fails or blocks on the WAL lock. Handle by requesting Electron's single-instance lock (`app.requestSingleInstanceLock()`) in `main.ts` — **not currently requested**, and worth doing on its own merits regardless of this engine. | +| **E8** | Slot reused: account A removed, account B added into A's freed `cookieSlot` | §5.3's resolve-by-cookie-then-verify-against-session makes this a no-op: B's cookie yields B's `accountId`, so B's store opens. A's store is already gone via §5.5's `removeAccount` purge. This row exists because resolving *by slot* would have been the natural shortcut and would have merged two accounts' mail. | +| **E9** | Engine's WS connection succeeds while the renderer's fails (the expected steady state, §1.3) | Correct and intended. Renderer keeps SSE for its list; engine uses WS for its cursors; **neither reads the other's state** (§2.5 rule 3). No notification is fired by the engine (§2.5 rule 2). | +| **E10** | Both connections wake on the same delivery | Both do their own work; the renderer refreshes the visible list, the engine advances its cursors. Duplicate *fetches*, never duplicate *writes* — they own disjoint state (M§5.7). | +| **E11** | Engine and renderer both refresh an OAuth access token, and the server rotates refresh tokens (`app/api/auth/token/route.ts:104-106`) | **Real hazard.** Two independent refreshers can invalidate each other's grant and log the user out. Rule: **the engine never refreshes independently.** It obtains tokens only through the existing `PUT /api/auth/token` route, in-process, so there is exactly one refresher and one rotation writer — the route. If that proves insufficient under concurrency, serialise it with a per-slot lock in the route itself. | +| **E12** | Worker thread crashes (OOM on a huge body, native fault) | Cycle counts as `failed`, cursor unchanged (M§7.1 — a crash is not StateInvalid), worker respawned with backoff, escalation ladder applies via the cursor's counters. Never a purge. | + +--- + +## 10. Required changes outside the engine + +### 10.1 A server-side JMAP layer **[new]** + +The engine cannot use `lib/jmap/client.ts`: it is a browser-`fetch` renderer class holding +credentials in memory (§1.2), and importing it server-side would drag the whole 7413-line surface +into the server bundle. It needs a small, focused JMAP client of its own under `lib/sync/jmap/`, +with **only** what M§12.1/§12.2 specify: + +- Typed results, not `null`-collapsing: `JmapResult`, `JmapMethodError` with M§12.1's + `JmapMethodErrorType` union including `'unknown'` defaulting to ServerTransient. **M's D5 is the + bug that caused its D4; building the taxonomy in from the first commit is how it never exists + here.** +- `getEmailChangesResult` / `getMailboxChangesResult` returning **branded** `ChangesState`, plus + `updatedProperties: string[] | null` on the Mailbox result (RFC 8621 §2.2). +- `getEmailProperties(ids, properties, accountId)` returning its `state` as a **`SnapshotState`**, so + M's D4 shape is a compile error rather than a code review question. +- `getMailboxProperties` for the `updatedProperties` patch path. +- `queryEmailWindow({after, before, limit, sort, anchor, anchorOffset})` for §4.6's keyset scan, + surfacing `anchorNotFound` distinctly. +- `captureStates(accountId)` — the one-request `Mailbox/get{ids:[]}` + `Email/get{ids:[]}` pair of + §4.4, returning branded `SnapshotState`s. +- Request-level error parsing (RFC 8620 §3.6.1 `application/problem+json`, `urn:…:error:limit` with + `limit: maxSizeRequest | maxCallsInRequest | maxConcurrentRequests | rateLimit`), an `AbortSignal`, + and a per-request timeout. Note `client.ts`'s plain-`fetch` path has no timeout today, so a hung + socket hangs a cycle — do not reproduce that. +- A header-capable WebSocket (`ws`) for §2.5 rule 1, with `WebSocketPushEnable`. +- `RateLimitError` + `Retry-After` handling equivalent to `client.ts:54-62`. + +**No `as ChangesState` / `as SnapshotState` cast may exist outside this layer's response parsers** +(M§6.3). Worth an eslint `no-restricted-syntax` rule. + +### 10.2 Settings + +`stores/settings-store.ts` gains, per account: `offlineCacheEnabled` (default **off**, per the +program decision M records), `offlineEnvelopeDays`, `offlineBodyDays`, `offlineMaxMB`. Enabling and +disabling both need confirming copy — disabling **purges** (§5.5). + +### 10.3 Electron shell + +`electron/main.ts`: pass `VNCMAIL_DESKTOP_STORE_DIR` on spawn (§2.4); resolve/create the per-account +wrapped key after `whenReady()` and hand it to the server process (§6.2); add T11's `powerMonitor` +hooks and T12's cooperative `before-quit`; request the single-instance lock (E7). `electron/preload.ts` +gains **nothing** — the renderer talks to the engine over HTTP, not IPC. + +### 10.4 UI + +New: an offline-status surface (phase, coverage, last error, storage used, "sync now", +"clear offline mail") reading `GET /api/offline/status`; and an offline read path in the mail list / +message view that falls back to `/api/offline/emails` and `/api/offline/email/:id` when the JMAP +request fails and the account has a materialised store. `stores/email-store.ts` is the natural place +for the fallback, mirroring where the mobile app does it — and, per M§9.5, it must check +`offlineCacheEnabled` **before** calling anything that could materialise a store. + +### 10.5 Packaging + +`serverExternalPackages: ['@signalapp/sqlcipher']` in `next.config.ts`; verification (and if needed a +copy step in `scripts/assemble-standalone.mjs`) that `prebuilds/**/*.node` reaches +`.next/standalone/node_modules`; a CI assertion that the packaged app can open an encrypted store on +every matrix OS. + +--- + +## 11. Test plan + +The `[QA]` gate. `apply.ts` being pure is what makes most of it cheap — that is why it is a hard +requirement (§4.1). + +**Unit, no network (vitest, already the repo's runner):** M§13's unit list in full — every M§11 row +expressible as `apply(localState, page, fetched) → mutations`; the cursor state machine's eight +rules; `classify()` over the whole taxonomy including the unknown-type default; the escalation ladder +asserted **monotonically non-increasing** across a range of `maxObjectsInGet` including values below +250 and below 25 (MR V2); backoff monotonic/jittered/capped with `Retry-After` override; retention +F23/F23B/F24/F24B/F25 and the F44 clock-jump guard; reconcile floor pinning (F38) and the reconcile +ceiling (F39). Minus the outbox-durability tests, which have no subject here (§5.4). + +**Type-level, compiled by `npm run typecheck`** (M§13's insistence that a type test only earns its +keep if a regression fails the build): `advanceCursor` rejects a `SnapshotState`; a plain object +literal is rejected where `EnumerationCommitment` is expected; no `as ChangesState`/`as SnapshotState` +cast exists outside the JMAP layer. + +**`SyncStore` contract tests against both backends** (`store-memory`, `store-sqlite`), including +M's S1 lost-update sequence (F37) and `clearRecords` clearing the body queue (F35). + +**Encryption, new and non-negotiable (§3.1's landmine):** + +- After a write-and-close, the raw `.db` bytes contain **no** canary string and the header is **not** + `SQLite format 3`. +- Opening with a wrong key fails; with the right key succeeds. +- `open()` throws if `PRAGMA cipher_version` is empty — i.e. the assertion of §7.1 actually fires if + someone swaps in a non-cipher binding. +- The format marker: mismatched/stale/absent ⇒ `purgeAccount('store-format-change')` at launch + **before** any cycle; a crash between materialising a store and writing its marker leaves a + mismatch (safe), not a false match (M§8.4.1). + +**Integration against real Stalwart — cheap here, unlike mobile.** `integration/docker-compose.yml` ++ `integration/tests/` already exist in this repo with a real Stalwart, real SMTP injection and an +Electron spec (§1.6). Extend with M§13's integration list, all of which apply: + +- Bootstrap → deliver mail *during* the coverage scan → assert the first delta cycle picks it up. + M calls this the highest-value test in the list and it is the §4.4 ordering test. +- Multi-page drain with `maxChanges` forced to 2; kill the server child between pages; relaunch; + assert convergence with no duplicates or omissions (F1/E1). +- Flag toggle from a second client → assert the envelope's `keywords` update and **no body refetch** + (a network assertion, not just a state assertion — this is the §4.5 3-property rule). +- Mailbox delete with `onDestroyRemoveEmails` both true and false (F7). +- Force `cannotCalculateChanges` → assert reconcile runs, records stay readable throughout, delta + keeps flowing during the enumeration (F49), and the sweep deletes exactly the server-absent ids. +- **Widen retention mid-reconcile** → assert nothing in the gap is deleted (F38). M calls this the + test for its worst potential data-loss bug. +- Two-account isolation, plus an explicit regression for M's D6: interleave account switching with + in-flight fetches, assert no row lands under the wrong account. Add E8: remove an account, add a + different one that lands in the freed `cookieSlot`, assert no bleed. +- Purge: kill mid-purge, relaunch, assert no records and no surviving cursor (F4/F22). +- **E6, the hosted-deployment gate:** boot the standalone server *without* the marker env var, hit + every `/api/offline/**` route, assert 404 and assert **no file was created** anywhere. +- **E9/E10:** with the engine's WS connection live, assert exactly one OS notification per delivery + and that the renderer's path is the one that fired it. + +**Electron-level (`npm run test:electron`, the existing required CI gate):** the packaged app opens +an encrypted store on each matrix OS (E2 negative case: a build with the binding deliberately +removed still launches and works online-only); the Linux runner asserts the `basic_text` refusal path +(E3) since a GitHub Linux runner has no keyring — a free, realistic test of the exact configuration +§6.3.1 is about. + +**Property/fuzz (M§13, cheap and high yield):** generate random legal change pages with M§5.4's +permitted overlaps and random kill points; assert the store converges to the same state as a +from-scratch bootstrap. + +--- + +## 12. Rollout, and the verify-first gate + +M§14's shape, with M's own lesson applied: its V4 finding was that a whole staging decision rested on +an untested premise (`expo-sqlite` works in Expo Go). The premises here have been tested (§3.1) — +**except the packaging ones**, which cannot be tested without installing into this repo and building. +So Stage A exists for exactly those. + +**Stage A — packaging and key storage, before a line of engine code.** *All of it is verification; +none of it is engine logic. If any item fails, the design changes before it is built, not after.* + +1. Add `@signalapp/sqlcipher` + `serverExternalPackages`. Run `npm run build:standalone` and assert + `prebuilds/-/@signalapp+sqlcipher.node` is present under + `.next/standalone/node_modules`. If NFT dropped it, add the copy step to + `assemble-standalone.mjs`. +2. Open an encrypted database from an `app/api/**` route in a **packaged** (`--dir`) build on macOS, + and confirm the canary/header assertions of §11 against the real file. Repeat on Windows and Linux + in CI. +3. Confirm cross-arch macOS packaging ships the right `.node` in each of the x64 and arm64 outputs + (§3.3.4). +4. Resolve a `safeStorage`-wrapped key in `main.ts` and get it into the server process (§6.2); pick + between the extra-fd and nonce-loopback variants on what actually proves cleaner. +5. On Linux, assert `getSelectedStorageBackend()` and that `basic_text` takes the refusal path (E3). +6. **Simulate an `electron-updater` upgrade of an unsigned build and check Keychain access survives** + (E5, §6.3.3). This is the one item that could plausibly change the shipping plan — if an unsigned + build loses its key on every update, the encrypted store should probably wait for step 9 + (signing), and the human should be told so rather than shipping a store that re-bootstraps + monthly. + +**Stage B — pure logic, no engine.** `states.ts` (with M's `Symbol()` note), `errors.ts`, `apply.ts`, +`retention.ts`, fully unit-tested. Type-level tests wired into `npm run typecheck`. + +**Stage C — store.** `SyncStore` + `store-memory.ts` + `store-sqlite.ts` + the format marker + the +contract tests against both backends + the §11 encryption tests. + +**Stage D — JMAP layer** (§10.1), with the taxonomy and branded returns. Includes the header-capable +WebSocket, tested against the real fixture — this is where §2.5 rule 1 gets proven or disproven, and +if the fixture's `/jmap/ws` behaves like the sandbox's (§1.3) this is where we find out that +server-side WS works. + +**Stage E — cursors + delta drain** (A1/A2). **Stage F — coverage + bootstrap** (B). **Stage G — +bodies** (C1, C2). **Stage H — triggers, routes, UI.** + +Feature flag: `offlineCacheEnabled`, default off, per account (§10.2). It gates route registration +and trigger registration, not just the engine body. + +The Electron smoke gate (`npm run test:electron`) and the integration suite must stay green at every +stage. + +--- + +## 13. Summary of key decisions + +1. **The engine and the SQLite file live in the standalone Next.js server process (option A)**, + on a `worker_threads` Worker, not on the request loop. Decisive reasons: the per-account + credentials are *already there* in httpOnly encrypted cookies (§1.2), so nothing secret crosses a + process boundary; and a Node process can put an `Authorization` header on a WebSocket upgrade, + which is the exact thing that makes RFC 8887 push unreachable from the renderer today + (`client.ts:6038-6059`). Option B was rejected because it can only be built by moving credentials + into a process that currently holds none — the change `client.ts` explicitly declined. Option C + was rejected because WASM SQLite needs `'wasm-unsafe-eval'` added to the **product-wide** CSP + (`proxy.ts`), and its only encrypted backends are small third-party WASM builds. +2. **The renderer's push pipeline does not change.** The engine gets its own header-capable + connection; `StateChange` is a wake signal and never a cursor (M§10.4); the engine **never fires a + notification** — `newEmailNotification` stays the single source; and the two cursors never read + each other's state. Two sockets per account is the deliberate price of that isolation. Collapsing + to one (renderer listening to the local server instead of Stalwart) is a *follow-up*, gated on the + engine's connection being proven. +3. **SQLCipher ships on day one, via `@signalapp/sqlcipher@4.0.3`.** Verified in this environment + against Electron 43.2.0: N-API prebuilds load with **no rebuild** in both the main process and + `ELECTRON_RUN_AS_NODE`, real SQLCipher 4.10.0, encrypted file header, wrong key rejected, FTS5 + present, AGPL-3.0-only matching this repo. The mobile design's plaintext-first phase existed + solely because Expo Go cannot load SQLCipher; **that constraint has no Electron analogue**, and + importing the staging anyway would mean shipping a plaintext mailbox for a phase plus building and + discarding a migration path to leave it. +4. **`node:sqlite` rejected** (no encryption at any price — `PRAGMA key` is a *silent no-op* that + leaves the mailbox in cleartext; and stability 1.2/RC in Node 24, which is what Electron 43 + bundles). **`better-sqlite3-multiple-ciphers` rejected**: newest release has Electron prebuilds + up to ABI 146, Electron 43 needs 148, so it means a C++ toolchain on every machine — and that lag + recurs at every Electron major by construction. Plain `better-sqlite3@13.0.2` is the fallback + only. +5. **Every store open asserts `PRAGMA cipher_version` is non-empty, and a test asserts a canary is + absent from the raw file bytes.** Silent-plaintext is the sharpest landmine found in this + investigation and it has no error to notice. +6. **The hosted-deployment gate is part of the design, not a convention.** The same server process + runs in Docker for many users. One env var (`VNCMAIL_DESKTOP_STORE_DIR`, set only by `main.ts`) + both enables the engine and supplies its path; the routes 404 without it; a test asserts no file + is created without it (E6). +7. **Keys: `safeStorage`** (built in, no `keytar`), per account, wrapped and written under the store + directory; the unwrapped key goes main → **server** process only, never to the renderer. + Consulting `getSelectedStorageBackend()` is mandatory: Linux returns + `isEncryptionAvailable() === true` while using a *public hardcoded password* (`basic_text`), which + is worse than an honest failure. +8. **Account identity is `AccountEntry.id` (`username@host`), never `cookieSlot`.** Slots are + recycled; resolution is cookie → `decryptSession` → `generateAccountId` → cross-check against the + session's confirmed username, and every commit re-validates `(accountId, epoch)` (E8). +9. **v1 desktop offline is read-only.** This repo has no outbox and no optimistic mutation layer, so + M§5.6's read-time overlay has nothing to overlay. When offline mutations land, M§5.6/§5.6.1 is the + design — including its durability requirement — and a write-through into `envelope`/`body` remains + forbidden. +10. **The engine syncs all logged-in accounts, not just the active one** — a capability the mobile + engine lacks because its JMAP client is a renderer singleton. Consequence: M§7.2's jitter matters + more here, since T4/T11 fire for every account at once. +11. **Everything else is the mobile design, deliberately unchanged**: the three state machines with + sequential execution (I11), independent envelope/body retention tiers, cursor provenance as an + ordering rule with branded types and an unforgeable `EnumerationCommitment`, capture-cursors- + before-enumerate, the 3-property `updated` fetch, cursor-last, the seven-class error taxonomy + with exactly one class moving a cursor, the pinned reconcile sweep floor, the monotonically + shrinking `maxChanges` ladder with per-cursor counters, no-deletion-by-inference, account-scoped + primary keys, the purge ordering (key before file), lazy materialisation, and the F1–F49 failure + table. + +### Open questions for the human + +1. **If Stage A item 6 shows an unsigned build loses its Keychain item across updates**, does the + encrypted store ship anyway (re-bootstrapping after each update — bandwidth, not data loss), or + wait on VNCprodbuild step 9 (Apple Developer ID, already a human-owned purchase)? This is the one + Stage A outcome that could change the plan rather than just an implementation detail. +2. **Linux with no keyring (`basic_text`, §6.3.1 / E3):** refuse outright, or offer an explicit + opt-in that records the downgrade? Refusing is the safe default and what this design specifies; + an opt-in is defensible for a single-user machine with full-disk encryption. Affects a real + segment — AppImage users on minimal window managers. +3. **Concrete retention defaults** — `offlineEnvelopeDays` ≫ `offlineBodyDays` and the MB cap on + bodies only are settled (M§2.1); the *numbers* are not recorded anywhere. Two constants; the + design is value-independent. Same open question M ends on. +4. **Does the follow-up in §2.5 (collapse to one socket per account by having the renderer listen to + the local server instead of Stalwart) get scheduled?** It would retire the renderer's WS + circuit-breaker path as dead code and halve the connection count, but it makes a working + notification path depend on the new one, so it is deliberately not in v1. +5. **Should this land as its own PR ahead of the offline engine?** Stage A is pure verification and + §10.3's `app.requestSingleInstanceLock()` (E7) plus §8.4's cooperative quit are improvements to + the shell on their own merits, independent of any offline store. + +None of these blocks starting Stage A. + +--- + +## 14. What was verified, and what was not + +Stated explicitly, in M's spirit — its V4 finding was precisely that an untested premise had been +presented as settled. + +**Verified by execution in this environment, against `electron@43.2.0`:** Electron's bundled Node +(24.18.0) and ABI (148/napi 10); its bundled SQLite (3.53.1, FTS5 on); `node:sqlite`'s presence, +absence of encryption, and the *silent* no-op of `PRAGMA key` including the plaintext canary in the +file bytes; `better-sqlite3@13.0.2`'s in-tarball N-API prebuilds and successful load in both process +modes; `@signalapp/sqlcipher@4.0.3`'s load in Electron, SQLCipher 4.10.0, encrypted header, absent +canary, wrong-key rejection, right-key read-back, and FTS5; `safeStorage.isEncryptionAvailable()`, +round-trip and `v10` ciphertext prefix on macOS. + +**Verified by reading published metadata:** `better-sqlite3` 13's move off `prebuild-install`; +`better-sqlite3-multiple-ciphers`' Electron ABI coverage (121–146, no 148) and release cadence; +the existence and provenance of `@7mind.io/sqlcipher-wasm` and `@aztec/sqlite3mc-wasm`; `node:sqlite`'s +stability index (1.2, RC) in Node 24; `safeStorage`'s Linux `basic_text` fallback and +`getSelectedStorageBackend()` values. + +**NOT verified — flagged for Stage A, in descending order of how much they could change the design:** + +1. Whether an **unsigned** Electron app retains its macOS Keychain item across an `electron-updater` + upgrade (§6.3.3, E5). Not documented by Electron either way. Could change *when* the encrypted + store ships. +2. Whether Next.js **output file tracing** carries `@signalapp/sqlcipher`'s `prebuilds/` into + `.next/standalone/node_modules` (§2.1-against-2, §3.3.2). Fallback is a copy step in a script + that already exists for exactly this class of omission. +3. Whether the cross-arch macOS build ships the correct per-arch `.node` (§3.3.4). +4. Whether the integration fixture's Stalwart `/jmap/ws` accepts a header-authenticated upgrade — the + sandbox's does *require* the header (§1.3), which is what makes the server-side connection work in + principle, but it has not been driven from Node here. Stage D. +5. The choice of mechanism for getting the unwrapped key from `main.ts` into the server process + (extra fd vs. nonce-authenticated loopback) — deliberately left to whichever proves cleaner + against a packaged build (§6.2).