fix(mail-index): real end-to-end verification, and the three bugs it found

Adds integration/tests/12-electron-mail-index.spec.ts (3 tests, all passing
against the real Stalwart fixture) and fixes what running it exposed. None of
these were visible from reading the code.

1. JMAP session fetch never followed a redirect. Stalwart 307-redirects
   /.well-known/jmap to /jmap/session, and fetchJmapSession used
   `redirect: 'manual'` and treated any non-2xx as failure - so every reindex
   died with "JMAP session fetch failed (307)". Now follows up to 3 hops and
   REFUSES to follow off-origin, because the user's credentials ride on every
   hop; a blind `redirect: 'follow'` would hand the Authorization header to
   whatever host a misconfigured session pointed at. Same bound and same
   reasoning as lib/auth/verify-jmap-auth.ts.

2. The fd-3 key channel could only be adopted once per process, but its state
   was module-scoped. Next re-evaluates route modules, so a second instance hit
   `Could not open fd 3: Error: open EEXIST` from libuv. State moved to a
   Symbol on globalThis - the one place in a Node process that survives module
   re-evaluation.

3. Next's output file tracing does NOT carry @signalapp/sqlcipher's prebuilds/
   into .next/standalone. It traced the package's JS and its node-gyp-build
   dependency, but node-gyp-build resolves the .node binary by scanning a
   directory at runtime, which no static tracer can follow - so `require()`
   would have failed in every packaged build. scripts/assemble-standalone.mjs
   now copies it, alongside the public/ and .next/static copies it already does
   for the same "standalone output omits things" reason. All six platform/arch
   prebuilds are copied, not just this host's, because electron-builder
   cross-builds the x64 and arm64 macOS targets from one runner.

The three tests, and why it takes three - two constraints made a single
configuration impossible, and both were measured rather than assumed:

  * The renderer cannot reach this fixture from a production build. Its CSP
    pins connect-src to `'self' https: wss:` and the fixture's Stalwart is
    plain HTTP. NODE_ENV=development at RUNTIME does not help: `next build`
    INLINES process.env.NODE_ENV into the compiled middleware, so proxy.ts's
    `isDev` is frozen at build time (observed: a standalone server started with
    NODE_ENV=development still served the production CSP).
  * The fd-3 channel cannot survive `next dev`, which forks its server with an
    IPC channel that claims fd 3 (EEXIST); fd 4 there is not a pipe either
    (ENOTTY).

  So: PIPELINE drives the real standalone server over HTTP from Node with a
  real fd-3 key channel (no browser, so no CSP) and asserts a real SMTP
  delivery is findable by a word from its BODY, with a real snippet and
  contextBlock, idempotent catch-up, working type filters, and - reading the
  raw bytes of the .db AND its -wal - that nothing is recoverable in cleartext.
  TRIGGER proves the event-driven wiring: a real delivery makes the renderer
  POST /api/offline/reindex off its live push. WIRING launches the real shell
  with no ELECTRON_LOAD_URL and asserts the routes are reachable (401, not 404
  or 503) with real safeStorage behind them.

Each test now gets its own --user-data-dir. That is load-bearing, not hygiene:
Electron reuses one profile across launches, and a leftover jmap_stalwart_ctx
cookie from an earlier run made the WIRING test's 401 assertion pass as a 200.

Verified: typecheck clean; unit suite 2379 tests with the SAME 3 pre-existing
failures as the base commit b15098a6 (2 builtin-themes, 1 jmap-client-
resilience) and 48 net new passing; both `docker build`s succeed; the
hosted-deployment gate returns 404 with an empty body and materialises no file
in the production image; e2e/electron-smoke 4/4; 11-electron-notification
still passes.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Bernd Rodler
2026-08-04 23:43:52 +02:00
co-authored by Claude Sonnet 5
parent 7e9aefcfa1
commit 0271df4338
6 changed files with 635 additions and 31 deletions
+35 -4
View File
@@ -75,14 +75,45 @@ async function fetchWithTimeout(url: string, init: RequestInit): Promise<Respons
}
}
/** Stalwart 307-redirects /.well-known/jmap to /jmap/session. */
const MAX_REDIRECTS = 3;
export async function fetchJmapSession(serverUrl: string, authHeader: string): Promise<JmapSessionInfo> {
const response = await fetchWithTimeout(`${serverUrl.replace(/\/+$/, '')}/.well-known/jmap`, {
method: 'GET',
headers: { Authorization: authHeader },
});
const base = serverUrl.replace(/\/+$/, '');
const origin = new URL(base).origin;
let currentUrl = `${base}/.well-known/jmap`;
let response: Response | undefined;
// Redirects must be followed EXPLICITLY, not with `redirect: 'follow'`: we
// attach the user's credentials to every hop, so each one has to be checked to
// still be on the origin we authenticated against. A blind follow would hand
// the Authorization header to whatever host a misconfigured or hostile session
// pointed at. Same reasoning (and same bound) as lib/auth/verify-jmap-auth.ts.
for (let hop = 0; hop <= MAX_REDIRECTS; hop++) {
response = await fetchWithTimeout(currentUrl, {
method: 'GET',
headers: { Authorization: authHeader },
});
if (response.status < 300 || response.status >= 400) break;
const location = response.headers.get('location');
if (!location) throw new JmapIndexError('JMAP session redirect had no Location header');
const next = new URL(location, currentUrl);
if (next.origin !== origin) {
throw new JmapIndexError(
`JMAP session redirected off-origin (${next.origin}); refusing to send credentials there`,
);
}
currentUrl = next.toString();
}
if (!response) throw new JmapIndexError('JMAP session fetch produced no response');
if (response.status === 401 || response.status === 403) {
throw new JmapIndexError('JMAP authentication failed', 401);
}
if (response.status >= 300 && response.status < 400) {
throw new JmapIndexError('Too many redirects fetching the JMAP session');
}
if (!response.ok) {
throw new JmapIndexError(`JMAP session fetch failed (${response.status})`);
}
+52 -23
View File
@@ -36,21 +36,49 @@ interface Pending {
timer: NodeJS.Timeout;
}
let socket: net.Socket | null = null;
let nextId = 1;
const pending = new Map<number, Pending>();
let readBuffer = '';
/**
* Channel state lives on `globalThis`, NOT in module scope.
*
* A file descriptor can be adopted as a socket exactly ONCE per process: a
* second `new net.Socket({ fd })` for an fd this process already owns throws
* `EEXIST` from libuv's uv_pipe_open. Module scope is not once-per-process -
* Next re-evaluates route modules (dev HMR, and separate module instances
* across route bundles), so a module-scoped `let socket` produced exactly that
* crash: `Could not open fd 3: Error: open EEXIST`, found by the integration
* test rather than by reading the code.
*
* A Symbol key on globalThis is the one place in a Node process that survives
* module re-evaluation, so adoption genuinely happens once.
*/
interface ChannelState {
socket: net.Socket | null;
nextId: number;
pending: Map<number, Pending>;
readBuffer: string;
}
function failAll(error: Error): void {
for (const [, p] of pending) {
const STATE_KEY = Symbol.for('vncmail.mailIndex.keyChannel');
function state(): ChannelState {
const holder = globalThis as unknown as Record<symbol, ChannelState | undefined>;
const existing = holder[STATE_KEY];
if (existing) return existing;
const created: ChannelState = { socket: null, nextId: 1, pending: new Map(), readBuffer: '' };
holder[STATE_KEY] = created;
return created;
}
function failAll(s: ChannelState, error: Error): void {
for (const [, p] of s.pending) {
clearTimeout(p.timer);
p.reject(error);
}
pending.clear();
s.pending.clear();
}
function getSocket(): net.Socket {
if (socket && !socket.destroyed) return socket;
const s = state();
if (s.socket && !s.socket.destroyed) return s.socket;
const raw = process.env[KEY_FD_ENV]?.trim();
const fd = raw ? Number(raw) : NaN;
@@ -73,12 +101,12 @@ function getSocket(): net.Socket {
created.unref();
created.on('data', (chunk: Buffer) => {
readBuffer += chunk.toString('utf8');
if (readBuffer.length > 64 * 1024) readBuffer = '';
s.readBuffer += chunk.toString('utf8');
if (s.readBuffer.length > 64 * 1024) s.readBuffer = '';
let newline: number;
while ((newline = readBuffer.indexOf('\n')) >= 0) {
const line = readBuffer.slice(0, newline);
readBuffer = readBuffer.slice(newline + 1);
while ((newline = s.readBuffer.indexOf('\n')) >= 0) {
const line = s.readBuffer.slice(0, newline);
s.readBuffer = s.readBuffer.slice(newline + 1);
if (!line.trim()) continue;
let msg: { id?: unknown; ok?: unknown; key?: unknown; code?: unknown; error?: unknown };
try {
@@ -88,9 +116,9 @@ function getSocket(): net.Socket {
}
const id = typeof msg.id === 'number' ? msg.id : null;
if (id === null) continue;
const p = pending.get(id);
const p = s.pending.get(id);
if (!p) continue;
pending.delete(id);
s.pending.delete(id);
clearTimeout(p.timer);
if (msg.ok === true) {
p.resolve({ key: typeof msg.key === 'string' ? msg.key : undefined });
@@ -102,32 +130,33 @@ function getSocket(): net.Socket {
});
const onGone = (error?: Error) => {
socket = null;
readBuffer = '';
failAll(error ?? new IndexKeyError('no-channel', 'Key service channel closed'));
s.socket = null;
s.readBuffer = '';
failAll(s, error ?? new IndexKeyError('no-channel', 'Key service channel closed'));
};
created.on('close', () => onGone());
created.on('error', (error) => onGone(new IndexKeyError('no-channel', String(error))));
socket = created;
s.socket = created;
return created;
}
function request(op: 'getIndexKey' | 'deleteIndexKey', accountId: string): Promise<{ key?: string }> {
const sock = getSocket();
const id = nextId++;
const s = state();
const id = s.nextId++;
return new Promise<{ key?: string }>((resolve, reject) => {
const timer = setTimeout(() => {
pending.delete(id);
s.pending.delete(id);
reject(new IndexKeyError('timeout', `Key service did not answer within ${REQUEST_TIMEOUT_MS}ms`));
}, REQUEST_TIMEOUT_MS);
// Don't let a pending key request keep the process alive either.
timer.unref?.();
pending.set(id, { resolve, reject, timer });
s.pending.set(id, { resolve, reject, timer });
try {
sock.write(`${JSON.stringify({ id, op, accountId })}\n`);
} catch (error) {
pending.delete(id);
s.pending.delete(id);
clearTimeout(timer);
reject(new IndexKeyError('no-channel', `Could not write to the key service: ${String(error)}`));
}