Files
SRCmail/lib/mail-index/jmap.ts
T
Bernd RodlerandClaude Sonnet 5 0271df4338 fix(mail-index): real end-to-end verification, and the three bugs it found
Adds integration/tests/12-electron-mail-index.spec.ts (3 tests, all passing
against the real Stalwart fixture) and fixes what running it exposed. None of
these were visible from reading the code.

1. JMAP session fetch never followed a redirect. Stalwart 307-redirects
   /.well-known/jmap to /jmap/session, and fetchJmapSession used
   `redirect: 'manual'` and treated any non-2xx as failure - so every reindex
   died with "JMAP session fetch failed (307)". Now follows up to 3 hops and
   REFUSES to follow off-origin, because the user's credentials ride on every
   hop; a blind `redirect: 'follow'` would hand the Authorization header to
   whatever host a misconfigured session pointed at. Same bound and same
   reasoning as lib/auth/verify-jmap-auth.ts.

2. The fd-3 key channel could only be adopted once per process, but its state
   was module-scoped. Next re-evaluates route modules, so a second instance hit
   `Could not open fd 3: Error: open EEXIST` from libuv. State moved to a
   Symbol on globalThis - the one place in a Node process that survives module
   re-evaluation.

3. Next's output file tracing does NOT carry @signalapp/sqlcipher's prebuilds/
   into .next/standalone. It traced the package's JS and its node-gyp-build
   dependency, but node-gyp-build resolves the .node binary by scanning a
   directory at runtime, which no static tracer can follow - so `require()`
   would have failed in every packaged build. scripts/assemble-standalone.mjs
   now copies it, alongside the public/ and .next/static copies it already does
   for the same "standalone output omits things" reason. All six platform/arch
   prebuilds are copied, not just this host's, because electron-builder
   cross-builds the x64 and arm64 macOS targets from one runner.

The three tests, and why it takes three - two constraints made a single
configuration impossible, and both were measured rather than assumed:

  * The renderer cannot reach this fixture from a production build. Its CSP
    pins connect-src to `'self' https: wss:` and the fixture's Stalwart is
    plain HTTP. NODE_ENV=development at RUNTIME does not help: `next build`
    INLINES process.env.NODE_ENV into the compiled middleware, so proxy.ts's
    `isDev` is frozen at build time (observed: a standalone server started with
    NODE_ENV=development still served the production CSP).
  * The fd-3 channel cannot survive `next dev`, which forks its server with an
    IPC channel that claims fd 3 (EEXIST); fd 4 there is not a pipe either
    (ENOTTY).

  So: PIPELINE drives the real standalone server over HTTP from Node with a
  real fd-3 key channel (no browser, so no CSP) and asserts a real SMTP
  delivery is findable by a word from its BODY, with a real snippet and
  contextBlock, idempotent catch-up, working type filters, and - reading the
  raw bytes of the .db AND its -wal - that nothing is recoverable in cleartext.
  TRIGGER proves the event-driven wiring: a real delivery makes the renderer
  POST /api/offline/reindex off its live push. WIRING launches the real shell
  with no ELECTRON_LOAD_URL and asserts the routes are reachable (401, not 404
  or 503) with real safeStorage behind them.

Each test now gets its own --user-data-dir. That is load-bearing, not hygiene:
Electron reuses one profile across launches, and a leftover jmap_stalwart_ctx
cookie from an earlier run made the WIRING test's 401 assertion pass as a 200.

Verified: typecheck clean; unit suite 2379 tests with the SAME 3 pre-existing
failures as the base commit b15098a6 (2 builtin-themes, 1 jmap-client-
resilience) and 48 net new passing; both `docker build`s succeed; the
hosted-deployment gate returns 404 with an empty body and materialises no file
in the production image; e2e/electron-smoke 4/4; 11-electron-notification
still passes.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-04 23:43:52 +02:00

389 lines
15 KiB
TypeScript

// A deliberately tiny server-side JMAP client, used only by the indexer.
//
// WHY NOT REUSE lib/jmap/client.ts: that class is a 7400-line renderer object.
// It holds credentials in instance fields, uses `btoa`, opens EventSource /
// WebSocket push connections, and wires itself into Zustand stores and toast
// notifications. Importing it into an API route would drag all of that into the
// server bundle for the sake of four method calls. The existing server-side
// JMAP code in this repo (lib/auth/verify-jmap-auth.ts) already sets the
// precedent: plain fetch + an Authorization header.
//
// Everything here is stateless - the caller supplies the auth header per call,
// so there is no resident credential and nothing to invalidate.
import type { CalendarEvent, ContactCard, Email, FileNode } from '@/lib/jmap/types';
const REQUEST_TIMEOUT_MS = 30_000;
export const CAP_CORE = 'urn:ietf:params:jmap:core';
export const CAP_MAIL = 'urn:ietf:params:jmap:mail';
export const CAP_CALENDARS = 'urn:ietf:params:jmap:calendars';
export const CAP_CONTACTS = 'urn:ietf:params:jmap:contacts';
export const CAP_FILENODE = 'urn:ietf:params:jmap:filenode';
export class JmapIndexError extends Error {
status: number;
constructor(message: string, status = 502) {
super(message);
this.name = 'JmapIndexError';
this.status = status;
}
}
export interface JmapSessionInfo {
apiUrl: string;
/** Server-confirmed authenticated login (JMAP Session.username). */
username?: string;
primaryAccounts: Record<string, string>;
accounts: Record<string, { name?: string; isPersonal?: boolean; accountCapabilities?: Record<string, unknown> }>;
capabilities: Record<string, unknown>;
}
/**
* Pins a URL advertised by the session to the origin we authenticated against.
*
* `lib/jmap/client.ts` does the same thing in its rewriteSessionUrls() for the
* renderer's benefit. Server-side it is a security control, not a convenience:
* we attach the user's credentials to this URL, so a session document that
* advertised an `apiUrl` on someone else's host would turn this into a
* credential-leaking SSRF. Keep the path and query, take the origin from the
* server URL we were configured with.
*/
function pinToServerOrigin(advertised: string, serverUrl: string): string {
const base = new URL(serverUrl);
let target: URL;
try {
target = new URL(advertised, base);
} catch {
throw new JmapIndexError('JMAP session advertised an unusable apiUrl');
}
return `${base.origin}${target.pathname}${target.search}`;
}
async function fetchWithTimeout(url: string, init: RequestInit): Promise<Response> {
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), REQUEST_TIMEOUT_MS);
try {
return await fetch(url, { ...init, signal: controller.signal, redirect: 'manual' });
} catch (error) {
if (error instanceof Error && error.name === 'AbortError') {
throw new JmapIndexError('JMAP request timed out', 504);
}
throw new JmapIndexError(`JMAP request failed: ${String(error)}`);
} finally {
clearTimeout(timer);
}
}
/** Stalwart 307-redirects /.well-known/jmap to /jmap/session. */
const MAX_REDIRECTS = 3;
export async function fetchJmapSession(serverUrl: string, authHeader: string): Promise<JmapSessionInfo> {
const base = serverUrl.replace(/\/+$/, '');
const origin = new URL(base).origin;
let currentUrl = `${base}/.well-known/jmap`;
let response: Response | undefined;
// Redirects must be followed EXPLICITLY, not with `redirect: 'follow'`: we
// attach the user's credentials to every hop, so each one has to be checked to
// still be on the origin we authenticated against. A blind follow would hand
// the Authorization header to whatever host a misconfigured or hostile session
// pointed at. Same reasoning (and same bound) as lib/auth/verify-jmap-auth.ts.
for (let hop = 0; hop <= MAX_REDIRECTS; hop++) {
response = await fetchWithTimeout(currentUrl, {
method: 'GET',
headers: { Authorization: authHeader },
});
if (response.status < 300 || response.status >= 400) break;
const location = response.headers.get('location');
if (!location) throw new JmapIndexError('JMAP session redirect had no Location header');
const next = new URL(location, currentUrl);
if (next.origin !== origin) {
throw new JmapIndexError(
`JMAP session redirected off-origin (${next.origin}); refusing to send credentials there`,
);
}
currentUrl = next.toString();
}
if (!response) throw new JmapIndexError('JMAP session fetch produced no response');
if (response.status === 401 || response.status === 403) {
throw new JmapIndexError('JMAP authentication failed', 401);
}
if (response.status >= 300 && response.status < 400) {
throw new JmapIndexError('Too many redirects fetching the JMAP session');
}
if (!response.ok) {
throw new JmapIndexError(`JMAP session fetch failed (${response.status})`);
}
const raw = (await response.json().catch(() => null)) as Record<string, unknown> | null;
if (!raw || typeof raw.apiUrl !== 'string') {
throw new JmapIndexError('Invalid JMAP session response');
}
return {
apiUrl: pinToServerOrigin(raw.apiUrl, serverUrl),
username: typeof raw.username === 'string' ? raw.username : undefined,
primaryAccounts: (raw.primaryAccounts as Record<string, string>) ?? {},
accounts: (raw.accounts as JmapSessionInfo['accounts']) ?? {},
capabilities: (raw.capabilities as Record<string, unknown>) ?? {},
};
}
type MethodCall = [string, Record<string, unknown>, string];
/** Raw method-response tuple, `[name, args, callId]`. `name` is 'error' on failure. */
type MethodResponse = [string, Record<string, unknown>, string];
export async function jmapRequest(
session: JmapSessionInfo,
authHeader: string,
using: readonly string[],
methodCalls: readonly MethodCall[],
): Promise<MethodResponse[]> {
const response = await fetchWithTimeout(session.apiUrl, {
method: 'POST',
headers: { Authorization: authHeader, 'Content-Type': 'application/json' },
body: JSON.stringify({ using, methodCalls }),
});
if (response.status === 401 || response.status === 403) {
throw new JmapIndexError('JMAP authentication failed', 401);
}
if (response.status === 429) {
throw new JmapIndexError('JMAP server is rate limiting', 429);
}
if (!response.ok) {
throw new JmapIndexError(`JMAP request failed (${response.status})`);
}
const data = (await response.json().catch(() => null)) as { methodResponses?: MethodResponse[] } | null;
if (!data || !Array.isArray(data.methodResponses)) {
throw new JmapIndexError('Invalid JMAP response envelope');
}
return data.methodResponses;
}
function firstResult(responses: MethodResponse[], expected: string): Record<string, unknown> | null {
for (const [name, args] of responses) {
if (name === expected) return args;
// A method-level error is not fatal for an INDEX: a server that doesn't
// support one data type should not fail the whole reindex. The caller
// treats null as "nothing to index for this type".
if (name === 'error') return null;
}
return null;
}
function idsOf(args: Record<string, unknown> | null): string[] {
const ids = args?.ids;
return Array.isArray(ids) ? ids.filter((v): v is string => typeof v === 'string') : [];
}
function listOf<T>(args: Record<string, unknown> | null): T[] {
const list = args?.list;
return Array.isArray(list) ? (list as T[]) : [];
}
export function accountIdFor(session: JmapSessionInfo, capability: string): string | null {
const id = session.primaryAccounts[capability];
return typeof id === 'string' && id.length > 0 ? id : null;
}
export function hasCapability(session: JmapSessionInfo, capability: string): boolean {
return Object.prototype.hasOwnProperty.call(session.capabilities, capability);
}
/** Per-ACCOUNT capability, mirroring client.ts's supportsFiles() (#563: a server can advertise it while an account has it revoked). */
export function accountHasCapability(
session: JmapSessionInfo,
accountId: string,
capability: string,
): boolean {
const account = session.accounts[accountId];
if (!account) return false;
if (account.accountCapabilities && Object.prototype.hasOwnProperty.call(account.accountCapabilities, capability)) {
return true;
}
return account.isPersonal === false;
}
// ── mail ────────────────────────────────────────────────────────────────────
/** Properties needed to build a mail IndexDoc. Bodies come via bodyValues. */
const EMAIL_INDEX_PROPERTIES = [
'id', 'threadId', 'mailboxIds', 'keywords', 'size', 'receivedAt',
'from', 'to', 'cc', 'subject', 'preview', 'hasAttachment',
'textBody', 'htmlBody', 'bodyValues',
] as const;
export async function getEmailsForIndex(
session: JmapSessionInfo,
authHeader: string,
accountId: string,
ids: readonly string[],
maxBodyBytes: number,
): Promise<Email[]> {
if (ids.length === 0) return [];
const responses = await jmapRequest(session, authHeader, [CAP_CORE, CAP_MAIL], [
['Email/get', {
accountId,
ids: [...ids],
properties: [...EMAIL_INDEX_PROPERTIES],
// Without these two the bodyValues map comes back EMPTY and every
// indexed body would silently fall back to `preview`.
fetchTextBodyValues: true,
fetchHTMLBodyValues: true,
maxBodyValueBytes: maxBodyBytes,
}, 'g'],
]);
return listOf<Email>(firstResult(responses, 'Email/get'));
}
export async function queryRecentEmailIds(
session: JmapSessionInfo,
authHeader: string,
accountId: string,
afterIso: string,
limit: number,
): Promise<string[]> {
const responses = await jmapRequest(session, authHeader, [CAP_CORE, CAP_MAIL], [
['Email/query', {
accountId,
filter: { after: afterIso },
sort: [{ property: 'receivedAt', isAscending: false }],
limit,
calculateTotal: false,
}, 'q'],
]);
return idsOf(firstResult(responses, 'Email/query'));
}
// ── calendar ────────────────────────────────────────────────────────────────
export async function getCalendarEventsForIndex(
session: JmapSessionInfo,
authHeader: string,
accountId: string,
ids: readonly string[],
): Promise<CalendarEvent[]> {
if (ids.length === 0) return [];
const responses = await jmapRequest(session, authHeader, [CAP_CORE, CAP_CALENDARS], [
['CalendarEvent/get', { accountId, ids: [...ids] }, 'g'],
]);
return listOf<CalendarEvent>(firstResult(responses, 'CalendarEvent/get'));
}
export async function queryCalendarEventIds(
session: JmapSessionInfo,
authHeader: string,
accountId: string,
afterIso: string,
beforeIso: string,
limit: number,
): Promise<string[]> {
const responses = await jmapRequest(session, authHeader, [CAP_CORE, CAP_CALENDARS], [
['CalendarEvent/query', {
accountId,
// LocalDateTime, per the note in lib/jmap/client.ts:307-312 - Stalwart
// parses these without a zone suffix and ignores unparseable values.
filter: { after: toLocalDateTime(afterIso), before: toLocalDateTime(beforeIso) },
limit,
calculateTotal: false,
}, 'q'],
]);
return idsOf(firstResult(responses, 'CalendarEvent/query'));
}
/** JSCalendar LocalDateTime: `YYYY-MM-DDTHH:MM:SS`, no zone designator. */
function toLocalDateTime(iso: string): string {
return iso.replace(/\.\d+/, '').replace(/Z$/, '').slice(0, 19);
}
// ── contacts ────────────────────────────────────────────────────────────────
export async function getContactsForIndex(
session: JmapSessionInfo,
authHeader: string,
accountId: string,
ids: readonly string[],
): Promise<ContactCard[]> {
if (ids.length === 0) return [];
const responses = await jmapRequest(session, authHeader, [CAP_CORE, CAP_CONTACTS], [
['ContactCard/get', { accountId, ids: [...ids] }, 'g'],
]);
return listOf<ContactCard>(firstResult(responses, 'ContactCard/get'));
}
export async function queryContactIds(
session: JmapSessionInfo,
authHeader: string,
accountId: string,
limit: number,
): Promise<string[]> {
const responses = await jmapRequest(session, authHeader, [CAP_CORE, CAP_CONTACTS], [
['ContactCard/query', { accountId, limit, calculateTotal: false }, 'q'],
]);
return idsOf(firstResult(responses, 'ContactCard/query'));
}
// ── files ───────────────────────────────────────────────────────────────────
const FILENODE_INDEX_PROPERTIES = [
'id', 'parentId', 'name', 'type', 'blobId', 'size', 'created', 'modified',
] as const;
export async function getFilesForIndex(
session: JmapSessionInfo,
authHeader: string,
accountId: string,
ids: readonly string[],
): Promise<FileNode[]> {
if (ids.length === 0) return [];
const responses = await jmapRequest(session, authHeader, [CAP_CORE], [
['FileNode/get', { accountId, ids: [...ids], properties: [...FILENODE_INDEX_PROPERTIES] }, 'g'],
]);
return listOf<FileNode>(firstResult(responses, 'FileNode/get'));
}
export async function queryFileIds(
session: JmapSessionInfo,
authHeader: string,
accountId: string,
limit: number,
): Promise<string[]> {
const responses = await jmapRequest(session, authHeader, [CAP_CORE], [
['FileNode/query', { accountId, filter: {}, limit, calculateTotal: false }, 'q'],
]);
return idsOf(firstResult(responses, 'FileNode/query'));
}
/**
* Builds `id -> "Parent/Child"` paths for the given nodes, walking `parentId`
* upward. FileNode only knows its parent, so the caller has to assemble this;
* unresolvable ancestors just truncate the path rather than failing.
*/
export function buildFilePaths(nodes: readonly FileNode[]): Map<string, string> {
const byId = new Map(nodes.map((n) => [n.id, n]));
const cache = new Map<string, string>();
const resolve = (id: string, depth: number): string => {
if (depth > 32) return '';
const cached = cache.get(id);
if (cached !== undefined) return cached;
const node = byId.get(id);
if (!node) return '';
const parent = node.parentId ? resolve(node.parentId, depth + 1) : '';
const full = parent ? `${parent}/${node.name}` : node.name;
cache.set(id, full);
return full;
};
const out = new Map<string, string>();
for (const n of nodes) {
// The document's own `path` metadata is its PARENT directory chain, so a
// search for "Invoices" matches files inside it without the filename
// being duplicated into the body.
out.set(n.id, n.parentId ? resolve(n.parentId, 0) : '');
}
return out;
}