Adds integration/tests/12-electron-mail-index.spec.ts (3 tests, all passing
against the real Stalwart fixture) and fixes what running it exposed. None of
these were visible from reading the code.
1. JMAP session fetch never followed a redirect. Stalwart 307-redirects
/.well-known/jmap to /jmap/session, and fetchJmapSession used
`redirect: 'manual'` and treated any non-2xx as failure - so every reindex
died with "JMAP session fetch failed (307)". Now follows up to 3 hops and
REFUSES to follow off-origin, because the user's credentials ride on every
hop; a blind `redirect: 'follow'` would hand the Authorization header to
whatever host a misconfigured session pointed at. Same bound and same
reasoning as lib/auth/verify-jmap-auth.ts.
2. The fd-3 key channel could only be adopted once per process, but its state
was module-scoped. Next re-evaluates route modules, so a second instance hit
`Could not open fd 3: Error: open EEXIST` from libuv. State moved to a
Symbol on globalThis - the one place in a Node process that survives module
re-evaluation.
3. Next's output file tracing does NOT carry @signalapp/sqlcipher's prebuilds/
into .next/standalone. It traced the package's JS and its node-gyp-build
dependency, but node-gyp-build resolves the .node binary by scanning a
directory at runtime, which no static tracer can follow - so `require()`
would have failed in every packaged build. scripts/assemble-standalone.mjs
now copies it, alongside the public/ and .next/static copies it already does
for the same "standalone output omits things" reason. All six platform/arch
prebuilds are copied, not just this host's, because electron-builder
cross-builds the x64 and arm64 macOS targets from one runner.
The three tests, and why it takes three - two constraints made a single
configuration impossible, and both were measured rather than assumed:
* The renderer cannot reach this fixture from a production build. Its CSP
pins connect-src to `'self' https: wss:` and the fixture's Stalwart is
plain HTTP. NODE_ENV=development at RUNTIME does not help: `next build`
INLINES process.env.NODE_ENV into the compiled middleware, so proxy.ts's
`isDev` is frozen at build time (observed: a standalone server started with
NODE_ENV=development still served the production CSP).
* The fd-3 channel cannot survive `next dev`, which forks its server with an
IPC channel that claims fd 3 (EEXIST); fd 4 there is not a pipe either
(ENOTTY).
So: PIPELINE drives the real standalone server over HTTP from Node with a
real fd-3 key channel (no browser, so no CSP) and asserts a real SMTP
delivery is findable by a word from its BODY, with a real snippet and
contextBlock, idempotent catch-up, working type filters, and - reading the
raw bytes of the .db AND its -wal - that nothing is recoverable in cleartext.
TRIGGER proves the event-driven wiring: a real delivery makes the renderer
POST /api/offline/reindex off its live push. WIRING launches the real shell
with no ELECTRON_LOAD_URL and asserts the routes are reachable (401, not 404
or 503) with real safeStorage behind them.
Each test now gets its own --user-data-dir. That is load-bearing, not hygiene:
Electron reuses one profile across launches, and a leftover jmap_stalwart_ctx
cookie from an earlier run made the WIRING test's 401 assertion pass as a 200.
Verified: typecheck clean; unit suite 2379 tests with the SAME 3 pre-existing
failures as the base commit b15098a6 (2 builtin-themes, 1 jmap-client-
resilience) and 48 net new passing; both `docker build`s succeed; the
hosted-deployment gate returns 404 with an empty body and materialises no file
in the production image; e2e/electron-smoke 4/4; 11-electron-notification
still passes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
389 lines
15 KiB
TypeScript
389 lines
15 KiB
TypeScript
// A deliberately tiny server-side JMAP client, used only by the indexer.
|
|
//
|
|
// WHY NOT REUSE lib/jmap/client.ts: that class is a 7400-line renderer object.
|
|
// It holds credentials in instance fields, uses `btoa`, opens EventSource /
|
|
// WebSocket push connections, and wires itself into Zustand stores and toast
|
|
// notifications. Importing it into an API route would drag all of that into the
|
|
// server bundle for the sake of four method calls. The existing server-side
|
|
// JMAP code in this repo (lib/auth/verify-jmap-auth.ts) already sets the
|
|
// precedent: plain fetch + an Authorization header.
|
|
//
|
|
// Everything here is stateless - the caller supplies the auth header per call,
|
|
// so there is no resident credential and nothing to invalidate.
|
|
|
|
import type { CalendarEvent, ContactCard, Email, FileNode } from '@/lib/jmap/types';
|
|
|
|
const REQUEST_TIMEOUT_MS = 30_000;
|
|
|
|
export const CAP_CORE = 'urn:ietf:params:jmap:core';
|
|
export const CAP_MAIL = 'urn:ietf:params:jmap:mail';
|
|
export const CAP_CALENDARS = 'urn:ietf:params:jmap:calendars';
|
|
export const CAP_CONTACTS = 'urn:ietf:params:jmap:contacts';
|
|
export const CAP_FILENODE = 'urn:ietf:params:jmap:filenode';
|
|
|
|
export class JmapIndexError extends Error {
|
|
status: number;
|
|
constructor(message: string, status = 502) {
|
|
super(message);
|
|
this.name = 'JmapIndexError';
|
|
this.status = status;
|
|
}
|
|
}
|
|
|
|
export interface JmapSessionInfo {
|
|
apiUrl: string;
|
|
/** Server-confirmed authenticated login (JMAP Session.username). */
|
|
username?: string;
|
|
primaryAccounts: Record<string, string>;
|
|
accounts: Record<string, { name?: string; isPersonal?: boolean; accountCapabilities?: Record<string, unknown> }>;
|
|
capabilities: Record<string, unknown>;
|
|
}
|
|
|
|
/**
|
|
* Pins a URL advertised by the session to the origin we authenticated against.
|
|
*
|
|
* `lib/jmap/client.ts` does the same thing in its rewriteSessionUrls() for the
|
|
* renderer's benefit. Server-side it is a security control, not a convenience:
|
|
* we attach the user's credentials to this URL, so a session document that
|
|
* advertised an `apiUrl` on someone else's host would turn this into a
|
|
* credential-leaking SSRF. Keep the path and query, take the origin from the
|
|
* server URL we were configured with.
|
|
*/
|
|
function pinToServerOrigin(advertised: string, serverUrl: string): string {
|
|
const base = new URL(serverUrl);
|
|
let target: URL;
|
|
try {
|
|
target = new URL(advertised, base);
|
|
} catch {
|
|
throw new JmapIndexError('JMAP session advertised an unusable apiUrl');
|
|
}
|
|
return `${base.origin}${target.pathname}${target.search}`;
|
|
}
|
|
|
|
async function fetchWithTimeout(url: string, init: RequestInit): Promise<Response> {
|
|
const controller = new AbortController();
|
|
const timer = setTimeout(() => controller.abort(), REQUEST_TIMEOUT_MS);
|
|
try {
|
|
return await fetch(url, { ...init, signal: controller.signal, redirect: 'manual' });
|
|
} catch (error) {
|
|
if (error instanceof Error && error.name === 'AbortError') {
|
|
throw new JmapIndexError('JMAP request timed out', 504);
|
|
}
|
|
throw new JmapIndexError(`JMAP request failed: ${String(error)}`);
|
|
} finally {
|
|
clearTimeout(timer);
|
|
}
|
|
}
|
|
|
|
/** Stalwart 307-redirects /.well-known/jmap to /jmap/session. */
|
|
const MAX_REDIRECTS = 3;
|
|
|
|
export async function fetchJmapSession(serverUrl: string, authHeader: string): Promise<JmapSessionInfo> {
|
|
const base = serverUrl.replace(/\/+$/, '');
|
|
const origin = new URL(base).origin;
|
|
let currentUrl = `${base}/.well-known/jmap`;
|
|
let response: Response | undefined;
|
|
|
|
// Redirects must be followed EXPLICITLY, not with `redirect: 'follow'`: we
|
|
// attach the user's credentials to every hop, so each one has to be checked to
|
|
// still be on the origin we authenticated against. A blind follow would hand
|
|
// the Authorization header to whatever host a misconfigured or hostile session
|
|
// pointed at. Same reasoning (and same bound) as lib/auth/verify-jmap-auth.ts.
|
|
for (let hop = 0; hop <= MAX_REDIRECTS; hop++) {
|
|
response = await fetchWithTimeout(currentUrl, {
|
|
method: 'GET',
|
|
headers: { Authorization: authHeader },
|
|
});
|
|
if (response.status < 300 || response.status >= 400) break;
|
|
|
|
const location = response.headers.get('location');
|
|
if (!location) throw new JmapIndexError('JMAP session redirect had no Location header');
|
|
const next = new URL(location, currentUrl);
|
|
if (next.origin !== origin) {
|
|
throw new JmapIndexError(
|
|
`JMAP session redirected off-origin (${next.origin}); refusing to send credentials there`,
|
|
);
|
|
}
|
|
currentUrl = next.toString();
|
|
}
|
|
|
|
if (!response) throw new JmapIndexError('JMAP session fetch produced no response');
|
|
if (response.status === 401 || response.status === 403) {
|
|
throw new JmapIndexError('JMAP authentication failed', 401);
|
|
}
|
|
if (response.status >= 300 && response.status < 400) {
|
|
throw new JmapIndexError('Too many redirects fetching the JMAP session');
|
|
}
|
|
if (!response.ok) {
|
|
throw new JmapIndexError(`JMAP session fetch failed (${response.status})`);
|
|
}
|
|
const raw = (await response.json().catch(() => null)) as Record<string, unknown> | null;
|
|
if (!raw || typeof raw.apiUrl !== 'string') {
|
|
throw new JmapIndexError('Invalid JMAP session response');
|
|
}
|
|
return {
|
|
apiUrl: pinToServerOrigin(raw.apiUrl, serverUrl),
|
|
username: typeof raw.username === 'string' ? raw.username : undefined,
|
|
primaryAccounts: (raw.primaryAccounts as Record<string, string>) ?? {},
|
|
accounts: (raw.accounts as JmapSessionInfo['accounts']) ?? {},
|
|
capabilities: (raw.capabilities as Record<string, unknown>) ?? {},
|
|
};
|
|
}
|
|
|
|
type MethodCall = [string, Record<string, unknown>, string];
|
|
|
|
/** Raw method-response tuple, `[name, args, callId]`. `name` is 'error' on failure. */
|
|
type MethodResponse = [string, Record<string, unknown>, string];
|
|
|
|
export async function jmapRequest(
|
|
session: JmapSessionInfo,
|
|
authHeader: string,
|
|
using: readonly string[],
|
|
methodCalls: readonly MethodCall[],
|
|
): Promise<MethodResponse[]> {
|
|
const response = await fetchWithTimeout(session.apiUrl, {
|
|
method: 'POST',
|
|
headers: { Authorization: authHeader, 'Content-Type': 'application/json' },
|
|
body: JSON.stringify({ using, methodCalls }),
|
|
});
|
|
if (response.status === 401 || response.status === 403) {
|
|
throw new JmapIndexError('JMAP authentication failed', 401);
|
|
}
|
|
if (response.status === 429) {
|
|
throw new JmapIndexError('JMAP server is rate limiting', 429);
|
|
}
|
|
if (!response.ok) {
|
|
throw new JmapIndexError(`JMAP request failed (${response.status})`);
|
|
}
|
|
const data = (await response.json().catch(() => null)) as { methodResponses?: MethodResponse[] } | null;
|
|
if (!data || !Array.isArray(data.methodResponses)) {
|
|
throw new JmapIndexError('Invalid JMAP response envelope');
|
|
}
|
|
return data.methodResponses;
|
|
}
|
|
|
|
function firstResult(responses: MethodResponse[], expected: string): Record<string, unknown> | null {
|
|
for (const [name, args] of responses) {
|
|
if (name === expected) return args;
|
|
// A method-level error is not fatal for an INDEX: a server that doesn't
|
|
// support one data type should not fail the whole reindex. The caller
|
|
// treats null as "nothing to index for this type".
|
|
if (name === 'error') return null;
|
|
}
|
|
return null;
|
|
}
|
|
|
|
function idsOf(args: Record<string, unknown> | null): string[] {
|
|
const ids = args?.ids;
|
|
return Array.isArray(ids) ? ids.filter((v): v is string => typeof v === 'string') : [];
|
|
}
|
|
|
|
function listOf<T>(args: Record<string, unknown> | null): T[] {
|
|
const list = args?.list;
|
|
return Array.isArray(list) ? (list as T[]) : [];
|
|
}
|
|
|
|
export function accountIdFor(session: JmapSessionInfo, capability: string): string | null {
|
|
const id = session.primaryAccounts[capability];
|
|
return typeof id === 'string' && id.length > 0 ? id : null;
|
|
}
|
|
|
|
export function hasCapability(session: JmapSessionInfo, capability: string): boolean {
|
|
return Object.prototype.hasOwnProperty.call(session.capabilities, capability);
|
|
}
|
|
|
|
/** Per-ACCOUNT capability, mirroring client.ts's supportsFiles() (#563: a server can advertise it while an account has it revoked). */
|
|
export function accountHasCapability(
|
|
session: JmapSessionInfo,
|
|
accountId: string,
|
|
capability: string,
|
|
): boolean {
|
|
const account = session.accounts[accountId];
|
|
if (!account) return false;
|
|
if (account.accountCapabilities && Object.prototype.hasOwnProperty.call(account.accountCapabilities, capability)) {
|
|
return true;
|
|
}
|
|
return account.isPersonal === false;
|
|
}
|
|
|
|
// ── mail ────────────────────────────────────────────────────────────────────
|
|
|
|
/** Properties needed to build a mail IndexDoc. Bodies come via bodyValues. */
|
|
const EMAIL_INDEX_PROPERTIES = [
|
|
'id', 'threadId', 'mailboxIds', 'keywords', 'size', 'receivedAt',
|
|
'from', 'to', 'cc', 'subject', 'preview', 'hasAttachment',
|
|
'textBody', 'htmlBody', 'bodyValues',
|
|
] as const;
|
|
|
|
export async function getEmailsForIndex(
|
|
session: JmapSessionInfo,
|
|
authHeader: string,
|
|
accountId: string,
|
|
ids: readonly string[],
|
|
maxBodyBytes: number,
|
|
): Promise<Email[]> {
|
|
if (ids.length === 0) return [];
|
|
const responses = await jmapRequest(session, authHeader, [CAP_CORE, CAP_MAIL], [
|
|
['Email/get', {
|
|
accountId,
|
|
ids: [...ids],
|
|
properties: [...EMAIL_INDEX_PROPERTIES],
|
|
// Without these two the bodyValues map comes back EMPTY and every
|
|
// indexed body would silently fall back to `preview`.
|
|
fetchTextBodyValues: true,
|
|
fetchHTMLBodyValues: true,
|
|
maxBodyValueBytes: maxBodyBytes,
|
|
}, 'g'],
|
|
]);
|
|
return listOf<Email>(firstResult(responses, 'Email/get'));
|
|
}
|
|
|
|
export async function queryRecentEmailIds(
|
|
session: JmapSessionInfo,
|
|
authHeader: string,
|
|
accountId: string,
|
|
afterIso: string,
|
|
limit: number,
|
|
): Promise<string[]> {
|
|
const responses = await jmapRequest(session, authHeader, [CAP_CORE, CAP_MAIL], [
|
|
['Email/query', {
|
|
accountId,
|
|
filter: { after: afterIso },
|
|
sort: [{ property: 'receivedAt', isAscending: false }],
|
|
limit,
|
|
calculateTotal: false,
|
|
}, 'q'],
|
|
]);
|
|
return idsOf(firstResult(responses, 'Email/query'));
|
|
}
|
|
|
|
// ── calendar ────────────────────────────────────────────────────────────────
|
|
|
|
export async function getCalendarEventsForIndex(
|
|
session: JmapSessionInfo,
|
|
authHeader: string,
|
|
accountId: string,
|
|
ids: readonly string[],
|
|
): Promise<CalendarEvent[]> {
|
|
if (ids.length === 0) return [];
|
|
const responses = await jmapRequest(session, authHeader, [CAP_CORE, CAP_CALENDARS], [
|
|
['CalendarEvent/get', { accountId, ids: [...ids] }, 'g'],
|
|
]);
|
|
return listOf<CalendarEvent>(firstResult(responses, 'CalendarEvent/get'));
|
|
}
|
|
|
|
export async function queryCalendarEventIds(
|
|
session: JmapSessionInfo,
|
|
authHeader: string,
|
|
accountId: string,
|
|
afterIso: string,
|
|
beforeIso: string,
|
|
limit: number,
|
|
): Promise<string[]> {
|
|
const responses = await jmapRequest(session, authHeader, [CAP_CORE, CAP_CALENDARS], [
|
|
['CalendarEvent/query', {
|
|
accountId,
|
|
// LocalDateTime, per the note in lib/jmap/client.ts:307-312 - Stalwart
|
|
// parses these without a zone suffix and ignores unparseable values.
|
|
filter: { after: toLocalDateTime(afterIso), before: toLocalDateTime(beforeIso) },
|
|
limit,
|
|
calculateTotal: false,
|
|
}, 'q'],
|
|
]);
|
|
return idsOf(firstResult(responses, 'CalendarEvent/query'));
|
|
}
|
|
|
|
/** JSCalendar LocalDateTime: `YYYY-MM-DDTHH:MM:SS`, no zone designator. */
|
|
function toLocalDateTime(iso: string): string {
|
|
return iso.replace(/\.\d+/, '').replace(/Z$/, '').slice(0, 19);
|
|
}
|
|
|
|
// ── contacts ────────────────────────────────────────────────────────────────
|
|
|
|
export async function getContactsForIndex(
|
|
session: JmapSessionInfo,
|
|
authHeader: string,
|
|
accountId: string,
|
|
ids: readonly string[],
|
|
): Promise<ContactCard[]> {
|
|
if (ids.length === 0) return [];
|
|
const responses = await jmapRequest(session, authHeader, [CAP_CORE, CAP_CONTACTS], [
|
|
['ContactCard/get', { accountId, ids: [...ids] }, 'g'],
|
|
]);
|
|
return listOf<ContactCard>(firstResult(responses, 'ContactCard/get'));
|
|
}
|
|
|
|
export async function queryContactIds(
|
|
session: JmapSessionInfo,
|
|
authHeader: string,
|
|
accountId: string,
|
|
limit: number,
|
|
): Promise<string[]> {
|
|
const responses = await jmapRequest(session, authHeader, [CAP_CORE, CAP_CONTACTS], [
|
|
['ContactCard/query', { accountId, limit, calculateTotal: false }, 'q'],
|
|
]);
|
|
return idsOf(firstResult(responses, 'ContactCard/query'));
|
|
}
|
|
|
|
// ── files ───────────────────────────────────────────────────────────────────
|
|
|
|
const FILENODE_INDEX_PROPERTIES = [
|
|
'id', 'parentId', 'name', 'type', 'blobId', 'size', 'created', 'modified',
|
|
] as const;
|
|
|
|
export async function getFilesForIndex(
|
|
session: JmapSessionInfo,
|
|
authHeader: string,
|
|
accountId: string,
|
|
ids: readonly string[],
|
|
): Promise<FileNode[]> {
|
|
if (ids.length === 0) return [];
|
|
const responses = await jmapRequest(session, authHeader, [CAP_CORE], [
|
|
['FileNode/get', { accountId, ids: [...ids], properties: [...FILENODE_INDEX_PROPERTIES] }, 'g'],
|
|
]);
|
|
return listOf<FileNode>(firstResult(responses, 'FileNode/get'));
|
|
}
|
|
|
|
export async function queryFileIds(
|
|
session: JmapSessionInfo,
|
|
authHeader: string,
|
|
accountId: string,
|
|
limit: number,
|
|
): Promise<string[]> {
|
|
const responses = await jmapRequest(session, authHeader, [CAP_CORE], [
|
|
['FileNode/query', { accountId, filter: {}, limit, calculateTotal: false }, 'q'],
|
|
]);
|
|
return idsOf(firstResult(responses, 'FileNode/query'));
|
|
}
|
|
|
|
/**
|
|
* Builds `id -> "Parent/Child"` paths for the given nodes, walking `parentId`
|
|
* upward. FileNode only knows its parent, so the caller has to assemble this;
|
|
* unresolvable ancestors just truncate the path rather than failing.
|
|
*/
|
|
export function buildFilePaths(nodes: readonly FileNode[]): Map<string, string> {
|
|
const byId = new Map(nodes.map((n) => [n.id, n]));
|
|
const cache = new Map<string, string>();
|
|
|
|
const resolve = (id: string, depth: number): string => {
|
|
if (depth > 32) return '';
|
|
const cached = cache.get(id);
|
|
if (cached !== undefined) return cached;
|
|
const node = byId.get(id);
|
|
if (!node) return '';
|
|
const parent = node.parentId ? resolve(node.parentId, depth + 1) : '';
|
|
const full = parent ? `${parent}/${node.name}` : node.name;
|
|
cache.set(id, full);
|
|
return full;
|
|
};
|
|
|
|
const out = new Map<string, string>();
|
|
for (const n of nodes) {
|
|
// The document's own `path` metadata is its PARENT directory chain, so a
|
|
// search for "Invoices" matches files inside it without the filename
|
|
// being duplicated into the body.
|
|
out.set(n.id, n.parentId ? resolve(n.parentId, 0) : '');
|
|
}
|
|
return out;
|
|
}
|