Files
SRCmail/lib/mail-index-client.ts
T
Bernd RodlerandClaude Sonnet 5 b966d285a9 feat(mail-index): encrypted SQLite/FTS5 index over mail, calendar, contacts, files
An on-device, SQLCipher-encrypted full-text index the app can retrieve from to
feed an LLM ("prompt against"), for the Electron desktop shell only.

Shape: no persistent background worker and no resident credential. Indexing is
a normal request-scoped API route, triggered by the renderer's EXISTING live
JMAP push connection - so it reacts to each delivery/change rather than polling.

- lib/mail-index/binding.ts   guarded require of the optional native binding
- lib/mail-index/paths.ts     the VNCMAIL_DESKTOP_STORE_DIR gate + hashed paths
- lib/mail-index/store.ts     schema, upsert, FTS5 search, encryption assertion
- lib/mail-index/extract.ts   PURE JMAP-object -> document extractors
- lib/mail-index/jmap.ts      minimal stateless server-side JMAP client
- lib/mail-index/key.ts       per-job key fetch over the inherited fd
- lib/mail-index/reindex.ts   the job + slot->account resolution
- electron/key-service.ts     safeStorage wrap/unwrap, served over fd 3
- app/api/offline/reindex     POST, event-driven + catch-up
- app/api/offline/search      GET, the retrieval surface (hits + contextBlock)
- lib/mail-index-client.ts    renderer client; StateChange -> index call
- components/settings/local-index-settings.tsx  status + manual catch-up

Decisions worth knowing:

* `@signalapp/sqlcipher` is an OPTIONAL dependency with a guarded runtime
  require. It publishes six N-API prebuilds and NO build sources, and both
  Dockerfiles are node:24-alpine (musl, no matching prebuild) - as a hard
  dependency it would break the production image and the integration fixture's
  webmail container, neither of which wants this feature.

* Credentials come from the existing per-slot encrypted `jmap_stalwart_ctx`
  cookie via lib/stalwart/credentials.ts - the same helper /api/settings and
  /api/push/preview already use. It carries a ready-made header for basic AND
  bearer accounts, so the indexer never touches the OAuth refresh-token cookie;
  a server-side refresh would rotate a token into a response nobody reads and
  silently log the user out.

* The encryption key crosses main -> server over an INHERITED FILE DESCRIPTOR,
  never an environment variable: env is readable by any process running as the
  same OS user, which would defeat using the OS keychain at all. Fetched per
  job and zeroed after, so there is no long-lived key copy.

* safeStorage's Linux `basic_text` backend (no keyring) is treated as refusal,
  not degradation - it "encrypts" with a hardcoded public password, which would
  look like an encrypted mailbox while providing nothing.
  getSelectedStorageBackend() is Linux-only and platform-guarded.

* Every store open asserts `PRAGMA cipher_version` returns a non-empty STRING,
  not merely a row: a non-cipher binding returns ZERO ROWS, so a row-count check
  would pass vacuously while writing the mailbox to disk in cleartext.

* Files are indexed by name/path/date/size only - NOT by extracted content.
  Text extraction from arbitrary PDFs/office documents is a separate problem.

* Account-scoped composite keys `(jmap_account_id, content_type, id)` are kept
  even though there is one file per account: one login exposes delegated/shared
  JMAP accounts too, and JMAP ids are unique only within an account.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-04 23:16:22 +02:00

200 lines
7.4 KiB
TypeScript

// Renderer-side client for the encrypted local search index.
//
// The index is EVENT-DRIVEN: the renderer already holds the live JMAP push
// connection (WebSocket -> SSE -> polling, `lib/jmap/client.ts`'s
// setupPushNotifications), so the moment a StateChange announces new mail, a
// calendar change, a contact edit or a file upload, this posts to the reindex
// route. No polling loop, no background worker, no long-lived credential -
// just one more authenticated fetch from the place the push already arrives.
//
// Every function here is best-effort and never throws: a search index failing
// to update must never break the mail UI.
import { apiFetch } from '@/lib/browser-navigation';
import { debug } from '@/lib/debug';
import type { StateChange } from '@/lib/jmap/types';
export type IndexContentType = 'mail' | 'calendar' | 'contact' | 'file';
export interface IndexRunResult {
ok: boolean;
written?: Partial<Record<IndexContentType, number>>;
skipped?: IndexContentType[];
errors?: Array<{ contentType: IndexContentType; message: string }>;
durationMs?: number;
/** Set when the feature isn't available (not the desktop shell, no keyring, no binding). */
unavailable?: boolean;
error?: string;
}
/**
* Maps JMAP `StateChange` type keys onto our content types.
*
* The transport is already type-generic - the WebSocket handler
* (`client.ts:6308-6316`) and the SSE handler (`:6505`) pass the whole
* `changed` map through untouched, and the WS subscribes with
* `dataTypes: null` (every type) - so anything the server pushes arrives here.
*
* `Mailbox` is deliberately NOT mapped: a Mailbox state change is usually just
* an unread-count move, and it fires constantly. `Email` covers the cases that
* change indexable content.
*/
const STATE_TYPE_TO_CONTENT: Record<string, IndexContentType> = {
Email: 'mail',
Calendar: 'calendar',
CalendarEvent: 'calendar',
ContactCard: 'contact',
AddressBook: 'contact',
FileNode: 'file',
};
export function contentTypesFromStateChange(change: StateChange): IndexContentType[] {
const out = new Set<IndexContentType>();
for (const perAccount of Object.values(change.changed ?? {})) {
for (const stateType of Object.keys(perAccount ?? {})) {
const mapped = STATE_TYPE_TO_CONTENT[stateType];
if (mapped) out.add(mapped);
}
}
return [...out];
}
export interface IndexRequestOptions {
types?: readonly IndexContentType[];
/**
* Per-type ids to index. Supply them whenever the renderer already knows
* which objects changed - it turns the call into a couple of `Foo/get`s
* instead of a windowed query. Mail is the frequent case and the one where
* this matters.
*/
ids?: Partial<Record<IndexContentType, string[]>>;
/** Backfill the recent window for every supported type, and prune. */
catchUp?: boolean;
/** Cookie slot of the account to index. Defaults to the server's first signed-in slot. */
slot?: number;
}
let inFlight: Promise<IndexRunResult> | null = null;
/** Set once the server says the feature isn't there, so we stop asking. */
let knownUnavailable = false;
/**
* Posts one index request. Single-flighted: a burst of deliveries coalesces
* into the in-flight call rather than queueing N overlapping SQLite writers.
*/
export async function requestIndex(options: IndexRequestOptions = {}): Promise<IndexRunResult> {
if (knownUnavailable) return { ok: false, unavailable: true };
if (inFlight) return inFlight;
const query = typeof options.slot === 'number' ? `?slot=${options.slot}` : '';
const run = (async (): Promise<IndexRunResult> => {
try {
const response = await apiFetch(`/api/offline/reindex${query}`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
types: options.types,
ids: options.ids,
catchUp: options.catchUp === true,
}),
});
// 404 = not the desktop shell (or the feature is gated off). Permanent for
// this page load; stop asking so a busy mailbox doesn't post per delivery.
if (response.status === 404) {
knownUnavailable = true;
return { ok: false, unavailable: true };
}
if (response.status === 503) {
// No keyring / no native binding / no key channel. Also permanent for
// this session, and the message is worth surfacing in Settings.
knownUnavailable = true;
const body = await response.json().catch(() => ({}));
return { ok: false, unavailable: true, error: body?.error };
}
if (!response.ok) {
const body = await response.json().catch(() => ({}));
return { ok: false, error: body?.error || `HTTP ${response.status}` };
}
const body = await response.json();
debug.log('push', '[index] reindex done', body?.written, body?.errors);
return {
ok: true,
written: body?.written,
skipped: body?.skipped,
errors: body?.errors,
durationMs: body?.durationMs,
};
} catch (error) {
return { ok: false, error: error instanceof Error ? error.message : String(error) };
} finally {
inFlight = null;
}
})();
inFlight = run;
return run;
}
/**
* The event-driven entry point, called from the push handler.
*
* `mailIds` lets the caller hand over the ids it already has (the refreshed
* mailbox page), so the frequent mail case costs one `Email/get` rather than a
* 30-day query. The other three types are rare events (a contact edit, a file
* upload, a calendar change), so they fall back to their own bounded queries.
*/
export function indexOnStateChange(
change: StateChange,
opts: { mailIds?: string[]; slot?: number } = {},
): void {
if (knownUnavailable) return;
const types = contentTypesFromStateChange(change);
if (types.length === 0) return;
const ids: Partial<Record<IndexContentType, string[]>> = {};
if (types.includes('mail') && opts.mailIds && opts.mailIds.length > 0) {
ids.mail = opts.mailIds.slice(0, 100);
}
// Fire-and-forget on purpose: this runs inside the push handler, and the mail
// UI must not wait on a search index.
void requestIndex({ types, ids: Object.keys(ids).length > 0 ? ids : undefined, slot: opts.slot });
}
/**
* Launch-time catch-up: backfills whatever changed while the app was closed,
* for which no push event was ever delivered. Also the recovery path for the
* polling transport, which has no signal for contacts or files at all
* (`client.ts`'s buildStatePollingRequest polls Mailbox/Email/Calendar/
* CalendarEvent/SieveScript only).
*/
export async function catchUpIndex(slot?: number): Promise<IndexRunResult> {
return requestIndex({ catchUp: true, slot });
}
export interface IndexStats {
contentType: string;
count: number;
newest: string | null;
indexedAt: number | null;
}
/** Reads per-type counts without searching. Used by the Settings panel. */
export async function fetchIndexStats(slot?: number): Promise<IndexStats[] | null> {
const slotQuery = typeof slot === 'number' ? `&slot=${slot}` : '';
try {
const response = await apiFetch(`/api/offline/search?stats=true${slotQuery}`);
if (!response.ok) return null;
const body = await response.json();
return Array.isArray(body?.stats) ? (body.stats as IndexStats[]) : [];
} catch {
return null;
}
}
/** Resets the "don't ask again" latch - e.g. after the user signs in again. */
export function resetIndexAvailability(): void {
knownUnavailable = false;
}