feat(mail-index): encrypted SQLite/FTS5 index over mail, calendar, contacts, files

An on-device, SQLCipher-encrypted full-text index the app can retrieve from to
feed an LLM ("prompt against"), for the Electron desktop shell only.

Shape: no persistent background worker and no resident credential. Indexing is
a normal request-scoped API route, triggered by the renderer's EXISTING live
JMAP push connection - so it reacts to each delivery/change rather than polling.

- lib/mail-index/binding.ts   guarded require of the optional native binding
- lib/mail-index/paths.ts     the VNCMAIL_DESKTOP_STORE_DIR gate + hashed paths
- lib/mail-index/store.ts     schema, upsert, FTS5 search, encryption assertion
- lib/mail-index/extract.ts   PURE JMAP-object -> document extractors
- lib/mail-index/jmap.ts      minimal stateless server-side JMAP client
- lib/mail-index/key.ts       per-job key fetch over the inherited fd
- lib/mail-index/reindex.ts   the job + slot->account resolution
- electron/key-service.ts     safeStorage wrap/unwrap, served over fd 3
- app/api/offline/reindex     POST, event-driven + catch-up
- app/api/offline/search      GET, the retrieval surface (hits + contextBlock)
- lib/mail-index-client.ts    renderer client; StateChange -> index call
- components/settings/local-index-settings.tsx  status + manual catch-up

Decisions worth knowing:

* `@signalapp/sqlcipher` is an OPTIONAL dependency with a guarded runtime
  require. It publishes six N-API prebuilds and NO build sources, and both
  Dockerfiles are node:24-alpine (musl, no matching prebuild) - as a hard
  dependency it would break the production image and the integration fixture's
  webmail container, neither of which wants this feature.

* Credentials come from the existing per-slot encrypted `jmap_stalwart_ctx`
  cookie via lib/stalwart/credentials.ts - the same helper /api/settings and
  /api/push/preview already use. It carries a ready-made header for basic AND
  bearer accounts, so the indexer never touches the OAuth refresh-token cookie;
  a server-side refresh would rotate a token into a response nobody reads and
  silently log the user out.

* The encryption key crosses main -> server over an INHERITED FILE DESCRIPTOR,
  never an environment variable: env is readable by any process running as the
  same OS user, which would defeat using the OS keychain at all. Fetched per
  job and zeroed after, so there is no long-lived key copy.

* safeStorage's Linux `basic_text` backend (no keyring) is treated as refusal,
  not degradation - it "encrypts" with a hardcoded public password, which would
  look like an encrypted mailbox while providing nothing.
  getSelectedStorageBackend() is Linux-only and platform-guarded.

* Every store open asserts `PRAGMA cipher_version` returns a non-empty STRING,
  not merely a row: a non-cipher binding returns ZERO ROWS, so a row-count check
  would pass vacuously while writing the mailbox to disk in cleartext.

* Files are indexed by name/path/date/size only - NOT by extracted content.
  Text extraction from arbitrary PDFs/office documents is a separate problem.

* Account-scoped composite keys `(jmap_account_id, content_type, id)` are kept
  even though there is one file per account: one login exposes delegated/shared
  JMAP accounts too, and JMAP ids are unique only within an account.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

app/(main)/[locale]/page.tsx

@@ -1063,7 +1063,30 @@ export default function Home() {
       debug.log('push', `[Push] Push notifications enabled for ${cleanups.length} account(s)`);
     }
 
+    // CATCH-UP for the desktop shell's local search index. The index's normal
+    // trigger is a push StateChange (stores/email-store.ts's handleStateChange),
+    // but nothing was pushed while the app was closed - and the polling
+    // transport has no signal for contacts or files at all (client.ts's
+    // buildStatePollingRequest covers Mailbox/Email/Calendar/CalendarEvent/
+    // SieveScript only). So backfill a bounded recent window once per session,
+    // after push is wired. Fire-and-forget; a no-op outside Electron.
+    const catchUpTimer = setTimeout(() => {
+      void (async () => {
+        try {
+          const { catchUpIndex } = await import('@/lib/mail-index-client');
+          await catchUpIndex(
+            useAccountStore.getState().getActiveAccount()?.cookieSlot,
+          );
+        } catch {
+          /* the index is optional */
+        }
+      })();
+      // Deliberately after the initial mailbox fetch settles: the catch-up is a
+      // background nicety and must not compete with first paint.
+    }, 4000);
+
     return () => {
+      clearTimeout(catchUpTimer);
       cleanups.forEach((fn) => fn());
     };
   }, [isAuthenticated, client, activeAccountId, connectedAccountsSignature, handleStateChange, setPushConnected, buildPopulatedUnifiedAccounts, refreshCrossCounts, refreshUnifiedCounts]);

app/api/offline/reindex/route.ts

@@ -0,0 +1,110 @@
+// POST /api/offline/reindex - write mail/calendar/contacts/files into the
+// encrypted local search index for the calling session's account.
+//
+// The PRIMARY caller is the renderer's live JMAP push handler: when a
+// StateChange arrives it posts the ids that changed, so indexing is reactive to
+// each delivery rather than periodic. `{ catchUp: true }` (no ids) is the
+// fallback used at app launch to backfill whatever changed while the app was
+// closed.
+//
+// GATED: returns 404 unless VNCMAIL_DESKTOP_STORE_DIR is set, which only
+// electron/main.ts does. The same standalone server artifact runs in the
+// multi-tenant production Docker image, where this feature must not exist at
+// all - 404 rather than 403 so nothing learns the route is there.
+import { NextRequest, NextResponse } from 'next/server';
+import { logger } from '@/lib/logger';
+import { isSqlcipherAvailable } from '@/lib/mail-index/binding';
+import { hasKeyChannel, IndexKeyError } from '@/lib/mail-index/key';
+import { getStoreDir } from '@/lib/mail-index/paths';
+import {
+  IndexSessionError, MAX_IDS_PER_CALL, resolveIndexSession, runIndex,
+  type IndexRequest,
+} from '@/lib/mail-index/reindex';
+import { CONTENT_TYPES, isContentType, type ContentType } from '@/lib/mail-index/store';
+import { JmapIndexError } from '@/lib/mail-index/jmap';
+
+export const runtime = 'nodejs';
+export const dynamic = 'force-dynamic';
+
+function parseIdMap(raw: unknown): Partial<Record<ContentType, string[]>> | undefined {
+  if (!raw || typeof raw !== 'object' || Array.isArray(raw)) return undefined;
+  const out: Partial<Record<ContentType, string[]>> = {};
+  for (const [key, value] of Object.entries(raw as Record<string, unknown>)) {
+    if (!isContentType(key) || !Array.isArray(value)) continue;
+    const ids = value
+      .filter((v): v is string => typeof v === 'string' && v.length > 0 && v.length <= 256)
+      .slice(0, MAX_IDS_PER_CALL);
+    if (ids.length > 0) out[key] = ids;
+  }
+  return Object.keys(out).length > 0 ? out : undefined;
+}
+
+export async function POST(request: NextRequest) {
+  if (!getStoreDir()) {
+    return new NextResponse(null, { status: 404 });
+  }
+  if (!hasKeyChannel()) {
+    return NextResponse.json(
+      { error: 'The local index has no key channel in this process.', code: 'no-key-channel' },
+      { status: 503 },
+    );
+  }
+  if (!isSqlcipherAvailable()) {
+    // The native binding is an optionalDependency, so "not installed" is a
+    // normal state on platforms without a prebuild - not an error to log loudly.
+    return NextResponse.json(
+      { error: 'Encrypted local index is unavailable on this platform.', code: 'no-binding' },
+      { status: 503 },
+    );
+  }
+
+  let body: Record<string, unknown> = {};
+  try {
+    const text = await request.text();
+    if (text.trim()) body = JSON.parse(text) as Record<string, unknown>;
+  } catch {
+    return NextResponse.json({ error: 'Malformed JSON body' }, { status: 400 });
+  }
+
+  const rawTypes = Array.isArray(body.types) ? body.types.filter(isContentType) : [];
+  const req: IndexRequest = {
+    types: rawTypes.length > 0 ? rawTypes : undefined,
+    ids: parseIdMap(body.ids),
+    removed: parseIdMap(body.removed),
+    // Pruning is a catch-up concern; a single-delivery call shouldn't scan.
+    prune: body.catchUp === true,
+  };
+
+  try {
+    const session = await resolveIndexSession(request);
+    const result = await runIndex(session, req);
+    return NextResponse.json(
+      {
+        ok: true,
+        written: result.written,
+        skipped: result.skipped,
+        errors: result.errors,
+        durationMs: result.durationMs,
+        types: CONTENT_TYPES,
+      },
+      { headers: { 'Cache-Control': 'no-store' } },
+    );
+  } catch (error) {
+    if (error instanceof IndexSessionError) {
+      return NextResponse.json({ error: error.message }, { status: error.status });
+    }
+    if (error instanceof JmapIndexError) {
+      return NextResponse.json({ error: error.message }, { status: error.status });
+    }
+    if (error instanceof IndexKeyError) {
+      // no-secure-storage is the Linux-without-a-keyring refusal: a real,
+      // expected outcome with a user-facing explanation, not a server fault.
+      const status = error.code === 'no-secure-storage' ? 503 : 500;
+      return NextResponse.json({ error: error.message, code: error.code }, { status });
+    }
+    logger.error('mail-index reindex failed', {
+      error: error instanceof Error ? error.message : String(error),
+    });
+    return NextResponse.json({ error: 'Reindex failed' }, { status: 500 });
+  }
+}

app/api/offline/search/route.ts

@@ -0,0 +1,117 @@
+// GET /api/offline/search?q=...&types=mail,calendar&limit=20
+//
+// THE RETRIEVAL SURFACE. This is what an AI/RAG feature calls to gather
+// relevant context from the user's own mail, calendar, contacts and files
+// before prompting a model - hence the `snippet` on every hit and the
+// `contextBlock` convenience field, which is the same information already
+// flattened into text a prompt can carry directly.
+//
+// Read-only: it never touches the network and never writes. Gated identically
+// to the reindex route.
+import { NextRequest, NextResponse } from 'next/server';
+import { logger } from '@/lib/logger';
+import { isSqlcipherAvailable } from '@/lib/mail-index/binding';
+import { hasKeyChannel, IndexKeyError, withIndexKey } from '@/lib/mail-index/key';
+import { getStoreDir } from '@/lib/mail-index/paths';
+import { IndexSessionError, resolveIndexSession } from '@/lib/mail-index/reindex';
+import {
+  isContentType, MailIndex, MailIndexUnavailableError, type ContentType, type SearchHit,
+} from '@/lib/mail-index/store';
+
+export const runtime = 'nodejs';
+export const dynamic = 'force-dynamic';
+
+/**
+ * One hit as a plain text block, ready to be concatenated into a prompt.
+ * Kept server-side so every caller (a chat feature, a future agent, a test)
+ * formats context the same way rather than each inventing its own.
+ */
+function toContextBlock(hit: SearchHit): string {
+  const label: Record<ContentType, string> = {
+    mail: 'EMAIL', calendar: 'CALENDAR EVENT', contact: 'CONTACT', file: 'FILE',
+  };
+  const lines = [`[${label[hit.contentType]}] ${hit.title}`];
+  if (hit.occurredAt) lines.push(`Date: ${hit.occurredAt}`);
+  if (hit.people) lines.push(`People: ${hit.people}`);
+  const path = hit.metadata?.path;
+  if (typeof path === 'string' && path) lines.push(`Path: ${path}`);
+  if (hit.snippet) lines.push(`Excerpt: ${hit.snippet}`);
+  return lines.join('\n');
+}
+
+export async function GET(request: NextRequest) {
+  if (!getStoreDir()) {
+    return new NextResponse(null, { status: 404 });
+  }
+  if (!hasKeyChannel() || !isSqlcipherAvailable()) {
+    return NextResponse.json(
+      { error: 'Encrypted local index is unavailable in this process.', code: 'unavailable' },
+      { status: 503 },
+    );
+  }
+
+  const params = request.nextUrl.searchParams;
+  const query = (params.get('q') ?? '').trim();
+  const wantStats = params.get('stats') === 'true';
+
+  if (!query && !wantStats) {
+    return NextResponse.json({ error: 'Missing q parameter' }, { status: 400 });
+  }
+  if (query.length > 512) {
+    return NextResponse.json({ error: 'Query too long' }, { status: 400 });
+  }
+
+  const types = (params.get('types') ?? '')
+    .split(',')
+    .map((t) => t.trim())
+    .filter(isContentType);
+
+  const limitRaw = Number(params.get('limit') ?? '20');
+  const limit = Number.isFinite(limitRaw) ? Math.min(Math.max(Math.trunc(limitRaw), 1), 100) : 20;
+
+  try {
+    const session = await resolveIndexSession(request);
+    const storeDir = getStoreDir();
+    if (!storeDir) return new NextResponse(null, { status: 404 });
+
+    const payload = await withIndexKey(session.accountId, (key) => {
+      const index = MailIndex.open({ storeDir, accountId: session.accountId, key });
+      try {
+        const stats = index.stats();
+        if (!query) return { hits: [] as SearchHit[], stats };
+        return { hits: index.search({ query, types, limit }), stats: wantStats ? stats : undefined };
+      } finally {
+        index.close();
+      }
+    });
+
+    return NextResponse.json(
+      {
+        ok: true,
+        query,
+        types: types.length > 0 ? types : 'all',
+        count: payload.hits.length,
+        hits: payload.hits,
+        // Everything a prompt needs, pre-joined in rank order.
+        contextBlock: payload.hits.map(toContextBlock).join('\n\n---\n\n'),
+        ...(payload.stats ? { stats: payload.stats } : {}),
+      },
+      { headers: { 'Cache-Control': 'no-store' } },
+    );
+  } catch (error) {
+    if (error instanceof IndexSessionError) {
+      return NextResponse.json({ error: error.message }, { status: error.status });
+    }
+    if (error instanceof IndexKeyError) {
+      const status = error.code === 'no-secure-storage' ? 503 : 500;
+      return NextResponse.json({ error: error.message, code: error.code }, { status });
+    }
+    if (error instanceof MailIndexUnavailableError) {
+      return NextResponse.json({ error: error.message, code: 'unavailable' }, { status: 503 });
+    }
+    logger.error('mail-index search failed', {
+      error: error instanceof Error ? error.message : String(error),
+    });
+    return NextResponse.json({ error: 'Search failed' }, { status: 500 });
+  }
+}