feat(mail-index): encrypted SQLite/FTS5 index over mail, calendar, contacts, files
An on-device, SQLCipher-encrypted full-text index the app can retrieve from to
feed an LLM ("prompt against"), for the Electron desktop shell only.
Shape: no persistent background worker and no resident credential. Indexing is
a normal request-scoped API route, triggered by the renderer's EXISTING live
JMAP push connection - so it reacts to each delivery/change rather than polling.
- lib/mail-index/binding.ts guarded require of the optional native binding
- lib/mail-index/paths.ts the VNCMAIL_DESKTOP_STORE_DIR gate + hashed paths
- lib/mail-index/store.ts schema, upsert, FTS5 search, encryption assertion
- lib/mail-index/extract.ts PURE JMAP-object -> document extractors
- lib/mail-index/jmap.ts minimal stateless server-side JMAP client
- lib/mail-index/key.ts per-job key fetch over the inherited fd
- lib/mail-index/reindex.ts the job + slot->account resolution
- electron/key-service.ts safeStorage wrap/unwrap, served over fd 3
- app/api/offline/reindex POST, event-driven + catch-up
- app/api/offline/search GET, the retrieval surface (hits + contextBlock)
- lib/mail-index-client.ts renderer client; StateChange -> index call
- components/settings/local-index-settings.tsx status + manual catch-up
Decisions worth knowing:
* `@signalapp/sqlcipher` is an OPTIONAL dependency with a guarded runtime
require. It publishes six N-API prebuilds and NO build sources, and both
Dockerfiles are node:24-alpine (musl, no matching prebuild) - as a hard
dependency it would break the production image and the integration fixture's
webmail container, neither of which wants this feature.
* Credentials come from the existing per-slot encrypted `jmap_stalwart_ctx`
cookie via lib/stalwart/credentials.ts - the same helper /api/settings and
/api/push/preview already use. It carries a ready-made header for basic AND
bearer accounts, so the indexer never touches the OAuth refresh-token cookie;
a server-side refresh would rotate a token into a response nobody reads and
silently log the user out.
* The encryption key crosses main -> server over an INHERITED FILE DESCRIPTOR,
never an environment variable: env is readable by any process running as the
same OS user, which would defeat using the OS keychain at all. Fetched per
job and zeroed after, so there is no long-lived key copy.
* safeStorage's Linux `basic_text` backend (no keyring) is treated as refusal,
not degradation - it "encrypts" with a hardcoded public password, which would
look like an encrypted mailbox while providing nothing.
getSelectedStorageBackend() is Linux-only and platform-guarded.
* Every store open asserts `PRAGMA cipher_version` returns a non-empty STRING,
not merely a row: a non-cipher binding returns ZERO ROWS, so a row-count check
would pass vacuously while writing the mailbox to disk in cleartext.
* Files are indexed by name/path/date/size only - NOT by extracted content.
Text extraction from arbitrary PDFs/office documents is a separate problem.
* Account-scoped composite keys `(jmap_account_id, content_type, id)` are kept
even though there is one file per account: one login exposes delegated/shared
JMAP accounts too, and JMAP ids are unique only within an account.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Bernd Rodler
co-authored by
Claude Sonnet 5
Claude Sonnet 5
parent
16466c7296
commit
b966d285a9
@@ -0,0 +1,332 @@
|
||||
// The index jobs.
|
||||
//
|
||||
// TWO SHAPES, both plain request-scoped work - there is no background worker,
|
||||
// no cursor, no retry ladder and no resident credential anywhere:
|
||||
//
|
||||
// 1. `indexDocuments()` - the PRIMARY path. The renderer's live JMAP push
|
||||
// connection sees a StateChange, and calls the route with the ids that
|
||||
// changed (or with no ids, meaning "refetch what's recent for this type").
|
||||
// One or a handful of objects, fetched and upserted.
|
||||
// 2. `catchUpAll()` - the FALLBACK. On app launch, backfill a bounded recent
|
||||
// window for every supported type, because anything that changed while the
|
||||
// app was closed produced no push event.
|
||||
//
|
||||
// Staleness between refreshes is acceptable by design: this is a search index
|
||||
// for a retrieval/AI feature, not a mail replica.
|
||||
|
||||
import type { NextRequest } from 'next/server';
|
||||
import { generateAccountId } from '@/lib/account-utils';
|
||||
import { getStalwartCredentials } from '@/lib/stalwart/credentials';
|
||||
import { logger } from '@/lib/logger';
|
||||
import {
|
||||
accountHasCapability, accountIdFor, buildFilePaths, CAP_CALENDARS, CAP_CONTACTS,
|
||||
CAP_FILENODE, CAP_MAIL, fetchJmapSession, getCalendarEventsForIndex, getContactsForIndex,
|
||||
getEmailsForIndex, getFilesForIndex, hasCapability, JmapIndexError, queryCalendarEventIds,
|
||||
queryContactIds, queryFileIds, queryRecentEmailIds, type JmapSessionInfo,
|
||||
} from './jmap';
|
||||
import { extractCalendarEvent, extractContact, extractFile, extractMail } from './extract';
|
||||
import { withIndexKey } from './key';
|
||||
import { getStoreDir } from './paths';
|
||||
import { MailIndex, type ContentType, type IndexDoc } from './store';
|
||||
|
||||
/**
|
||||
* Bounded window. Small on purpose: this is the first cut of a retrieval index,
|
||||
* and a wide window turns "index on every delivery" into a slow request. The
|
||||
* event-driven path indexes single objects, so the window only bounds catch-up.
|
||||
*/
|
||||
export const INDEX_WINDOW_DAYS = 30;
|
||||
/** Calendar looks forward as well as back - upcoming events are the useful ones. */
|
||||
export const CALENDAR_FORWARD_DAYS = 180;
|
||||
/** Per-type ceiling for one catch-up pass. */
|
||||
export const CATCHUP_MAX_PER_TYPE = 500;
|
||||
/** Ids accepted in one event-driven call. A push reports a handful, not thousands. */
|
||||
export const MAX_IDS_PER_CALL = 200;
|
||||
/** Cap on body bytes requested per message from the server. */
|
||||
export const MAX_BODY_VALUE_BYTES = 256_000;
|
||||
/** Contacts and files have no useful date filter, so they are simply capped. */
|
||||
export const CONTACTS_MAX = 2_000;
|
||||
export const FILES_MAX = 2_000;
|
||||
|
||||
export interface IndexSession {
|
||||
serverUrl: string;
|
||||
authHeader: string;
|
||||
username: string;
|
||||
slot: number;
|
||||
/** `username@host` - the durable per-account key. NEVER the cookie slot. */
|
||||
accountId: string;
|
||||
}
|
||||
|
||||
export class IndexSessionError extends Error {
|
||||
status: number;
|
||||
constructor(message: string, status: number) {
|
||||
super(message);
|
||||
this.name = 'IndexSessionError';
|
||||
this.status = status;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves the calling request to an account and a usable Authorization header.
|
||||
*
|
||||
* Uses the SAME per-slot encrypted `jmap_stalwart_ctx` cookie that
|
||||
* `/api/settings`, `/api/push/preview` and `/api/plugin-approval-status`
|
||||
* already read (`lib/stalwart/credentials.ts`). That cookie is written by
|
||||
* `/api/auth/stalwart-context`, which the renderer syncs on every login,
|
||||
* session restore, SSO callback, account switch and token refresh
|
||||
* (`stores/auth-store.ts`, 10 call sites), and it carries a ready-made header
|
||||
* for BOTH basic and bearer accounts.
|
||||
*
|
||||
* Why this matters beyond convenience: it means the indexer never touches the
|
||||
* OAuth refresh-token cookie. A server-side refresh would rotate the token into
|
||||
* a response nobody reads while the browser kept the superseded one, and the
|
||||
* next real refresh would then fail and log the user out. Reading an
|
||||
* already-minted header cannot cause that.
|
||||
*/
|
||||
export async function resolveIndexSession(request: NextRequest): Promise<IndexSession> {
|
||||
const credentials = await getStalwartCredentials(request);
|
||||
if (!credentials) {
|
||||
throw new IndexSessionError('No JMAP auth context for this account; sign in again.', 401);
|
||||
}
|
||||
const accountId = generateAccountId(credentials.username, credentials.serverUrl);
|
||||
return { ...credentials, accountId };
|
||||
}
|
||||
|
||||
export interface IndexResult {
|
||||
accountId: string;
|
||||
/** Per-type counts of documents written. */
|
||||
written: Partial<Record<ContentType, number>>;
|
||||
/** Types the server (or this account) doesn't support, so nothing was attempted. */
|
||||
skipped: ContentType[];
|
||||
/** Non-fatal per-type failures. One broken type must not fail the whole call. */
|
||||
errors: Array<{ contentType: ContentType; message: string }>;
|
||||
durationMs: number;
|
||||
}
|
||||
|
||||
function isoDaysFromNow(days: number): string {
|
||||
return new Date(Date.now() + days * 24 * 60 * 60 * 1000).toISOString();
|
||||
}
|
||||
|
||||
/**
|
||||
* Which types this session can actually index. Calendar/contacts are session
|
||||
* capabilities; files is a PER-ACCOUNT capability (a server can advertise
|
||||
* filenode while a specific account has it revoked - #563).
|
||||
*/
|
||||
export function supportedTypes(session: JmapSessionInfo): {
|
||||
supported: ContentType[];
|
||||
skipped: ContentType[];
|
||||
accountIds: Partial<Record<ContentType, string>>;
|
||||
} {
|
||||
const supported: ContentType[] = [];
|
||||
const skipped: ContentType[] = [];
|
||||
const accountIds: Partial<Record<ContentType, string>> = {};
|
||||
|
||||
const mailAccount = accountIdFor(session, CAP_MAIL);
|
||||
if (mailAccount) { supported.push('mail'); accountIds.mail = mailAccount; }
|
||||
else skipped.push('mail');
|
||||
|
||||
const calAccount = accountIdFor(session, CAP_CALENDARS);
|
||||
if (calAccount && hasCapability(session, CAP_CALENDARS)) {
|
||||
supported.push('calendar'); accountIds.calendar = calAccount;
|
||||
} else skipped.push('calendar');
|
||||
|
||||
const contactAccount = accountIdFor(session, CAP_CONTACTS);
|
||||
if (contactAccount && hasCapability(session, CAP_CONTACTS)) {
|
||||
supported.push('contact'); accountIds.contact = contactAccount;
|
||||
} else skipped.push('contact');
|
||||
|
||||
// Files fall back to the mail account id: Stalwart exposes FileNode on the
|
||||
// same account and does not always list a primaryAccounts entry for it.
|
||||
const fileAccount = accountIdFor(session, CAP_FILENODE) ?? mailAccount;
|
||||
if (fileAccount && accountHasCapability(session, fileAccount, CAP_FILENODE)) {
|
||||
supported.push('file'); accountIds.file = fileAccount;
|
||||
} else skipped.push('file');
|
||||
|
||||
return { supported, skipped, accountIds };
|
||||
}
|
||||
|
||||
interface FetchArgs {
|
||||
session: JmapSessionInfo;
|
||||
authHeader: string;
|
||||
jmapAccountId: string;
|
||||
ids: readonly string[] | null;
|
||||
}
|
||||
|
||||
/** Fetches and flattens one content type. `ids === null` means "the recent window". */
|
||||
async function fetchDocs(contentType: ContentType, args: FetchArgs): Promise<IndexDoc[]> {
|
||||
const { session, authHeader, jmapAccountId, ids } = args;
|
||||
|
||||
switch (contentType) {
|
||||
case 'mail': {
|
||||
const targetIds = ids ?? await queryRecentEmailIds(
|
||||
session, authHeader, jmapAccountId,
|
||||
isoDaysFromNow(-INDEX_WINDOW_DAYS), CATCHUP_MAX_PER_TYPE,
|
||||
);
|
||||
const docs: IndexDoc[] = [];
|
||||
// Chunked because bodies are big: one Email/get for 500 messages with
|
||||
// full bodies would be an enormous response.
|
||||
for (let i = 0; i < targetIds.length; i += 25) {
|
||||
const emails = await getEmailsForIndex(
|
||||
session, authHeader, jmapAccountId, targetIds.slice(i, i + 25), MAX_BODY_VALUE_BYTES,
|
||||
);
|
||||
for (const email of emails) docs.push(extractMail(jmapAccountId, email));
|
||||
}
|
||||
return docs;
|
||||
}
|
||||
case 'calendar': {
|
||||
const targetIds = ids ?? await queryCalendarEventIds(
|
||||
session, authHeader, jmapAccountId,
|
||||
isoDaysFromNow(-INDEX_WINDOW_DAYS), isoDaysFromNow(CALENDAR_FORWARD_DAYS),
|
||||
CATCHUP_MAX_PER_TYPE,
|
||||
);
|
||||
const docs: IndexDoc[] = [];
|
||||
for (let i = 0; i < targetIds.length; i += 50) {
|
||||
const events = await getCalendarEventsForIndex(
|
||||
session, authHeader, jmapAccountId, targetIds.slice(i, i + 50),
|
||||
);
|
||||
for (const event of events) docs.push(extractCalendarEvent(jmapAccountId, event));
|
||||
}
|
||||
return docs;
|
||||
}
|
||||
case 'contact': {
|
||||
const targetIds = ids ?? await queryContactIds(session, authHeader, jmapAccountId, CONTACTS_MAX);
|
||||
const docs: IndexDoc[] = [];
|
||||
for (let i = 0; i < targetIds.length; i += 100) {
|
||||
const cards = await getContactsForIndex(
|
||||
session, authHeader, jmapAccountId, targetIds.slice(i, i + 100),
|
||||
);
|
||||
for (const card of cards) docs.push(extractContact(jmapAccountId, card));
|
||||
}
|
||||
return docs;
|
||||
}
|
||||
case 'file': {
|
||||
const targetIds = ids ?? await queryFileIds(session, authHeader, jmapAccountId, FILES_MAX);
|
||||
const nodes = [];
|
||||
for (let i = 0; i < targetIds.length; i += 100) {
|
||||
nodes.push(...await getFilesForIndex(
|
||||
session, authHeader, jmapAccountId, targetIds.slice(i, i + 100),
|
||||
));
|
||||
}
|
||||
// Paths need the whole set in hand, so this one can't stream per chunk.
|
||||
const paths = buildFilePaths(nodes);
|
||||
return nodes
|
||||
// Directories are indexed too: "what's in the Invoices folder" is a
|
||||
// real query, and a folder row is a few bytes.
|
||||
.map((node) => extractFile(jmapAccountId, node, { path: paths.get(node.id) }));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export interface IndexRequest {
|
||||
/** Types to touch. Empty means every supported type. */
|
||||
types?: readonly ContentType[];
|
||||
/**
|
||||
* Per-type ids to index. Omitted/empty for a type means "refetch that type's
|
||||
* recent window" (the catch-up shape).
|
||||
*/
|
||||
ids?: Partial<Record<ContentType, readonly string[]>>;
|
||||
/** Per-type ids to REMOVE (a JMAP `destroyed`). */
|
||||
removed?: Partial<Record<ContentType, readonly string[]>>;
|
||||
/** Drop documents outside the retention window after writing. */
|
||||
prune?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs one index pass. Opens the encrypted store, fetches, upserts, closes.
|
||||
*
|
||||
* The key is fetched from the main process for the duration of this call only
|
||||
* (`withIndexKey`) and zeroed afterwards - there is no cached handle and no
|
||||
* resident key.
|
||||
*/
|
||||
export async function runIndex(
|
||||
indexSession: IndexSession,
|
||||
req: IndexRequest,
|
||||
): Promise<IndexResult> {
|
||||
const started = Date.now();
|
||||
const storeDir = getStoreDir();
|
||||
if (!storeDir) {
|
||||
throw new IndexSessionError('The local index is not enabled in this deployment.', 404);
|
||||
}
|
||||
|
||||
const session = await fetchJmapSession(indexSession.serverUrl, indexSession.authHeader);
|
||||
|
||||
// Identity cross-check. `generateAccountId` used the username from the auth
|
||||
// context cookie; the server may canonicalise a short login (`linus`) to a
|
||||
// full address (`linus@example.com`) - which is exactly why AccountEntry
|
||||
// carries `serverIdentifiers`. Accept either form, reject anything else
|
||||
// rather than writing one account's mail into another's file.
|
||||
if (session.username) {
|
||||
const serverAccountId = generateAccountId(session.username, indexSession.serverUrl);
|
||||
if (serverAccountId !== indexSession.accountId) {
|
||||
const shortMatches = session.username.split('@')[0] === indexSession.username.split('@')[0];
|
||||
if (!shortMatches) {
|
||||
throw new IndexSessionError(
|
||||
'The JMAP session belongs to a different account than the request cookie.',
|
||||
409,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const { supported, skipped, accountIds } = supportedTypes(session);
|
||||
const requested = req.types && req.types.length > 0 ? req.types : supported;
|
||||
const types = requested.filter((t) => supported.includes(t));
|
||||
const notAttempted = [...new Set([...skipped, ...requested.filter((t) => !supported.includes(t))])];
|
||||
|
||||
const written: Partial<Record<ContentType, number>> = {};
|
||||
const errors: IndexResult['errors'] = [];
|
||||
|
||||
await withIndexKey(indexSession.accountId, async (key) => {
|
||||
const index = MailIndex.open({ storeDir, accountId: indexSession.accountId, key });
|
||||
try {
|
||||
for (const contentType of types) {
|
||||
const jmapAccountId = accountIds[contentType];
|
||||
if (!jmapAccountId) continue;
|
||||
try {
|
||||
const removed = req.removed?.[contentType];
|
||||
if (removed && removed.length > 0) {
|
||||
index.remove(jmapAccountId, contentType, removed.slice(0, MAX_IDS_PER_CALL));
|
||||
}
|
||||
|
||||
const requestedIds = req.ids?.[contentType];
|
||||
const ids = requestedIds && requestedIds.length > 0
|
||||
? requestedIds.slice(0, MAX_IDS_PER_CALL)
|
||||
: null;
|
||||
|
||||
const docs = await fetchDocs(contentType, {
|
||||
session, authHeader: indexSession.authHeader, jmapAccountId, ids,
|
||||
});
|
||||
written[contentType] = index.upsert(docs);
|
||||
|
||||
if (req.prune && contentType === 'mail') {
|
||||
// Only mail prunes by date: calendar's window looks forward,
|
||||
// contacts have no date, and file rows are metadata-sized.
|
||||
index.pruneOlderThan(jmapAccountId, 'mail', isoDaysFromNow(-INDEX_WINDOW_DAYS));
|
||||
}
|
||||
} catch (error) {
|
||||
// One unsupported or misbehaving type must not fail the others.
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
errors.push({ contentType, message });
|
||||
if (error instanceof JmapIndexError && error.status === 401) throw error;
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
index.close();
|
||||
}
|
||||
});
|
||||
|
||||
const result: IndexResult = {
|
||||
accountId: indexSession.accountId,
|
||||
written,
|
||||
skipped: notAttempted,
|
||||
errors,
|
||||
durationMs: Date.now() - started,
|
||||
};
|
||||
logger.info('mail-index: pass complete', {
|
||||
slot: indexSession.slot,
|
||||
written: JSON.stringify(written),
|
||||
skipped: notAttempted.join(',') || 'none',
|
||||
errors: errors.length,
|
||||
durationMs: result.durationMs,
|
||||
});
|
||||
return result;
|
||||
}
|
||||
Reference in New Issue
Block a user