Files
SRCmail/lib/mail-index-client.ts
T
Bernd Rodler 87336981d3 feat(mail-index): 1-year retention by default + a recency retrieval leg
The two things that made real questions fail against a correctly-populated
index, both fixed at the root.

RETENTION (A1). `INDEX_WINDOW_DAYS = 30` was not merely a fetch bound — catch-up
also PRUNED mail older than it, so "summarise everything from July" was
unanswerable in August because the rows had been deleted, while the UI said
only that nothing matched. Now a user-visible setting (Settings → About &
Data): 30 days / 3 months / 1 year / everything, defaulting to 1 YEAR per the
product owner. The window bounds the fetch AND the prune from one value so the
two can never disagree and delete what was just written; "everything" skips
pruning entirely rather than falling back to some default bound. The per-pass
ceiling scales with the window (500/30d, hard cap 20k) because 500 messages is
right for a month and nonsense for "everything". Email/query now omits the
`after` filter entirely when unbounded — Stalwart rejects a malformed filter
rather than treating `undefined` as unset.

RECENCY (A2). Keyword search structurally cannot answer a question about WHEN:
bm25 ranks by term overlap, so "who sent the last email" matches documents
containing the word "last", and "all mails in July" matches documents
containing "July" — not documents dated in July. Both were asked by a real
user and both failed. New lib/mail-index/recency.ts detects time intent
(English + German, since the UI ships German) and turns it into a date RANGE;
new MailIndex.recent() answers it with an ordered scan over the already-indexed
`occurred_at`. The route ADDS these hits to the keyword hits rather than
replacing them — "what did the last mail from Anna say" is both kinds of
question at once.

Timezone subtlety worth knowing: bounds are built from LOCAL calendar
boundaries and serialised as UTC instants, so "July" covers the user's July.
A mail at 00:30 local on 1 July belongs to it even though its stored UTC
timestamp reads 30 June. My first test asserted the ISO string prefix, which
would have enshrined the opposite and passed only in UTC — the tests now
assert the local-time property instead.

SCOPE, stated by the product owner and now enforced structurally: the
assistant only ever sees the mailbox the user is signed in to. Both retrieval
legs resolve the active account (local leg by cookie slot, server leg by the
session's own JMAP account); there is deliberately no fan-out across connected
or shared mailboxes, and adding one would be a policy change, not a feature.

Gate: tsc clean, eslint clean, 2520/2520 tests (8 new for recency intent), build clean.
2026-08-07 09:45:10 +02:00

228 lines
8.8 KiB
TypeScript

// Renderer-side client for the encrypted local search index.
//
// The index is EVENT-DRIVEN: the renderer already holds the live JMAP push
// connection (WebSocket -> SSE -> polling, `lib/jmap/client.ts`'s
// setupPushNotifications), so the moment a StateChange announces new mail, a
// calendar change, a contact edit or a file upload, this posts to the reindex
// route. No polling loop, no background worker, no long-lived credential -
// just one more authenticated fetch from the place the push already arrives.
//
// Every function here is best-effort and never throws: a search index failing
// to update must never break the mail UI.
import { apiFetch } from '@/lib/browser-navigation';
import { debug } from '@/lib/debug';
import type { StateChange } from '@/lib/jmap/types';
export type IndexContentType = 'mail' | 'calendar' | 'contact' | 'file';
export interface IndexRunResult {
ok: boolean;
written?: Partial<Record<IndexContentType, number>>;
skipped?: IndexContentType[];
errors?: Array<{ contentType: IndexContentType; message: string }>;
durationMs?: number;
/** Set when the feature isn't available (not the desktop shell, no keyring, no binding). */
unavailable?: boolean;
error?: string;
}
/**
* Maps JMAP `StateChange` type keys onto our content types.
*
* The transport is already type-generic - the WebSocket handler
* (`client.ts:6308-6316`) and the SSE handler (`:6505`) pass the whole
* `changed` map through untouched, and the WS subscribes with
* `dataTypes: null` (every type) - so anything the server pushes arrives here.
*
* `Mailbox` is deliberately NOT mapped: a Mailbox state change is usually just
* an unread-count move, and it fires constantly. `Email` covers the cases that
* change indexable content.
*/
const STATE_TYPE_TO_CONTENT: Record<string, IndexContentType> = {
Email: 'mail',
Calendar: 'calendar',
CalendarEvent: 'calendar',
ContactCard: 'contact',
AddressBook: 'contact',
FileNode: 'file',
};
export function contentTypesFromStateChange(change: StateChange): IndexContentType[] {
const out = new Set<IndexContentType>();
for (const perAccount of Object.values(change.changed ?? {})) {
for (const stateType of Object.keys(perAccount ?? {})) {
const mapped = STATE_TYPE_TO_CONTENT[stateType];
if (mapped) out.add(mapped);
}
}
return [...out];
}
export interface IndexRequestOptions {
types?: readonly IndexContentType[];
/**
* Per-type ids to index. Supply them whenever the renderer already knows
* which objects changed - it turns the call into a couple of `Foo/get`s
* instead of a windowed query. Mail is the frequent case and the one where
* this matters.
*/
ids?: Partial<Record<IndexContentType, string[]>>;
/** Backfill the recent window for every supported type, and prune. */
catchUp?: boolean;
/** Retention window in days, or null to keep everything. Omitted = server
* default (1 year). Bounds the fetch AND the prune together. */
windowDays?: number | null;
/** Cookie slot of the account to index. Defaults to the server's first signed-in slot. */
slot?: number;
}
let inFlight: Promise<IndexRunResult> | null = null;
/** Set once the server says the feature isn't there, so we stop asking. */
let knownUnavailable = false;
/**
* Posts one index request. Single-flighted: a burst of deliveries coalesces
* into the in-flight call rather than queueing N overlapping SQLite writers.
*/
export async function requestIndex(options: IndexRequestOptions = {}): Promise<IndexRunResult> {
if (knownUnavailable) return { ok: false, unavailable: true };
if (inFlight) return inFlight;
const query = typeof options.slot === 'number' ? `?slot=${options.slot}` : '';
const run = (async (): Promise<IndexRunResult> => {
try {
const response = await apiFetch(`/api/offline/reindex${query}`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
types: options.types,
ids: options.ids,
catchUp: options.catchUp === true,
...(options.windowDays !== undefined ? { windowDays: options.windowDays } : {}),
}),
});
// 404 = not the desktop shell (or the feature is gated off). Permanent for
// this page load; stop asking so a busy mailbox doesn't post per delivery.
if (response.status === 404) {
knownUnavailable = true;
return { ok: false, unavailable: true };
}
if (response.status === 503) {
// No keyring / no native binding / no key channel. Also permanent for
// this session, and the message is worth surfacing in Settings.
knownUnavailable = true;
const body = await response.json().catch(() => ({}));
return { ok: false, unavailable: true, error: body?.error };
}
if (!response.ok) {
const body = await response.json().catch(() => ({}));
return { ok: false, error: body?.error || `HTTP ${response.status}` };
}
const body = await response.json();
debug.log('push', '[index] reindex done', body?.written, body?.errors);
return {
ok: true,
written: body?.written,
skipped: body?.skipped,
errors: body?.errors,
durationMs: body?.durationMs,
};
} catch (error) {
return { ok: false, error: error instanceof Error ? error.message : String(error) };
} finally {
inFlight = null;
}
})();
inFlight = run;
return run;
}
/**
* The event-driven entry point, called from the push handler.
*
* `mailIds` lets the caller hand over the ids it already has (the refreshed
* mailbox page), so the frequent mail case costs one `Email/get` rather than a
* 30-day query. The other three types are rare events (a contact edit, a file
* upload, a calendar change), so they fall back to their own bounded queries.
*/
export function indexOnStateChange(
change: StateChange,
opts: { mailIds?: string[]; slot?: number } = {},
): void {
if (knownUnavailable) return;
const types = contentTypesFromStateChange(change);
if (types.length === 0) return;
const ids: Partial<Record<IndexContentType, string[]>> = {};
if (types.includes('mail') && opts.mailIds && opts.mailIds.length > 0) {
ids.mail = opts.mailIds.slice(0, 100);
}
// Fire-and-forget on purpose: this runs inside the push handler, and the mail
// UI must not wait on a search index.
void requestIndex({ types, ids: Object.keys(ids).length > 0 ? ids : undefined, slot: opts.slot });
}
/**
* Launch-time catch-up: backfills whatever changed while the app was closed,
* for which no push event was ever delivered. Also the recovery path for the
* polling transport, which has no signal for contacts or files at all
* (`client.ts`'s buildStatePollingRequest polls Mailbox/Email/Calendar/
* CalendarEvent/SieveScript only).
*/
export async function catchUpIndex(slot?: number, windowDays?: number | null): Promise<IndexRunResult> {
return requestIndex({ catchUp: true, slot, windowDays });
}
// ── Retention setting ────────────────────────────────────────────────────
// Renderer-owned: the renderer already drives every index run, so keeping the
// choice here avoids a second source of truth on the server that could drift
// out of step with what the user last picked.
const RETENTION_KEY = 'vncmail:index:retention-days';
/** Matches DEFAULT_RETENTION_WINDOW_DAYS in lib/mail-index/reindex.ts. */
export const DEFAULT_RETENTION_DAYS = 365;
/** `null` = keep everything. */
export function getRetentionDays(): number | null {
if (typeof window === 'undefined') return DEFAULT_RETENTION_DAYS;
const raw = window.localStorage.getItem(RETENTION_KEY);
if (raw === null) return DEFAULT_RETENTION_DAYS;
if (raw === 'forever') return null;
const parsed = Number.parseInt(raw, 10);
return Number.isFinite(parsed) ? parsed : DEFAULT_RETENTION_DAYS;
}
export function setRetentionDays(days: number | null): void {
if (typeof window === 'undefined') return;
window.localStorage.setItem(RETENTION_KEY, days === null ? 'forever' : String(days));
}
export interface IndexStats {
contentType: string;
count: number;
newest: string | null;
indexedAt: number | null;
}
/** Reads per-type counts without searching. Used by the Settings panel. */
export async function fetchIndexStats(slot?: number): Promise<IndexStats[] | null> {
const slotQuery = typeof slot === 'number' ? `&slot=${slot}` : '';
try {
const response = await apiFetch(`/api/offline/search?stats=true${slotQuery}`);
if (!response.ok) return null;
const body = await response.json();
return Array.isArray(body?.stats) ? (body.stats as IndexStats[]) : [];
} catch {
return null;
}
}
/** Resets the "don't ask again" latch - e.g. after the user signs in again. */
export function resetIndexAvailability(): void {
knownUnavailable = false;
}