The two things that made real questions fail against a correctly-populated index, both fixed at the root. RETENTION (A1). `INDEX_WINDOW_DAYS = 30` was not merely a fetch bound — catch-up also PRUNED mail older than it, so "summarise everything from July" was unanswerable in August because the rows had been deleted, while the UI said only that nothing matched. Now a user-visible setting (Settings → About & Data): 30 days / 3 months / 1 year / everything, defaulting to 1 YEAR per the product owner. The window bounds the fetch AND the prune from one value so the two can never disagree and delete what was just written; "everything" skips pruning entirely rather than falling back to some default bound. The per-pass ceiling scales with the window (500/30d, hard cap 20k) because 500 messages is right for a month and nonsense for "everything". Email/query now omits the `after` filter entirely when unbounded — Stalwart rejects a malformed filter rather than treating `undefined` as unset. RECENCY (A2). Keyword search structurally cannot answer a question about WHEN: bm25 ranks by term overlap, so "who sent the last email" matches documents containing the word "last", and "all mails in July" matches documents containing "July" — not documents dated in July. Both were asked by a real user and both failed. New lib/mail-index/recency.ts detects time intent (English + German, since the UI ships German) and turns it into a date RANGE; new MailIndex.recent() answers it with an ordered scan over the already-indexed `occurred_at`. The route ADDS these hits to the keyword hits rather than replacing them — "what did the last mail from Anna say" is both kinds of question at once. Timezone subtlety worth knowing: bounds are built from LOCAL calendar boundaries and serialised as UTC instants, so "July" covers the user's July. A mail at 00:30 local on 1 July belongs to it even though its stored UTC timestamp reads 30 June. My first test asserted the ISO string prefix, which would have enshrined the opposite and passed only in UTC — the tests now assert the local-time property instead. SCOPE, stated by the product owner and now enforced structurally: the assistant only ever sees the mailbox the user is signed in to. Both retrieval legs resolve the active account (local leg by cookie slot, server leg by the session's own JMAP account); there is deliberately no fan-out across connected or shared mailboxes, and adding one would be a policy change, not a feature. Gate: tsc clean, eslint clean, 2520/2520 tests (8 new for recency intent), build clean.
598 lines
24 KiB
TypeScript
598 lines
24 KiB
TypeScript
// The encrypted local search index: schema, open/close, upsert, search.
|
|
//
|
|
// One SQLite (SQLCipher) file per account. Rows are ALSO account-scoped
|
|
// internally - `(jmap_account_id, content_type, id)` - because a single login
|
|
// exposes the user's own JMAP account plus every delegated/shared account, and
|
|
// JMAP ids are unique only WITHIN an account (this codebase already works
|
|
// around that collision in `lib/jmap/client.ts:388`'s namespaceMailboxIds).
|
|
// One file per account keeps purge trivial; the composite key keeps
|
|
// delegated accounts from merging inside it.
|
|
//
|
|
// This is a SEARCH INDEX, not a mail replica. It is allowed to be stale, it is
|
|
// allowed to be incomplete, and it can be discarded and rebuilt at any time -
|
|
// which is why the schema-version mismatch path below simply drops everything
|
|
// rather than migrating.
|
|
|
|
import fs from 'node:fs';
|
|
import path from 'node:path';
|
|
import { loadSqlcipher, type SqlcipherConstructor, type SqlcipherDatabase } from './binding';
|
|
import { dbSiblings, indexDbPath } from './paths';
|
|
|
|
export const SCHEMA_VERSION = 1;
|
|
|
|
export type ContentType = 'mail' | 'calendar' | 'contact' | 'file';
|
|
|
|
export const CONTENT_TYPES: readonly ContentType[] = ['mail', 'calendar', 'contact', 'file'];
|
|
|
|
export function isContentType(v: unknown): v is ContentType {
|
|
return typeof v === 'string' && (CONTENT_TYPES as readonly string[]).includes(v);
|
|
}
|
|
|
|
/**
|
|
* One indexable thing, already flattened to text. Produced by the pure
|
|
* extractors in `extract.ts` so that every JMAP-shape decision is unit-testable
|
|
* without a database or a server.
|
|
*/
|
|
export interface IndexDoc {
|
|
jmapAccountId: string;
|
|
contentType: ContentType;
|
|
/** JMAP id. Unique only within (jmapAccountId, contentType). */
|
|
id: string;
|
|
/** Subject / event title / contact display name / filename. */
|
|
title: string;
|
|
/** Addresses and names: sender+recipients, attendees, contact emails/phones, owner. */
|
|
people: string;
|
|
/** The bulk searchable text. Plain text only - never HTML. */
|
|
body: string;
|
|
/** ISO 8601, or null when the type has no meaningful date. Drives recency ordering. */
|
|
occurredAt: string | null;
|
|
/** Small type-specific extras returned verbatim to the caller (never searched). */
|
|
metadata: Record<string, unknown>;
|
|
}
|
|
|
|
export interface SearchHit {
|
|
contentType: ContentType;
|
|
id: string;
|
|
jmapAccountId: string;
|
|
title: string;
|
|
people: string;
|
|
occurredAt: string | null;
|
|
metadata: Record<string, unknown>;
|
|
/** FTS5 bm25 score. Lower is a better match (bm25 returns negative values). */
|
|
score: number;
|
|
/** Highlighted excerpt from the body, for feeding an LLM as context. */
|
|
snippet: string;
|
|
}
|
|
|
|
const DDL = `
|
|
CREATE TABLE IF NOT EXISTS doc (
|
|
jmap_account_id TEXT NOT NULL,
|
|
content_type TEXT NOT NULL,
|
|
id TEXT NOT NULL,
|
|
title TEXT NOT NULL DEFAULT '',
|
|
people TEXT NOT NULL DEFAULT '',
|
|
body TEXT NOT NULL DEFAULT '',
|
|
occurred_at TEXT,
|
|
metadata_json TEXT NOT NULL DEFAULT '{}',
|
|
indexed_at INTEGER NOT NULL,
|
|
PRIMARY KEY (jmap_account_id, content_type, id)
|
|
);
|
|
CREATE INDEX IF NOT EXISTS doc_recent
|
|
ON doc(jmap_account_id, content_type, occurred_at DESC);
|
|
|
|
-- Standalone (not external-content) FTS5: the text is duplicated into this
|
|
-- table and kept in step manually on upsert. External content would avoid the
|
|
-- duplication but requires deleting the old FTS row using its OLD column
|
|
-- values, which an upsert does not have to hand - a well-known source of
|
|
-- silently-stale FTS rows. At this scale (a bounded recent window) the
|
|
-- duplication is the cheaper correctness trade.
|
|
CREATE VIRTUAL TABLE IF NOT EXISTS doc_fts USING fts5(
|
|
title, people, body,
|
|
tokenize='unicode61 remove_diacritics 2'
|
|
);
|
|
|
|
CREATE TABLE IF NOT EXISTS meta (k TEXT PRIMARY KEY, v TEXT NOT NULL);
|
|
`;
|
|
|
|
export class MailIndexUnavailableError extends Error {
|
|
constructor(message: string) {
|
|
super(message);
|
|
this.name = 'MailIndexUnavailableError';
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Assert that the file we just opened is REALLY encrypted.
|
|
*
|
|
* This is not defensive boilerplate, it guards the sharpest landmine found
|
|
* while designing this: on both `node:sqlite` and plain `better-sqlite3`,
|
|
* `PRAGMA key = ...` is **silently accepted and does nothing** - no error, a
|
|
* working database, and the mail sitting on disk in cleartext. Verified by
|
|
* writing a file and recovering a canary string from the raw bytes.
|
|
*
|
|
* The check is on the VALUE, not the row count: a non-cipher binding returns
|
|
* ZERO ROWS for `PRAGMA cipher_version`, so a naive `!== ''` comparison over a
|
|
* missing row passes vacuously. Require a non-empty string.
|
|
*/
|
|
function assertEncrypted(db: SqlcipherDatabase, dbPath: string): void {
|
|
const rows = db.pragma('cipher_version');
|
|
const value =
|
|
Array.isArray(rows) && rows.length > 0 && rows[0] && typeof rows[0] === 'object'
|
|
? (rows[0] as Record<string, unknown>).cipher_version
|
|
: undefined;
|
|
if (typeof value !== 'string' || value.trim().length === 0) {
|
|
db.close();
|
|
throw new MailIndexUnavailableError(
|
|
`Refusing to use ${path.basename(dbPath)}: the SQLite binding reports no SQLCipher ` +
|
|
`support (PRAGMA cipher_version returned ${JSON.stringify(rows)}), so the index ` +
|
|
`would be written in cleartext.`,
|
|
);
|
|
}
|
|
}
|
|
|
|
export interface OpenOptions {
|
|
storeDir: string;
|
|
accountId: string;
|
|
/** Raw 32-byte key. Used as SQLCipher's raw key (no KDF) via `PRAGMA key = "x'..'"`. */
|
|
key: Buffer;
|
|
}
|
|
|
|
/**
|
|
* Opens the connection, sets the SQLCipher key, verifies real encryption, and
|
|
* applies the fixed pragmas - the sequence both the first attempt and the
|
|
* wrong-key retry in `open()` need identically.
|
|
*
|
|
* On ANY failure the just-opened connection is closed before the error
|
|
* propagates. This matters beyond `assertEncrypted`'s own failure (which
|
|
* already closes): a bare `db.pragma(...)` throwing - SQLITE_BUSY, a full disk
|
|
* on the first WAL write, anything - must not leak the native handle either,
|
|
* which a `try` wrapped around only some of these calls previously missed.
|
|
*
|
|
* Exported so the cleanup guarantee can be unit-tested against a fake
|
|
* `SqlcipherDatabase` - a real double failure (wrong key, THEN a pragma
|
|
* failure on the freshly rebuilt file) is not practically reproducible
|
|
* against the real binding.
|
|
*/
|
|
export function openKeyed(
|
|
Database: SqlcipherConstructor,
|
|
dbPath: string,
|
|
key: Buffer,
|
|
): { db: SqlcipherDatabase; version: number | null } {
|
|
const db = new Database(dbPath);
|
|
try {
|
|
// The key pragma must be the FIRST statement on the connection. Hex form
|
|
// means SQLCipher uses these 32 bytes as the raw key with no KDF, which is
|
|
// right for a random key (a passphrase would want the KDF).
|
|
db.pragma(`key = "x'${key.toString('hex')}'"`);
|
|
assertEncrypted(db, dbPath);
|
|
db.pragma('journal_mode = WAL');
|
|
db.pragma('synchronous = NORMAL');
|
|
// The offline replica (lib/offline-replica/**) is a SECOND connection to
|
|
// this same file, writing disjoint tables. WAL lets a writer and readers
|
|
// coexist, but two WRITERS get SQLITE_BUSY immediately without this - and
|
|
// both subsystems are driven by the same renderer push handler, so they
|
|
// genuinely do overlap.
|
|
db.pragma('busy_timeout = 8000');
|
|
return { db, version: readSchemaVersion(db) };
|
|
} catch (error) {
|
|
// Idempotent: assertEncrypted already closed on its own failure, so this
|
|
// is a harmless no-op in that case.
|
|
try { db.close(); } catch { /* already closed */ }
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
export class MailIndex {
|
|
private constructor(
|
|
private readonly db: SqlcipherDatabase,
|
|
readonly dbPath: string,
|
|
) {}
|
|
|
|
/**
|
|
* Opens (creating if needed) the account's index. Throws
|
|
* MailIndexUnavailableError when the native binding is absent or the file is
|
|
* not actually encrypted; the caller turns the feature off rather than
|
|
* falling back to something unencrypted.
|
|
*/
|
|
static open({ storeDir, accountId, key }: OpenOptions): MailIndex {
|
|
const Database = loadSqlcipher();
|
|
if (!Database) {
|
|
throw new MailIndexUnavailableError(
|
|
'@signalapp/sqlcipher is not installed for this platform (it is an optional dependency).',
|
|
);
|
|
}
|
|
if (key.length !== 32) {
|
|
throw new MailIndexUnavailableError(`Index key must be 32 bytes, got ${key.length}.`);
|
|
}
|
|
|
|
const dbPath = indexDbPath(storeDir, accountId);
|
|
fs.mkdirSync(path.dirname(dbPath), { recursive: true, mode: 0o700 });
|
|
|
|
// A wrong key surfaces here rather than at open: SQLCipher only reads the
|
|
// header lazily. Treat it as "unreadable" and rebuild from scratch - the
|
|
// index is derived data, so there is nothing to recover and never anything
|
|
// to prompt the user for (the key was never a user secret).
|
|
let opened: { db: SqlcipherDatabase; version: number | null };
|
|
try {
|
|
opened = openKeyed(Database, dbPath, key);
|
|
} catch {
|
|
// `openKeyed` guarantees the failed connection above is already closed,
|
|
// so there is nothing to clean up here before retrying on a fresh file.
|
|
for (const f of dbSiblings(dbPath)) {
|
|
try { fs.rmSync(f, { force: true }); } catch { /* best effort */ }
|
|
}
|
|
opened = openKeyed(Database, dbPath, key);
|
|
opened.version = null; // fresh file - nothing to read
|
|
}
|
|
const db = opened.db;
|
|
let version = opened.version;
|
|
|
|
if (version !== null && version !== SCHEMA_VERSION) {
|
|
// Rebuildable derived data: drop, don't migrate.
|
|
db.exec('DROP TABLE IF EXISTS doc_fts; DROP TABLE IF EXISTS doc; DROP TABLE IF EXISTS meta;');
|
|
version = null;
|
|
}
|
|
if (version === null) {
|
|
db.exec(DDL);
|
|
db.prepare('INSERT OR REPLACE INTO meta (k, v) VALUES (?, ?)').run([
|
|
'schema_version',
|
|
String(SCHEMA_VERSION),
|
|
]);
|
|
}
|
|
|
|
return new MailIndex(db, dbPath);
|
|
}
|
|
|
|
close(): void {
|
|
try { this.db.close(); } catch { /* already closed */ }
|
|
}
|
|
|
|
/**
|
|
* Upserts documents and keeps the FTS rows in step. Returns the number of
|
|
* rows written. One transaction for the whole batch - a partially-applied
|
|
* batch is harmless (it is an index) but a transaction is faster.
|
|
*/
|
|
upsert(docs: readonly IndexDoc[]): number {
|
|
if (docs.length === 0) return 0;
|
|
|
|
const upsertDoc = this.db.prepare(`
|
|
INSERT INTO doc (jmap_account_id, content_type, id, title, people, body,
|
|
occurred_at, metadata_json, indexed_at)
|
|
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
|
|
ON CONFLICT(jmap_account_id, content_type, id) DO UPDATE SET
|
|
title = excluded.title, people = excluded.people, body = excluded.body,
|
|
occurred_at = excluded.occurred_at, metadata_json = excluded.metadata_json,
|
|
indexed_at = excluded.indexed_at
|
|
RETURNING rowid
|
|
`);
|
|
const deleteFts = this.db.prepare('DELETE FROM doc_fts WHERE rowid = ?');
|
|
const insertFts = this.db.prepare(
|
|
'INSERT INTO doc_fts (rowid, title, people, body) VALUES (?, ?, ?, ?)',
|
|
);
|
|
|
|
const now = Date.now();
|
|
let written = 0;
|
|
this.db.exec('BEGIN');
|
|
try {
|
|
for (const d of docs) {
|
|
const row = upsertDoc.get([
|
|
d.jmapAccountId, d.contentType, d.id,
|
|
d.title, d.people, d.body,
|
|
d.occurredAt, JSON.stringify(d.metadata ?? {}), now,
|
|
]);
|
|
const rowid = row?.rowid;
|
|
if (typeof rowid !== 'number') continue;
|
|
// ON CONFLICT preserves the rowid, so delete-then-insert replaces the
|
|
// old FTS row rather than accumulating duplicates for one document.
|
|
deleteFts.run([rowid]);
|
|
insertFts.run([rowid, d.title, d.people, d.body]);
|
|
written++;
|
|
}
|
|
this.db.exec('COMMIT');
|
|
} catch (error) {
|
|
this.db.exec('ROLLBACK');
|
|
throw error;
|
|
}
|
|
return written;
|
|
}
|
|
|
|
/** Removes documents by id (a JMAP `destroyed` id, or a stale row). */
|
|
remove(jmapAccountId: string, contentType: ContentType, ids: readonly string[]): number {
|
|
if (ids.length === 0) return 0;
|
|
const findRow = this.db.prepare(
|
|
'SELECT rowid FROM doc WHERE jmap_account_id = ? AND content_type = ? AND id = ?',
|
|
);
|
|
const deleteFts = this.db.prepare('DELETE FROM doc_fts WHERE rowid = ?');
|
|
const deleteDoc = this.db.prepare(
|
|
'DELETE FROM doc WHERE jmap_account_id = ? AND content_type = ? AND id = ?',
|
|
);
|
|
let removed = 0;
|
|
this.db.exec('BEGIN');
|
|
try {
|
|
for (const id of ids) {
|
|
const row = findRow.get([jmapAccountId, contentType, id]);
|
|
if (typeof row?.rowid === 'number') deleteFts.run([row.rowid]);
|
|
removed += deleteDoc.run([jmapAccountId, contentType, id]).changes;
|
|
}
|
|
this.db.exec('COMMIT');
|
|
} catch (error) {
|
|
this.db.exec('ROLLBACK');
|
|
throw error;
|
|
}
|
|
return removed;
|
|
}
|
|
|
|
/**
|
|
* Full-text search - the retrieval surface an AI feature calls to gather
|
|
* context. `types` empty/omitted searches everything.
|
|
*/
|
|
search(opts: {
|
|
query: string;
|
|
types?: readonly ContentType[];
|
|
limit?: number;
|
|
snippetTokens?: number;
|
|
/** 'and' (default): every token required - a deliberate search-box query.
|
|
* 'any': stop words dropped, remaining tokens OR-joined, ranked by bm25 -
|
|
* a natural-language question (see toFtsMatchQueryAny's docstring). */
|
|
mode?: 'and' | 'any';
|
|
}): SearchHit[] {
|
|
const match = opts.mode === 'any' ? toFtsMatchQueryAny(opts.query) : toFtsMatchQuery(opts.query);
|
|
if (!match) return [];
|
|
|
|
const limit = Math.min(Math.max(opts.limit ?? 20, 1), 200);
|
|
const tokens = Math.min(Math.max(opts.snippetTokens ?? 24, 4), 64);
|
|
const types = opts.types && opts.types.length > 0 ? opts.types : null;
|
|
const typeFilter = types ? ` AND d.content_type IN (${types.map(() => '?').join(',')})` : '';
|
|
|
|
// bm25 weights: a hit in the title or in a name/address is a stronger
|
|
// signal than one in a long body, and for RAG the title is what makes a
|
|
// retrieved chunk recognisable.
|
|
const rows = this.db
|
|
.prepare(`
|
|
SELECT d.content_type, d.id, d.jmap_account_id, d.title, d.people,
|
|
d.occurred_at, d.metadata_json,
|
|
bm25(doc_fts, 8.0, 4.0, 1.0) AS score,
|
|
snippet(doc_fts, 2, '[', ']', '…', ${tokens}) AS snip
|
|
FROM doc_fts
|
|
JOIN doc d ON d.rowid = doc_fts.rowid
|
|
WHERE doc_fts MATCH ?${typeFilter}
|
|
ORDER BY score ASC, d.occurred_at DESC
|
|
LIMIT ?
|
|
`)
|
|
.all([match, ...(types ?? []), limit]);
|
|
|
|
return rows.map((r) => ({
|
|
contentType: String(r.content_type) as ContentType,
|
|
id: String(r.id),
|
|
jmapAccountId: String(r.jmap_account_id),
|
|
title: String(r.title ?? ''),
|
|
people: String(r.people ?? ''),
|
|
occurredAt: r.occurred_at === null || r.occurred_at === undefined ? null : String(r.occurred_at),
|
|
metadata: safeParseObject(r.metadata_json),
|
|
score: typeof r.score === 'number' ? r.score : 0,
|
|
snippet: String(r.snip ?? ''),
|
|
}));
|
|
}
|
|
|
|
/**
|
|
* Newest documents by date, ignoring keyword relevance entirely.
|
|
*
|
|
* The retrieval leg for RECENCY questions — "what was the last mail", "who
|
|
* wrote most recently", "everything from July". Full-text search cannot
|
|
* answer those even with a perfect index: bm25 ranks by term overlap and has
|
|
* no notion of "latest", so "the last mail" matches documents containing the
|
|
* word "last". Two real questions failed exactly that way before this
|
|
* existed. `doc(jmap_account_id, content_type, occurred_at DESC)` is already
|
|
* indexed, so this is an ordered range scan, not a table sweep.
|
|
*
|
|
* `since`/`until` are ISO strings, both optional — a month-name question
|
|
* becomes a bounded range, a bare "latest" becomes an unbounded top-N.
|
|
*/
|
|
recent(opts: {
|
|
types?: readonly ContentType[];
|
|
limit?: number;
|
|
since?: string;
|
|
until?: string;
|
|
snippetChars?: number;
|
|
}): SearchHit[] {
|
|
const limit = Math.min(Math.max(opts.limit ?? 10, 1), 200);
|
|
const snippetChars = Math.min(Math.max(opts.snippetChars ?? 400, 80), 2000);
|
|
const types = opts.types && opts.types.length > 0 ? opts.types : null;
|
|
|
|
const where: string[] = ['d.occurred_at IS NOT NULL'];
|
|
const params: Array<string | number> = [];
|
|
if (types) {
|
|
where.push(`d.content_type IN (${types.map(() => '?').join(',')})`);
|
|
params.push(...types);
|
|
}
|
|
if (opts.since) { where.push('d.occurred_at >= ?'); params.push(opts.since); }
|
|
if (opts.until) { where.push('d.occurred_at <= ?'); params.push(opts.until); }
|
|
|
|
const rows = this.db
|
|
.prepare(`
|
|
SELECT d.content_type, d.id, d.jmap_account_id, d.title, d.people,
|
|
d.occurred_at, d.metadata_json,
|
|
substr(f.body, 1, ${snippetChars}) AS snip
|
|
FROM doc d
|
|
LEFT JOIN doc_fts f ON f.rowid = d.rowid
|
|
WHERE ${where.join(' AND ')}
|
|
ORDER BY d.occurred_at DESC
|
|
LIMIT ?
|
|
`)
|
|
.all([...params, limit]);
|
|
|
|
return rows.map((r) => ({
|
|
contentType: String(r.content_type) as ContentType,
|
|
id: String(r.id),
|
|
jmapAccountId: String(r.jmap_account_id),
|
|
title: String(r.title ?? ''),
|
|
people: String(r.people ?? ''),
|
|
occurredAt: r.occurred_at === null || r.occurred_at === undefined ? null : String(r.occurred_at),
|
|
metadata: safeParseObject(r.metadata_json),
|
|
// No bm25 score here: these are ordered by time, not relevance, and
|
|
// faking a relevance number would let the fusion step rank them as if
|
|
// they had been scored.
|
|
score: 0,
|
|
snippet: String(r.snip ?? ''),
|
|
}));
|
|
}
|
|
|
|
/** Per-type counts and freshness, for the Settings UI and for debugging. */
|
|
stats(): Array<{ contentType: string; count: number; newest: string | null; indexedAt: number | null }> {
|
|
return this.db
|
|
.prepare(`
|
|
SELECT content_type, COUNT(*) AS n, MAX(occurred_at) AS newest, MAX(indexed_at) AS indexed
|
|
FROM doc GROUP BY content_type ORDER BY content_type
|
|
`)
|
|
.all()
|
|
.map((r) => ({
|
|
contentType: String(r.content_type),
|
|
count: Number(r.n ?? 0),
|
|
newest: r.newest === null || r.newest === undefined ? null : String(r.newest),
|
|
indexedAt: typeof r.indexed === 'number' ? r.indexed : null,
|
|
}));
|
|
}
|
|
|
|
/** Ids already present, so a catch-up pass can skip re-fetching bodies. */
|
|
existingIds(jmapAccountId: string, contentType: ContentType): Set<string> {
|
|
const rows = this.db
|
|
.prepare('SELECT id FROM doc WHERE jmap_account_id = ? AND content_type = ?')
|
|
.all([jmapAccountId, contentType]);
|
|
return new Set(rows.map((r) => String(r.id)));
|
|
}
|
|
|
|
/** Drops documents older than the retention floor for a type. */
|
|
pruneOlderThan(jmapAccountId: string, contentType: ContentType, isoFloor: string): number {
|
|
const rows = this.db
|
|
.prepare(`
|
|
SELECT rowid FROM doc
|
|
WHERE jmap_account_id = ? AND content_type = ?
|
|
AND occurred_at IS NOT NULL AND occurred_at < ?
|
|
`)
|
|
.all([jmapAccountId, contentType, isoFloor]);
|
|
if (rows.length === 0) return 0;
|
|
const deleteFts = this.db.prepare('DELETE FROM doc_fts WHERE rowid = ?');
|
|
const deleteDoc = this.db.prepare('DELETE FROM doc WHERE rowid = ?');
|
|
this.db.exec('BEGIN');
|
|
try {
|
|
for (const r of rows) {
|
|
deleteFts.run([r.rowid]);
|
|
deleteDoc.run([r.rowid]);
|
|
}
|
|
this.db.exec('COMMIT');
|
|
} catch (error) {
|
|
this.db.exec('ROLLBACK');
|
|
throw error;
|
|
}
|
|
return rows.length;
|
|
}
|
|
}
|
|
|
|
function readSchemaVersion(db: SqlcipherDatabase): number | null {
|
|
try {
|
|
const row = db.prepare("SELECT v FROM meta WHERE k = 'schema_version'").get();
|
|
if (!row || row.v === undefined) return null;
|
|
const n = Number(row.v);
|
|
return Number.isFinite(n) ? n : null;
|
|
} catch {
|
|
// `meta` doesn't exist yet - a fresh file.
|
|
return null;
|
|
}
|
|
}
|
|
|
|
function safeParseObject(v: unknown): Record<string, unknown> {
|
|
if (typeof v !== 'string') return {};
|
|
try {
|
|
const parsed = JSON.parse(v);
|
|
return parsed && typeof parsed === 'object' && !Array.isArray(parsed)
|
|
? (parsed as Record<string, unknown>)
|
|
: {};
|
|
} catch {
|
|
return {};
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Splits and safely quotes raw text into FTS5-safe tokens, shared by both
|
|
* query-builders below. Split on anything that isn't a word character or an
|
|
* intra-word mark - keeps unicode letters (so "Müller" and "東京" survive)
|
|
* via the u flag. Every token is quoted (making it a literal phrase) so a
|
|
* bare `"` or a stray `*`/`NEAR`/`:` in user input can never raise FTS5's own
|
|
* `fts5: syntax error` - that would turn a normal search into a 500.
|
|
*/
|
|
function quoteFtsTokens(raw: string): string[] {
|
|
return raw
|
|
.normalize('NFC')
|
|
.split(/[^\p{L}\p{N}_@.'-]+/u)
|
|
.map((t) => t.replace(/^['-]+|['-]+$/g, ''))
|
|
.filter((t) => t.length > 0)
|
|
.slice(0, 24)
|
|
.map((t, i, all) => {
|
|
const quoted = `"${t.replace(/"/g, '""')}"`;
|
|
// Prefix-match only the final token, and only if it's long enough to not
|
|
// match half the mailbox.
|
|
return i === all.length - 1 && t.length >= 3 ? `${quoted}*` : quoted;
|
|
});
|
|
}
|
|
|
|
/**
|
|
* Turns arbitrary user text into a safe FTS5 MATCH expression, every token
|
|
* required (AND-joined). Right for a deliberate, short search-box query,
|
|
* where requiring every word is what makes results precise as you type.
|
|
*
|
|
* Exported for unit testing - it is the one piece of this file with no
|
|
* database dependency and the most ways to be wrong.
|
|
*/
|
|
export function toFtsMatchQuery(raw: string): string | null {
|
|
if (typeof raw !== 'string') return null;
|
|
const tokens = quoteFtsTokens(raw);
|
|
if (tokens.length === 0) return null;
|
|
return tokens.join(' AND ');
|
|
}
|
|
|
|
// A minimal, well-known set of English function words that carry no
|
|
// retrieval signal - kept out of toFtsMatchQueryAny's OR expression so they
|
|
// don't drown out the bm25 ranking's actual signal (see below). Deliberately
|
|
// NOT applied inside quoteFtsTokens/toFtsMatchQuery: that function's own
|
|
// tests rely on "AND"/"OR"/"NOT" surviving verbatim as literal search terms
|
|
// (FTS5-keyword-injection safety) - a different concern from this one's job
|
|
// of turning a natural-language QUESTION into a good search.
|
|
const RETRIEVAL_STOP_WORDS = new Set([
|
|
'a', 'an', 'the', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'am',
|
|
'and', 'or', 'but', 'if', 'then', 'than', 'so', 'because',
|
|
'for', 'of', 'to', 'in', 'on', 'at', 'by', 'with', 'from', 'as', 'about', 'into', 'over', 'after', 'before',
|
|
'that', 'this', 'these', 'those', 'what', 'when', 'where', 'who', 'whom', 'which', 'why', 'how',
|
|
'do', 'does', 'did', 'doing', 'done',
|
|
'can', 'could', 'will', 'would', 'shall', 'should', 'may', 'might', 'must',
|
|
'i', 'you', 'he', 'she', 'it', 'we', 'they', 'my', 'your', 'his', 'her', 'its', 'our', 'their', 'me', 'him', 'us', 'them',
|
|
'not', 'no',
|
|
]);
|
|
|
|
/**
|
|
* Turns a natural-language QUESTION into a lenient FTS5 MATCH expression:
|
|
* stop words dropped, remaining tokens OR-joined so bm25 ranks by how many
|
|
* content words matched instead of requiring every one of them present.
|
|
*
|
|
* toFtsMatchQuery's strict AND is wrong for this shape of input: a real
|
|
* question like "When is check-in for the Villa sul Lago booking?" shares
|
|
* almost none of its own function words ("when"/"is"/"for"/"the") with the
|
|
* document that actually answers it, so ANDing every token together returns
|
|
* nothing - confirmed live: 0 hits for the full question, 2 correct hits for
|
|
* the same index once reduced to "Villa sul Lago check-in". The one real
|
|
* caller of `/api/offline/search?q=...` is exactly this AI-question shape
|
|
* (see that route's own header - no manual search-box UI hits it today), so
|
|
* this is the query builder that route now uses, not toFtsMatchQuery.
|
|
*/
|
|
export function toFtsMatchQueryAny(raw: string): string | null {
|
|
if (typeof raw !== 'string') return null;
|
|
const withoutStopWords = raw
|
|
.split(/\s+/)
|
|
.filter((w) => w.length > 0 && !RETRIEVAL_STOP_WORDS.has(w.toLowerCase().replace(/^[^\p{L}\p{N}]+|[^\p{L}\p{N}]+$/gu, '')))
|
|
.join(' ');
|
|
// Every word was a stop word (e.g. "What is this?") - fall back to the
|
|
// original text rather than searching for literally nothing.
|
|
const tokens = quoteFtsTokens(withoutStopWords.length > 0 ? withoutStopWords : raw);
|
|
if (tokens.length === 0) return null;
|
|
return tokens.join(' OR ');
|
|
}
|