src-advisory/SRCmail
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(mail-index): local AI retrieval returned 0 hits for real questions — AND-every-token FTS matching killed on stop words

Browse Source 
Found by a new real Electron e2e test built specifically to prove the
`local` AI class genuinely works end-to-end in the packaged desktop shell:
a real local Ollama model answering a real question, grounded in the real
encrypted SQLite/FTS5 mail index — not a browser tab, not a mock.

First run surfaced a genuine bug: toFtsMatchQuery() AND-joins every token,
which is right for a deliberate search-box query but wrong for the natural-
language questions the AI retrieval surface (/api/offline/search — see its
own module header, "THE RETRIEVAL SURFACE") actually receives. "When is
check-in for the Villa sul Lago booking, and what time?" shares almost none
of its own function words with the email that answers it, so ANDing every
token — including "when"/"is"/"for"/"the"/"and"/"what" — returned 0 hits
against an index that correctly returns the right email for "Villa sul Lago
check-in".

Fix: new toFtsMatchQueryAny() (lib/mail-index/store.ts) — drops a small,
well-known English stop-word list, OR-joins what's left, and lets the
existing bm25 ranking pick the winner among partial matches. Deliberately a
NEW function, not a change to toFtsMatchQuery itself: that one's own tests
rely on "AND"/"OR"/"NOT" surviving verbatim as literal search terms
(FTS5-keyword-injection safety) — a different guarantee than this one's job
of turning a question into a good search. search() gains a `mode: 'and' |
'any'` option (default 'and', so every existing caller is unaffected); the
offline-search route passes 'any', since its one real caller is exactly
this AI-question shape.

Also added, to make the e2e test possible at all: electron/main.ts's
VNCMAIL_TEST_FIXED_PORT — a narrow, off-by-default escape hatch so
DEV_MOCK_JMAP's JMAP_SERVER_URL can point at this same standalone server's
own /api/dev-jmap. Needed because the encrypted index's key channel
(fd-3/safeStorage) only gets wired up in startStandaloneServer()'s own
random-port launch path, never when ELECTRON_LOAD_URL bypasses it for a
plain `next dev` target — so this was the only way to exercise the real
index without a full Stalwart+SMTP Docker fixture.

Verified live in the real packaged Electron shell, not just unit tests:
real dev-mode login, real multi-round /api/offline/sync + /api/offline/reindex
(39 mail/35 calendar/23 contacts indexed), real local-discovery banner
(11 real Ollama models on this machine), real "Connect", a real question
through the real Settings UI, a real direct renderer->Ollama /api/chat call
(confirmed via network log, never proxied through this app's backend), and
the model's own answer citing the exact right fact: "Saturday 28 March at
15:00" — a fact that exists nowhere except in the one indexed email.

4 new unit tests for toFtsMatchQueryAny. Full gate: tsc clean, eslint
clean, 2502/2502 tests passing, build clean, e2e/electron-ai-local-index.spec.ts
passing against the real standalone server + real Electron + real Ollama.
This commit is contained in:
Bernd Rodler
2026-08-06 18:09:14 +02:00
parent 2a8778c905
commit 7b2047681e
6 changed files with 369 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files
lib/mail-index/store.ts
+79 -24
View File
@@ -331,8 +331,12 @@ export class MailIndex {
types?: readonly ContentType[];
limit?: number;
snippetTokens?: number;
/** 'and' (default): every token required - a deliberate search-box query.
* 'any': stop words dropped, remaining tokens OR-joined, ranked by bm25 -
* a natural-language question (see toFtsMatchQueryAny's docstring). */
mode?: 'and' | 'any';
}): SearchHit[] {
const match = toFtsMatchQuery(opts.query);
const match = opts.mode === 'any' ? toFtsMatchQueryAny(opts.query) : toFtsMatchQuery(opts.query);
if (!match) return [];
const limit = Math.min(Math.max(opts.limit ?? 20, 1), 200);
@@ -446,34 +450,85 @@ function safeParseObject(v: unknown): Record<string, unknown> {
}
/**
* Turns arbitrary user text into a safe FTS5 MATCH expression.
*
* FTS5's query syntax is not SQL, so parameter binding does NOT protect it: a
* bare `"` or a stray `*`/`NEAR`/`:` in user input raises
* `fts5: syntax error`, which would turn a normal search box into a 500. Every
* token is quoted (making it a literal phrase) and a trailing `*` is added to
* the last token so typing continues to match as the user types.
* Splits and safely quotes raw text into FTS5-safe tokens, shared by both
* query-builders below. Split on anything that isn't a word character or an
* intra-word mark - keeps unicode letters (so "Müller" and "東京" survive)
* via the u flag. Every token is quoted (making it a literal phrase) so a
* bare `"` or a stray `*`/`NEAR`/`:` in user input can never raise FTS5's own
* `fts5: syntax error` - that would turn a normal search into a 500.
*/
function quoteFtsTokens(raw: string): string[] {
return raw
.normalize('NFC')
.split(/[^\p{L}\p{N}_@.'-]+/u)
.map((t) => t.replace(/^['-]+|['-]+$/g, ''))
.filter((t) => t.length > 0)
.slice(0, 24)
.map((t, i, all) => {
const quoted = `"${t.replace(/"/g, '""')}"`;
// Prefix-match only the final token, and only if it's long enough to not
// match half the mailbox.
return i === all.length - 1 && t.length >= 3 ? `${quoted}*` : quoted;
});
}
/**
* Turns arbitrary user text into a safe FTS5 MATCH expression, every token
* required (AND-joined). Right for a deliberate, short search-box query,
* where requiring every word is what makes results precise as you type.
*
* Exported for unit testing - it is the one piece of this file with no
* database dependency and the most ways to be wrong.
*/
export function toFtsMatchQuery(raw: string): string | null {
if (typeof raw !== 'string') return null;
// Split on anything that isn't a word character or an intra-word mark. Keeps
// unicode letters (so "Müller" and "東京" survive) via the u flag.
const tokens = raw
.normalize('NFC')
.split(/[^\p{L}\p{N}_@.'-]+/u)
.map((t) => t.replace(/^['-]+|['-]+$/g, ''))
.filter((t) => t.length > 0)
.slice(0, 24);
const tokens = quoteFtsTokens(raw);
if (tokens.length === 0) return null;
return tokens
.map((t, i) => {
const quoted = `"${t.replace(/"/g, '""')}"`;
// Prefix-match only the final token, and only if it's long enough to not
// match half the mailbox.
return i === tokens.length - 1 && t.length >= 3 ? `${quoted}*` : quoted;
})
.join(' AND ');
return tokens.join(' AND ');
}
// A minimal, well-known set of English function words that carry no
// retrieval signal - kept out of toFtsMatchQueryAny's OR expression so they
// don't drown out the bm25 ranking's actual signal (see below). Deliberately
// NOT applied inside quoteFtsTokens/toFtsMatchQuery: that function's own
// tests rely on "AND"/"OR"/"NOT" surviving verbatim as literal search terms
// (FTS5-keyword-injection safety) - a different concern from this one's job
// of turning a natural-language QUESTION into a good search.
const RETRIEVAL_STOP_WORDS = new Set([
'a', 'an', 'the', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'am',
'and', 'or', 'but', 'if', 'then', 'than', 'so', 'because',
'for', 'of', 'to', 'in', 'on', 'at', 'by', 'with', 'from', 'as', 'about', 'into', 'over', 'after', 'before',
'that', 'this', 'these', 'those', 'what', 'when', 'where', 'who', 'whom', 'which', 'why', 'how',
'do', 'does', 'did', 'doing', 'done',
'can', 'could', 'will', 'would', 'shall', 'should', 'may', 'might', 'must',
'i', 'you', 'he', 'she', 'it', 'we', 'they', 'my', 'your', 'his', 'her', 'its', 'our', 'their', 'me', 'him', 'us', 'them',
'not', 'no',
]);
/**
* Turns a natural-language QUESTION into a lenient FTS5 MATCH expression:
* stop words dropped, remaining tokens OR-joined so bm25 ranks by how many
* content words matched instead of requiring every one of them present.
*
* toFtsMatchQuery's strict AND is wrong for this shape of input: a real
* question like "When is check-in for the Villa sul Lago booking?" shares
* almost none of its own function words ("when"/"is"/"for"/"the") with the
* document that actually answers it, so ANDing every token together returns
* nothing - confirmed live: 0 hits for the full question, 2 correct hits for
* the same index once reduced to "Villa sul Lago check-in". The one real
* caller of `/api/offline/search?q=...` is exactly this AI-question shape
* (see that route's own header - no manual search-box UI hits it today), so
* this is the query builder that route now uses, not toFtsMatchQuery.
*/
export function toFtsMatchQueryAny(raw: string): string | null {
if (typeof raw !== 'string') return null;
const withoutStopWords = raw
.split(/\s+/)
.filter((w) => w.length > 0 && !RETRIEVAL_STOP_WORDS.has(w.toLowerCase().replace(/^[^\p{L}\p{N}]+|[^\p{L}\p{N}]+$/gu, '')))
.join(' ');
// Every word was a stop word (e.g. "What is this?") - fall back to the
// original text rather than searching for literally nothing.
const tokens = quoteFtsTokens(withoutStopWords.length > 0 ? withoutStopWords : raw);
if (tokens.length === 0) return null;
return tokens.join(' OR ');
}