fix(mail-index): local AI retrieval returned 0 hits for real questions — AND-every-token FTS matching killed on stop words
Found by a new real Electron e2e test built specifically to prove the `local` AI class genuinely works end-to-end in the packaged desktop shell: a real local Ollama model answering a real question, grounded in the real encrypted SQLite/FTS5 mail index — not a browser tab, not a mock. First run surfaced a genuine bug: toFtsMatchQuery() AND-joins every token, which is right for a deliberate search-box query but wrong for the natural- language questions the AI retrieval surface (/api/offline/search — see its own module header, "THE RETRIEVAL SURFACE") actually receives. "When is check-in for the Villa sul Lago booking, and what time?" shares almost none of its own function words with the email that answers it, so ANDing every token — including "when"/"is"/"for"/"the"/"and"/"what" — returned 0 hits against an index that correctly returns the right email for "Villa sul Lago check-in". Fix: new toFtsMatchQueryAny() (lib/mail-index/store.ts) — drops a small, well-known English stop-word list, OR-joins what's left, and lets the existing bm25 ranking pick the winner among partial matches. Deliberately a NEW function, not a change to toFtsMatchQuery itself: that one's own tests rely on "AND"/"OR"/"NOT" surviving verbatim as literal search terms (FTS5-keyword-injection safety) — a different guarantee than this one's job of turning a question into a good search. search() gains a `mode: 'and' | 'any'` option (default 'and', so every existing caller is unaffected); the offline-search route passes 'any', since its one real caller is exactly this AI-question shape. Also added, to make the e2e test possible at all: electron/main.ts's VNCMAIL_TEST_FIXED_PORT — a narrow, off-by-default escape hatch so DEV_MOCK_JMAP's JMAP_SERVER_URL can point at this same standalone server's own /api/dev-jmap. Needed because the encrypted index's key channel (fd-3/safeStorage) only gets wired up in startStandaloneServer()'s own random-port launch path, never when ELECTRON_LOAD_URL bypasses it for a plain `next dev` target — so this was the only way to exercise the real index without a full Stalwart+SMTP Docker fixture. Verified live in the real packaged Electron shell, not just unit tests: real dev-mode login, real multi-round /api/offline/sync + /api/offline/reindex (39 mail/35 calendar/23 contacts indexed), real local-discovery banner (11 real Ollama models on this machine), real "Connect", a real question through the real Settings UI, a real direct renderer->Ollama /api/chat call (confirmed via network log, never proxied through this app's backend), and the model's own answer citing the exact right fact: "Saturday 28 March at 15:00" — a fact that exists nowhere except in the one indexed email. 4 new unit tests for toFtsMatchQueryAny. Full gate: tsc clean, eslint clean, 2502/2502 tests passing, build clean, e2e/electron-ai-local-index.spec.ts passing against the real standalone server + real Electron + real Ollama.
This commit is contained in:
Bernd Rodler
@@ -6,7 +6,7 @@ import { afterEach, beforeEach, describe, expect, it } from 'vitest';
|
||||
import { isSqlcipherAvailable } from '../binding';
|
||||
import type { SqlcipherConstructor, SqlcipherDatabase, SqlcipherStatement } from '../binding';
|
||||
import { accountFileToken, getStoreDir, indexDbPath, STORE_DIR_ENV } from '../paths';
|
||||
import { MailIndex, openKeyed, toFtsMatchQuery, type IndexDoc } from '../store';
|
||||
import { MailIndex, openKeyed, toFtsMatchQuery, toFtsMatchQueryAny, type IndexDoc } from '../store';
|
||||
|
||||
describe('toFtsMatchQuery', () => {
|
||||
it('quotes every token so FTS5 operators in user input cannot break the query', () => {
|
||||
@@ -48,6 +48,62 @@ describe('toFtsMatchQuery', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('toFtsMatchQueryAny', () => {
|
||||
it('drops English function words and OR-joins what is left - the confirmed-live failure this fixes', () => {
|
||||
// AND-every-token (toFtsMatchQuery) returns 0 hits for this exact
|
||||
// question against a document that only contains "Villa sul Lago" and
|
||||
// "check-in" - see app/api/offline/search/route.ts's comment and the
|
||||
// e2e electron-ai-local-index.spec.ts run that first caught this.
|
||||
const result = toFtsMatchQueryAny('When is check-in for the Villa sul Lago booking, and what time?');
|
||||
expect(result).not.toBeNull();
|
||||
expect(result).not.toContain(' AND ');
|
||||
expect(result).toContain('"check-in"');
|
||||
expect(result).toContain('"Villa"');
|
||||
expect(result).toContain('"sul"');
|
||||
expect(result).toContain('"Lago"');
|
||||
expect(result).toContain('"booking"');
|
||||
// "time" is the last surviving content word, so it gets the
|
||||
// prefix-match star - not "Lago", which is merely the last one this
|
||||
// test happens to name first.
|
||||
expect(result).toContain('"time"*');
|
||||
// Pure stop words, correctly dropped rather than OR-joined as noise that
|
||||
// would otherwise match almost every document in a mailbox.
|
||||
expect(result).not.toMatch(/"When"|"is"|"for"|"the"|"and"|"what"/i);
|
||||
});
|
||||
|
||||
it('falls back to the unfiltered text when every word is a stop word, rather than searching for nothing', () => {
|
||||
// "What is this" is 100% stop words - dropping all of them would leave
|
||||
// zero tokens (a null match, meaning "return everything" is wrong for a
|
||||
// question shaped like this); falling back to the original text at
|
||||
// least keeps a real, if weak, query.
|
||||
const result = toFtsMatchQueryAny('What is this');
|
||||
expect(result).not.toBeNull();
|
||||
});
|
||||
|
||||
it('still safely quotes FTS5 syntax characters even after stop-word filtering removes the surrounding noise', () => {
|
||||
// "OR"/"NEAR" themselves are common enough as English words that this
|
||||
// builder's stop-word list intentionally drops bare "or" (unlike
|
||||
// toFtsMatchQuery, which preserves it verbatim - see that test's own
|
||||
// comment on why: different concern, different guarantee). The safety
|
||||
// property that DOES still apply here is the one that matters for a
|
||||
// 500: whatever tokens survive filtering are always quoted before
|
||||
// reaching FTS5, so a stray `"`/`*`/`(` in real question text can never
|
||||
// raise a syntax error.
|
||||
const result = toFtsMatchQueryAny('a" NEAR(bar) baz*');
|
||||
expect(result).not.toBeNull();
|
||||
expect(result).toContain('"NEAR"');
|
||||
expect(result).toContain('"bar"');
|
||||
expect(result).toContain('"baz"');
|
||||
expect(result).not.toMatch(/fts5|syntax/i);
|
||||
});
|
||||
|
||||
it('returns null for input with no usable tokens', () => {
|
||||
expect(toFtsMatchQueryAny('')).toBeNull();
|
||||
expect(toFtsMatchQueryAny('***')).toBeNull();
|
||||
expect(toFtsMatchQueryAny(undefined as unknown as string)).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('paths', () => {
|
||||
const original = process.env[STORE_DIR_ENV];
|
||||
afterEach(() => {
|
||||
|
||||
+79
-24
@@ -331,8 +331,12 @@ export class MailIndex {
|
||||
types?: readonly ContentType[];
|
||||
limit?: number;
|
||||
snippetTokens?: number;
|
||||
/** 'and' (default): every token required - a deliberate search-box query.
|
||||
* 'any': stop words dropped, remaining tokens OR-joined, ranked by bm25 -
|
||||
* a natural-language question (see toFtsMatchQueryAny's docstring). */
|
||||
mode?: 'and' | 'any';
|
||||
}): SearchHit[] {
|
||||
const match = toFtsMatchQuery(opts.query);
|
||||
const match = opts.mode === 'any' ? toFtsMatchQueryAny(opts.query) : toFtsMatchQuery(opts.query);
|
||||
if (!match) return [];
|
||||
|
||||
const limit = Math.min(Math.max(opts.limit ?? 20, 1), 200);
|
||||
@@ -446,34 +450,85 @@ function safeParseObject(v: unknown): Record<string, unknown> {
|
||||
}
|
||||
|
||||
/**
|
||||
* Turns arbitrary user text into a safe FTS5 MATCH expression.
|
||||
*
|
||||
* FTS5's query syntax is not SQL, so parameter binding does NOT protect it: a
|
||||
* bare `"` or a stray `*`/`NEAR`/`:` in user input raises
|
||||
* `fts5: syntax error`, which would turn a normal search box into a 500. Every
|
||||
* token is quoted (making it a literal phrase) and a trailing `*` is added to
|
||||
* the last token so typing continues to match as the user types.
|
||||
* Splits and safely quotes raw text into FTS5-safe tokens, shared by both
|
||||
* query-builders below. Split on anything that isn't a word character or an
|
||||
* intra-word mark - keeps unicode letters (so "Müller" and "東京" survive)
|
||||
* via the u flag. Every token is quoted (making it a literal phrase) so a
|
||||
* bare `"` or a stray `*`/`NEAR`/`:` in user input can never raise FTS5's own
|
||||
* `fts5: syntax error` - that would turn a normal search into a 500.
|
||||
*/
|
||||
function quoteFtsTokens(raw: string): string[] {
|
||||
return raw
|
||||
.normalize('NFC')
|
||||
.split(/[^\p{L}\p{N}_@.'-]+/u)
|
||||
.map((t) => t.replace(/^['-]+|['-]+$/g, ''))
|
||||
.filter((t) => t.length > 0)
|
||||
.slice(0, 24)
|
||||
.map((t, i, all) => {
|
||||
const quoted = `"${t.replace(/"/g, '""')}"`;
|
||||
// Prefix-match only the final token, and only if it's long enough to not
|
||||
// match half the mailbox.
|
||||
return i === all.length - 1 && t.length >= 3 ? `${quoted}*` : quoted;
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Turns arbitrary user text into a safe FTS5 MATCH expression, every token
|
||||
* required (AND-joined). Right for a deliberate, short search-box query,
|
||||
* where requiring every word is what makes results precise as you type.
|
||||
*
|
||||
* Exported for unit testing - it is the one piece of this file with no
|
||||
* database dependency and the most ways to be wrong.
|
||||
*/
|
||||
export function toFtsMatchQuery(raw: string): string | null {
|
||||
if (typeof raw !== 'string') return null;
|
||||
// Split on anything that isn't a word character or an intra-word mark. Keeps
|
||||
// unicode letters (so "Müller" and "東京" survive) via the u flag.
|
||||
const tokens = raw
|
||||
.normalize('NFC')
|
||||
.split(/[^\p{L}\p{N}_@.'-]+/u)
|
||||
.map((t) => t.replace(/^['-]+|['-]+$/g, ''))
|
||||
.filter((t) => t.length > 0)
|
||||
.slice(0, 24);
|
||||
const tokens = quoteFtsTokens(raw);
|
||||
if (tokens.length === 0) return null;
|
||||
return tokens
|
||||
.map((t, i) => {
|
||||
const quoted = `"${t.replace(/"/g, '""')}"`;
|
||||
// Prefix-match only the final token, and only if it's long enough to not
|
||||
// match half the mailbox.
|
||||
return i === tokens.length - 1 && t.length >= 3 ? `${quoted}*` : quoted;
|
||||
})
|
||||
.join(' AND ');
|
||||
return tokens.join(' AND ');
|
||||
}
|
||||
|
||||
// A minimal, well-known set of English function words that carry no
|
||||
// retrieval signal - kept out of toFtsMatchQueryAny's OR expression so they
|
||||
// don't drown out the bm25 ranking's actual signal (see below). Deliberately
|
||||
// NOT applied inside quoteFtsTokens/toFtsMatchQuery: that function's own
|
||||
// tests rely on "AND"/"OR"/"NOT" surviving verbatim as literal search terms
|
||||
// (FTS5-keyword-injection safety) - a different concern from this one's job
|
||||
// of turning a natural-language QUESTION into a good search.
|
||||
const RETRIEVAL_STOP_WORDS = new Set([
|
||||
'a', 'an', 'the', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'am',
|
||||
'and', 'or', 'but', 'if', 'then', 'than', 'so', 'because',
|
||||
'for', 'of', 'to', 'in', 'on', 'at', 'by', 'with', 'from', 'as', 'about', 'into', 'over', 'after', 'before',
|
||||
'that', 'this', 'these', 'those', 'what', 'when', 'where', 'who', 'whom', 'which', 'why', 'how',
|
||||
'do', 'does', 'did', 'doing', 'done',
|
||||
'can', 'could', 'will', 'would', 'shall', 'should', 'may', 'might', 'must',
|
||||
'i', 'you', 'he', 'she', 'it', 'we', 'they', 'my', 'your', 'his', 'her', 'its', 'our', 'their', 'me', 'him', 'us', 'them',
|
||||
'not', 'no',
|
||||
]);
|
||||
|
||||
/**
|
||||
* Turns a natural-language QUESTION into a lenient FTS5 MATCH expression:
|
||||
* stop words dropped, remaining tokens OR-joined so bm25 ranks by how many
|
||||
* content words matched instead of requiring every one of them present.
|
||||
*
|
||||
* toFtsMatchQuery's strict AND is wrong for this shape of input: a real
|
||||
* question like "When is check-in for the Villa sul Lago booking?" shares
|
||||
* almost none of its own function words ("when"/"is"/"for"/"the") with the
|
||||
* document that actually answers it, so ANDing every token together returns
|
||||
* nothing - confirmed live: 0 hits for the full question, 2 correct hits for
|
||||
* the same index once reduced to "Villa sul Lago check-in". The one real
|
||||
* caller of `/api/offline/search?q=...` is exactly this AI-question shape
|
||||
* (see that route's own header - no manual search-box UI hits it today), so
|
||||
* this is the query builder that route now uses, not toFtsMatchQuery.
|
||||
*/
|
||||
export function toFtsMatchQueryAny(raw: string): string | null {
|
||||
if (typeof raw !== 'string') return null;
|
||||
const withoutStopWords = raw
|
||||
.split(/\s+/)
|
||||
.filter((w) => w.length > 0 && !RETRIEVAL_STOP_WORDS.has(w.toLowerCase().replace(/^[^\p{L}\p{N}]+|[^\p{L}\p{N}]+$/gu, '')))
|
||||
.join(' ');
|
||||
// Every word was a stop word (e.g. "What is this?") - fall back to the
|
||||
// original text rather than searching for literally nothing.
|
||||
const tokens = quoteFtsTokens(withoutStopWords.length > 0 ? withoutStopWords : raw);
|
||||
if (tokens.length === 0) return null;
|
||||
return tokens.join(' OR ');
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user