fix(mail-index): local AI retrieval returned 0 hits for real questions — AND-every-token FTS matching killed on stop words
Found by a new real Electron e2e test built specifically to prove the `local` AI class genuinely works end-to-end in the packaged desktop shell: a real local Ollama model answering a real question, grounded in the real encrypted SQLite/FTS5 mail index — not a browser tab, not a mock. First run surfaced a genuine bug: toFtsMatchQuery() AND-joins every token, which is right for a deliberate search-box query but wrong for the natural- language questions the AI retrieval surface (/api/offline/search — see its own module header, "THE RETRIEVAL SURFACE") actually receives. "When is check-in for the Villa sul Lago booking, and what time?" shares almost none of its own function words with the email that answers it, so ANDing every token — including "when"/"is"/"for"/"the"/"and"/"what" — returned 0 hits against an index that correctly returns the right email for "Villa sul Lago check-in". Fix: new toFtsMatchQueryAny() (lib/mail-index/store.ts) — drops a small, well-known English stop-word list, OR-joins what's left, and lets the existing bm25 ranking pick the winner among partial matches. Deliberately a NEW function, not a change to toFtsMatchQuery itself: that one's own tests rely on "AND"/"OR"/"NOT" surviving verbatim as literal search terms (FTS5-keyword-injection safety) — a different guarantee than this one's job of turning a question into a good search. search() gains a `mode: 'and' | 'any'` option (default 'and', so every existing caller is unaffected); the offline-search route passes 'any', since its one real caller is exactly this AI-question shape. Also added, to make the e2e test possible at all: electron/main.ts's VNCMAIL_TEST_FIXED_PORT — a narrow, off-by-default escape hatch so DEV_MOCK_JMAP's JMAP_SERVER_URL can point at this same standalone server's own /api/dev-jmap. Needed because the encrypted index's key channel (fd-3/safeStorage) only gets wired up in startStandaloneServer()'s own random-port launch path, never when ELECTRON_LOAD_URL bypasses it for a plain `next dev` target — so this was the only way to exercise the real index without a full Stalwart+SMTP Docker fixture. Verified live in the real packaged Electron shell, not just unit tests: real dev-mode login, real multi-round /api/offline/sync + /api/offline/reindex (39 mail/35 calendar/23 contacts indexed), real local-discovery banner (11 real Ollama models on this machine), real "Connect", a real question through the real Settings UI, a real direct renderer->Ollama /api/chat call (confirmed via network log, never proxied through this app's backend), and the model's own answer citing the exact right fact: "Saturday 28 March at 15:00" — a fact that exists nowhere except in the one indexed email. 4 new unit tests for toFtsMatchQueryAny. Full gate: tsc clean, eslint clean, 2502/2502 tests passing, build clean, e2e/electron-ai-local-index.spec.ts passing against the real standalone server + real Electron + real Ollama.
This commit is contained in:
@@ -6,7 +6,7 @@ import { afterEach, beforeEach, describe, expect, it } from 'vitest';
|
||||
import { isSqlcipherAvailable } from '../binding';
|
||||
import type { SqlcipherConstructor, SqlcipherDatabase, SqlcipherStatement } from '../binding';
|
||||
import { accountFileToken, getStoreDir, indexDbPath, STORE_DIR_ENV } from '../paths';
|
||||
import { MailIndex, openKeyed, toFtsMatchQuery, type IndexDoc } from '../store';
|
||||
import { MailIndex, openKeyed, toFtsMatchQuery, toFtsMatchQueryAny, type IndexDoc } from '../store';
|
||||
|
||||
describe('toFtsMatchQuery', () => {
|
||||
it('quotes every token so FTS5 operators in user input cannot break the query', () => {
|
||||
@@ -48,6 +48,62 @@ describe('toFtsMatchQuery', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('toFtsMatchQueryAny', () => {
|
||||
it('drops English function words and OR-joins what is left - the confirmed-live failure this fixes', () => {
|
||||
// AND-every-token (toFtsMatchQuery) returns 0 hits for this exact
|
||||
// question against a document that only contains "Villa sul Lago" and
|
||||
// "check-in" - see app/api/offline/search/route.ts's comment and the
|
||||
// e2e electron-ai-local-index.spec.ts run that first caught this.
|
||||
const result = toFtsMatchQueryAny('When is check-in for the Villa sul Lago booking, and what time?');
|
||||
expect(result).not.toBeNull();
|
||||
expect(result).not.toContain(' AND ');
|
||||
expect(result).toContain('"check-in"');
|
||||
expect(result).toContain('"Villa"');
|
||||
expect(result).toContain('"sul"');
|
||||
expect(result).toContain('"Lago"');
|
||||
expect(result).toContain('"booking"');
|
||||
// "time" is the last surviving content word, so it gets the
|
||||
// prefix-match star - not "Lago", which is merely the last one this
|
||||
// test happens to name first.
|
||||
expect(result).toContain('"time"*');
|
||||
// Pure stop words, correctly dropped rather than OR-joined as noise that
|
||||
// would otherwise match almost every document in a mailbox.
|
||||
expect(result).not.toMatch(/"When"|"is"|"for"|"the"|"and"|"what"/i);
|
||||
});
|
||||
|
||||
it('falls back to the unfiltered text when every word is a stop word, rather than searching for nothing', () => {
|
||||
// "What is this" is 100% stop words - dropping all of them would leave
|
||||
// zero tokens (a null match, meaning "return everything" is wrong for a
|
||||
// question shaped like this); falling back to the original text at
|
||||
// least keeps a real, if weak, query.
|
||||
const result = toFtsMatchQueryAny('What is this');
|
||||
expect(result).not.toBeNull();
|
||||
});
|
||||
|
||||
it('still safely quotes FTS5 syntax characters even after stop-word filtering removes the surrounding noise', () => {
|
||||
// "OR"/"NEAR" themselves are common enough as English words that this
|
||||
// builder's stop-word list intentionally drops bare "or" (unlike
|
||||
// toFtsMatchQuery, which preserves it verbatim - see that test's own
|
||||
// comment on why: different concern, different guarantee). The safety
|
||||
// property that DOES still apply here is the one that matters for a
|
||||
// 500: whatever tokens survive filtering are always quoted before
|
||||
// reaching FTS5, so a stray `"`/`*`/`(` in real question text can never
|
||||
// raise a syntax error.
|
||||
const result = toFtsMatchQueryAny('a" NEAR(bar) baz*');
|
||||
expect(result).not.toBeNull();
|
||||
expect(result).toContain('"NEAR"');
|
||||
expect(result).toContain('"bar"');
|
||||
expect(result).toContain('"baz"');
|
||||
expect(result).not.toMatch(/fts5|syntax/i);
|
||||
});
|
||||
|
||||
it('returns null for input with no usable tokens', () => {
|
||||
expect(toFtsMatchQueryAny('')).toBeNull();
|
||||
expect(toFtsMatchQueryAny('***')).toBeNull();
|
||||
expect(toFtsMatchQueryAny(undefined as unknown as string)).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('paths', () => {
|
||||
const original = process.env[STORE_DIR_ENV];
|
||||
afterEach(() => {
|
||||
|
||||
Reference in New Issue
Block a user