fix(mail-index): local AI retrieval returned 0 hits for real questions — AND-every-token FTS matching killed on stop words

Found by a new real Electron e2e test built specifically to prove the
`local` AI class genuinely works end-to-end in the packaged desktop shell:
a real local Ollama model answering a real question, grounded in the real
encrypted SQLite/FTS5 mail index — not a browser tab, not a mock.

First run surfaced a genuine bug: toFtsMatchQuery() AND-joins every token,
which is right for a deliberate search-box query but wrong for the natural-
language questions the AI retrieval surface (/api/offline/search — see its
own module header, "THE RETRIEVAL SURFACE") actually receives. "When is
check-in for the Villa sul Lago booking, and what time?" shares almost none
of its own function words with the email that answers it, so ANDing every
token — including "when"/"is"/"for"/"the"/"and"/"what" — returned 0 hits
against an index that correctly returns the right email for "Villa sul Lago
check-in".

Fix: new toFtsMatchQueryAny() (lib/mail-index/store.ts) — drops a small,
well-known English stop-word list, OR-joins what's left, and lets the
existing bm25 ranking pick the winner among partial matches. Deliberately a
NEW function, not a change to toFtsMatchQuery itself: that one's own tests
rely on "AND"/"OR"/"NOT" surviving verbatim as literal search terms
(FTS5-keyword-injection safety) — a different guarantee than this one's job
of turning a question into a good search. search() gains a `mode: 'and' |
'any'` option (default 'and', so every existing caller is unaffected); the
offline-search route passes 'any', since its one real caller is exactly
this AI-question shape.

Also added, to make the e2e test possible at all: electron/main.ts's
VNCMAIL_TEST_FIXED_PORT — a narrow, off-by-default escape hatch so
DEV_MOCK_JMAP's JMAP_SERVER_URL can point at this same standalone server's
own /api/dev-jmap. Needed because the encrypted index's key channel
(fd-3/safeStorage) only gets wired up in startStandaloneServer()'s own
random-port launch path, never when ELECTRON_LOAD_URL bypasses it for a
plain `next dev` target — so this was the only way to exercise the real
index without a full Stalwart+SMTP Docker fixture.

Verified live in the real packaged Electron shell, not just unit tests:
real dev-mode login, real multi-round /api/offline/sync + /api/offline/reindex
(39 mail/35 calendar/23 contacts indexed), real local-discovery banner
(11 real Ollama models on this machine), real "Connect", a real question
through the real Settings UI, a real direct renderer->Ollama /api/chat call
(confirmed via network log, never proxied through this app's backend), and
the model's own answer citing the exact right fact: "Saturday 28 March at
15:00" — a fact that exists nowhere except in the one indexed email.

4 new unit tests for toFtsMatchQueryAny. Full gate: tsc clean, eslint
clean, 2502/2502 tests passing, build clean, e2e/electron-ai-local-index.spec.ts
passing against the real standalone server + real Electron + real Ollama.
This commit is contained in:
Bernd Rodler
2026-08-06 18:09:14 +02:00
parent 2a8778c905
commit 7b2047681e
6 changed files with 369 additions and 27 deletions
+12 -1
View File
@@ -173,7 +173,18 @@ async function startStandaloneServer(): Promise<string> {
);
}
const port = await getFreePort();
// Normally a random free port, chosen fresh every launch - JMAP_SERVER_URL
// never needs to reference it back (a real deployment's Stalwart lives at
// its own fixed address). VNCMAIL_TEST_FIXED_PORT is a narrow escape hatch
// for e2e tests that DO need to know the port ahead of time - specifically
// to point DEV_MOCK_JMAP's JMAP_SERVER_URL at this same standalone server's
// own /api/dev-jmap route, which is the only way to exercise the real
// encrypted offline index (lib/mail-index/**) without a real Stalwart
// fixture: that index's key channel only gets wired up in this function,
// never when ELECTRON_LOAD_URL bypasses it for a plain `next dev` target.
// Unset in every normal launch, so this changes nothing outside a test run.
const fixedPort = process.env.VNCMAIL_TEST_FIXED_PORT ? Number(process.env.VNCMAIL_TEST_FIXED_PORT) : null;
const port = fixedPort && Number.isInteger(fixedPort) ? fixedPort : await getFreePort();
const url = `http://127.0.0.1:${port}`;
const storeDir = getIndexStoreDir();