Found by a new real Electron e2e test built specifically to prove the
`local` AI class genuinely works end-to-end in the packaged desktop shell:
a real local Ollama model answering a real question, grounded in the real
encrypted SQLite/FTS5 mail index — not a browser tab, not a mock.
First run surfaced a genuine bug: toFtsMatchQuery() AND-joins every token,
which is right for a deliberate search-box query but wrong for the natural-
language questions the AI retrieval surface (/api/offline/search — see its
own module header, "THE RETRIEVAL SURFACE") actually receives. "When is
check-in for the Villa sul Lago booking, and what time?" shares almost none
of its own function words with the email that answers it, so ANDing every
token — including "when"/"is"/"for"/"the"/"and"/"what" — returned 0 hits
against an index that correctly returns the right email for "Villa sul Lago
check-in".
Fix: new toFtsMatchQueryAny() (lib/mail-index/store.ts) — drops a small,
well-known English stop-word list, OR-joins what's left, and lets the
existing bm25 ranking pick the winner among partial matches. Deliberately a
NEW function, not a change to toFtsMatchQuery itself: that one's own tests
rely on "AND"/"OR"/"NOT" surviving verbatim as literal search terms
(FTS5-keyword-injection safety) — a different guarantee than this one's job
of turning a question into a good search. search() gains a `mode: 'and' |
'any'` option (default 'and', so every existing caller is unaffected); the
offline-search route passes 'any', since its one real caller is exactly
this AI-question shape.
Also added, to make the e2e test possible at all: electron/main.ts's
VNCMAIL_TEST_FIXED_PORT — a narrow, off-by-default escape hatch so
DEV_MOCK_JMAP's JMAP_SERVER_URL can point at this same standalone server's
own /api/dev-jmap. Needed because the encrypted index's key channel
(fd-3/safeStorage) only gets wired up in startStandaloneServer()'s own
random-port launch path, never when ELECTRON_LOAD_URL bypasses it for a
plain `next dev` target — so this was the only way to exercise the real
index without a full Stalwart+SMTP Docker fixture.
Verified live in the real packaged Electron shell, not just unit tests:
real dev-mode login, real multi-round /api/offline/sync + /api/offline/reindex
(39 mail/35 calendar/23 contacts indexed), real local-discovery banner
(11 real Ollama models on this machine), real "Connect", a real question
through the real Settings UI, a real direct renderer->Ollama /api/chat call
(confirmed via network log, never proxied through this app's backend), and
the model's own answer citing the exact right fact: "Saturday 28 March at
15:00" — a fact that exists nowhere except in the one indexed email.
4 new unit tests for toFtsMatchQueryAny. Full gate: tsc clean, eslint
clean, 2502/2502 tests passing, build clean, e2e/electron-ai-local-index.spec.ts
passing against the real standalone server + real Electron + real Ollama.
QA pass on the encrypted mail index found two real gaps beyond what the
prior end-to-end fix pass caught:
1. store.ts's MailIndex.open() only wrapped SOME of the post-key pragma
calls in a try/catch before this: the first attempt's `key` pragma and
assertEncrypted() ran outside any try at all, and the wrong-key retry
repeated the same gap. Any pragma throwing there (SQLITE_BUSY, a full
disk on the first WAL write) leaked the native SQLite handle instead of
closing it. Factored the open+key+verify+pragma sequence into openKeyed(),
which guarantees a close before rethrowing on any failure, and reused it
for both the first attempt and the retry.
2. reindex.ts never removed a deleted contact or file from the index. The
`removed` field exists in the API and is fully tested at the store layer,
but nothing in the renderer populates it, so a deleted contact/file stayed
searchable - and retrievable by the AI feature - indefinitely. Mail and
calendar can't use the same fix (their queries are date-windowed, so an id
missing from one fetch may just be outside the window), but contacts/files
have no date filter - a catch-up fetch that comes back under its cap IS
the complete set, so anything locally indexed but absent from it is safely
known to be deleted. Added strayIdsAfterCatchUp() and wired it into the
catch-up path for those two types only.
Also read binding.ts, key.ts, paths.ts, jmap.ts, extract.ts, the FTS5
query builder, and both /api/offline/{search,reindex} routes end to end;
no other concrete bugs found there. Full findings reported separately.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
48 assertions. The pure extractors and toFtsMatchQuery need no database; the
store tests run against REAL SQLCipher and skip themselves when the optional
native binding is absent (e.g. Alpine/musl), which is the same guard the
runtime uses.
The two that matter most:
* "writes an ENCRYPTED file" reads the raw bytes back and asserts a canary
string is absent. This is the assertion that catches `PRAGMA key` silently
doing nothing - a plain-SQLite binding leaves the mailbox in cleartext with
no error anywhere, so a functional test alone would pass.
* "upserting the same id REPLACES the FTS row" - the FTS table is maintained by
hand (standalone, not external-content), so a missed delete leaves the OLD
body permanently searchable. The test asserts the old text stops matching,
not just that the new text starts.
Also covered: FTS5 MATCH injection (its grammar is not protected by SQL
parameter binding, so a bare quote would 500 the search route), account-scoped
keys not merging two accounts' identical JMAP ids, title-over-body bm25
weighting, and the hosted-deployment env gate rejecting a relative path.
Note: lib/__tests__/builtin-themes.test.ts has 2 pre-existing failures on this
branch (theme author "VNC" vs. expected "Built-in", from the earlier rebrand) -
verified failing identically at b15098a6, before any of this work.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Bernd Rodler