87336981d3885bff37eb7cb099aa250a05f63d44
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Bernd Rodler
|
87336981d3 |
feat(mail-index): 1-year retention by default + a recency retrieval leg
The two things that made real questions fail against a correctly-populated index, both fixed at the root. RETENTION (A1). `INDEX_WINDOW_DAYS = 30` was not merely a fetch bound — catch-up also PRUNED mail older than it, so "summarise everything from July" was unanswerable in August because the rows had been deleted, while the UI said only that nothing matched. Now a user-visible setting (Settings → About & Data): 30 days / 3 months / 1 year / everything, defaulting to 1 YEAR per the product owner. The window bounds the fetch AND the prune from one value so the two can never disagree and delete what was just written; "everything" skips pruning entirely rather than falling back to some default bound. The per-pass ceiling scales with the window (500/30d, hard cap 20k) because 500 messages is right for a month and nonsense for "everything". Email/query now omits the `after` filter entirely when unbounded — Stalwart rejects a malformed filter rather than treating `undefined` as unset. RECENCY (A2). Keyword search structurally cannot answer a question about WHEN: bm25 ranks by term overlap, so "who sent the last email" matches documents containing the word "last", and "all mails in July" matches documents containing "July" — not documents dated in July. Both were asked by a real user and both failed. New lib/mail-index/recency.ts detects time intent (English + German, since the UI ships German) and turns it into a date RANGE; new MailIndex.recent() answers it with an ordered scan over the already-indexed `occurred_at`. The route ADDS these hits to the keyword hits rather than replacing them — "what did the last mail from Anna say" is both kinds of question at once. Timezone subtlety worth knowing: bounds are built from LOCAL calendar boundaries and serialised as UTC instants, so "July" covers the user's July. A mail at 00:30 local on 1 July belongs to it even though its stored UTC timestamp reads 30 June. My first test asserted the ISO string prefix, which would have enshrined the opposite and passed only in UTC — the tests now assert the local-time property instead. SCOPE, stated by the product owner and now enforced structurally: the assistant only ever sees the mailbox the user is signed in to. Both retrieval legs resolve the active account (local leg by cookie slot, server leg by the session's own JMAP account); there is deliberately no fan-out across connected or shared mailboxes, and adding one would be a policy change, not a feature. Gate: tsc clean, eslint clean, 2520/2520 tests (8 new for recency intent), build clean. |
||
|
Bernd Rodler
|
7b2047681e |
fix(mail-index): local AI retrieval returned 0 hits for real questions — AND-every-token FTS matching killed on stop words
Found by a new real Electron e2e test built specifically to prove the `local` AI class genuinely works end-to-end in the packaged desktop shell: a real local Ollama model answering a real question, grounded in the real encrypted SQLite/FTS5 mail index — not a browser tab, not a mock. First run surfaced a genuine bug: toFtsMatchQuery() AND-joins every token, which is right for a deliberate search-box query but wrong for the natural- language questions the AI retrieval surface (/api/offline/search — see its own module header, "THE RETRIEVAL SURFACE") actually receives. "When is check-in for the Villa sul Lago booking, and what time?" shares almost none of its own function words with the email that answers it, so ANDing every token — including "when"/"is"/"for"/"the"/"and"/"what" — returned 0 hits against an index that correctly returns the right email for "Villa sul Lago check-in". Fix: new toFtsMatchQueryAny() (lib/mail-index/store.ts) — drops a small, well-known English stop-word list, OR-joins what's left, and lets the existing bm25 ranking pick the winner among partial matches. Deliberately a NEW function, not a change to toFtsMatchQuery itself: that one's own tests rely on "AND"/"OR"/"NOT" surviving verbatim as literal search terms (FTS5-keyword-injection safety) — a different guarantee than this one's job of turning a question into a good search. search() gains a `mode: 'and' | 'any'` option (default 'and', so every existing caller is unaffected); the offline-search route passes 'any', since its one real caller is exactly this AI-question shape. Also added, to make the e2e test possible at all: electron/main.ts's VNCMAIL_TEST_FIXED_PORT — a narrow, off-by-default escape hatch so DEV_MOCK_JMAP's JMAP_SERVER_URL can point at this same standalone server's own /api/dev-jmap. Needed because the encrypted index's key channel (fd-3/safeStorage) only gets wired up in startStandaloneServer()'s own random-port launch path, never when ELECTRON_LOAD_URL bypasses it for a plain `next dev` target — so this was the only way to exercise the real index without a full Stalwart+SMTP Docker fixture. Verified live in the real packaged Electron shell, not just unit tests: real dev-mode login, real multi-round /api/offline/sync + /api/offline/reindex (39 mail/35 calendar/23 contacts indexed), real local-discovery banner (11 real Ollama models on this machine), real "Connect", a real question through the real Settings UI, a real direct renderer->Ollama /api/chat call (confirmed via network log, never proxied through this app's backend), and the model's own answer citing the exact right fact: "Saturday 28 March at 15:00" — a fact that exists nowhere except in the one indexed email. 4 new unit tests for toFtsMatchQueryAny. Full gate: tsc clean, eslint clean, 2502/2502 tests passing, build clean, e2e/electron-ai-local-index.spec.ts passing against the real standalone server + real Electron + real Ollama. |
||
|
Bernd RodlerandClaude Sonnet 5
|
bde8455df5 |
fix(mail-index): close handle on every pragma failure, reconcile contact/file deletes
QA pass on the encrypted mail index found two real gaps beyond what the
prior end-to-end fix pass caught:
1. store.ts's MailIndex.open() only wrapped SOME of the post-key pragma
calls in a try/catch before this: the first attempt's `key` pragma and
assertEncrypted() ran outside any try at all, and the wrong-key retry
repeated the same gap. Any pragma throwing there (SQLITE_BUSY, a full
disk on the first WAL write) leaked the native SQLite handle instead of
closing it. Factored the open+key+verify+pragma sequence into openKeyed(),
which guarantees a close before rethrowing on any failure, and reused it
for both the first attempt and the retry.
2. reindex.ts never removed a deleted contact or file from the index. The
`removed` field exists in the API and is fully tested at the store layer,
but nothing in the renderer populates it, so a deleted contact/file stayed
searchable - and retrievable by the AI feature - indefinitely. Mail and
calendar can't use the same fix (their queries are date-windowed, so an id
missing from one fetch may just be outside the window), but contacts/files
have no date filter - a catch-up fetch that comes back under its cap IS
the complete set, so anything locally indexed but absent from it is safely
known to be deleted. Added strayIdsAfterCatchUp() and wired it into the
catch-up path for those two types only.
Also read binding.ts, key.ts, paths.ts, jmap.ts, extract.ts, the FTS5
query builder, and both /api/offline/{search,reindex} routes end to end;
no other concrete bugs found there. Full findings reported separately.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
||
|
Bernd RodlerandClaude Sonnet 5
|
7e9aefcfa1 |
test(mail-index): unit tests for the extractors, FTS query builder and store
48 assertions. The pure extractors and toFtsMatchQuery need no database; the
store tests run against REAL SQLCipher and skip themselves when the optional
native binding is absent (e.g. Alpine/musl), which is the same guard the
runtime uses.
The two that matter most:
* "writes an ENCRYPTED file" reads the raw bytes back and asserts a canary
string is absent. This is the assertion that catches `PRAGMA key` silently
doing nothing - a plain-SQLite binding leaves the mailbox in cleartext with
no error anywhere, so a functional test alone would pass.
* "upserting the same id REPLACES the FTS row" - the FTS table is maintained by
hand (standalone, not external-content), so a missed delete leaves the OLD
body permanently searchable. The test asserts the old text stops matching,
not just that the new text starts.
Also covered: FTS5 MATCH injection (its grammar is not protected by SQL
parameter binding, so a bare quote would 500 the search route), account-scoped
keys not merging two accounts' identical JMAP ids, title-over-body bm25
weighting, and the hosted-deployment env gate rejecting a relative path.
Note: lib/__tests__/builtin-themes.test.ts has 2 pre-existing failures on this
branch (theme author "VNC" vs. expected "Built-in", from the earlier rebrand) -
verified failing identically at
|