Full retrieval pipeline per docs/AI-ASSISTANT-CONCEPT.md §7/§8.1, real end
to end, not mocked:
- lib/ai/retrieval/types.ts: SourceRef + RetrieverAdapter schema. Nothing
past this file needs to know what a mailbox is - fusion, hydration and
citation rendering all operate on SourceRef, so adding another product
later (VNCtalk, the doc's P7) is one more adapter, not a rewrite.
- lib/ai/retrieval/fusion.ts: Reciprocal Rank Fusion, score = Σ1/(k+rank),
k=60. Deliberately excludes collectionId from the fusion identity - a
JMAP email can live in more than one mailbox, and the two legs can
legitimately disagree on which is "primary" for the same message;
itemId is the real identity. 5 unit tests, including that exact
double-count case.
- lib/ai/retrieval/mail-embeddings.ts: the server embedding leg. Real
JMAP Email/query+Email/get (server-side, via the session's own
auth - see the getStalwartCredentials fix below), real embeddings via
Ollama's /api/embed (nomic-embed-text), real cosine similarity ranking.
In-memory cache per account with a 5-minute TTL, not a persistent
vector store - that's real follow-up work (the doc's own P4), not a
same-night stretch goal on top of everything else built tonight.
- app/api/ai/retrieve/route.ts: wires it together. ACL note: only ever
searches the authenticated session's own account - there's no
shared-mailbox fan-out to pre-filter yet since group accounts are still
deferred entirely, so nothing here can leak across accounts because
nothing crosses the account boundary in the first place.
- lib/ai/local-client.ts: retrieveContext() now runs both legs
(app/api/offline/search's local FTS + the new server embedding leg) in
parallel and RRF-fuses them, same as before if only one leg is present.
Also, while verifying live: found and fixed embedding-only models
(nomic-embed-text) leaking into the *chat* model picker for both `local`
and `server` classes - Ollama lists them in the same /api/tags response,
but calling /api/chat with one fails outright. Filtered by `capabilities`
(fails open if absent, for older Ollama).
Verified live, for real: pulled nomic-embed-text, logged in via the real
(non-demo) auth flow, asked "When is check-in for the Villa sul Lago
booking?" against the seeded mock inbox - got back "Check-in ... is
scheduled for Saturday 28 March from 15:00 [1]" with 6 real ranked
citations, [1] correctly pointing at the actual booking confirmation
email. Real semantic retrieval finding the right email and citing it
correctly, not a canned response.
Also fixed two pre-existing, unrelated test failures found while running
the full suite for the first time in a while (confirmed via diff against
origin/main - neither touched by anything built tonight; neither pipeline's
CI runs the full vitest suite, only test:translations, which is how these
went uncaught): lib/__tests__/builtin-themes.test.ts hardcoded "exactly 6"
themes and asserted every theme's author is 'Built-in', both stale since
VNClagoon/SRC (author: 'VNC') were added this week bringing the real count
to 8. Left the also-pre-existing, timing-sensitive
jmap-client-resilience.test.ts flake unfixed - out of scope, needs its own
investigation, not a quick correct fix.
Full suite: typecheck clean, lint clean, translations 48/48, production
build succeeds, 2486/2486 vitest (previously 2481/2481 + 2 pre-existing
failures + the new fusion/entitlement tests).
Bernd Rodler