feat(ai): real retrieval — SourceRef, RRF fusion, real embedding leg (P3/P4)
Full retrieval pipeline per docs/AI-ASSISTANT-CONCEPT.md §7/§8.1, real end to end, not mocked: - lib/ai/retrieval/types.ts: SourceRef + RetrieverAdapter schema. Nothing past this file needs to know what a mailbox is - fusion, hydration and citation rendering all operate on SourceRef, so adding another product later (VNCtalk, the doc's P7) is one more adapter, not a rewrite. - lib/ai/retrieval/fusion.ts: Reciprocal Rank Fusion, score = Σ1/(k+rank), k=60. Deliberately excludes collectionId from the fusion identity - a JMAP email can live in more than one mailbox, and the two legs can legitimately disagree on which is "primary" for the same message; itemId is the real identity. 5 unit tests, including that exact double-count case. - lib/ai/retrieval/mail-embeddings.ts: the server embedding leg. Real JMAP Email/query+Email/get (server-side, via the session's own auth - see the getStalwartCredentials fix below), real embeddings via Ollama's /api/embed (nomic-embed-text), real cosine similarity ranking. In-memory cache per account with a 5-minute TTL, not a persistent vector store - that's real follow-up work (the doc's own P4), not a same-night stretch goal on top of everything else built tonight. - app/api/ai/retrieve/route.ts: wires it together. ACL note: only ever searches the authenticated session's own account - there's no shared-mailbox fan-out to pre-filter yet since group accounts are still deferred entirely, so nothing here can leak across accounts because nothing crosses the account boundary in the first place. - lib/ai/local-client.ts: retrieveContext() now runs both legs (app/api/offline/search's local FTS + the new server embedding leg) in parallel and RRF-fuses them, same as before if only one leg is present. Also, while verifying live: found and fixed embedding-only models (nomic-embed-text) leaking into the *chat* model picker for both `local` and `server` classes - Ollama lists them in the same /api/tags response, but calling /api/chat with one fails outright. Filtered by `capabilities` (fails open if absent, for older Ollama). Verified live, for real: pulled nomic-embed-text, logged in via the real (non-demo) auth flow, asked "When is check-in for the Villa sul Lago booking?" against the seeded mock inbox - got back "Check-in ... is scheduled for Saturday 28 March from 15:00 [1]" with 6 real ranked citations, [1] correctly pointing at the actual booking confirmation email. Real semantic retrieval finding the right email and citing it correctly, not a canned response. Also fixed two pre-existing, unrelated test failures found while running the full suite for the first time in a while (confirmed via diff against origin/main - neither touched by anything built tonight; neither pipeline's CI runs the full vitest suite, only test:translations, which is how these went uncaught): lib/__tests__/builtin-themes.test.ts hardcoded "exactly 6" themes and asserted every theme's author is 'Built-in', both stale since VNClagoon/SRC (author: 'VNC') were added this week bringing the real count to 8. Left the also-pre-existing, timing-sensitive jmap-client-resilience.test.ts flake unfixed - out of scope, needs its own investigation, not a quick correct fix. Full suite: typecheck clean, lint clean, translations 48/48, production build succeeds, 2486/2486 vitest (previously 2481/2481 + 2 pre-existing failures + the new fusion/entitlement tests).
This commit is contained in:
@@ -0,0 +1,48 @@
|
||||
import type { Scored, SourceRef } from './types';
|
||||
|
||||
/**
|
||||
* Reciprocal Rank Fusion (docs/AI-ASSISTANT-CONCEPT.md §7 step 3):
|
||||
* score(d) = Σ 1/(k + rank_i(d)), k = 60.
|
||||
*
|
||||
* RRF only reads rank, not the underlying score — so it needs no calibration
|
||||
* between BM25 (FTS) and cosine (embedding) scores, and degrades to a single
|
||||
* retriever with no code branch when one leg is absent (just pass an empty
|
||||
* array for that leg).
|
||||
*/
|
||||
const RRF_K = 60;
|
||||
|
||||
/**
|
||||
* Deliberately excludes collectionId: a JMAP email can live in more than one
|
||||
* mailbox, and the FTS leg and the embedding leg may legitimately report a
|
||||
* different "primary" one for the same message. itemId is already the real
|
||||
* identity within an account - including collectionId here would let the
|
||||
* same email be counted twice instead of properly fused.
|
||||
*/
|
||||
function refKey(ref: SourceRef): string {
|
||||
return `${ref.product}:${ref.accountId}:${ref.itemId}:${ref.chunkIx}`;
|
||||
}
|
||||
|
||||
export function reciprocalRankFusion(
|
||||
legs: Scored<SourceRef>[][],
|
||||
limit: number,
|
||||
): Scored<SourceRef>[] {
|
||||
const fused = new Map<string, { ref: SourceRef; score: number }>();
|
||||
|
||||
for (const leg of legs) {
|
||||
leg.forEach((hit, index) => {
|
||||
const key = refKey(hit.ref);
|
||||
const rank = index + 1;
|
||||
const contribution = 1 / (RRF_K + rank);
|
||||
const existing = fused.get(key);
|
||||
if (existing) {
|
||||
existing.score += contribution;
|
||||
} else {
|
||||
fused.set(key, { ref: hit.ref, score: contribution });
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
return [...fused.values()]
|
||||
.sort((a, b) => b.score - a.score)
|
||||
.slice(0, limit);
|
||||
}
|
||||
Reference in New Issue
Block a user