fix(mail-index): local AI retrieval returned 0 hits for real questions — AND-every-token FTS matching killed on stop words
Found by a new real Electron e2e test built specifically to prove the `local` AI class genuinely works end-to-end in the packaged desktop shell: a real local Ollama model answering a real question, grounded in the real encrypted SQLite/FTS5 mail index — not a browser tab, not a mock. First run surfaced a genuine bug: toFtsMatchQuery() AND-joins every token, which is right for a deliberate search-box query but wrong for the natural- language questions the AI retrieval surface (/api/offline/search — see its own module header, "THE RETRIEVAL SURFACE") actually receives. "When is check-in for the Villa sul Lago booking, and what time?" shares almost none of its own function words with the email that answers it, so ANDing every token — including "when"/"is"/"for"/"the"/"and"/"what" — returned 0 hits against an index that correctly returns the right email for "Villa sul Lago check-in". Fix: new toFtsMatchQueryAny() (lib/mail-index/store.ts) — drops a small, well-known English stop-word list, OR-joins what's left, and lets the existing bm25 ranking pick the winner among partial matches. Deliberately a NEW function, not a change to toFtsMatchQuery itself: that one's own tests rely on "AND"/"OR"/"NOT" surviving verbatim as literal search terms (FTS5-keyword-injection safety) — a different guarantee than this one's job of turning a question into a good search. search() gains a `mode: 'and' | 'any'` option (default 'and', so every existing caller is unaffected); the offline-search route passes 'any', since its one real caller is exactly this AI-question shape. Also added, to make the e2e test possible at all: electron/main.ts's VNCMAIL_TEST_FIXED_PORT — a narrow, off-by-default escape hatch so DEV_MOCK_JMAP's JMAP_SERVER_URL can point at this same standalone server's own /api/dev-jmap. Needed because the encrypted index's key channel (fd-3/safeStorage) only gets wired up in startStandaloneServer()'s own random-port launch path, never when ELECTRON_LOAD_URL bypasses it for a plain `next dev` target — so this was the only way to exercise the real index without a full Stalwart+SMTP Docker fixture. Verified live in the real packaged Electron shell, not just unit tests: real dev-mode login, real multi-round /api/offline/sync + /api/offline/reindex (39 mail/35 calendar/23 contacts indexed), real local-discovery banner (11 real Ollama models on this machine), real "Connect", a real question through the real Settings UI, a real direct renderer->Ollama /api/chat call (confirmed via network log, never proxied through this app's backend), and the model's own answer citing the exact right fact: "Saturday 28 March at 15:00" — a fact that exists nowhere except in the one indexed email. 4 new unit tests for toFtsMatchQueryAny. Full gate: tsc clean, eslint clean, 2502/2502 tests passing, build clean, e2e/electron-ai-local-index.spec.ts passing against the real standalone server + real Electron + real Ollama.
This commit is contained in:
@@ -79,7 +79,12 @@ export async function GET(request: NextRequest) {
|
|||||||
try {
|
try {
|
||||||
const stats = index.stats();
|
const stats = index.stats();
|
||||||
if (!query) return { hits: [] as SearchHit[], stats };
|
if (!query) return { hits: [] as SearchHit[], stats };
|
||||||
return { hits: index.search({ query, types, limit }), stats: wantStats ? stats : undefined };
|
// 'any': this route is the AI/RAG retrieval surface (see module
|
||||||
|
// header) - its one real caller sends natural-language questions,
|
||||||
|
// not deliberate search-box keywords, so strict AND-every-token
|
||||||
|
// matching (the default) drops nearly all of them. See
|
||||||
|
// toFtsMatchQueryAny's docstring for the confirmed-live failure.
|
||||||
|
return { hits: index.search({ query, types, limit, mode: 'any' }), stats: wantStats ? stats : undefined };
|
||||||
} finally {
|
} finally {
|
||||||
index.close();
|
index.close();
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,199 @@
|
|||||||
|
import { test, expect, _electron as electron } from '@playwright/test';
|
||||||
|
import type { ElectronApplication, Page } from '@playwright/test';
|
||||||
|
import fs from 'node:fs';
|
||||||
|
import os from 'node:os';
|
||||||
|
import path from 'node:path';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Proves the two hardest-to-fake claims about the AI Assistant's `local`
|
||||||
|
* class in the REAL packaged desktop shell, not a browser tab:
|
||||||
|
*
|
||||||
|
* 1. The LLM genuinely runs locally — a direct browser-side fetch to
|
||||||
|
* this machine's own Ollama (127.0.0.1:11434), never proxied through
|
||||||
|
* this app's backend.
|
||||||
|
* 2. It is genuinely grounded in the ENCRYPTED LOCAL SQLITE/FTS5 MAIL
|
||||||
|
* INDEX (lib/mail-index/**), not the separate real-JMAP-embeddings
|
||||||
|
* server leg (lib/ai/retrieval/mail-embeddings.ts) — AI_SERVER_BASE_URL
|
||||||
|
* is deliberately left UNSET here so only the local FTS leg can
|
||||||
|
* supply retrieval context. If this test passes, the local index
|
||||||
|
* leg is the only possible source of the grounded answer.
|
||||||
|
*
|
||||||
|
* Needs a real launch through electron/main.ts's startStandaloneServer(),
|
||||||
|
* not ELECTRON_LOAD_URL — that's the only code path that wires up the
|
||||||
|
* fd-3 key channel / safeStorage the encrypted index depends on (see
|
||||||
|
* integration/tests/12-electron-mail-index.spec.ts's header for the full
|
||||||
|
* reasoning). That function picks a random free port every launch, which
|
||||||
|
* would make it impossible to also point DEV_MOCK_JMAP's JMAP_SERVER_URL
|
||||||
|
* at this same server's own /api/dev-jmap route — hence
|
||||||
|
* VNCMAIL_TEST_FIXED_PORT, a narrow, off-by-default escape hatch added to
|
||||||
|
* electron/main.ts specifically to make this test possible without a real
|
||||||
|
* Stalwart fixture.
|
||||||
|
*
|
||||||
|
* Requires a real Ollama already running on this machine with at least one
|
||||||
|
* completion-capable model installed — skips (not fails) otherwise, since
|
||||||
|
* "no local LLM on this machine" is an environment fact, not a bug.
|
||||||
|
*/
|
||||||
|
|
||||||
|
const projectRoot = path.resolve(__dirname, '..');
|
||||||
|
const FIXED_PORT = 39217;
|
||||||
|
const ORIGIN = `http://127.0.0.1:${FIXED_PORT}`;
|
||||||
|
|
||||||
|
async function ollamaIsUp(): Promise<boolean> {
|
||||||
|
try {
|
||||||
|
const res = await fetch('http://127.0.0.1:11434/api/tags');
|
||||||
|
if (!res.ok) return false;
|
||||||
|
const body = (await res.json()) as { models?: Array<{ capabilities?: string[] }> };
|
||||||
|
return (body.models ?? []).some((m) => !m.capabilities || m.capabilities.includes('completion'));
|
||||||
|
} catch {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
test.describe('Electron desktop shell - local LLM answers from the real encrypted mail index', () => {
|
||||||
|
let electronApp: ElectronApplication;
|
||||||
|
let appWindow: Page;
|
||||||
|
let userDataDir: string;
|
||||||
|
|
||||||
|
test.beforeAll(async () => {
|
||||||
|
if (!(await ollamaIsUp())) {
|
||||||
|
test.skip(true, 'No local Ollama with a completion-capable model reachable on this machine — environment fact, not a failure.');
|
||||||
|
}
|
||||||
|
|
||||||
|
userDataDir = fs.mkdtempSync(path.join(os.tmpdir(), 'vncmail-ai-index-test-'));
|
||||||
|
|
||||||
|
electronApp = await electron.launch({
|
||||||
|
args: [projectRoot, `--user-data-dir=${userDataDir}`],
|
||||||
|
env: {
|
||||||
|
...process.env,
|
||||||
|
VNCMAIL_TEST_FIXED_PORT: String(FIXED_PORT),
|
||||||
|
DEV_MOCK_JMAP: 'true',
|
||||||
|
JMAP_SERVER_URL: `${ORIGIN}/api/dev-jmap`,
|
||||||
|
SESSION_SECRET: 'electron-ai-local-index-verify-32-chars-min',
|
||||||
|
// Deliberately UNSET: isolates grounding to the local FTS leg (see
|
||||||
|
// module header) — the server embeddings leg 404s cleanly instead
|
||||||
|
// of silently also being able to answer the question.
|
||||||
|
AI_SERVER_BASE_URL: '',
|
||||||
|
NODE_ENV: 'production',
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
appWindow = await electronApp.firstWindow();
|
||||||
|
await appWindow.waitForLoadState('domcontentloaded');
|
||||||
|
});
|
||||||
|
|
||||||
|
test.afterAll(async () => {
|
||||||
|
await electronApp?.close();
|
||||||
|
if (userDataDir) fs.rmSync(userDataDir, { recursive: true, force: true });
|
||||||
|
});
|
||||||
|
|
||||||
|
test('logs in, builds the real encrypted index, and a local Ollama model answers a mail question grounded in it', async () => {
|
||||||
|
// ── 1. Real dev-mode login (sets the real session cookie the offline
|
||||||
|
// index and every other server-side-identity feature need). ──
|
||||||
|
const devLoginContainer = appWindow.locator('div', { hasText: 'Dev mode - logging in as dev@localhost' }).last();
|
||||||
|
await devLoginContainer.getByRole('button').click();
|
||||||
|
await appWindow.waitForURL((url) => !url.pathname.includes('login'), { timeout: 20000 });
|
||||||
|
|
||||||
|
// ── 2. Build the real encrypted local index: delta-sync the mock
|
||||||
|
// account's mail into the replica store, then write it into SQLite/FTS5.
|
||||||
|
// Chains /api/offline/sync while unfinishedWork is true, capped so a
|
||||||
|
// real bug can't hang the test forever. ──
|
||||||
|
const syncOutcome = await appWindow.evaluate(async () => {
|
||||||
|
let unfinished = true;
|
||||||
|
let calls = 0;
|
||||||
|
const statuses: number[] = [];
|
||||||
|
while (unfinished && calls < 10) {
|
||||||
|
const res = await fetch('/api/offline/sync', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: '{}' });
|
||||||
|
statuses.push(res.status);
|
||||||
|
if (!res.ok) break;
|
||||||
|
const body = await res.json();
|
||||||
|
unfinished = body.unfinishedWork === true;
|
||||||
|
calls++;
|
||||||
|
}
|
||||||
|
const reindexRes = await fetch('/api/offline/reindex', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ catchUp: true }) });
|
||||||
|
return { syncStatuses: statuses, syncCalls: calls, reindexStatus: reindexRes.status, reindexBody: await reindexRes.json().catch(() => null) };
|
||||||
|
});
|
||||||
|
console.log('[ai-local-index] sync+reindex outcome:', JSON.stringify(syncOutcome));
|
||||||
|
expect(syncOutcome.syncStatuses.every((s) => s === 200)).toBe(true);
|
||||||
|
expect(syncOutcome.reindexStatus).toBe(200);
|
||||||
|
|
||||||
|
// ── 3. Prove the local index itself is real and queryable BEFORE
|
||||||
|
// touching the LLM at all — isolates "is the SQLite/FTS5 index working"
|
||||||
|
// from "did the model use it correctly". ──
|
||||||
|
const directSearch = await appWindow.evaluate(async () => {
|
||||||
|
const res = await fetch(`/api/offline/search?q=${encodeURIComponent('Villa sul Lago check-in')}&limit=6`);
|
||||||
|
return { status: res.status, body: await res.json().catch(() => null) };
|
||||||
|
});
|
||||||
|
console.log('[ai-local-index] direct /api/offline/search result:', JSON.stringify(directSearch.body));
|
||||||
|
expect(directSearch.status, 'the encrypted local index must be reachable (200), not 404 (feature disabled) or 503 (no key channel)').toBe(200);
|
||||||
|
expect(directSearch.body?.ok).toBe(true);
|
||||||
|
const hitTitles = (directSearch.body?.hits ?? []).map((h: { title?: string }) => h.title ?? '');
|
||||||
|
expect(hitTitles.some((t: string) => /villa sul lago/i.test(t)), `expected a "Villa sul Lago" hit in the real index, got: ${JSON.stringify(hitTitles)}`).toBe(true);
|
||||||
|
|
||||||
|
// ── 4. Navigate to the real AI Assistant settings UI and use the
|
||||||
|
// local-discovery "Connect" banner — the exact flow a real user takes,
|
||||||
|
// proving discovery -> connect -> ask works as one integrated feature.
|
||||||
|
// Deliberately an in-app SPA navigation (click the real sidebar link),
|
||||||
|
// NOT appWindow.goto() — a full page reload drops whatever client-only
|
||||||
|
// session state the dev-mode login established (confirmed: goto('/settings')
|
||||||
|
// bounces straight back to /login even though the JMAP session cookie
|
||||||
|
// from step 2/3 is still valid), so the click is load-bearing, not
|
||||||
|
// cosmetic. ──
|
||||||
|
await appWindow.locator('a[href="/settings"], a[href*="/settings"]').first().click();
|
||||||
|
const searchBox = appWindow.locator('input[type="search"]').first();
|
||||||
|
await searchBox.fill('AI Assistant');
|
||||||
|
await appWindow.getByRole('button', { name: 'AI Assistant' }).click();
|
||||||
|
|
||||||
|
const connectButton = appWindow.getByRole('button', { name: /Connect/i });
|
||||||
|
await expect(connectButton, 'the local-discovery banner should appear since a real Ollama is running on this machine').toBeVisible({ timeout: 10000 });
|
||||||
|
const bannerText = await appWindow.locator('text=Local AI found on this machine').locator('..').innerText();
|
||||||
|
console.log('[ai-local-index] discovery banner text:', bannerText);
|
||||||
|
await connectButton.click();
|
||||||
|
|
||||||
|
// ── 5. Ask a question only answerable by combining the local LLM
|
||||||
|
// with the local index's actual content. ──
|
||||||
|
const questionBox = appWindow.getByPlaceholder(/What did legal say/i);
|
||||||
|
await questionBox.fill('When is check-in for the Villa sul Lago booking, and what time?');
|
||||||
|
const askButton = appWindow.getByRole('button', { name: /^Ask$/ });
|
||||||
|
await expect(askButton, 'Ask must be enabled immediately after Connect pre-fills provider+model').toBeEnabled({ timeout: 5000 });
|
||||||
|
|
||||||
|
const chatRequests: string[] = [];
|
||||||
|
const offlineSearchCalls: Array<{ url: string; status: number; body: unknown }> = [];
|
||||||
|
appWindow.on('request', (req) => {
|
||||||
|
if (req.url().includes('11434')) chatRequests.push(`${req.method()} ${req.url()}`);
|
||||||
|
});
|
||||||
|
appWindow.on('response', async (res) => {
|
||||||
|
if (res.url().includes('/api/offline/search')) {
|
||||||
|
offlineSearchCalls.push({ url: res.url(), status: res.status(), body: await res.json().catch(() => null) });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// Log the exact prompt Ollama actually received, straight from the
|
||||||
|
// request body — the ground truth for "did retrieval even fire".
|
||||||
|
const ollamaChatPayloads: unknown[] = [];
|
||||||
|
await appWindow.route('**/api/chat', async (route) => {
|
||||||
|
try {
|
||||||
|
ollamaChatPayloads.push(JSON.parse(route.request().postData() ?? 'null'));
|
||||||
|
} catch { /* ignore parse failure, still let the request through */ }
|
||||||
|
await route.continue();
|
||||||
|
});
|
||||||
|
|
||||||
|
await askButton.click();
|
||||||
|
|
||||||
|
const answerLocator = appWindow.locator('p.whitespace-pre-wrap').first();
|
||||||
|
await expect(answerLocator, 'the local Ollama model should produce an answer within a generous timeout').toBeVisible({ timeout: 60000 });
|
||||||
|
const answerText = await answerLocator.innerText();
|
||||||
|
console.log('[ai-local-index] final answer:', answerText);
|
||||||
|
console.log('[ai-local-index] direct-to-Ollama requests observed:', chatRequests);
|
||||||
|
console.log('[ai-local-index] /api/offline/search calls during Ask:', JSON.stringify(offlineSearchCalls));
|
||||||
|
console.log('[ai-local-index] exact payload(s) sent to Ollama /api/chat:', JSON.stringify(ollamaChatPayloads));
|
||||||
|
|
||||||
|
// The real proof: the model's own words contain the fact that only
|
||||||
|
// exists in the indexed email (28 March, 15:00), and the request log
|
||||||
|
// shows the renderer talked to Ollama's loopback address directly.
|
||||||
|
expect(answerText).toMatch(/28\s*march|march\s*28/i);
|
||||||
|
expect(answerText).toMatch(/15:00|3\s*pm|3:00\s*pm/i);
|
||||||
|
expect(chatRequests.some((r) => r.includes('/api/chat')), `expected a direct renderer -> Ollama /api/chat request, saw: ${JSON.stringify(chatRequests)}`).toBe(true);
|
||||||
|
|
||||||
|
await appWindow.screenshot({ path: path.join(projectRoot, 'electron-ai-local-index-result.png'), fullPage: true });
|
||||||
|
});
|
||||||
|
});
|
||||||
+12
-1
@@ -173,7 +173,18 @@ async function startStandaloneServer(): Promise<string> {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
const port = await getFreePort();
|
// Normally a random free port, chosen fresh every launch - JMAP_SERVER_URL
|
||||||
|
// never needs to reference it back (a real deployment's Stalwart lives at
|
||||||
|
// its own fixed address). VNCMAIL_TEST_FIXED_PORT is a narrow escape hatch
|
||||||
|
// for e2e tests that DO need to know the port ahead of time - specifically
|
||||||
|
// to point DEV_MOCK_JMAP's JMAP_SERVER_URL at this same standalone server's
|
||||||
|
// own /api/dev-jmap route, which is the only way to exercise the real
|
||||||
|
// encrypted offline index (lib/mail-index/**) without a real Stalwart
|
||||||
|
// fixture: that index's key channel only gets wired up in this function,
|
||||||
|
// never when ELECTRON_LOAD_URL bypasses it for a plain `next dev` target.
|
||||||
|
// Unset in every normal launch, so this changes nothing outside a test run.
|
||||||
|
const fixedPort = process.env.VNCMAIL_TEST_FIXED_PORT ? Number(process.env.VNCMAIL_TEST_FIXED_PORT) : null;
|
||||||
|
const port = fixedPort && Number.isInteger(fixedPort) ? fixedPort : await getFreePort();
|
||||||
const url = `http://127.0.0.1:${port}`;
|
const url = `http://127.0.0.1:${port}`;
|
||||||
|
|
||||||
const storeDir = getIndexStoreDir();
|
const storeDir = getIndexStoreDir();
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ import { afterEach, beforeEach, describe, expect, it } from 'vitest';
|
|||||||
import { isSqlcipherAvailable } from '../binding';
|
import { isSqlcipherAvailable } from '../binding';
|
||||||
import type { SqlcipherConstructor, SqlcipherDatabase, SqlcipherStatement } from '../binding';
|
import type { SqlcipherConstructor, SqlcipherDatabase, SqlcipherStatement } from '../binding';
|
||||||
import { accountFileToken, getStoreDir, indexDbPath, STORE_DIR_ENV } from '../paths';
|
import { accountFileToken, getStoreDir, indexDbPath, STORE_DIR_ENV } from '../paths';
|
||||||
import { MailIndex, openKeyed, toFtsMatchQuery, type IndexDoc } from '../store';
|
import { MailIndex, openKeyed, toFtsMatchQuery, toFtsMatchQueryAny, type IndexDoc } from '../store';
|
||||||
|
|
||||||
describe('toFtsMatchQuery', () => {
|
describe('toFtsMatchQuery', () => {
|
||||||
it('quotes every token so FTS5 operators in user input cannot break the query', () => {
|
it('quotes every token so FTS5 operators in user input cannot break the query', () => {
|
||||||
@@ -48,6 +48,62 @@ describe('toFtsMatchQuery', () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
describe('toFtsMatchQueryAny', () => {
|
||||||
|
it('drops English function words and OR-joins what is left - the confirmed-live failure this fixes', () => {
|
||||||
|
// AND-every-token (toFtsMatchQuery) returns 0 hits for this exact
|
||||||
|
// question against a document that only contains "Villa sul Lago" and
|
||||||
|
// "check-in" - see app/api/offline/search/route.ts's comment and the
|
||||||
|
// e2e electron-ai-local-index.spec.ts run that first caught this.
|
||||||
|
const result = toFtsMatchQueryAny('When is check-in for the Villa sul Lago booking, and what time?');
|
||||||
|
expect(result).not.toBeNull();
|
||||||
|
expect(result).not.toContain(' AND ');
|
||||||
|
expect(result).toContain('"check-in"');
|
||||||
|
expect(result).toContain('"Villa"');
|
||||||
|
expect(result).toContain('"sul"');
|
||||||
|
expect(result).toContain('"Lago"');
|
||||||
|
expect(result).toContain('"booking"');
|
||||||
|
// "time" is the last surviving content word, so it gets the
|
||||||
|
// prefix-match star - not "Lago", which is merely the last one this
|
||||||
|
// test happens to name first.
|
||||||
|
expect(result).toContain('"time"*');
|
||||||
|
// Pure stop words, correctly dropped rather than OR-joined as noise that
|
||||||
|
// would otherwise match almost every document in a mailbox.
|
||||||
|
expect(result).not.toMatch(/"When"|"is"|"for"|"the"|"and"|"what"/i);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('falls back to the unfiltered text when every word is a stop word, rather than searching for nothing', () => {
|
||||||
|
// "What is this" is 100% stop words - dropping all of them would leave
|
||||||
|
// zero tokens (a null match, meaning "return everything" is wrong for a
|
||||||
|
// question shaped like this); falling back to the original text at
|
||||||
|
// least keeps a real, if weak, query.
|
||||||
|
const result = toFtsMatchQueryAny('What is this');
|
||||||
|
expect(result).not.toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('still safely quotes FTS5 syntax characters even after stop-word filtering removes the surrounding noise', () => {
|
||||||
|
// "OR"/"NEAR" themselves are common enough as English words that this
|
||||||
|
// builder's stop-word list intentionally drops bare "or" (unlike
|
||||||
|
// toFtsMatchQuery, which preserves it verbatim - see that test's own
|
||||||
|
// comment on why: different concern, different guarantee). The safety
|
||||||
|
// property that DOES still apply here is the one that matters for a
|
||||||
|
// 500: whatever tokens survive filtering are always quoted before
|
||||||
|
// reaching FTS5, so a stray `"`/`*`/`(` in real question text can never
|
||||||
|
// raise a syntax error.
|
||||||
|
const result = toFtsMatchQueryAny('a" NEAR(bar) baz*');
|
||||||
|
expect(result).not.toBeNull();
|
||||||
|
expect(result).toContain('"NEAR"');
|
||||||
|
expect(result).toContain('"bar"');
|
||||||
|
expect(result).toContain('"baz"');
|
||||||
|
expect(result).not.toMatch(/fts5|syntax/i);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('returns null for input with no usable tokens', () => {
|
||||||
|
expect(toFtsMatchQueryAny('')).toBeNull();
|
||||||
|
expect(toFtsMatchQueryAny('***')).toBeNull();
|
||||||
|
expect(toFtsMatchQueryAny(undefined as unknown as string)).toBeNull();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
describe('paths', () => {
|
describe('paths', () => {
|
||||||
const original = process.env[STORE_DIR_ENV];
|
const original = process.env[STORE_DIR_ENV];
|
||||||
afterEach(() => {
|
afterEach(() => {
|
||||||
|
|||||||
+79
-24
@@ -331,8 +331,12 @@ export class MailIndex {
|
|||||||
types?: readonly ContentType[];
|
types?: readonly ContentType[];
|
||||||
limit?: number;
|
limit?: number;
|
||||||
snippetTokens?: number;
|
snippetTokens?: number;
|
||||||
|
/** 'and' (default): every token required - a deliberate search-box query.
|
||||||
|
* 'any': stop words dropped, remaining tokens OR-joined, ranked by bm25 -
|
||||||
|
* a natural-language question (see toFtsMatchQueryAny's docstring). */
|
||||||
|
mode?: 'and' | 'any';
|
||||||
}): SearchHit[] {
|
}): SearchHit[] {
|
||||||
const match = toFtsMatchQuery(opts.query);
|
const match = opts.mode === 'any' ? toFtsMatchQueryAny(opts.query) : toFtsMatchQuery(opts.query);
|
||||||
if (!match) return [];
|
if (!match) return [];
|
||||||
|
|
||||||
const limit = Math.min(Math.max(opts.limit ?? 20, 1), 200);
|
const limit = Math.min(Math.max(opts.limit ?? 20, 1), 200);
|
||||||
@@ -446,34 +450,85 @@ function safeParseObject(v: unknown): Record<string, unknown> {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Turns arbitrary user text into a safe FTS5 MATCH expression.
|
* Splits and safely quotes raw text into FTS5-safe tokens, shared by both
|
||||||
*
|
* query-builders below. Split on anything that isn't a word character or an
|
||||||
* FTS5's query syntax is not SQL, so parameter binding does NOT protect it: a
|
* intra-word mark - keeps unicode letters (so "Müller" and "東京" survive)
|
||||||
* bare `"` or a stray `*`/`NEAR`/`:` in user input raises
|
* via the u flag. Every token is quoted (making it a literal phrase) so a
|
||||||
* `fts5: syntax error`, which would turn a normal search box into a 500. Every
|
* bare `"` or a stray `*`/`NEAR`/`:` in user input can never raise FTS5's own
|
||||||
* token is quoted (making it a literal phrase) and a trailing `*` is added to
|
* `fts5: syntax error` - that would turn a normal search into a 500.
|
||||||
* the last token so typing continues to match as the user types.
|
*/
|
||||||
|
function quoteFtsTokens(raw: string): string[] {
|
||||||
|
return raw
|
||||||
|
.normalize('NFC')
|
||||||
|
.split(/[^\p{L}\p{N}_@.'-]+/u)
|
||||||
|
.map((t) => t.replace(/^['-]+|['-]+$/g, ''))
|
||||||
|
.filter((t) => t.length > 0)
|
||||||
|
.slice(0, 24)
|
||||||
|
.map((t, i, all) => {
|
||||||
|
const quoted = `"${t.replace(/"/g, '""')}"`;
|
||||||
|
// Prefix-match only the final token, and only if it's long enough to not
|
||||||
|
// match half the mailbox.
|
||||||
|
return i === all.length - 1 && t.length >= 3 ? `${quoted}*` : quoted;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Turns arbitrary user text into a safe FTS5 MATCH expression, every token
|
||||||
|
* required (AND-joined). Right for a deliberate, short search-box query,
|
||||||
|
* where requiring every word is what makes results precise as you type.
|
||||||
*
|
*
|
||||||
* Exported for unit testing - it is the one piece of this file with no
|
* Exported for unit testing - it is the one piece of this file with no
|
||||||
* database dependency and the most ways to be wrong.
|
* database dependency and the most ways to be wrong.
|
||||||
*/
|
*/
|
||||||
export function toFtsMatchQuery(raw: string): string | null {
|
export function toFtsMatchQuery(raw: string): string | null {
|
||||||
if (typeof raw !== 'string') return null;
|
if (typeof raw !== 'string') return null;
|
||||||
// Split on anything that isn't a word character or an intra-word mark. Keeps
|
const tokens = quoteFtsTokens(raw);
|
||||||
// unicode letters (so "Müller" and "東京" survive) via the u flag.
|
|
||||||
const tokens = raw
|
|
||||||
.normalize('NFC')
|
|
||||||
.split(/[^\p{L}\p{N}_@.'-]+/u)
|
|
||||||
.map((t) => t.replace(/^['-]+|['-]+$/g, ''))
|
|
||||||
.filter((t) => t.length > 0)
|
|
||||||
.slice(0, 24);
|
|
||||||
if (tokens.length === 0) return null;
|
if (tokens.length === 0) return null;
|
||||||
return tokens
|
return tokens.join(' AND ');
|
||||||
.map((t, i) => {
|
}
|
||||||
const quoted = `"${t.replace(/"/g, '""')}"`;
|
|
||||||
// Prefix-match only the final token, and only if it's long enough to not
|
// A minimal, well-known set of English function words that carry no
|
||||||
// match half the mailbox.
|
// retrieval signal - kept out of toFtsMatchQueryAny's OR expression so they
|
||||||
return i === tokens.length - 1 && t.length >= 3 ? `${quoted}*` : quoted;
|
// don't drown out the bm25 ranking's actual signal (see below). Deliberately
|
||||||
})
|
// NOT applied inside quoteFtsTokens/toFtsMatchQuery: that function's own
|
||||||
.join(' AND ');
|
// tests rely on "AND"/"OR"/"NOT" surviving verbatim as literal search terms
|
||||||
|
// (FTS5-keyword-injection safety) - a different concern from this one's job
|
||||||
|
// of turning a natural-language QUESTION into a good search.
|
||||||
|
const RETRIEVAL_STOP_WORDS = new Set([
|
||||||
|
'a', 'an', 'the', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'am',
|
||||||
|
'and', 'or', 'but', 'if', 'then', 'than', 'so', 'because',
|
||||||
|
'for', 'of', 'to', 'in', 'on', 'at', 'by', 'with', 'from', 'as', 'about', 'into', 'over', 'after', 'before',
|
||||||
|
'that', 'this', 'these', 'those', 'what', 'when', 'where', 'who', 'whom', 'which', 'why', 'how',
|
||||||
|
'do', 'does', 'did', 'doing', 'done',
|
||||||
|
'can', 'could', 'will', 'would', 'shall', 'should', 'may', 'might', 'must',
|
||||||
|
'i', 'you', 'he', 'she', 'it', 'we', 'they', 'my', 'your', 'his', 'her', 'its', 'our', 'their', 'me', 'him', 'us', 'them',
|
||||||
|
'not', 'no',
|
||||||
|
]);
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Turns a natural-language QUESTION into a lenient FTS5 MATCH expression:
|
||||||
|
* stop words dropped, remaining tokens OR-joined so bm25 ranks by how many
|
||||||
|
* content words matched instead of requiring every one of them present.
|
||||||
|
*
|
||||||
|
* toFtsMatchQuery's strict AND is wrong for this shape of input: a real
|
||||||
|
* question like "When is check-in for the Villa sul Lago booking?" shares
|
||||||
|
* almost none of its own function words ("when"/"is"/"for"/"the") with the
|
||||||
|
* document that actually answers it, so ANDing every token together returns
|
||||||
|
* nothing - confirmed live: 0 hits for the full question, 2 correct hits for
|
||||||
|
* the same index once reduced to "Villa sul Lago check-in". The one real
|
||||||
|
* caller of `/api/offline/search?q=...` is exactly this AI-question shape
|
||||||
|
* (see that route's own header - no manual search-box UI hits it today), so
|
||||||
|
* this is the query builder that route now uses, not toFtsMatchQuery.
|
||||||
|
*/
|
||||||
|
export function toFtsMatchQueryAny(raw: string): string | null {
|
||||||
|
if (typeof raw !== 'string') return null;
|
||||||
|
const withoutStopWords = raw
|
||||||
|
.split(/\s+/)
|
||||||
|
.filter((w) => w.length > 0 && !RETRIEVAL_STOP_WORDS.has(w.toLowerCase().replace(/^[^\p{L}\p{N}]+|[^\p{L}\p{N}]+$/gu, '')))
|
||||||
|
.join(' ');
|
||||||
|
// Every word was a stop word (e.g. "What is this?") - fall back to the
|
||||||
|
// original text rather than searching for literally nothing.
|
||||||
|
const tokens = quoteFtsTokens(withoutStopWords.length > 0 ? withoutStopWords : raw);
|
||||||
|
if (tokens.length === 0) return null;
|
||||||
|
return tokens.join(' OR ');
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,16 @@
|
|||||||
|
import { defineConfig } from '@playwright/test';
|
||||||
|
|
||||||
|
// Separate from playwright.electron.config.ts (which hardcodes testMatch to
|
||||||
|
// electron-smoke.spec.ts) purely so this one test can get a longer timeout —
|
||||||
|
// real Ollama inference plus a real multi-round offline sync/reindex chain
|
||||||
|
// legitimately takes longer than the smoke suite's 60s budget.
|
||||||
|
export default defineConfig({
|
||||||
|
testDir: './e2e',
|
||||||
|
testMatch: 'electron-ai-local-index.spec.ts',
|
||||||
|
timeout: 120000,
|
||||||
|
retries: 0,
|
||||||
|
use: {
|
||||||
|
trace: 'retain-on-failure',
|
||||||
|
},
|
||||||
|
workers: 1,
|
||||||
|
});
|
||||||
Reference in New Issue
Block a user