Bernd Rodler
87336981d3
The two things that made real questions fail against a correctly-populated index, both fixed at the root. RETENTION (A1). `INDEX_WINDOW_DAYS = 30` was not merely a fetch bound — catch-up also PRUNED mail older than it, so "summarise everything from July" was unanswerable in August because the rows had been deleted, while the UI said only that nothing matched. Now a user-visible setting (Settings → About & Data): 30 days / 3 months / 1 year / everything, defaulting to 1 YEAR per the product owner. The window bounds the fetch AND the prune from one value so the two can never disagree and delete what was just written; "everything" skips pruning entirely rather than falling back to some default bound. The per-pass ceiling scales with the window (500/30d, hard cap 20k) because 500 messages is right for a month and nonsense for "everything". Email/query now omits the `after` filter entirely when unbounded — Stalwart rejects a malformed filter rather than treating `undefined` as unset. RECENCY (A2). Keyword search structurally cannot answer a question about WHEN: bm25 ranks by term overlap, so "who sent the last email" matches documents containing the word "last", and "all mails in July" matches documents containing "July" — not documents dated in July. Both were asked by a real user and both failed. New lib/mail-index/recency.ts detects time intent (English + German, since the UI ships German) and turns it into a date RANGE; new MailIndex.recent() answers it with an ordered scan over the already-indexed `occurred_at`. The route ADDS these hits to the keyword hits rather than replacing them — "what did the last mail from Anna say" is both kinds of question at once. Timezone subtlety worth knowing: bounds are built from LOCAL calendar boundaries and serialised as UTC instants, so "July" covers the user's July. A mail at 00:30 local on 1 July belongs to it even though its stored UTC timestamp reads 30 June. My first test asserted the ISO string prefix, which would have enshrined the opposite and passed only in UTC — the tests now assert the local-time property instead. SCOPE, stated by the product owner and now enforced structurally: the assistant only ever sees the mailbox the user is signed in to. Both retrieval legs resolve the active account (local leg by cookie slot, server leg by the session's own JMAP account); there is deliberately no fan-out across connected or shared mailboxes, and adding one would be a policy change, not a feature. Gate: tsc clean, eslint clean, 2520/2520 tests (8 new for recency intent), build clean.
76 lines
3.7 KiB
TypeScript
76 lines
3.7 KiB
TypeScript
import { describe, expect, it } from 'vitest';
|
|
import { detectRecencyIntent } from '../recency';
|
|
|
|
// Fixed "now" so the month-name branch is deterministic: 2026-08-07.
|
|
const NOW = new Date('2026-08-07T10:00:00.000Z');
|
|
|
|
describe('detectRecencyIntent', () => {
|
|
it('recognises the two questions that actually failed against a populated index', () => {
|
|
// Both were asked by a real user and both returned nothing useful, because
|
|
// bm25 matched the WORDS "last"/"July" rather than the dates.
|
|
expect(detectRecencyIntent('who sent the last email', NOW)).not.toBeNull();
|
|
expect(detectRecencyIntent('summarize the input of all mails sent in July', NOW)).not.toBeNull();
|
|
});
|
|
|
|
it('turns a month name into a bounded range, not a top-N', () => {
|
|
const intent = detectRecencyIntent('summarize all mails sent in July', NOW);
|
|
// Asserted in LOCAL time on purpose. The bounds are local midnight
|
|
// rendered as UTC instants, so west of UTC the ISO string reads as the
|
|
// previous month - and that is correct, not a bug: a mail that arrived
|
|
// 00:30 local on 1 July belongs to the user's July even though its UTC
|
|
// timestamp says 30 June. Asserting the ISO prefix would enshrine the
|
|
// wrong semantics and pass only in UTC.
|
|
const since = new Date(intent!.since!);
|
|
const until = new Date(intent!.until!);
|
|
expect(since.getMonth()).toBe(6); // local July...
|
|
expect(since.getDate()).toBe(1); // ...starting on the 1st
|
|
expect(until.getMonth()).toBe(7); // exclusive upper bound = local 1 Aug
|
|
expect(until.getDate()).toBe(1);
|
|
});
|
|
|
|
it('reads a month later in the year as LAST year', () => {
|
|
// Asked in August, "December" cannot mean four months from now.
|
|
const intent = detectRecencyIntent('what came in December?', NOW);
|
|
const since = new Date(intent!.since!);
|
|
expect(since.getFullYear()).toBe(2025);
|
|
expect(since.getMonth()).toBe(11);
|
|
});
|
|
|
|
it('handles today and yesterday as distinct bounded days', () => {
|
|
const today = detectRecencyIntent('anything today?', NOW);
|
|
expect(today?.since).toBeDefined();
|
|
expect(today?.until).toBeUndefined();
|
|
|
|
const yesterday = detectRecencyIntent('what arrived yesterday', NOW);
|
|
expect(yesterday?.since).toBeDefined();
|
|
expect(yesterday?.until).toBeDefined();
|
|
expect(new Date(yesterday!.until!).getTime()).toBeGreaterThan(new Date(yesterday!.since!).getTime());
|
|
});
|
|
|
|
it('understands German recency wording — the app ships a German UI', () => {
|
|
expect(detectRecencyIntent('welche war die letzte Mail?', NOW)).not.toBeNull();
|
|
expect(detectRecencyIntent('was kam heute an', NOW)).not.toBeNull();
|
|
const juli = detectRecencyIntent('Mails aus Juli zusammenfassen', NOW);
|
|
expect(new Date(juli!.since!).getMonth()).toBe(6);
|
|
});
|
|
|
|
it('gives an unbounded top-N when recency is implied but no period named', () => {
|
|
const intent = detectRecencyIntent('what is the newest message', NOW);
|
|
expect(intent?.since).toBeUndefined();
|
|
expect(intent?.until).toBeUndefined();
|
|
expect(intent?.limit).toBeGreaterThan(0);
|
|
});
|
|
|
|
it('does NOT fire on pure content questions — those are keyword search\'s job', () => {
|
|
expect(detectRecencyIntent('what did Anna say about the invoice?', NOW)).toBeNull();
|
|
expect(detectRecencyIntent('when is check-in for the Villa sul Lago booking?', NOW)).toBeNull();
|
|
expect(detectRecencyIntent('find the contract with Bechtle', NOW)).toBeNull();
|
|
});
|
|
|
|
it('does not treat a word merely CONTAINING a keyword as recency', () => {
|
|
// "newsletter" contains "new"; "lastly" contains "last". Word boundaries
|
|
// matter or half a mailbox reads as a time question.
|
|
expect(detectRecencyIntent('unsubscribe from the newsletter', NOW)).toBeNull();
|
|
});
|
|
});
|