Bernd Rodler
87336981d3
The two things that made real questions fail against a correctly-populated index, both fixed at the root. RETENTION (A1). `INDEX_WINDOW_DAYS = 30` was not merely a fetch bound — catch-up also PRUNED mail older than it, so "summarise everything from July" was unanswerable in August because the rows had been deleted, while the UI said only that nothing matched. Now a user-visible setting (Settings → About & Data): 30 days / 3 months / 1 year / everything, defaulting to 1 YEAR per the product owner. The window bounds the fetch AND the prune from one value so the two can never disagree and delete what was just written; "everything" skips pruning entirely rather than falling back to some default bound. The per-pass ceiling scales with the window (500/30d, hard cap 20k) because 500 messages is right for a month and nonsense for "everything". Email/query now omits the `after` filter entirely when unbounded — Stalwart rejects a malformed filter rather than treating `undefined` as unset. RECENCY (A2). Keyword search structurally cannot answer a question about WHEN: bm25 ranks by term overlap, so "who sent the last email" matches documents containing the word "last", and "all mails in July" matches documents containing "July" — not documents dated in July. Both were asked by a real user and both failed. New lib/mail-index/recency.ts detects time intent (English + German, since the UI ships German) and turns it into a date RANGE; new MailIndex.recent() answers it with an ordered scan over the already-indexed `occurred_at`. The route ADDS these hits to the keyword hits rather than replacing them — "what did the last mail from Anna say" is both kinds of question at once. Timezone subtlety worth knowing: bounds are built from LOCAL calendar boundaries and serialised as UTC instants, so "July" covers the user's July. A mail at 00:30 local on 1 July belongs to it even though its stored UTC timestamp reads 30 June. My first test asserted the ISO string prefix, which would have enshrined the opposite and passed only in UTC — the tests now assert the local-time property instead. SCOPE, stated by the product owner and now enforced structurally: the assistant only ever sees the mailbox the user is signed in to. Both retrieval legs resolve the active account (local leg by cookie slot, server leg by the session's own JMAP account); there is deliberately no fan-out across connected or shared mailboxes, and adding one would be a policy change, not a feature. Gate: tsc clean, eslint clean, 2520/2520 tests (8 new for recency intent), build clean.
93 lines
4.2 KiB
TypeScript
93 lines
4.2 KiB
TypeScript
// Recency-intent detection for the retrieval layer.
|
|
//
|
|
// Keyword search cannot answer a question about WHEN. bm25 ranks by term
|
|
// overlap, so "what was the last mail I received" matches documents that
|
|
// happen to contain the word "last", and "summarise everything from July"
|
|
// matches documents containing "July" — not documents dated in July. Both were
|
|
// asked by a real user against a correctly-populated index and both returned
|
|
// nothing useful, which is what this module exists to fix: it decides when a
|
|
// question is really a date question, and turns it into a date RANGE the index
|
|
// can answer with an ordered scan over `occurred_at`.
|
|
//
|
|
// Deliberately a heuristic on English/German keywords rather than an LLM call:
|
|
// it runs on every question, must be instant, and a false positive is cheap
|
|
// (the recency hits are fused with the keyword hits, not substituted for them).
|
|
|
|
// TIMEZONE NOTE: all bounds are built from LOCAL calendar boundaries and then
|
|
// serialised as UTC instants. That is deliberate — "July" means the user's
|
|
// July, so a mail received 00:30 local on 1 July belongs to it even though its
|
|
// stored UTC timestamp reads 30 June. Building the bounds in UTC instead would
|
|
// silently drop the first/last hours of every named period for anyone not on
|
|
// UTC.
|
|
export interface RecencyIntent {
|
|
/** ISO lower bound, if the question named one. */
|
|
since?: string;
|
|
/** ISO upper bound, if the question named a closed period. */
|
|
until?: string;
|
|
/** How many documents the recency leg should contribute. */
|
|
limit: number;
|
|
}
|
|
|
|
const RECENCY_WORDS = [
|
|
// English
|
|
'last', 'latest', 'recent', 'recently', 'newest', 'new', 'today', 'yesterday',
|
|
'this week', 'this month', 'past week', 'past month', 'so far', 'just now', 'current',
|
|
// German — the app ships a German UI and users mix languages freely
|
|
'letzte', 'letzten', 'letzter', 'neueste', 'neuesten', 'neu', 'heute', 'gestern',
|
|
'diese woche', 'diesen monat', 'kürzlich', 'zuletzt', 'aktuell',
|
|
];
|
|
|
|
const MONTHS: Record<string, number> = {
|
|
january: 0, february: 1, march: 2, april: 3, may: 4, june: 5,
|
|
july: 6, august: 7, september: 8, october: 9, november: 10, december: 11,
|
|
januar: 0, februar: 1, märz: 2, maerz: 2, mai: 4, juni: 5,
|
|
juli: 6, oktober: 9, dezember: 11,
|
|
};
|
|
|
|
function startOfDay(d: Date): Date {
|
|
const c = new Date(d);
|
|
c.setHours(0, 0, 0, 0);
|
|
return c;
|
|
}
|
|
|
|
/**
|
|
* @param now injected so the behaviour is testable and deterministic — the
|
|
* month-name branch depends on "which year is it" and must not be a coin
|
|
* flip in a test suite.
|
|
*/
|
|
export function detectRecencyIntent(question: string, now: Date = new Date()): RecencyIntent | null {
|
|
const q = question.toLowerCase();
|
|
|
|
// A named month wins over generic recency words: "everything from July" is a
|
|
// bounded range, which is far more useful than "the newest N".
|
|
for (const [name, monthIndex] of Object.entries(MONTHS)) {
|
|
if (!new RegExp(`\\b${name}\\b`).test(q)) continue;
|
|
// A month later than the current one must mean LAST year — "July" asked in
|
|
// March means the July that already happened, not one nine months away.
|
|
const year = monthIndex > now.getMonth() ? now.getFullYear() - 1 : now.getFullYear();
|
|
const since = new Date(year, monthIndex, 1, 0, 0, 0, 0);
|
|
const until = new Date(year, monthIndex + 1, 1, 0, 0, 0, 0);
|
|
return { since: since.toISOString(), until: until.toISOString(), limit: 40 };
|
|
}
|
|
|
|
if (/\btoday\b|\bheute\b/.test(q)) {
|
|
return { since: startOfDay(now).toISOString(), limit: 25 };
|
|
}
|
|
if (/\byesterday\b|\bgestern\b/.test(q)) {
|
|
const start = startOfDay(new Date(now.getTime() - 86_400_000));
|
|
return { since: start.toISOString(), until: startOfDay(now).toISOString(), limit: 25 };
|
|
}
|
|
if (/this week|past week|diese woche|letzte woche/.test(q)) {
|
|
return { since: startOfDay(new Date(now.getTime() - 7 * 86_400_000)).toISOString(), limit: 40 };
|
|
}
|
|
if (/this month|past month|diesen monat|letzten monat/.test(q)) {
|
|
return { since: startOfDay(new Date(now.getTime() - 30 * 86_400_000)).toISOString(), limit: 40 };
|
|
}
|
|
|
|
if (RECENCY_WORDS.some((w) => (w.includes(' ') ? q.includes(w) : new RegExp(`\\b${w}\\b`).test(q)))) {
|
|
// No period named — "the last mail", "what's new". Unbounded top-N.
|
|
return { limit: 15 };
|
|
}
|
|
return null;
|
|
}
|