feat(mail-index): 1-year retention by default + a recency retrieval leg

The two things that made real questions fail against a correctly-populated
index, both fixed at the root.

RETENTION (A1). `INDEX_WINDOW_DAYS = 30` was not merely a fetch bound — catch-up
also PRUNED mail older than it, so "summarise everything from July" was
unanswerable in August because the rows had been deleted, while the UI said
only that nothing matched. Now a user-visible setting (Settings → About &
Data): 30 days / 3 months / 1 year / everything, defaulting to 1 YEAR per the
product owner. The window bounds the fetch AND the prune from one value so the
two can never disagree and delete what was just written; "everything" skips
pruning entirely rather than falling back to some default bound. The per-pass
ceiling scales with the window (500/30d, hard cap 20k) because 500 messages is
right for a month and nonsense for "everything". Email/query now omits the
`after` filter entirely when unbounded — Stalwart rejects a malformed filter
rather than treating `undefined` as unset.

RECENCY (A2). Keyword search structurally cannot answer a question about WHEN:
bm25 ranks by term overlap, so "who sent the last email" matches documents
containing the word "last", and "all mails in July" matches documents
containing "July" — not documents dated in July. Both were asked by a real
user and both failed. New lib/mail-index/recency.ts detects time intent
(English + German, since the UI ships German) and turns it into a date RANGE;
new MailIndex.recent() answers it with an ordered scan over the already-indexed
`occurred_at`. The route ADDS these hits to the keyword hits rather than
replacing them — "what did the last mail from Anna say" is both kinds of
question at once.

Timezone subtlety worth knowing: bounds are built from LOCAL calendar
boundaries and serialised as UTC instants, so "July" covers the user's July.
A mail at 00:30 local on 1 July belongs to it even though its stored UTC
timestamp reads 30 June. My first test asserted the ISO string prefix, which
would have enshrined the opposite and passed only in UTC — the tests now
assert the local-time property instead.

SCOPE, stated by the product owner and now enforced structurally: the
assistant only ever sees the mailbox the user is signed in to. Both retrieval
legs resolve the active account (local leg by cookie slot, server leg by the
session's own JMAP account); there is deliberately no fan-out across connected
or shared mailboxes, and adding one would be a policy change, not a feature.

Gate: tsc clean, eslint clean, 2520/2520 tests (8 new for recency intent), build clean.
This commit is contained in:
Bernd Rodler
2026-08-07 09:45:10 +02:00
parent f6fc34fab3
commit 87336981d3
11 changed files with 406 additions and 22 deletions
+2 -1
View File
@@ -1088,9 +1088,10 @@ export default function Home() {
const runCatchUp = async (attempt: number) => {
if (catchUpCancelled) return;
try {
const { catchUpIndex } = await import('@/lib/mail-index-client');
const { catchUpIndex, getRetentionDays } = await import('@/lib/mail-index-client');
const result = await catchUpIndex(
useAccountStore.getState().getActiveAccount()?.cookieSlot,
getRetentionDays(),
);
if (!result.ok && !result.unavailable && attempt + 1 < catchUpRetryDelaysMs.length) {
catchUpTimer = setTimeout(() => void runCatchUp(attempt + 1), catchUpRetryDelaysMs[attempt + 1]);
+5 -1
View File
@@ -17,7 +17,7 @@ import { isSqlcipherAvailable } from '@/lib/mail-index/binding';
import { hasKeyChannel, IndexKeyError } from '@/lib/mail-index/key';
import { getStoreDir } from '@/lib/mail-index/paths';
import {
IndexSessionError, MAX_IDS_PER_CALL, resolveIndexSession, runIndex,
IndexSessionError, MAX_IDS_PER_CALL, normalizeWindowDays, resolveIndexSession, runIndex,
type IndexRequest,
} from '@/lib/mail-index/reindex';
import { CONTENT_TYPES, isContentType, type ContentType } from '@/lib/mail-index/store';
@@ -73,6 +73,10 @@ export async function POST(request: NextRequest) {
removed: parseIdMap(body.removed),
// Pruning is a catch-up concern; a single-delivery call shouldn't scan.
prune: body.catchUp === true,
// `undefined` (absent) means "use the default"; an explicit null means
// keep everything. normalizeWindowDays() in runIndex clamps anything
// unexpected, since this value drives deletion.
windowDays: body.windowDays === undefined ? undefined : normalizeWindowDays(body.windowDays),
};
try {
+28 -1
View File
@@ -17,6 +17,7 @@ import { IndexSessionError, resolveIndexSession } from '@/lib/mail-index/reindex
import {
isContentType, MailIndex, MailIndexUnavailableError, type ContentType, type SearchHit,
} from '@/lib/mail-index/store';
import { detectRecencyIntent } from '@/lib/mail-index/recency';
export const runtime = 'nodejs';
export const dynamic = 'force-dynamic';
@@ -84,7 +85,29 @@ export async function GET(request: NextRequest) {
// not deliberate search-box keywords, so strict AND-every-token
// matching (the default) drops nearly all of them. See
// toFtsMatchQueryAny's docstring for the confirmed-live failure.
return { hits: index.search({ query, types, limit, mode: 'any' }), stats: wantStats ? stats : undefined };
const keywordHits = index.search({ query, types, limit, mode: 'any' });
// RECENCY leg. Keyword search structurally cannot answer "the last
// mail" or "everything from July" (see lib/mail-index/recency.ts), so
// when the question is really about time, add a date-ordered slice.
// ADDED to the keyword hits rather than replacing them: "what did the
// last mail from Anna say" is both a time question and a content one.
const intent = detectRecencyIntent(query);
if (!intent) {
return { hits: keywordHits, stats: wantStats ? stats : undefined };
}
const recentHits = index.recent({
types, limit: Math.min(intent.limit, limit * 3), since: intent.since, until: intent.until,
});
const seen = new Set(keywordHits.map((h) => `${h.contentType}:${h.id}`));
const merged = [...keywordHits];
for (const hit of recentHits) {
const key = `${hit.contentType}:${hit.id}`;
if (seen.has(key)) continue;
seen.add(key);
merged.push(hit);
}
return { hits: merged, stats: wantStats ? stats : undefined, recency: intent };
} finally {
index.close();
}
@@ -100,6 +123,10 @@ export async function GET(request: NextRequest) {
// Everything a prompt needs, pre-joined in rank order.
contextBlock: payload.hits.map(toContextBlock).join('\n\n---\n\n'),
...(payload.stats ? { stats: payload.stats } : {}),
// Present when the question was read as a time question — lets the
// client say "these are the newest N" instead of implying relevance
// ranking it did not do.
...(payload.recency ? { recency: payload.recency } : {}),
},
{ headers: { 'Cache-Control': 'no-store' } },
);