4653de6d308d5d49264546cdbb4537fcc522ae9e
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Bernd Rodler
|
87336981d3 |
feat(mail-index): 1-year retention by default + a recency retrieval leg
The two things that made real questions fail against a correctly-populated index, both fixed at the root. RETENTION (A1). `INDEX_WINDOW_DAYS = 30` was not merely a fetch bound — catch-up also PRUNED mail older than it, so "summarise everything from July" was unanswerable in August because the rows had been deleted, while the UI said only that nothing matched. Now a user-visible setting (Settings → About & Data): 30 days / 3 months / 1 year / everything, defaulting to 1 YEAR per the product owner. The window bounds the fetch AND the prune from one value so the two can never disagree and delete what was just written; "everything" skips pruning entirely rather than falling back to some default bound. The per-pass ceiling scales with the window (500/30d, hard cap 20k) because 500 messages is right for a month and nonsense for "everything". Email/query now omits the `after` filter entirely when unbounded — Stalwart rejects a malformed filter rather than treating `undefined` as unset. RECENCY (A2). Keyword search structurally cannot answer a question about WHEN: bm25 ranks by term overlap, so "who sent the last email" matches documents containing the word "last", and "all mails in July" matches documents containing "July" — not documents dated in July. Both were asked by a real user and both failed. New lib/mail-index/recency.ts detects time intent (English + German, since the UI ships German) and turns it into a date RANGE; new MailIndex.recent() answers it with an ordered scan over the already-indexed `occurred_at`. The route ADDS these hits to the keyword hits rather than replacing them — "what did the last mail from Anna say" is both kinds of question at once. Timezone subtlety worth knowing: bounds are built from LOCAL calendar boundaries and serialised as UTC instants, so "July" covers the user's July. A mail at 00:30 local on 1 July belongs to it even though its stored UTC timestamp reads 30 June. My first test asserted the ISO string prefix, which would have enshrined the opposite and passed only in UTC — the tests now assert the local-time property instead. SCOPE, stated by the product owner and now enforced structurally: the assistant only ever sees the mailbox the user is signed in to. Both retrieval legs resolve the active account (local leg by cookie slot, server leg by the session's own JMAP account); there is deliberately no fan-out across connected or shared mailboxes, and adding one would be a policy change, not a feature. Gate: tsc clean, eslint clean, 2520/2520 tests (8 new for recency intent), build clean. |
||
|
Bernd RodlerandClaude Sonnet 5
|
b966d285a9 |
feat(mail-index): encrypted SQLite/FTS5 index over mail, calendar, contacts, files
An on-device, SQLCipher-encrypted full-text index the app can retrieve from to
feed an LLM ("prompt against"), for the Electron desktop shell only.
Shape: no persistent background worker and no resident credential. Indexing is
a normal request-scoped API route, triggered by the renderer's EXISTING live
JMAP push connection - so it reacts to each delivery/change rather than polling.
- lib/mail-index/binding.ts guarded require of the optional native binding
- lib/mail-index/paths.ts the VNCMAIL_DESKTOP_STORE_DIR gate + hashed paths
- lib/mail-index/store.ts schema, upsert, FTS5 search, encryption assertion
- lib/mail-index/extract.ts PURE JMAP-object -> document extractors
- lib/mail-index/jmap.ts minimal stateless server-side JMAP client
- lib/mail-index/key.ts per-job key fetch over the inherited fd
- lib/mail-index/reindex.ts the job + slot->account resolution
- electron/key-service.ts safeStorage wrap/unwrap, served over fd 3
- app/api/offline/reindex POST, event-driven + catch-up
- app/api/offline/search GET, the retrieval surface (hits + contextBlock)
- lib/mail-index-client.ts renderer client; StateChange -> index call
- components/settings/local-index-settings.tsx status + manual catch-up
Decisions worth knowing:
* `@signalapp/sqlcipher` is an OPTIONAL dependency with a guarded runtime
require. It publishes six N-API prebuilds and NO build sources, and both
Dockerfiles are node:24-alpine (musl, no matching prebuild) - as a hard
dependency it would break the production image and the integration fixture's
webmail container, neither of which wants this feature.
* Credentials come from the existing per-slot encrypted `jmap_stalwart_ctx`
cookie via lib/stalwart/credentials.ts - the same helper /api/settings and
/api/push/preview already use. It carries a ready-made header for basic AND
bearer accounts, so the indexer never touches the OAuth refresh-token cookie;
a server-side refresh would rotate a token into a response nobody reads and
silently log the user out.
* The encryption key crosses main -> server over an INHERITED FILE DESCRIPTOR,
never an environment variable: env is readable by any process running as the
same OS user, which would defeat using the OS keychain at all. Fetched per
job and zeroed after, so there is no long-lived key copy.
* safeStorage's Linux `basic_text` backend (no keyring) is treated as refusal,
not degradation - it "encrypts" with a hardcoded public password, which would
look like an encrypted mailbox while providing nothing.
getSelectedStorageBackend() is Linux-only and platform-guarded.
* Every store open asserts `PRAGMA cipher_version` returns a non-empty STRING,
not merely a row: a non-cipher binding returns ZERO ROWS, so a row-count check
would pass vacuously while writing the mailbox to disk in cleartext.
* Files are indexed by name/path/date/size only - NOT by extracted content.
Text extraction from arbitrary PDFs/office documents is a separate problem.
* Account-scoped composite keys `(jmap_account_id, content_type, id)` are kept
even though there is one file per account: one login exposes delegated/shared
JMAP accounts too, and JMAP ids are unique only within an account.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|