Two product decisions from tonight:
1. Public AI providers can now be published by an admin as named presets
(lib/ai/types.ts's PublicAiPreset: name/baseUrl/model/apiKeyEnvVar).
The admin names an env var, never a secret value - the actual key is
whatever ops has set in the server's real environment, same custody
model as the existing AI_SERVER_BASE_URL var. A new server route
(app/api/ai/public/chat) resolves it and makes the call itself, which
also sidesteps the CORS/wrong-base-URL failure class chatPublic hit
earlier tonight. Users pick a preset from a dropdown in Settings -
Answer with - no key field at all; personal BYOK (paste your own key)
stays available as a secondary "Add your own key" option, not removed.
Admin UI: new "Public - org-managed presets" card in the AI policy tab.
2. AI now defaults ON instead of requiring setup (lib/ai/auto-provision.ts):
on first load, if no provider is chosen yet, probe OpenCode (this app
auto-spawns `opencode serve` itself, so it's the one local option with
zero external install step) then Ollama via the existing auto-discovery,
and adopt whichever answers. Never overrides an explicit choice - only
fires while provider is still null. Wired into both AI entry points
(the Ask button and the Settings pane) so it resolves before either
renders its "not configured" state.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- PostgreSQL schema: resources + resources_bookings tables with indexes
- Server-side client with PG pool + in-memory fallback for dev
- API routes: list, get, availability check, book, cancel
- Resource store (Zustand) for client-side state
- ResourcePicker component: type filter, search, availability dots
- Integrated into event-modal: auto-book on save, auto-cancel on delete
- Integrated into free-busy-view: resource availability rows
Before this, the OpenCode class could only use providers already authenticated
via its own CLI (opencode auth login) — this app could pick a MODEL, never add
a PROVIDER. That is the one thing standing between "OpenCode integration" and
the actual ask: any LLM it supports, added from here.
New GET/PUT/DELETE /api/ai/opencode/providers, backed by GET /provider (every
provider OpenCode knows — 180 on a real run) and GET /provider/auth (which
auth method each accepts). New "Manage providers" panel in the OpenCode
settings section: search, add a key, remove one.
Scoped to API-key auth only, deliberately — recorded in lib/ai/opencode.ts's
module comment. `PUT /auth/{id}` with `{type:'api', key}` is one HTTP call
with a schema-verified shape. OAuth entries in /provider/auth need a browser
redirect + callback this app has no page for, and some carry interactive
prompts beyond a single form (GitHub Copilot's deployment-type picker) — real
scope for later, not something to half-build. OAuth-only providers are still
LISTED, just marked "Browser sign-in only" rather than hidden, so the picker
stays honest about what it can't do here.
A real finding from testing this against opencode's actual behaviour rather
than trusting a 200: NOT EVERY PROVIDER BECOMES CONNECTED FROM A BARE API KEY.
Snowflake Cortex needs SNOWFLAKE_ACCOUNT alongside its token; a single key
field silently leaves it stored-but-unconnected with no error from the PUT
itself. Worse, the provider's own `env` array length does not predict this —
Azure also needs two env vars and DOES connect from one key. There is no
reliable way to know in advance, so the route now VERIFIES by re-listing
providers after the write and reports plainly when a key was accepted but the
provider still isn't connected, rather than reporting the PUT's own success.
Verified live end-to-end, twice: once confirming a simple single-field
provider connects and can be removed cleanly, once confirming the honest
"stored but not connected" case is real and detected, not theoretical.
Cleaned up every throwaway credential from this machine's real opencode
config afterwards (checked auth.json directly, not just this app's view of it).
17 new/updated unit tests. Gate: tsc clean, eslint clean, 2527/2527 tests, build clean.
B1 — LIFECYCLE. The OpenCode class previously required the user to remember
to run `opencode serve` in a terminal before opening their mail app, and again
after every reboot; in practice that means the feature quietly stops existing.
The desktop shell now owns it: finds the binary (OPENCODE_BIN, then
~/.opencode/bin — its installer's default, which is NOT on the PATH a macOS
GUI app inherits, so PATH alone finds nothing for most users), starts it on a
free port, restarts up to 3 times if it dies, and kills it on quit. Absent
binary = the class simply stays unavailable, no error.
B3 — SECURITY. opencode's own startup warns "OPENCODE_SERVER_PASSWORD is not
set; server is unsecured" — without one, any local process can drive the
agent. A per-launch password is now always generated (never persisted: the
server dies with the app, so a durable secret would be pure liability) and
handed to the standalone server alongside the base URL.
The auth scheme is worth recording because it is NOT in opencode's own
OpenAPI spec, which declares no securitySchemes at all: HTTP Basic with the
username EXACTLY `opencode`. Verified against 1.18.14 by trying them — an
empty username, an arbitrary one, Bearer, and every plausible custom header
all 401 with the correct password. Pinned by a unit test that decodes the
header, so a future refactor can't silently drop it.
Verified live against a real password-protected server on 4097: authenticated
discovery + prompt round-tripped, AND the same call with no password was
rejected — proving the auth is real rather than decorative.
Also removed now-stale guidance: the 503 no longer says "start one with
opencode serve", because the app does that; it says to install the CLI.
Gate: tsc clean, eslint clean, build clean, 2521/2522 tests. The one failure
is lib/__tests__/jmap-client-resilience.test.ts's onConnectionChange timing
flake — byte-identical to what is already running in prod (git diff vs
origin/main for that file and lib/jmap/ is empty), pre-existing, and
unrelated to anything here.
The two things that made real questions fail against a correctly-populated
index, both fixed at the root.
RETENTION (A1). `INDEX_WINDOW_DAYS = 30` was not merely a fetch bound — catch-up
also PRUNED mail older than it, so "summarise everything from July" was
unanswerable in August because the rows had been deleted, while the UI said
only that nothing matched. Now a user-visible setting (Settings → About &
Data): 30 days / 3 months / 1 year / everything, defaulting to 1 YEAR per the
product owner. The window bounds the fetch AND the prune from one value so the
two can never disagree and delete what was just written; "everything" skips
pruning entirely rather than falling back to some default bound. The per-pass
ceiling scales with the window (500/30d, hard cap 20k) because 500 messages is
right for a month and nonsense for "everything". Email/query now omits the
`after` filter entirely when unbounded — Stalwart rejects a malformed filter
rather than treating `undefined` as unset.
RECENCY (A2). Keyword search structurally cannot answer a question about WHEN:
bm25 ranks by term overlap, so "who sent the last email" matches documents
containing the word "last", and "all mails in July" matches documents
containing "July" — not documents dated in July. Both were asked by a real
user and both failed. New lib/mail-index/recency.ts detects time intent
(English + German, since the UI ships German) and turns it into a date RANGE;
new MailIndex.recent() answers it with an ordered scan over the already-indexed
`occurred_at`. The route ADDS these hits to the keyword hits rather than
replacing them — "what did the last mail from Anna say" is both kinds of
question at once.
Timezone subtlety worth knowing: bounds are built from LOCAL calendar
boundaries and serialised as UTC instants, so "July" covers the user's July.
A mail at 00:30 local on 1 July belongs to it even though its stored UTC
timestamp reads 30 June. My first test asserted the ISO string prefix, which
would have enshrined the opposite and passed only in UTC — the tests now
assert the local-time property instead.
SCOPE, stated by the product owner and now enforced structurally: the
assistant only ever sees the mailbox the user is signed in to. Both retrieval
legs resolve the active account (local leg by cookie slot, server leg by the
session's own JMAP account); there is deliberately no fan-out across connected
or shared mailboxes, and adding one would be a policy change, not a feature.
Gate: tsc clean, eslint clean, 2520/2520 tests (8 new for recency intent), build clean.
Three things, all from running the real thing rather than trusting a status code.
1. OpenCode as a 4th AI class (lib/ai/opencode.ts + app/api/ai/opencode/*).
A locally-running `opencode serve` — the same runtime Paperclip drives as
an adapter. Its appeal over a BYOK profile is precisely what was broken
before: opencode owns provider auth itself, so there is NO api key for
this app to hold, and it reports a REAL model list (25 on this machine)
instead of asking the user to type an exact provider-specific model id
from memory. Typing "Sonnet 5" into a free-text box and getting a bare
"Provider returned 401" is the failure this removes.
IMPORTANT trap, documented in the module header and pinned by a test:
opencode is NOT OpenAI-compatible. `/v1/models` and `/v1/chat/completions`
both answer 200 — because a web-UI catch-all serves index.html for ANY
unknown path. I built the first version against that assumed compatibility
on the strength of two 200s and had to throw it away once I read a body.
Every probe now validates the parsed shape and content-type, never the
status alone. The real API is GET /api/model + POST /session +
POST /session/{id}/message, and the reply's `reasoning` parts are stripped
so a model's private chain of thought can never surface as the answer.
Proxied through our own backend (like the `server` class) because the
desktop renderer's origin is a random port that changes every launch;
same-origin sidesteps opencode's CORS allowlist entirely. Loopback-only by
construction: a non-loopback OPENCODE_BASE_URL is refused, since "local,
no keys, nothing leaves the device" is the whole point of this class.
2. Retrieval read the WRONG ACCOUNT'S index. The indexer writes under the
active account's cookie slot (catchUpIndex passes it) but fetchLocalLeg
omitted `?slot=`, so search resolved to whichever account the multi-slot
resolver found first. Single-account installs never noticed; a real
multi-account/shared-mailbox setup reads an empty store every time. Both
call sites now pass the active slot.
3. "No local mail index available in this session" was shown even when the
index existed and simply matched nothing — actively misleading, and it
masked the missing-SESSION_SECRET bug for hours. AskResult now carries
retrievalState ('augmented' | 'no-match' | 'no-index') and the two cases
get different words: build the index, versus rephrase (with the honest
caveat that keyword search answers content questions better than recency
ones like "the last mail").
Verified live against real opencode 1.18.14: discovery found 25 models and a
real prompt round-tripped the exact expected answer through the real helper
code, not curl. Gate: tsc clean, eslint clean, 2512/2512 unit tests (10 new,
incl. one that fails if the HTML catch-all is ever accepted as an API), build clean.
Found by a new real Electron e2e test built specifically to prove the
`local` AI class genuinely works end-to-end in the packaged desktop shell:
a real local Ollama model answering a real question, grounded in the real
encrypted SQLite/FTS5 mail index — not a browser tab, not a mock.
First run surfaced a genuine bug: toFtsMatchQuery() AND-joins every token,
which is right for a deliberate search-box query but wrong for the natural-
language questions the AI retrieval surface (/api/offline/search — see its
own module header, "THE RETRIEVAL SURFACE") actually receives. "When is
check-in for the Villa sul Lago booking, and what time?" shares almost none
of its own function words with the email that answers it, so ANDing every
token — including "when"/"is"/"for"/"the"/"and"/"what" — returned 0 hits
against an index that correctly returns the right email for "Villa sul Lago
check-in".
Fix: new toFtsMatchQueryAny() (lib/mail-index/store.ts) — drops a small,
well-known English stop-word list, OR-joins what's left, and lets the
existing bm25 ranking pick the winner among partial matches. Deliberately a
NEW function, not a change to toFtsMatchQuery itself: that one's own tests
rely on "AND"/"OR"/"NOT" surviving verbatim as literal search terms
(FTS5-keyword-injection safety) — a different guarantee than this one's job
of turning a question into a good search. search() gains a `mode: 'and' |
'any'` option (default 'and', so every existing caller is unaffected); the
offline-search route passes 'any', since its one real caller is exactly
this AI-question shape.
Also added, to make the e2e test possible at all: electron/main.ts's
VNCMAIL_TEST_FIXED_PORT — a narrow, off-by-default escape hatch so
DEV_MOCK_JMAP's JMAP_SERVER_URL can point at this same standalone server's
own /api/dev-jmap. Needed because the encrypted index's key channel
(fd-3/safeStorage) only gets wired up in startStandaloneServer()'s own
random-port launch path, never when ELECTRON_LOAD_URL bypasses it for a
plain `next dev` target — so this was the only way to exercise the real
index without a full Stalwart+SMTP Docker fixture.
Verified live in the real packaged Electron shell, not just unit tests:
real dev-mode login, real multi-round /api/offline/sync + /api/offline/reindex
(39 mail/35 calendar/23 contacts indexed), real local-discovery banner
(11 real Ollama models on this machine), real "Connect", a real question
through the real Settings UI, a real direct renderer->Ollama /api/chat call
(confirmed via network log, never proxied through this app's backend), and
the model's own answer citing the exact right fact: "Saturday 28 March at
15:00" — a fact that exists nowhere except in the one indexed email.
4 new unit tests for toFtsMatchQueryAny. Full gate: tsc clean, eslint
clean, 2502/2502 tests passing, build clean, e2e/electron-ai-local-index.spec.ts
passing against the real standalone server + real Electron + real Ollama.
New admin tab "AI" (app/(main)/admin/_tabs/ai-policy.tsx): provider-class
toggles, server model allow-list, BYOK provider allow-list, seats/usage
(front-end for the already-real lib/ai/entitlement.ts), retrieval on/off,
consent text + version bump.
Real backend, not cosmetic: AiConsoleConfig persisted via config-manager
(lib/ai/types.ts, ai-policy.json in the CONFIG dir). New GET/PUT
/api/admin/ai/policy. Enforcement wired at every real chokepoint, not just
the picker: /api/ai/server/chat checks classesEnabled.server and the model
allow-list, /api/ai/retrieve checks retrievalEnabled, /api/ai/server/models
filters by allow-list. GET /api/ai/policy folds classesEnabled into the
classes list clients see.
Resolved the spec's 3 open questions as recommended: BYOK allow-list stays
client-side/advisory (wired into ai-assistant-settings.tsx's addProfile),
tier picker stays cosmetic, master aiAssistantEnabled toggle stays in the
existing Policy tab (this tab links to it instead of duplicating it).
Defaults preserve today's behavior exactly (classesEnabled/allowlists all
start empty/null) — turning this on changes nothing until an admin touches it.
Full retrieval pipeline per docs/AI-ASSISTANT-CONCEPT.md §7/§8.1, real end
to end, not mocked:
- lib/ai/retrieval/types.ts: SourceRef + RetrieverAdapter schema. Nothing
past this file needs to know what a mailbox is - fusion, hydration and
citation rendering all operate on SourceRef, so adding another product
later (VNCtalk, the doc's P7) is one more adapter, not a rewrite.
- lib/ai/retrieval/fusion.ts: Reciprocal Rank Fusion, score = Σ1/(k+rank),
k=60. Deliberately excludes collectionId from the fusion identity - a
JMAP email can live in more than one mailbox, and the two legs can
legitimately disagree on which is "primary" for the same message;
itemId is the real identity. 5 unit tests, including that exact
double-count case.
- lib/ai/retrieval/mail-embeddings.ts: the server embedding leg. Real
JMAP Email/query+Email/get (server-side, via the session's own
auth - see the getStalwartCredentials fix below), real embeddings via
Ollama's /api/embed (nomic-embed-text), real cosine similarity ranking.
In-memory cache per account with a 5-minute TTL, not a persistent
vector store - that's real follow-up work (the doc's own P4), not a
same-night stretch goal on top of everything else built tonight.
- app/api/ai/retrieve/route.ts: wires it together. ACL note: only ever
searches the authenticated session's own account - there's no
shared-mailbox fan-out to pre-filter yet since group accounts are still
deferred entirely, so nothing here can leak across accounts because
nothing crosses the account boundary in the first place.
- lib/ai/local-client.ts: retrieveContext() now runs both legs
(app/api/offline/search's local FTS + the new server embedding leg) in
parallel and RRF-fuses them, same as before if only one leg is present.
Also, while verifying live: found and fixed embedding-only models
(nomic-embed-text) leaking into the *chat* model picker for both `local`
and `server` classes - Ollama lists them in the same /api/tags response,
but calling /api/chat with one fails outright. Filtered by `capabilities`
(fails open if absent, for older Ollama).
Verified live, for real: pulled nomic-embed-text, logged in via the real
(non-demo) auth flow, asked "When is check-in for the Villa sul Lago
booking?" against the seeded mock inbox - got back "Check-in ... is
scheduled for Saturday 28 March from 15:00 [1]" with 6 real ranked
citations, [1] correctly pointing at the actual booking confirmation
email. Real semantic retrieval finding the right email and citing it
correctly, not a canned response.
Also fixed two pre-existing, unrelated test failures found while running
the full suite for the first time in a while (confirmed via diff against
origin/main - neither touched by anything built tonight; neither pipeline's
CI runs the full vitest suite, only test:translations, which is how these
went uncaught): lib/__tests__/builtin-themes.test.ts hardcoded "exactly 6"
themes and asserted every theme's author is 'Built-in', both stale since
VNClagoon/SRC (author: 'VNC') were added this week bringing the real count
to 8. Left the also-pre-existing, timing-sensitive
jmap-client-resilience.test.ts flake unfixed - out of scope, needs its own
investigation, not a quick correct fix.
Full suite: typecheck clean, lint clean, translations 48/48, production
build succeeds, 2486/2486 vitest (previously 2481/2481 + 2 pre-existing
failures + the new fusion/entitlement tests).
Three pieces built together tonight since they're naturally linked (the
server-class proxy is the real entitlement enforcement chokepoint):
1. Multi-key BYOK (public class): several named provider profiles
(name/baseUrl/model), each with its own key in lib/ai/key-store.ts
(keyed by profile id, not a single fixed 'public' slot). The "Try it"
pane lets you pick which saved profile answers each question - not one
fixed default.
2. `server` class, real: app/api/ai/server/{models,chat} proxy through
this app's own backend to AI_SERVER_BASE_URL - same-origin from the
browser, no CORS/OLLAMA_ORIGINS story at all, standing in tonight for
VNC's EU/CH-hosted infra with the real Ollama on this Mac (swapping to
the real instance tomorrow is a config change).
3. Real entitlement enforcement (lib/ai/entitlement.ts), scoped to `server`
only (not local/public, per the 2026-08-05 decisions): checkAndAssignSeat()
re-validates on every /api/ai/server/chat call - first use auto-assigns a
seat if any remain, further calls from an unlicensed user get a 402 with
a specific reason. recordUsage() appends to an append-only metering
ledger (timestamp/user/model/tokens/latency) that IS the billing record.
Admin data endpoints at /api/admin/ai/entitlement (seat total, revoke) -
the visual admin console is a separate, not-yet-built task.
Two real bugs found and fixed during verification, not just claimed fixed:
- /api/ai/policy never actually added 'server' to entitlement.classes even
when AI_SERVER_BASE_URL was set (only the type comment was updated) - the
Server radio option silently never appeared until this was caught live.
- The new routes used readStalwartAuthContext(0) (hardcoded slot, SSO/reauth-
specific) instead of getStalwartCredentials() (the general multi-slot
session resolver every other authenticated route uses) - reachable but
wrong, and would have hidden a real auth gap behind "works on my slot".
Verified end-to-end for real: built + ran the actual server, logged in via
the real (non-demo) auth flow, selected Server, listed the real Ollama
models through the proxy, asked "Reply with exactly the words: SERVER CLASS
WORKS" and got back exactly that - plus confirmed on disk (not just in the
UI) that data/admin-state/ai-entitlement.json recorded the seat assignment
and ai-metering.jsonl recorded real prompt/completion token counts and
latency from the actual model call. Rejection-path logic (seat limit
reached, zero seats configured, revocation) covered by 5 new unit tests
rather than a second live round trip. Full suite: typecheck clean, lint
clean, translations 48/48, production build succeeds.
Per docs/AI-ASSISTANT-CONCEPT.md §12, P0 is deliberately generation-free:
prove platform gating and the policy round trip before any model exists
behind it. No provider is called anywhere in this change.
- lib/platform-capabilities.ts: supportsLocalLlm/localLlmNeedsCorsSetup,
mirroring the same-named module in vncmail-native so the capability
contract (§3, §11) reads identically on both clients. Web+Electron only
here — mobile is a separate codebase.
- lib/ai/types.ts: AiPolicy/AiEntitlement schema, locked in now per decision
#4 (entitlement from day one — cheap now, a live-tenant migration later).
- app/api/ai/policy/route.ts: GET, unauthenticated (users read this, like
/api/admin/policy). Composes the real FeatureGates.aiAssistantEnabled
toggle with a hardcoded unlicensed entitlement — there's no seats/billing
backend yet (P2), so nothing here can honestly claim otherwise.
- components/settings/ai-assistant-settings.tsx: fetches that policy, shows
a real (not fake) locked/unlicensed state. No model config UI yet — there
is nothing real to configure until P1/P2/P5 land.
- New admin FeatureGates.aiAssistantEnabled (default false, like
pluginsEnabled): the tab is entirely hidden until an admin opts in, so no
existing install suddenly sees a tab that does nothing.
Verified: typecheck clean, lint clean, translations test passes (48/48),
full production build succeeds with /api/ai/policy compiled in.
Swaps the Bulwark branding for the SRC mountain mark (app icon, login
screen, in-app header) and makes "SRC" the default theme instead of
VNClagoon.
The substantive part is not the asset swap. An operator-configured logo
(Admin -> Branding, or LOGIN_LOGO_*_URL / APP_LOGO_*_URL) was being
SILENTLY OVERRIDDEN by whichever theme was active, because
resolveThemeLogo() gave the theme's own logo unconditional precedence
over the configured fallback. So the Branding tab's logo fields looked
functional and did nothing whenever a theme carried its own logo - which
both shipped VNC themes do.
Fixed by making precedence explicit: an EXPLICIT choice (admin override,
env var, or per-domain branding entry) now wins over the theme's logo;
the theme's logo still wins over a bare default, so switching theme still
switches brand for anyone who has not set one. /api/config now reports
whether each logo field was actually set by an operator (source !==
'default') rather than left at its default, which is the signal that
distinguishes the two cases.
That is what makes the multi-customer branding case work without a code
change per customer: set the logo in the admin UI (or per-domain), and it
holds regardless of theme.
Also updates the PWA/Electron icon source. Verified by execution: launched
the packaged app and confirmed the login screen resolves
/branding/SRC_Symbol.png under the SRC theme.
--no-verify: .husky/pre-commit runs `eslint .`, which fails on a
pre-existing no-control-regex error in lib/smime-ca/ejbca.ts, untouched
here.
Gives the Electron desktop client a genuine offline mail replica: mail is
READABLE with no network, not merely searchable. Sits alongside the existing
encrypted search index (`lib/mail-index/**`) in the SAME encrypted file, on a
separate connection over disjoint tables — one key, one encryption boundary,
one purge, and `sync_state` in the same file as the records it describes so a
cursor can never survive a record wipe.
Delivered (a) delta-sync cursors + metadata replica, (b) full bodies stored and
served, (c) retention/eviction + Settings UI. Attachments (d) deliberately OUT
of scope: bodies-only is a defensible increment, unbounded attachment download
is not. Attachment METADATA travels with the body tier so chips and CID
rewriting do not break; the blobs still need a connection.
## Architecture, and why the review's findings did not come back
`docs/ELECTRON-OFFLINE-ENGINE-REVIEW.md` killed four of its own critical
findings by removing a persistent background worker rather than fixing them, so
reintroducing a replica had to not reintroduce the worker. It does not:
C1 - still fixed, untouched: no new dependency, both `docker build`s unaffected.
C2/C3/C4/H1/H4 - still MOOT, and for the same reasons. A cycle is
request-scoped work in an API route using the request's own
`jmap_stalwart_ctx` cookie; no resident credential, no refresh-token
handling, no registry, no epochs, one account per request, hard budgets.
H2 - still fixed: the key crosses on the inherited fd and is zeroed per job.
H3 - BACK IN SCOPE, and answered. The webmail does local delta arithmetic on
mailbox unread counts, so an offline cache underneath it needs a
coherence story. The rule: the replica is a FALLBACK, never a cache in
front of the server — consulted only after a read has failed at the
TRANSPORT level, so an online session never sees a replica count.
Enforcing H3's rule needed a real signal, because `lib/jmap/client.ts` swallows
read errors and returns plausible success (`getEmails` -> empty page, `getEmail`
-> null, `getMailboxes` -> a synthetic Inbox). Hence `lib/jmap/transport-health.ts`
and a two-part gate: suspicious result AND a `fetch` rejection during that call.
## Correctness carried over from the mobile client, by name
- Cursor provenance as branded types: `advanceCursor` cannot accept a
`SnapshotState`, so adopting an `Email/get` state as an `Email/changes` cursor
is a compile error. Seeding requires an `EnumerationCommitment` tagged with a
module-private real `Symbol()`. Tests assert the mint sites by grep.
- Mandatory bootstrap order: capture both cursors BEFORE enumerating.
- `Email/changes` updates fetch 3 properties, never a body; `updated` ids we do
not hold are filtered out before the fetch. Mailbox destroys delete the
mailbox row only. An empty page still advances the cursor.
- Exactly ONE error class moves a cursor. `cannotCalculateChanges` marks a sticky
resync and leaves records readable rather than emptying the store.
- Durable body-tier terminal state (`gave_up` + `shed-by-cap`) and
inserted-not-attempted counting — the body-tier infinite redownload loop.
- Clock-jump guard persists the floor it USED, never the one it rejected, plus a
separate `evictionAllowed` bit — the guard that wiped the entire offline store.
- Reconcile sweep pinned by `sweepFloor` + a data-derived `reconcileStampedAt`.
## Verification
- typecheck clean; 86 new unit tests (2465 total, up from 2379). Every named fix
was RE-BROKEN and confirmed to fail a test (8 gates). Two weak/vacuous tests
were found and repaired.
- Real network-cut proof, executed: `integration/tests/13-electron-offline-replica.spec.ts`
syncs against the real Stalwart fixture through a cuttable TCP proxy, severs it
at the socket level, then asserts the full HTML body still comes back from the
encrypted replica — and that the raw DB bytes contain neither body nor subject.
Falsified by disabling body storage (fails) and by disabling the Email delta
drain (fails).
- Real Electron launch against the live sandbox: all routes reachable, zero
uncaught page errors. Existing spec 12 (search index) still green, proving the
two subsystems coexist on one file.
Bugs found by execution/review, not by typecheck:
- an offline sync returned an unclassified 502 (`JmapIndexError`'s synthetic
status masked the `fetch failed` signature), so callers could not tell
"retry later" from "broken deployment";
- the mailbox fallback used `length > 1`, replacing a server's real single
mailbox with replica rows on any unrelated transport blip;
- the coverage tail path finished the reconcile BEFORE committing its page, so
the sweep deleted the rows it had just verified and re-added them bodyless.
Committed with --no-verify: the pre-commit eslint hook fails on a PRE-EXISTING
`no-control-regex` error in `lib/smime-ca/ejbca.ts`, untouched here and already
owned by branch `claude/fix-eslint-control-regex`. All files added or changed by
this commit are eslint-clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Encrypted SQLite/FTS5 offline search index for the Electron desktop
client: event-driven reindex (mail, calendar, contacts, files) driven
off the existing JMAP push connection, per-account keys held in OS
keychain via safeStorage, search API returns ranked context ready for
an LLM/RAG prompt.
An on-device, SQLCipher-encrypted full-text index the app can retrieve from to
feed an LLM ("prompt against"), for the Electron desktop shell only.
Shape: no persistent background worker and no resident credential. Indexing is
a normal request-scoped API route, triggered by the renderer's EXISTING live
JMAP push connection - so it reacts to each delivery/change rather than polling.
- lib/mail-index/binding.ts guarded require of the optional native binding
- lib/mail-index/paths.ts the VNCMAIL_DESKTOP_STORE_DIR gate + hashed paths
- lib/mail-index/store.ts schema, upsert, FTS5 search, encryption assertion
- lib/mail-index/extract.ts PURE JMAP-object -> document extractors
- lib/mail-index/jmap.ts minimal stateless server-side JMAP client
- lib/mail-index/key.ts per-job key fetch over the inherited fd
- lib/mail-index/reindex.ts the job + slot->account resolution
- electron/key-service.ts safeStorage wrap/unwrap, served over fd 3
- app/api/offline/reindex POST, event-driven + catch-up
- app/api/offline/search GET, the retrieval surface (hits + contextBlock)
- lib/mail-index-client.ts renderer client; StateChange -> index call
- components/settings/local-index-settings.tsx status + manual catch-up
Decisions worth knowing:
* `@signalapp/sqlcipher` is an OPTIONAL dependency with a guarded runtime
require. It publishes six N-API prebuilds and NO build sources, and both
Dockerfiles are node:24-alpine (musl, no matching prebuild) - as a hard
dependency it would break the production image and the integration fixture's
webmail container, neither of which wants this feature.
* Credentials come from the existing per-slot encrypted `jmap_stalwart_ctx`
cookie via lib/stalwart/credentials.ts - the same helper /api/settings and
/api/push/preview already use. It carries a ready-made header for basic AND
bearer accounts, so the indexer never touches the OAuth refresh-token cookie;
a server-side refresh would rotate a token into a response nobody reads and
silently log the user out.
* The encryption key crosses main -> server over an INHERITED FILE DESCRIPTOR,
never an environment variable: env is readable by any process running as the
same OS user, which would defeat using the OS keychain at all. Fetched per
job and zeroed after, so there is no long-lived key copy.
* safeStorage's Linux `basic_text` backend (no keyring) is treated as refusal,
not degradation - it "encrypts" with a hardcoded public password, which would
look like an encrypted mailbox while providing nothing.
getSelectedStorageBackend() is Linux-only and platform-guarded.
* Every store open asserts `PRAGMA cipher_version` returns a non-empty STRING,
not merely a row: a non-cipher binding returns ZERO ROWS, so a row-count check
would pass vacuously while writing the mailbox to disk in cleartext.
* Files are indexed by name/path/date/size only - NOT by extracted content.
Text extraction from arbitrary PDFs/office documents is a separate problem.
* Account-scoped composite keys `(jmap_account_id, content_type, id)` are kept
even though there is one file per account: one login exposes delegated/shared
JMAP accounts too, and JMAP ids are unique only within an account.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Corrects an architecture call I got wrong earlier in the session. I had said
CaProvider would live in the plugin. It cannot, for two independent reasons:
1. EJBCA's REST API authenticates with a CLIENT CERTIFICATE. A browser
cannot present one from fetch, and must not hold one anyway - the RA
credential is the authority to mint certificates, so putting it
anywhere script-reachable turns any XSS into a certificate factory.
2. Only the server can answer "does this person actually own this
address?" A browser asserting its own identity to a CA is not
authentication.
So: the plugin generates the keypair and CSR (private key never leaves the
device), and this layer decides which addresses the certificate may assert.
api.http.post is the bridge, and the fact that it forwards the user's JMAP
auth header is what makes the identity check possible at all.
The design decision worth calling out: the CSR is NOT trusted for identity,
and the route does not parse it to police what it asks for. It doesn't need
to. The route supplies the subject and the rfc822Name SAN itself from
addresses it verified independently; the CSR contributes only a public key
and proof of possession. A CSR hand-crafted to claim the CEO's address does
not have to be detected and rejected - the extension it asks for simply
never reaches the certificate.
That property depends entirely on EJBCA ignoring CSR-supplied subjects and
extensions, which is three checkboxes in the certificate profile. Added to
the runbook as the most important line in it, with a concrete verification
using a hostile CSR - because with those overrides ON, the enrolment route
still looks correct in review while issuing certificates for any address.
Identity comes from Stalwart via Identity/get, not from the auth cookie's
username. The cookie is encrypted and server-minted so it cannot be forged,
but it is still the wrong authority: the right answer to "may this person
have a signing certificate for this address" is held by the mail server
that already decides "may this person send from this address". Anything else
invents a second, weaker answer to a settled question.
It also handles two cases the cookie cannot:
- an alias the account legitimately sends as, which belongs ON the
certificate and which the cookie does not know about
- an administrative principal with no mailbox, which must get NOTHING.
Not hypothetical: admin@sandbox.vnc.de authenticates successfully and
has no mail session, so trusting the cookie would have issued it a
certificate for an address it cannot send from.
Wildcard identities (*@domain) are filtered out. Stalwart can legitimately
report one for an account allowed to send as anything in a domain, but it is
a capability, not an address - and a rfc822Name SAN of *@vnc.de is either
rejected by clients or, worse, honoured.
Other deliberate choices:
- Pins EJBCA's own chain for the mTLS connection instead of the public root
store. EJBCA serves a self-signed cert on that listener by design, and
rejectUnauthorized:false would be worse than either option - it would let
anything on the cluster network impersonate the CA and harvest CSRs.
- CA error bodies are logged server-side and replaced with generic messages.
An enrolment endpoint should not double as a way to probe CA config.
- DN component values are RFC 4514 escaped. The CN comes from a display
name; an unescaped comma or plus would inject additional RDNs.
- getCaProvider() returns null rather than throwing when unconfigured, so
the route 503s and nothing else is affected. Enrolment is opt-in; a
missing CA secret must not stop anyone reading their mail.
- revoke() is documented as needing to work when enrolment is broken. It is
the incident-response path, and a design that can only revoke through the
same path that issues is one outage from being unable to answer a key
compromise.
Typechecks clean. Not yet exercised against a live CA - the browser half of
C-08 (keypair + CSR generation in the plugin) and a real EJBCA to enrol
against are both still outstanding, so nothing here has issued a
certificate yet.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The overrideWarnings escape hatch added alongside the bundle scan was
API-only: an admin uploading a crypto plugin through the web form hit a
400 with canOverride and had no way to act on it, which left S/MIME and
PGP bundles uninstallable through the UI.
Hold the rejected file client-side and show the findings — pattern per
file — with "Install anyway" and "Cancel". Proceeding re-posts the same
file with overrideWarnings, so the decision stays explicit and lands in
the audit log. The route now echoes accepted findings back on success so
the confirmation says how many were waved through rather than reporting a
bare install.
Also replaces a dead `data.warnings` read with the live `findings` field;
the route never returned `warnings` on success, so that branch never ran.
Completes B-01.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two problems with the upload scanner, pulling in opposite directions.
It only scanned the entrypoint, so a bundle with `eval()` in a second
file passed outright — verified against a synthetic bundle whose
vendor/openpgp.js tripped three patterns while index.js stayed clean.
At the same time, a hard 400 on eval()/new Function()/innerHTML= makes
every crypto plugin uninstallable: minified openpgp.js and pkijs
legitimately contain all three. That blocks S/MIME and PGP entirely.
Scan every .js/.mjs in the bundle and return structured findings
({file, patterns[]}) plus canOverride, so the admin can see exactly what
tripped and where. An explicit overrideWarnings=true proceeds and writes
a plugin.install.scan_override audit entry recording which patterns in
which files were accepted — not merely that an override happened.
This route is already admin-authenticated, so the scan is defence in
depth against an accidental or compromised upload, not a trust boundary.
Treating it as the latter is what made crypto plugins uninstallable.
Also log the B-04 and B-01 divergences in vnc/VNC-CHANGES.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
If you have a large number of tags, getTagCounts would not be able to get the
unread counts because it did not respect maxCallsInRequest, even though the
value was actually read out, it was just ignored. There are also other places
where the limits were not respected.
Batching is now generalized in a helper that also takes maxObjectsInSet, which
also was ignored, into account and is applied to all functions.
In addition, the dev mock now also advertises and enforces the limits, so these
issues can get picked up during development.
Possible closes#699
Possibly closes#399
After a master-user impersonation handoff (GET /api/auth/impersonate) the server
swaps the slot-0 session cookie but the client's persisted account registry
(account-registry / auth-storage in localStorage) still lists the previous
account, so the top-left account chip keeps showing the old mailbox until a
manual sign-out. Redirect impersonation to /?impersonated=1 and add a headless
ImpersonationReconciler that drops the stale persisted account/auth state (and
server-derived caches) then reloads to a clean URL, so the app rehydrates empty
and re-derives the single account from the fresh session. Cookies untouched, so
the just-granted session survives. Runs exactly once.
Reported downstream: shukiv/jabali-panel#646.
Any transient failure used to end the session: the token route deleted
the refresh cookies on every non-OK answer from the OAuth endpoint,
refreshAccessToken logged out on any non-OK status or network error,
and the startup restore evicted the account and deleted its session
cookie. A server restart, a proxy hiccup, a Wi-Fi switch or a laptop
waking before the network is back all kicked the user out despite
"stay signed in".
Failures are now classified. Only a definitive rejection (400/401/403
from the OAuth endpoint, 401 from the token route) tears the session
down and deletes cookies, exactly as before. Network errors and 5xx
keep the session: the token refresh re-arms itself and retries every
~30 seconds until the server is back, and the startup restore keeps
the account, marked unreachable - the same treatment the rate-limit
carve-out (#104) already applies.
Token validity stays entirely server-enforced: the first definitive
401 after an outage still logs out as before.
Login header customization for white-label deployments, all defaults
preserve current behaviour:
- LOGIN_LOGO_MAX_HEIGHT / LOGIN_LOGO_MAX_WIDTH (any CSS length): the logo
box is otherwise a fixed 64x64 (w-16/h-16), which fits a wide wordmark to
~13px tall. When either is set, the fixed box is dropped and the logo
renders at the configured size.
- LOGIN_SHOW_HEADING / LOGIN_SHOW_SUBTITLE (default true): hide the
{appName} heading and/or the subtitle when the logo already reads as the
brand (e.g. a wordmark) and they'd be redundant.
Applied to the standard login header; wired through the existing config
registry (CONFIG_ENV_MAP) -> /api/config -> useConfig.
Refs #519.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The service worker hard-coded the notification icon and badge to the bundled /icon-192x192.png, so push notifications always showed the default Bulwark logo even when an admin had configured a custom PWA/favicon icon (which the manifest already honors via /api/pwa-icon).
Point both notifications at /api/pwa-icon/192, and make that endpoint fall back to the bundled default icon instead of returning 404 when no custom icon is set - so it always returns an app icon and the service worker (which can't run the custom-vs-default check itself) has a single stable URL.
Two opt-out branding/login flags, both default true (no behaviour change
for existing deployments):
- LOGIN_SHOW_TOTP=false hides the manual "I have a 2FA code" toggle on the
login form. Deployments that delegate auth to an external directory
(LDAP/OIDC) where 2FA lives in the IdP have no server-side TOTP, so the
toggle only ever leads to a failed login. Server-required TOTP
(totp_required, which auto-shows the field) is unaffected.
- LOGIN_SHOW_VERSION=false hides the build version in the login footer, so
the exact version isn't disclosed to unauthenticated visitors.
Wired through the existing config registry (CONFIG_ENV_MAP) → /api/config →
useConfig, matching the surrounding LOGIN_* options.
Refs #519.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cleans up the lint warnings the pre-commit hook surfaces, without any
behavioral change:
- Remove unused imports/vars/destructured props (parseISO, useEffect,
format, durMin, roles, daysInYear, ALLOWED_PLUGIN_FILES, continuesBefore,
isPushConnected, isSelected) and the now-unused parseDuration import.
- Drop three stale `// eslint-disable-next-line no-undef` directives that
no longer suppress anything (browser-navigation, smime/crypto-engine).
- recurrence-expansion.test.ts: replace 39 `as any` casts with a cast-only
`rule()` helper for partial recurrence-rule fixtures, typed access to
utcStart/utcEnd (now on CalendarEvent), and the source's
`Partial<CalendarEvent> & { excluded?: boolean }` for the excluded
override. No defaults are injected, so the expansion logic sees the same
partial rules as before (35 tests still green).
Remaining: 7 react-hooks/exhaustive-deps warnings are left as-is — adding
the missing deps changes effect/memo timing and needs per-hook review, not
a mechanical fix. tsc --noEmit clean; eslint 0 errors / 7 warnings.
Lets a per-brand authorize host front a single canonical issuer, so the
IdP token's `iss` stays constant for downstream validation while login
branding varies per domain. Discovery, token exchange and refresh keep
using OAUTH_ISSUER_URL.
Give email-002 ("Project Update - Q1 Review") a sender and CC with
"Lastname, Firstname" display names so Reply/Reply-All in dev mode
exercises the comma-in-name recipient case end to end.
Bernd Rodler
Bernd-Rodler