a622e3755bba38a4e7bd933c16bc80fe1e2e4f36
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Bernd Rodler
|
62b0455388 |
feat(ai): OpenCode provider management (B4) — "any LLM OpenCode supports", from inside VNCmail+
Before this, the OpenCode class could only use providers already authenticated
via its own CLI (opencode auth login) — this app could pick a MODEL, never add
a PROVIDER. That is the one thing standing between "OpenCode integration" and
the actual ask: any LLM it supports, added from here.
New GET/PUT/DELETE /api/ai/opencode/providers, backed by GET /provider (every
provider OpenCode knows — 180 on a real run) and GET /provider/auth (which
auth method each accepts). New "Manage providers" panel in the OpenCode
settings section: search, add a key, remove one.
Scoped to API-key auth only, deliberately — recorded in lib/ai/opencode.ts's
module comment. `PUT /auth/{id}` with `{type:'api', key}` is one HTTP call
with a schema-verified shape. OAuth entries in /provider/auth need a browser
redirect + callback this app has no page for, and some carry interactive
prompts beyond a single form (GitHub Copilot's deployment-type picker) — real
scope for later, not something to half-build. OAuth-only providers are still
LISTED, just marked "Browser sign-in only" rather than hidden, so the picker
stays honest about what it can't do here.
A real finding from testing this against opencode's actual behaviour rather
than trusting a 200: NOT EVERY PROVIDER BECOMES CONNECTED FROM A BARE API KEY.
Snowflake Cortex needs SNOWFLAKE_ACCOUNT alongside its token; a single key
field silently leaves it stored-but-unconnected with no error from the PUT
itself. Worse, the provider's own `env` array length does not predict this —
Azure also needs two env vars and DOES connect from one key. There is no
reliable way to know in advance, so the route now VERIFIES by re-listing
providers after the write and reports plainly when a key was accepted but the
provider still isn't connected, rather than reporting the PUT's own success.
Verified live end-to-end, twice: once confirming a simple single-field
provider connects and can be removed cleanly, once confirming the honest
"stored but not connected" case is real and detected, not theoretical.
Cleaned up every throwaway credential from this machine's real opencode
config afterwards (checked auth.json directly, not just this app's view of it).
17 new/updated unit tests. Gate: tsc clean, eslint clean, 2527/2527 tests, build clean.
|
||
|
Bernd Rodler
|
bd778adf12 |
feat(electron): supervise a password-protected opencode server (B1+B3)
B1 — LIFECYCLE. The OpenCode class previously required the user to remember to run `opencode serve` in a terminal before opening their mail app, and again after every reboot; in practice that means the feature quietly stops existing. The desktop shell now owns it: finds the binary (OPENCODE_BIN, then ~/.opencode/bin — its installer's default, which is NOT on the PATH a macOS GUI app inherits, so PATH alone finds nothing for most users), starts it on a free port, restarts up to 3 times if it dies, and kills it on quit. Absent binary = the class simply stays unavailable, no error. B3 — SECURITY. opencode's own startup warns "OPENCODE_SERVER_PASSWORD is not set; server is unsecured" — without one, any local process can drive the agent. A per-launch password is now always generated (never persisted: the server dies with the app, so a durable secret would be pure liability) and handed to the standalone server alongside the base URL. The auth scheme is worth recording because it is NOT in opencode's own OpenAPI spec, which declares no securitySchemes at all: HTTP Basic with the username EXACTLY `opencode`. Verified against 1.18.14 by trying them — an empty username, an arbitrary one, Bearer, and every plausible custom header all 401 with the correct password. Pinned by a unit test that decodes the header, so a future refactor can't silently drop it. Verified live against a real password-protected server on 4097: authenticated discovery + prompt round-tripped, AND the same call with no password was rejected — proving the auth is real rather than decorative. Also removed now-stale guidance: the 503 no longer says "start one with opencode serve", because the app does that; it says to install the CLI. Gate: tsc clean, eslint clean, build clean, 2521/2522 tests. The one failure is lib/__tests__/jmap-client-resilience.test.ts's onConnectionChange timing flake — byte-identical to what is already running in prod (git diff vs origin/main for that file and lib/jmap/ is empty), pre-existing, and unrelated to anything here. |
||
|
Bernd Rodler
|
87336981d3 |
feat(mail-index): 1-year retention by default + a recency retrieval leg
The two things that made real questions fail against a correctly-populated index, both fixed at the root. RETENTION (A1). `INDEX_WINDOW_DAYS = 30` was not merely a fetch bound — catch-up also PRUNED mail older than it, so "summarise everything from July" was unanswerable in August because the rows had been deleted, while the UI said only that nothing matched. Now a user-visible setting (Settings → About & Data): 30 days / 3 months / 1 year / everything, defaulting to 1 YEAR per the product owner. The window bounds the fetch AND the prune from one value so the two can never disagree and delete what was just written; "everything" skips pruning entirely rather than falling back to some default bound. The per-pass ceiling scales with the window (500/30d, hard cap 20k) because 500 messages is right for a month and nonsense for "everything". Email/query now omits the `after` filter entirely when unbounded — Stalwart rejects a malformed filter rather than treating `undefined` as unset. RECENCY (A2). Keyword search structurally cannot answer a question about WHEN: bm25 ranks by term overlap, so "who sent the last email" matches documents containing the word "last", and "all mails in July" matches documents containing "July" — not documents dated in July. Both were asked by a real user and both failed. New lib/mail-index/recency.ts detects time intent (English + German, since the UI ships German) and turns it into a date RANGE; new MailIndex.recent() answers it with an ordered scan over the already-indexed `occurred_at`. The route ADDS these hits to the keyword hits rather than replacing them — "what did the last mail from Anna say" is both kinds of question at once. Timezone subtlety worth knowing: bounds are built from LOCAL calendar boundaries and serialised as UTC instants, so "July" covers the user's July. A mail at 00:30 local on 1 July belongs to it even though its stored UTC timestamp reads 30 June. My first test asserted the ISO string prefix, which would have enshrined the opposite and passed only in UTC — the tests now assert the local-time property instead. SCOPE, stated by the product owner and now enforced structurally: the assistant only ever sees the mailbox the user is signed in to. Both retrieval legs resolve the active account (local leg by cookie slot, server leg by the session's own JMAP account); there is deliberately no fan-out across connected or shared mailboxes, and adding one would be a policy change, not a feature. Gate: tsc clean, eslint clean, 2520/2520 tests (8 new for recency intent), build clean. |
||
|
Bernd Rodler
|
98dcd3b1e9 |
feat(ai): OpenCode provider class; fix retrieval reading the wrong account's index
Three things, all from running the real thing rather than trusting a status code.
1. OpenCode as a 4th AI class (lib/ai/opencode.ts + app/api/ai/opencode/*).
A locally-running `opencode serve` — the same runtime Paperclip drives as
an adapter. Its appeal over a BYOK profile is precisely what was broken
before: opencode owns provider auth itself, so there is NO api key for
this app to hold, and it reports a REAL model list (25 on this machine)
instead of asking the user to type an exact provider-specific model id
from memory. Typing "Sonnet 5" into a free-text box and getting a bare
"Provider returned 401" is the failure this removes.
IMPORTANT trap, documented in the module header and pinned by a test:
opencode is NOT OpenAI-compatible. `/v1/models` and `/v1/chat/completions`
both answer 200 — because a web-UI catch-all serves index.html for ANY
unknown path. I built the first version against that assumed compatibility
on the strength of two 200s and had to throw it away once I read a body.
Every probe now validates the parsed shape and content-type, never the
status alone. The real API is GET /api/model + POST /session +
POST /session/{id}/message, and the reply's `reasoning` parts are stripped
so a model's private chain of thought can never surface as the answer.
Proxied through our own backend (like the `server` class) because the
desktop renderer's origin is a random port that changes every launch;
same-origin sidesteps opencode's CORS allowlist entirely. Loopback-only by
construction: a non-loopback OPENCODE_BASE_URL is refused, since "local,
no keys, nothing leaves the device" is the whole point of this class.
2. Retrieval read the WRONG ACCOUNT'S index. The indexer writes under the
active account's cookie slot (catchUpIndex passes it) but fetchLocalLeg
omitted `?slot=`, so search resolved to whichever account the multi-slot
resolver found first. Single-account installs never noticed; a real
multi-account/shared-mailbox setup reads an empty store every time. Both
call sites now pass the active slot.
3. "No local mail index available in this session" was shown even when the
index existed and simply matched nothing — actively misleading, and it
masked the missing-SESSION_SECRET bug for hours. AskResult now carries
retrievalState ('augmented' | 'no-match' | 'no-index') and the two cases
get different words: build the index, versus rephrase (with the honest
caveat that keyword search answers content questions better than recency
ones like "the last mail").
Verified live against real opencode 1.18.14: discovery found 25 models and a
real prompt round-tripped the exact expected answer through the real helper
code, not curl. Gate: tsc clean, eslint clean, 2512/2512 unit tests (10 new,
incl. one that fails if the HTML catch-all is ever accepted as an API), build clean.
|
||
|
Bernd Rodler
|
2dc224e882 |
feat(ai): local LLM auto-discovery — find a running Ollama, suggest connecting
New lib/ai/local-discovery.ts: one /api/tags query against the loopback
addresses Ollama binds to (127.0.0.1/localhost), no follow-up /api/show
round trips needed — the tags response already carries capabilities, size,
and parameter_size, enough to recommend a default model. Picks the
smallest non-"thinking" chat-capable model for the fastest first response
("Connect" pre-fills provider+baseUrl+model in one click), and separately
surfaces the largest as a "higher quality" alternative.
New banner in ai-assistant-settings.tsx: fires when Local isn't yet
configured, offers one-click Connect or a persisted "Not now" dismissal.
13 new unit tests using this machine's actual Ollama /api/tags response
(11 real installed models — qwen2.5:32b, llama3.2, deepseek-r1 x2,
gemma4 x3, hermes3, qwen3, qwen3.5, nomic-embed-text) as literal fixtures,
per the explicit instruction to use this machine as the test case:
confirms exactly one query is required, the heuristic recommends
llama3.2:latest (fastest) / qwen2.5:32b (largest) on this real fleet,
never recommends an embedding-only model, and degrades correctly when a
candidate base URL is unreachable.
Full QA gate: tsc clean, eslint clean, 2498/2498 tests passing, build clean.
Also live-verified in a real browser session against this machine's real
Ollama — the banner rendered with exactly these two model names.
|
||
|
Bernd Rodler
|
30e5059b94 |
feat(admin): build the AI Policy console (§6) — approved, spec now implemented
New admin tab "AI" (app/(main)/admin/_tabs/ai-policy.tsx): provider-class toggles, server model allow-list, BYOK provider allow-list, seats/usage (front-end for the already-real lib/ai/entitlement.ts), retrieval on/off, consent text + version bump. Real backend, not cosmetic: AiConsoleConfig persisted via config-manager (lib/ai/types.ts, ai-policy.json in the CONFIG dir). New GET/PUT /api/admin/ai/policy. Enforcement wired at every real chokepoint, not just the picker: /api/ai/server/chat checks classesEnabled.server and the model allow-list, /api/ai/retrieve checks retrievalEnabled, /api/ai/server/models filters by allow-list. GET /api/ai/policy folds classesEnabled into the classes list clients see. Resolved the spec's 3 open questions as recommended: BYOK allow-list stays client-side/advisory (wired into ai-assistant-settings.tsx's addProfile), tier picker stays cosmetic, master aiAssistantEnabled toggle stays in the existing Policy tab (this tab links to it instead of duplicating it). Defaults preserve today's behavior exactly (classesEnabled/allowlists all start empty/null) — turning this on changes nothing until an admin touches it. |
||
|
Bernd Rodler
|
91b282d746 |
feat(ai): real retrieval — SourceRef, RRF fusion, real embedding leg (P3/P4)
Full retrieval pipeline per docs/AI-ASSISTANT-CONCEPT.md §7/§8.1, real end to end, not mocked: - lib/ai/retrieval/types.ts: SourceRef + RetrieverAdapter schema. Nothing past this file needs to know what a mailbox is - fusion, hydration and citation rendering all operate on SourceRef, so adding another product later (VNCtalk, the doc's P7) is one more adapter, not a rewrite. - lib/ai/retrieval/fusion.ts: Reciprocal Rank Fusion, score = Σ1/(k+rank), k=60. Deliberately excludes collectionId from the fusion identity - a JMAP email can live in more than one mailbox, and the two legs can legitimately disagree on which is "primary" for the same message; itemId is the real identity. 5 unit tests, including that exact double-count case. - lib/ai/retrieval/mail-embeddings.ts: the server embedding leg. Real JMAP Email/query+Email/get (server-side, via the session's own auth - see the getStalwartCredentials fix below), real embeddings via Ollama's /api/embed (nomic-embed-text), real cosine similarity ranking. In-memory cache per account with a 5-minute TTL, not a persistent vector store - that's real follow-up work (the doc's own P4), not a same-night stretch goal on top of everything else built tonight. - app/api/ai/retrieve/route.ts: wires it together. ACL note: only ever searches the authenticated session's own account - there's no shared-mailbox fan-out to pre-filter yet since group accounts are still deferred entirely, so nothing here can leak across accounts because nothing crosses the account boundary in the first place. - lib/ai/local-client.ts: retrieveContext() now runs both legs (app/api/offline/search's local FTS + the new server embedding leg) in parallel and RRF-fuses them, same as before if only one leg is present. Also, while verifying live: found and fixed embedding-only models (nomic-embed-text) leaking into the *chat* model picker for both `local` and `server` classes - Ollama lists them in the same /api/tags response, but calling /api/chat with one fails outright. Filtered by `capabilities` (fails open if absent, for older Ollama). Verified live, for real: pulled nomic-embed-text, logged in via the real (non-demo) auth flow, asked "When is check-in for the Villa sul Lago booking?" against the seeded mock inbox - got back "Check-in ... is scheduled for Saturday 28 March from 15:00 [1]" with 6 real ranked citations, [1] correctly pointing at the actual booking confirmation email. Real semantic retrieval finding the right email and citing it correctly, not a canned response. Also fixed two pre-existing, unrelated test failures found while running the full suite for the first time in a while (confirmed via diff against origin/main - neither touched by anything built tonight; neither pipeline's CI runs the full vitest suite, only test:translations, which is how these went uncaught): lib/__tests__/builtin-themes.test.ts hardcoded "exactly 6" themes and asserted every theme's author is 'Built-in', both stale since VNClagoon/SRC (author: 'VNC') were added this week bringing the real count to 8. Left the also-pre-existing, timing-sensitive jmap-client-resilience.test.ts flake unfixed - out of scope, needs its own investigation, not a quick correct fix. Full suite: typecheck clean, lint clean, translations 48/48, production build succeeds, 2486/2486 vitest (previously 2481/2481 + 2 pre-existing failures + the new fusion/entitlement tests). |
||
|
Bernd Rodler
|
dda7adf565 |
feat(ai): multi-key BYOK, real server class, real entitlement enforcement
Three pieces built together tonight since they're naturally linked (the
server-class proxy is the real entitlement enforcement chokepoint):
1. Multi-key BYOK (public class): several named provider profiles
(name/baseUrl/model), each with its own key in lib/ai/key-store.ts
(keyed by profile id, not a single fixed 'public' slot). The "Try it"
pane lets you pick which saved profile answers each question - not one
fixed default.
2. `server` class, real: app/api/ai/server/{models,chat} proxy through
this app's own backend to AI_SERVER_BASE_URL - same-origin from the
browser, no CORS/OLLAMA_ORIGINS story at all, standing in tonight for
VNC's EU/CH-hosted infra with the real Ollama on this Mac (swapping to
the real instance tomorrow is a config change).
3. Real entitlement enforcement (lib/ai/entitlement.ts), scoped to `server`
only (not local/public, per the 2026-08-05 decisions): checkAndAssignSeat()
re-validates on every /api/ai/server/chat call - first use auto-assigns a
seat if any remain, further calls from an unlicensed user get a 402 with
a specific reason. recordUsage() appends to an append-only metering
ledger (timestamp/user/model/tokens/latency) that IS the billing record.
Admin data endpoints at /api/admin/ai/entitlement (seat total, revoke) -
the visual admin console is a separate, not-yet-built task.
Two real bugs found and fixed during verification, not just claimed fixed:
- /api/ai/policy never actually added 'server' to entitlement.classes even
when AI_SERVER_BASE_URL was set (only the type comment was updated) - the
Server radio option silently never appeared until this was caught live.
- The new routes used readStalwartAuthContext(0) (hardcoded slot, SSO/reauth-
specific) instead of getStalwartCredentials() (the general multi-slot
session resolver every other authenticated route uses) - reachable but
wrong, and would have hidden a real auth gap behind "works on my slot".
Verified end-to-end for real: built + ran the actual server, logged in via
the real (non-demo) auth flow, selected Server, listed the real Ollama
models through the proxy, asked "Reply with exactly the words: SERVER CLASS
WORKS" and got back exactly that - plus confirmed on disk (not just in the
UI) that data/admin-state/ai-entitlement.json recorded the seat assignment
and ai-metering.jsonl recorded real prompt/completion token counts and
latency from the actual model call. Rejection-path logic (seat limit
reached, zero seats configured, revocation) covered by 5 new unit tests
rather than a second live round trip. Full suite: typecheck clean, lint
clean, translations 48/48, production build succeeds.
|
||
|
Bernd Rodler
|
5d7ae230ce |
feat(ai): real local Ollama chat + BYOK public provider
Decisions 2026-08-05 evening (reprioritizing docs/AI-ASSISTANT-CONCEPT.md's original P1/P2 server-first sequencing to local-first, since a real Ollama instance already runs on this Mac with a full model set): - `local` ships free, no entitlement check — always available wherever supportsLocalLlm() is true. - `public` (BYOK) is available too, explicitly unmonitored for now — no seats/metering/consent backend. This reverses the concept doc's decision #1 (server-side-only key custody): the client holds its own key, matching vncmail-native's existing pattern. - `server` (VNC-hosted) stays unwired client-side; that infra is "this MacBook tonight, the dev k8s cluster tomorrow." New: - lib/ai/local-client.ts: listLocalModels/testLocalConnection/chatLocal/ chatPublic, ported near-verbatim from vncmail-native's proven src/api/ai.ts. Direct browser-side fetch, not proxied through this app's own server — a server-side proxy would reach the *server's* loopback, not the user's own laptop, which defeats the point of "local" once this app is hosted remotely. - lib/ai/key-store.ts: client-held BYOK storage (localStorage — this repo's existing convention for client state, no OS keychain reachable from a browser tab). - lib/ai/local-settings.ts: isolated persistence for provider/model/base-URL choices. Deliberately NOT folded into stores/settings-store.ts, which has a hand-maintained export/import enumeration this prototype-scope state doesn't belong in yet. - Retrieval reuses this app's own already-built app/api/offline/search (encrypted SQLite/FTS5 mail index) as context when available, and degrades to unaugmented chat — not an error — when it 404s/503s (no index in this session, e.g. plain browser rather than Electron). Rewrote components/settings/ai-assistant-settings.tsx: provider picker, local runtime config (base URL, model list/refresh, test connection with a CORS-aware diagnostic per the concept doc's own note on the browser row), public BYOK config (base URL, model, key, client-side consent toggle), and a working Ask box. Verified: typecheck clean, lint clean, translations pass, production build succeeds. Live-tested against the real Ollama on this machine (confirmed running: qwen2.5:32b, gemma4, deepseek-r1, llama3.2, hermes3, qwen3) via a local server + demo-mode session — admin flag round-trips correctly, the pane renders both provider options, and the CORS-diagnostic path fires correctly on a real (if here environment-sandboxed, not Ollama-side) connection failure. Full success end-to-end still wants a real, unsandboxed browser tab against this Mac's loopback to close out. |
||
|
Bernd Rodler
|
2a35019b21 |
feat(ai): P0 client scaffolding — capability flags, settings pane, policy fetch
Per docs/AI-ASSISTANT-CONCEPT.md §12, P0 is deliberately generation-free: prove platform gating and the policy round trip before any model exists behind it. No provider is called anywhere in this change. - lib/platform-capabilities.ts: supportsLocalLlm/localLlmNeedsCorsSetup, mirroring the same-named module in vncmail-native so the capability contract (§3, §11) reads identically on both clients. Web+Electron only here — mobile is a separate codebase. - lib/ai/types.ts: AiPolicy/AiEntitlement schema, locked in now per decision #4 (entitlement from day one — cheap now, a live-tenant migration later). - app/api/ai/policy/route.ts: GET, unauthenticated (users read this, like /api/admin/policy). Composes the real FeatureGates.aiAssistantEnabled toggle with a hardcoded unlicensed entitlement — there's no seats/billing backend yet (P2), so nothing here can honestly claim otherwise. - components/settings/ai-assistant-settings.tsx: fetches that policy, shows a real (not fake) locked/unlicensed state. No model config UI yet — there is nothing real to configure until P1/P2/P5 land. - New admin FeatureGates.aiAssistantEnabled (default false, like pluginsEnabled): the tab is entirely hidden until an admin opts in, so no existing install suddenly sees a tab that does nothing. Verified: typecheck clean, lint clean, translations test passes (48/48), full production build succeeds with /api/ai/policy compiled in. |