# WebSocket Performance & Drop Analysis

## Executive Summary

User reports of **slow WebSocket connections** and **sudden drops** on the Prosody 13.0.6 installation have three root causes in this repo:

1. **Synchronous PostgreSQL reads on every message stanza** (`mod_vnc_fcm`, `mod_vnc_muc_fcm`) block Prosody's single event loop, causing global latency spikes.
2. **Unbounded WebSocket fragment buffer** (`mod_websocket.patch`) allows unbounded memory growth per connection with no timeout or size limit.
3. **Missing transport keepalive** means NATs, ingress controllers, and load balancers silently drop idle WebSocket connections after 60–300 s.

Secondary factors: SMACKS session limits are tight for mobile churn, and `trusted_proxies` is misconfigured for Kubernetes.

---

## Findings

### 1. Synchronous DB reads on the message hot path (HIGH IMPACT)

**Location:** `vnctalk/mod_vnc_fcm.lua`, `vnctalk/mod_vnc_muc_fcm.lua`

Both modules hook every `message/bare`, `message/full`, `pre-message/*`, and `muc-broadcast-message` event at priority 2. On **every** stanza they perform synchronous SQL lookups:

- `private_storage:get(userName)`
- `fcm_token_store:get(userName)`
- `vcard_storage:get(userName)` (via `getDisplayName()`)

For MUC messages, `mod_vnc_muc_fcm` iterates over **all room affiliations** and repeats the above lookups per affiliate. A single message in a 100-member room can trigger **hundreds** of blocking PostgreSQL queries inside Prosody's single-threaded Lua event loop.

**Result:** Any slowdown in PostgreSQL or spike in message volume blocks *all* connections (WebSocket, BOSH, c2s). Clients see latency, stalls, and eventual timeouts/drops.

### 2. Unbounded WebSocket fragment buffer (HIGH IMPACT — FIXED)

**Location:** `patches/mod_websocket.patch`

Upstream 13.0.6 removed continuation-frame support. The patch restores it by accumulating fragments in a `dataBuffer` table local to each connection. The original patch had **no** maximum size, fragment count limit, or timeout.

**Fix applied:** The patch now caps the accumulated message size via the configurable `websocket_max_message_size` option (default **2 MB**). If exceeded, the connection is closed with WebSocket close code 1009 ("Message too big") and the buffer is reset. This prevents unbounded memory growth from buggy or malicious clients that never send a FIN frame.

### 3. Missing WebSocket keepalive (HIGH IMPACT — easy fix)

**Location:** `config/prosody.cfg.lua.template`

The template has no keepalive configuration for the WebSocket transport:

- No `ping_interval` for `mod_ping` (loaded but only replies to client pings by default).
- No transport-level WebSocket ping/pong interval from Prosody.
- No TCP keepalive tuning.

Kubernetes ingress controllers (nginx, Traefik, AWS ALB) and corporate NATs typically drop idle TCP/WebSocket connections after **60–300 s**. When a client is idle (e.g., reading a chat), the connection is silently closed by the intermediary. The client sees a "sudden drop" only when it next tries to send.

### 4. SMACKS session limits too tight for mobile churn (MEDIUM IMPACT)

**Location:** `config/prosody.cfg.lua.template`, `config/startup.sh`

Defaults set in `startup.sh`:

```
smacks_hibernation_time = 300
smacks_max_old_sessions = 10
```

Mobile clients frequently background/foreground and reconnect. If a user has more than 10 old sessions within the 5-minute hibernation window, the oldest are evicted. The client must perform a full rebind instead of a fast SMACKS resume, which feels like a slow reconnect or a drop.

### 5. trusted_proxies misconfigured for Kubernetes (LOW DIRECT IMPACT)

**Location:** `config/prosody.cfg.lua.template`

```lua
trusted_proxies = { "127.0.0.1" }
```

In Kubernetes the ingress controller is **not** localhost. Prosody ignores `X-Forwarded-For` and sees every client as the ingress IP. This breaks IP-based diagnostics and could interact with future rate-limiting, but does not directly cause drops today.

---

## Recommended Fixes

### Immediate (low effort, high reward)

1. **Enable proactive XMPP pings** to keep the WebSocket alive through NATs/LBs. Add to the global config:
   ```lua
   ping_interval = 60
   ping_timeout = 120
   ```
   Also align the ingress controller idle timeout (e.g., nginx `proxy-read-timeout`) to be > 120 s.

2. **Add bounds to `mod_websocket.patch`** — **DONE**. The patch now enforces a configurable `websocket_max_message_size` (default 2 MB). Connections exceeding the limit are closed with code 1009 and the buffer is reset.

3. **Raise SMACKS limits for mobile clients**. In `startup.sh` or the template:
   ```lua
   smacks_max_old_sessions = 50
   ```
   Consider whether `smacks_hibernation_time = 300` is appropriate; some mobile deployments use 600–1800 s.

### Short-term (requires code changes)

4. **Cache FCM token lookups in memory**. The `mod_vnc_fcm` modules read the same private-data / token rows repeatedly. Add an in-memory LRU cache (e.g., 5-minute TTL) around:
   - `getNotifyOptionsForUser()`
   - `getDisplayName()`

   This avoids synchronous PostgreSQL round-trips on the hot path.

5. **Defer or batch MUC FCM notifications**. In `mod_vnc_muc_fcm`, iterating all affiliations and doing a DB read + HTTP request per member inside `muc-broadcast-message` blocks the room. Move the loop into an async timer or a background queue so the broadcast event returns immediately.

6. **Fix `trusted_proxies`**. Set it to the cluster CIDR(s) or the ingress controller's source IPs, e.g.:
   ```lua
   trusted_proxies = { "127.0.0.1", "10.0.0.0/8" }
   ```

### Diagnostic

7. **Watch `log_slow_events`**. The module is enabled with a 1.5 s threshold. If PostgreSQL or the FCM path is the bottleneck, the logs will show `Slow event 'message/bare'` or `Slow event 'muc-broadcast-message'`. Use this to confirm fix #4/#5 before and after.

---

## Files Involved

| File | Relevance |
|------|-----------|
| `patches/mod_websocket.patch` | Unbounded fragment accumulation |
| `vnctalk/mod_vnc_fcm.lua` | Synchronous DB reads on every message |
| `vnctalk/mod_vnc_muc_fcm.lua` | Synchronous DB reads + loop over MUC affiliations |
| `config/prosody.cfg.lua.template` | Missing `ping_interval`, wrong `trusted_proxies` |
| `config/startup.sh` | `SMACKS_MAX_UNACKED_STANZAS`, `SMACKS_HIBERNATION_TIME` defaults |
| `vnctalk/mod_log_slow_events.lua` | Already logging; verify thresholds |