Stefan-Sanger
a4f17e2506
The mod_websocket continuation-frame patch accumulated fragments in an
unbounded dataBuffer with no size limit, fragment count limit, or
timeout. A buggy or malicious client sending endless continuation
frames without FIN could exhaust memory and degrade the entire server.
Add a configurable websocket_max_message_size option (default 2 MB)
that closes the connection with code 1009 ("Message too big") and
resets the buffer when exceeded. Also add wss-perf-analysis.md
documenting the WebSocket performance and drop investigation.
Part-of: <http://gitlab.vnc.biz/uxf/vnctalk-prosody/-/merge_requests/8>
126 lines
6.3 KiB
Markdown
126 lines
6.3 KiB
Markdown
# WebSocket Performance & Drop Analysis
|
||
|
||
## Executive Summary
|
||
|
||
User reports of **slow WebSocket connections** and **sudden drops** on the Prosody 13.0.6 installation have three root causes in this repo:
|
||
|
||
1. **Synchronous PostgreSQL reads on every message stanza** (`mod_vnc_fcm`, `mod_vnc_muc_fcm`) block Prosody's single event loop, causing global latency spikes.
|
||
2. **Unbounded WebSocket fragment buffer** (`mod_websocket.patch`) allows unbounded memory growth per connection with no timeout or size limit.
|
||
3. **Missing transport keepalive** means NATs, ingress controllers, and load balancers silently drop idle WebSocket connections after 60–300 s.
|
||
|
||
Secondary factors: SMACKS session limits are tight for mobile churn, and `trusted_proxies` is misconfigured for Kubernetes.
|
||
|
||
---
|
||
|
||
## Findings
|
||
|
||
### 1. Synchronous DB reads on the message hot path (HIGH IMPACT)
|
||
|
||
**Location:** `vnctalk/mod_vnc_fcm.lua`, `vnctalk/mod_vnc_muc_fcm.lua`
|
||
|
||
Both modules hook every `message/bare`, `message/full`, `pre-message/*`, and `muc-broadcast-message` event at priority 2. On **every** stanza they perform synchronous SQL lookups:
|
||
|
||
- `private_storage:get(userName)`
|
||
- `fcm_token_store:get(userName)`
|
||
- `vcard_storage:get(userName)` (via `getDisplayName()`)
|
||
|
||
For MUC messages, `mod_vnc_muc_fcm` iterates over **all room affiliations** and repeats the above lookups per affiliate. A single message in a 100-member room can trigger **hundreds** of blocking PostgreSQL queries inside Prosody's single-threaded Lua event loop.
|
||
|
||
**Result:** Any slowdown in PostgreSQL or spike in message volume blocks *all* connections (WebSocket, BOSH, c2s). Clients see latency, stalls, and eventual timeouts/drops.
|
||
|
||
### 2. Unbounded WebSocket fragment buffer (HIGH IMPACT — FIXED)
|
||
|
||
**Location:** `patches/mod_websocket.patch`
|
||
|
||
Upstream 13.0.6 removed continuation-frame support. The patch restores it by accumulating fragments in a `dataBuffer` table local to each connection. The original patch had **no** maximum size, fragment count limit, or timeout.
|
||
|
||
**Fix applied:** The patch now caps the accumulated message size via the configurable `websocket_max_message_size` option (default **2 MB**). If exceeded, the connection is closed with WebSocket close code 1009 ("Message too big") and the buffer is reset. This prevents unbounded memory growth from buggy or malicious clients that never send a FIN frame.
|
||
|
||
### 3. Missing WebSocket keepalive (HIGH IMPACT — easy fix)
|
||
|
||
**Location:** `config/prosody.cfg.lua.template`
|
||
|
||
The template has no keepalive configuration for the WebSocket transport:
|
||
|
||
- No `ping_interval` for `mod_ping` (loaded but only replies to client pings by default).
|
||
- No transport-level WebSocket ping/pong interval from Prosody.
|
||
- No TCP keepalive tuning.
|
||
|
||
Kubernetes ingress controllers (nginx, Traefik, AWS ALB) and corporate NATs typically drop idle TCP/WebSocket connections after **60–300 s**. When a client is idle (e.g., reading a chat), the connection is silently closed by the intermediary. The client sees a "sudden drop" only when it next tries to send.
|
||
|
||
### 4. SMACKS session limits too tight for mobile churn (MEDIUM IMPACT)
|
||
|
||
**Location:** `config/prosody.cfg.lua.template`, `config/startup.sh`
|
||
|
||
Defaults set in `startup.sh`:
|
||
|
||
```
|
||
smacks_hibernation_time = 300
|
||
smacks_max_old_sessions = 10
|
||
```
|
||
|
||
Mobile clients frequently background/foreground and reconnect. If a user has more than 10 old sessions within the 5-minute hibernation window, the oldest are evicted. The client must perform a full rebind instead of a fast SMACKS resume, which feels like a slow reconnect or a drop.
|
||
|
||
### 5. trusted_proxies misconfigured for Kubernetes (LOW DIRECT IMPACT)
|
||
|
||
**Location:** `config/prosody.cfg.lua.template`
|
||
|
||
```lua
|
||
trusted_proxies = { "127.0.0.1" }
|
||
```
|
||
|
||
In Kubernetes the ingress controller is **not** localhost. Prosody ignores `X-Forwarded-For` and sees every client as the ingress IP. This breaks IP-based diagnostics and could interact with future rate-limiting, but does not directly cause drops today.
|
||
|
||
---
|
||
|
||
## Recommended Fixes
|
||
|
||
### Immediate (low effort, high reward)
|
||
|
||
1. **Enable proactive XMPP pings** to keep the WebSocket alive through NATs/LBs. Add to the global config:
|
||
```lua
|
||
ping_interval = 60
|
||
ping_timeout = 120
|
||
```
|
||
Also align the ingress controller idle timeout (e.g., nginx `proxy-read-timeout`) to be > 120 s.
|
||
|
||
2. **Add bounds to `mod_websocket.patch`** — **DONE**. The patch now enforces a configurable `websocket_max_message_size` (default 2 MB). Connections exceeding the limit are closed with code 1009 and the buffer is reset.
|
||
|
||
3. **Raise SMACKS limits for mobile clients**. In `startup.sh` or the template:
|
||
```lua
|
||
smacks_max_old_sessions = 50
|
||
```
|
||
Consider whether `smacks_hibernation_time = 300` is appropriate; some mobile deployments use 600–1800 s.
|
||
|
||
### Short-term (requires code changes)
|
||
|
||
4. **Cache FCM token lookups in memory**. The `mod_vnc_fcm` modules read the same private-data / token rows repeatedly. Add an in-memory LRU cache (e.g., 5-minute TTL) around:
|
||
- `getNotifyOptionsForUser()`
|
||
- `getDisplayName()`
|
||
|
||
This avoids synchronous PostgreSQL round-trips on the hot path.
|
||
|
||
5. **Defer or batch MUC FCM notifications**. In `mod_vnc_muc_fcm`, iterating all affiliations and doing a DB read + HTTP request per member inside `muc-broadcast-message` blocks the room. Move the loop into an async timer or a background queue so the broadcast event returns immediately.
|
||
|
||
6. **Fix `trusted_proxies`**. Set it to the cluster CIDR(s) or the ingress controller's source IPs, e.g.:
|
||
```lua
|
||
trusted_proxies = { "127.0.0.1", "10.0.0.0/8" }
|
||
```
|
||
|
||
### Diagnostic
|
||
|
||
7. **Watch `log_slow_events`**. The module is enabled with a 1.5 s threshold. If PostgreSQL or the FCM path is the bottleneck, the logs will show `Slow event 'message/bare'` or `Slow event 'muc-broadcast-message'`. Use this to confirm fix #4/#5 before and after.
|
||
|
||
---
|
||
|
||
## Files Involved
|
||
|
||
| File | Relevance |
|
||
|------|-----------|
|
||
| `patches/mod_websocket.patch` | Unbounded fragment accumulation |
|
||
| `vnctalk/mod_vnc_fcm.lua` | Synchronous DB reads on every message |
|
||
| `vnctalk/mod_vnc_muc_fcm.lua` | Synchronous DB reads + loop over MUC affiliations |
|
||
| `config/prosody.cfg.lua.template` | Missing `ping_interval`, wrong `trusted_proxies` |
|
||
| `config/startup.sh` | `SMACKS_MAX_UNACKED_STANZAS`, `SMACKS_HIBERNATION_TIME` defaults |
|
||
| `vnctalk/mod_log_slow_events.lua` | Already logging; verify thresholds |
|