vnctalk-prosody/wss-perf-analysis.md

WebSocket Performance & Drop Analysis

Executive Summary

User reports of slow WebSocket connections and sudden drops on the Prosody 13.0.6 installation have three root causes in this repo:

1. Synchronous PostgreSQL reads on every message stanza (mod_vnc_fcm, mod_vnc_muc_fcm) block Prosody's single event loop, causing global latency spikes.
2. Unbounded WebSocket fragment buffer (mod_websocket.patch) allows unbounded memory growth per connection with no timeout or size limit.
3. Missing transport keepalive means NATs, ingress controllers, and load balancers silently drop idle WebSocket connections after 60–300 s.

Secondary factors: SMACKS session limits are tight for mobile churn, and trusted_proxies is misconfigured for Kubernetes.

---

Findings

1. Synchronous DB reads on the message hot path (HIGH IMPACT)

Location: vnctalk/mod_vnc_fcm.lua, vnctalk/mod_vnc_muc_fcm.lua

Both modules hook every message/bare, message/full, pre-message/*, and muc-broadcast-message event at priority 2. On every stanza they perform synchronous SQL lookups:

• private_storage:get(userName)
• fcm_token_store:get(userName)
• vcard_storage:get(userName) (via getDisplayName())

For MUC messages, mod_vnc_muc_fcm iterates over all room affiliations and repeats the above lookups per affiliate. A single message in a 100-member room can trigger hundreds of blocking PostgreSQL queries inside Prosody's single-threaded Lua event loop.

Result: Any slowdown in PostgreSQL or spike in message volume blocks all connections (WebSocket, BOSH, c2s). Clients see latency, stalls, and eventual timeouts/drops.

2. Unbounded WebSocket fragment buffer (HIGH IMPACT — FIXED)

Location: patches/mod_websocket.patch

Upstream 13.0.6 removed continuation-frame support. The patch restores it by accumulating fragments in a dataBuffer table local to each connection. The original patch had no maximum size, fragment count limit, or timeout.

Fix applied: The patch now caps the accumulated message size via the configurable websocket_max_message_size option (default 2 MB). If exceeded, the connection is closed with WebSocket close code 1009 ("Message too big") and the buffer is reset. This prevents unbounded memory growth from buggy or malicious clients that never send a FIN frame.

3. Missing WebSocket keepalive (HIGH IMPACT — easy fix)

Location: config/prosody.cfg.lua.template

The template has no keepalive configuration for the WebSocket transport:

• No ping_interval for mod_ping (loaded but only replies to client pings by default).
• No transport-level WebSocket ping/pong interval from Prosody.
• No TCP keepalive tuning.

Kubernetes ingress controllers (nginx, Traefik, AWS ALB) and corporate NATs typically drop idle TCP/WebSocket connections after 60–300 s. When a client is idle (e.g., reading a chat), the connection is silently closed by the intermediary. The client sees a "sudden drop" only when it next tries to send.

4. SMACKS session limits too tight for mobile churn (MEDIUM IMPACT)

Location: config/prosody.cfg.lua.template, config/startup.sh

Defaults set in startup.sh:

smacks_hibernation_time = 300
smacks_max_old_sessions = 10

Mobile clients frequently background/foreground and reconnect. If a user has more than 10 old sessions within the 5-minute hibernation window, the oldest are evicted. The client must perform a full rebind instead of a fast SMACKS resume, which feels like a slow reconnect or a drop.

5. trusted_proxies misconfigured for Kubernetes (LOW DIRECT IMPACT)

Location: config/prosody.cfg.lua.template

trusted_proxies = { "127.0.0.1" }

In Kubernetes the ingress controller is not localhost. Prosody ignores X-Forwarded-For and sees every client as the ingress IP. This breaks IP-based diagnostics and could interact with future rate-limiting, but does not directly cause drops today.

---

Recommended Fixes

Immediate (low effort, high reward)

1. Enable proactive XMPP pings to keep the WebSocket alive through NATs/LBs. Add to the global config:

   ping_interval = 60
   ping_timeout = 120
   

   Also align the ingress controller idle timeout (e.g., nginx proxy-read-timeout) to be > 120 s.

2. Add bounds to mod_websocket.patch — DONE. The patch now enforces a configurable websocket_max_message_size (default 2 MB). Connections exceeding the limit are closed with code 1009 and the buffer is reset.

3. Raise SMACKS limits for mobile clients. In startup.sh or the template:

   smacks_max_old_sessions = 50
   

   Consider whether smacks_hibernation_time = 300 is appropriate; some mobile deployments use 600–1800 s.

Short-term (requires code changes)

4. Cache FCM token lookups in memory. The mod_vnc_fcm modules read the same private-data / token rows repeatedly. Add an in-memory LRU cache (e.g., 5-minute TTL) around:

   • getNotifyOptionsForUser()
   • getDisplayName()

   This avoids synchronous PostgreSQL round-trips on the hot path.

5. Defer or batch MUC FCM notifications. In mod_vnc_muc_fcm, iterating all affiliations and doing a DB read + HTTP request per member inside muc-broadcast-message blocks the room. Move the loop into an async timer or a background queue so the broadcast event returns immediately.

6. Fix trusted_proxies. Set it to the cluster CIDR(s) or the ingress controller's source IPs, e.g.:

   trusted_proxies = { "127.0.0.1", "10.0.0.0/8" }
   

Diagnostic

7. Watch log_slow_events. The module is enabled with a 1.5 s threshold. If PostgreSQL or the FCM path is the bottleneck, the logs will show Slow event 'message/bare' or Slow event 'muc-broadcast-message'. Use this to confirm fix #4/#5 before and after.

---

Files Involved

File	Relevance
patches/mod_websocket.patch	Unbounded fragment accumulation
vnctalk/mod_vnc_fcm.lua	Synchronous DB reads on every message
vnctalk/mod_vnc_muc_fcm.lua	Synchronous DB reads + loop over MUC affiliations
config/prosody.cfg.lua.template	Missing ping_interval, wrong trusted_proxies
config/startup.sh	SMACKS_MAX_UNACKED_STANZAS, SMACKS_HIBERNATION_TIME defaults
vnctalk/mod_log_slow_events.lua	Already logging; verify thresholds