# Prosody Upgrade Plan: 0.11.6 → 13.0.6 Stepwise upgrade of the vnctalk-prosody image, its patches to upstream Prosody, and the custom VNCtalk modules. Verification at each milestone is done with the existing external testsuite. All claims below about upstream code were verified against the pristine source trees of prosody-0.11.6, prosody-0.12.5, and prosody-13.0.6 (not guessed from release notes). ## Strategy Three build milestones, each producing a deployable image and a green testsuite run before the next one starts: | Milestone | Prosody | Lua | Purpose | |-----------|---------|-----|---------| | M1 | 0.12.5 (latest 0.12.x at execution time) | 5.2 (unchanged) | Mandatory intermediate: DB schema migration path, first re-port of patches, config modernization | | M2 | 0.12.5 | 5.4 | Isolate the interpreter change from the Prosody change | | M3 | 13.0.6 | 5.4 | Target version, second (smaller) re-port of patches | Rationale: - 0.11 → 13.0 in one jump conflates too many variables (patch drift, config renames, DB schema, Lua version). 0.12 is the supported upgrade path for the SQL schema migration. - 13.0.6 still supports Lua 5.2 (`configure --lua-version=5.2|5.3|5.4`), so the Lua bump is technically optional — but Lua 5.2 is on its way out upstream and in Alpine, so do it as its own milestone (M2) where a failure can only be Lua-related. - Each milestone lands as its own MR on `main`, gets a `development-$SHA` image via the existing CI, is run against the external testsuite, and soaks before the next milestone begins. ## Phase 0: Groundwork (before M1) 1. **Convert patches from full-file copies to unified diffs.** ✅ DONE. `patches/` now holds 11 `.patch` files (unified diffs against pristine 0.11.6, documented in `patches/README.md`); the Dockerfile builder stage applies them with `patch -p1 --fuzz=0` before `./configure && make install`. Verified: the resulting image's installed Prosody tree (`/usr/local/lib/prosody` + `/usr/local/bin`, 269 files) is byte-identical to the image built from the old copy-based mechanism. 2. **Record the baseline.** ✅ DONE. Image build verified; compose-harness testsuite green against the converted image (76 passed, 5 skipped after post-M1 fixes). 3. **Snapshot the database schema.** ✅ N/A — the PostgreSQL schema migration is a no-op (`upgrade_table` in `mod_storage_sql.lua` is MySQL-only). A pre-upgrade `pg_dump` is still recommended as a rollback point before any production deploy. ### Decisions (already made) These constraints shape the dispositions below: 1. An external service telnets into the admin console (port 5582) to run commands — the console must stay reachable on all interfaces across the upgrade. 2. The `muc-config-sub-mitted` event fired by the muc.lib patch **is consumed externally** — the hunk must be preserved through both re-ports. 3. `legacyauth` is no longer required — remove it at M1. 4. The external avatar upload in `mod_vcard_muc` stays — the fork is kept; 13.0's core MUC vcard support does not replace it. 5. `component_secret` may be randomized — fold into M1's startup/template edits. ## Patch inventory and disposition Verified deltas of each file in `patches/` vs pristine 0.11.6, and what to do with each: | Patch | What it actually changes | Disposition | |-------|--------------------------|-------------| | `mod_carbons.lua` | +1 line: hook `vnc-rest-message` so REST-injected messages get carbon-copied | **Re-apply** at M1 and M3. Trivial: the hook block is byte-identical in 13.0.6 (`plugins/mod_carbons.lua:166-171`). | | `mod_mam.lua` | (a) hooks `vnc-rest-message` (stores under *sender* via new `vnc_rest` flag), (b) `shall_store()` short-circuited to `true` — required because users live in the external HTTP auth backend so `um.user_exists()` can't be trusted, (c) only archives stanzas that have a `` | **Re-port** at M1 and M3. `shall_store()` and the hook points still exist in 13.0.6 (`plugins/mod_mam/mod_mam.lua:285,439`). Re-express as minimal hunks; the 0.11 patch also deleted upstream code that should stay. | | `mod_muc_mam.lua` | Disables `schedule_cleanup()` and archive deletion on room destroy; adds info-level logging | **Re-port, shrink.** The info-level log lines were debug aids — drop them. Check whether `muc_log_expires_after = "never"` (already set) makes the cleanup disable redundant in 0.12+/13.0 before keeping those hunks. | | `mod_muc.lua` (muc/) | Registers `handle_unregister_iq` on room_mt + routes `iq-set/bare/xmpp:vnctalk:unregister:query`; adds unused `dumpTable` debug helper | **Re-port** minus `dumpTable`. Routing table and `room_mt` wiring still look the same in 13.0.6. | | `register.lib.lua` (muc/) | Adds `handle_unregister_iq`: fires `vnc-muc-kick` (consumed by `mod_vnc_track_kicks`) and removes the user's affiliation | **Re-port.** 13.0.6 exports the same three functions, so the addition applies almost cleanly. Note it pokes the internal `room._affiliations` table — still present in 13.0.6 muc.lib, but re-check at each milestone. | | `muc.lib.lua` | The big fork: (a) `broadcast()` also routes stanzas to *offline remote* affiliated members, (b) suppresses the unavailable self-presence in `publicise_occupant_status`, (c) `handle_groupchat_to_room` lets owner/admin/member post without being a present occupant (skips `muc-occupant-groupchat` for them, guards `occupant.nick` against nil), (d) fires extra `muc-config-sub-mitted` event — **consumed by an external service, must be kept**, (e) operator-precedence bugfix in the semi-anonymous role-change branch | **Re-port (a)-(d) at M1 and M3 — this is the single biggest effort item.** `muc.lib.lua` was heavily refactored in both 0.12 and 13.0 (e.g. `handle_groupchat_to_room` moved logic into a `muc-occupant-groupchat` hook at module level, 13.0.6 `muc.lib.lua:1199`). Drop (e): fixed upstream in 0.12 (verified at 0.12.5 `muc.lib.lua:1506`). Cover (d) explicitly in the external testsuite so a silent porting mistake can't drop the event. | | `hidden.lib.lua` | Hides the "publicly searchable" room config option for *everyone* when `restrict_room_creation`-style restriction applies (removes the admin exception) | **Re-port.** Small; the surrounding code switches to the roles/permissions API in 13.0 — re-express against that. | | `mod_muc_unique.lua` | Rewrites the muc#unique handler; adds a bare-JID variant answering `item-not-found` | **Re-port.** Upstream file is nearly unchanged in 13.0.6. | | `moduleapi.lua` | Adds `module:open_host_store(host, name, type)` | **Drop the core patch at M1.** Rewrite the only consumers (`mod_vnc_muc_fcm.lua:42-45`, `mod_vnc_muc_fcm_hin.lua:41-44`) to call `require "core.storagemanager".open(host, name, type)` directly — same one-liner, no core fork needed. | | `mod_admin_telnet.lua` | Console listens on `*` instead of `127.0.0.1` | **Drop the patch at M1, replace with config.** An external service depends on telnet access (Decision 1), so the exposure stays — but it doesn't need a core fork: the telnet service registers as `name = "console"`, and portmanager resolves `console_interfaces` from config (verified 13.0.6 `core/portmanager.lua:94-100`). Set `console_interfaces = { "*" }` in the template. | | `portmanager.lua` | Changes `default_local_interfaces` from loopback to `*`/`::` | **Drop the patch at M1.** It only existed to expose the telnet console; `console_interfaces = { "*" }` covers that alone. Verify after the switch that no *other* private service was relying on the widened default (compare `netstat` listeners in the old vs new container). | Note: `mod_admin_telnet` and core `usermanager.is_admin()` are **deprecated but still present in 13.0.6** — neither forces an immediate rewrite, but both should be migrated during M3 (see below). ## Custom module inventory (`vnctalk/`) Two categories with different upgrade treatment: ### Stale copies of community modules — refresh, don't port These are snapshots of prosody-modules. At M1 and again at M3, `diff` each against the current prosody-modules tip: if unmodified, replace wholesale with the current upstream version; if locally modified, extract the local delta first (same treatment as `patches/`). - `mod_smacks` + `mod_smacks_offline` — **delete at M1.** The bundled copy is the ~2020 community fork; 0.12 moved an improved mod_smacks into core. `mod_smacks_offline` is already a no-op shim (logs "no longer required"). Keep `"smacks"` in `modules_enabled`; audit the `smacks_*` options in the config template against 0.12/13.0 core option names (the bundled fork changed some defaults, e.g. hibernation 600 vs core 300 — the template pins explicit values, so mostly a verification task). - `mod_auth_http_async` — auth against the hybrida HTTP backend. Load-bearing. Verify `util.sasl`/`util.async` usage per milestone; `net.http.request(url, ex, callback)` keeps its old signature in 13.0.6, so likely low-effort. Compare with the current community version. - Likely-unmodified community snapshots to refresh: `mod_auto_accept_subscriptions`, `mod_discoitems`, `mod_filter_chatstates`, `mod_http_altconnect`, `mod_http_index`, `mod_http_upload_external`, `mod_idlecompat`, `mod_webpresence`, `mod_log_slow_events`, `mod_s2s_keepalive`, `mod_roster_command`, `mod_carbons_copies`, `mod_auth_any`, `mod_alias`. - `mod_vcard_muc` — **keep the fork** (Decision 4: the external avatar upload to `avatar_upload_url` via `net.http` stays). Diff against the community version to isolate the local delta, then carry that delta forward on top of the current upstream module. At M3, make sure the in-core MUC vcard support (`plugins/muc/vcard.lib.lua`, new in 13.0) doesn't double-handle the same iq events — disable or reconcile one of the two. ### VNC-original modules (`mod_vnc_*`, `mod_http_rest`) — port Version-sensitive API usage found by scan (everything else in these modules uses stable APIs — `util.stanza`, `module:hook`, `module:get_option_*`): | Module | Risk | Action | |--------|------|--------| | `mod_vnc_muc_fcm`, `mod_vnc_muc_fcm_hin` | Use the patched-in `module:open_host_store()` | Rewrite to `storagemanager.open()` at M1 (see moduleapi row above) | | `mod_vnc_broadcast` | `require "core.usermanager".is_admin` — deprecated in 13.0 (removal controlled by `strict_deprecate_is_admin`) | Works through M3 with a deprecation warning; port to the roles API (`module:may` / `usermanager.get_jid_role`) during M3. Do **not** set `strict_deprecate_is_admin` | | `mod_http_rest` | Fires `vnc-rest-message` (consumed by the mam/carbons patches); uses `usermanager.test_password` | `test_password` still exists in 13.0; keep the custom event wiring in sync with the re-ported patches | | `mod_vnc_fcm`, `mod_vnc_fcm_hin` | Hook `muc-room-changed` — an event **no Prosody version fires** (checked 0.11.6/0.12.5/13.0.6) and nothing in this repo fires | Dead code; ignore for the upgrade, delete when convenient | | All MUC-related `mod_vnc_*` | Hook `muc-broadcast-message`, `muc-config-submitted`, `muc-disco#info`, `muc-invite`, `muc-occupant-session-new` | All verified still fired in 0.12.5 and 13.0.6 — re-test, no rewrite expected | | Modules doing outbound HTTP (fcm, delfile, vcard_avatar, auth) | `net.http.request` old signature | Unchanged in 13.0.6; no action, just testsuite coverage | ## Milestone 1: 0.11.6 → 0.12.6, Lua 5.2 — ✅ COMPLETE Status: all repo changes done and verified. 0.12.6 was latest 0.12.x at execution time. Compose-harness testsuite green (80 passed, 5 skipped); external testsuite run against a dev deployment successful; telnet console regression test passed. The PostgreSQL schema migration is a no-op (`upgrade_table` is MySQL-only); tables are auto-created on startup via `sql_manage_tables = true`. Discovered during execution (now part of the changes): - 0.12 switched the default IDN library to ICU, which broke at runtime (`U_FILE_ACCESS_ERROR` plus a segfault) because the image ships no ICU data — pinned `--idn-library=idn`. - `http` (5280) became a *private* service in 0.12 and bound loopback-only — this, not just the telnet console, is what `portmanager.patch` had been widening. Fixed via `http_interfaces = { "*", "::" }`. - `mod_admin_socket` is a hard dependency of the telnet console in 0.12 and its unix socket defaults into the unwritable data dir — pointed `admin_socket` at `/var/log/prosody/`. - Latent `startup.sh` bug: the `log_slow_events_threshold` fallback wasn't exported, so the rendered config was syntactically invalid whenever the env var was absent — fixed. Discovered during post-M1 testsuite review (fixed in commits after `0ea4d0a`): - `run_as_root = true` was removed as a "leftover", but production and the compose harness run as root (`user: "0:0"`) because `startup.sh` writes into root-owned `/etc/prosody/`. Without it, `mod_posix` calls `prosody.shutdown("Refusing to run as root")` during startup; the `server-stopping` event deactivates c2s (port 5222) before the shutdown itself errors out (`prosody.main_thread` is nil during module init), leaving Prosody running without c2s. Re-added. - `default_storage = "sql"` was removed as "redundant" with `storage = "sql"`. However, the MUC component sets `storage = { muc_log = "sql" }` (a table); `storagemanager.get_driver` falls back to `default_storage` (or `"internal"`) for stores not listed in the table. Without `default_storage = "sql"`, the `kick` store on the MUC component used internal storage, whose 0.12 archive driver requires stanza objects — `mod_vnc_track_kicks` passes a plain string, causing `unsupported-datatype` errors. Re-added. - Three `test_08_image_patches` entries checked for patches that M1 intentionally dropped (`moduleapi`, `mod_admin_telnet`, `muc.lib dumpTable`). Replaced with markers that verify the config-based replacements (`console_interfaces`, `http_interfaces`) and the `storagemanager.open()` rewrite in `mod_vnc_muc_fcm.lua`. 1. **Dockerfile**: bump source URL/version; `./configure` flags unchanged (`--sysconfdir`, `--no-example-certs` still valid). Keep all lua5.2 Alpine packages as-is. ✅ 2. **Patches**: regenerate every patch against the 0.12.6 tree per the disposition table. Deleted at this milestone: `moduleapi.lua`, `mod_admin_telnet.lua`, `portmanager.lua` patches and the `muc.lib.lua` precedence hunk and `dumpTable` debug helper. Patches are applied via glob (`patches/*.patch`), so no Dockerfile change was needed. ✅ 3. **Delete** `vnctalk/mod_smacks/` and `vnctalk/mod_smacks_offline/`. ✅ 4. **Config template** (`config/prosody.cfg.lua.template`): - `cross_domain_bosh` / `cross_domain_websocket`: deprecated in 0.12 (still shimmed with a warning — verified in 13.0.6 `mod_bosh.lua:47`, `mod_websocket.lua:36`). Migrate to `http_cors_override` / `access_control_allow_origins` now so M3 starts clean. ✅ - `bosh_ports` → serve BOSH via `http_ports = { 5280 }` (mod_http consolidation). ✅ - Add `console_interfaces = { "*" }` — replaces the two dropped interface patches and keeps the admin console reachable for the external service that telnets in (Decision 1). ✅ - Remove `"legacyauth"` from `modules_enabled` (Decision 3). ✅ - Randomize `component_secret` (Decision 5): generate it in `startup.sh` (e.g. `openssl rand -hex 24`, overridable via an env var for setups where the component connects from outside the pod) and substitute it into the template like the other variables. ✅ - `run_as_root = true`: initially removed as a "leftover", then **re-added** — production and the compose harness run as root; without it `mod_posix` deactivates c2s. ✅ - `default_storage = "sql"`: initially removed as "redundant", then **re-added** — the MUC component's `storage = { muc_log = "sql" }` table needs it as the fallback for non-listed stores (e.g. `kick`). ✅ - `daemonize = false` is deprecated; startup already passes `-F`, so remove the option when `prosodyctl check config` complains. ✅ - Run `prosodyctl check config` inside the built image and fix every warning — this is the cheap way to catch renamed options (also covers the mod_muc_mam/`muc_log_*` option names). ✅ 5. **Database**: the 0.11 → 0.12 archive schema migration is required and **one-way**. Procedure: dump the Postgres DB → restore into a scratch DB → run `prosodyctl mod_storage_sql upgrade` against it → point a dev deployment at it → testsuite. Only then schedule the real migration (with a fresh dump) for the dev/prod rollouts. ✅ **N/A for PostgreSQL** — `upgrade_table()` in `mod_storage_sql.lua` is MySQL-only; for PG the command is a no-op. Table creation is already automatic on every startup via `sql_manage_tables = true` (`CREATE TABLE IF NOT EXISTS`). A pre-upgrade `pg_dump` is still recommended as a rollback point. 6. **Verify**: image builds, `./test.sh` smoke run, `prosodyctl check` clean, external testsuite green — with particular attention to the flows the patches exist for: REST-injected message archiving + carbons, MUC posting by non-present members, offline-member broadcast, MUC unregister/kick, MAM of users that only exist in the HTTP auth backend, and delivery of the `muc-config-sub-mitted` event to its external consumer (Decision 2). ✅ **Done** — compose-harness testsuite green (80 passed, 5 skipped); external testsuite run against a dev deployment successful. 7. **Telnet console regression test**: 0.12 reimplemented `mod_admin_telnet` on top of `mod_admin_shell`, so command syntax and output formatting changed. Exercise the exact commands the external telnet service runs (Decision 1) against the M1 image before rollout. ✅ **Done** — manual test passed. ## Milestone 2: Lua 5.2 → 5.4 (Prosody stays 0.12.6) — ✅ COMPLETE Status: Dockerfile updated, image builds, `prosodyctl check config` passes, compose-harness testsuite green (80 passed, 6 skipped, 0 failed). External testsuite also passed (48 passed, 30 skipped — skips are infrastructure prerequisites: no compose/docker exec, no MOCK_URL, no PG/telnet port-forwards; `test_vcard_fallback` now skipped pending module enablement). No DB changes. 1. **Alpine packages**: ✅ Done. All `lua5.2-*` packages mapped to `lua5.4-*` equivalents. - Available and swapped: `lua5.4`, `lua5.4-socket`, `lua5.4-dbi-postgresql`, `lua5.4-expat`, `lua5.4-sql-postgres`, `lua5.4-filesize`, `lua5.4-lpeg`, `lua5.4-hiredis`, `lua5.4-filesystem`, `lua5.4-ldap`, `lua5.4-sec`, `lua5.4-lzlib`, `lua5.4-cjson`, `lua5.4-dev` (builder only). - Dropped (unused): `lua5.2-lpeg_patterns`, `lua5.2-rapidjson`, `lua5.2-redis` — verified not required by any `require` in Prosody 0.12.6 or any vnctalk module. - Dropped (unnecessary): `lua5.2-bitop` — Prosody's `util.bitcompat` uses `util.bit53` (native Lua 5.3/5.4 bitwise operators) when `bit32` is unavailable. - Dropped (never used): `luarocks5.2` — installed but no luarocks commands were ever run. - Fixed: duplicate `lua5.2-socket` entry removed. 2. **Configure**: ✅ Done. `./configure --lua-version=5.4 --idn-library=idn ...`. 3. **Code audit** for 5.2 → 5.4 breakage: ✅ Clean. - No `bit32`/`bitop` usage in any custom module (checked) — `lua5.2-bitop` can be dropped. - `unpack` vs `table.unpack`: every Prosody file that uses `unpack` has `local unpack = table.unpack or unpack` — resolves to `table.unpack` on Lua 5.4. No bare `unpack` in any vnctalk module or patch. - Lua 5.4 `string.format("%d", x)`: the only `%d` usage in vnctalk modules is `mod_http_upload_external.lua:43` with `filesize` (integer from HTTP headers). Prosody core's `%d` usages are all with integer values (counts, timestamps). No breakage. 4. **Verify**: ✅ Done. Compose-harness: 80 passed, 6 skipped, 0 failed. External: all runnable tests passed; skips are infrastructure prerequisites (no compose exec, no MOCK_URL, no PG/telnet port-forwards); `test_vcard_fallback` skipped pending `mod_vnc_vcard_fallback` enablement in the config template. ## Milestone 3: 0.12.6 → 13.0.6 — ✅ COMPLETE (repo changes; manual tasks pending) Status: Dockerfile bumped to 13.0.6, all 8 patches re-ported, config modernized. `prosodyctl check config` passes (all checks passed). Compose-harness testsuite green (80 passed, 6 skipped, 0 failed). No DB schema migration needed for PostgreSQL (same as M1 — `upgrade_table` is MySQL-only). Manual tasks before rollout: run external testsuite against a dev deployment; exercise the telnet console command set (same as M1 §7). See `m1-manual-tasks.md` §2 and §3 (the same procedures apply). 1. **Dockerfile**: ✅ Done. Bumped to `prosody-13.0.6.tar.gz`; `--lua-version=5.4` and `--idn-library=idn` unchanged. `lua-unbound` and `lua-readline` not added (optional; Prosody falls back and logs a warning). 2. **Patches**: ✅ Done. All 8 patches re-ported against 13.0.6. - `muc.lib.patch`: all 6 hunks applied with offset 7 — no re-port needed. - `hidden.lib.patch`: re-ported — 13.0 uses `module:may(":create-public-room")` instead of `um_is_admin`; changed to `if restrict_public then` (same intent: hide the public-room option for everyone when `restrict_public` is true). - `mod_muc.patch`: re-ported hunk 1 — 13.0 added `restrict_pm` between `register` and `presence_broadcast`; updated context. Hunk 2 applied with offset 17. - `mod_muc_unique.patch`: re-ported — 13.0 uses `require "prosody.util.stanza"` (namespaced); updated context. - `mod_carbons.patch`, `mod_mam.patch`, `mod_muc_mam.patch`, `register.lib.patch`: applied with line offsets, no re-port needed. 3. **Namespaced requires**: ✅ No action needed. Old-style `require "util.stanza"` in vnctalk modules works via 13.0's compat loader. No bulk rewrite required. 4. **Roles/permissions**: ✅ No action needed. `mod_vnc_broadcast` uses `usermanager.is_admin` which is deprecated but still works (logs a warning). `strict_deprecate_is_admin` is not set. The `hidden.lib` patch was re-ported against the new `module:may` API. 5. **mod_posix**: ✅ Done. Removed from `modules_enabled` — 13.0 absorbed signal handling, pidfile writing, and the `run_as_root` check into core (`util/startup.lua`). 6. **Database**: ✅ N/A for PostgreSQL — `upgrade_table` in `mod_storage_sql.lua` is MySQL-only (same as M1). Tables auto-created on startup via `sql_manage_tables = true`. 7. **Config**: ✅ Done. `prosodyctl check config` passes. Remaining warnings: - `mod_admin_telnet` deprecation — kept intentionally (external service telnets into 5582; `mod_admin_shell` uses a unix socket, not telnet). - `default_storage` — false positive (needed for MUC component's table storage override; the check only sees the global `storage = "sql"` string). - `mod_pubsub` moved from `modules_enabled` to a `Component` — 13.0 requires pubsub to be loaded as a component, not a module. 8. **Healthcheck / telnet console**: ✅ Unchanged. `mod_admin_telnet` (port 5582) still ships in 13.0.6 and works. `healthcheck.sh` unchanged. Telnet console regression test is a manual task (same as M1 §7). 9. **Verify**: ✅ Compose-harness: 80 passed, 6 skipped, 0 failed. External testsuite run is manual. ## Rollout and rollback (per milestone) 1. MR → merge to `main` → CI builds `development-$SHA` and Trivy-scans it (expect the scan to improve dramatically; 0.11.6 is EOL with known CVEs). 2. Deploy to dev, run the external testsuite, soak. 3. Promote via the existing tag flow (`prod-` / `stable-` re-tag the dev image). 4. **Rollback hazard is the DB, not the image**: M1 and M3 perform one-way SQL schema upgrades. Rolling back the image after the schema upgrade requires restoring the pre-upgrade dump. Always: fresh dump immediately before deploying a schema-upgrading milestone, and keep the previous image tag pinned (`development-` of the prior milestone) for emergencies.