vnclagoon/vnctalk-prosody
14
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
26e7512b24e1a57ea8fd4ef9b5dbc35177d9b935
vnctalk-prosody/upgrade-plan.md
T
Stefan-Sanger 26e7512b24 feat: upgrade Prosody 0.12.6 → 13.0.6 (milestone 3) 
Bump Dockerfile to prosody-13.0.6; all 8 patches re-ported against 13.0.6.

Re-ported patches (3 changed, 5 applied with offset):
- hidden.lib.patch: 13.0 uses module:may() instead of um_is_admin; changed
  to 'if restrict_public then' (same intent: hide option for everyone)
- mod_muc.patch: 13.0 added restrict_pm between register and
  presence_broadcast; updated hunk 1 context
- mod_muc_unique.patch: 13.0 uses 'require "prosody.util.stanza"'
  (namespaced); updated context
- muc.lib, mod_carbons, mod_mam, mod_muc_mam, register.lib: applied
  with line offsets, no re-port needed

Config changes:
- Remove mod_posix from modules_enabled (13.0 absorbed signal handling,
  pidfile, and run_as_root check into core util/startup.lua)
- Move pubsub from modules_enabled to Component (13.0 requires pubsub
  to be loaded as a component, not a module)

Test fix:
- test_muc_fcm_push_to_offline_member: wait for count=2 captures instead
  of 1 — mod_vnc_muc_fcm pushes to ALL affiliated members (including
  sender, because it can't see main-host sessions from the MUC
  component); the test was racing on which push arrived first

Verified: prosodyctl check config passes; compose-harness testsuite
green (80 passed, 6 skipped, 0 failed).

Part-of: <http://gitlab.vnc.biz/uxf/vnctalk-prosody/-/merge_requests/3>
2026-07-15 17:58:02 +02:00

24 KiB
Raw Blame History

Prosody Upgrade Plan: 0.11.6 → 13.0.6

Stepwise upgrade of the vnctalk-prosody image, its patches to upstream Prosody, and the custom VNCtalk modules. Verification at each milestone is done with the existing external testsuite.

All claims below about upstream code were verified against the pristine source trees of prosody-0.11.6, prosody-0.12.5, and prosody-13.0.6 (not guessed from release notes).

Strategy

Three build milestones, each producing a deployable image and a green testsuite run before the next one starts:

Milestone Prosody Lua Purpose
M1 0.12.5 (latest 0.12.x at execution time) 5.2 (unchanged) Mandatory intermediate: DB schema migration path, first re-port of patches, config modernization
M2 0.12.5 5.4 Isolate the interpreter change from the Prosody change
M3 13.0.6 5.4 Target version, second (smaller) re-port of patches

Rationale:

  • 0.11 → 13.0 in one jump conflates too many variables (patch drift, config renames, DB schema, Lua version). 0.12 is the supported upgrade path for the SQL schema migration.
  • 13.0.6 still supports Lua 5.2 (configure --lua-version=5.2|5.3|5.4), so the Lua bump is technically optional — but Lua 5.2 is on its way out upstream and in Alpine, so do it as its own milestone (M2) where a failure can only be Lua-related.
  • Each milestone lands as its own MR on main, gets a development-$SHA image via the existing CI, is run against the external testsuite, and soaks before the next milestone begins.

Phase 0: Groundwork (before M1)

  1. Convert patches from full-file copies to unified diffs. ✅ DONE. patches/ now holds 11 .patch files (unified diffs against pristine 0.11.6, documented in patches/README.md); the Dockerfile builder stage applies them with patch -p1 --fuzz=0 before ./configure && make install. Verified: the resulting image's installed Prosody tree (/usr/local/lib/prosody + /usr/local/bin, 269 files) is byte-identical to the image built from the old copy-based mechanism.
  2. Record the baseline. ✅ DONE. Image build verified; compose-harness testsuite green against the converted image (76 passed, 5 skipped after post-M1 fixes).
  3. Snapshot the database schema. ✅ N/A — the PostgreSQL schema migration is a no-op (upgrade_table in mod_storage_sql.lua is MySQL-only). A pre-upgrade pg_dump is still recommended as a rollback point before any production deploy.

Decisions (already made)

These constraints shape the dispositions below:

  1. An external service telnets into the admin console (port 5582) to run commands — the console must stay reachable on all interfaces across the upgrade.
  2. The muc-config-sub-mitted event fired by the muc.lib patch is consumed externally — the hunk must be preserved through both re-ports.
  3. legacyauth is no longer required — remove it at M1.
  4. The external avatar upload in mod_vcard_muc stays — the fork is kept; 13.0's core MUC vcard support does not replace it.
  5. component_secret may be randomized — fold into M1's startup/template edits.

Patch inventory and disposition

Verified deltas of each file in patches/ vs pristine 0.11.6, and what to do with each:

Patch What it actually changes Disposition
mod_carbons.lua +1 line: hook vnc-rest-message so REST-injected messages get carbon-copied Re-apply at M1 and M3. Trivial: the hook block is byte-identical in 13.0.6 (plugins/mod_carbons.lua:166-171).
mod_mam.lua (a) hooks vnc-rest-message (stores under sender via new vnc_rest flag), (b) shall_store() short-circuited to true — required because users live in the external HTTP auth backend so um.user_exists() can't be trusted, (c) only archives stanzas that have a <body> Re-port at M1 and M3. shall_store() and the hook points still exist in 13.0.6 (plugins/mod_mam/mod_mam.lua:285,439). Re-express as minimal hunks; the 0.11 patch also deleted upstream code that should stay.
mod_muc_mam.lua Disables schedule_cleanup() and archive deletion on room destroy; adds info-level logging Re-port, shrink. The info-level log lines were debug aids — drop them. Check whether muc_log_expires_after = "never" (already set) makes the cleanup disable redundant in 0.12+/13.0 before keeping those hunks.
mod_muc.lua (muc/) Registers handle_unregister_iq on room_mt + routes iq-set/bare/xmpp:vnctalk:unregister:query; adds unused dumpTable debug helper Re-port minus dumpTable. Routing table and room_mt wiring still look the same in 13.0.6.
register.lib.lua (muc/) Adds handle_unregister_iq: fires vnc-muc-kick (consumed by mod_vnc_track_kicks) and removes the user's affiliation Re-port. 13.0.6 exports the same three functions, so the addition applies almost cleanly. Note it pokes the internal room._affiliations table — still present in 13.0.6 muc.lib, but re-check at each milestone.
muc.lib.lua The big fork: (a) broadcast() also routes stanzas to offline remote affiliated members, (b) suppresses the unavailable self-presence in publicise_occupant_status, (c) handle_groupchat_to_room lets owner/admin/member post without being a present occupant (skips muc-occupant-groupchat for them, guards occupant.nick against nil), (d) fires extra muc-config-sub-mitted event — consumed by an external service, must be kept, (e) operator-precedence bugfix in the semi-anonymous role-change branch Re-port (a)-(d) at M1 and M3 — this is the single biggest effort item. muc.lib.lua was heavily refactored in both 0.12 and 13.0 (e.g. handle_groupchat_to_room moved logic into a muc-occupant-groupchat hook at module level, 13.0.6 muc.lib.lua:1199). Drop (e): fixed upstream in 0.12 (verified at 0.12.5 muc.lib.lua:1506). Cover (d) explicitly in the external testsuite so a silent porting mistake can't drop the event.
hidden.lib.lua Hides the "publicly searchable" room config option for everyone when restrict_room_creation-style restriction applies (removes the admin exception) Re-port. Small; the surrounding code switches to the roles/permissions API in 13.0 — re-express against that.
mod_muc_unique.lua Rewrites the muc#unique handler; adds a bare-JID variant answering item-not-found Re-port. Upstream file is nearly unchanged in 13.0.6.
moduleapi.lua Adds module:open_host_store(host, name, type) Drop the core patch at M1. Rewrite the only consumers (mod_vnc_muc_fcm.lua:42-45, mod_vnc_muc_fcm_hin.lua:41-44) to call require "core.storagemanager".open(host, name, type) directly — same one-liner, no core fork needed.
mod_admin_telnet.lua Console listens on * instead of 127.0.0.1 Drop the patch at M1, replace with config. An external service depends on telnet access (Decision 1), so the exposure stays — but it doesn't need a core fork: the telnet service registers as name = "console", and portmanager resolves console_interfaces from config (verified 13.0.6 core/portmanager.lua:94-100). Set console_interfaces = { "*" } in the template.
portmanager.lua Changes default_local_interfaces from loopback to */:: Drop the patch at M1. It only existed to expose the telnet console; console_interfaces = { "*" } covers that alone. Verify after the switch that no other private service was relying on the widened default (compare netstat listeners in the old vs new container).

Note: mod_admin_telnet and core usermanager.is_admin() are deprecated but still present in 13.0.6 — neither forces an immediate rewrite, but both should be migrated during M3 (see below).

Custom module inventory (vnctalk/)

Two categories with different upgrade treatment:

Stale copies of community modules — refresh, don't port

These are snapshots of prosody-modules. At M1 and again at M3, diff each against the current prosody-modules tip: if unmodified, replace wholesale with the current upstream version; if locally modified, extract the local delta first (same treatment as patches/).

  • mod_smacks + mod_smacks_offline — delete at M1. The bundled copy is the ~2020 community fork; 0.12 moved an improved mod_smacks into core. mod_smacks_offline is already a no-op shim (logs "no longer required"). Keep "smacks" in modules_enabled; audit the smacks_* options in the config template against 0.12/13.0 core option names (the bundled fork changed some defaults, e.g. hibernation 600 vs core 300 — the template pins explicit values, so mostly a verification task).
  • mod_auth_http_async — auth against the hybrida HTTP backend. Load-bearing. Verify util.sasl/util.async usage per milestone; net.http.request(url, ex, callback) keeps its old signature in 13.0.6, so likely low-effort. Compare with the current community version.
  • Likely-unmodified community snapshots to refresh: mod_auto_accept_subscriptions, mod_discoitems, mod_filter_chatstates, mod_http_altconnect, mod_http_index, mod_http_upload_external, mod_idlecompat, mod_webpresence, mod_log_slow_events, mod_s2s_keepalive, mod_roster_command, mod_carbons_copies, mod_auth_any, mod_alias.
  • mod_vcard_muc — keep the fork (Decision 4: the external avatar upload to avatar_upload_url via net.http stays). Diff against the community version to isolate the local delta, then carry that delta forward on top of the current upstream module. At M3, make sure the in-core MUC vcard support (plugins/muc/vcard.lib.lua, new in 13.0) doesn't double-handle the same iq events — disable or reconcile one of the two.

VNC-original modules (mod_vnc_*, mod_http_rest) — port

Version-sensitive API usage found by scan (everything else in these modules uses stable APIs — util.stanza, module:hook, module:get_option_*):

Module Risk Action
mod_vnc_muc_fcm, mod_vnc_muc_fcm_hin Use the patched-in module:open_host_store() Rewrite to storagemanager.open() at M1 (see moduleapi row above)
mod_vnc_broadcast require "core.usermanager".is_admin — deprecated in 13.0 (removal controlled by strict_deprecate_is_admin) Works through M3 with a deprecation warning; port to the roles API (module:may / usermanager.get_jid_role) during M3. Do not set strict_deprecate_is_admin
mod_http_rest Fires vnc-rest-message (consumed by the mam/carbons patches); uses usermanager.test_password test_password still exists in 13.0; keep the custom event wiring in sync with the re-ported patches
mod_vnc_fcm, mod_vnc_fcm_hin Hook muc-room-changed — an event no Prosody version fires (checked 0.11.6/0.12.5/13.0.6) and nothing in this repo fires Dead code; ignore for the upgrade, delete when convenient
All MUC-related mod_vnc_* Hook muc-broadcast-message, muc-config-submitted, muc-disco#info, muc-invite, muc-occupant-session-new All verified still fired in 0.12.5 and 13.0.6 — re-test, no rewrite expected
Modules doing outbound HTTP (fcm, delfile, vcard_avatar, auth) net.http.request old signature Unchanged in 13.0.6; no action, just testsuite coverage

Milestone 1: 0.11.6 → 0.12.6, Lua 5.2 — ✅ COMPLETE

Status: all repo changes done and verified. 0.12.6 was latest 0.12.x at execution time. Compose-harness testsuite green (80 passed, 5 skipped); external testsuite run against a dev deployment successful; telnet console regression test passed. The PostgreSQL schema migration is a no-op (upgrade_table is MySQL-only); tables are auto-created on startup via sql_manage_tables = true.

Discovered during execution (now part of the changes):

  • 0.12 switched the default IDN library to ICU, which broke at runtime (U_FILE_ACCESS_ERROR plus a segfault) because the image ships no ICU data — pinned --idn-library=idn.
  • http (5280) became a private service in 0.12 and bound loopback-only — this, not just the telnet console, is what portmanager.patch had been widening. Fixed via http_interfaces = { "*", "::" }.
  • mod_admin_socket is a hard dependency of the telnet console in 0.12 and its unix socket defaults into the unwritable data dir — pointed admin_socket at /var/log/prosody/.
  • Latent startup.sh bug: the log_slow_events_threshold fallback wasn't exported, so the rendered config was syntactically invalid whenever the env var was absent — fixed.

Discovered during post-M1 testsuite review (fixed in commits after 0ea4d0a):

  • run_as_root = true was removed as a "leftover", but production and the compose harness run as root (user: "0:0") because startup.sh writes into root-owned /etc/prosody/. Without it, mod_posix calls prosody.shutdown("Refusing to run as root") during startup; the server-stopping event deactivates c2s (port 5222) before the shutdown itself errors out (prosody.main_thread is nil during module init), leaving Prosody running without c2s. Re-added.
  • default_storage = "sql" was removed as "redundant" with storage = "sql". However, the MUC component sets storage = { muc_log = "sql" } (a table); storagemanager.get_driver falls back to default_storage (or "internal") for stores not listed in the table. Without default_storage = "sql", the kick store on the MUC component used internal storage, whose 0.12 archive driver requires stanza objects — mod_vnc_track_kicks passes a plain string, causing unsupported-datatype errors. Re-added.
  • Three test_08_image_patches entries checked for patches that M1 intentionally dropped (moduleapi, mod_admin_telnet, muc.lib dumpTable). Replaced with markers that verify the config-based replacements (console_interfaces, http_interfaces) and the storagemanager.open() rewrite in mod_vnc_muc_fcm.lua.
  1. Dockerfile: bump source URL/version; ./configure flags unchanged (--sysconfdir, --no-example-certs still valid). Keep all lua5.2 Alpine packages as-is. ✅
  2. Patches: regenerate every patch against the 0.12.6 tree per the disposition table. Deleted at this milestone: moduleapi.lua, mod_admin_telnet.lua, portmanager.lua patches and the muc.lib.lua precedence hunk and dumpTable debug helper. Patches are applied via glob (patches/*.patch), so no Dockerfile change was needed. ✅
  3. Delete vnctalk/mod_smacks/ and vnctalk/mod_smacks_offline/. ✅
  4. Config template (config/prosody.cfg.lua.template):
    • cross_domain_bosh / cross_domain_websocket: deprecated in 0.12 (still shimmed with a warning — verified in 13.0.6 mod_bosh.lua:47, mod_websocket.lua:36). Migrate to http_cors_override / access_control_allow_origins now so M3 starts clean. ✅
    • bosh_ports → serve BOSH via http_ports = { 5280 } (mod_http consolidation). ✅
    • Add console_interfaces = { "*" } — replaces the two dropped interface patches and keeps the admin console reachable for the external service that telnets in (Decision 1). ✅
    • Remove "legacyauth" from modules_enabled (Decision 3). ✅
    • Randomize component_secret (Decision 5): generate it in startup.sh (e.g. openssl rand -hex 24, overridable via an env var for setups where the component connects from outside the pod) and substitute it into the template like the other variables. ✅
    • run_as_root = true: initially removed as a "leftover", then re-added — production and the compose harness run as root; without it mod_posix deactivates c2s. ✅
    • default_storage = "sql": initially removed as "redundant", then re-added — the MUC component's storage = { muc_log = "sql" } table needs it as the fallback for non-listed stores (e.g. kick). ✅
    • daemonize = false is deprecated; startup already passes -F, so remove the option when prosodyctl check config complains. ✅
    • Run prosodyctl check config inside the built image and fix every warning — this is the cheap way to catch renamed options (also covers the mod_muc_mam/muc_log_* option names). ✅
  5. Database: the 0.11 → 0.12 archive schema migration is required and one-way. Procedure: dump the Postgres DB → restore into a scratch DB → run prosodyctl mod_storage_sql upgrade against it → point a dev deployment at it → testsuite. Only then schedule the real migration (with a fresh dump) for the dev/prod rollouts. ✅ N/A for PostgreSQL — upgrade_table() in mod_storage_sql.lua is MySQL-only; for PG the command is a no-op. Table creation is already automatic on every startup via sql_manage_tables = true (CREATE TABLE IF NOT EXISTS). A pre-upgrade pg_dump is still recommended as a rollback point.
  6. Verify: image builds, ./test.sh smoke run, prosodyctl check clean, external testsuite green — with particular attention to the flows the patches exist for: REST-injected message archiving + carbons, MUC posting by non-present members, offline-member broadcast, MUC unregister/kick, MAM of users that only exist in the HTTP auth backend, and delivery of the muc-config-sub-mitted event to its external consumer (Decision 2). ✅ Done — compose-harness testsuite green (80 passed, 5 skipped); external testsuite run against a dev deployment successful.
  7. Telnet console regression test: 0.12 reimplemented mod_admin_telnet on top of mod_admin_shell, so command syntax and output formatting changed. Exercise the exact commands the external telnet service runs (Decision 1) against the M1 image before rollout. ✅ Done — manual test passed.

Milestone 2: Lua 5.2 → 5.4 (Prosody stays 0.12.6) — ✅ COMPLETE

Status: Dockerfile updated, image builds, prosodyctl check config passes, compose-harness testsuite green (80 passed, 6 skipped, 0 failed). External testsuite also passed (48 passed, 30 skipped — skips are infrastructure prerequisites: no compose/docker exec, no MOCK_URL, no PG/telnet port-forwards; test_vcard_fallback now skipped pending module enablement). No DB changes.

  1. Alpine packages: ✅ Done. All lua5.2-* packages mapped to lua5.4-* equivalents.
    • Available and swapped: lua5.4, lua5.4-socket, lua5.4-dbi-postgresql, lua5.4-expat, lua5.4-sql-postgres, lua5.4-filesize, lua5.4-lpeg, lua5.4-hiredis, lua5.4-filesystem, lua5.4-ldap, lua5.4-sec, lua5.4-lzlib, lua5.4-cjson, lua5.4-dev (builder only).
    • Dropped (unused): lua5.2-lpeg_patterns, lua5.2-rapidjson, lua5.2-redis — verified not required by any require in Prosody 0.12.6 or any vnctalk module.
    • Dropped (unnecessary): lua5.2-bitop — Prosody's util.bitcompat uses util.bit53 (native Lua 5.3/5.4 bitwise operators) when bit32 is unavailable.
    • Dropped (never used): luarocks5.2 — installed but no luarocks commands were ever run.
    • Fixed: duplicate lua5.2-socket entry removed.
  2. Configure: ✅ Done. ./configure --lua-version=5.4 --idn-library=idn ....
  3. Code audit for 5.2 → 5.4 breakage: ✅ Clean.
    • No bit32/bitop usage in any custom module (checked) — lua5.2-bitop can be dropped.
    • unpack vs table.unpack: every Prosody file that uses unpack has local unpack = table.unpack or unpack — resolves to table.unpack on Lua 5.4. No bare unpack in any vnctalk module or patch.
    • Lua 5.4 string.format("%d", x): the only %d usage in vnctalk modules is mod_http_upload_external.lua:43 with filesize (integer from HTTP headers). Prosody core's %d usages are all with integer values (counts, timestamps). No breakage.
  4. Verify: ✅ Done. Compose-harness: 80 passed, 6 skipped, 0 failed. External: all runnable tests passed; skips are infrastructure prerequisites (no compose exec, no MOCK_URL, no PG/telnet port-forwards); test_vcard_fallback skipped pending mod_vnc_vcard_fallback enablement in the config template.

Milestone 3: 0.12.6 → 13.0.6 — ✅ COMPLETE (repo changes; manual tasks pending)

Status: Dockerfile bumped to 13.0.6, all 8 patches re-ported, config modernized. prosodyctl check config passes (all checks passed). Compose-harness testsuite green (80 passed, 6 skipped, 0 failed). No DB schema migration needed for PostgreSQL (same as M1 — upgrade_table is MySQL-only).

Manual tasks before rollout: run external testsuite against a dev deployment; exercise the telnet console command set (same as M1 §7). See m1-manual-tasks.md §2 and §3 (the same procedures apply).

  1. Dockerfile: ✅ Done. Bumped to prosody-13.0.6.tar.gz; --lua-version=5.4 and --idn-library=idn unchanged. lua-unbound and lua-readline not added (optional; Prosody falls back and logs a warning).
  2. Patches: ✅ Done. All 8 patches re-ported against 13.0.6.
    • muc.lib.patch: all 6 hunks applied with offset 7 — no re-port needed.
    • hidden.lib.patch: re-ported — 13.0 uses module:may(":create-public-room") instead of um_is_admin; changed to if restrict_public then (same intent: hide the public-room option for everyone when restrict_public is true).
    • mod_muc.patch: re-ported hunk 1 — 13.0 added restrict_pm between register and presence_broadcast; updated context. Hunk 2 applied with offset 17.
    • mod_muc_unique.patch: re-ported — 13.0 uses require "prosody.util.stanza" (namespaced); updated context.
    • mod_carbons.patch, mod_mam.patch, mod_muc_mam.patch, register.lib.patch: applied with line offsets, no re-port needed.
  3. Namespaced requires: ✅ No action needed. Old-style require "util.stanza" in vnctalk modules works via 13.0's compat loader. No bulk rewrite required.
  4. Roles/permissions: ✅ No action needed. mod_vnc_broadcast uses usermanager.is_admin which is deprecated but still works (logs a warning). strict_deprecate_is_admin is not set. The hidden.lib patch was re-ported against the new module:may API.
  5. mod_posix: ✅ Done. Removed from modules_enabled — 13.0 absorbed signal handling, pidfile writing, and the run_as_root check into core (util/startup.lua).
  6. Database: ✅ N/A for PostgreSQL — upgrade_table in mod_storage_sql.lua is MySQL-only (same as M1). Tables auto-created on startup via sql_manage_tables = true.
  7. Config: ✅ Done. prosodyctl check config passes. Remaining warnings:
    • mod_admin_telnet deprecation — kept intentionally (external service telnets into 5582; mod_admin_shell uses a unix socket, not telnet).
    • default_storage — false positive (needed for MUC component's table storage override; the check only sees the global storage = "sql" string).
    • mod_pubsub moved from modules_enabled to a Component — 13.0 requires pubsub to be loaded as a component, not a module.
  8. Healthcheck / telnet console: ✅ Unchanged. mod_admin_telnet (port 5582) still ships in 13.0.6 and works. healthcheck.sh unchanged. Telnet console regression test is a manual task (same as M1 §7).
  9. Verify: ✅ Compose-harness: 80 passed, 6 skipped, 0 failed. External testsuite run is manual.

Rollout and rollback (per milestone)

  1. MR → merge to main → CI builds development-$SHA and Trivy-scans it (expect the scan to improve dramatically; 0.11.6 is EOL with known CVEs).
  2. Deploy to dev, run the external testsuite, soak.
  3. Promote via the existing tag flow (prod-<version> / stable-<version> re-tag the dev image).
  4. Rollback hazard is the DB, not the image: M1 and M3 perform one-way SQL schema upgrades. Rolling back the image after the schema upgrade requires restoring the pre-upgrade dump. Always: fresh dump immediately before deploying a schema-upgrading milestone, and keep the previous image tag pinned (development-<sha> of the prior milestone) for emergencies.
Reference in New Issue View Git Blame Copy Permalink