docs: reconcile AGENTS.md with Prosody 13.0.6 + Lua 5.4 and current tooling

Part-of: <http://gitlab.vnc.biz/uxf/vnctalk-prosody/-/merge_requests/3>

AGENTS.md

@@ -2,50 +2,77 @@
 
 ## What this repo is
 
-Dockerized Prosody 0.11.6 XMPP server for VNCtalk. It builds Prosody from source, applies patches to core upstream files, and bundles custom Lua modules. There is no language package manager or test framework.
+Dockerized Prosody **13.0.6** XMPP server for VNCtalk, built against **Lua 5.4** on Alpine 3.23. The Dockerfile compiles Prosody from source, applies `.patch` unified diffs to upstream core files, and layers on custom Lua modules from `vnctalk/`. There is no Lua package manager, linter, or unit-test framework; verification is image-build + an optional pytest + slixmpp integration suite against a running server.
+
+> **Version drift warning:** `patches/README.md`, `CLAUDE.md`, and `helm/prosody/Chart.yaml` (`appVersion: 0.11.6`) still reference older Prosody versions. The **Dockerfile is the source of truth** (downloads `prosody-13.0.6.tar.gz`, `--lua-version=5.4`). `argo/prosody.yaml` overrides the helm tag to `release-13.0.6-vnc`. Trust the Dockerfile when docs disagree.
 
 ## Build & verification
 
-- **Only build command:** `docker build -t <tag> .`
-- No `make`, `npm`, rockspec tests, or linting exists.
-- `test.sh` is a **manual integration helper** that runs the container with a long list of required env vars. It is not an automated test suite.
-- **Python test suite** lives in `tests/`. It is an optional pytest/slixmpp integration suite used to verify a running XMPP server against VNCtalk requirements.
-  - Set up: `python3 -m venv tests/venv && source tests/venv/bin/activate && pip install slixmpp pytest pytest-asyncio aiohttp lxml psycopg2-binary asyncpg`
-  - Run dry (no server): `pytest tests/ -v --skip-live -c tests/pytest.ini`
-  - Run against a server: set `XMPP_JID`, `XMPP_PASSWORD`, `XMPP_DOMAIN`, `MUC_DOMAIN`, etc. For full coverage also provide `XMPP_JID2`/`XMPP_PASSWORD2` (distinct account), `PG_HOST`/`PG_USER`/`PG_PASSWORD`, and `REST_USER`/`REST_PASSWORD` if the `/rest` endpoint is behind a reverse proxy with HTTP Basic Auth. Add `--verify-ssl` only if the server has a real certificate (tests disable verification by default because containers use self-signed certs). Then `pytest tests/ -v -c tests/pytest.ini`
-  - See `tests/AUDIT.md` for a gap analysis and `tests/MANUAL_TESTS.md` for contracts that require manual verification.
+- **Build:** `docker build -t <tag> .` (multi-stage Alpine; builder compiles Prosody, final stage copies `/usr/local/`).
+- **Build gotcha:** `./configure --idn-library=idn` is mandatory. Prosody 0.12+ defaults to ICU, but the runtime image ships no ICU data files (`U_FILE_ACCESS_ERROR`); libidn is used instead. The builder also passes `--lua-version=5.4`.
+- **Docker-compose test harness** (`docker-compose.yml`): postgres + an aiohttp mock (stands in for auth/FCM/file-share/avatar backends) + prosody. Convenience wrappers in the `Makefile`:
+  - `make up` — build + start postgres/mocks/prosody (run pytest from host against published ports).
+  - `make down` / `make logs`.
+  - `make test` runs the suite in the `tester` service, but `tester` is gated behind the `ci` profile (`profiles: ["ci"]`) — invoke as `docker compose --profile ci run --rm tester pytest tests/ -v -c tests/pytest.ini`. Easier: use `run-compose-tests.sh` (see below).
+- **`run-compose-tests.sh`** is the recommended one-shot runner: creates `tests/venv`, brings up the stack, waits for prosody health, and runs the full suite from the host with all env vars set. Flags: `--no-down` (leave stack up), `--down` (tear down only). It auto-remaps telnet to 5583 if 5582 is taken (e.g. by a stale kubectl port-forward).
+- **`run-tests.sh` / `run-all-tests.sh`** target an **external microlab deployment** (hardcoded hosts/creds), not the compose harness. They require an SSH tunnel to PostgreSQL (`ssh -p 60024 … -L 14322:127.0.0.1:5432`) and a `kubectl port-forward services/telnet 5582:5582 -n prosody`. Not for local image testing.
+- `test.sh` is a **manual smoke test**: `./test.sh <image-tag>` runs the image with example env vars. Not automated.
+- **Python integration suite** in `tests/` (pytest + slixmpp, `~86` tests across `test_01`–`test_12`). See `tests/README.md`, `tests/AUDIT.md`, `tests/MANUAL_TESTS.md`.
+  - Set up: `python3 -m venv tests/venv && source tests/venv/bin/activate && pip install -r tests/requirements.txt`
+  - **Always pass `-c tests/pytest.ini`** — it sets `asyncio_mode = auto`; without it collection fails with an asyncio error.
+  - Dry run (no server, skips live-only tests): `pytest tests/ -v --skip-live -c tests/pytest.ini`
+  - Single file: `pytest tests/test_05_patches.py -v -c tests/pytest.ini`
+  - Collect-only sanity check: `pytest tests/ --collect-only -c tests/pytest.ini`
+  - Against a server: set `XMPP_HOST`, `XMPP_PORT`, `XMPP_JID`, `XMPP_PASSWORD`, `XMPP_DOMAIN`, `MUC_DOMAIN`. For full coverage also `XMPP_JID2`/`XMPP_PASSWORD2` (distinct account), `REST_URL`, `BOSH_URL`, `WS_URL`, `ADMIN_TELNET_HOST`/`ADMIN_TELNET_PORT`, `PG_HOST`/`PG_PORT`/`PG_USER`/`PG_PASSWORD`/`PG_DB`, `MOCK_URL` (side-effect assertions), and `REST_USER`/`REST_PASSWORD` if `/rest` is behind HTTP Basic Auth. SSL verification is off by default (containers use self-signed certs); add `--verify-ssl` only with a real cert.
 
 ## Code layout
 
 | Path | Purpose |
 |------|---------|
-| `patches/` | Overrides for upstream Prosody core files (e.g., `mod_mam.lua`, `mod_muc.lua`, `moduleapi.lua`). Copied over the upstream source during the Docker build. |
-| `vnctalk/` | Custom Prosody Lua modules. Copied wholesale to `/usr/local/lib/prosody/modules/` in the image. |
-| `config/` | Runtime templates and shell scripts. |
+| `Dockerfile` | Multi-stage build. Builder applies `patches/*.patch` to the extracted Prosody source, then `./configure --idn-library=idn --lua-version=5.4 && make install`. Final stage copies the built tree + `vnctalk/` modules + `config/`. |
+| `patches/` | Unified diffs (`a/...`/`b/...` headers, paths relative to the Prosody source root) applied in the builder with `patch -p1 --fuzz=0` **before** configure. Applied via glob — **adding/removing a `.patch` needs no Dockerfile change.** `--fuzz=0` means upstream drift fails the build loudly. `patches/README.md` documents each patch's intent (but mislabels the base version — see drift warning above); `upgrade-plan.md` tracks per-version disposition. |
+| `vnctalk/` | VNCtalk-original Prosody modules, copied wholesale to `/usr/local/lib/prosody/modules/`. |
+| `config/` | Runtime templates and shell scripts (rendered/run at container start). |
+| `tests/` | pytest suite + `mocks/` aiohttp mock service + `Dockerfile` for the tester image. |
+| `helm/prosody/` | Helm chart for Kubernetes deployment (`Chart.yaml`, `values.yaml`, `templates/`). `appVersion` in `Chart.yaml` is stale; the actual image tag is set in `argo/prosody.yaml`. |
+| `argo/prosody.yaml` | ArgoCD `Application` manifest — the canonical deploy config (image tag, ingress, env). Useful reference for production wiring. |
+| `db-customization/` | PostgreSQL schema/migration SQL for the Prosody 13 DB (`prosody-13-migration-once.sql`, rules/triggers, no-owner variants). Applied out-of-band, not by the container. |
+| `scan/` | CI artifact output (Trivy image-scan results), not source. |
+| `upgrade-plan.md`, `PATCHES_AND_MODULES.md` | Reference docs tracking patch/module disposition across versions. |
+| Root `*.log`, `testrun-*.md`, `m1-manual-tasks.md`, `db-change-analysis.md`, `vc-set.xml` | Working scratch / run-report artifacts from past test sessions, not source. |
 
-- `patches.list` is **stale and unused** by the build; the Dockerfile hardcodes each `cp` command. If you add a patch, update the Dockerfile `RUN cp …` block.
+### Patch vs. module — where a change belongs
+
+- **`patches/`** for changes to *upstream* Prosody behavior (core or bundled modules: `mod_mam`, `mod_muc`, `mod_carbons`, `muc.lib.lua`, `register.lib.lua`, `hidden.lib.lua`, `mod_muc_unique`, …). Regenerate with `diff -u pristine/<path> patched/<path>` keeping the `a/<path>`/`b/<path>` header form. Document intent in `patches/README.md`.
+- **`vnctalk/`** for VNCtalk-specific new functionality (`mod_vnc_*`, `mod_http_rest`, `mod_alias`, `mod_webpresence`, …).
+- When adding a module of either kind, also enable it in `config/prosody.cfg.lua.template` (global `modules_enabled`, or the `modules_enabled` of the specific host/component), or it installs but stays inactive.
 
 ## Configuration
 
-- The real config source is `config/prosody.cfg.lua.template`. At container startup, `startup.sh` renders it with `envsubst` to `/etc/prosody/prosody.cfg.lua`.
-- **Edit the template**, not a generated `.cfg.lua` file.
-- Template variables use `${VAR}` syntax. Required runtime env vars include: `prosodyDomain`, `prosodyDBhost`, `prosodyDBname`, `prosodyDBuser`, `prosodyDBpass`, `hybridaAuthUrl`, `fcmApiKey`, `fcmApiUrl`, `fileShareBaseUrl`, `fileShareSecret`, `avatarUploadUrl`, `avatarUploadUser`, `avatarUploadPass`.
+- The real config source is `config/prosody.cfg.lua.template`. At startup `startup.sh` renders it with `envsubst` to `/etc/prosody/prosody.cfg.lua`. **Edit the template, not a generated `.cfg.lua`.**
+- Template variables use `${VAR}` syntax. The full set (verify here before assuming a knob doesn't exist): `prosodyDomain`, `prosodyDBhost`, `prosodyDBname`, `prosodyDBuser`, `prosodyDBpass`, `prosodyDBport`, `hybridaAuthUrl`, `fcmApiKey`, `fcm_api_url`, `del_api_url`, `fileShareBaseUrl`, `fileShareSecret`, `avatarUploadUrl`, `avatarUploadUser`, `avatarUploadPass`, `componentSecret`, `DEFAULT_JITSI_CONFERENCE`, `log_slow_events_threshold`, `SMACKS_HIBERNATION_TIME`, `C2S_STANZA_SIZE_LIMIT`, `S2S_STANZA_SIZE_LIMIT`, `PROSODY_LOG_LEVEL` (set by `startup.sh` from `PROSODY_DEBUG=true`).
+- **Env-var naming gotcha:** the template uses snake_case `${fcm_api_url}` / `${del_api_url}`. `test.sh` historically passes camelCase `fcmApiUrl`/`fcmDelUrl`, which `envsubst` silently leaves unsubstituted. The compose harness uses the correct snake_case names — match the template.
+- The template defines **two VirtualHosts + three Components**: `VirtualHost "${prosodyDomain}"` (HTTP-async auth), `VirtualHost "anon.${prosodyDomain}"` (anonymous auth), `Component "conference.${prosodyDomain}" "muc"` (MUC, own module list + MAM config), `Component "pubsub.${prosodyDomain}" "pubsub"`, and `Component "broadcast@${prosodyDomain}" "vnc_broadcast"`. Most module/config changes target one specific host/component, not the global list.
+- `startup.sh` appends defaults to the rendered config when env vars are unset: `SMACKS_HIBERNATION_TIME` (300s; compose lowers to 10s so SMACKS-expiry tests don't wait 5 min), a random `componentSecret` (port 5347), `DEFAULT_JITSI_CONFERENCE`, `C2S/S2S_STANZA_SIZE_LIMIT` (5 MB), and `PROSODY_LOG_LEVEL`.
+- The config sets `run_as_root = true` and `console_interfaces = { "*" }` / `http_interfaces = { "*", "::" }` — required because production and the compose harness run as root and expose the telnet/HTTP ports beyond loopback.
 
 ## Runtime quirks
 
-- `startup.sh` (main container) generates certs, renders config, and runs `/usr/local/bin/prosody -F`.
-- `startup-sidecar.sh` runs a separate Node `http-server` on port 8080 for static files (redirect page, status page). This is **not** the XMPP server.
-- `healthcheck.sh` compares `/etc/tls-update/tls.crt` with the currently loaded cert. If they differ, it exits **2** (not 1) to force a container restart so the new cert is picked up. It then checks TCP port 5582 (admin telnet).
+- `startup.sh` (main `CMD`): writes TLS cert/key (from `$prosodySSLcert`/`$prosodySSLkey` env, else a baked-in dev default, overridden by `/etc/tls-update/tls.{crt,key}` if present), renders config, then runs `/usr/local/bin/prosody -F` (also tails `/var/log/prosody/prosody.log`).
+- `startup-sidecar.sh` is a separate entrypoint for a static-file/redirect sidecar (`http-server` on :8080). **Not** the XMPP server; not invoked by the main `CMD`.
+- `healthcheck.sh`: if `/etc/tls-update/tls.crt` differs from the loaded cert, exits **2** (not 1) to force a container restart so the new cert is picked up. Otherwise checks the admin telnet port 5582 via `check_tcp`.
+- The image declares `USER prosody`, but `startup.sh` writes certs/config into root-owned `/etc/prosody`. Production and the compose harness run as root (`user: "0:0"`).
 
 ## CI / deploy
 
-- GitLab CI (`.gitlab-ci.yml`). Builds push to Google Container Registry (`eu.gcr.io`).
-- `main` branch builds a `development` image.
-- Git tags matching `prod-*` or `stable-*` trigger promotion jobs that **retag** the existing dev image rather than rebuilding.
+- GitLab CI (`.gitlab-ci.yml`). `main` builds + pushes to `eu.gcr.io/vnc-development/vnctalk-prosody:development-$CI_COMMIT_SHORT_SHA`, then Trivy-scans the saved image.
+- Tags `prod-*` retag the matching `development-<version>` to `eu.gcr.io/vnc-dev-275609/vnctalk-prosody:production-<version>`. Tags `stable-*` retag to `eu.gcr.io/vnc-stable/vnctalk-prosody:stable-<version>`. **No rebuild on promotion** — the `<version>` suffix must match an existing dev tag.
+- Kubernetes deploy is via the Helm chart in `helm/prosody/`, synced by ArgoCD (`argo/prosody.yaml`). The Argo manifest sets the live image tag (`release-13.0.6-vnc`) and TLS-update sidecar (`tlsUpdates`).
 
 ## Module conventions
 
-- Lua 5.2. Modules follow Prosody conventions (`module:hook`, `module:open_store`, `module:get_option_string`, `module:provides`).
-- Authentication is delegated to an external HTTP endpoint via `mod_auth_http_async`.
-- Push notifications are handled by `mod_vnc_fcm` (and `mod_vnc_fcm_hin`) using a configurable FCM proxy URL.
-- `mod_http_rest` exposes an HTTP endpoint at `/rest` that accepts `text/xml` bodies and injects them as XMPP stanzas into the server.
+- Lua 5.4. Modules follow Prosody conventions (`module:hook`, `module:open_store`, `module:get_option_string`, `module:provides`).
+- Authentication is delegated to an external HTTP endpoint via `mod_auth_http_async` (`hybridaAuthUrl`) on the main VirtualHost; the global default is `internal_hashed`.
+- Push notifications: `mod_vnc_fcm` / `mod_vnc_fcm_hin` over a configurable FCM proxy URL (`fcm_api_url`).
+- `mod_http_rest` exposes `/rest` accepting `text/xml` bodies, injected as XMPP stanzas (fires `vnc-rest-message`, consumed by the `mod_carbons`/`mod_mam` patches).
+- The `mod_mam` patch always stores (`shall_store → true`) and only archives stanzas with a `<body>`, because users live in the external HTTP auth backend so Prosody's `user_exists()` can't be trusted.