Evaluate replacing ActiveMQ with MQTT (Mosquitto) before #194's broker authentication

Summary

ECCE's programs exchange status messages through an ActiveMQ broker (Java, Debian's ActiveMQ "Classic" 5.17). On the client, a Java relay (JMSDispatcher) sits between the programs and the broker.

This issue evaluates replacing both with an MQTT broker, Mosquitto, and with direct connections from ECCE's programs through its C library. It is to be decided before the broker authentication of #194 is implemented, so that the authentication is written once, for the broker that stays.

Why consider it

No Java on the server or the client. The relay is the main obstacle to native macOS and Windows clients (Platform-agnostic ECCE: what a native macOS and Windows port would take #133).
Packaging. Mosquitto is packaged for Debian and in EPEL for RHEL/Rocky, with official macOS and Windows builds. ActiveMQ has no RHEL package and is installed by hand there.
Separation of users in the broker itself. Mosquitto's pattern access rules (for example, write access only below ecce/%u/#) give each user exclusive publication rights to their own topics. This is the separation Broker has no authentication: any local user can send job-control messages as another user (shared-broker mode makes it official) #194 requires, which on ActiveMQ would be built from JAAS authentication plus checks in every receiver.
Resources. Mosquitto is a small C process; each ActiveMQ broker starts a 512 MB Java heap for a low message rate.

What changes

ECCE's messaging classes (JMSMessage, JMSPublisher, JMSSubscriber) are reimplemented on libmosquitto, and the Java relay is removed.
Selectors. ECCE selects messages by properties (for example username='me'). MQTT has no selectors, so that information moves into the topic hierarchy, e.g. ecce/<user>/url_state. Session-local messages never reach the broker.
Event loops. The connections are integrated with the wx event loop, and with the job store's X Toolkit loop until that loop is replaced.
Deployment and tests. The three deployment modes (per-user, central server, shared system broker) and their tests (tests/apps/session_end.py).
Authentication. The data server's per-user accounts, through a Mosquitto authentication plugin or a password file, as planned in Broker has no authentication: any local user can send job-control messages as another user (shared-broker mode makes it official) #194.

A broker change alters what an administrator installs, so it belongs in 9.0.

Evaluation (one day)

Map every topic in data/client/config/ecce_messages, and every selector ECCE uses, onto MQTT topics, and list anything that does not map.
Send one message round trip (an Organizer state change received by Builder) through Mosquitto, with per-user access rules, and show that a second user cannot publish into the first user's topics.

Outcome:

If both succeed, Broker has no authentication: any local user can send job-control messages as another user (shared-broker mode makes it official) #194 is implemented on Mosquitto and ActiveMQ is retired in 9.0. The estimate is 8–12 days, which replaces most of Broker has no authentication: any local user can send job-control messages as another user (shared-broker mode makes it official) #194's ActiveMQ plan (about 4 days of it) and the planned direct STOMP access (v9 roadmap: the revamped ECCE (server/client split, new transport, native clients) #186).
Otherwise, Broker has no authentication: any local user can send job-control messages as another user (shared-broker mode makes it official) #194 proceeds on ActiveMQ as proposed there.

@mattasplund

Possible later step: job monitoring over MQTT (from #194)

Raised by @mattasplund in #194. If Mosquitto is adopted, eccejobmonitor could publish its message blocks to the broker instead of writing them to stdout for eccejobstore to read over the ssh connection. A running job's results would then no longer depend on that connection and the client staying up.

Constraint: compute nodes often have no outbound network, or reach only the login node. Direct publication is therefore optional, and the ssh/stdio path remains the guaranteed route. A port forwarded through the ssh session is a middle option.
Open question: which side parses. Today eccejobstore on the client parses the blocks and writes the data server. With the monitor publishing, something must still consume the messages while no client is running, for example a consumer next to the data server.
Access rules: a job publishes only under its owner's topics, using the same per-user rules as the rest of the evaluation.

Not part of the one-day evaluation. To be designed after it, if MQTT is chosen.

I did a bit of searching, and if we change the job monitoring to MQTT, there are a couple of interesting options. One, bish-bosh https://github.com/raphaelcohn/bish-bosh is a minimal shell script based MQTT client. The other is paho-mqtt https://pypi.org/project/paho-mqtt/is a python based client. I think it is fair to assume that any reasonable compute server would have python.

@mattasplund

@mattasplund Thanks, both noted. The client library is the easy part, and pure-Python or pure-Perl clients can be shipped with the job the way eccejobmonitor is copied to the run directory, so nothing needs to be installed on the cluster. paho-mqtt is the more likely choice: bish-bosh depends on whichever socket tools the node has, and TLS is harder there. Whichever client, two questions decide the design: whether the compute node can reach the broker (otherwise it falls back to the ssh path, or a port forwarded through it), and how a job gets a credential that lets it publish only under its owner's topics, which would be a per-job token issued at launch rather than the user's password. These go into the design after the evaluation.

Wrappers: ECCE_LOCAL_DATA skips the data server (#216)

@ohlincha

Evaluation result

Both criteria succeed, so #194 will be implemented on Mosquitto, and ActiveMQ and the Java relay will be retired in 9.0.

1. Mapping. Every topic in data/client/config/ecce_messages and every JMS selector maps onto MQTT topics, ecce/<user>/<topic>, with session messages under ecce/<user>/session/<host>_<display>/<topic>. Nothing that ECCE uses is lost.

username='<me>' becomes the user level of the topic. It is enforced by the broker rule pattern readwrite ecce/%u/#, so the sender's identity can be trusted.
The filter that drops a process's own messages becomes MQTT 5 No Local.
targetid/targetname are carried as MQTT 5 user properties and checked by the receiver.
JMSReplyTo is unused, and the 4096-byte UDP packet limit disappears.

2. Round trip. A prototype with the same calls as JMSPublisher/WxJMSSubscriber uses ECCE's own JMSMessage.C unmodified, on libmosquitto. An Organizer-style ecce_url_state message reached a Builder-style wx subscriber in two integrations: a library thread handing over with CallAfter, and AddSourceForFD. A poll() loop stood in for the job store's Xt loop. The broker ran with allow_anonymous false, a password file and per-user access rules. It refused every publication by a second user into the first user's topics (MQTT 5 reason code 135):

a state change, including a fake killed state;
ecce_ejs_kill for the first user's job;
ecce_quit into the first user's session.

A wildcard subscription to another user's topics received nothing, and a wrong password was refused at connect. 19/19 checks passed, repeatedly, against Debian's packages.

Measurements.

An idle Mosquitto uses 8 MB, against ActiveMQ's 512 MB Java heap.
EPEL 9 has mosquitto 2.0.22 and mosquitto-devel.
Debian needs mosquitto, libmosquitto1 and libmosquitto-dev (to build).

Pitfalls found.

Mosquitto grants a wildcard subscription that its access rules then serve nothing on, so tests must check that messages are delivered, not the subscription reply.
In acl_file, topic lines above the first user line apply only to anonymous clients.
mosquitto_passwd cannot read a password from stdin.
ecce_auth_changed currently carries the password. The port drops it, and receivers use the session credential store instead.

Decisions (@ohlincha, 2026-10-04)

Visibility. ecce_url_* and ecce_ejs_kill are readable by the owning user only. Today they reach every user on a shared broker. ecce_machreg_changed stays site-wide.
Session-local messages (app launch, quit) go through the broker, under the session's own subtree; there is no broker per session. In central-server mode, launching an app therefore requires the server's broker to be reachable.
Broker accounts come from a password file written by ecce-dataserver-adduser. An authentication plugin that checks the data server directly may follow later.
Login timing. The gateway asks for the login at session start, and the apps and eccejobstore reuse it from the session's credential store. Per-user mode needs no credential: its broker listens on a Unix socket in a directory only that user can read.
wx integration: the libmosquitto thread plus wxApp::CallAfter, which also suits native macOS and Windows clients. The job store's Xt loop watches the socket directly.

Deployment modes

Mode	Broker
Per-user	`mosquitto` on a Unix socket in the state directory, mode 0700, no password
Central server (`-remote`)	run by the server account, TLS on 8883, clients log in with their data server account
Shared broker (`siteconfig/SharedBroker`)	`ecce-broker.service` runs `mosquitto` with ECCE's configuration, authenticating the same way

Estimate: about 10.5 days

Work	Days
Messaging library: one connection per process, topic map, reconnect and resubscribe	2
Event loops, gateway start and session end without the relay, `auth_changed`	1.5
Remove Java and ActiveMQ; gateway and broker scripts, service unit, package dependencies	2
Authentication: access rules, password file, credential timing, TLS for central-server mode	1.5
Tests: `tests/jms`, `session_end.py` in all three modes, launch harness, teaching set	2
Documentation	0.5
Live verification with two real accounts	1

This replaces the ActiveMQ part of #194 and the planned STOMP access (#186). 8.x clients will not be able to talk to a 9.x broker, so this is a breaking change for 9.0. Job monitoring over MQTT (see above) is a separate design, after the port.

Messaging on libmosquitto: JMS publisher/subscriber over MQTT 5 (#213)

Gateway scripts on a per-user mosquitto, no relay (#213)

Remove ActiveMQ and the Java relay; per-session broker files (#213)

Authenticated TCP brokers: accounts from the data server, modes 2 and…

Central broker checks logins against the data server's users file (#213…

Docs: GETTING_STARTED, README and ENVIRONMENT for Mosquitto (#213, #194)

Carry the 8.x central-server mark over at the first 9.x start (#213)

Private broker: the topic user is 'local' (#216, #213)

The evaluation passed and the port is released in 9.0.0-alpha.3: ECCE's programs use libmosquitto directly, and ActiveMQ and the Java relay are removed (cfa748f to 9ce722f).

Merge wip/213-stage5: container harness for the central-server deploy…

Stage 5 done: tests/containers/central_server_test.py (merged 8c13ce5) runs the central-server deployment in rootless podman: one server container (ecce-remote-setup --server, data server, broker), two client containers as separate Unix accounts with real ecce -remote sessions on their own Xvfb, and ecce-broker.service under systemd. 65/65 checks pass (WebDAV and broker isolation checked by delivery, session end of one client, systemd start/reload/restart-after-kill/stop). About 5 min per run after the first image build. Run before releases next to tests/teaching (see tests/containers/README.md). The systemd container needs --cap-add SYS_ADMIN for DynamicUser under rootless podman; the unit file itself is unchanged. Not covered: two physical machines, TLS, a real ECCE session on a shared (mode 3) broker.

Return in a message box presses the button marked default, not Cancel…

Navigation Menu

Repository navigation

Evaluate replacing ActiveMQ with MQTT (Mosquitto) before #194's broker authentication #213

Summary

Why consider it

What changes

Evaluation (one day)

Possible later step: job monitoring over MQTT (from #194)

Evaluation result

Decisions (@ohlincha, 2026-10-04)

Deployment modes

Estimate: about 10.5 days

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Evaluate replacing ActiveMQ with MQTT (Mosquitto) before #194's broker authentication #213

Description

Summary

Why consider it

What changes

Evaluation (one day)

Activity

ohlincha commented on Oct 1, 2026

Possible later step: job monitoring over MQTT (from #194)

mattasplund commented on Oct 2, 2026

ohlincha commented on Oct 2, 2026

ohlincha commented on Oct 4, 2026

Evaluation result

Decisions (@ohlincha, 2026-10-04)

Deployment modes

Estimate: about 10.5 days

ohlincha commented on Oct 4, 2026

ohlincha commented on Oct 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions