Local broker with a remote data server (cloud/VM data server deployment) #246

Summary

Allow a client to use a per-user local broker together with a remote data server, with jobs running on registered remote machines.

Current state

There are three deployment modes (GETTING_STARTED, "Deployment modes"):

1. local: per-user broker on a Unix socket, local data (FileEDSI);
2. central server (-remote): the server's broker (TCP 8088) and its data server;
3. shared system broker.

There is no way to combine a local broker with a remote data server.

Motivation

A small group can run a data server on a rented cloud VM or a campus virtual server. Keeping the broker on the client means:

• one fewer port open on the server;
• broker passwords never leave the client;
• the server needs only the HTTP(S) data server.

Compute on the same or another host is a registered machine, as today.

To establish

• Which messages in mode 2 require a broker shared between users? Machine-registration change notices are one. Do any others exist, or are all topics per user (ecce/<user>/)?
• Does anything server-side (data server, scripts) publish to the broker? If so, it has no path to a client-local broker.
• Job monitoring: eccejobstore/eccejobmonitor run on the client and reach the compute host over ssh. With the client closed, monitoring pauses and resumes on the next start. Document this. Whether a server-side monitor is wanted is a separate question.
• How it is selected: an option of "Connect to a server", or a site/user setting. The first-start question should not gain a third choice unless needed.

Constraint

Mode 2 (central server with its broker) stays supported unchanged.