Job progress in the Organizer: SCF, optimisation and frequency progress, and jobs out of contact #284
Description
Show how a running calculation is getting on, for all codes, in the Organizer. Agreed 2026-10-11.
Data path
- Each code's
.descgets[PROGRESS]entries that match its per-cycle lines (SCF cycle and ΔE, optimisation step, energy and convergence criteria, frequency point k of 6N). The job monitor already reads the output as it is written. - The monitor sends progress messages throttled to at most one per second. They are not stored as properties; only the last snapshot is kept with the calculation, so a restarted Organizer still shows where the job got to.
- The job store relays them on a broker topic
ecce_url_progress, on the owner's own topic (as decided for Evaluate replacing ActiveMQ with MQTT (Mosquitto) before #194's broker authentication #213), so on a central server users see only their own jobs. - ECCE-QM writes its own machine-readable lines, e.g.
progress scf cycle=12 energy=… dE=… grad=…,progress opt step=4 energy=… maxforce=…/…,progress freq point=14/54. Its SCF history is kept (small) for a convergence plot.
Display
- Summary pane: a Progress box between the launcher and viewer rows, shown once a job has sent progress, with only the lines that apply: optimisation step, energy and a small energy plot, each convergence criterion against its target (✓/✗); SCF cycle and ΔE with a log-scale plot; frequency point k of N with a bar and a time estimate; "updated n s ago". After the job, one or two lines (converged in n steps, final energy, real/imaginary frequencies, ZPE).
- Project table (project selected; Options › Project View): a Progress column after State in the Basic, Launch and Detail views, one short line per running job (
Freq 14/54with a bar,Opt step 4 · SCF 10,⚠ no contact for 17 min), empty for other states. Filled from the live messages and the snapshot, not from a stored property.
Out of contact
- No heartbeat for about 3 intervals: the state shows "running?" with a hollow amber circle; the Progress box greys the last known progress, says when the job was last heard from and that it may still be running.
- Actions: Reconnect (existing Run Management › Reconnect), Check job on machine (process or queue job present, output last written), Show output so far (tail of the output).
- A missing heartbeat alone never sets a final state: Completed/Failed only from the output or the machine check, System Failure only when the job is gone without usable output.
- On reconnect the monitor reads on from its bookmark, so stored properties catch up; intermediate SCF progress is not replayed (except ECCE-QM's stored history).
- Related: Jobs submitted and monitored by the ECCE server, not the client #208 (monitoring by the server).
Order
- ECCE-QM end to end (progress lines, message, relay, summary box, frequency bar, snapshot), to settle the design.
- ORCA, Gaussian 16, NWChem, MOPAC patterns; the table column.
- Out-of-contact handling and the three actions.
- Later: Quantum ESPRESSO, GROMACS, CP2K (k-points/ionic steps, MD steps), and a live convergence plot in the Viewer.
Activity
ECCE-QM: progress lines for the job monitor (#284)
eccejobmonitor: [PROGRESS] entries and the jmPROG message (#284)
Job store: relay progress as ecce_url_progress, store the last snapsh…
Organizer: Progress box in the calculation summary (#284)
ohlincha commentedon Oct 11, 2026
Phase 1 on main (2751daf, c5a2c5b, f939235, 6b798bb): ECCE-QM progress lines; [PROGRESS] entries in .desc files (regex → fields, no script, never consume a line); the job monitor sends jmPROG at most once a second when the job store asks for it (-progress 1), so mixed versions are safe; the job store relays it as ecce_url_progress on the owner's topic and keeps the last snapshot as calculation metadata (at most every 10 s, and at the end); the Organizer's summary pane shows the Progress box and repaints only the box. Checked with a real ECCE-QM water GeoVib through the job store (18–20 messages, ≥1 s apart, monotonic, final snapshot stored) and a headless Organizer test. Not tested: central-server and shared-broker modes, Windows. Next: ORCA, Gaussian, NWChem, MOPAC, Quantum ESPRESSO and GROMACS patterns and the project-table column (phase 2), then out-of-contact handling (phase 3).
ohlincha commentedon Oct 11, 2026
Agreed 2026-10-11 (phase 3): the Progress box is collapsible. Collapsed it is one live line, the same text as the project-table column ("▸ Progress Opt step 4 · SCF 10 · updated 2 s ago"); click the heading to toggle; the state is remembered per user, expanded by default. Out of contact, the collapsed line turns amber ("⚠ no contact for 17 min") without forcing the box open.
ORCA: [PROGRESS] entries for SCF, optimisation and numerical frequenc…
Gaussian 16/09: [PROGRESS] entries for SCF, optimisation and frequenc…
NWChem: [PROGRESS] entries; monitor reads stdout for progress, Rememb…
MOPAC: [PROGRESS] entries; box shows target-less values and a bare 'f…
Quantum ESPRESSO: [PROGRESS] entries for SCF and relax/vc-relax steps (…
GROMACS: [PROGRESS] entries (step k of nsteps); ctest progress-codes (#…
Project table: Progress column, one cell per progress message (#284)
Job store: out of contact is a notice, silence reconnects with the jo…
Check Job on Machine: whether the job and its output are still there (#…
Organizer: "running?" out of contact, its three actions, and a foldin…
Tests and notes for out of contact (#284, #208)
ohlincha commentedon Oct 11, 2026
Phase 3 on main (f5c446f, a216bea, b167dc0, f9d7541): the job store detects silence (no heartbeat for max(60 s, 3 heartbeats): 60 s local, 180 s remote) and sends contact lost/ok on ecce_url_progress, stored as ecce:contact. "running?" (hollow amber ring) is a display state; the stored run state stays Running. The Progress box greys the last progress, says when the job was last heard from, and offers Reconnect, Check Job on Machine (new Run Management item; reports only) and Show Output So Far. The box folds to one line, remembered per user. A heartbeat timeout now keeps the job Running and reconnects (was Waiting → Submitted); reconnect after max(180 s, contact time + 120 s), i.e. 300 s instead of 180 s for remote machines. Tested with a real ECCE-QM job: monitor stopped and resumed; job killed while the monitor was stopped (ends Unsuccessful, not System Failure); silence past the reconnect time (stays Running, completes). Not tested: remote ssh or queue machines, central-server and shared-broker modes, Windows.
Progress box: repaint the parent when the box changes height, so a va…
ohlincha commentedon Oct 12, 2026
Change to the display after the job ends (Andy, 2026-10-12): keep the Progress box as it was during the run, with its plots (SCF convergence, optimisation energy and criteria, frequency points), when the job finishes. Do not replace it with the one or two summary lines; add the final lines (converged in n steps, final energy, real/imaginary frequencies, ZPE) below the plots instead. The finished box is a useful overview of how the SCF and the geometry converged.
Organizer: a finished job keeps its Progress plots, the outcome goes …
Organizer: build the progress record of a finished calculation that h…
ohlincha commentedon Oct 12, 2026
Released in 9.0.0: a finished job keeps its Progress plots, with the outcome below them (8cd2c2a), and finished calculations without a record get one built from their stored output (5887cf6). MOPAC and other codes without progress patterns follow in #303.