Symptom
Running experiment_run_e2e.py with --streaming_engine precompute (SketchDB experiment mode), the experiment never finishes. remote_monitor.py on the remote node loops indefinitely printing:
Waiting for prometheus client container to stop running...
Investigating on the node:
docker ps -a shows sketchdb-prometheusclient and sketchdb-queryengine-rust both still Up (not exited).- Inside the prometheus-client container,
main_prometheus_client.py is alive but has accumulated almost no CPU time over the run — it appears to be blocked, not doing work.
- It holds one long-lived ESTABLISHED TCP connection to the query engine's HTTP query port (
:8088) and never writes any output to its output file or container logs.
remote_monitor.py's own exit check (docker inspect on the container) is behaving correctly — the container really is still running, so this isn't a bug in the monitor's polling logic itself.
Net effect: the client (and therefore the whole e2e experiment run) hangs indefinitely and has to be killed manually.
Symptom
Running
experiment_run_e2e.pywith--streaming_engine precompute(SketchDB experiment mode), the experiment never finishes.remote_monitor.pyon the remote node loops indefinitely printing:Investigating on the node:
docker ps -ashowssketchdb-prometheusclientandsketchdb-queryengine-rustboth stillUp(not exited).main_prometheus_client.pyis alive but has accumulated almost no CPU time over the run — it appears to be blocked, not doing work.:8088) and never writes any output to its output file or container logs.remote_monitor.py's own exit check (docker inspecton the container) is behaving correctly — the container really is still running, so this isn't a bug in the monitor's polling logic itself.Net effect: the client (and therefore the whole e2e experiment run) hangs indefinitely and has to be killed manually.