I've observed this in long experiments with large query lookback periods. There are long multiple instances of:
ssh -o StrictHostKeyChecking=no milindsr@node0.scratch1.cloudmigration-PG0.utah.cloudlab.us "pgrep -f remote_monitor.py" Remote monitor is still running. Will check again in 10 seconds
Not a problem wrt functionality but figuring this out will speed up running of experiments.
My hypothesis was that this was because of remote_monitor and/or PrometheusClient dumping large files and taking time to do so.
However, remote_monitor's monitor_output.json is small and PrometheusClient has been updated to stream results to file instead of dumping them all at once after the end of an experiment.
I've observed this in long experiments with large query lookback periods. There are long multiple instances of:
ssh -o StrictHostKeyChecking=no milindsr@node0.scratch1.cloudmigration-PG0.utah.cloudlab.us "pgrep -f remote_monitor.py" Remote monitor is still running. Will check again in 10 secondsNot a problem wrt functionality but figuring this out will speed up running of experiments.
My hypothesis was that this was because of remote_monitor and/or PrometheusClient dumping large files and taking time to do so.
However, remote_monitor's
monitor_output.jsonis small and PrometheusClient has been updated to stream results to file instead of dumping them all at once after the end of an experiment.