Jenkins sometimes does not show "Resume build" option

[trivikr]

Refs: nodejs/node#65652 (comment)

[trivikr]

When this happens, collaborators have to start a new CI.

[sxa]

Odd. Both seem to have been started manually by @jasnell and on the same PR so there's no obvious reason why one would show it and the other wouldn't. I'm not sure what we can do about this but I've never noticed the absence of the button before although James mentioned he has seen it before. I've "locked" both of the jobs referenced in that issue for now in case we can do some diagnosis on it:

• https://ci.nodejs.org/job/node-test-pull-request/76776/ (with resume button)
• https://ci.nodejs.org/job/node-test-pull-request/76787/ (no resume button)

I will also note that the way we are doing our PR test builds is with a deprecated/unmaintained jenkins plugin so there will be limited chance of getting it addressed if the failure is there.

[sxa]

Also spotted on two of @codebytere's recent jobs (both on separate PRs): https://ci.nodejs.org/job/node-test-pull-request/76791/ and https://ci.nodejs.org/job/node-test-pull-request/76792/

[sxa]

The ones without the resume button do not seem to have this section in the build's build.xml:

<     <com.tikal.jenkins.plugins.multijob.MultiJobResumeBuild>
<       <run class="com.tikal.jenkins.plugins.multijob.MultiJobBuild" reference="../../.."/>
<     </com.tikal.jenkins.plugins.multijob.MultiJobResumeBuild>

None of the PR test jobs between 76789 and 76795 had the resume link. All of those were started by @codebytere although the one before those at https://ci.nodejs.org/job/node-test-pull-request/76788/ was also by her and has resume build ... although unlike the ones that don't have it 76788 was started from a resume build link 🤔

[richardlau]

I think that in order for the Resume Build button to appear, the node-test-pull-request job has to complete (the spawned child jobs can fail). I think this is related to #4453 -- if the node-test-pull-request job gets disconnected, it doesn't add the Resume Build button.

[codebytere]

I think @richardlau's read is right from what i can see in the plugin: MultiJobBuild$MultiJobRunnerImpl.run() only does getBuild().addAction(new MultiJobResumeBuild(...)) after super.run(listener) returns normally with a non-SUCCESS result. If super.run() throws, the action is never attached and there's no Resume link.

Every no-resume build i checked (76787, 76791, 76792, 76955) has the same ending in its console: the child node-test-commit finishes, the parent prints Starting to gather test results!, and then AbstractBuild$AbstractBuildExecution.run -> Launcher$RemoteLauncher.kill -> Channel.call throws ChannelClosedException on JNLP4-connect connection from 67.158.54.159, which is test-mnx-ubuntu2204-x64-1 (jenkins-workspace). The parent held an executor there for the whole 2-10 h, the agent's remoting channel reconnected at some point in between (the traces show two different Channel@… ids across the week), and the end-of-build process-tree kill on the stale channel is what throws. Builds that ended FAILURE or ABORTED without that trace (76776, 76788, 76967, 76993, 77016) all have /resume/; 76966, which i aborted by hand before node-test-commit had started, lacks it for the same reason via the abort's interrupt. Who started the run, or whether it came in through ncu-ci, looks like coincidence; what the no-resume builds share is that agent's connection not surviving the wait.

The same exception also explains the original report in #4453 (parent red with all children green): the throw happens after the children are collected, so the parent is marked FAILURE regardless of their results.

One thing that made this harder to see from scripts: MultiJobResumeBuild isn't @Exported, so api/json?tree=actions[_class] never lists it even when the link is there; checking for /resume/ in the build page HTML is the only way i found.

Possible directions, for whoever knows the agent setup better than i do: find out why that MNX host's JNLP channel drops (it's also the box from #4442), or pin node-test-pull-request / node-test-commit parents to an agent local to the controller since they only do the git/rebase work and then sit waiting.