Repository navigation
Artifacts stop pulling without visible failure #2156
Description
Sometimes, artifacts get stuck in a state of being "pulled" but no network traffic occurs. Buildstream will never finish or stop trying to pull these artifacts. These pulls never hit timeouts or retries. I can even turn off my network connection and buildstream will still present these as being actively pulled for hours, never showing a failure (even though the network is down).
I have hesitated to open this issue for a while because, unfortunately, I have not found clear reproduction steps for this. This has just occasionally happened for months with no clear pattern. Nothing is logged with default logging options.
Activity
wsalmonct commentedon Jul 27, 2026
This is often the case for bugs between buildstream and buildbox.. They can be very frustrating!
Its worth noting that all the pulling happens in buildbox-casd.
So ether buildbox-casd has not noticed, or its failed to let buildstream know.
@juergbi @ColdrickSotK is it posible for buildbox-casd to fail a pull and not let buildstream know?
juergbi commentedon Jul 31, 2026
Have you already tried setting
keepalive-time(and possibly alsorequest-timeout) as documented in https://docs.buildstream.build/2.7/using_config.html#connection-configuration? gRPC doesn't enable keepalive by default but maybe we should.juergbi commentedon Jul 31, 2026
Too aggressive keepalive can cause problems depending on server and possible proxy. I think a default of 5 min (which is also the server side default for
GRPC_ARG_HTTP2_MIN_RECV_PING_INTERVAL_WITHOUT_DATA_MS) may be reasonable for BuildStream, though, especially considering that this will only apply to active sessions, not some persistent background daemon. It may make sense to keep keepalive disabled by default in buildbox-casd upstream, though, as buildbox-casd may be used as a persistent background daemon where regular pings may not be desired.ada-magicat commentedon Jul 31, 2026
@juergbi
We do this in g-b-m, yes: https://gitlab.gnome.org/GNOME/gnome-build-meta/-/blob/3c6c56f3dfd49073343c1f84a264542b494e3a90/project.conf#L11
I've seen the bug happen before and after we set those limits though.
Indeed, setting it too low makes me get nonsense pull failures (as in, it "times out" while pulling normally)