bst shell should be able to stage the specified element on top of a base element

See original issue on GitLab
In GitLab by [Gitlab user @tlater] on Jun 12, 2018, 12:32

Background

I find that I avoid using type: runtime on very low-level elements most of the time, even if they are technically runtime dependencies. This is because they tend to be large and contain dependencies mostly available on systems I'll want to deploy software to. It doesn't seem useful for bst checkout to checkout something like /bin/sh, for example.

This works for anything I want to run in CI, or actually use for deployable builds, but as soon as I start developing I want to run bst shell on whatever I am working on.

bst shell currently requires the runtime of any element I want to shell into to contain a /bin/sh, and therefore consistently breaks unless I add type: runtime to my base platform dependency. As a result I end up constantly adding and removing dependencies for debugging.

There is bst shell --sysroot, but it requires a checkout of the base platform, which makes it cumbersome to use for debugging. It also doesn't update with potential changes to the base platform, which makes it feel wrong for this use case.

Acceptance Criteria

We should have a way to avoid specifying type: runtime on base platforms with bst shell. Options include:

Adding a --platform option which specifies a dependency to stage the given element on top of
Allow specifying multiple elements

This is still very up for discussion, please do comment :)

Update from Laurence, February 2019: This was discussed at the Gathering in Jan 2019 and a summary mail sent to the list which was discussed a little.

Update

This is to be handled simply by allowing multiple elements to be specified to bst shell, allowing one to stage multiple things. For the case of bst shell --build, it is easy enough to consider the first specified element to be "the build element", as mentioned elsewhere in the comments below, there can only be one build environment.

A note about CLI consistency, is that this is much the same issue as #771, and should be handled the same way.

Allowing multiple targets being specified here will not cause breakage in the CLI api, and as such we will not be blocking on this for the 2.0 milestone.

In GitLab by [Gitlab user @cs-shadow] on Jun 12, 2018, 14:55

I would quite like something like this too.

I would much prefer a solution where bst shell <element> "just works" without the need for any other options as that would be the more intuitive behavior IMHO.

To that end, another idea that comes to mind is defining something like base-runtime-platform(s) in project.conf which will get staged automatically anytime you run bst shell on any element. This would make sense for projects where almost all the elements will have the same base platform for runtime.

The same idea could be extended to build-time dependencies too. If all of .bst files in a project are going to depend on the same base platform, then having a dependency on it in each of the file seems overly verbose. This could be solved if there was a declare a base-build-platform(s) property in project.conf

cc [Gitlab user @sstriker] as we were discussing the same issue a while back.

In GitLab by [Gitlab user @LaurenceUrhegyi] on Jun 18, 2018, 16:15

marked this issue as related to #425

In GitLab by [Gitlab user @LaurenceUrhegyi] on Jun 18, 2018, 16:16

marked this issue as related to #425

In GitLab by [Gitlab user @LaurenceUrhegyi] on Jun 18, 2018, 16:16

removed the relation with #425

In GitLab by [Gitlab user @jennis] on Sep 18, 2018, 12:40

I would much prefer a solution where bst shell <element> "just works" without the need for any other options as that would be the more intuitive behavior IMHO.

+1, especially when you build an application in BuildStream, as one of the first things you're likely to do is bst shell application.bst and try to test it within the shell.

Here is my current workaround, which as you will be able to see, is much less than ideal:

At the moment, I'm currently building vim.bst, which, within it's pipeline, contains no runtime dependencies that provide the sh command, thus I cannot bst shell into it.
If I want to shell into something where I can execute vim --version or something similar, I'm currently creating a stack element that looks something like this:

kind: stack

runtime-depends:
- vim.bst
- platform.bst

I then bst build vim-stack.bst, then bst shell vim-stack.bst

^ Not ideal...

The same idea could be extended to build-time dependencies too. If all of .bst files in a project are going to depend on the same base platform, then having a dependency on it in each of the file seems overly verbose. This could be solved if there was a declare a base-build-platform(s) property in project.conf

Again +1.

In GitLab by [Gitlab user @tristanvb] on Sep 18, 2018, 15:24

Note that something similar was discussed on the mailing list

Basically, developers want to include their own personal tooling in their shell, and might have some tooling which is compatible with a runtime they are using inside bst shell.

The idea is to allow either overlay or underlay of a user specified filesystem into the shell; so that it is staged along with the regular BuildStream staging mechanism, so that a user can provide their own input to the shell, without using a --mount.

For this issue, I think we need to ensure that the use cases being employed here are in fact the intended use of dependencies.

If an element requires another to run, it should have a runtime dependency
Dependencies are both build + run by default
More fragmented decoupling is achieved with split-rules, not the dependency model

As far as I can see, it can happen that you have a situation where you want to shell into something without a dependency on the shell, this can be tricky to implement here as it may imply cyclic dependencies (does the shell inadvertently also depend on this element I want to shell into ?)

In GitLab by [Gitlab user @richardmaw-codethink] on Sep 28, 2018, 16:14

From my reading of the discussion, it's actually covered 4 different issues:

The need to stage more dependencies for a useful shell session.

You need to have a shell to have a useful shell session,
but you can build an application that doesn't use the shell at runtime,
therefore having the shell isn't a valid runtime dependency.

This can be generalised to more than just the "platform",
e.g. staging test dependencies if the purpose of running the shell is to do testing,
though testing on its own isn't the best example
as there's been discussion of making that a first-class feature
for the purposes of passing/failing builds rather than punting the decision to running a shell.
Adding tooling that isn't defined in .bst files to the shell

This is a natural extension to wanting to stage more things,
and could be used in some circumstances to solve the same problems as (1),
but there's other cases where (1) would be more convenient.

There doesn't appear to be an issue specifically for this.
Being able to cheaply export to a Docker image.

The argument for this appears to be that it's a problem with a known solution under Docker.
Given Docker doesn't allow you to combine multiple images to create a container
I can't see this ever becoming a zero-copy operation,
so there'll be value in bst shell being able to make the container if it can do so quicker.

https://gitlab.com/BuildStream/buildstream/issues/349 already exists for this.
Being able to use Docker as a container runtime backend.

Depending how it's implemented it might make (3) easier, but otherwise is unrelated.

There doesn't appear to already be an issue for this, but [Gitlab user @cs]-shadow was looking at it earlier.

In GitLab by [Gitlab user @richardmaw-codethink] on Sep 28, 2018, 17:42

This is in bikeshedding territory,
but the presence of base-build-platform and base-runtime-platform
suggests a symmetry with build-dependencies and runtime-dependencies to me
and that it adds dependencies to every element,
despite such an idea only working with a systemd-style
NoDefaultDependencies=yes option.

It may be possible to do base-build-platform behaviour with element overrides
https://buildstream.gitlab.io/-/buildstream/-/jobs/58667641/artifacts/public/projectconf.html#element-overrides
setting replacing the dependencies for the default set,
and using the merge syntax to add extra dependencies on top of the default set.

project.conf

elements:
  ...
    depends:
    - platform.bst

platform.bst

depends: []

foo.bst

depends:
  (>):
  - barlib.bst

We might be able to solve both this, and have a default target for shell
by merging the elements from the command-line with those in project.conf,
but possibly not both at the same time.

The configuration in project.conf may look something like this:

shell:
  elements:
    (<):
    - platform.bst
    (>):
    - debugger.bst

I suspect the best way to work out a good interface will be to try it
and then see how people feel about it.
I expect to get to this after I've looked at
separating sandboxing errors from build errors.

In GitLab by [Gitlab user @richardmaw-codethink] on Oct 5, 2018, 10:41

assigned to [Gitlab user @richardmaw-codethink]

In GitLab by [Gitlab user @richardmaw-codethink] on Oct 29, 2018, 14:18

mentioned in merge request !909

In GitLab by [Gitlab user @LaurenceUrhegyi] on Jan 31, 2019, 11:22

Discussion from London Gathering, January 2019:

Not very controversial for multiple runtime shells.
For build shells, we can’t really have multiple build environments, let’s be clear that the use-case is to have exactly one element that sets the build environment but the ability to add more elements into the shell as runtime.
Angelos highlighted a central use-case where people may been working on a library AND multiple apps at the same time. Coming in and out of shells for that might be a pain?
Valentin raised the point that if you modify a new library you’ll have to rebuild it anyway, in that light, using another shell isn’t really an extra inconvenience
For many users' use-case, any testing of anything that has been built, needs to happen inside the shell.
One problem, how do we know what’s already been staged? Need to expand the API to figure our shared dependencies? Complicated addition required for this use-case…?
We didn’t feel too comfortable to commit to that (yet).
Chandan re-raised the issue that shell is too slow to invoke.
Re-staging things seems to be the biggest issue.
Click won’t let us: bst shell --build one.bst two.bst three.bst now-give-me-this-runtime.bst and-this-runtime.bst
Perhaps elements we want for runtime are the default arguments, which can be unlimited, and the build element (or elements…?) must be specified with --build this.bst --build that.bst
Can split this into two steps (where the first is much more attainable):
Be able to stage multiple runtime elements and JUST ONE build element
Editing multiple elements in the shell at the same time (i.e. allowing for multiple build elements)
There will be some complications with the --deps option, we’d have to allow this to support limited options.

In GitLab by [Gitlab user @LaurenceUrhegyi] on Feb 14, 2019, 08:23

changed the description

In GitLab by [Gitlab user @LaurenceUrhegyi] on Feb 14, 2019, 08:24

changed the description

In GitLab by [Gitlab user @jennis] on Feb 15, 2019, 15:04

Link to the most recent summary email: https://mail.gnome.org/archives/buildstream-list/2019-February/msg00001.html

Hi everyone, is there any update on this? I think this could be very helpful in many situations, for instance I might want to tweak a kernel configuration in freedesktop-sdk inside a shell with the build tree, but ncurses or other dev-dependencies are missing in the shell. IIUC in this issue the discussion is about proposing both a way to specify project-wise shell dependencies and a way to specify additional shell-dependencies in .bst elements. I think both would be very useful.

Hi everyone, is there any update on this? I think this could be very helpful in many situations, for instance I might want to tweak a kernel configuration in freedesktop-sdk inside a shell with the build tree, but ncurses or other dev-dependencies are missing in the shell. IIUC in this issue the discussion is about proposing both a way to specify project-wise shell dependencies and a way to specify additional shell-dependencies in .bst elements. I think both would be very useful.

I think the initial feature is to allow the functionality first via the CLI, for both build shells and runtime shells.

The main initial push for this feature is two fold:

to allow things like debugging to occur with existing built artifacts, without needing to define specific elements and build again in order to bring your debugging tools to life the shell
general IDE enablement, we had some DX folks at the time thinking hard about what will be necessary to create a great DX with buildstream

I'd say that any implementation around this will be welcome so long as it is a general implementation that does not stand in the way of these goals.

Project configurations for specifying shell dependencies on a project wide or element wide basis, sounds like sugar on the cake, something simple to add (most complicated part is settling on a sensible API) after having the initial CLI functionality settled.

I don't expect you would get any pushback on patches towards this :)

I took a look at this at some point. I'll post my notes here in case someone wants to pick it up.

The main blocker is that click (the library used to handle the command line) doesn't support more than one set of variable-size positional arguments (I forgot the exact term).

bst shell supports passing a command to it (which can be more than one argument), trying to make the element support more than one argument can't be handled by click. We can try to do this manually (as in have a single set of position arguments and then have bst split it), but that needs careful consideration.

Currently we support:

bst shell element.bst
bst shell (in a workspace)
bst shell element.bst command to run
bst shell element.bst -- command to run

We need to keep support for these and add support for specifying more than one element.

One thing we could consider is not having it as a positional argument, but as a flag.

bst shell --with strace.bst element.bst strace command to run

We could also make the -- mandatory if there are more than one element (which probably makes more sense), or try to split based on whether the name ends with .bst.

Unless we go for the "flag" solution, we need to implement it manually and fight with click to get the help text right.

Investigation into previous work on the implementation for this:

Old 2018 Gitlab MR by Richard Maw for Allow staging multiple elements in `bst shell`
https://gitlab.com/BuildStream/buildstream/-/merge_requests/909
https://github.com/apache/buildstream/tree/richardmaw/shell-multi-stage is the branch in the github repo, the MR was not migrated from gitlab.
Looks like implementation was finished and under review, but was closed in 2019 due to inactivity.
Due to age of patches and being written for Buildstream1, they likely won't work with Buildstream2. The branch is 4864 commits behind current main, 12 ahead.

@nathanwilliams-ct

@nathanwilliams-ct I suggest starting over and implementing #422 (comment). I hope the change related to the integration commands wouldn't be needed now, as that's a pretty invasive change (also staging works differently now, so maybe it's not needed at all)

Investigation into implementation:

Shell looks like it already uses a function that could stage artifacts from multiple elements, so minimal work needed there
There’s some input validation logic that will needs re-working to handle multiple elements
CLI changes should be relatively straightforward going down the flag approach to maintain existing UX, rather than introducing multi arguments that would change existing UX

Navigation Menu

Repository navigation

`bst shell` should be able to stage the specified element on top of a base element #422

Background

Acceptance Criteria

Update

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

bst shell should be able to stage the specified element on top of a base element #422

Description

Background

Acceptance Criteria

Update

Activity

BuildStream-Migration-Bot commented on Feb 3, 2021

BuildStream-Migration-Bot commented on Feb 3, 2021

BuildStream-Migration-Bot commented on Feb 3, 2021

BuildStream-Migration-Bot commented on Feb 3, 2021

BuildStream-Migration-Bot commented on Feb 3, 2021

BuildStream-Migration-Bot commented on Feb 3, 2021

BuildStream-Migration-Bot commented on Feb 3, 2021

BuildStream-Migration-Bot commented on Feb 3, 2021

BuildStream-Migration-Bot commented on Feb 3, 2021

BuildStream-Migration-Bot commented on Feb 3, 2021

BuildStream-Migration-Bot commented on Feb 3, 2021

BuildStream-Migration-Bot commented on Feb 3, 2021

BuildStream-Migration-Bot commented on Feb 3, 2021

BuildStream-Migration-Bot commented on Feb 3, 2021

mattmart3 commented on Mar 4, 2025

gtristan commented on Mar 5, 2025

abderrahim commented on May 20, 2026

nathanwilliams-ct commented on Jun 26, 2026

abderrahim commented on Jun 26, 2026

nathanwilliams-ct commented on Jul 2, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`bst shell` should be able to stage the specified element on top of a base element #422