Unable to mount sysfs (EPERM) during sys-container initialization

[rodnymolina]

Ran into this issue while trying to launch K8s PODs as system containers.

Within one of the privileged containers acting as K8s nodes, we attempt to launch a POD making use of sysbox as the runtime.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nginx-sysbox-1
spec:
  runtimeClassName: sysbox-runc
  containers:
  - name: nginx
    image: nginx
EOF

• Sysbox-runc returns an EPERM error while trying to build the sandbox (pause) container:

rmolina@heavy-vm-bionic:~/wsp/04-26-2020/kind$ kubectl describe pod nginx-sysbox-1
...
Events:
  Type     Reason                  Age   From                  Message
  ----     ------                  ----  ----                  -------
  Normal   Scheduled               13s   default-scheduler     Successfully assigned default/nginx-sysbox-1 to kind-worker
  Warning  FailedCreatePodSandBox  10s   kubelet, kind-worker  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: OCI runtime create failed: container_linux.go:364: starting container process caused "process_linux.go:532: container init caused \"rootfs_linux.go:58: setting up rootfs mounts caused \\\"rootfs_linux.go:884: mounting \\\\\\\"sysfs\\\\\\\" to rootfs \\\\\\\"/run/containerd/io.containerd.runtime.v1.linux/k8s.io/834a03c15468dedaf708cb525f0fe3016d61aabf7e40e29dd2cf57475ad18ee0/rootfs\\\\\\\" at \\\\\\\"sys\\\\\\\" caused \\\\\\\"operation not permitted\\\\\\\"\\\"\"": unknown

• We reproduce the same exact issue when replacing kubectl cli interface with crictl tool for POD creation:

root@kind-worker:~/kindestnode-rodny# crictl run --runtime=sysbox-runc container-config.json pod-config.json
FATA[3001] Running container failed: run pod sandbox failed: rpc error: code = Unknown desc = failed to create containerd task: OCI runtime create failed: container_linux.go:364: starting container process caused "process_linux.go:532: container init caused \"rootfs_linux.go:58: setting up rootfs mounts caused \\\"rootfs_linux.go:884: mounting \\\\\\\"sysfs\\\\\\\" to rootfs \\\\\\\"/run/containerd/io.containerd.runtime.v1.linux/k8s.io/617d677a65ffb36b8872629d3894b23ec5f47501dc4e54fb0bb8337006d5d508/rootfs\\\\\\\" at \\\\\\\"sys\\\\\\\" caused \\\\\\\"operation not permitted\\\\\\\"\\\"\"": unknown

• Let's place a breakpoint right before the sysfs mount instruction to have a chance at peaking the state of the incipient container. [ all the instructions below are executed within the 'sysbox-runc init' context ]

rmolina@heavy-vm-bionic:~/wsp/04-26-2020/sysbox/kindestnode-rodny$ ps -ef | egrep "sysbox-runc init"
231072    6149  6066  0 23:41 ?        00:00:00 /usr/local/sbin/sysbox-runc init

rmolina@heavy-vm-bionic:~/wsp/04-26-2020/sysbox/kindestnode-rodny$ sudo nsenter -t 6149 -a bash
root@kind-worker:/# findmnt
TARGET                                                                                 SOURCE                                             FSTYPE    OPTIONS
/                                                                                      overlay                                            overlay   rw,relatime,lowerdir=/var/lib/docker/overlay2/l/KMVXHZYLEJZPKOU2TTPZUFSHBF:/var/lib/docker/ove
|-/proc                                                                                proc                                               proc      rw,nosuid,nodev,noexec,relatime
| `-/proc/sys/fs/binfmt_misc                                                           systemd-1                                          autofs    rw,relatime,fd=39,pgrp=0,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=94183586
|-/dev                                                                                 tmpfs                                              tmpfs     rw,nosuid,size=65536k,mode=755
| |-/dev/pts                                                                           devpts                                             devpts    rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666
| |-/dev/mqueue                                                                        mqueue                                             mqueue    rw,nosuid,nodev,noexec,relatime
| |-/dev/shm                                                                           shm                                                tmpfs     rw,nosuid,nodev,noexec,relatime,size=65536k
| |-/dev/console                                                                       devpts[/0]                                         devpts    rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666
| `-/dev/hugepages                                                                     hugetlbfs                                          hugetlbfs rw,relatime,pagesize=2M
|-/sys                                                                                 sysfs                                              sysfs     rw,nosuid,nodev,noexec,relatime
| |-/sys/fs/cgroup                                                                     tmpfs                                              tmpfs     ro,nosuid,nodev,noexec,mode=755
| | |-/sys/fs/cgroup/systemd                                                           cgroup[/../../..]                                  cgroup    rw,nosuid,nodev,noexec,relatime,xattr,name=systemd
| | |-/sys/fs/cgroup/devices                                                           cgroup[/../../..]                                  cgroup    rw,nosuid,nodev,noexec,relatime,devices
| | |-/sys/fs/cgroup/cpuset                                                            cgroup[/../../..]                                  cgroup    rw,nosuid,nodev,noexec,relatime,cpuset
| | |-/sys/fs/cgroup/cpu,cpuacct                                                       cgroup[/../../..]                                  cgroup    rw,nosuid,nodev,noexec,relatime,cpu,cpuacct
| | |-/sys/fs/cgroup/blkio                                                             cgroup[/../../..]                                  cgroup    rw,nosuid,nodev,noexec,relatime,blkio
| | |-/sys/fs/cgroup/memory                                                            cgroup[/../../..]                                  cgroup    rw,nosuid,nodev,noexec,relatime,memory
| | |-/sys/fs/cgroup/hugetlb                                                           cgroup[/../../..]                                  cgroup    rw,nosuid,nodev,noexec,relatime,hugetlb
| | |-/sys/fs/cgroup/rdma                                                              cgroup                                             cgroup    rw,nosuid,nodev,noexec,relatime,rdma
| | |-/sys/fs/cgroup/net_cls,net_prio                                                  cgroup[/../../..]                                  cgroup    rw,nosuid,nodev,noexec,relatime,net_cls,net_prio
| | |-/sys/fs/cgroup/freezer                                                           cgroup[/../../..]                                  cgroup    rw,nosuid,nodev,noexec,relatime,freezer
| | |-/sys/fs/cgroup/pids                                                              cgroup[/../../..]                                  cgroup    rw,nosuid,nodev,noexec,relatime,pids
| | |-/sys/fs/cgroup/perf_event                                                        cgroup[/../../..]                                  cgroup    rw,nosuid,nodev,noexec,relatime,perf_event
| | `-/sys/fs/cgroup/unified                                                           cgroup2[/../../../..]                              cgroup2   rw,nosuid,nodev,noexec,relatime,nsdelegate
| |-/sys/kernel/debug                                                                  debugfs                                            debugfs   rw,nosuid,nodev,noexec,relatime
| `-/sys/fs/fuse/connections                                                           fusectl                                            fusectl   rw,nosuid,nodev,noexec,relatime
|-/run                                                                                 tmpfs                                              tmpfs     rw,nosuid,nodev,noexec,relatime
| |-/run/lock                                                                          tmpfs                                              tmpfs     rw,nosuid,nodev,noexec,relatime,size=5120k
| |-/run/containerd/io.containerd.grpc.v1.cri/sandboxes/a5ba02ce9661b191df25560ca67694f2b00601b887db936c3d38d882082e837a/shm
| |                                                                                    shm                                                tmpfs     rw,nosuid,nodev,noexec,relatime,size=65536k
| |-/run/containerd/io.containerd.runtime.v2.task/k8s.io/a5ba02ce9661b191df25560ca67694f2b00601b887db936c3d38d882082e837a/rootfs
| |                                                                                    overlay                                            overlay   rw,relatime,lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1/fs
| |-/run/containerd/io.containerd.grpc.v1.cri/sandboxes/a9294afb871b35503d3111c60346f3c69d67757d52032ef6018f3fa782a2f02b/shm
| |                                                                                    shm                                                tmpfs     rw,nosuid,nodev,noexec,relatime,size=65536k
| |-/run/containerd/io.containerd.runtime.v2.task/k8s.io/a9294afb871b35503d3111c60346f3c69d67757d52032ef6018f3fa782a2f02b/rootfs
| |                                                                                    overlay                                            overlay   rw,relatime,lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1/fs
| |-/run/containerd/io.containerd.runtime.v2.task/k8s.io/6f2abce13e25839fca0689db233038506514c1c342586c1a33505665769211c5/rootfs
| |                                                                                    overlay                                            overlay   rw,relatime,lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/16/f
| |-/run/containerd/io.containerd.runtime.v2.task/k8s.io/225162f17f1794063d62865b698bddf1cdac1d5e0569c51e033b9f1ae6e100a1/rootfs
| |                                                                                    overlay                                            overlay   rw,relatime,lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/17/f
| |-/run/containerd/io.containerd.runtime.v1.linux/k8s.io/.a0127560524578936a435db0554660ed362eb19ac1e7dfff1ceb9306bbd5d137/rootfs
| |                                                                                    overlay                                            overlay   rw,relatime,lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1/fs
| | `-/run/containerd/io.containerd.runtime.v1.linux/k8s.io/.a0127560524578936a435db0554660ed362eb19ac1e7dfff1ceb9306bbd5d137/rootfs
| |                                                                                    /run/containerd/io.containerd.runtime.v1.linux/k8s.io/a0127560524578936a435db0554660ed362eb19ac1e7dfff1ceb9306bbd5d137/rootfs
| |                                                                                                                                       shiftfs   rw,relatime,mark
| |-/run/netns/cni-2b9b363a-c622-4b7f-017c-08688285fd7c                                nsfs[net:[4026532972]]                             nsfs      rw
| |-/run/containerd/io.containerd.grpc.v1.cri/sandboxes/617d677a65ffb36b8872629d3894b23ec5f47501dc4e54fb0bb8337006d5d508/shm
| |                                                                                    shm                                                tmpfs     rw,nosuid,nodev,noexec,relatime,size=65536k
| | `-/run/containerd/io.containerd.grpc.v1.cri/sandboxes/617d677a65ffb36b8872629d3894b23ec5f47501dc4e54fb0bb8337006d5d508/shm
| |                                                                                    /run/containerd/io.containerd.grpc.v1.cri/sandboxes/617d677a65ffb36b8872629d3894b23ec5f47501dc4e54fb0bb8337006d5d508/shm
| |                                                                                                                                       shiftfs   rw,relatime,mark
| |   `-/run/containerd/io.containerd.grpc.v1.cri/sandboxes/617d677a65ffb36b8872629d3894b23ec5f47501dc4e54fb0bb8337006d5d508/shm
| |                                                                                    /run/containerd/io.containerd.grpc.v1.cri/sandboxes/617d677a65ffb36b8872629d3894b23ec5f47501dc4e54fb0bb8337006d5d508/shm
| |                                                                                                                                       shiftfs   rw,relatime
| `-/run/containerd/io.containerd.runtime.v1.linux/k8s.io/617d677a65ffb36b8872629d3894b23ec5f47501dc4e54fb0bb8337006d5d508/rootfs
|                                                                                      overlay                                            overlay   rw,relatime,lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1/fs
|   `-/run/containerd/io.containerd.runtime.v1.linux/k8s.io/617d677a65ffb36b8872629d3894b23ec5f47501dc4e54fb0bb8337006d5d508/rootfs
|                                                                                      /run/containerd/io.containerd.runtime.v1.linux/k8s.io/617d677a65ffb36b8872629d3894b23ec5f47501dc4e54fb0bb8337006d5d508/rootfs
|                                                                                                                                         shiftfs   rw,relatime,mark
|     `-/run/containerd/io.containerd.runtime.v1.linux/k8s.io/617d677a65ffb36b8872629d3894b23ec5f47501dc4e54fb0bb8337006d5d508/rootfs
|                                                                                      /run/containerd/io.containerd.runtime.v1.linux/k8s.io/617d677a65ffb36b8872629d3894b23ec5f47501dc4e54fb0bb8337006d5d508/rootfs
|                                                                                                                                         shiftfs   rw,relatime,mark
|       `-/run/containerd/io.containerd.runtime.v1.linux/k8s.io/617d677a65ffb36b8872629d3894b23ec5f47501dc4e54fb0bb8337006d5d508/rootfs
|                                                                                      .                                                  shiftfs   rw,relatime
|-/tmp                                                                                 tmpfs                                              tmpfs     rw,nosuid,nodev,noexec,relatime
|-/root/nestybox/sysbox                                                                /dev/sda3[/home/rmolina/wsp/04-26-2020/sysbox]     ext4      rw,relatime,errors=remount-ro
|-/var                                                                                 /dev/sda3[/var/lib/docker/volumes/8e4ddb7f4ee97e215df9d125de2bb10dd2e6cd9f3303e109e3ba1b552db24f1f/_data]
|                                                                                                                                         ext4      rw,relatime,errors=remount-ro
| |-/var/lib/kubelet                                                                   /dev/sda3[/var/lib/docker/volumes/8e4ddb7f4ee97e215df9d125de2bb10dd2e6cd9f3303e109e3ba1b552db24f1f/_data/lib/kubelet]
| |                                                                                                                                       ext4      rw,relatime,errors=remount-ro
| | |-/var/lib/kubelet/pods/4a12d0af-20af-48bc-8297-3f63a2be05d9/volumes/kubernetes.io~secret/kube-proxy-token-ll9tc
| | |                                                                                  tmpfs                                              tmpfs     rw,relatime
| | `-/var/lib/kubelet/pods/ae9626cb-2388-44fb-8bd6-0aaca66dc88d/volumes/kubernetes.io~secret/kindnet-token-hcnt8
| |                                                                                    tmpfs                                              tmpfs     rw,relatime
| `-/var/lib/sysboxfs/617d677a65ffb36b8872629d3894b23ec5f47501dc4e54fb0bb8337006d5d508 sysboxfs                                           fuse      rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
|-/usr/lib/modules                                                                     /dev/sda3[/lib/modules]                            ext4      ro,relatime,errors=remount-ro
| `-/usr/lib/modules/5.3.0-46-generic                                                  /lib/modules/5.3.0-46-generic                      shiftfs   rw,relatime,mark
|   `-/usr/lib/modules/5.3.0-46-generic                                                /lib/modules/5.3.0-46-generic                      shiftfs   rw,relatime
|-/usr/src/linux-headers-5.3.0-46                                                      /dev/sda3[/usr/src/linux-headers-5.3.0-46]         ext4      ro,relatime,errors=remount-ro
| `-/usr/src/linux-headers-5.3.0-46                                                    /usr/src/linux-headers-5.3.0-46                    shiftfs   rw,relatime,mark
|   `-/usr/src/linux-headers-5.3.0-46                                                  /usr/src/linux-headers-5.3.0-46                    shiftfs   rw,relatime
|-/usr/src/linux-headers-5.3.0-46-generic                                              /dev/sda3[/usr/src/linux-headers-5.3.0-46-generic] ext4      ro,relatime,errors=remount-ro
| `-/usr/src/linux-headers-5.3.0-46-generic                                            /usr/src/linux-headers-5.3.0-46-generic            shiftfs   rw,relatime,mark
|   `-/usr/src/linux-headers-5.3.0-46-generic                                          /usr/src/linux-headers-5.3.0-46-generic            shiftfs   rw,relatime
|-/etc/resolv.conf                                                                     /dev/sda3[/var/lib/docker/containers/693268e234832ba5b97d02763bd8ab37a236610024382f9c9c4cec561ffb9e29/resolv.conf]
|                                                                                                                                         ext4      rw,relatime,errors=remount-ro
|-/etc/hostname                                                                        /dev/sda3[/var/lib/docker/containers/693268e234832ba5b97d02763bd8ab37a236610024382f9c9c4cec561ffb9e29/hostname]
|                                                                                                                                         ext4      rw,relatime,errors=remount-ro
`-/etc/hosts                                                                           /dev/sda3[/var/lib/docker/containers/693268e234832ba5b97d02763bd8ab37a236610024382f9c9c4cec561ffb9e29/hosts]
                                                                                                                                          ext4      rw,relatime,errors=remount-ro
root@kind-worker:/#

• Once again, an EPERM is returned if we manually attempt to mount sysfs in the container's rootfs:

root@kind-worker:/# mount -t sysfs sysfs /run/containerd/io.containerd.runtime.v1.linux/k8s.io/617d677a65ffb36b8872629d3894b23ec5f47501dc4e54fb0bb8337006d5d508/rootfs/sys
mount: /run/containerd/io.containerd.runtime.v1.linux/k8s.io/617d677a65ffb36b8872629d3894b23ec5f47501dc4e54fb0bb8337006d5d508/rootfs/sys: permission denied.
root@kind-worker:/#

• However, no issue is observed when trying to mount procfs:

root@kind-worker:/# mkdir /run/containerd/io.containerd.runtime.v1.linux/k8s.io/617d677a65ffb36b8872629d3894b23ec5f47501dc4e54fb0bb8337006d5d508/rootfs/proc
root@kind-worker:/#
root@kind-worker:/# ls -lrt /run/containerd/io.containerd.runtime.v1.linux/k8s.io/617d677a65ffb36b8872629d3894b23ec5f47501dc4e54fb0bb8337006d5d508/rootfs
total 740
-rwxr-xr-x 1 root root 742472 Dec 20  2017 pause
drwxr-xr-x 2 root root   4096 May  8 23:41 dev
drwxr-xr-x 2 root root   4096 May  8 23:41 sys
drwxr-xr-x 2 root root   4096 May  8 23:44 proc
root@kind-worker:/#
root@kind-worker:/# mount -t proc proc /run/containerd/io.containerd.runtime.v1.linux/k8s.io/617d677a65ffb36b8872629d3894b23ec5f47501dc4e54fb0bb8337006d5d508/rootfs/proc
root@kind-worker:/# findmnt
TARGET                                                                                 SOURCE                                             FSTYPE    OPTIONS
/                                                                                      overlay                                            overlay   rw,relatime,lowerdir=/var/lib/docker/overlay2/l/KMVXHZYLEJZPKOU2TTPZUFSHBF:/var/lib/docker/ove
|-/proc                                                                                proc                                               proc      rw,nosuid,nodev,noexec,relatime
...
|       `-/run/containerd/io.containerd.runtime.v1.linux/k8s.io/617d677a65ffb36b8872629d3894b23ec5f47501dc4e54fb0bb8337006d5d508/rootfs
|                                                                                      .                                                  shiftfs   rw,relatime
|         `-/run/containerd/io.containerd.runtime.v1.linux/k8s.io/617d677a65ffb36b8872629d3894b23ec5f47501dc4e54fb0bb8337006d5d508/rootfs/proc
|                                                                                      proc                                               proc      rw,relatime
...

• Notice that there's nothing wrong with the 'rootfs/sys' folder created by sysbox-runc; the same issues is observed with a manually created 'sys' folder:

root@kind-worker:/# mkdir /run/containerd/io.containerd.runtime.v1.linux/k8s.io/617d677a65ffb36b8872629d3894b23ec5f47501dc4e54fb0bb8337006d5d508/rootfs/sysnew
root@kind-worker:/# mount -t sysfs sysfs /run/containerd/io.containerd.runtime.v1.linux/k8s.io/617d677a65ffb36b8872629d3894b23ec5f47501dc4e54fb0bb8337006d5d508/rootfs/sysnew
mount: /run/containerd/io.containerd.runtime.v1.linux/k8s.io/617d677a65ffb36b8872629d3894b23ec5f47501dc4e54fb0bb8337006d5d508/rootfs/sysnew: permission denied.
root@kind-worker:/#

• Obviously, the problem is not with the rootfs path but with sys file-system itself. For some reason kernel is preventing sysfs from being mounted.

[rodnymolina]

Issue closed by mistake. Re-opening.

[rodnymolina]

Technically speaking, we don't really need these bind-mounts (see examples two comments below) created by Kind folks to operate a Kind cluster. We could explore alternative solutions to workaround this problem ... but there's more ... see issue #669.

[rodnymolina]

As it could be expected, the particular setup described above (L1 priv container) has no bearing in the behavior being observed. IOW, the same exact behavior is observed in the host FS (L0):

rmolina@heavy-vm-bionic:~/wsp$ touch /tmp/testing.txt
rmolina@heavy-vm-bionic:~/wsp$ sudo mount -o bind /tmp/testing.txt /sys/devices/system/node/online
rmolina@heavy-vm-bionic:~/wsp$

rmolina@heavy-vm-bionic:~/wsp$ sudo unshare -m -u -i -n -p -U -f -r bash -c "mkdir /root/sys && mount -t sysfs sysfs /root/sys"
mount: /root/sys: permission denied.

rmolina@heavy-vm-bionic:~/wsp$ sudo umount /sys/devices/system/node/online

rmolina@heavy-vm-bionic:~/wsp$ sudo rm -rf /root/sys

rmolina@heavy-vm-bionic:~/wsp$ sudo unshare -m -u -i -n -p -U -f -r bash -c "mkdir /root/sys && mount -t sysfs sysfs /root/sys"
rmolina@heavy-vm-bionic:~/wsp$

[rodnymolina]

I was able to reproduce problem by invoking sysbox-runc directly. There seems to be some kernel-level restrictions being imposed at the time sysfs is mounted.

In this particular case, the mere act of bind-mounting any resource into the privilege-container' sysfs (running at L1 level), is somehow impacting our capability of mounting sysfs at L2 level.

The "bind-mounts" that i'm referring to are defined by Kind's entrypoint script as shown below:

fix_product_name() {
  # this is a small fix to hide the underlying hardware and fix issue #426
  # https://github.com/kubernetes-sigs/kind/issues/426
  if [[ -f /sys/class/dmi/id/product_name ]]; then
    echo 'INFO: faking /sys/class/dmi/id/product_name to be "kind"'
    echo 'kind' > /kind/product_name
    mount -o ro,bind /kind/product_name /sys/class/dmi/id/product_name
  fi
}

fix_product_uuid() {
  # The system UUID is usually read from DMI via sysfs, the problem is that
  # in the kind case this means that all (container) nodes share the same
  # system/product uuid, as they share the same DMI.
  # Note: The UUID is read from DMI, this tool is overwriting the sysfs files
  # which should fix the attached issue, but this workaround does not address
  # the issue if a tool is reading directly from DMI.
  # https://github.com/kubernetes-sigs/kind/issues/1027
  [[ ! -f /kind/product_uuid ]] && cat /proc/sys/kernel/random/uuid > /kind/product_uuid
  if [[ -f /sys/class/dmi/id/product_uuid ]]; then
    echo 'INFO: faking /sys/class/dmi/id/product_uuid to be random'
    mount -o ro,bind /kind/product_uuid /sys/class/dmi/id/product_uuid
  fi
  if [[ -f /sys/devices/virtual/dmi/id/product_uuid ]]; then
    echo 'INFO: faking /sys/devices/virtual/dmi/id/product_uuid as well'
    mount -o ro,bind /kind/product_uuid /sys/devices/virtual/dmi/id/product_uuid
  fi
}

Without these two functions (above) sysbox-runc works as expected. In fact, problem can be easily reproduced in any privileged container; for conveniency let's use our own sysbox's test-shell-installer container:

• Working case:

rmolina@dev-vm1:~/wsp/05-07-2020/sysbox$ make test-shell-installer
...

root@sysbox-installer-test:~/ubuntu# findmnt
...
|-/sys                                sysfs                                              sysfs     rw,nosuid,nodev,noexec,relatime
| |-/sys/kernel/debug                 debugfs                                            debugfs   rw,relatime
| |-/sys/kernel/config                configfs                                           configfs  rw,relatime
| |-/sys/fs/fuse/connections          fusectl                                            fusectl   rw,relatime
| `-/sys/fs/cgroup                    tmpfs                                              tmpfs     ro,nosuid,nodev,noexec,mode=755
|   |-/sys/fs/cgroup/unified          cgroup                                             cgroup2   rw,nosuid,nodev,noexec,relatime,nsdelegate
|   |-/sys/fs/cgroup/systemd          cgroup[/docker/fd1df2905da572db64c60b09418703ddce115f54364799d52925e232ed76fd22]
|   |                                                                                    cgroup    rw,nosuid,nodev,noexec,relatime,xattr,name=systemd
|   |-/sys/fs/cgroup/devices          cgroup[/docker/fd1df2905da572db64c60b09418703ddce115f54364799d52925e232ed76fd22]
|   |                                                                                    cgroup    rw,nosuid,nodev,noexec,relatime,devices
|   |-/sys/fs/cgroup/cpuset           cgroup[/docker/fd1df2905da572db64c60b09418703ddce115f54364799d52925e232ed76fd22]
|   |                                                                                    cgroup    rw,nosuid,nodev,noexec,relatime,cpuset
|   |-/sys/fs/cgroup/cpu,cpuacct      cgroup[/docker/fd1df2905da572db64c60b09418703ddce115f54364799d52925e232ed76fd22]
|   |                                                                                    cgroup    rw,nosuid,nodev,noexec,relatime,cpu,cpuacct
|   |-/sys/fs/cgroup/blkio            cgroup[/docker/fd1df2905da572db64c60b09418703ddce115f54364799d52925e232ed76fd22]
|   |                                                                                    cgroup    rw,nosuid,nodev,noexec,relatime,blkio
|   |-/sys/fs/cgroup/memory           cgroup[/docker/fd1df2905da572db64c60b09418703ddce115f54364799d52925e232ed76fd22]
|   |                                                                                    cgroup    rw,nosuid,nodev,noexec,relatime,memory
|   |-/sys/fs/cgroup/hugetlb          cgroup[/docker/fd1df2905da572db64c60b09418703ddce115f54364799d52925e232ed76fd22]
|   |                                                                                    cgroup    rw,nosuid,nodev,noexec,relatime,hugetlb
|   |-/sys/fs/cgroup/rdma             cgroup                                             cgroup    rw,nosuid,nodev,noexec,relatime,rdma
|   |-/sys/fs/cgroup/net_cls,net_prio cgroup[/docker/fd1df2905da572db64c60b09418703ddce115f54364799d52925e232ed76fd22]
|   |                                                                                    cgroup    rw,nosuid,nodev,noexec,relatime,net_cls,net_prio
|   |-/sys/fs/cgroup/freezer          cgroup[/docker/fd1df2905da572db64c60b09418703ddce115f54364799d52925e232ed76fd22]
|   |                                                                                    cgroup    rw,nosuid,nodev,noexec,relatime,freezer
|   |-/sys/fs/cgroup/pids             cgroup[/docker/fd1df2905da572db64c60b09418703ddce115f54364799d52925e232ed76fd22]
|   |                                                                                    cgroup    rw,nosuid,nodev,noexec,relatime,pids
|   `-/sys/fs/cgroup/perf_event       cgroup[/docker/fd1df2905da572db64c60b09418703ddce115f54364799d52925e232ed76fd22]
|                                                                                        cgroup    rw,nosuid,nodev,noexec,relatime,perf_event
...


root@sysbox-installer-test:~/ mkdir /root/ubuntu
root@sysbox-installer-test:~/ cd /root/ubuntu
root@sysbox-installer-test:~/ubuntu# mkdir rootfs
root@sysbox-installer-test:~/ubuntu# docker export $(docker create ubuntu:latest) | tar -C rootfs -xvf -
root@sysbox-installer-test:~/ubuntu# sysbox-runc spec

root@sysbox-installer-test:~/ubuntu# sysbox-runc run ubuntu-1
#

• Non-working case:

root@sysbox-installer-test:~/nestybox/sysbox/ubuntu# mount -o ro,bind /testing.txt /sys/class/dmi/id/product_name

root@sysbox-installer-test:~/nestybox/sysbox/ubuntu# findmnt
...
root@sysbox-installer-test:~/nestybox/sysbox/ubuntu# findmnt
TARGET                                SOURCE                                                                           FSTYPE    OPTIONS
/                                     overlay                                                                          overlay   rw,relatime,lowerdir=/var/lib/docker/overlay2/l/HZPTQBEENTJFHNBV75OFS2JP3N:/var/lib/docker/overlay2/l/VJSI6XDBSXN
|-/proc                               proc                                                                             proc      rw,nosuid,nodev,noexec,relatime
| `-/proc/sys/fs/binfmt_misc          systemd-1                                                                        autofs    rw,relatime,fd=25,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=123960382
|-/dev                                tmpfs                                                                            tmpfs     rw,nosuid,size=65536k,mode=755
| |-/dev/hugepages                    hugetlbfs                                                                        hugetlbfs rw,relatime,pagesize=2M
| |-/dev/pts                          devpts                                                                           devpts    rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666
| |-/dev/mqueue                       mqueue                                                                           mqueue    rw,nosuid,nodev,noexec,relatime
| `-/dev/shm                          shm                                                                              tmpfs     rw,nosuid,nodev,noexec,relatime,size=65536k
|-/sys                                sysfs                                                                            sysfs     rw,nosuid,nodev,noexec,relatime
| |-/sys/kernel/debug                 debugfs                                                                          debugfs   rw,relatime
| |-/sys/kernel/config                configfs                                                                         configfs  rw,relatime
| |-/sys/fs/fuse/connections          fusectl                                                                          fusectl   rw,relatime
| |-/sys/fs/cgroup                    tmpfs                                                                            tmpfs     ro,nosuid,nodev,noexec,mode=755
| | |-/sys/fs/cgroup/unified          cgroup                                                                           cgroup2   rw,nosuid,nodev,noexec,relatime,nsdelegate
| | |-/sys/fs/cgroup/systemd          cgroup[/docker/ecdba2b418af97affcbb2a536b3ad493b04fa345b0e641d985b4b7039423c886] cgroup    rw,nosuid,nodev,noexec,relatime,xattr,name=systemd
| | |-/sys/fs/cgroup/devices          cgroup[/docker/ecdba2b418af97affcbb2a536b3ad493b04fa345b0e641d985b4b7039423c886] cgroup    rw,nosuid,nodev,noexec,relatime,devices
| | |-/sys/fs/cgroup/cpuset           cgroup[/docker/ecdba2b418af97affcbb2a536b3ad493b04fa345b0e641d985b4b7039423c886] cgroup    rw,nosuid,nodev,noexec,relatime,cpuset
| | |-/sys/fs/cgroup/cpu,cpuacct      cgroup[/docker/ecdba2b418af97affcbb2a536b3ad493b04fa345b0e641d985b4b7039423c886] cgroup    rw,nosuid,nodev,noexec,relatime,cpu,cpuacct
| | |-/sys/fs/cgroup/blkio            cgroup[/docker/ecdba2b418af97affcbb2a536b3ad493b04fa345b0e641d985b4b7039423c886] cgroup    rw,nosuid,nodev,noexec,relatime,blkio
| | |-/sys/fs/cgroup/memory           cgroup[/docker/ecdba2b418af97affcbb2a536b3ad493b04fa345b0e641d985b4b7039423c886] cgroup    rw,nosuid,nodev,noexec,relatime,memory
| | |-/sys/fs/cgroup/hugetlb          cgroup[/docker/ecdba2b418af97affcbb2a536b3ad493b04fa345b0e641d985b4b7039423c886] cgroup    rw,nosuid,nodev,noexec,relatime,hugetlb
| | |-/sys/fs/cgroup/rdma             cgroup                                                                           cgroup    rw,nosuid,nodev,noexec,relatime,rdma
| | |-/sys/fs/cgroup/net_cls,net_prio cgroup[/docker/ecdba2b418af97affcbb2a536b3ad493b04fa345b0e641d985b4b7039423c886] cgroup    rw,nosuid,nodev,noexec,relatime,net_cls,net_prio
| | |-/sys/fs/cgroup/freezer          cgroup[/docker/ecdba2b418af97affcbb2a536b3ad493b04fa345b0e641d985b4b7039423c886] cgroup    rw,nosuid,nodev,noexec,relatime,freezer
| | |-/sys/fs/cgroup/pids             cgroup[/docker/ecdba2b418af97affcbb2a536b3ad493b04fa345b0e641d985b4b7039423c886] cgroup    rw,nosuid,nodev,noexec,relatime,pids
| | `-/sys/fs/cgroup/perf_event       cgroup[/docker/ecdba2b418af97affcbb2a536b3ad493b04fa345b0e641d985b4b7039423c886] cgroup    rw,nosuid,nodev,noexec,relatime,perf_event
| `-/sys/devices/virtual/dmi/id/product_name
|                                     overlay[/testing.txt]                                                            overlay   ro,relatime,lowerdir=/var/lib/docker/overlay2/l/HZPTQBEENTJFHNBV75OFS2JP3N:/var/lib/docker/overlay2/l/VJSI6XDBSXN
|-/tmp                                tmpfs                                                                            tmpfs     rw,nosuid,nodev,noexec,relatime

root@sysbox-installer-test:~/nestybox/sysbox/ubuntu# sysbox-runc run ubuntu-1
container_linux.go:364: starting container process caused "process_linux.go:533: container init caused \"rootfs_linux.go:58: setting up rootfs mounts caused \\\"rootfs_linux.go:928: mounting \\\\\\\"sysfs\\\\\\\" to rootfs \\\\\\\"/root/nestybox/sysbox/ubuntu/rootfs\\\\\\\" at \\\\\\\"sys\\\\\\\" caused \\\\\\\"operation not permitted\\\\\\\"\\\"\""
root@sysbox-installer-test:~/nestybox/sysbox/ubuntu#
...

There's nothing special with the elected mountpoint ("product_name"), nor with the fstype being mounted (overlayfs). The moment any resource is bind-mounted over the existing sysfs hierarchy (L1 privileged container), subsequent sysfs mounts stop working. See another example below of a tmpfs resource being bind-mounted over a different sysfs node:

root@sysbox-installer-test:~/nestybox/sysbox/ubuntu# mount -o bind /tmp/testing.txt /sys/devices/system/node/online

root@sysbox-installer-test:~/nestybox/sysbox/ubuntu# findmnt | egrep online
| `-/sys/devices/system/node/online       tmpfs[/testing.txt]                                                                                                tmpfs     rw,nosuid,nodev,noexec,relatime

root@sysbox-installer-test:~/nestybox/sysbox/ubuntu# sysbox-runc run ubuntu-1
container_linux.go:364: starting container process caused "process_linux.go:533: container init caused \"rootfs_linux.go:58: setting up rootfs mounts caused \\\"rootfs_linux.go:928: mounting \\\\\\\"sysfs\\\\\\\" to rootfs \\\\\\\"/root/nestybox/sysbox/ubuntu/rootfs\\\\\\\" at \\\\\\\"sys\\\\\\\" caused \\\\\\\"operation not permitted\\\\\\\"\\\"\""

root@sysbox-installer-test:~/nestybox/sysbox/ubuntu# umount /sys/devices/system/node/online

root@sysbox-installer-test:~/nestybox/sysbox/ubuntu# sysbox-runc run ubuntu-1
#

Finally, notice that this behavior has nothing to do with sysbox-runc. The same is observed with a regular unshare instruction:

root@sysbox-installer-test:~/nestybox/sysbox/ubuntu# unshare -m -u -i -n -p -U -f -r bash -c "mkdir /root/sys && mount -t sysfs sysfs /root/sys"
mount: /root/sys: permission denied.

root@sysbox-installer-test:~/nestybox/sysbox/ubuntu# umount /sys/devices/system/node/online

root@sysbox-installer-test:~/nestybox/sysbox/ubuntu# rm -rf /root/sys
root@sysbox-installer-test:~/nestybox/sysbox/ubuntu# unshare -m -u -i -n -p -U -f -r bash -c "mkdir /root/sys && mount -t sysfs sysfs /root/sys"
root@sysbox-installer-test:~/nestybox/sysbox/ubuntu#

[rodnymolina]

Ref #64.

[ctalledo]

This issue is very likely a duplicate of issue #67. The reason that sysfs mounts are failing when k8s deploys pods with Sysbox is that sysbox is creating the user-ns after the net-ns has been created. This somehow impacts mounting of sysfs inside said user-ns.

For example:

$ sudo unshare -n bash                                                                                                                                                                                                                                                                                                        
$ unshare -U -m -i -p -u -C -f -r --mount-proc bash                                                                                                                                                                                                                                                                           
$ mount -t sysfs sysfs sys                                                                                                                                                                                                                                                                                                    
mount: /home/cesar//rootfs/sys: permission denied                                                                                                                                                                                                                                                                  
$ unshare -n bash                                                                                                                                                                                                                                                                                                             
$ mount -t sysfs sysfs sys                                                                                                                                                                                                                                                                                                    
(no problem)                                                                                                                                                                                                                                                                                                                  

I don't believe the problem has anything to do with host sysfs submounts as described by this PR. Thus, I am closing this PR and we will track this via issue #67.

If we find later that host sysfs submounts are causing problems when mounting sysfs to a container's root filesystem, we will re-open.

[rodnymolina]

For the record, I ran into this one again (three years later). I can confirm that the problem has nothing to do with Sysbox (it's reproduced with a regular privileged container), and as explained above, it's indeed related to the specific sysfs bind-mounts that kind is creating.

Problem can be easily reproduced by following these steps:

ubuntu@ip-172-31-0-219:~$ docker run -it --rm --privileged ubuntu:jammy bash
root@d01d7ef07662:/# touch test-file

root@d01d7ef07662:/# mount -o rw,bind test-file /sys/class/dmi/id/product_name

root@d01d7ef07662:/# unshare -m -n -u -i -p -U -f -r bash -c "mkdir -p /root/sys && mount -t sysfs sysfs /root/sys"
mount: /root/sys: permission denied.
root@d01d7ef07662:/#

[ctalledo]

Hi @rodnymolina,

root@d01d7ef07662:/# mount -o rw,bind test-file /sys/class/dmi/id/product_name

root@d01d7ef07662:/# unshare -m -n -u -i -p -U -f -r bash -c "mkdir -p /root/sys && mount -t sysfs sysfs /root/sys"
mount: /root/sys: permission denied.

Looks like the bind-mount of test-file onto /sys/class/dmi/id/product_name is causing the subsequent mount of sysfs inside the unshared namespaces to fail. I assume that without that bind-mount things work (correct me if I am wrong).

I wonder then if the bind mount's "propagation type" (default shared IIRC) is causing the problem? Try changing the propagation type of the bind mount by inserting this command in between:

$ mount --make-rprivate /sys/class/dmi/id/product_name

Not sure if that would help, but it's worth a try.