fix(sandbox): skip read-only mounts during recursive chown of /sandbox by varshaprasad96 · Pull Request #2341 · NVIDIA/OpenShell

varshaprasad96 · 2026-07-17T05:55:50Z

Summary

Fix sandbox supervisor crash (EROFS: Read-only file system) when the recursive chown /sandbox encounters read-only submounts. This is common in gVisor-based Kubernetes deployments where read-only volume mounts enforce per-directory immutability (Landlock is unavailable under gVisor).

Related Issue

Closes #2294

Changes

Mount-boundary detection: Compare st_dev of each path against the root /sandbox device — paths on a different filesystem are skipped entirely, avoiding the chown call on nested read-only mounts
EROFS tolerance: If chown still returns EROFS (e.g. the root mount itself is read-only), the error is logged at debug level and startup continues
Refactored chown_sandbox_home: Split into entry-point (symlink guard + root_dev capture) and chown_recursive (inner walk with st_dev and EROFS guards)
Added unit tests: chown_recursive_skips_different_device and chown_sandbox_home_does_not_chown_across_device_boundary; strengthened existing symlink rejection test

Testing

mise run pre-commit passes (Rust lint/format/clippy clean; unrelated Python proto env issue)
Unit tests added/updated — 6 chown-related tests pass
Manually verified on a kind cluster with a read-only PVC at /sandbox/readonly-data: pod starts successfully, writable paths owned by sandbox user, read-only mount retains root ownership
E2E tests added/updated — Kubernetes e2e coverage is a separate follow-up

Checklist

Follows Conventional Commits
Commits are signed off (DCO)

copy-pr-bot · 2026-07-17T05:55:53Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

The sandbox supervisor crashed with EROFS when the recursive chown of /sandbox encountered read-only submounts. This is common in gVisor-based Kubernetes deployments where read-only volume mounts are the only way to enforce per-directory immutability (Landlock is unavailable under gVisor). The fix adds two guards to the ownership walk: 1. Mount-boundary detection via st_dev comparison — paths on a different filesystem than /sandbox are skipped entirely, avoiding the chown call on nested read-only mounts. 2. EROFS tolerance — if chown still returns EROFS (e.g. the root mount itself is read-only), the error is logged at debug level and startup continues. Symlink skipping (already present) is preserved and extracted into the recursive walker for consistency. Manually verified on a kind cluster with a read-only PVC mounted at /sandbox/readonly-data: the pod starts successfully, writable paths are owned by the sandbox user, and the read-only mount retains root ownership. Kubernetes e2e test coverage is a separate follow-up. Closes NVIDIA#2294 Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

elezar · 2026-07-17T10:00:35Z

+    let meta = std::fs::symlink_metadata(path).into_diagnostic()?;
+


Can we move the symlink check that is currently performed on child in the loop below to here?

Done in 4cd9059. Moved the symlink check into the top of chown_recursive (reusing the symlink_metadata call already there), which also eliminated the redundant per-child symlink_metadata in the loop

Move the per-child symlink guard from the directory iteration loop into the top of chown_recursive, reusing the symlink_metadata call that is already performed there. This centralizes all three skip guards (symlink, mount boundary, EROFS) in one place and eliminates a redundant symlink_metadata call per child entry. Suggested-by: elezar Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

When chown returns EROFS on a directory, the code previously returned early without recursing into children. This skipped writable submounts nested under a read-only directory on the same device (e.g., a writable emptyDir inside a read-only ConfigMap mount sharing the same st_dev). Now the EROFS case falls through to the directory recursion so that writable children are still chowned. Also fixes the doc comment which said EROFS errors are "logged as warnings" when they are actually logged at debug level. Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

bjw123 · 2026-07-20T06:39:11Z

Thank you @varshaprasad96 i did some validation testing and this MR does indeed fix the bug. Looking forward to it getting merged :)

elezar · 2026-07-20T11:22:08Z

/ok-to-test 3d079d5

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

johntmyers

gator-agent

PR Review Status

Validation: This is a concentrated fix for the confirmed Kubernetes sandbox startup bug in #2294.
Head SHA: 888f91a47dfd8f14f86986356a385b24cd805184

Review findings:

Two actionable findings are attached inline: a writable cross-filesystem mount regression and missing coverage for the actual EROFS recovery path.

Thanks @elezar. I checked your request to centralize the symlink check; the current head resolves it in chown_recursive and preserves the no-follow behavior.

Docs: Fern docs are not required because the intended fix does not change a user-facing command, configuration, or workflow.

Next state: gator:in-review; author changes are required before pipeline monitoring.

Remove the unconditional st_dev boundary check that prevented chown on writable PVC mounts with a different device ID. Rely solely on EROFS errors to skip read-only paths while continuing traversal into children and siblings. Inject the chown operation into chown_recursive to enable unit testing the EROFS recovery path without requiring root or mount privileges. Signed-off-by: Varsha Prasad Narsing <vnarsing@nvidia.com> Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

github-actions · 2026-07-21T18:43:41Z

Label test:e2e applied, but pull-request/2341 is at 3d079d5 while the PR head is 0869591. A maintainer needs to comment /ok to test 08695910c048edf59643e547bef38b1af61bf38a to refresh the mirror. Once the mirror catches up, re-run Branch E2E Checks from the Actions tab.

johntmyers · 2026-07-21T18:43:43Z

/ok to test 0869591

johntmyers

gator-agent

PR Review Status

Validation: This is a concentrated fix for the confirmed Kubernetes sandbox startup bug in #2294.
Head SHA: 08695910c048edf59643e547bef38b1af61bf38a

Review findings:

No blocking findings remain.
The independent review found only a non-blocking follow-up suggestion to add Kubernetes E2E coverage for a real read-only submount.

Thanks @varshaprasad96. I verified that the current update resolves both prior gator findings: it removes the cross-device skip that could leave writable PVCs root-owned, and the injected chown test now exercises EROFS recovery while confirming traversal continues. Non-EROFS failures still propagate, and symlinks remain skipped.

Docs: Fern docs are not required because this restores startup behavior without changing a command, configuration field, API, or documented workflow.

E2E: test:e2e is applied for the sandbox-lifecycle change. The current-head mirror refresh was authorized with /ok to test 08695910c048edf59643e547bef38b1af61bf38a; required checks should run after the mirror catches up.

Next state: gator:watch-pipeline.

Signed-off-by: Varsha Prasad Narsing <vnarsing@nvidia.com> Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

johntmyers

gator-agent

PR Review Status

Validation: This remains a concentrated fix for the confirmed Kubernetes sandbox startup bug in #2294.
Head SHA: 2250cb23c012f9da545b81bfeb9d0b099406e115

Review findings:

One blocking high-severity security finding is attached inline: the root-privileged path-based ownership walk remains vulnerable to a symlink swap on concurrently mutable mounted content.
A real Kubernetes read-only-mount regression test remains a useful non-blocking follow-up; the injected unit test cannot exercise actual mount, gVisor, or LSM behavior.

Docs: Fern docs are not required because this restores behavior already promised for read-only PVC configuration and adds no command, option, API, or workflow.

Next state: gator:in-review; author changes are required before pipeline monitoring resumes.

johntmyers · 2026-07-21T19:07:54Z

+        return Ok(());
+    }
+
+    if let Err(e) = do_chown(path, uid, gid) {


gator-agent

High: CWE-367/CWE-59: symlink_metadata does not make this following chown race-safe. A PVC mounted read-only here can still be changed by another pod; after EROFS, the new descent lets an attacker swap a child or ancestor for a symlink and make the root supervisor chown an arbitrary file in its namespace. Minimally stop descending that directory after EROFS while continuing its siblings. If nested writable submounts must remain supported, use an fd-relative walker with no-follow opens and fchown, then add a deterministic symlink-swap regression test.

johntmyers · 2026-07-22T04:26:15Z

/ok to test 2250cb2

varshaprasad96 requested review from a team, derekwaynecarr, maxamillion and mrunalp as code owners July 17, 2026 05:55

varshaprasad96 mentioned this pull request Jul 17, 2026

bug: Kubernetes sandbox crashes EROFS — recursive chown /sandbox fails on read-only submounts (0.0.82) #2294

Open

varshaprasad96 force-pushed the fix/sandbox-erofs-readonly-mounts branch from 6dadfae to 90fae13 Compare July 17, 2026 06:00

elezar reviewed Jul 17, 2026

View reviewed changes

varshaprasad96 added 2 commits July 17, 2026 13:08

fix(sandbox): allow similar_names clippy lint in chown test

888f91a

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

johntmyers self-assigned this Jul 21, 2026

johntmyers reviewed Jul 21, 2026

View reviewed changes

Comment thread crates/openshell-supervisor-process/src/process.rs Outdated

Comment thread crates/openshell-supervisor-process/src/process.rs

johntmyers added the gator:in-review Gator is reviewing or awaiting PR review feedback label Jul 21, 2026

johntmyers added the test:e2e Requires end-to-end coverage label Jul 21, 2026

johntmyers reviewed Jul 21, 2026

View reviewed changes

johntmyers added gator:watch-pipeline Gator is monitoring PR CI/CD status and removed gator:in-review Gator is reviewing or awaiting PR review feedback labels Jul 21, 2026

style(sandbox): apply rustfmt to chown test closures

2250cb2

Signed-off-by: Varsha Prasad Narsing <vnarsing@nvidia.com> Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

johntmyers reviewed Jul 21, 2026

View reviewed changes

johntmyers added gator:in-review Gator is reviewing or awaiting PR review feedback and removed gator:watch-pipeline Gator is monitoring PR CI/CD status labels Jul 21, 2026

		let meta = std::fs::symlink_metadata(path).into_diagnostic()?;

Uh oh!

Conversation

varshaprasad96 commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Testing

Checklist

Uh oh!

copy-pr-bot Bot commented Jul 17, 2026

Uh oh!

elezar Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

varshaprasad96 Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bjw123 commented Jul 20, 2026

Uh oh!

elezar commented Jul 20, 2026

Uh oh!

johntmyers left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 21, 2026

Uh oh!

johntmyers commented Jul 21, 2026

Uh oh!

johntmyers left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

johntmyers left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

johntmyers Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

johntmyers commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

varshaprasad96 commented Jul 17, 2026 •

edited

Loading

varshaprasad96 Jul 18, 2026 •

edited

Loading