`local_runtime_repo` writes absolute path of Xcode on macOS

[erikkerber]

🐞 bug report

(Sorry for all the annoying macOS specific bugs!)

Affected Rule

The issue is caused by the rule: `local_runtime_repo`

Is this a regression?

Yes(?)

Description

Starting with this change: https://github.com/bazel-contrib/rules_python/pull/2760/files#diff-33a51eb016b4b2a4b53d940f07c5388bd572ef61689051bc6f266ea7e5533c72R99

If running on macOS, /usr/bin/python3 is a system shim that will run a Python runtime located within Xcode. The path of Xcode will then show up in base_executable:

❯ /usr/bin/python3 python/private/get_local_runtime_info.py
{"major": 3, "minor": 9, "micro": 6, "include": "/Library/Python/3.9/include", "implementation_name": "cpython", "base_executable": "/Applications/Xcode-26.0.0-Beta.app/Contents/Developer/usr/bin/python3", "LDLIBRARY": "Python3.framework/Versions/3.9/Python3", "LIBDIR": "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib", "INSTSONAME": "Python3.framework/Versions/3.9/Python3", "PY3LIBRARY": "", "SHLIB_SUFFIX": ".so"}

This path then ends up in the run script for i.e. py_binary:

76:  PYTHON_BINARY = '/Applications/Xcode-26.0.0-Beta.app/Contents/Developer/usr/bin/python3'

🔬 Minimal Reproduction

1. macOS
2. local_runtime_repo setup with interpreter_path = "/usr/bin/python3"
3. build a py_binary target and observe the resulting run script
4. Change/move your Xcode version or path and rebuild. The py_binary path will be unchanged.

[aignas]

No need to be sorry, thanks for reporting this - it helps others who search for a similar issue.

What is the expected behaviour here?

[pennig]

Expected behavior is either:

• PYTHON_BINARY remains as /usr/bin/python3 and isn't resolved to the honest binary inside of Xcode.
• rules_python becomes sensitive to DEVELOPER_DIR and/or XCODE_VERSION (common env vars associated with the selected Xcode installation), so if those change it invalidates the bootstrap script's cache and it gets regenerated.

[aignas]

In my view the DEVELOPER_DIR and XCODE_VERSION awarness on osx would be preferred, because otherwise if the Python version is changed underthehood, the cache within bazel cannot be invalidated. Am I getting it right here?

If so, PRs on DEVELOPER_DIR and XCODE_VERSION awarness would be welcome! I don't use a Mac these days, so I cannot fully test or develop this.

[erikkerber]

@rickeylev Matt's PR did help builds not point to an invalid/non-existing Python path when using macOS, but the path still exists in the py_binary file.

This means that the same Xcode, placed in different directories, will result in cache misses when the Xcode path is different across machines. Mostly important when a py_binary is then an input to another rule.

Minor housekeeping question: think this should be re-opened? I can write another cache-oriented issue too.

[aignas]

Thanks for getting back here and checking. Any extra information from your last debugging might be useful!

[erikkerber]

Yes! I noticed this debugging some cache misses and discussed it a bit in the Bazel Slack workspace

The original change that led to this nuance on macOS remains—base_executable resolves to a full Xcode path on macOS, even if the "local" python points to the (shim) /usr/bin/python3.

The original change included this comment, which tells me the solution isn't simply "reverting the change" which would of course regress what it was originally solving with venv.

# We use base_executable because we want the path within a Python
# installation directory ("PYTHONHOME"). The problems with sys.executable
# are:
# * If we're in an activated venv, then we don't want the venv's
#   `bin/python3` path to be used -- it isn't an actual Python installation.
# * If sys.executable is a wrapper (e.g. pyenv), then (1) it may not be
#   located within an actual Python installation directory, and (2) it
#   can interfer with Python recognizing when it's within a venv.
#
# In some cases, it may be a symlink (usually e.g. `python3->python3.12`),
# but we don't realpath() it to respect what it has decided is the
# appropriate path.