Support universal requirements for multiple python versions

[keith]

🐞 bug report

Affected Rule

pip.parse

Is this a regression?

I don't think so

Description

Some dependencies can vary based on the current python version. In this case for requirements.txt files generated by uv or similar tools, the python version is encoded in the requirements.txt:

numpy==2.0.2 ; python_full_version < '3.10'
    # via -r requirements.in
numpy==2.2.5 ; python_full_version >= '3.10'
    # via -r requirements.in

In this case if you attempt to use this requirements file with pip.parse you get this error:

ERROR: /home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python+/python/private/pypi/extension.bzl:224:25: Traceback (most recent call last):
        File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python+/python/private/pypi/extension.bzl", line 607, column 25, in _pip_impl
                mods = parse_modules(module_ctx)
        File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python+/python/private/pypi/extension.bzl", line 478, column 36, in parse_modules
                out = _create_whl_repos(
        File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python+/python/private/pypi/extension.bzl", line 224, column 25, in _create_whl_repos
                fail("Attempting to creating a duplicate library {} for {}".format(
Error in fail: Attempting to creating a duplicate library pip_311_numpy_linux_aarch64_linux_arm_linux_ppc_linux_s390x_linux_x86_64_osx_aarch64_osx_x86_64_windows_x86_64 for numpy

Ideally rules_python would know that there should only be a single repo here since these constraints are mutually exclusive

🔬 Minimal Reproduction

bazel build ... in #2798

🌍 Your Environment

Operating System:

  
linux x86_64
  

Output of bazel version:

  
8.2.1
  

Rules_python version:

c981569

[aignas]

For experimental_index_url code path where the sha256 is included in the repo name, this will not be an issue.

[baxelrod-bdai]

I am also getting this error.

Minimal reproduction

requirements.in has just:

ipython >= 5

and BUILD.bazel has:

load("@rules_uv//uv:pip.bzl", "pip_compile")

pip_compile(
    name = "generate_requirements",
    extra_args = [
        "--universal",
    ],
    requirements_in = "requirements.in",
    requirements_txt = "requirements.lock",
)

bazel run //:generate_requirements runs fine, and creates a lock file with:

ipython==8.35.0 ; python_full_version < '3.11' \
    --hash=sha256:d200b7d93c3f5883fc36ab9ce28a18249c7706e51347681f80a0aef9895f2520 \
    --hash=sha256:e6b7470468ba6f1f0a7b116bb688a3ece2f13e2f94138e508201fad677a788ba
    # via -r requirements.in
ipython==9.1.0 ; python_full_version >= '3.11' \
    --hash=sha256:2df07257ec2f84a6b346b8d83100bcf8fa501c6e01ab75cd3799b0bb253b3d2a \
    --hash=sha256:a47e13a5e05e02f3b8e1e7a0f9db372199fe8c3763532fe7a1e0379e4e135f16
    # via -r requirements.in

But when you try to bazel run anything else, I get the crash:

ERROR: /home/baxelrod/.cache/bazel/_bazel_baxelrod/223f490fa0c2f2ec38707ddac3dd6272/external/rules_python+/python/private/pypi/extension.bzl:231:25: Traceback (most recent call last):
	File "/home/baxelrod/.cache/bazel/_bazel_baxelrod/223f490fa0c2f2ec38707ddac3dd6272/external/rules_python+/python/private/pypi/extension.bzl", line 611, column 25, in _pip_impl
		mods = parse_modules(module_ctx)
	File "/home/baxelrod/.cache/bazel/_bazel_baxelrod/223f490fa0c2f2ec38707ddac3dd6272/external/rules_python+/python/private/pypi/extension.bzl", line 482, column 36, in parse_modules
		out = _create_whl_repos(
	File "/home/baxelrod/.cache/bazel/_bazel_baxelrod/223f490fa0c2f2ec38707ddac3dd6272/external/rules_python+/python/private/pypi/extension.bzl", line 231, column 25, in _create_whl_repos
		fail("Attempting to creating a duplicate library {} for {}".format(
Error in fail: Attempting to creating a duplicate library pip_310_ipython_linux_aarch64_linux_arm_linux_ppc_linux_s390x_linux_x86_64_osx_aarch64_osx_x86_64_windows_x86_64 for ipython
ERROR: error evaluating module extension @@rules_python+//python/extensions:pip.bzl%pip

Versions

Bazelisk version: v1.19.0
Build label: 8.1.1

bazel_dep(name = "rules_python", version = "1.4.0-rc1")
bazel_dep(name = "rules_uv", version = "0.68.0")

Linux baxelrod-ThinkPad 6.8.0-57-generic #59~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Mar 19 17:07:41 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Notes

I tried adding this to my .bazelrc to force things to use a single Python version, but no luck.

build --action_env=PYTHON_VERSION=3.10
run --action_env=PYTHON_VERSION=3.10

The problem goes away if I don't specify --universal in the uv pip compile arguments. Because then no markers are added in the lock file.

[baxelrod-bdai]

@aignas can you explain your comment? Is this a workaround?

For experimental_index_url code path where the sha256 is included in the repo name, this will not be an issue.

[keith]

with your requirements file, since it has hashes, if you set experimental_index_url = "https://pypi.org/simple", on pip.parse it will not fail with this error.

In my case it seems using experimental_index_url causes a 50% regression in dependency download time, but I plan to file a separate issue for that.

I am also curious if the plan is to require this codepath for this fix, in which case maybe we should add a more clear error

[hartikainen]

For what it's worth, we're having similar issues with jax, when using jax[cuda12] for linux_x86_64 and jax for {linux,macos}_arm64. I created a reproduction at main...hartikainen:rules_python:bug/duplicate-library.

Here's a log from running this on macos_arm64. Note that this is with experimental_index_url = "https://pypi.org/simple" .

Error log

$ bazel run //:main
WARNING: Build option --@@rules_python+//python/config_settings:venvs_use_declare_symlink has changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed).
ERROR: /private/var/tmp/_bazel_user/9567799826d8cc6628021154b048c90b/external/rules_python+/python/private/pypi/extension.bzl:266:21: Traceback (most recent call last):
        File "/private/var/tmp/_bazel_user/9567799826d8cc6628021154b048c90b/external/rules_python+/python/private/pypi/extension.bzl", line 625, column 25, in _pip_impl
                mods = parse_modules(module_ctx, enable_pipstar = rp_config.enable_pipstar)
        File "/private/var/tmp/_bazel_user/9567799826d8cc6628021154b048c90b/external/rules_python+/python/private/pypi/extension.bzl", line 501, column 36, in parse_modules
                out = _create_whl_repos(
        File "/private/var/tmp/_bazel_user/9567799826d8cc6628021154b048c90b/external/rules_python+/python/private/pypi/extension.bzl", line 266, column 21, in _create_whl_repos
                fail("attempting to create a duplicate library {} for {}".format(
Error in fail: attempting to create a duplicate library pypi_312_jax_py3_none_any_62833036 for jax
ERROR: Analysis of target '//:main' failed; build aborted: error evaluating module extension @@rules_python+//python/extensions:pip.bzl%pip
INFO: Elapsed time: 5.041s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully
ERROR: Build failed. Not running target
FAILED:
    Fetching ...tension @@rules_python+//python/extensions:pip.bzl%pip; Fetch package lists from PyPI index 4s

The same run works fine if I comment out the linux_x86_64 line from requirements_by_platform.

Working diff

modified   examples/bzlmod/multi-platform/MODULE.bazel
@@ -26,7 +26,7 @@ pip.parse(
     python_version = "3.12",
     requirements_by_platform = {
         "requirements.linux_arm64.txt": "linux_aarch64",
-        "requirements.linux_x86_64.txt": "linux_x86_64",
+        # "requirements.linux_x86_64.txt": "linux_x86_64",
         "requirements.macos_arm64.txt": "osx_aarch64",
     },
 )

Another workaround that seems to fix this to comment out one of the hashes from requirements.linux_x86_64.txt. Maybe this is what @aignas suggested above?

Workaround

modified   examples/bzlmod/multi-platform/requirements.linux_x86_64.txt
@@ -49,8 +49,7 @@ jax==0.7.0 \
     #   -r requirements.in
     #   mujoco-mjx
 jax[cuda12]==0.7.0 \
-    --hash=sha256:4dd8924f171ed73a4f1a6191e2f800ae1745069989b69fabc45593d6b6504003 \
-    --hash=sha256:62833036cbaf4641d66ae94c61c0446890a91b2c0d153946583a0ebe04877a76
+    --hash=sha256:4dd8924f171ed73a4f1a6191e2f800ae1745069989b69fabc45593d6b6504003
     # via -r requirements.in
 jax-cuda12-pjrt==0.7.0 \
     --hash=sha256:742cc260d697771cb883084398748a007d233286aa3fe5fc152a74c073d59a24 \

I'm not sure how good of a solution this is as I would prefer not to touch the requirements locks manually. Is there any known solution or other workaround for this?

[aignas]

The PR that is waiting for review (#3058) should hopefully fix some of the errors that are described in here, could you please test that? It is not fixing the problem directly though.

That said, I think this is a bug and it would be great to add another test case in https://github.com/bazel-contrib/rules_python/blob/main/tests/pypi/extension/extension_tests.bzl

We already have a test where we use different extras in a single requirements file (see the optimum test case), but for the case that you are describing we don't have a test.

[hartikainen]

I tested my setup on top of aignas@5fd3c86 (see aignas/rules_python@exp/pypi-simplify-freethreaded...hartikainen:rules_python:bug/duplicate-library+5fd3c86) but the same issue persists.

I created a test from my case in aignas#2.

[aignas]

Please test the latest PR, if it doesn't fix the issue (even after merged), I'll reopen this ticket.

[hartikainen]

My test case (aignas#2 #3193) still fails. Perhaps that's a different issue and should be reported separately?