Repository navigation
Closed
Labels
bugSomething isn't workingSomething isn't workingcuda.pathfinderEverything related to the cuda.pathfinder moduleEverything related to the cuda.pathfinder module
Milestone
Description
Is this a duplicate?
- I confirmed there appear to be no duplicate issues for this request and that I agree to the Code of Conduct
Area
cuda.pathfinder
Is your feature request related to a problem? Please describe.
numba-cuda currently supports functioning with only system packages and no nvvm/nvrtc wheels. Today, I can't find nvvm from the system using cuda-pathfinder:
root@machine:/# ls /usr/local/cuda/nvvm/lib64/libnvvm.so
/usr/local/cuda/nvvm/lib64/libnvvm.so
>>> from cuda.pathfinder import load_nvidia_dynamic_lib
>>> load_nvidia_dynamic_lib('nvvm')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/pyenv/versions/3.12.11/lib/python3.12/site-packages/cuda/pathfinder/_dynamic_libs/load_nvidia_dynamic_lib.py", line 140, in load_nvidia_dynamic_lib
return _load_lib_no_cache(libname)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/pyenv/versions/3.12.11/lib/python3.12/site-packages/cuda/pathfinder/_dynamic_libs/load_nvidia_dynamic_lib.py", line 57, in _load_lib_no_cache
finder.raise_not_found_error()
File "/pyenv/versions/3.12.11/lib/python3.12/site-packages/cuda/pathfinder/_dynamic_libs/find_nvidia_dynamic_lib.py", line 210, in raise_not_found_error
raise DynamicLibNotFoundError(f'Failure finding "{self.lib_searched_for}": {err}\n{att}')
cuda.pathfinder._dynamic_libs.load_dl_common.DynamicLibNotFoundError: Failure finding "libnvvm.so": No such file: libnvvm.so*, No such file: libnvvm.so*, No such file: libnvvm.so*, No such file: libnvvm.so*
Describe the solution you'd like
I'd like load_nvidia_dynamic_lib to be able to load the library at the above path.
Describe alternatives you've considered
numba-cuda currently implements this handling so we could theoretically maintain it, but then it somewhat defeats the purpose of adopting pathfinder.
Additional context
No response
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't workingcuda.pathfinderEverything related to the cuda.pathfinder moduleEverything related to the cuda.pathfinder module
Activity
pathfinderusable as a full replacement forcuda_paths.pyinnumba-cuda#1036leofang commentedon Oct 21, 2025
With which version of pathfinder does this happen? I thought we've made a special case for nvvm precisely because it does not show up in the system search path. We also covered this case in the CI (
$CUDA_PATH/nvvm/lib64/is NOT added to$LD_LIBRARY_PATH).brandon-b-miller commentedon Oct 21, 2025
The error is present for
1.3.1. The issue may be reproduced insidewith
Then
rwgk commentedon Oct 21, 2025
In the meantime (between #441 and #1038) it became certain to me that we need to tie the discovery of
nvvmin some way to the discovery another library, at the point where we reach thesystem-searchstage, butnvvmis not found there. — That's actually what led me to propose the mechanism under #1038, although that mechanism is powerful beyond just findingnvvmin a consistent way.As soon as I'm done with the CTK-next work, I'll jump on this. I'll probably solve this issue with a "scoped search lite" approach.
kkraus14 commentedon Oct 27, 2025
This seems like another case where libraries are in
/usr/local/cuda*but are not necessarily in the ldconfig and that the expectation is we still search and find them within there regardlessrwgk commentedon Oct 27, 2025
@kkraus14 wrote:
Yes. I was thinking
libvvmis the only such case, by design.It's a long-established
numba-cudafeature thatlibnvvmis found, unless we are OK with a regression from a user perspective, we have to support it.Tangential, but maybe useful for me to know long-term:
I'm only aware of one other such case, but we're considering that a bug:
cuda-python/cuda_pathfinder/cuda/pathfinder/_dynamic_libs/load_dl_linux.py
Lines 179 to 195 in 53f5680
Are there any other cases?
rwgk commentedon Oct 27, 2025
I just looked around, I think the bug that prompted us to add the
_work_around_known_bugs()was fixed already.I ran into a similar problem elsewhere (internal) and wrongly assumed it's still the same bug.
I still need to understand why exactly I was seeing that similar problem.
leofang commentedon Nov 4, 2025
Yes. But, this question still needs to be answered:
Why didn't we run into any issue so far, in our
local-ctkCI pipelines? My impression is that we already addressed this. Would be nice to understand it before implementing any fix. Maybe there is nothing to fix?rwgk commentedon Nov 4, 2025
They all set
CUDA_HOME,libnvvmis found that way.leofang commentedon Nov 17, 2025
I see, this is what I missed, thanks!
cpcloud commentedon Feb 4, 2026
This is now supported and in use in numba-cuda. Closing, thanks all!