Cannot locate native library

[librasteve]

My kit:

Welcome to Rakudo™ Star v2024.10.
Implementing the Raku® Programming Language v6.d.
Built on MoarVM version 2024.10.

On ubuntu

PRETTY_NAME="Ubuntu 22.04.5 LTS"

zef

0.22.4

I am trying to

zef install Inline::Python --exclude="python3"

And getting this error on the first test t/call.t...

===> Testing: Inline::Python:ver<0.5>:auth<cpan:NINE>
[Inline::Python] Cannot locate native library '/tmp/.zef.1731085229.69/1731085236.69.5931.3643253181135/resources/15A7732F70D51B37276F0C1819FD5F
[Inline::Python] 091BA26D2D.so': libpython3.11.so.1.0: cannot open shared object file: No such file or directory
[Inline::Python]   in method setup at /home/jovyan/rakudo/share/perl6/core/sources/14F0F0534C870E1E63E36F719EE99DE84801C5C3 (NativeCall) line 359
[Inline::Python]   in method setup at /home/jovy
[Inline::Python] an/rakudo/share/perl6/core/sources/14F0F0534C870E1E63E36F719EE99DE84801C5C3 (NativeCall) line 399
[Inline::Python]   in sub raku-nativecall at /home/jovyan/rakudo/share/perl6/core/sources/5228944CC4FA77F123C90F521225D03ABE23DF88 (NativeCall::Dispatcher) line 50
[Inline::Python]   in method BUILD at /home/jovyan/rakudo/share/perl6/site/sources/37653EDB728A26F20B16AF45FDA81F139277FD9B (Inline::Python) line 537
[Inline::Python]   in block <unit> at t/call.t line 8
[Inline::Python] 
[Inline::Python] Some exceptions were thrown in END blocks:
[Inline::Python]   X::AdHoc: Cannot locate native library '/tmp/.zef.1731085229.69/1731085236.69.5931.3643253181135/resources/15A7732F70D51B37276F0C1819FD5F091BA26D2D.so': libpython3.11.so.1.0: cannot open shared object file: No such file or directory
[Inline::Python]       in method setup at /home/jovyan/rakudo/share/perl6/core/sources/14F0F0534C870E1E63E36F719EE99DE84801C5C3 (NativeCall) line 359
[Inline::Python]       in method setup at /home/jovyan/rakudo/share/perl6/core/sources/14F0F0534C870E1E63E36F719EE99D
[Inline::Python] E84801C5C3 (NativeCall) line 399
[Inline::Python]       in sub raku-nativecall at /home/jovyan/rakudo/share/perl6/core/sources/5228944CC4FA77F123C90F521225D03ABE23DF88 (NativeCall::Dispatcher) line 50
[Inline::Python]       in block  at /home/jovyan/rakudo/share/perl6/site/sources/37653EDB728A26F20B16AF45FDA81F139277FD9B (Inline::Python) line 648
[Inline::Python]       in any <main> at /home/jovyan/rakudo/bin/../share/perl6/runtime/perl6.moarvm line 1
[Inline::Python]       in any <entry> at /home/jovyan/rakudo/bin/../share/perl6/runtime/perl6.moarvm line 1
[Inline::Python] 
[Inline::Python] t/call.t ................ Dubious, test returned 1

So I did a
zef install Inline::Python -v --exclude="python3" --/test

And then ran this snippet:

#!/usr/bin/env perl6

use v6;
use Inline::Python;

say "1..13";

my $py = Inline::Python.new();

And got a clearer error:

(base) root@a8e00c935f1b:~/Inline-Python# raku -I. scum.raku 
1..13
Cannot locate native library '/home/jovyan/Inline-Python/resources/libraries/libpyhelper.so': /home/jovyan/Inline-Python/resources/libraries/libpyhelper.so: cannot open shared object file: No such file or directory
  in method setup at /home/jovyan/rakudo/share/perl6/core/sources/14F0F0534C870E1E63E36F719EE99DE84801C5C3 (NativeCall) line 359
  in method setup at /home/jovyan/rakudo/share/perl6/core/sources/14F0F0534C870E1E63E36F719EE99DE84801C5C3 (NativeCall) line 399
  in sub raku-nativecall at /home/jovyan/rakudo/share/perl6/core/sources/5228944CC4FA77F123C90F521225D03ABE23DF88 (NativeCall::Dispatcher) line 50
  in method BUILD at /home/jovyan/Inline-Python/lib/Inline/Python.pm6 (Inline::Python) line 537
  in block <unit> at scum.raku line 8

Some exceptions were thrown in END blocks:
  X::AdHoc: Cannot locate native library '/home/jovyan/Inline-Python/resources/libraries/libpyhelper.so': /home/jovyan/Inline-Python/resources/libraries/libpyhelper.so: cannot open shared object file: No such file or directory
      in method setup at /home/jovyan/rakudo/share/perl6/core/sources/14F0F0534C870E1E63E36F719EE99DE84801C5C3 (NativeCall) line 359
      in method setup at /home/jovyan/rakudo/share/perl6/core/sources/14F0F0534C870E1E63E36F719EE99DE84801C5C3 (NativeCall) line 399
      in sub raku-nativecall at /home/jovyan/rakudo/share/perl6/core/sources/5228944CC4FA77F123C90F521225D03ABE23DF88 (NativeCall::Dispatcher) line 50
      in block  at /home/jovyan/Inline-Python/lib/Inline/Python.pm6 (Inline::Python) line 648
      in any <main> at /home/jovyan/rakudo/bin/../share/perl6/runtime/perl6.moarvm line 1
      in any <entry> at /home/jovyan/rakudo/bin/../share/perl6/runtime/perl6.moarvm line 1

Here's the source

https://github.com/niner/Inline-Python/blob/05806b0aee2ded79391b63ee5537b7277d17a6b6/lib/Inline/Python.pm6#L10

There is no dir /home/jovyan/Inline-Python/resources at all which makes me suspect the
"builder": "Distribution::Builder::MakeFromJSON"

Please can someone who knows how to get libpyhelper.so to the right place please take a look...?

[librasteve]

One way to replicate my environment is to run this Dockerfile...

FROM librasteve/rakudo:scipy

#RUN    zef install https://github.com/niner/Inline-Python.git --exclude="python3"
RUN    zef install --/test Text::CSV

ENTRYPOINT ["/bin/bash"]

So go docker build -t test . and then docker run -it test and do the commented out command from the command line.

[librasteve]

OK ... try to boil this down ... I isolated the gcc issued by Distribution::Build and ran by hand...

~/Inline-Python# gcc pyhelper.c -I/opt/conda/include/python3.11 -I/opt/conda/include/python3.11  -Wsign-compare -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -isystem /opt/conda/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/python-split_1696333997367/work=/usr/local/src/conda/python-3.11.6 -fdebug-prefix-map=/opt/conda=/usr/local/src/conda-prefix -DNDEBUG -fwrapv -O3 -Wall -L/opt/conda/lib/python3.11/config-3.11-aarch64-linux-gnu -L/opt/conda/lib -lpython3.11 -lpthread -ldl  -lutil -lm  -shared -fPIC -o resources/libraries/libpyhelper.so```

and got this 

pyhelper.c: In function ‘py_init_python’:
pyhelper.c:19:5: warning: ‘Py_SetProgramName’ is deprecated [-Wdeprecated-declarations]
19 | Py_SetProgramName(L"python");
| ^~~~~~~~~~~~~~~~~
In file included from /opt/conda/include/python3.11/Python.h:94,
from pyhelper.c:1:
/opt/conda/include/python3.11/pylifecycle.h:37:38: note: declared here
37 | Py_DEPRECATED(3.11) PyAPI_FUNC(void) Py_SetProgramName(const wchar_t );
| ^~~~~~~~~~~~~~~~~
pyhelper.c:23:5: warning: ‘PySys_SetArgv’ is deprecated [-Wdeprecated-declarations]
23 | PySys_SetArgv(_python_argc, _python_argv); / Tk needs this */
| ^~~~~~~~~~~~~
In file included from /opt/conda/include/python3.11/Python.h:96,
from pyhelper.c:1:
/opt/conda/include/python3.11/sysmodule.h:13:38: note: declared here
13 | Py_DEPRECATED(3.11) PyAPI_FUNC(void) PySys_SetArgv(int, wchar_t **);
| ^~~~~~~~~~~~~
pyhelper.c: In function ‘py_string_as_string’:
pyhelper.c:146:12: warning: return discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
146 | return PyUnicode_AsUTF8(obj);
| ^~~~~~~~~~~~~~~~~~~~~
pyhelper.c: In function ‘py_raise_missing_method’:
pyhelper.c:262:30: warning: initialization discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
262 | char *c_class_name = PyUnicode_AsUTF8(class_name);
| ^~~~~~~~~~~~~~~~
pyhelper.c: In function ‘perl6_invoke’:
pyhelper.c:362:18: warning: initialization discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
362 | char * buf = PyUnicode_AsUTF8AndSize(name, &length);
| ^~~~~~~~~~~~~~~~~~~~~~~
pyhelper.c: In function ‘py_call_method’:
pyhelper.c:323:12: warning: ‘py_retval’ may be used uninitialized in this function [-Wmaybe-uninitialized]
323 | return py_retval;
| ^~~~~~~~~
/usr/bin/ld: cannot open output file resources/libraries/libpyhelper.so: No such file or directory
collect2: error: ld returned 1 exit status
(base) root@3e1dd9b36003:~/Inline-Python# make clean all
make: *** No rule to make target 'clean'. Stop.

[librasteve]

then I went

(base) root@3e1dd9b36003:~/Inline-Python# mkdir -p resources/libraries
(base) root@3e1dd9b36003:~/Inline-Python# gcc pyhelper.c -I/opt/conda/include/python3.11 -I/opt/conda/include/python3.11  -Wsign-compare -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -isystem /opt/conda/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/python-split_1696333997367/work=/usr/local/src/conda/python-3.11.6 -fdebug-prefix-map=/opt/conda=/usr/local/src/conda-prefix -DNDEBUG -fwrapv -O3 -Wall -L/opt/conda/lib/python3.11/config-3.11-aarch64-linux-gnu -L/opt/conda/lib -lpython3.11 -lpthread -ldl  -lutil -lm  -shared -fPIC -o resources/libraries/libpyhelper.so
pyhelper.c: In function ‘py_init_python’:
pyhelper.c:19:5: warning: ‘Py_SetProgramName’ is deprecated [-Wdeprecated-declarations]
   19 |     Py_SetProgramName(L"python");
      |     ^~~~~~~~~~~~~~~~~
In file included from /opt/conda/include/python3.11/Python.h:94,
                 from pyhelper.c:1:
/opt/conda/include/python3.11/pylifecycle.h:37:38: note: declared here
   37 | Py_DEPRECATED(3.11) PyAPI_FUNC(void) Py_SetProgramName(const wchar_t *);
      |                                      ^~~~~~~~~~~~~~~~~
pyhelper.c:23:5: warning: ‘PySys_SetArgv’ is deprecated [-Wdeprecated-declarations]
   23 |     PySys_SetArgv(_python_argc, _python_argv);  /* Tk needs this */
      |     ^~~~~~~~~~~~~
In file included from /opt/conda/include/python3.11/Python.h:96,
                 from pyhelper.c:1:
/opt/conda/include/python3.11/sysmodule.h:13:38: note: declared here
   13 | Py_DEPRECATED(3.11) PyAPI_FUNC(void) PySys_SetArgv(int, wchar_t **);
      |                                      ^~~~~~~~~~~~~
pyhelper.c: In function ‘py_string_as_string’:
pyhelper.c:146:12: warning: return discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
  146 |     return PyUnicode_AsUTF8(obj);
      |            ^~~~~~~~~~~~~~~~~~~~~
pyhelper.c: In function ‘py_raise_missing_method’:
pyhelper.c:262:30: warning: initialization discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
  262 |         char *c_class_name = PyUnicode_AsUTF8(class_name);
      |                              ^~~~~~~~~~~~~~~~
pyhelper.c: In function ‘perl6_invoke’:
pyhelper.c:362:18: warning: initialization discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
  362 |     char * buf = PyUnicode_AsUTF8AndSize(name, &length);
      |                  ^~~~~~~~~~~~~~~~~~~~~~~
pyhelper.c: In function ‘py_call_method’:
pyhelper.c:323:12: warning: ‘py_retval’ may be used uninitialized in this function [-Wmaybe-uninitialized]
  323 |     return py_retval;
      |            ^~~~~~~~~
(base) root@3e1dd9b36003:~/Inline-Python# ls
Build.pm  LICENSE  MANIFEST  META6.json  Makefile.in  README.md  configure.pl6  df-amd  df-arm  lib  named.py  pyhelper.c  resources  t
(base) root@3e1dd9b36003:~/Inline-Python# cd resources/libraries/
(base) root@3e1dd9b36003:~/Inline-Python/resources/libraries# ls
libpyhelper.so

[librasteve]

OK

I found this https://stackoverflow.com/questions/43333207/python-error-while-loading-shared-libraries-libpython3-4m-so-1-0-cannot-open ... which said go

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/conda/lib

^^ the actual path is Python installation dependent

Now I get this in /t

(base) root@3e1dd9b36003:~/Inline-Python/t# prove6 *
call.t ................ ok 
call_back.t ........... ok  
callables.t ........... ok
date.t ................ ok
eval.t ................ ok
eval_return_values.t .. ok
exceptions.t .......... ok
===SORRY!=== Error while compiling /home/jovyan/Inline-Python/t/import.t
Could not find Precomp in:
    /home/jovyan/Inline-Python/t/t/lib
    /home/jovyan/.raku
    /home/jovyan/rakudo/share/perl6/site
    /home/jovyan/rakudo/share/perl6/vendor
    /home/jovyan/rakudo/share/perl6/core
    CompUnit::Repository::AbsolutePath<4728144613136>
    CompUnit::Repository::NQP<4728145398800>
    CompUnit::Repository::Perl5<4728145398840>
at /home/jovyan/Inline-Python/t/import.t:6
import.t .............. Dubious, test returned 1
No subtests run
inherit.t ............. ok
invoke.t .............. ok
matplotlib.t .......... ok
p6_to_py.t ............ ok 
py_to_p6.t ............ ok
All tests successful.

Test Summary Report
-------------------
import.t  (Wstat: 256 Tests: 0 Failed: 0)
Non-zero exit status: 1
  Parse errors: No plan found in TAP output
Files=13, Tests=379,  2 wallclock secs
Result: FAILED

[librasteve]

OK, now I did a clean install of Inline::Python using zef (after rm -rf the ~/Inline-Python/resources/libraries) ...

all works fine except for import.t

[librasteve]

I have raised a new Issue for the import.t failure

Meantime please go --/test

[librasteve]

fwiw it has been pointed out that the line export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/conda/lib (or some better version that picks the path from which python could be put in Makefile or in configure.pl6) ...