How to work with Python packages via e.g. venvs, uv, or conda?

[leftaroundabout]

A large part of the attractiveness of calling Python (for me, the only reason really!) are the packages available on PyPI. Particularly the uv package manager makes it very quick to access all kinds of libraries with functionality that to this day has alas no native Haskell implementation.

The obvious question is: can we access such packages from Haskell using inline-python? I suppose this should be done in some Python virtual environment. The naïve ways don't seem to work, as in, neither running the Haskell processes inside uv run nor after activating a Conda environment do I actually get to use the associated Python interpreters or installed packages. Instead,

runPy [pymain| import sys; print(sys.version) |]

always only shows me the default Python version installed in the operating system.

Is there a way, or is this a fundamental limitation of the way inline-python uses Python as a library?

[Shimuuar]

Thank you for bringing this up. I'm however have only vague understanding of inner working of this tools.

inline-python will use interpreter it has been linked with (libpython3). It should be possible to override that path using LD_LIBRARY_PATH/LD_PRELOAD. I don't think uv sets them, so you can try to set set these variables manually to check whether it will work

[leftaroundabout]

Compiling with

env LD_PRELOAD="~/miniconda3/.../libpython3.14.so" LD_LIBRARY_PATH="~/miniconda3/.../lib/" cabal test

seems to have some kind of effect. The resulting executable won't run then, though:

*** stack smashing detected ***: terminated

Any idea what to do about that?

[Shimuuar]

Sorry for long silence. I was able to run test suite using python installed from conda by setting both LD_PRELOAD and PYTHONHOME:

export LD_PRELOAD=$HOME/.conda/envs/ihs2/lib/libpython3.12.so
export PYTHONHOME=$HOME/.conda/envs/ihs2/

tests were build using python 3.13. Running with conda's 3.12 works, except it segfaults when working with Integers. As it should, since 3.13 introduced new API call. And 3.14 segfaults during initialization

It looks like binary compatibility is just not good enough. So library should be built with python version it should be using. At the moment it relies on pkg-config in order to determine path to libraries, flags, etc. I need to think what's better way to do this

[leftaroundabout]

Interesting that at least one version seems to work. That could already be very useful. Does seem like this would lead to very brittle setups though. It's certainly a tough problem!

What I'm wondering is whether loading Python as an .so library is really the best approach. I've made some attempts myself a while ago to instead run a Python process as a server that only interacts with Haskell via TCP. That would sidestep much of the binary compatibility issues and also be much more flexible with regards to running one or more Pythons on remote machines - but of course it would also incur its own problems. If nothing else, at least a lot of communication overhead.

Currently I have no time for working on any of this...

[Shimuuar]

I finally got some time to work on this. If one to inspect output of pkg-config he'll find that it links with specific python version: -lpython3.13. So it seems python provides strong source compatibility but ABI is not so stable. Only solution is to build inline-python with python provided by uv/conda/etc. I made branch python3-config. It which uses python3-config to obtain necessary compiler/liker flags. I was able to successfully build inline-python with python installed by conda.

What I'm wondering is whether loading Python as an .so library is really the best approach. I've made some attempts myself a while ago to instead run a Python process as a server that only interacts with Haskell via TCP.

Yes that's viable approach with different set of tradeoffs. You have to define serialization for values, it's impossible to pass functions. AFAIR there's such library in common lisp but I've never used it

[leftaroundabout]

Thanks!

So, am I understanding it right: Cabal looks up the currently active Python venv during configuration, then hard-bakes the path into the config files and executable so this can be run regardless of which Conda environment is active, and only re-running cabal configure in a different environment would change the Python?

[Shimuuar]

Almost. You still need to use LD_LIBRARY_PATH to point to python3.so. Maybe it's possible to bake that path into executable but I don't know how. And yes you'll have to rebuild inline-python and everything that depends on it to use another environment. Maybe you don't necessarily need rebuilding for environments with same python version