BUG: Binary Builds Deadlock due to OpenBLAS threading issue with fork

Describe the issue:

Consider the snippet shown below.

On my system with numpy installed via pip this deadlocks with GDB showing an issue in the embedded OpenBLAS where it is stuck on a mutex. Rerunning with export OMP_NUM_THREADS=1 resolves the issue. I suspect this is due to a recent change in how the binary distributions are prepared (maybe previously they were not using multithreaded OpenBLAS?) . The above snippet should be fine, however, as it is the parent which is continuing to use NumPy and its threads should not have changed. Maybe OpenBLAS has a bad atfork handler?

If this is intended behaviour (NumPy not being compatible with any scripts which explicitly call fork) then it might be worth using os.register_at_fork to ensure an exception is thrown.

Reproduce the code example:

import numpy as np
import os

# Do some algebra
A = np.random.randn(216, 216)
np.linalg.inv(A)

# Fork but have the child do nothing
if (pid := os.fork()) != 0:
    # Deadlock!
    np.linalg.inv(A)

    # Wait for the child
    os.waitpid(pid, 0)

Error message:

Python and NumPy Versions:

Build Dependencies:
  blas:
    detection method: pkgconfig
    found: true
    include directory: /opt/_internal/cpython-3.14.0/lib/python3.14/site-packages/scipy_openblas64/include
    lib directory: /opt/_internal/cpython-3.14.0/lib/python3.14/site-packages/scipy_openblas64/lib
    name: scipy-openblas
    openblas configuration: OpenBLAS 0.3.30  USE64BITINT DYNAMIC_ARCH NO_AFFINITY
      Haswell MAX_THREADS=64
    pc file directory: /project/.openblas
    version: 0.3.30
  lapack:
    detection method: pkgconfig
    found: true
    include directory: /opt/_internal/cpython-3.14.0/lib/python3.14/site-packages/scipy_openblas64/include
    lib directory: /opt/_internal/cpython-3.14.0/lib/python3.14/site-packages/scipy_openblas64/lib
    name: scipy-openblas
    openblas configuration: OpenBLAS 0.3.30  USE64BITINT DYNAMIC_ARCH NO_AFFINITY
      Haswell MAX_THREADS=64
    pc file directory: /project/.openblas
    version: 0.3.30
Compilers:
  c:
    commands: cc
    linker: ld.bfd
    name: gcc
    version: 14.2.1
  c++:
    commands: c++
    linker: ld.bfd
    name: gcc
    version: 14.2.1
  cython:
    commands: cython
    linker: cython
    name: cython
    version: 3.1.4
Machine Information:
  build:
    cpu: x86_64
    endian: little
    family: x86_64
    system: linux
  host:
    cpu: x86_64
    endian: little
    family: x86_64
    system: linux
Python Information:
  path: /tmp/build-env-qywnc8nt/bin/python
  version: '3.14'
SIMD Extensions:
  baseline:
  - SSE
  - SSE2
  - SSE3
  found:
  - SSSE3
  - SSE41
  - POPCNT
  - SSE42
  - AVX
  - F16C
  - FMA3
  - AVX2
  not found:
  - AVX512F
  - AVX512CD
  - AVX512_KNL
  - AVX512_KNM
  - AVX512_SKX
  - AVX512_CLX
  - AVX512_CNL
  - AVX512_ICL
  - AVX512_SPR

Runtime Environment:

No response

Context for the issue:

No response

I can confirm the deadlock, on CPython3.14 and 3.13 (and maybe others). I used 3.13 so I could go back in versions to see where it started. numpy==2.3.1 does not deadlock, numpy2.3.2 does. The corresponding versions of OpenBLAS are 0.3.29 (does not deadlock), 0.3.30 (does deadlock). Maybe connected to #29391.

@martin-frbg

Bisecting points to OpenMathLib/OpenBLAS#5170, which fixed some other threading problems. The backtrace at the hang is this. I am a bit confused why there multiple calls to dgetrf_parallel in the stack. @martin-frbg any thoughts?

(gdb) bt
#0  futex_wait (private=0, expected=2, futex_word=0x7ffff71efa80 <server_lock>) at ../sysdeps/nptl/futex-internal.h:146
#1  __GI___lll_lock_wait (futex=futex@entry=0x7ffff71efa80 <server_lock>, private=0) at ./nptl/lowlevellock.c:49
#2  0x00007ffff7ca0101 in lll_mutex_lock_optimized (mutex=0x7ffff71efa80 <server_lock>) at ./nptl/pthread_mutex_lock.c:48
#3  ___pthread_mutex_lock (mutex=0x7ffff71efa80 <server_lock>) at ./nptl/pthread_mutex_lock.c:93
#4  0x00007ffff605b388 in blas_thread_init.part () from /tmp/venv313/lib/python3.13/site-packages/scipy_openblas64/lib/libscipy_openblas64_.so
#5  0x00007ffff605b93b in exec_blas_async () from /tmp/venv313/lib/python3.13/site-packages/scipy_openblas64/lib/libscipy_openblas64_.so
#6  0x00007ffff6060fb3 in dgetrf_parallel () from /tmp/venv313/lib/python3.13/site-packages/scipy_openblas64/lib/libscipy_openblas64_.so
#7  0x00007ffff60607bd in dgetrf_parallel () from /tmp/venv313/lib/python3.13/site-packages/scipy_openblas64/lib/libscipy_openblas64_.so
#8  0x00007ffff60607bd in dgetrf_parallel () from /tmp/venv313/lib/python3.13/site-packages/scipy_openblas64/lib/libscipy_openblas64_.so
#9  0x00007ffff60607bd in dgetrf_parallel () from /tmp/venv313/lib/python3.13/site-packages/scipy_openblas64/lib/libscipy_openblas64_.so
#10 0x00007ffff5cfb890 in scipy_dgesv_64_ () from /tmp/venv313/lib/python3.13/site-packages/scipy_openblas64/lib/libscipy_openblas64_.so
#11 0x00007ffff784b660 in call_gesv (params=0x7fffffff79f0) at ../numpy/linalg/umath_linalg.cpp:1674
#12 inv<double> (args=0x7fffb911a720, dimensions=<optimized out>, steps=<optimized out>, __NPY_UNUSED_TAGGEDfunc=<optimized out>)
    at ../numpy/linalg/umath_linalg.cpp:1857
#13 0x00007fffb9873b39 in generic_wrapped_legacy_loop (__NPY_UNUSED_TAGGEDcontext=<optimized out>, data=<optimized out>, dimensions=<optimized out>, 
    strides=<optimized out>, auxdata=0x7fffb915e070) at ../numpy/_core/src/umath/legacy_array_method.c:98

Bisecting points to OpenMathLib/OpenBLAS#5170, which fixed some other threading problems. The backtrace at the hang is this. I am a bit confused why there multiple calls to dgetrf_parallel in the stack. @martin-frbg any thoughts?

I believe they're using a recursive implementation. See:

https://github.com/OpenMathLib/OpenBLAS/blob/develop/lapack/getrf/getrf_parallel.c#L678

which calls:

https://github.com/OpenMathLib/OpenBLAS/blob/develop/lapack/getrf/getrf_parallel.c#L764

I have always found the threading support in OpenBLAS to be somewhat twitchy. Likely not something I'd trust without a huge test suite.

Did anyone had the capacity to try this on MKL or Accelerate platforms to single out OpenBLAS?

Seems to run fine with numpy main (5fe514b), python 3.12, and

$ mamba list |grep mkl
# packages in environment at /home/ev-br/.conda/envs/numpy-dev-mkl:
mkl                       2025.3.0           h0e700b2_462    conda-forge
mkl-devel                 2025.3.0           ha770c72_462    conda-forge
mkl-include               2025.3.0           hf2ce2f3_462    conda-forge

$ cat fork.py 
import numpy as np
import os

# Do some algebra
A = np.random.randn(216, 216)
np.linalg.inv(A)

# Fork but have the child do nothing
if (pid := os.fork()) != 0:
    # Deadlock!
    np.linalg.inv(A)

    # Wait for the child
    os.waitpid(pid, 0)

$ time for i in `seq 1 100`; do python -P fork.py; done

real	0m11.428s
user	0m7.336s
sys	0m3.927s

On the same machine, does hang with

$ mamba list |grep openblas
libblas                   3.9.0           34_h59b9bed_openblas    conda-forge
libcblas                  3.9.0           34_he106b2a_openblas    conda-forge
liblapack                 3.9.0           34_h7ac8fdf_openblas    conda-forge
libopenblas               0.3.30          pthreads_h94d23a6_1    conda-forge
openblas                  0.3.30          pthreads_h6ec200e_1    conda-forge
$ mamba list |grep numpy
numpy                     2.2.6           py311h5d046bc_0    conda-forge

Having a reproducer for Linux/x86_64 (as opposed to a hang happening exclusively on OSX/x86_64 as with the related scipy issue) should help, thanks.
And regarding the perceived twitchyness - the problem is (largely) that thread safety was a very late consideration - with the original GotoBLAS and any subsequent OpenBLAS until the beginning of my involvement about 10 years ago, one had to use OpenMP and hope for the best, while the pthreads build was racy as hell

What is surprising is that it is the parent which has issues. When forking one would expect the child to get confused (since all of the other threads it thinks it spawned in the pool are now gone). But the parent should usually be fine even if it doesn't take any action. This makes me wonder if an atfork handler is at fault.

Also, if it makes any difference the code seems to work with lower thread counts i.e., export OMP_NUM_THREADS=2 does not exhibit the issue, but larger numbers do (and I was running on a 20 'core' machine).

Yes, I'm also thinking that it is the atfork handler that causes the deadlock - it uses the same (changed in PR5170) atomic ordering constraints as the main blas_server loop while it tries to shut down the thread pool.
And I guess higher thread count simply translates to a higher probability that races/clashes occur (which would also be a main reason for the blatant lack of thread safety having no bearing on the early success of GotoBLAS/OpenBLAS - not much of a race when one can have at most 4 contestants)

Hmm, seems the hang occurs after the atfork handler has returned.

That makes sense to me. From the backtrace it is making it to the inversion function. However, my working assumption is that the atfork handler is messing up the threadpool which causes subsequent calls to deadlock.

@lesteve

I mentioned this issue on scipy/scipy#23686, which is the same problem on SciPy. @lesteve commented with a C reproducer there that only uses OpenBLAS. I can confirm that the small C reproducer hangs on Ubuntu 24.04 x86_64 just like in this issue with NumPy. Here is a shortened incomplete version of the reproducer.

int main() {
    int64_t m = 200, n = 200;
    int64_t lda = m;
    int64_t info;
    int64_t ipiv[200];

    // array is an identity matrix
    double arr[200*200];
    for (int i = 0; i < m*n; i += n + 1) {
        arr[i] = 1.0;
    }

    printf("before fork\n");
    pid_t pid = fork();
    printf("after fork\n");
    if (pid == 0) {
        printf("inside child\n");
        exit(0);
    } else {
        wait(NULL);
    }

    printf("before dgetrf\n");
    dgetrf_(&m, &n, arr, &lda, ipiv, &info);
    printf("after dgetrf\n");

When I add some printing at various calls inside OpenBLAS, I see:

installing atfork handler in memory::openblas_fork_handler 2015
in blas_thread_init
before fork
in blas_thread_shutdown
after fork
after fork
inside child
in blas_thread_shutdown
before dgetrf
in blas_thread_init
<hangs>

~~Is it expected to see the additional call to blas_thread_shutdown?~~ (Edit: yes, that is a consequence of exit()ing the child) In any case, the hang seems to be in blas_thread_init at the call to LOCK_COMMAND(&server_lock);.

It would be interesting to know the state of server_lock at the start (so when blas_thread_init is called for the very first time), before atfork, and post atfork. If the atfork handler is cleaning everything up before we fork (so bringing down the thread pool) then the variables should be reset to their initial states. I am guessing this is not happening and this is what is causing us to deadlock.

Let's move further discussion to OpenMathLib/OpenBLAS#5520

Given the issue now appears to be fixed on the OpenBLAS side what are the next steps? Is it worth reverting the OpenBLAS version used for the NumPy builds in the interim whilst we wait for a new release of OpenBLAS to be made?

@mattip

Hi @mattip - thanks for the above. However, I'm still getting issues with Freddie's initial test code on my Mac with Python 3.12 and Numpy 2.4.1. Specifically, it works fine with OMP_NUM_THREADS=1 but hangs if OMP_NUM_THREADS is not set.

>>> import numpy as np
>>> print(np.__version__)
2.4.1
>>> print(np.show_config())
Build Dependencies:
  blas:
    detection method: pkgconfig
    found: true
    include directory: /private/var/folders/pr/_3h4tm913_l56k6hrhpm89z00000gn/T/cibw-run-d6u3niut/cp312-macosx_arm64/build/venv/lib/python3.12/site-packages/scipy_openblas64/include
    lib directory: /private/var/folders/pr/_3h4tm913_l56k6hrhpm89z00000gn/T/cibw-run-d6u3niut/cp312-macosx_arm64/build/venv/lib/python3.12/site-packages/scipy_openblas64/lib
    name: scipy-openblas
    openblas configuration: OpenBLAS 0.3.30  USE64BITINT DYNAMIC_ARCH NO_AFFINITY
      neoversen1 MAX_THREADS=64
    pc file directory: /Users/runner/work/numpy-release/numpy-release/.openblas
    version: 0.3.30
  lapack:
    detection method: pkgconfig
    found: true
    include directory: /private/var/folders/pr/_3h4tm913_l56k6hrhpm89z00000gn/T/cibw-run-d6u3niut/cp312-macosx_arm64/build/venv/lib/python3.12/site-packages/scipy_openblas64/include
    lib directory: /private/var/folders/pr/_3h4tm913_l56k6hrhpm89z00000gn/T/cibw-run-d6u3niut/cp312-macosx_arm64/build/venv/lib/python3.12/site-packages/scipy_openblas64/lib
    name: scipy-openblas
    openblas configuration: OpenBLAS 0.3.30  USE64BITINT DYNAMIC_ARCH NO_AFFINITY
      neoversen1 MAX_THREADS=64
    pc file directory: /Users/runner/work/numpy-release/numpy-release/.openblas
    version: 0.3.30
Compilers:
  c:
    commands: cc
    linker: ld64
    name: clang
    version: 15.0.0
  c++:
    commands: c++
    linker: ld64
    name: clang
    version: 15.0.0
  cython:
    commands: cython
    linker: cython
    name: cython
    version: 3.2.4
Machine Information:
  build:
    cpu: aarch64
    endian: little
    family: aarch64
    system: darwin
  host:
    cpu: aarch64
    endian: little
    family: aarch64
    system: darwin
Python Information:
  path: /private/var/folders/pr/_3h4tm913_l56k6hrhpm89z00000gn/T/build-env-ljnyeb_0/bin/python
  version: '3.12'
SIMD Extensions:
  baseline:
  - NEON
  - NEON_FP16
  - NEON_VFPV4
  - ASIMD
  found:
  - ASIMDHP
  - ASIMDDP
  not found:
  - ASIMDFHM

What version of scipy-openblas64 are you using? (i.e. what version is at /private/var/folders/pr/_3h4tm913_l56k6hrhpm89z00000gn/T/cibw-run-d6u3niut/cp312-macosx_arm64/build/venv/lib/python3.12/site-packages/scipy_openblas64? When I use the PyPI provided wheel (pip install numpy==2.4.1) I do not see the hang

I couldn't work out how to get the version, but I have MD5 (libscipy_openblas64_.dylib) = ffc464ddedf3038f3c928b6ea9a33999 if that helps? The .dylib file lives inside the venv on my system: venv/lib/python3.12/site-packages/numpy/.dylibs/libscipy_openblas64_.dylib (the path in the above post is apparently "a build-time path that got baked into the wheel metadata when the NumPy wheel was produced" according to ChatGPT).

This is confusing. You say you have venv/lib/python3.12/site-packages/numpy/.dylibs/libscipy_openblas64_.dylib, which indicates you are using numpy from PyPI. But show_config is showing /private/var/folders/pr/_3h4tm913_l56k6hrhpm89z00000gn/T/cibw-run-d6u3niut/cp312-macosx_arm64/build/venv/lib/python3.12/site-packages/scipy_openblas64/lib which would indicate you also have scipy-openblas64 installed from a run of cibuildwheel. Could describe how you put together this NumPy environment?

Thanks for the reply. I just used "pip install numpy==2.4.1". I tried again in a completely fresh venv and have the same issues.

Please open a new issue, and clearly indicate your operating system, platform, and what you are doing to produce the hang. Above you said

When I use the PyPI provided wheel (pip install numpy==2.4.1) I do not see the hang

but then

I tried again in a completely fresh venv and have the same issues.

Thanks. I'll open a new issue. Just FYI, I have consistently been seeing the issue my side, I think

When I use the PyPI provided wheel (pip install numpy==2.4.1) I do not see the hang

was a comment in one of your posts.

Hi, I'm seeing this issue with numpy 2.4.1 + Python 3.13.11. This is on an arm64 system running Ubuntu 24.04.

Is there a work around or a fix that you suggest?

See #30733

Navigation Menu

Repository navigation

BUG: Binary Builds Deadlock due to OpenBLAS threading issue with fork #30092

Describe the issue:

Reproduce the code example:

Error message:

Python and NumPy Versions:

Runtime Environment:

Context for the issue:

1 remaining item

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

BUG: Binary Builds Deadlock due to OpenBLAS threading issue with fork #30092

Description

Describe the issue:

Reproduce the code example:

Error message:

Python and NumPy Versions:

Runtime Environment:

Context for the issue:

Activity

mattip commented on Oct 29, 2025

mattip commented on Oct 29, 2025

FreddieWitherden commented on Oct 29, 2025

ilayn commented on Oct 29, 2025

ev-br commented on Oct 29, 2025

martin-frbg commented on Oct 29, 2025

FreddieWitherden commented on Oct 29, 2025

martin-frbg commented on Oct 29, 2025

martin-frbg commented on Oct 29, 2025

FreddieWitherden commented on Oct 29, 2025

mattip commented on Oct 30, 2025

FreddieWitherden commented on Oct 30, 2025

mattip commented on Oct 30, 2025

FreddieWitherden commented on Oct 31, 2025

1 remaining item

pv101 commented on Jan 19, 2026

mattip commented on Jan 20, 2026

pv101 commented on Jan 21, 2026

mattip commented on Jan 22, 2026

pv101 commented on Jan 25, 2026

mattip commented on Jan 26, 2026

pv101 commented on Jan 27, 2026

WillTrojak commented on Jan 30, 2026

mattip commented on Jan 30, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions