Repository navigation
Deadlock in threaded dgemv #660
Description
I've compiled OpenBLAS dfe1eef and I'm linking to it from a JNI interface and calling it from the JVM. It was compiled using USE_THREAD=1 and NO_AFFINITY=0 using gcc 4.4.7. After approximately 5 days of processing between 5 and 10 dgemv calls per second, I saw a deadlock occurring with a thread consuming 100% usage of a CPU core.
I grabbed this backtrace; this was the only thread with symbols in libblas.so:
Thread 1 (process 60961):
#0 0x00007fde1dce2287 in sched_yield () from /lib64/libc.so.6
#1 0x00007fdcca2f5ba5 in exec_blas_async_wait () from ./libblas.so.3
#2 0x00007fdcca2f6272 in exec_blas () from ./libblas.so.3
#3 0x00007fdcca132a74 in dgemv_thread_n () from ./libblas.so.3
#4 0x00007fdcca0f4934 in cblas_dgemv () from ./libblas.so.3
#5 0x00007fdccb31dafb in Java_com_github_fommil_netlib_NativeSystemBLAS_dgemv_1offsets () from /tmp/jniloader5273540829126619406netlib-native_system-linux-x86_64.so
#6 0x00007fde094c04a9 in ?? ()
#7 0x0000000000000000 in ?? ()
./libblas.so.3 is
lrwxrwxrwx 1 wgood users 35 Sep 23 15:16 libblas.so.3 -> libopenblas_sandybridgep-r0.2.14.so
make output looks like this:
OpenBLAS build complete. (BLAS CBLAS LAPACK LAPACKE)
OS ... Linux
Architecture ... x86_64
BINARY ... 64bit
C compiler ... GCC (command line : gcc)
Fortran compiler ... GFORTRAN (command line : gfortran)
Library Name ... libopenblas_sandybridgep-r0.2.14.a (Multi threaded; Max num-threads is 24)
The JVM is only calling dgemv from a single thread at any given point in time.
I will continue to watch for this; I suspect it will happen again in the next day or so although not all JVM processes failed after 5 days. My long term plan was to migrate to a single-threaded BLAS and do my own threading and that will fix this issue as well, but I'll try to see if I can learn anything more about this issue in the meanwhile.
Activity
xianyi commentedon Oct 5, 2015
Do you know what is the input size for dgemv? On the machine, is there only one JVM process calling OpenBLAS?
bkgood commentedon Oct 5, 2015
The matrix is approx 50k rows x 10k cols (row count is ~49k and col count is precisely 10k) and there's a single thread in a single process calling OpenBLAS.
brada4 commentedon Dec 1, 2015
If sched_yield uses 100% of CPU something is badly broken with your system's process scheduler and timers
Can you post to gist
gdb> attach
gdb >thread apply all backtrace
gdb> detach
gdb> quit
jbaiter commentedon Oct 27, 2016
I have the exact same problem with OpenBLAS called from numpy, same backtrace.
Here's the full backtrace as suggested by @brada4: https://gist.github.com/jbaiter/8dd11939bedfa9d9bf7e417c86f02bd9
The version is 0.2.18-1ubuntu1 running on Linux 4.4.0-38-generic.
martin-frbg commentedon Oct 27, 2016
Could be related to races like
==7085== Possible data race during write of size 8 at 0x641C00 by thread #2
==7085== Locks held: none
==7085== at 0x432F91: blas_thread_server (blas_server.c:438)
==7085== by 0x4C2F736: mythread_wrapper (hg_intercepts.c:234)
==7085== by 0x54680A3: start_thread (pthread_create.c:309)
==7085==
==7085== This conflicts with a previous read of size 8 by thread #1
==7085== Locks held: none
==7085== at 0x4333F2: exec_blas_async_wait (blas_server.c:726)
==7085== by 0x4335A5: exec_blas (blas_server.c:795)
==7085== by 0x42C3DE: sgbmv_thread_n (gbmv_thread.c:260)
==7085== by 0x416480: sgbmv_ (gbmv.c:240)
==7085== by 0x40D3A1: schk1_ (sblat2.f:592)
==7085== by 0x413F6F: MAIN__ (sblat2.f:305)
==7085== by 0x4148FE: main (sblat2.f:390)
==7085== Address 0x641c00 is 0 bytes inside data symbol "thread_status"
seen in #995 - I am a bit unsure how to (sanely) put a lock around or into the "while" loop at 726
berke commentedon Oct 31, 2016
Had the same problem, via Lapack, using libopenblas 0.2.19-1 on Debian, on Intel(R) Xeon(R) CPU E5-2687W v3 @ 3.10GHz with 20 cores. The main thread got stuck after a few hours on exec_blas_async_wait called from dsytri from Lapack. Kernel was 4.5.0-2-amd64 #1 SMP Debian 4.5.3-2 (2016-05-08) x86_64 GNU/Linux. The other threads were all in pthread_cond_destroy.
(gdb) bt
#0 0x00007f2c9704b308 in exec_blas_async_wait () from /usr/lib/libopenblas.so.0
#1 0x00007f2c9704b3a7 in exec_blas () from /usr/lib/libopenblas.so.0
#2 0x00007f2c96e7a2ee in dsymv_thread_U () from /usr/lib/libopenblas.so.0
#3 0x00007f2c9a6cb043 in dsymv_ () from /usr/lib/libblas.so.3
#4 0x00007f2c9a12e3ff in dsytri_ () from /usr/lib/liblapack.so.3
martin-frbg commentedon Oct 31, 2016
I am making extremely slow progress, if any, as thread programming is not exactly my forte. Part of the problem now seems to be that blas_thread_server appears to be "thinking" in terms of cpus while exec_blas_async_wait is iterating over threads so I have a hard time putting mutex locks in the latter on the right item to achieve anything.
martin-frbg commentedon Jan 4, 2017
Would either of you be willing to retest with the changed blas_server.c and memory.c linked from #995 (or point me to some self-contained test to check this myself) please ?
sparseinference commentedon Feb 2, 2017
I think I just ran into this problem too. I'd be very happy for a workaround ..
I have a very simple test case with numpy 1.12.0 in Python 3.6.
It hangs at the call to
dot.Here's the backtrace:
berke commentedon Feb 2, 2017
@martin-frbg I don't have an easy-to-reproduce case as it happened after a very long computation launched by someone else, and the code has evolved since so that it doesn't make as many calls now. If the fix works on @sparseinference's case I would like to start using it on a development machine to catch some regressions, but there are no guarantees that I would run into the bug if it isn't fixed.
sparseinference commentedon Feb 2, 2017
I can sacrifice some performance but still make stuff work with:
export OPENBLAS_NUM_THREADS=1
berke commentedon Feb 2, 2017
Same here, we've been running with OPENBLAS_NUM_THREADS=1 since November and had no issues. I parallelize at the process level.
martin-frbg commentedon Feb 2, 2017
@sparceinference which version of OpenBLAS do you see this with - one of the releases (including latest 0.2.19) or with current git develop tree ? The fix (if it is one) is only in the latter so far.
sparseinference commentedon Feb 3, 2017
@martin-frbg - I think it's 2.18 . The library loaded is
libopenblasp-r0.2.18.so.21 remaining items
martin-frbg commentedon Mar 13, 2018
Possibly the cpuid instruction used in the x86_64 "WhereAmI()" is unsuitable for "modern" big Intel systems. https://software.intel.com/en-us/articles/intel-64-architecture-processor-topology-enumeration
discusses means to enumerate processor topology but I do not have time to look into this right now.
dodomorandi commentedon Mar 13, 2018
I can show you as much information as possible, but I don't any idea of how I could find possible issue with OS (nevertheless, the machine works flawlessly, I just do not know what to search for to discover anomalous behaviours).
This is from lscpu:
Feel free to ask me any specific information.
However, to discover a bit the behaviour of my machine, I wrote a simple program in C++. It creates a pool of threads and every thread get the cpu index and the acpi id from
sched_getcpuand thecpuidinstruction and stores them. As soon I spawned the thread, I use thepthread_setaffinity_npto set the thread affinity. Then I join the threads and I check if the ids (both CPU and ACPI) are unique or not.What I found is that this was leading to spurious id collisions, for both types of ids. After a bit I understood the problem: the main thread was not set as affine. As soon as I set the main thread affine using the same procedure, I stopped having id collisions.
The question is: could be exactly this the problem we are encountering?
dodomorandi commentedon Mar 13, 2018
Sorry, it took me some time to write down the previous reply. Maybe we don't need to touch the
WhereAmIfunction, for now. But I will wait for your opinion to have a better idea.martin-frbg commentedon Mar 15, 2018
(Un?)fortunately it seems to me that you may well be more experienced with multithreading in general and NUMA architectures in particular than me. As far as I can tell, gotoblas_affinity_init() in init.c does a sched_setaffinity() for itself before enumerating the cores with whereAmI(), and I assume this should be equivalent to what your test code is doing with the non- (or less-)portable pthread_setaffinity_np() function that appears to be a GNU extension
brada4 commentedon Mar 15, 2018
can you add "numactl -H" output? I have not seen out of order core numbering recently.
Initial report is on small systems, nothing to do with NUMA.
IMO trying to override OpenMP thread placement is wrong.
dodomorandi commentedon Mar 15, 2018
Sure!
You are absolutely right, i just took the first one in my test... just because ;) I replaced it with the
sched_setaffinityfor coherence.Nevertheless, I want to update you on my tests: I started thinking that the problem was about thread switching in critical sections (not the one the could create a data race, but something like a couple of instruction one expects to be executed without an interruption), but now I am not sure. The test I have got now is much more similar to what is going on inside OpenBLAS -- a
sched_setaffinityis called inside the thread. I tried to perform the following things inside the spawned thread, in order:sched_setaffinityis called to make the thread affine to the CPU equal to the thread indexI see that it is always true that the first ID is (generally) different from the second, which in turn is always equal to the third. This means that, at least in this machine with this OS, the rescheduling happens as soon as the
sched_setaffinityis called. I just wanted to be sure that something unexpected could happen in this context.Oh, the test written in this way never leads to ACPI ID collisions. Probably I am going toward the wrong direction 😞
dodomorandi commentedon Apr 4, 2018
I have been quite busy, but I took a bit of time to make a simple check. I initialized all the values of
common->cpu_infoinsidedrivers/other/init.cwithULONG_MAX. Then I tried to put around some assertions, just to check that thecpu_infoparameter was written, and I see these throwing.I am not completely sure this could be the right way to perform this check, but surely @martin-frbg has a better knowledge of the code. Martin, if you have the possibility to give it a look, maybe we could be on the right direction. Thanks! 😄
martin-frbg commentedon Apr 5, 2018
Not sure what you want me to do here, are you seeing active cpu nodes that never had their cpu_info set to a sensible value ? From the git history, the last person to do any serious work on the NUMA mapping seems to have been ashwinyes (remapping cpus to keep all tasks on a common node if possible). I do note there are a few printfs to dump the cpu_info already in the code that should become active when compiling with DEBUG=1.
dodomorandi commentedon Apr 5, 2018
I think I fixed the issue. As I said previously, I made a fork where you can go and perform some tests.
Let's explain what I found.
numa_mappingsets thecommon->cpu_infoarray using the countercount, incremented only when an assignment is performed. When hyperthreading is disabled, we are only assigning onlycommon -> num_procs / 2.At the same time, inside
local_cpu_map,mappingis used as a counter of mapped cores, whilecpuis incremented in any case. This means thatmappingcan only reachcommon -> num_procs / 2with disabled hyperthreading, butcpucan exceed this value. Therefore, herecpu_mapping[mapping]must be set related tocommon->cpu_info[mapping], notcommon->cpu_info[cpu].At the same time,
cpu_sub_mappingmaps to the complete sets of cores, therefore using it to take elements fromcommon->cpu_infoshould only bring to use uninitialized values.The commit I pushed to my repo seems to fix the deadlock. I would really like to use the thread sanitizer to perform additional tests, but as suggested previously by @martin-frbg, this seems to introduce problems. This could also mean that there are still some issues, but it is difficult to say.
If you have the possibility to perform some test, it would be really nice. In theory things should work flawlessly if previously were working, but at the same time who was experiencing the deadlock now should be able to run applications and tests without problems.
martin-frbg commentedon Apr 5, 2018
Thanks for the detailed explanation - in retrospect it makes sense that a misbehaviour in the mapping would be related to the dubious nature of the hyperthreading semi-cores. (Which would also prevent
its showing up on other big systems that use a different architecture)
In my defense, I had looked at your fork but did not see a commit that contained the initializing of
common->cpu_infothat you had mentioned, then I had a look at the code paths that read or write this variable and became confused rather quickly... still after reading your patch I am unsure if dropping the cpu_sub_mapping would let the code still work as intended when hyperthreading is enabled (but I would take your word for it). Maybe I should just read what you wrote aloud three more times to get it.As an alternative to thread sanitizer, perhaps helgrind from the valgrind package can provide additional insight (but there too, the environmental changes necessary for observing may change the system under observation)
dodomorandi commentedon Apr 5, 2018
Actually, this is my fault: I did not include the initialization and the assertions in the commit of the patch, just to focus on the fix. But you are right, it is useful to have a commit with these, I will make one.
After I wrote the previous comment, I remembered I can use valgrind with helgrind. And the result is different from the thread sanitizer (which lead to a strange deadlock during the destruction phase), but it still involves issues. In details, the
openblas_utestaborts on the check I put insidegotoblas_set_affinityto avoid node remapping. But I am not able to reproduce this situation without valgrind (until now).Even if changing the mapping behaviour seems to solve the deadlock problem, I want to be sure I did not introduce other issues, or, if there are other related bugs left, I would like to fix them. The problems with the thread sanitizer and valgrind leave me very suspicious.
dodomorandi commentedon Apr 6, 2018
I made some real case tests, and I find that the issues are still there 😢. I am going to continue searching for bugs.
martin-frbg commentedon Apr 6, 2018
Ouch, too bad. Still the same symptoms though, and are you testing with dgemv (as was the original issue here) ? It is possible that the bug I tracked down in #1497 (using the return value from blas_quickdivide to calculate blocksizes without checking that it is non-zero ) may be present in some of the level3 BLAS files as well.
martin-frbg commentedon Dec 2, 2018
Hopefilly this is now resolved with PR #1865, the fix for #1844 (where it was found that an old optimization (#532) for inputs with small M and large N had introduced local static storage that broke reentrancy)