Repository navigation
[regression] 0.3.29 test failure on ARMV6 #5104
Description
openblas 0.3.29 fails to build on Debian's armhf machine: https://buildd.debian.org/status/fetch.php?pkg=openblas&arch=armhf&ver=0.3.29%2Bds-1&stamp=1738175608&raw=0
It seems to be a regression in the cblas_stbmv function:
******* FATAL ERROR - COMPUTED RESULT IS LESS THAN HALF ACCURATE *******
EXPECTED RESULT COMPUTED RESULT
1 0.157343 0.127373
2 0.00000 0.00000
3 0.447053 0.447053
4 0.277223 0.277223
5 0.806693 0.806693
******* cblas_stbmv FAILED ON CALL NUMBER:
580: cblas_stbmv ( CblasUpper, CblasNoTrans, CblasUnit,
5, 0, A, 2, X,-1) .
******* FATAL ERROR - TESTS ABANDONED *******
ERROR STOP
Debian uses a modified source tree. But I can reproduce this problem with the upstream git repo:
TARGET=ARMV6 make
The error message reads:
******* FATAL ERROR - COMPUTED RESULT IS LESS THAN HALF ACCURATE *******
EXPECTED RESULT COMPUTED RESULT
1 0.245164 0.245164
2 0.00000 0.00000
3 0.539071 0.539071
4 1.22516 1.22516
5 0.601097E-01 0.374625E-01
******* cblas_stbmv FAILED ON CALL NUMBER:
582: cblas_stbmv ( CblasUpper, CblasNoTrans, CblasNonUnit,
5, 0, A, 2, X, 1) .
******* cblas_stbmv FAILED ON CALL NUMBER:
2: cblas_stbmv ( CblasUpper, CblasNoTrans, CblasUnit,
1, 0, A, 2, X, 1) .
******* FATAL ERROR - TESTS ABANDONED *******
ERROR STOP
This is a regression because 0.3.28 does not fail the test. I checked the diff between 0.3.28 and 0.3.29 but did not find anything straightforward. Any idea?
Activity
cdluminate commentedon Jan 31, 2025
It seems to be a threading issue:
But the test can pass in serial mode:
Update: seems to be pthread related. OpenMP is fine:
martin-frbg commentedon Feb 1, 2025
Curious that you would get an error in tbmv (only). Did you change compilers between 0.3.28 and 29 perchance ?
cdluminate commentedon Feb 1, 2025
I did not change the compiler. It's gcc-14
14.2.0-16from Debian unstable.martin-frbg commentedon Feb 2, 2025
I see - still seems weird that it is STBMV of all functions. I won't be able to reproduce this or eventually locate the commit that caused this until late next week. (Nothing immediately obvious that wouldn't also mess up all of GEMM)
martin-frbg commentedon Feb 2, 2025
I'm increasingly certain that none of the files/functions relevant for multithreaded TBMV were changed between 0.3.28 and 0.3.29... could be this is an older bug, a missing memory barrier somewhere or something like that, that has only a low probability to happen. Is the armhf machine an actual hardware or something like qemu ?
cdluminate commentedon Feb 2, 2025
The
armhfmachine is a real arm64 machine withNeoverse-N1CPU. (Debian's infrastructure is all real machines. I tested on Debian's porterbox which is also real machine).Recently I have seen some regression bugs in gcc-14 which leads to internal compiler error when compiling pytorch (downgrading to gcc-13 solves the issue). Here, with the speculation that it might be something wrong from the toolchain side, I tested with gcc-13 but the same issue persists.
martin-frbg commentedon Feb 2, 2025
Docker container running a 32bit OS on the armv8 hardware ?
cdluminate commentedon Feb 2, 2025
It's
schroot(https://wiki.debian.org/Schroot), a kind of chroot. It is also the backend of Debian's official build machines (throughsbuild). I guess running this in docker may get something similar.I'm trying to do a bisect. Will update later.
martin-frbg commentedon Feb 2, 2025
I'm on a train with variable network quality, but from checking file dates and commit logs anything remotely relevant would have happened before 0.3.28.
cdluminate commentedon Feb 2, 2025
My local git bisect result is
By looking at d9f368d, I agree with you that the problem is introduced somewhere in the past.
xmirya commentedon Feb 2, 2025
Might be unrelated, but: OpenBLAS got recently updated to 0.3.29 in FreeBSD ports as well, and i'm experiencing a similar threading/timing test failure building it locally (via FBSD ports infrastructure), but it's
amd64:./sblat2 < ./sblat2.datfails withOMP_NUM_THREADS=2(or any non-1 value), but passes withOMP_NUM_THREADS=1.The compiler is
gcc13, it fails the same with OpenMP on or off, regardless of-Olevel.Another clue is that the official FreeBSD package (from the same port/recipe) is already available for this update, which means the same port/recipe was successfully built by the official FreeBSD build cluster - likely much newer/faster hardware than my local (quite dated) machine.
martin-frbg commentedon Feb 8, 2025
@xmirya probably unrelated given the difference in architectures, but what is your hardware please (the optimized BLAS kernels are different for individual cpu models) ? (SIGBUS instead of simply producing bad results could mean an access to unaligned data where the instruction used requires data alignment)
xmirya commentedon Feb 8, 2025
It's i5-2410M, it has SSE* and AVX, but no AVX2 or higher, the build is done with
added to
Makefile.rules(disabling OpenMP and/or addingNO_AVX=1makes no difference)martin-frbg commentedon Feb 8, 2025
@xmirya that would appear to be a standard Sandy Bridge target - I cannot reproduce the failure on Ryzen5 but can take an actual SandyBridge system out of storage sometime next week
xmirya commentedon Feb 9, 2025
Thanks, would be grateful if you could
martin-frbg commentedon Feb 20, 2025
Can reproduce the ARMV6 issue (on a first generation Asus Tinkerboard, Cortex-A12, so natively ARMV7 building for TARGET=ARMV6). The error only happens in "some" runs of the test, which might explain why it has not come up earlier. Curiously, I have not gotten the corresponding DTBMV test to fail, although both share the same level2 driver.
martin-frbg commentedon Feb 21, 2025
Interestingly, it is only the CBLAS version of the test that fails, making a fault in the test code itself likely - internally, both CBLAS and BLAS version of the call take the exact same code path after reshuffling of the arguments.
xmirya commentedon Feb 24, 2025
Filed this as a separate issue: #5147