BLAS : Program is Terminated. Because you tried to allocate too many memory regions.

I used openblas for blas/lapack functions in the erkale project, and it fails. erkale's author says that openblas is broken, see susilehtola/erkale#29 (comment)

Well, you know, he may be right. But it would certainly help if we knew the version of OpenBLAS that you are currently using, and a bit more about what erkale does. I guess erkale itself is multithreading and thus calling into OpenBLAS from multiple threads, which causes problems that I am currently trying to fix.

openblas-0.2.20_3,1 on FreeBSD
erkale experiences this problem when run with multithreading using OpenMP.

Could you try with current develop branch ("soon" to be 0.3.4) please ? Apart from a number of fixes, that one has a new compile-time parameter NUM_PARALLEL to reduce the risk of running out of unique thread pointers in what looks like your use case.

Alternatively rebuild package with OpenMP that should be more moderate in openmp program and not try to spawn n^2 threads

erkale is built with both OpenMP and OpenBLAS.
I'm trying to fix test failures in its parallel version.

The message

Because you tried to allocate too many memory regions.

begs the question "How many regions were allocated?"

Also with the ever increasing computing power, what does "too many" really mean? Why is this limitation imposed?

It is a fixed table of regions.... 1-2 are consumed per parallel thread
what was built into system package derived from NUM_THREADS at build time got exceeded.
You can throw in any value you prefer where recent improvement was made:
#1858

Why can't you reallocate it dynamically when exceeded instead of fixing it once and for good during build?

This limit is directly related to the NUM_THREADS parameter set at build time (which defaults to the number of cores detected on the build host ). There has been a recent attempt to rewrite the memory allocation logic (that dates back to K. Goto's original libGotoBLAS of 10+ years ago) using thread-local storage. Unfortunately the reimplementation met a number of unexpected corner cases and it is unclear if it is safe to use as the default in its current state. See option USE_TLS in current develop.

Indeed OpenBLAS FreeBSD port would be built with 16 malloc slots only.
https://svnweb.freebsd.org/ports/head/math/openblas/Makefile?view=markup#l61
PR above is meant to plainly mask such cases so that ages old limitation does not hurt every other user.
I'd recommend to change package makefile with like 64 threads (128 slots), and use OpenMP , since you wrap library in OpenMP and OMP openblas reduces threading if called in OMP parallel section.

I'll change the limit in the port for now.

But no matter what the limit value would be set, this problem will come back because the number of threads shouldn't even in theory be tied to the number of CPUs in general (threads can be half-idle for example). This needs to be solved.

math/openblas: Increase the limit for the number of threads to unbrea…

@yurivict

@yurivict
I think you got the message into right ears.

You are wrong about number of threads. The constraining resource here is CPU cache, OpenBLAS( or MKL for that sake) would operate on limited amount of data , fitting in L1d/L2/L3/L4 caches. Obvious if 2 threads of a kind meet on same core they go with 10-20x slower memory accesses from main memory and performance goes 10x down.
What is wrong here is that number of memory buffers is compiled in, and bound to build CPUs, and hurts people oversubscribing CPU cores (I mean caches)

You assume that all threads are CPU-intense. But some threads might be idle. Some might work on separate data sets while using only 10% of CPU each. Some people create threads per connection, etc. All sorts of use models can take place.

I see now that OPENMP isn't a default option in the port, changes that I made only apply to the non-default case. I'll move them to be for all versions.

@martin-frbg

@martin-frbg
dgetrf zherk dsyrk are not guarded against early threading.
Ignore me if I dont produce PRs today.
@yurivict
Following changes will need to go to 0.3.4 on top of what we did here:
DYNAMIC_OLDER=1 should supplement amd64/x86_64 DYNAMIC_ARCH=1

No more need for -frecursive, it is now in right place.

There will be AVX-512 (Skylake-X) support , both FreeBSD clang and gcc can compile it, so new option for that(?).

In principle it builds with clang+flang(once later works) too, if you want to experiment in other side of OMP world, but not required at all.

I think we cannot improve anything here, but feel free to report if you stumble on anything similarily weird.

dgetrf zherk dsyrk are not guarded against early threading.
Ignore me if I dont produce PRs today.

Could you create a separate issue for that please (I assume with "early threading" you mean inefficient multithreading for tiny problem sizes (and not something leading to catastrophic failure), but I am guaranteed to lose my mind if I try to look into that today).

@yurivict

@yurivict (not related to current issue at all) is it possible to get to FreeBSD something like linux pax-utils, i.e. lddtree to find 2 distinct OMP imports and symtree to quickly list imported functions per library?

@brada4

@brada4 Is it this package: https://www.freshports.org/sysutils/pax-utils ?

FYI You can use Repology website to search for packages by name in different systems: https://repology.org/

Installed, thanks :-)

math/openblas: Use options FCOMMON_OPT=-frecursive MAKE_NB_JOBS=-1 fo…

Closing as the crucial change, adding -frecursive to the gfortran options, was released with 0.3.4 already.

@brada4

@brada4 Is there any 'harm' on increasing the number of memory buffers to large number?

From the memory.c:

local_memory_table = (struct alloc_t **)malloc(sizeof(struct alloc_t *) * NUM_BUFFERS);
memset(local_memory_table, 0, sizeof(struct alloc_t *) * NUM_BUFFERS);

It seems that it's just 64 * NUM_BUFFERS bytes in memory.

From the discussions above, I can't quite relate the local_memory_table with fitting data into CPU cache. Is there any harm to have a very large NUMBER_THREADS in build time, say 4,096 for a 96 core CPU, and call OpenBLAS in 128 concurrent threads? In such way, our program won't be terminated with an outburst of number of concurrent threads.

Very bad to steal closed unrelated thread...
Open a new one if you want to discuss what is not clear from discussion in(unrelated to this) #1858

Navigation Menu

Repository navigation

BLAS : Program is Terminated. Because you tried to allocate too many memory regions. #1882

41 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

BLAS : Program is Terminated. Because you tried to allocate too many memory regions. #1882

Description

Activity

martin-frbg commented on Nov 22, 2018

yurivict commented on Nov 22, 2018

martin-frbg commented on Nov 22, 2018

brada4 commented on Nov 22, 2018

yurivict commented on Nov 22, 2018

yurivict commented on Nov 22, 2018

brada4 commented on Nov 22, 2018

yurivict commented on Nov 22, 2018

martin-frbg commented on Nov 22, 2018

brada4 commented on Nov 22, 2018

yurivict commented on Nov 23, 2018

brada4 commented on Nov 23, 2018

yurivict commented on Nov 23, 2018

41 remaining items

yurivict commented on Nov 26, 2018

brada4 commented on Nov 26, 2018

martin-frbg commented on Nov 26, 2018

brada4 commented on Nov 26, 2018

yurivict commented on Nov 26, 2018

yurivict commented on Nov 26, 2018

brada4 commented on Nov 26, 2018

martin-frbg commented on Dec 30, 2018

zhilians commented on Jan 3, 2019

brada4 commented on Jan 3, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions