Optimizing for POWER8 on big-endian

We're preparing an update of OpenBLAS port in FreeBSD (to 0.3.7) and I'm doing tests on POWER. Previously, we optimized for PPC970 with an option to optimize for POWER6. Now I also tested newer CPU's (they didn't work before). POWER9 still fails, but it looks like OpenBLAS built for POWER8 passes all tests. This is all done on big-endian.

Can I conclude that it's safe to optimize for POWER8 on big-endian variant?

From my latest tests Power9 passes blas-tester for little endian. Just I should lookat lapack failures.
I think its good idea to start with power8 blas 1/2 which is the same on power9

POWER9 probably passes on LE, since this is where most attention is at, but FreeBSD is currently BE-only. If POWER9 works as well, even better, but it currently fails.

pkubaj where do you get failures with big-endian POWER9 ? There were some recent (post-0.3.7) fixes (#2233, #2263,#2269) for getting the POWER9 code compiled correctly with both gcc 9.2 and various older versions of gcc and binutils but I am not sure if anyone looked for potential issues with endianness yet.

This test fails when optimizing for POWER9 on BE:

TEST 21/23 potrf:smoketest_trivial [FAIL]
  ERR: test_potrs.c:513  L s(0,0) difference: 1.7

Other than this one, there may be other failing later.

@quickwritereader

I suspect this may actually be SGEMM failing (test calls SPORTF followed by SGEMM then compares against initial input), there will probably be corresponding failures in the lapack-derived test/ctest as well. Unfortunately my current OpenPower host does not appear to provide big-endian POWER9, @quickwritereader does yours ?

@quickwritereader

I suspect this may actually be SGEMM failing (test calls SPORTF followed by SGEMM then compares against initial input), there will probably be corresponding failures in the lapack-derived test/ctest as well. Unfortunately my current OpenPower host does not appear to provide big-endian POWER9, @quickwritereader does yours ?

If you have a bare-metal host, you can create VM's in any endianness on POWER. So you can create a BE VM on LE host.

If that's not possible, I can give you SSH access to my host (just note that it runs FreeBSD).

I did not consider endiannes as my goal was little endian. For that one should change swap masks and and also check save macroses. Eapecially with single precisions.

@martin-frbg

@martin-frbg I think we can request big endian one. But the current one is a little endian

Right now I see my recent fixes with the gcc7-generated assembly backfiring on power8be - at the very least the assembler is not going to accept the .localentry and .size statements generated by the little-endian compiler...

I think you don't actually mean endianess here, but ABI version (ELFv2). This concept is often mixed even by IBM.
They seem to consider big-endian only ELFv1 and little-endian only ELFv2. In fact this is only an ABI and nothing prevents you from creating big-endian system with ABI ELFv2 or little-endian ELFv1.

Creating little-endian OS with ELFv1 is pointless nowadays, but OpenSUSE had a LE ELFv1 variant a few years ago, when ELFv2 was only starting.

However, there are actively developed systems that are BE for POWER and use ELFv2 ABI, e.g. Adelie Linux (BE exclusively on PPC) or Void Linux (it has variants for both LE and BE, but all are ELFv2).
FreeBSD is currently ELFv1, but it's switching to ELFv2 soon and it will stay on BE. I tested both 0.3.7 and git from develop branch optimized for POWER8 and they seem to build and pass tests just fine on my experimental system with ELFv2.

After your message, I also tested develop on ELFv1 system and it fails to build, but 0.3.7 builds there just fine (for POWER8, I use GCC 9.2), but fails with tests:

TEST 23/23 kernel_regress:skx_avx [FAIL]
  ERR: test_kernel_regress.c:49  expected 0.000e+00, got 6.694e+02 (diff -6.694e+02, tol 1.000e-10)

If there are no plans to fix it I think we'll just offer an option to optimize for POWER8 only on ELFv2.

I also think that you shouldn't "fix" compiler problems with such workarounds. OpenBLAS users probably use the latest compilers anyway to squeeze more performance and GCC 9 works.

What I wrote may be a little unclear, so a short summary (all variants are BE):
ELFv1, 0.3.7: works with POWER6, doesn't work with POWER8 (test failure), POWER9,
ELFv1: develop: works with POWER6, doesn't work with POWER8 (assembly failure you highligted) and POWER9,
ELFv2, 0.3.7: works with POWER6, POWER8, doesn't work with POWER9,
ELFv2, develop: works with POWER6, POWER8, doesn't work with POWER9,

So POWER6 works everywhere, POWER9 doesn't work everywhere, POWER8 works on ELFv2.

Thanks for the explanation - ELFv1 it is then (and I am checking if the problem can be papered over with a few #if _CALL_ELF ==2). Not that I was confusing concepts here, I simply did not know any better - my only previous encounter with PPC was with the venerable RS/6000 "some" years ago.

AFAIK the proper check is #if defined(_CALL_ELF) && (_CALL_ELF == 2), because a compiler might only support ELFv1 (and not define _CALL_ELF, because when this compiler was released, there was no ELFv2).

@pkubaj

Right, but I wanted to try first if my trivial fix was sufficient - unfortunately it is not (segfaults in all the min/max/axpy functions that I tampered with to work around miscompilation by older gcc). Guess I will either have to try and merge the differences seen in the respective gcc-generated assembly file for big-endian, or revert my kludgy fix for #2254 before a 0.3.8 release.
Unfortunately - and this is relevant to 0.3.7 as well - running make lapack-test on big-endian POWER8 shows 1199 cases of numerical error for double precision real and 526 for double-precision complex (in addition to the "familiar" one error each in single precision real&complex that is most likely spurious). This is with an OpenSUSE "tumbleweed" rolling release containing gcc 9.2.1, glibc 2.30.
@pkubaj have you checked the results of make lapack-test (i.e. running the testsuite from the netlib reference version of LAPACK) ? (The tests for complex try to put a huge array of test data on the stack, so you may need to increase the default stack size limit before running them)

I have checked it now:

                        -->   LAPACK TESTING SUMMARY  <--
SUMMARY                 nb test run     numerical error         other error
================        ===========     =================       ================
REAL                    1284869         1       (0.000%)        5       (0.000%)
DOUBLE PRECISION        1088441         1182    (0.109%)        5       (0.000%)
COMPLEX                 751052          1       (0.000%)        10      (0.001%)
COMPLEX16               554591          558     (0.101%)        15      (0.003%)

--> ALL PRECISIONS      3678953         1742    (0.047%)        35      (0.001%)

I also did those tests for POWER6:

                        -->   LAPACK TESTING SUMMARY  <--
SUMMARY                 nb test run     numerical error         other error
================        ===========     =================       ================
REAL                    1284869         1       (0.000%)        1       (0.000%)
DOUBLE PRECISION        1293457         0       (0.000%)        1       (0.000%)
COMPLEX                 741512          7       (0.001%)        2       (0.000%)
COMPLEX16               748336          4       (0.001%)        2       (0.000%)

--> ALL PRECISIONS      4068174         12      (0.000%)        6       (0.000%)

So yeah, I think we'll just stay with PPC970 and option for POWER6.

Thanks for help!

Instead of reducing gcc optimization to O1 globally, it is sufficient to add a pragma to caxpy.c and disable the assembly microkernel in zdot.c.

Sorry for the long lag, I was busy with other things.

What do you mean by disabling assembly microkernel in zdot.c? https://github.com/xianyi/OpenBLAS/blob/develop/kernel/power/zdot.c has no assembly and https://github.com/xianyi/OpenBLAS/blob/develop/kernel/power/KERNEL.POWER8 doesn't list zdot.S.

zdot.c starts with an ifdef POWER8 (etc). ... include zdot_microk_power8.c - and that microk(ernel) file is where the inline assembly resides. BTW this is a pretty common motif in the kernels written by Werner Saar.

BTW, regarding those XERBLA test errors. By "apparently we are just doing a little better than expected with some iterative algorithm", do you mean those errors are safe to ignore? It would be nice to get this issue closed :)

I am guessing that they are safe to ignore, unfortunately there are no clear guidelines for how to assess testsuite failures (and the testsuite is not free from bugs itself). And if what I wrote above is still correct, reducing the gcc optimization level for caxpy and disabling the zdot microkernel took care of these anyway. Unfortunately my minicloud account seems to have expired in the meantime, and the gcc compile farm does not offer this combination of HW and OS, so it may take me a few days to get back to this. If you can test, adding

#if defined(__FreeBSD__)
#if defined(_CALL_ELF) && (_CALL_ELF == 2)
#pragma GCC optimize "O1"
#endif
#endif

at the start of caxpy.c and adding something like
#if _CALL_ELF != 2 or #if !defined(__FreeBSD__) to the mix of ifdefs around the microkernel include in zdot.c was what I had in mind.

I patched those files:

--- kernel/power/caxpy.c.orig   2020-06-14 20:03:04 UTC
+++ kernel/power/caxpy.c
@@ -24,6 +24,13 @@ CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONT
 OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
 USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 *****************************************************************************/
+
+#if defined(__FreeBSD__)
+#if defined(_CALL_ELF) && (_CALL_ELF == 2)
+#pragma GCC optimize "O1"
+#endif
+#endif
+
 #include "common.h"
 #ifndef HAVE_ASM_KERNEL
 #include <altivec.h>
--- kernel/power/zdot.c.orig    2020-08-24 14:36:15 UTC
+++ kernel/power/zdot.c
@@ -36,7 +36,7 @@ USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILI
 #include "common.h"


-#if defined(POWER8) || defined(POWER9)
+#if (defined(POWER8) || defined(POWER9)) && !defined(__FreeBSD__)
 #include "zdot_microk_power8.c"
 #endif

But still experience the same issue (that's on ELFv2):

                        -->   LAPACK TESTING SUMMARY  <--
SUMMARY                 nb test run     numerical error         other error
================        ===========     =================       ================
REAL                    1300419         1       (0.000%)        4       (0.000%)
DOUBLE PRECISION        1309007         0       (0.000%)        4       (0.000%)
COMPLEX                 768366          0       (0.000%)        8       (0.001%)
COMPLEX16               769178          0       (0.000%)        8       (0.001%)

--> ALL PRECISIONS      4146970         1       (0.000%)        24      (0.001%)

Since those failures are probably not an issue, I guess it's ok to close this.

Which gcc ? ISTR that patch was needed to work around a ton of "other error"s seen with 10.1

Ah, ok. I was still on 9.3.0. I'll test with 10.1.0.

Hm, same issue after applying that patch:

                        -->   LAPACK TESTING SUMMARY  <--
SUMMARY                 nb test run     numerical error         other error
================        ===========     =================       ================
REAL                    1300419         1       (0.000%)        4       (0.000%)
DOUBLE PRECISION        1309007         0       (0.000%)        4       (0.000%)
COMPLEX                 768366          0       (0.000%)        8       (0.001%)
COMPLEX16               769178          0       (0.000%)        8       (0.001%)

--> ALL PRECISIONS      4146970         1       (0.000%)        24      (0.001%)

Probably. What I meant was that it looked a lot worse with 10.1 before that patch.

Regained minicloud access, the result I get for the current develop branch (i.e. without patching caxpy and zdot) on POWER8 with FreeBSD13-CURRENT and now gcc-10.2 is

                        -->   LAPACK TESTING SUMMARY  <--
SUMMARY                 nb test run     numerical error         other error
================        ===========     =================       ================
REAL                    1300419         1       (0.000%)        0       (0.000%)
DOUBLE PRECISION        1309007         0       (0.000%)        0       (0.000%)
COMPLEX                 766602          1       (0.000%)        0       (0.000%)
COMPLEX16               769178          0       (0.000%)        0       (0.000%)

--> ALL PRECISIONS      4145206         2       (0.000%)        0       (0.000%)

(so interestingly no "other errors" counted but still about 1700 fewer COMPLEX tests completed compared to your latest results - incidentally exactly the same number as you saw on Jun 15 with gcc9.3 - and later with 10.1)
Adding the pragma GCC optimize "O1" to caxpy.c brought the COMPLEX count back to 768366 and removed the single numerical error, still without "other" errors there. (The zdot part of the patch had zero impact with gcc-10.2)

Thanks, I guess it's something about my environment then. I'll soon post an update to 0.3.10 for FreeBSD and add an option to use POWER8 kernels. Thanks!

Navigation Menu

Repository navigation

Optimizing for POWER8 on big-endian #2299

48 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Optimizing for POWER8 on big-endian #2299

Description

Activity

quickwritereader commented on Oct 31, 2019

pkubaj commented on Oct 31, 2019

martin-frbg commented on Oct 31, 2019

pkubaj commented on Oct 31, 2019

martin-frbg commented on Oct 31, 2019

pkubaj commented on Oct 31, 2019

quickwritereader commented on Oct 31, 2019

quickwritereader commented on Oct 31, 2019

martin-frbg commented on Oct 31, 2019

pkubaj commented on Oct 31, 2019

pkubaj commented on Oct 31, 2019

martin-frbg commented on Oct 31, 2019

pkubaj commented on Oct 31, 2019

martin-frbg commented on Nov 1, 2019

pkubaj commented on Nov 1, 2019

48 remaining items

martin-frbg commented on Jun 28, 2020

pkubaj commented on Aug 23, 2020

martin-frbg commented on Aug 23, 2020

pkubaj commented on Aug 23, 2020

martin-frbg commented on Aug 24, 2020

pkubaj commented on Aug 25, 2020

martin-frbg commented on Aug 25, 2020

pkubaj commented on Aug 25, 2020

pkubaj commented on Aug 26, 2020

martin-frbg commented on Aug 27, 2020

martin-frbg commented on Sep 3, 2020

pkubaj commented on Sep 4, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions