Skip to content

Performance of dcopy() #45

New issue
New issue
Closed
Closed
Performance of dcopy()#45
Labels
Feature request

Description

@ViralBShah
ViralBShah
opened on Jul 16, 2011

I notice that the performance of DCOPY for copying large vectors with stride 1 (size 1000000) is considerably slower than that of memcpy. Is this a known issue?

Activity

  1. xianyi commented on Jul 16, 2011

    @xianyi
    xianyi
    on Jul 16, 2011
    Collaborator

    Hi

    What's your platform?

    Thanks

    Xianyi

    2011/7/16 ViralBShah <
    reply@reply.github.com>

    I notice that the performance of DCOPY for copying large vectors with
    stride 1 (size 1000000) is considerably slower than that of memcpy. Is this
    a known issue?

    Reply to this email directly or view it on GitHub:
    #45

  2. ViralBShah commented on Jul 16, 2011

    @ViralBShah
    ViralBShah
    on Jul 16, 2011
    ContributorAuthor

    Mac, Intel Core 2 Duo.

    -viral

    On Jul 16, 2011, at 2:14 PM, xianyi wrote:

    Hi

    What's your platform?

    Thanks

    Xianyi

    2011/7/16 ViralBShah <
    reply@reply.github.com>

    I notice that the performance of DCOPY for copying large vectors with
    stride 1 (size 1000000) is considerably slower than that of memcpy. Is this
    a known issue?

    Reply to this email directly or view it on GitHub:
    #45

    Reply to this email directly or view it on GitHub:
    #45 (comment)

  3. ViralBShah commented on Jul 16, 2011

    @ViralBShah
    ViralBShah
    on Jul 16, 2011
    ContributorAuthor

    BTW, I observe the same on Opteron+Linux also.

    -viral

    On Jul 16, 2011, at 2:14 PM, xianyi wrote:

    Hi

    What's your platform?

    Thanks

    Xianyi

    2011/7/16 ViralBShah <
    reply@reply.github.com>

    I notice that the performance of DCOPY for copying large vectors with
    stride 1 (size 1000000) is considerably slower than that of memcpy. Is this
    a known issue?

    Reply to this email directly or view it on GitHub:
    #45

    Reply to this email directly or view it on GitHub:
    #45 (comment)

  4. ViralBShah commented on Jul 16, 2011

    @ViralBShah
    ViralBShah
    on Jul 16, 2011
    ContributorAuthor

    Would it help if I provide a standalone C program?

    Currently, I have noticed this in a matlab-like programming environment that we are developing.

    -viral

    On Jul 16, 2011, at 2:14 PM, xianyi wrote:

    Hi

    What's your platform?

    Thanks

    Xianyi

    2011/7/16 ViralBShah <
    reply@reply.github.com>

    I notice that the performance of DCOPY for copying large vectors with
    stride 1 (size 1000000) is considerably slower than that of memcpy. Is this
    a known issue?

    Reply to this email directly or view it on GitHub:
    #45

    Reply to this email directly or view it on GitHub:
    #45 (comment)

  5. xianyi commented on Jul 16, 2011

    @xianyi
    xianyi
    on Jul 16, 2011
    Collaborator

    Hi viral,

    Thank you for the report.Could you give me the performance result on your
    platform?
    I didn't modify the codes in dcopy function. I wonder the copy kernel has
    some optimizing rooms with stride 1 case.

    In addition, I knew a staff from Intel optimized memcpy function in glibc on
    Intel CPU.

    Thank you again.
    Xianyi

    2011/7/16 ViralBShah <
    reply@reply.github.com>

    BTW, I observe the same on Opteron+Linux also.

    -viral

    On Jul 16, 2011, at 2:14 PM, xianyi wrote:

    Hi

    What's your platform?

    Thanks

    Xianyi

    2011/7/16 ViralBShah <
    reply@reply.github.com>

    I notice that the performance of DCOPY for copying large vectors with
    stride 1 (size 1000000) is considerably slower than that of memcpy. Is
    this
    a known issue?

    Reply to this email directly or view it on GitHub:
    #45

    Reply to this email directly or view it on GitHub:
    #45 (comment)

    Reply to this email directly or view it on GitHub:
    #45 (comment)

  6. xianyi commented on Jul 16, 2011

    @xianyi
    xianyi
    on Jul 16, 2011
    Collaborator

    It's great to provide the C program. :)
    We need it to reproduce the problem.

    Thanks

    Xianyi Zhang

    2011/7/16 ViralBShah <
    reply@reply.github.com>

    Would it help if I provide a standalone C program?

    Currently, I have noticed this in a matlab-like programming environment
    that we are developing.

    -viral

    On Jul 16, 2011, at 2:14 PM, xianyi wrote:

    Hi

    What's your platform?

    Thanks

    Xianyi

    2011/7/16 ViralBShah <
    reply@reply.github.com>

    I notice that the performance of DCOPY for copying large vectors with
    stride 1 (size 1000000) is considerably slower than that of memcpy. Is
    this
    a known issue?

    Reply to this email directly or view it on GitHub:
    #45

    Reply to this email directly or view it on GitHub:
    #45 (comment)

    Reply to this email directly or view it on GitHub:
    #45 (comment)

  7. ViralBShah commented on Jul 16, 2011

    @ViralBShah
    ViralBShah
    on Jul 16, 2011
    ContributorAuthor

    Hi,

    On Mac OS X with Intel Core 2 Duo 2.1GHz (penryn), dcopy is much slower:
    $ ./copy
    memcpy: 3.531213 seconds
    dcopy: 5.212013 seconds

    On Linux with Opteron 2.4GHz, dcopy is much faster than glibc memcpy:

    $ ./copy
    memcpy: 5.210000 seconds
    dcopy: 3.730000 seconds

    Here is how to reproduce:

    #include <stdio.h>
    #include <string.h>
    #include <time.h>

    #define N 1000000
    #define NITER 1000

    int main(int ac, char*av[])
    {
    double *src;
    double *dest;
    double t_memcpy, t_dcopy;
    clock_t start, stop;
    int i, n=N, one=1;

    src = (double *) malloc ((size_t)N*sizeof(double));
dest = (double *) malloc ((size_t)N*sizeof(double));

start = clock();
for (i=0; i<NITER; ++i) {
    memcpy(dest, src, N*sizeof(double));

}
stop = clock();
t_memcpy = (stop-start) / (double)CLOCKS_PER_SEC;
printf("memcpy: %f\n", t_memcpy);

start = clock();
for (i=0; i<NITER; ++i) {
    dcopy_(&n, src, &one, dest, &one);
}
stop = clock();
t_dcopy = (stop-start) / (double)CLOCKS_PER_SEC;
printf("dcopy: %f\n", t_dcopy);

return 0;

    }

    -viral

    On Jul 16, 2011, at 2:34 PM, xianyi wrote:

    It's great to provide the C program. :)
    We need it to reproduce the problem.

    Thanks

    Xianyi Zhang

    2011/7/16 ViralBShah <
    reply@reply.github.com>

    Would it help if I provide a standalone C program?

    Currently, I have noticed this in a matlab-like programming environment
    that we are developing.

    -viral

    On Jul 16, 2011, at 2:14 PM, xianyi wrote:

    Hi

    What's your platform?

    Thanks

    Xianyi

    2011/7/16 ViralBShah <
    reply@reply.github.com>

    I notice that the performance of DCOPY for copying large vectors with
    stride 1 (size 1000000) is considerably slower than that of memcpy. Is
    this
    a known issue?

    Reply to this email directly or view it on GitHub:
    #45

    Reply to this email directly or view it on GitHub:
    #45 (comment)

    Reply to this email directly or view it on GitHub:
    #45 (comment)

    Reply to this email directly or view it on GitHub:
    #45 (comment)

  8. StefanKarpinski commented on Jul 16, 2011

    @StefanKarpinski
    StefanKarpinski
    on Jul 16, 2011
    Contributor

    If I'm correct that this is implemented in dcopyf.f, then it may fix the problem just to call memcpy at line 33 instead of trying to do the fancy unrolled copy. Seems like optimized system memcpy's are beating the Fortran code here.

  9. ViralBShah commented on Jul 17, 2011

    @ViralBShah
    ViralBShah
    on Jul 17, 2011
    ContributorAuthor

    On Opteron Linux, dcopy beats system memcpy in all cases.

    -viral

    On Jul 17, 2011, at 12:01 AM, StefanKarpinski wrote:

    If I'm correct that this is implemented in dcopyf.f, then it may fix the problem just to call memcpy at line 33 instead of trying to do the fancy unrolled copy. Seems like optimized system memcpy's are beating the Fortran code here.

    Reply to this email directly or view it on GitHub:
    #45 (comment)

  10. xianyi commented on Jul 18, 2011

    @xianyi
    xianyi
    on Jul 18, 2011
    Collaborator

    Hi Stefan,

    dcopyf.f is only reference implementation.
    The kernel codes are kernel/x86_64/copy_sse2.S and kernel/x86_64/copy_sse.S

    Thanks

    Xianyi

    2011/7/17 StefanKarpinski <
    reply@reply.github.com>

    If I'm correct that this is implemented in dcopyf.f, then
    it may fix the problem just to call memcpy at line 33
    instead of trying to do the fancy unrolled copy. Seems like optimized system
    memcpy's are beating the Fortran code here.

    Reply to this email directly or view it on GitHub:
    #45 (comment)

  11. ViralBShah commented on Jul 18, 2011

    @ViralBShah
    ViralBShah
    on Jul 18, 2011
    ContributorAuthor

    Which is why I was all the more puzzled why the memcpy was beating dcopy on Mac, even though on linux, dcopy seems to be faster.

    -viral

    On Jul 18, 2011, at 11:18 AM, xianyi wrote:

    Hi Stefan,

    dcopyf.f is only reference implementation.
    The kernel codes are kernel/x86_64/copy_sse2.S and kernel/x86_64/copy_sse.S

    Thanks

    Xianyi

    2011/7/17 StefanKarpinski <
    reply@reply.github.com>

    If I'm correct that this is implemented in dcopyf.f, then
    it may fix the problem just to call memcpy at line 33
    instead of trying to do the fancy unrolled copy. Seems like optimized system
    memcpy's are beating the Fortran code here.

    Reply to this email directly or view it on GitHub:
    #45 (comment)

    Reply to this email directly or view it on GitHub:
    #45 (comment)

  12. xianyi commented on Jul 18, 2011

    @xianyi
    xianyi
    on Jul 18, 2011
    Collaborator

    Hi Viral,
    Could you give me your glibc version ?

    My testbox is Intel X5470 (penryn), Ubuntu 10.10, glibc 2.12.1.

    The outputs of your codes are
    memcpy: 0.880000
    dcopy: 2.540000

    Thanks

    Xianyi

    Which is why I was all the more puzzled why the memcpy was beating dcopy on Mac, even though on linux, dcopy seems to be faster.

    -viral

    On Jul 18, 2011, at 11:18 AM, xianyi wrote:

    Hi Stefan,

    dcopyf.f is only reference implementation.
    The kernel codes are kernel/x86_64/copy_sse2.S and kernel/x86_64/copy_sse.S

    Thanks

    Xianyi

    2011/7/17 StefanKarpinski<
    reply@reply.github.com>

    If I'm correct that this is implemented in dcopyf.f, then
    it may fix the problem just to call memcpy at line 33
    instead of trying to do the fancy unrolled copy. Seems like optimized system
    memcpy's are beating the Fortran code here.

    Reply to this email directly or view it on GitHub:
    #45 (comment)

    Reply to this email directly or view it on GitHub:
    #45 (comment)

  13. ViralBShah commented on Jul 18, 2011

    @ViralBShah
    ViralBShah
    on Jul 18, 2011
    ContributorAuthor

    Ok, so dcopy is much slower on your computer as well. I wonder what's going on.

    On mac, I have 10.8.0 Darwin Kernel. The libc used in there is probably this one:

    http://www.opensource.apple.com/source/Libc/Libc-594.9.5/

    On the linux machine I reported, the glibc is 2.5.

    -viral

    On Jul 18, 2011, at 12:45 PM, xianyi wrote:

    Hi Viral,
    Could you give me your glibc version ?

    My testbox is Intel X5470 (penryn), Ubuntu 10.10, glibc 2.12.1.

    The outputs of your codes are
    memcpy: 0.880000
    dcopy: 2.540000

    Thanks

    Xianyi

    Which is why I was all the more puzzled why the memcpy was beating dcopy on Mac, even though on linux, dcopy seems to be faster.

    -viral

    On Jul 18, 2011, at 11:18 AM, xianyi wrote:

    Hi Stefan,

    dcopyf.f is only reference implementation.
    The kernel codes are kernel/x86_64/copy_sse2.S and kernel/x86_64/copy_sse.S

    Thanks

    Xianyi

    2011/7/17 StefanKarpinski<
    reply@reply.github.com>

    If I'm correct that this is implemented in dcopyf.f, then
    it may fix the problem just to call memcpy at line 33
    instead of trying to do the fancy unrolled copy. Seems like optimized system
    memcpy's are beating the Fortran code here.

    Reply to this email directly or view it on GitHub:
    #45 (comment)

    Reply to this email directly or view it on GitHub:
    #45 (comment)

    Reply to this email directly or view it on GitHub:
    #45 (comment)

  14. ViralBShah commented on Jul 18, 2011

    @ViralBShah
    ViralBShah
    on Jul 18, 2011
    ContributorAuthor

    I should mention that while my Mac experiment was on a macbook, the Opteron experiment is on a 32-core NUMA box, running Centos 5.6.

    -viral

    On Jul 18, 2011, at 12:45 PM, xianyi wrote:

    Hi Viral,
    Could you give me your glibc version ?

    My testbox is Intel X5470 (penryn), Ubuntu 10.10, glibc 2.12.1.

    The outputs of your codes are
    memcpy: 0.880000
    dcopy: 2.540000

    Thanks

    Xianyi

    Which is why I was all the more puzzled why the memcpy was beating dcopy on Mac, even though on linux, dcopy seems to be faster.

    -viral

    On Jul 18, 2011, at 11:18 AM, xianyi wrote:

    Hi Stefan,

    dcopyf.f is only reference implementation.
    The kernel codes are kernel/x86_64/copy_sse2.S and kernel/x86_64/copy_sse.S

    Thanks

    Xianyi

    2011/7/17 StefanKarpinski<
    reply@reply.github.com>

    If I'm correct that this is implemented in dcopyf.f, then
    it may fix the problem just to call memcpy at line 33
    instead of trying to do the fancy unrolled copy. Seems like optimized system
    memcpy's are beating the Fortran code here.

    Reply to this email directly or view it on GitHub:
    #45 (comment)

    Reply to this email directly or view it on GitHub:
    #45 (comment)

    Reply to this email directly or view it on GitHub:
    #45 (comment)

  15. ViralBShah commented on Jul 18, 2011

    @ViralBShah
    ViralBShah
    on Jul 18, 2011
    ContributorAuthor

    Do you have an idea of MKL performance on dcopy, and how it compares?

    -viral

    On Jul 18, 2011, at 12:45 PM, xianyi wrote:

    Hi Viral,
    Could you give me your glibc version ?

    My testbox is Intel X5470 (penryn), Ubuntu 10.10, glibc 2.12.1.

    The outputs of your codes are
    memcpy: 0.880000
    dcopy: 2.540000

    Thanks

    Xianyi

    Which is why I was all the more puzzled why the memcpy was beating dcopy on Mac, even though on linux, dcopy seems to be faster.

    -viral

    On Jul 18, 2011, at 11:18 AM, xianyi wrote:

    Hi Stefan,

    dcopyf.f is only reference implementation.
    The kernel codes are kernel/x86_64/copy_sse2.S and kernel/x86_64/copy_sse.S

    Thanks

    Xianyi

    2011/7/17 StefanKarpinski<
    reply@reply.github.com>

    If I'm correct that this is implemented in dcopyf.f, then
    it may fix the problem just to call memcpy at line 33
    instead of trying to do the fancy unrolled copy. Seems like optimized system
    memcpy's are beating the Fortran code here.

    Reply to this email directly or view it on GitHub:
    #45 (comment)

    Reply to this email directly or view it on GitHub:
    #45 (comment)

    Reply to this email directly or view it on GitHub:
    #45 (comment)

  16. xianyi commented on Jul 19, 2011

    @xianyi
    xianyi
    on Jul 19, 2011
    Collaborator

    Hi viral,

    Intel MKL 10.1.1

    ./test_copy_mkl
    memcpy: 0.870000
    dcopy: 2.550000

    Thanks

    Xianyi

  17. ViralBShah commented on Jul 19, 2011

    @ViralBShah
    ViralBShah
    on Jul 19, 2011
    ContributorAuthor

    Is it the case that dcopy is not used much, and hence nobody has spent time improving its performance?

    -viral

    On Jul 19, 2011, at 1:40 PM, xianyi wrote:

    Hi viral,

    Intel MKL 10.1.1

    ./test_copy_mkl
    memcpy: 0.870000
    dcopy: 2.550000

    Thanks

    Xianyi

    Reply to this email directly or view it on GitHub:
    #45 (comment)

  18. xianyi commented on Jul 20, 2011

    @xianyi
    xianyi
    on Jul 20, 2011
    Collaborator

    Hi viral,

    I didn't test dcopy of atlas or MKL new version. I think the performance of memcpy is very good.

    Xianyi

  19. ViralBShah commented on Jul 20, 2011

    @ViralBShah
    ViralBShah
    on Jul 20, 2011
    ContributorAuthor

    Hi Xianyi,

    The thing is that it is very good on large inputs, but on small inputs, openblas dcopy is very fast. Would be great if we can have the best of both worlds. Or just copy the memcpy source code from BSD into openblas.

    -viral

    On Jul 20, 2011, at 8:14 AM, xianyi wrote:

    Hi viral,

    I didn't test dcopy of atlas or MKL new version. I think the performance of memcpy is very good.

    Xianyi

    Reply to this email directly or view it on GitHub:
    #45 (comment)

  20. samueljohn commented on Aug 24, 2012

    @samueljohn
    samueljohn
    on Aug 24, 2012

    Interesting thread to read. Any updates on this matter?

  21. xianyi commented on Aug 24, 2012

    @xianyi
    xianyi
    on Aug 24, 2012
    Collaborator

    Hi @samueljohn ,

    We are doing some BLAS level 1 and 2 optimizations on Intel Sandy Bridge. For example, we try to improve the performance with Intel AVX instructions.

    Xianyi

  22. samueljohn commented on Aug 24, 2012

    @samueljohn
    samueljohn
    on Aug 24, 2012

    @xianyi cool!
    Since I used BLAS and LAPACK always indirectly via numpy/scipy and now julia, I am not an expert.
    I know as much as level 1 and 2 refers to vector x vector and vector x matrix operations.

    I can build openblas on my sandybridge i7 Mac perfectly (with clang and gfortran-4.7). The question I have is: Are all the passing tests at the end of the build process thorough enough to trust the build I did? I know there is some c and some fortran part, though I don't know which is the critical part, performance wise.

    Doesn't clang normally spit out AVX using code, only if I pass -march=corei7-avx or -march=native or with the explicit -mavx (guessed, was someling like this) switch? I don't see openblas adding these. Homebrew adds -march=native to the CFLAGS. Are you enabling these instruction per defining constants?

  23. xianyi commented on Aug 26, 2012

    @xianyi
    xianyi
    on Aug 26, 2012
    Collaborator

    Hi @samueljohn ,

    Yes, passing the tests means that OpenBLAS is built OK.

    The kernel part is assembly language. We implement the kernel by AVX assembly language on Intel Sandy Bridge. Thus, you don't need to enable "-mavx" on command line.

    Xianyi

  24. samueljohn commented on Aug 26, 2012

    @samueljohn
    samueljohn
    on Aug 26, 2012

    Thanks!

  25. wernsaar
    closed this as completedon Jun 4, 2014
  26. martin-frbg
    added a commit that references this issue on Apr 12, 2020

    Merge pull request #45 from xianyi/develop

    f73391c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Feature request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions