Skip to content

Benchmark results analysis for .NET Core 2.1 vs 3.0 #1194

Closed

Description

@adamsitnik

I took all of our benchmarks and run them for:

  1. .NET Core 2.1 using native CpuMathNative which uses SSE
  2. .NET Core 3.0 using new managed Hardware Intrinsics API which uses AVX with Tiered compilation enabled
  3. .NET Core 3.0 using new managed Hardware Intrinsics API which uses AVX with Tiered compilation disabled

I will add a separate comment for every benchmark in this issue with some analysis.

Environment info (updated):

BenchmarkDotNet=v0.11.1.786-nightly, OS=Windows 10.0.17134.285 (1803/April2018Update/Redstone4)
Intel Xeon CPU E5-1650 v4 3.60GHz, 1 CPU, 12 logical and 6 physical cores
Frequency=3507500 Hz, Resolution=285.1033 ns, Timer=TSC
.NET Core SDK=2.2.100-preview2-009404
  [Host]             : .NET Core 2.1.4 (CoreCLR 4.6.26814.03, CoreFX 4.6.26814.02), 64bit RyuJIT
  Core 2.1           : .NET Core 2.1.4 (CoreCLR 4.6.26814.03, CoreFX 4.6.26814.02), 64bit RyuJIT
  Core 3.0 NonTiered : .NET Core 3.0.0-preview1-27004-04 (CoreCLR 4.6.27003.04, CoreFX 4.6.27003.02), 64bit RyuJIT

Activity

  1. self-assigned this
    on Oct 9, 2018
  2. adamsitnik commented on Oct 9, 2018

    @adamsitnik
    MemberAuthor

    KMeansAndLogisticRegressionBench.TrainKMeansAndLR

    Job Toolchain Mean StdDev Scaled
    Core 2.1 netcoreapp2.1 496.564 ms 70.6073 ms 1.00
    Core 3.0 NonTiered netcoreapp3.0 595.176 ms 112.4640 ms 1.22
    Core 3.0 Tiered netcoreapp3.0 597.032 ms 106.1578 ms 1.22

    The profile shows that AddScaleSU which should be faster with AVX is actually slower and takes much more time. This can be confirmed running our CpuMathBenchmarks:

    ..\..\Tools\dotnetcli\dotnet.exe run -c Release-Intrinsics -f netcoreapp3.0 -- -F *.AddScaleSU --join -d

    Type Method Mean
    AvxPerformanceTests AddScaleSU 4.012 ms
    NativePerformanceTests AddScaleSU 2.966 ms
    SsePerformanceTests AddScaleSU 2.916 ms

    This issue has been spotted by @eerhardt in August #691 (comment)

    I reported issues #1195 and #1196 which got improved in #1200 but still exist.

    Edit: This benchmark is multimodal, I believe that it's very dependent on the alignment.

    Summary: 20% slower due to invalid AddScaleSU AVX implementation.

  3. adamsitnik commented on Oct 9, 2018

    @adamsitnik
    MemberAuthor
    MultiClassClassificationTest.Test_Multiclass_WikiDetox_BigramsAndTrichar_OVAAveragedPerceptron
    Job Mean Error StdDev Median Scaled
    Core 2.1 4,889.920 ms 73.2056 ms 68.4765 ms 4,882.153 ms 1.00
    Core 3.0 NonTiered 5,258.516 ms 99.5758 ms 93.1432 ms 5,253.034 ms 1.08
    Core 3.0 Tiered 5,284.164 ms 73.6200 ms 68.8642 ms 5,273.463 ms 1.08

    The PerfView diff for time to execute the benchmark once (the positive values mean more time spent in given method):

    Name Exc % Exc Inc % Inc
    module microsoft.ml.data <<microsoft.ml.data!SequencePool.GetCore>> 28,1 995 43,9 1 555
    module coreclr <<coreclr!Thread::intermediateThreadProc>> -22,8 -809 98,5 3 489
    LIB <<system.threading.tasks.parallel.il!Parallel.For>> 13,4 475 53,4 1 889
    OS <<kernelbase!SleepEx>> 13,2 466 13,2 466
    module ? <> 12,8 452,319 12,7 451,319
    LIB <<system.threading.tasks.parallel!Parallel.For>> -11,5 -406 -46,3 -1 640
    LIB <<system.private.corelib!System.Collections.Generic.Dictionary`2[System.Char,System.Char].FindEntry(!0)>> 10,8 382 10,9 386
    module microsoft.ml.cpumath <<microsoft.ml.cpumath!CpuMathUtils.DotProductSparse>> 10,8 382 11,0 388
    module microsoft.ml.core <<microsoft.ml.core!ReadOnlyMemoryUtils.FindInPool>> 9,0 319 11,5 408
    LIB <<system.private.corelib!Array.Sort>> 8,7 308 8,1 288
    module microsoft.ml.core <<microsoft.ml.core!Hashing.MurmurHash>> 8,4 296 15,1 533
    module cpumathnative <<cpumathnative!DotSU>> -8,3 -294 -8,4 -296
    module microsoft.ml.transforms <<microsoft.ml.transforms!Microsoft.ML.Runtime.Data.LpNormNormalizerTransform+<>c__DisplayClass30_0.b__5(value class Microsoft.ML.Runtime.Data.VBuffer`1&)>> 8,2 292 91,9 3 253
    module coreclr <<coreclr!ThreadNative::SpinWait>> -4,6 -163 -4,6 -163
    module clrcompression <<clrcompression!inflate>> -4,5 -160 -4,5 -160
    module clrcompression <<clrcompression!CompressionNative_Inflate>> 4,4 156 4,4 156
    module coreclr <<coreclr!JIT_GetSharedNonGCStaticBase_SingleAppDomain>> -4,0 -141 -4,0 -142
    OS <<kernelbase!GetQueuedCompletionStatus>> 4,0 141 4,0 141
    OS <<kernelbase!PostQueuedCompletionStatus>> 2,6 93 2,6 93
    module microsoft.ml.core <<microsoft.ml.core!Microsoft.ML.Runtime.Internal.Utilities.FixedSizeQueue`1[System.UInt32].AddLast(!0)>> 2,3 82 2,4 84
    module coreclr <<coreclr!JIT_NewArr1>> 2,1 76 2,3 81
    OS <<kernelbase!SwitchToThread>> 2,0 72 2,0 72
    module microsoft.ml.core <<microsoft.ml.core!ReadOnlyMemoryUtils.SplitOne>> 2,0 70 3,1 108

    Summary: 8% slower. Top method from the diff - SequencePool.GetCore was using same IL, but different runtimes.

    TODO: investigate if SequencePool.GetCore produces different asm for 3.0. The method is really hot on the profiles so any perf improvement will be great

  4. adamsitnik commented on Oct 9, 2018

    @adamsitnik
    MemberAuthor

    PredictionEngineBench.MakeIrisPredictions

    Job Mean StdDev Scaled
    Core 2.1 2.838 ms 0.0019 ms 1.00
    Core 3.0 NonTiered 3.433 ms 0.0026 ms 1.21
    Core 3.0 Tiered 3.452 ms 0.0017 ms 1.22

    Summary: 21% slower. PredictCore calling DotProductDense can't be inlined. The DotU is faster for AVX but we need to make sure we don't pay any extra cost for the call.

    image

    The issue allready exists #833

  5. helloguo commented on Oct 9, 2018

    @helloguo

    I'm curious if you saw the run to run variance when running TrainKMeansAndLR benchmark? I run it on my machine 10 times (.net core 3 and Tiered JIT enabled) and the perf number ranges from 466 ms to 533 ms.

  6. adamsitnik commented on Oct 10, 2018

    @adamsitnik
    MemberAuthor

    @helloguo the TrainKMeansAndLR is multimodal, I believe that it's very dependent on the alignment (/cc @tannergooding )

  7. tannergooding commented on Oct 10, 2018

    @tannergooding
    Member

    Yes, the current implementation does not do anything to deal with misaligned inputs and will incur a penalty anytime a read crosses a cache-line or page boundary. @Anipik is working on improving some of these algorithms using the template I provided (which will incur at most 2 unaligned reads for the input data).

  8. adamsitnik commented on Oct 10, 2018

    @adamsitnik
    MemberAuthor
    Microsoft.ML.Benchmarks.RankingTrain.TrainTest_Ranking_MSLRWeb10K_RawNumericFeatures_FastTreeRanking
    Job Mean StdDev Scaled
    Core 2.1 30.48 s 0.4394 s 1.00
    Core 3.0 NonTiered 30.36 s 0.6783 s 1.00

    no significant difference

  9. adamsitnik commented on Oct 10, 2018

    @adamsitnik
    MemberAuthor
    Microsoft.ML.Benchmarks.RankingTrain.TrainTest_Ranking_MSLRWeb10K_RawNumericFeatures_LightGBMRanking
    Job Mean StdDev Scaled
    Core 2.1 28.06 s 0.2593 s 1.00
    Core 3.0 NonTiered 28.54 s 0.7388 s 1.02

    no significant difference (the 2% here is noise)

  10. Anipik commented on Oct 10, 2018

    @Anipik
    Contributor
    Microsoft.ML.Benchmarks.RankingTrain.TrainTest_Ranking_MSLRWeb10K_RawNumericFeatures_LightGBMRanking
    Microsoft.ML.Benchmarks.RankingTrain.TrainTest_Ranking_MSLRWeb10K_RawNumericFeatures_FastTreeRanking
    

    These results are expected for these tests because they don't use anything from CpuMath library

  11. adamsitnik commented on Oct 10, 2018

    @adamsitnik
    MemberAuthor

    These results are expected for these tests because they don't use anything from CpuMath library

    good to know (I am still getting familiar with ML.NET), I also wanted to check if ML.NET can get affected by tiered jitting

  12. danmoseley commented on Oct 29, 2018

    @danmoseley

    It seems next step here is to re-measure after @Anipik is done.

  13. danmoseley commented on Oct 29, 2018

    @danmoseley

    Thank you for doing this verification @adamsitnik

  14. Anipik commented on Feb 27, 2019

    @Anipik
    Contributor

    @adamsitnik how did you get the asm for SequencePool.GetCore , this function is causing regressing in couple of benchmarks ? and any suggestions for improvements ?

  15. adamsitnik commented on Mar 1, 2019

    @adamsitnik
    MemberAuthor

    @Anipik the best way is to use the Checked CoreCLR build that you have been using recently to check the ASM of the Math helpers. But it might need using the 2.1 branch to test ;(

  16. ghost locked as resolved and limited conversation to collaborators on Mar 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions