You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Benchmark results analysis for .NET Core 2.1 vs 3.0 #1194
The profile shows that AddScaleSU which should be faster with AVX is actually slower and takes much more time. This can be confirmed running our CpuMathBenchmarks:
module microsoft.ml.transforms <<microsoft.ml.transforms!Microsoft.ML.Runtime.Data.LpNormNormalizerTransform+<>c__DisplayClass30_0.b__5(value class Microsoft.ML.Runtime.Data.VBuffer`1&)>>
Summary: 8% slower. Top method from the diff - SequencePool.GetCore was using same IL, but different runtimes.
TODO: investigate if SequencePool.GetCore produces different asm for 3.0. The method is really hot on the profiles so any perf improvement will be great
Summary: 21% slower. PredictCore calling DotProductDense can't be inlined. The DotU is faster for AVX but we need to make sure we don't pay any extra cost for the call.
I'm curious if you saw the run to run variance when running TrainKMeansAndLR benchmark? I run it on my machine 10 times (.net core 3 and Tiered JIT enabled) and the perf number ranges from 466 ms to 533 ms.
Yes, the current implementation does not do anything to deal with misaligned inputs and will incur a penalty anytime a read crosses a cache-line or page boundary. @Anipik is working on improving some of these algorithms using the template I provided (which will incur at most 2 unaligned reads for the input data).
@adamsitnik how did you get the asm for SequencePool.GetCore , this function is causing regressing in couple of benchmarks ? and any suggestions for improvements ?
@Anipik the best way is to use the Checked CoreCLR build that you have been using recently to check the ASM of the Math helpers. But it might need using the 2.1 branch to test ;(
Activity
adamsitnik commentedon Oct 9, 2018
KMeansAndLogisticRegressionBench.TrainKMeansAndLR
The profile shows that
AddScaleSUwhich should be faster with AVX is actually slower and takes much more time. This can be confirmed running our CpuMathBenchmarks:..\..\Tools\dotnetcli\dotnet.exe run -c Release-Intrinsics -f netcoreapp3.0 -- -F *.AddScaleSU --join -dThis issue has been spotted by @eerhardt in August #691 (comment)
I reported issues #1195 and #1196 which got improved in #1200 but still exist.
Edit: This benchmark is multimodal, I believe that it's very dependent on the alignment.
Summary: 20% slower due to invalid
AddScaleSUAVX implementation.adamsitnik commentedon Oct 9, 2018
MultiClassClassificationTest.Test_Multiclass_WikiDetox_BigramsAndTrichar_OVAAveragedPerceptron
The PerfView diff for time to execute the benchmark once (the positive values mean more time spent in given method):
Summary: 8% slower. Top method from the diff -
SequencePool.GetCorewas using same IL, but different runtimes.TODO: investigate if
SequencePool.GetCoreproduces different asm for 3.0. The method is really hot on the profiles so any perf improvement will be greatadamsitnik commentedon Oct 9, 2018
PredictionEngineBench.MakeIrisPredictions
Summary: 21% slower.
PredictCorecallingDotProductDensecan't be inlined. TheDotUis faster for AVX but we need to make sure we don't pay any extra cost for the call.The issue allready exists #833
helloguo commentedon Oct 9, 2018
I'm curious if you saw the run to run variance when running
TrainKMeansAndLRbenchmark? I run it on my machine 10 times (.net core 3 and Tiered JIT enabled) and the perf number ranges from 466 ms to 533 ms.adamsitnik commentedon Oct 10, 2018
@helloguo the
TrainKMeansAndLRis multimodal, I believe that it's very dependent on the alignment (/cc @tannergooding )tannergooding commentedon Oct 10, 2018
Yes, the current implementation does not do anything to deal with misaligned inputs and will incur a penalty anytime a read crosses a cache-line or page boundary. @Anipik is working on improving some of these algorithms using the template I provided (which will incur at most 2 unaligned reads for the input data).
adamsitnik commentedon Oct 10, 2018
Microsoft.ML.Benchmarks.RankingTrain.TrainTest_Ranking_MSLRWeb10K_RawNumericFeatures_FastTreeRanking
no significant difference
adamsitnik commentedon Oct 10, 2018
Microsoft.ML.Benchmarks.RankingTrain.TrainTest_Ranking_MSLRWeb10K_RawNumericFeatures_LightGBMRanking
no significant difference (the 2% here is noise)
Anipik commentedon Oct 10, 2018
These results are expected for these tests because they don't use anything from CpuMath library
adamsitnik commentedon Oct 10, 2018
good to know (I am still getting familiar with ML.NET), I also wanted to check if ML.NET can get affected by tiered jitting
danmoseley commentedon Oct 29, 2018
It seems next step here is to re-measure after @Anipik is done.
danmoseley commentedon Oct 29, 2018
Thank you for doing this verification @adamsitnik
Anipik commentedon Feb 27, 2019
@adamsitnik how did you get the asm for
SequencePool.GetCore, this function is causing regressing in couple of benchmarks ? and any suggestions for improvements ?adamsitnik commentedon Mar 1, 2019
@Anipik the best way is to use the Checked CoreCLR build that you have been using recently to check the ASM of the Math helpers. But it might need using the 2.1 branch to test ;(