Performance PR tests failing with "Could not find/download: ".NET Core SDK" with version = 5.0.100-preview.1.20125.8".

[sandreenko]

Error message:

[2020/02/25 22:26:10][INFO] dotnet-install: Downloading legacy link: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.1.20125.8/dotnet-dev-win-x86.5.0.100-preview.1.20125.8.zip
[2020/02/25 22:26:13][INFO] dotnet-install: Cannot download: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.1.20125.8/dotnet-dev-win-x86.5.0.100-preview.1.20125.8.zip
[2020/02/25 22:26:13][INFO] Could not find/download: ".NET Core SDK" with version = 5.0.100-preview.1.20125.8

Examples:
https://dev.azure.com/dnceng/public/_build/results?buildId=536416
https://dev.azure.com/dnceng/public/_build/results?buildId=536459

Looks like that blob was deleted, do we need to use a newer one?

@dotnet/runtime-infrastructure @dotnet/dotnet-perf

[brianrob]

@DrewScoggins can you please take a look?
cc: @billwert

[billwert]

We don't specify a blob when we run these tests - we rely on the dotnet-install.ps1/sh scripts to install the latest SDK. This failure indicates there was something wrong with that path - either a blob got deleted or there was a connectivity issue.

This issue seems to have passed - later runs are moving past this point. (We are hitting other transient nuget failures, however, like this.)

@dotnet/runtime-infrastructure Is there a "lkg" SDK package of .NET 5 we can use? We don't necessarily need to rely on the absolute latest build out of core-sdk.

[dagood]

For context, the logs above are missing the actual links, the first two:

[2020/02/25 22:32:08][INFO] dotnet-install: Downloading link: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.1.20125.8/dotnet-sdk-5.0.100-preview.1.20125.8-win-x86.zip
[2020/02/25 22:32:11][INFO] dotnet-install: Cannot download: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.1.20125.8/dotnet-sdk-5.0.100-preview.1.20125.8-win-x86.zip
[2020/02/25 22:32:11][INFO] dotnet-install: Downloading legacy link: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.1.20125.8/dotnet-dev-win-x86.5.0.100-preview.1.20125.8.zip
[2020/02/25 22:32:13][INFO] dotnet-install: Cannot download: https://dotnetcli.azureedge.net/dotnet/Sdk/5.0.100-preview.1.20125.8/dotnet-dev-win-x86.5.0.100-preview.1.20125.8.zip
[2020/02/25 22:32:13][INFO] Could not find/download: ".NET Core SDK" with version = 5.0.100-preview.1.20125.8

5.0.100-preview.1.20125.8 is this build, 20200225.8:
https://dev.azure.com/dnceng/internal/_build/results?buildId=536364&view=results

It got partway through publishing and failed, which I imagine can cause problems like this.

Related: dotnet/arcade#4922 /cc @mmitche

It doesn't look like Core-SDK is set up with proper per-channel custom build step architecture to prevent partial builds from being marked latest, but maybe there's something in the infra I'm missing.

Something interesting that I don't think is directly connected: you're getting a 5.0 p1 build, not 5.0 p2 as you'd expect from a master branch. It looks like the p1 branch is configured incorrectly, still publishing to the master channel: https://github.com/dotnet/core-sdk/blob/release/5.0.1xx-preview1/src/CopyToLatest/targets/BranchInfo.props (perma)

[mmitche]

I ran a second build that succeeded in publishing. It looks like the main issue should be solved by Cesar's recent changes that retry nuget pushes.

[DrewScoggins]

That last four jobs to finish in the dotnet-runtime-perf pipeline have completed successfully. They all took builds of the CLI that were newer than the broken 20125.8 build. Given that I feel like we can close this issue, or at the very least remove the blocking-clean-ci tag.

I have opened an issue on our side to see what we can do to harden our infra against failures in dotnet-install.

Also @dagood you mentioned

It doesn't look like Core-SDK is set up with proper per-channel custom build step architecture to prevent partial builds from being marked latest, but maybe there's something in the infra I'm missing.

Is this something that we are tracking anywhere, and at the very least are the core-sdk folks aware of this?

[dagood]

There's an issue open at dotnet/installer#6572 where I copied that text, but I'm not aware of any more in depth tracking.

[DrewScoggins]

OK, going to close this one for now as we are not blocking anymore.