Builds timing out on CI

[safern]

We're seeing multiple builds canceled due to "timeouts" when they're not really timeouts. This seems like a bug in AzDo that it is considering the "queue" time and time spent waiting for an agent as part of the timeout time.

##[error]The job running on agent NetCorePublic-Pool 172 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134
Pool: NetCorePublic-Pool
Agent: NetCorePublic-Pool 172
Started: Today at 1:54 PM
Duration: 15m 2s

As seen, the duration was really 15m 2s which is not over 60 mins.

Runtime build

Build	Kind	Timeline Record
721340	PR #38938	CoreCLR Product Build Windows_NT x64 release
721340	PR #38938	CoreCLR Product Build Windows_NT arm64 checked
721340	PR #38938	CoreCLR Product Build Windows_NT x86 release
721425	PR #38952	CoreCLR Product Build Windows_NT x64 release
721425	PR #38952	CoreCLR Product Build Windows_NT arm release
721425	PR #38952	CoreCLR Product Build Windows_NT arm64 release
721425	PR #38952	CoreCLR Product Build Windows_NT x86 release
721535	PR #38765	CoreCLR Product Build OSX x64 checked
721294	PR #38948	Mono Product Build Linux x64 debug
721294	PR #38948	CoreCLR Product Build Linux_musl x64 release
721294	PR #38948	CoreCLR Product Build Linux_musl x64 checked
721294	PR #38948	CoreCLR Product Build Linux x64 release
721294	PR #38948	CoreCLR Product Build Linux arm release
721294	PR #38948	Formatting Linux x64
721294	PR #38948	CoreCLR Product Build FreeBSD x64 release
721294	PR #38948	CoreCLR Product Build Linux arm64 release
721294	PR #38948	CoreCLR Product Build Linux_musl arm64 release
721294	PR #38948	Build Linux arm Debug AllSubsets_Mono
721294	PR #38948	Build Linux x64 Debug AllSubsets_Mono_LLVMJIT
721294	PR #38948	Mono Product Build Linux arm64 debug
721294	PR #38948	Mono Product Build Linux x64 release
721294	PR #38948	Build Linux_musl x64 Release AllSubsets_Mono
721294	PR #38948	Mono Product Build Linux arm64 release
721294	PR #38948	CoreCLR Product Build Linux arm64 checked
721294	PR #38948	CoreCLR Product Build Linux_musl arm64 checked
721294	PR #38948	CoreCLR Product Build Linux x64 checked
721294	PR #38948	CoreCLR Product Build Linux arm checked
721294	PR #38948	Build Linux x64 Release AllSubsets_Mono_LLVMAOT
721099	PR #38707	Mono Product Build Linux x64 debug
721099	PR #38707	CoreCLR Product Build Linux_musl x64 release
721099	PR #38707	CoreCLR Product Build Linux x64 release
721099	PR #38707	CoreCLR Product Build Linux arm release
721099	PR #38707	CoreCLR Product Build FreeBSD x64 release
721099	PR #38707	CoreCLR Product Build Linux arm64 release
721099	PR #38707	CoreCLR Product Build Linux_musl arm64 release
721099	PR #38707	Build Linux arm Debug AllSubsets_Mono
721099	PR #38707	Build Linux x64 Release AllSubsets_Mono_LLVMAOT
721099	PR #38707	Build Linux x64 Debug AllSubsets_Mono_LLVMJIT
721099	PR #38707	Mono Product Build Linux arm64 debug
721099	PR #38707	Mono Product Build Linux x64 release
721099	PR #38707	Build Linux_musl x64 Release AllSubsets_Mono
721099	PR #38707	Mono Product Build Linux arm64 release
721062	PR #38935	CoreCLR Product Build Linux_musl x64 release
721062	PR #38935	CoreCLR Product Build Linux x64 release
721062	PR #38935	Build Linux x64 Release AllSubsets_Mono_LLVMAOT
721160	PR #37971	CoreCLR Product Build Windows_NT x86 checked
721161	PR #38884	CoreCLR Product Build Windows_NT x64 release
721161	PR #38884	CoreCLR Product Build Windows_NT arm release
721161	PR #38884	CoreCLR Product Build Windows_NT x86 release
721161	PR #38884	CoreCLR Product Build Windows_NT arm64 release
721009	PR #38815	Build Linux arm Debug AllSubsets_Mono
721042	PR #38933	CoreCLR Product Build Linux arm release
720484	PR #38893	CoreCLR Product Build OSX x64 release
719881	PR #38728	CoreCLR Product Build Windows_NT x86 checked

Evaluated 100 builds
Impacted 12 bulids
Impacted 54 jobs

This is impacting around 12% of builds of runtime and 12% of runtime-live-build.

Runtime-live-build

Build	Kind	Timeline Record
721591	PR #38547	Build OSX x64 release Runtime_Debug
721451	PR #38894	Build OSX x64 release Runtime_Debug
721456	PR #38616	Build Windows_NT x64 debug Mono_Libraries
721456	PR #38616	Build Windows_NT x86 release Runtime_Debug
721295	PR #38948	Build Linux x64 debug RuntimeFlavor_Mono
721295	PR #38948	Build Linux x64 debug Runtime_Release
721295	PR #38948	Build Linux x64 release SourceBuild
721264	PR #38547	Build Linux x64 debug Runtime_Release
721264	PR #38547	Build Linux x64 release SourceBuild
721270	PR #38946	Build Linux x64 debug RuntimeFlavor_Mono
721270	PR #38946	Build Linux x64 debug Runtime_Release
721241	PR #32520	Build Windows_NT x64 debug Mono_Libraries
721063	PR #38935	Build Windows_NT x64 debug Mono_Libraries
721063	PR #38935	Build Linux x64 debug RuntimeFlavor_Mono
721063	PR #38935	Build Linux x64 debug Runtime_Release
721063	PR #38935	Build Linux x64 release SourceBuild
721100	PR #38707	Build Linux x64 debug RuntimeFlavor_Mono
721100	PR #38707	Build Linux x64 debug Runtime_Release
721043	PR #38933	Build Linux x64 release SourceBuild
721039	PR #37159	Build Linux x64 debug Runtime_Release
720680	PR #38917	Build OSX x64 release Runtime_Debug

Evaluated 100 builds
Impacted 12 bulids
Impacted 21 jobs

@MattGal is looking into this and opening a ticket on AzDo

cc: @dotnet/runtime-infrastructure

Tagging subscribers to this area: @ViktorHofer
Notify danmosemsft if you want to be subscribed.

[safern]

FYI: @stephentoub @jkotas

[safern]

I think I haven't seen this anymore. Closing