Many Android JIT tests failing in CI

[ViktorHofer]

https://dev.azure.com/dnceng/public/_build/results?buildId=873257&view=ms.vss-test-web.build-test-results-tab

Configuration: Mono Android x64 Release @ Ubuntu.1804.Amd64.Android.Open

@akoeplinger @steveisok is this known?

Runfo Tracking Issue: Many Android JIT tests failing in CI

Build	Definition	Kind	Run Name	Console	Core Dump	Test Results	Run Client
956620	runtime	PR 47182	net6.0-Android-Release-arm64-Mono_Release-Windows.10.Amd64.Android.Open	console.log			runclient.py
945073	runtime	PR 46810	net6.0-Android-Release-arm64-Mono_Release-Windows.10.Amd64.Android.Open	console.log			runclient.py
944893	runtime	PR 46810	net6.0-Android-Release-arm64-Mono_Release-Windows.10.Amd64.Android.Open	console.log			runclient.py

Build Result Summary

Day Hit Count	Week Hit Count	Month Hit Count
0	0	0

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ViktorHofer]

cc @SamMonoRT @vargaz

Tagging subscribers to this area:
See info in area-owners.md if you want to be subscribed.

[ViktorHofer]

@danmosemsft the bot isn't mentioning anyone, intentional?

[steveisok]

So, https://dev.azure.com/dnceng/public/_build/results?buildId=873257&view=ms.vss-test-web.build-test-results-tab is a result of an apk installation failure. That impacted 50+ tests on two runs from #38457

It looks like work items completed on the machine that failed afterwards, so my guess is that adb is in a bad state and may be corrected upon the work item finishing. We can look into making this more resilient.

The other issues linked in by runfo appear to be different. My guess on some would adb corruption occurring at different points in time.

[danmoseley]

@danmosemsft the bot isn't mentioning anyone, intentional?

The owner is @vargaz -- @vargaz do you want notifications for this area label?

[akoeplinger]

@ViktorHofer all of the failures from #43740 are known, we're iterating on replacing OpenSSL on Android there.

[ViktorHofer]

@steveisok what do you think would be a reasonable follow-up action? The day hit count is concerningly high. Assuming that disabling the Android leg in CI is a no go, should we move the job into a different pipeline that doesn't fail the build: #44031?

cc @safern @jkoritzinsky for opinions

[fanyang-mono]

I wonder why this only happened to runtime tests? Theoretically, library tests should be affected as well.

[steveisok]

I'm trying to sort through the recent data first and that may help what a good follow up action will be.