CI flakiness inventory: Aug 11-Sep 8, 2026

[simonrozsival]

Android framework version

net11.0-android (Preview)

Affected platform version

dotnet/android pull-request CI, public Azure DevOps dotnet-android pipeline, 2026-08-11 through 2026-09-08.

Description

This issue tracks flaky infrastructure and flaky tests observed across 242 PR pipeline runs from 2026-08-11 through 2026-09-08. Of these, 241 runs completed (123 red and 118 green); 1 run was still active at collection time.

Frequency is measured as build incidents / 241 completed PR runs and as the average incidents per week over this four-week interval. A build incident is one unique root signature in one build; duplicate configuration annotations are collapsed. Independent roots in the same build are counted separately. Failures determined to be caused by the PR are intentionally omitted from this issue.

The first PR pipeline run returned within the interval started at 2026-08-29 02:03 UTC; the Azure build query returned no PR runs from 2026-08-11 through 2026-08-29 before that run.

Summary: 18 infrastructure signatures / 71 incidents and 10 flaky-test signatures / 32 incidents.

Earlier inventories and focused trackers

• Flaky test analysis: Xamarin.Android-PR pipeline, April 10–24 2026 #11203 - April 10-24 flaky-test inventory.
• Flaky test analysis: Xamarin.Android-PR pipeline, April 27 – May 11 2026 (follow-up to #11203) #11320 - April 27-May 11 follow-up inventory; recorded the improvement after [tests] Skip PerformanceTest on slow CI machines using evaluation time #11249 and [build] Remove WearOS emulator CI lane #11257.
• [tests] test failures on dnceng-public #10832 - earlier broad dnceng-public test-failure tracker.
• CI: MAUI R2R Helix device-lab failures across Android queues #12658 - open MAUI R2R Helix device-lab tracker.
• Flaky JNI reference/peer-count assertions in shared device test process #12031 - open shared-process JNI reference/peer-count tracker.
• JnienvArrayMarshaling.GetObjectArray fails on CoreCLR with emulator 36.4.10 #10973 - GetObjectArray tracker, closed by [typemap] Fix trimmable peer creation race #12650.
• [Flaky test] InstallAndroidDependenciesTest (GoogleV2/Xamarin, CoreCLR) fails intermittently across PRs #11973 - InstallAndroidDependenciesTest tracker.
• [flaky test] FastTimingTests.ConcurrentEventsCanGrowAndDump loses dump completion marker in logcat #12491 - FastTiming logcat race, fixed by [tests] Read FastTiming dump from file #12492.

Recently merged fixes and recurrence

Failure family	Merged change	Evidence after merge	Assessment
GetObjectArray peer identity	#12650 on Sep 3 15:45 UTC	Last failure was Sep 3 09:25 UTC; 0 recurrences in 71 later completed builds. Earlier mitigations #12271 and #12357 had not eliminated it.	Likely fixed.
InstallAndroidDependenciesTest missing SharpZipLib/Polly	#12649 on Sep 3 04:59 UTC	The only current occurrence preceded the merge; 0 recurrences in 107 later completed builds. Earlier work #11974, #12059, and #12580 addressed separate retry/network paths.	Likely fixed.
CheckSignApk stale JAVA_HOME	#12603 on Sep 1 13:54 UTC	The only current occurrence preceded the merge; 0 recurrences in 167 later completed builds.	Likely fixed.
BuildReleaseArm64 apkdiff timeout	#12515 on Sep 1 10:14 UTC	Both real occurrences preceded the change from 30 to 120 seconds; 0 recurrences in 184 completed builds queued after merge. Build 1575474 was a false attribution: the test passed and MAUI Helix caused the red build.	Likely fixed.
BootstrapTasks libMono.Unix path evaluation	#12231 on Jul 27 12:28 UTC	The only occurrence was PR #12611 against a release branch that diverged before the fix. 313 other scannable completed builds after merge had zero exact recurrences.	Fixed on main; release-branch exception.
BindLibrary generated MemorySafetyRulesAttribute compile	#12700 on Sep 7 12:16 UTC, carrying dotnet/roslyn#85131	The only failure was already running with the older SDK when #12700 merged; 0 recurrences in 7 verified completed builds based directly on the fixed SDK merge.	Likely fixed.
MAUI R2R Helix	#12641 on Sep 3 16:00 UTC	The fork-token 401 path was fixed, but direct device-lab failures continued through Sep 8.	Partially fixed; #12658 remains ongoing.
Shared-process JNI GREF/peer assertions	#12032 and #12037 in July	Those PRs disabled two specific assertions. The enabled String overload and JavaSideActivation still failed in this interval.	Ongoing under #12031.
Logcat capture pressure/hangs	#12471 on Aug 21 and #12620 on Sep 2	Capture timeouts still occurred Sep 4 and Sep 7.	Mitigated, not fixed.
Previous inventory: performance and WearOS noise	#11249 and #11257 on May 1	Those failure families are absent from this interval.	Fixes are still holding.
Previous FastTiming logcat race	#12492 on Aug 25	No unrelated-PR recurrence was classified as flaky. One later failure occurred on a PR directly changing FastTiming and was classified PR-related.	Original flake fix appears to be holding.

First-attempt and unchanged-rerun reliability

Unit: one PR plus exact source SHA. Sequences containing any PR-related incident are excluded, so a later build is an unchanged rerun rather than a code-change build.

Outcome	Sequences	Share
Green on first attempt	109	70.3%
- clean green with no recorded flake	91	58.7%
- green with non-gating flake(s)	18	11.6%
Red solely from flakiness	46	29.7%

Of the 46 flake-red first attempts, 32 were infrastructure-only, 8 test-only, and 6 contained both.

Unchanged-rerun metric	Result
Flake-red sequences actually rerun at the same SHA	5 / 46 (10.9%)
Eventually green when rerun unchanged	4 / 5 (80.0%)
Median reruns to green among recovered sequences	1
Mean reruns to green among recovered sequences	1.25

Three recovered on the first rerun; one recovered on the second. One additional unchanged sequence was rerun once but hit another flake and remained red.

Recovered unchanged-SHA chains:

• PR [NativeAOT] Stop linking and shipping libc++ and libunwind #12523 (8e0c17a2): 1578195 failed -> 1578492 succeeded; 1 rerun(s) to green.
• PR [ci] Fix MAUI R2R Helix submissions for fork PRs #12641 (e4dc407b): 1578755 failed -> 1579094 failed -> 1579507 succeeded; 2 rerun(s) to green.
• PR [typemap] Diagnose unsupported export signatures #12597 (d5f41779): 1578760 failed -> 1579125 succeeded; 1 rerun(s) to green.
• PR [native] Remove std::format from the CoreCLR host #12534 (86271701): 1578694 failed -> 1579498 succeeded; 1 rerun(s) to green.

Most flake-red sequences were not rerun unchanged, so they are censored rather than counted as failed recoveries.

Flaky infrastructure

Failure	First seen	Last seen	Frequency	Red / non-gating green	PRs	Evidence
MAUI R2R Helix work item/submission failure	2026-08-30 18:14 UTC (1574227)	2026-09-08 07:04 UTC (1586410)	35 / 241 (14.5%; 8.75/week)	26 / 9	29	Status: Ongoing. Fork authentication was fixed by #12641, but direct device-lab failures continued through Sep 8 and are tracked by #12658. Recurring Helix work-item or submission failures crossed unrelated PRs and sometimes appeared as non-gating failures in green builds. Confidence: high.
Gradle daemon/cache/tooling failure	2026-08-30 18:14 UTC (1574227)	2026-09-02 21:26 UTC (1578693)	11 / 241 (4.6%; 2.75/week)	11 / 0	9	Status: Quiet since Sep 2; no merged fix was identified. Gradle daemons, cache transforms, or shared Java helper builds failed across unrelated PRs; unchanged retries later passed for this family. Confidence: medium.
MAUI restore requested an unpublished Microsoft.Extensions.Logging build	2026-08-31 19:40 UTC (1575484)	2026-09-04 06:10 UTC (1582480)	4 / 241 (1.7%; 1.00/week)	4 / 0	4	Status: Transient publication/feed lag; quiet since Sep 4 without a repository fix. NU1102 requested a dotnet11 package version newer than the nearest version available on the feed. Confidence: high.
Hosted agent disconnected or was deprovisioned	2026-08-31 16:56 UTC (1575255)	2026-09-07 07:14 UTC (1585360)	3 / 241 (1.2%; 0.75/week)	3 / 0	3	Status: Ongoing/recent; last observed Sep 7. Azure stopped hearing from the agent or the remote provider cancelled the machine. Confidence: high.
Azure artifact download had no route to host	2026-09-04 10:50 UTC (1582867)	2026-09-04 17:15 UTC (1583384)	2 / 241 (0.8%; 0.50/week)	2 / 0	2	Status: Transient cluster on Sep 4; no later recurrence in the interval. DownloadPipelineArtifact could not connect to dev.azure.com:443; one occurrence also timed out publishing telemetry. Confidence: high.
Logcat capture task timed out	2026-09-04 15:31 UTC (1583273)	2026-09-07 11:18 UTC (1585745)	2 / 241 (0.8%; 0.50/week)	2 / 0	2	Status: Ongoing/recent. It recurred after #12471 bounded capture and after #12620 reduced reference logcat traffic. One or more post-test capture-logcat tasks hit the task timeout. Confidence: high.
System.NetTests.SslTest.VerifyTrustedCertificates	2026-08-31 09:46 UTC (1574749)	2026-09-03 22:24 UTC (1581249)	2 / 241 (0.8%; 0.50/week)	1 / 1	2	Status: Quiet since Sep 3; no merged fix was identified. Certificate/network validation failed in an unrelated PR and did not gate the otherwise-successful build. Confidence: medium.
dotnet-test-slicer could not find its required .NET runtime	2026-08-31 07:47 UTC (1574631)	2026-08-31 13:40 UTC (1574989)	2 / 241 (0.8%; 0.50/week)	2 / 0	2	Status: Quiet since Aug 31; no repository fix was identified. The hosted macOS image launched dotnet-test-slicer without DOTNET_ROOT or a discoverable runtime. Confidence: high.
ADB install failed with a broken pipe	2026-09-01 04:31 UTC (1576002)	2026-09-01 04:31 UTC (1576002)	1 / 241 (0.4%; 0.25/week)	1 / 0	1	Status: Single transient occurrence on Sep 1. Device installation failed in adb client/server communication rather than in product compilation. Confidence: high.
Android toolchain cache download was rate-limited	2026-09-02 14:03 UTC (1578771)	2026-09-02 14:03 UTC (1578771)	1 / 241 (0.4%; 0.25/week)	1 / 0	1	Status: Single transient HTTP 429 occurrence on Sep 2. The shared cache step failed with HTTP 'Too Many Requests'. Confidence: high.
BootstrapTasks could not locate libMono.Unix.so	2026-09-01 04:45 UTC (1576007)	2026-09-01 04:45 UTC (1576007)	1 / 241 (0.4%; 0.25/week)	1 / 0	1	Status: Fixed by #12231 on main. The only occurrence came from a release branch that diverged before the fix; 313 other scannable completed builds after merge had zero exact recurrences. The release branch lacked #12231, so BootstrapTasks evaluated an empty PkgMono_Unix path and attempted a root-relative native-library copy. Confidence: high.
Git submodule checkout failed with exit code 128	2026-08-29 02:03 UTC (1573456)	2026-08-29 02:03 UTC (1573456)	1 / 241 (0.4%; 0.25/week)	1 / 0	1	Status: Single transient checkout occurrence on Aug 29. Repository checkout failed before product code was built. Confidence: high.
Hosted agent DNS failure while installing the .NET SDK	2026-09-02 07:35 UTC (1578195)	2026-09-02 07:35 UTC (1578195)	1 / 241 (0.4%; 0.25/week)	1 / 0	1	Status: Single transient DNS occurrence on Sep 2. curl exited with code 6 while the SDK installation step resolved its download host; the same SHA later passed. Confidence: high.
Hosted job exceeded the 240-minute limit	2026-09-02 21:21 UTC (1579664)	2026-09-02 21:21 UTC (1579664)	1 / 241 (0.4%; 0.25/week)	1 / 0	1	Status: Single occurrence on Sep 2, downstream of workload download stalls. The macOS job was terminated by Azure after the maximum hosted-job duration. Confidence: high.
NuGet workload-pack downloads timed out	2026-09-02 21:21 UTC (1579664)	2026-09-02 21:21 UTC (1579664)	1 / 241 (0.4%; 0.25/week)	1 / 0	1	Status: Single transient feed/download occurrence on Sep 2. Multiple package downloads received no data for 60 seconds and the operation was cancelled. Confidence: high.
Pipeline failed before any job or timeline record started	2026-09-04 21:12 UTC (1583713)	2026-09-04 21:12 UTC (1583713)	1 / 241 (0.4%; 0.25/week)	1 / 0	1	Status: Single Azure service/startup occurrence on Sep 4. The build began and ended at the same instant, with no GitHub check and no Azure timeline. Confidence: high.
Xamarin.Android.Build.Tests.BuildTest.Desugar(True,"",CoreCLR)	2026-09-01 22:23 UTC (1577672)	2026-09-01 22:23 UTC (1577672)	1 / 241 (0.4%; 0.25/week)	1 / 0	1	Status: Single external-download timeout on Sep 1. HttpClient timed out while the test fetched external Android artifacts; this is a network dependency failure. Confidence: high.
Xamarin.Android.NetTests.AndroidMessageHandlerTests.HttpContentStreamIsRewoundAfterCancellation	2026-09-07 07:02 UTC (1584741)	2026-09-07 07:02 UTC (1584741)	1 / 241 (0.4%; 0.25/week)	0 / 1	1	Status: Ongoing/recent; non-gating socket failure observed Sep 7. The failure was an IOException 'Broken pipe' during a green build, indicating transient socket infrastructure. Confidence: high.

Flaky tests

Failure	First seen	Last seen	Frequency	Red / non-gating green	PRs	Evidence
Android.RuntimeTests.JnienvArrayMarshaling.GetObjectArray	2026-08-29 04:44 UTC (1573562)	2026-09-03 09:25 UTC (1579507)	14 / 241 (5.8%; 3.50/week)	11 / 3	12	Status: Likely fixed by #12650: last failure preceded the merge, with 0 recurrences in 71 completed builds afterward. JNI object identity intermittently mismatched across many unrelated PRs, including green builds and unchanged retries. Confidence: high.
Xamarin.Android.JcwGenTests.BindingTests.JavaSideActivation	2026-09-01 22:13 UTC (1577025)	2026-09-07 13:57 UTC (1583153)	4 / 241 (1.7%; 1.00/week)	2 / 2	4	Status: Ongoing; last observed Sep 7 and covered by open tracker #12031. The GREF assertion recurred across unrelated PRs and in non-gating green builds. Confidence: high.
Xamarin.Android.Build.Tests.InstallAndRunTests.GradleFBProj(False,CoreCLR)	2026-09-01 18:37 UTC (1577327)	2026-09-04 14:38 UTC (1583179)	3 / 241 (1.2%; 0.75/week)	3 / 0	3	Status: Quiet since Sep 4; no merged fix was identified. The device test failed across unrelated PRs, and build 1577702 passed unchanged as build 1578074. Confidence: high.
Java.InteropTests.JniTypeUtf8Test.TryFindClass_String_DoesNotLeakGlobalRefs	2026-08-31 13:40 UTC (1574982)	2026-09-01 10:15 UTC (1576388)	2 / 241 (0.8%; 0.50/week)	2 / 0	2	Status: Still unresolved under #12031. #12032 disabled the different UTF-8 overload, not this String overload. The global-reference count assertion failed on unrelated native and typemap PRs. Confidence: high.
Xamarin.Android.Build.Tests.BuildTest2.BuildReleaseArm64(True,CoreCLR,True)	2026-08-31 13:40 UTC (1574985)	2026-08-31 20:14 UTC (1575529)	2 / 241 (0.8%; 0.50/week)	2 / 0	2	Status: Likely fixed by #12515: the two real occurrences used the old 30-second apkdiff timeout, with 0 recurrences in 184 completed post-merge builds. Build 1575474 was removed as a false attribution because the test passed and MAUI Helix caused the red build. The apkdiff process timed out under the former 30-second limit before producing a size-regression verdict. Confidence: medium.
Xamarin.Android.Build.Tests.InstallAndRunTests.DotNetRunBenchmarkDotNet	2026-08-31 07:54 UTC (1573240)	2026-09-03 11:11 UTC (1580340)	2 / 241 (0.8%; 0.50/week)	0 / 2	2	Status: Quiet since Sep 3; no merged fix was identified. The benchmark test failed only as a non-gating result in otherwise-green unrelated builds. Confidence: high.
Xamarin.Android.Build.Tests.InstallAndRunTests.GradleFBProj(True,NativeAOT)	2026-08-31 07:52 UTC (1573440)	2026-09-07 13:34 UTC (1583858)	2 / 241 (0.8%; 0.50/week)	0 / 2	2	Status: Ongoing/recent; last observed as a non-gating failure Sep 7. The device test appeared as a non-gating failure in otherwise-green unrelated builds. Confidence: high.
Xamarin.Android.Build.Tests.AndroidDependenciesTests.InstallAndroidDependenciesTest("GoogleV2",CoreCLR)	2026-09-02 21:22 UTC (1579349)	2026-09-02 21:22 UTC (1579349)	1 / 241 (0.4%; 0.25/week)	1 / 0	1	Status: Likely fixed by #12649: 0 recurrences in 107 completed builds after the runtime-dependency fix. The test host could not load ICSharpCode.SharpZipLib; the next build passed after only merging main. Confidence: high.
Xamarin.Android.Build.Tests.AndroidGradleProjectTests.BindLibrary(True,"Debug",True,CoreCLR)	2026-09-07 11:30 UTC (1584520)	2026-09-07 11:30 UTC (1584520)	1 / 241 (0.4%; 0.25/week)	0 / 1	1	Status: Likely fixed by #12700, which flowed the Roslyn fix from dotnet/roslyn#85131. The only failure used the older SDK; 0 recurrences in 7 verified post-merge SDK exposures. The generated binding compile hit the Roslyn MemorySafetyRulesAttribute decoding race fixed upstream by dotnet/roslyn#85131. Confidence: high.
Xamarin.Android.Build.Tests.PackagingTest.CheckSignApk(False,False,NativeAOT)	2026-08-29 06:09 UTC (1573574)	2026-08-29 06:09 UTC (1573574)	1 / 241 (0.4%; 0.25/week)	1 / 0	1	Status: Likely fixed by #12603: 0 recurrences in 167 completed builds after the SDK-state restoration fix. The test inherited an invalid JAVA_HOME from a previous test, indicating test isolation/order dependence. Confidence: high.

Steps to Reproduce

1. Enumerate all dotnet-android PR builds started during the interval.
2. Read GitHub aggregate-check annotations for failed-test names and messages.
3. Read the Azure timeline and failed task logs for infrastructure and failures omitted from aggregate annotations.
4. Collapse duplicate configuration annotations to one signature per build, then group identical signatures across builds.
5. Classify signatures using unchanged retry success, occurrence across unrelated PRs, green non-gating failures, task-log root causes, and overlap with PR changes.

Did you find any workaround?

Rerunning an unchanged SHA cleared several of these failures. That is only a tactical workaround; the recurring signatures above should be fixed or quarantined independently.

Relevant log output

Each row links the first and last Azure build where the signature was observed. The underlying catalog retains every affected build and PR.

[simonrozsival]

CI flake remediation plan

Goal

Reduce the current 29.7% flake-red first-attempt rate by assigning one independent
dotnet/android session and pull request to each actionable failure family.

Each child session must:

1. Re-read issue CI flakiness inventory: Aug 11-Sep 8, 2026 #12704 and the linked Azure examples.
2. Reproduce or establish the root cause from logs before changing behavior.
3. Prefer a root-cause fix; quarantine or retry only when the external dependency
   cannot be made deterministic.
4. Add focused coverage where practical.
5. Create a draft pull request whose title starts with [flaky-ci], has the
   flaky-ci label, and references CI flakiness inventory: Aug 11-Sep 8, 2026 #12704 plus any focused tracker.
6. Report the PR and remaining uncertainty to this coordinator.

Wave 1: highest impact

Workstream	Current signal	Intended PR
MAUI R2R Helix device-lab failures	35 incidents; ongoing; #12658	Add device preflight, cleanup/recovery, and actionable machine/device diagnostics without hiding real APK failures.
Gradle daemon/cache/tooling failures	11 incidents	Isolate shared Gradle state and make known daemon/cache corruption recoverable without retrying real Gradle build errors.
JavaSideActivation GREF assertion	4 incidents; ongoing; #12031	Replace exact process-global GREF equality with an isolated or operation-specific leak assertion.
GradleFBProj device tests	5 incidents across CoreCLR and NativeAOT variants	Stabilize app startup/device readiness and retain meaningful Gradle/Facebook project coverage.
Logcat capture timeouts	2 incidents after prior mitigations	Make capture bounded and non-gating while preserving failure diagnostics.
MAUI package publication lag	4 incidents	Make MAUI restore tolerate short dependency-publication skew without masking persistent package resolution errors.

Wave 2: remaining actionable failures

Workstream	Current signal	Intended PR
JniTypeUtf8 String GREF assertion	2 incidents; open under #12031	Isolate or redesign the enabled String-overload leak assertion.
BuildReleaseArm64 APK-size baseline	3 incidents	Make the comparison deterministic and explain real size regressions without stale-baseline flakes.
dotnet-test-slicer runtime discovery	2 incidents	Ensure the tool runs with an explicit compatible runtime on hosted macOS agents.
ADB install broken pipe	1 incident	Retry only recognized transport breakage and preserve real install failures.
Hosted dependency downloads	cache HTTP 429 plus NuGet workload-pack stalls/job timeout	Add bounded retries/backoff for transient package acquisition and avoid consuming the full hosted-job limit.
Desugar external artifact timeout	1 incident	Remove or harden the external network dependency, preferably through existing mirrors/caches.
Missing libMono.Unix test artifact	1 incident	Correct the test runtime copy-local dependency closure.
AndroidMessageHandler broken pipe	1 non-gating incident	Make the cancellation test server/client teardown deterministic without weakening assertions.
BenchmarkDotNet device test	2 non-gating incidents	Stabilize benchmark app startup/execution and improve failure diagnostics.
AndroidGradleProject BindLibrary	1 non-gating incident	Reproduce and remove the nondeterministic Gradle/binding fixture failure.
System.NetTests VerifyTrustedCertificates	2 incidents	Stabilize certificate/environment assumptions while preserving the trust validation coverage.

Verify, but do not create duplicate PRs

Failure	Existing fix	Verification
GetObjectArray peer identity	#12650	No recurrence in 71 completed post-merge builds.
InstallAndroidDependencies missing SharpZipLib/Polly	#12649	No recurrence in 107 completed post-merge builds.
CheckSignApk stale JAVA_HOME	#12603	No recurrence in 167 completed post-merge builds.
Original FastTiming logcat race	#12492	No unrelated-PR recurrence in the current catalog.

External-only incidents

Do not create speculative repository PRs for hosted-agent deprovisioning, Azure
no-route-to-host, DNS outages, pipeline startup failure, or isolated git checkout
failure. Keep them in #12704 for frequency tracking and escalate to the service/device-
lab owners when they recur.

Coordination

• All 17 implementation sessions branch independently from main.
• Sessions start in plan mode; the coordinator approves only plans that identify a
  concrete root cause and a reviewable fix.
• Overlapping pipeline files may produce merge conflicts; PRs remain independent and
  will be rebased after higher-impact fixes merge.
• Prioritize Wave 1 review and merge order by expected incident reduction.