[Android] System.Security.Cryptography.Tests RSA signature validation failures

[matouskozak]

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1018649

Build error leg or test failing: System.Security.Cryptography.Tests (RSA signature validation failures)

Affected CI jobs:

• android_arm_release_allsubsets_mono
• android_arm64_release_allsubsets_mono

Commit range: a6553ac646aa190f716787dcdf1dcc157b5f280b...44096c65c0e96e72b02073953c3b7e69a73db1e2

Error log

Exception messages: RsaPkcs1 signature validation failed when it should have succeeded.
Exception stack traces:    
at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests.VerifySignature(CertKind certKind, X509Certificate2 cert, ReadOnlySpan`1 data, ReadOnlySpan`1 signature, HashAlgorithmName hashAlgorithm) in /_/src/libraries/System.Security.Cryptography/tests/X509Certificates/CertificateCreation/CrlBuilderTests.cs:line 1693
at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests.<>c.<BuildEmpty_NoSubjectKeyIdentifier>b__21_0(CertKind certKind, X509Certificate2 cert, DateTimeOffset now) in /_/src/libraries/System.Security.Cryptography/tests/X509Certificates/CertificateCreation/CrlBuilderTests.cs:line 552
at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests.BuildCertificateAndRun(CertKind certKind, IEnumerable`1 extensions, Action`3 action, Boolean addSubjectKeyIdentifier, String callerName) in /_/src/libraries/System.Security.Cryptography/tests/X509Certificates/CertificateCreation/CrlBuilderTests.cs:line 1522

Error Message

{
  "ErrorMessage": "RsaPkcs1 signature validation failed when it should have succeeded.",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Issue Details

First Observed: April 10, 2025
Failing Tests:

1. CrlBuilderTests.BuildEmpty_NoSubjectKeyIdentifier(certKind: RsaPkcs1)
2. CrlBuilderTests.BuildEmpty_NoSubjectKeyIdentifier(certKind: RsaPss)
3. CrlBuilderTests.BuildEmpty(certKind: RsaPkcs1)
4. CrlBuilderTests.BuildEmpty(certKind: RsaPss)

Problem: RSA signature validation in X.509 certificate operations is failing consistently on Android platforms. Both PKCS#1 and PSS signature validation fail with the same error pattern.

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1018649
Error message validated: [RsaPkcs1 signature validation failed when it should have succeeded.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 4/17/2025 11:12:02 AM UTC

Report

Build	Definition	Test	Pull Request
1020296	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
1019924	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
1019283	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
1018649	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	4	4

[dotnet-policy-service]

Tagging subscribers to 'arch-android': @vitek-karas, @simonrozsival, @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[matouskozak]

Could it be caused by #114416 @bartonjs ?

[bartonjs]

Yeah, it moved a test that I think was already failing on (some versions of?) Android, but was suppressed by a mechanism other than something in the file. Specifically, they're using a static key that has an exponent value bigger than uint.MaxValue, which I was recently reminded was a problem on Android. I'll see if I can get these tests moved off of that key.