RSAAndroid fails to verify RSA signatures when the public exponent is bigger than 2^32

[runfoapp]

There are a bunch of cryptography failures on Android in rolling build 1905046. Here is an example of one:

https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-e21ac2efc94b48918e/Microsoft.Extensions.Caching.Memory.Tests/1/console.10ca6335.log?%3Fhelixlogtype%3Dresult

<test name="System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests.BuildEmptyRsaPss(hashName: \&quot;SHA256\&quot;)" type="System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests" method="BuildEmptyRsaPss" time="0.031689" result="Fail">
        <failure exception-type="Xunit.Sdk.TrueException">
          <message><![CDATA[Certificate's public key verifies the signature\nExpected: True\nActual:   False]]></message>
          <stack-trace><![CDATA[   at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests.<>c__DisplayClass17_0.<BuildEmptyRsaPss>b__0(X509Certificate2 cert, DateTimeOffset now) in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/CertificateCreation/CrlBuilderTests.cs:line 407
   at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests.BuildRsaCertificateAndRun(IEnumerable`1 extensions, Action`2 action, Boolean addSubjectKeyIdentifier, String callerName) in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/CertificateCreation/CrlBuilderTests.cs:line 1490
   at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests.BuildEmptyRsaPss(String hashName) in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/CertificateCreation/CrlBuilderTests.cs:line 382
   at System.Reflection.MethodInvoker.InterpretedInvoke(Object obj, Span`1 args, BindingFlags invokeAttr)]]></stack-trace>
        </failure>
      </test>

Runfo Tracking Issue: system.security.cryptography.x509certificates.tests work item

Build	Definition	Kind	Run Name

Build Result Summary

Day Hit Count	Week Hit Count	Month Hit Count
0	0	0

Tagging subscribers to this area: @dotnet/area-system-security, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Runfo Creating Tracking Issue (data being generated)

Author:	runfoapp[bot]
Assignees:	-
Labels:	area-System.Security
Milestone:	-

Tagging subscribers to 'arch-android': @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

Issue Details

---

There are a bunch of cryptography failures on Android in rolling build 1905046. Here is an example of one:

https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-e21ac2efc94b48918e/Microsoft.Extensions.Caching.Memory.Tests/1/console.10ca6335.log?%3Fhelixlogtype%3Dresult

<test name="System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests.BuildEmptyRsaPss(hashName: \&quot;SHA256\&quot;)" type="System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests" method="BuildEmptyRsaPss" time="0.031689" result="Fail">
        <failure exception-type="Xunit.Sdk.TrueException">
          <message><![CDATA[Certificate's public key verifies the signature\nExpected: True\nActual:   False]]></message>
          <stack-trace><![CDATA[   at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests.<>c__DisplayClass17_0.<BuildEmptyRsaPss>b__0(X509Certificate2 cert, DateTimeOffset now) in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/CertificateCreation/CrlBuilderTests.cs:line 407
   at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests.BuildRsaCertificateAndRun(IEnumerable`1 extensions, Action`2 action, Boolean addSubjectKeyIdentifier, String callerName) in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/CertificateCreation/CrlBuilderTests.cs:line 1490
   at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests.BuildEmptyRsaPss(String hashName) in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/CertificateCreation/CrlBuilderTests.cs:line 382
   at System.Reflection.MethodInvoker.InterpretedInvoke(Object obj, Span`1 args, BindingFlags invokeAttr)]]></stack-trace>
        </failure>
      </test>

Runfo Tracking Issue: system.security.cryptography.x509certificates.tests work item

Build	Definition	Kind	Run Name	Console	Core Dump	Test Results	Run Client
1905046	runtime-extra-platforms	Rolling	net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open	console.log		test results	runclient.py
1905046	runtime-extra-platforms	Rolling	net7.0-Android-Release-x64-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open	console.log		test results	runclient.py
1905046	runtime-extra-platforms	Rolling	net7.0-Android-Release-arm-Mono_Release-Windows.10.Amd64.Android.Open	console.log		test results	runclient.py
1905046	runtime-extra-platforms	Rolling	net7.0-Android-Release-arm64-Mono_Release-Windows.10.Amd64.Android.Open	console.log		test results	runclient.py
1904585	runtime-extra-platforms	Rolling	net7.0-Linux-Release-x64-CoreCLR_release-(Fedora.34.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-34-helix-20220523150939-4f64125	console.log			runclient.py
1903280	runtime-extra-platforms	PR 72832	net7.0-MacCatalyst-Release-arm64-Mono_Release-OSX.1200.Arm64.Open	console.log			runclient.py
1903274	runtime-extra-platforms	Rolling	net7.0-Linux-Release-x64-CoreCLR_release-(Fedora.34.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-34-helix-20220523150939-4f64125	console.log			runclient.py
1902034	runtime-extra-platforms	Rolling	net7.0-Linux-Release-x64-CoreCLR_release-(Fedora.34.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-34-helix-20220523150939-4f64125	console.log			runclient.py
1901750	runtime-extra-platforms	PR 72545	net7.0-MacCatalyst-Release-arm64-Mono_Release-OSX.1200.Arm64.Open	console.log			runclient.py
1900887	runtime-extra-platforms	Rolling	net7.0-Linux-Release-x64-CoreCLR_release-(Fedora.34.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-34-helix-20220523150939-4f64125	console.log			runclient.py
1900480	runtime-extra-platforms	Rolling	net7.0-Linux-Release-x64-CoreCLR_release-(Fedora.34.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-34-helix-20220523150939-4f64125	console.log			runclient.py
1900064	runtime-extra-platforms	Rolling	net7.0-Linux-Release-x64-CoreCLR_release-(Fedora.34.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-34-helix-20220523150939-4f64125	console.log			runclient.py

Build Result Summary

Day Hit Count	Week Hit Count	Month Hit Count
4	9	9

Author:	runfoapp[bot]
Assignees:	-
Labels:	area-System.Security, blocking-clean-ci, os-android, untriaged
Milestone:	-

[bartonjs]

The default RSA provider tests claim that Android supports both RSASSA-PSS and RSA with keys whose exponents are bigger than 32 bits; but this may be the only test that does both.

<test name="System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests.BuildEmptyRsaPss(hashName: \"SHA256\")" type="System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests" method="BuildEmptyRsaPss" time="0.0771638" result="Fail">
<failure exception-type="Xunit.Sdk.TrueException">
<message>
<![CDATA[ Certificate's public key verifies the signature\nExpected: True\nActual: False ]]>
</message>
<stack-trace>
<![CDATA[ at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests.<>c__DisplayClass17_0.<BuildEmptyRsaPss>b__0(X509Certificate2 cert, DateTimeOffset now) in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/CertificateCreation/CrlBuilderTests.cs:line 407 at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests.BuildRsaCertificateAndRun(IEnumerable`1 extensions, Action`2 action, Boolean addSubjectKeyIdentifier, String callerName) in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/CertificateCreation/CrlBuilderTests.cs:line 1490 at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests.BuildEmptyRsaPss(String hashName) in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/CertificateCreation/CrlBuilderTests.cs:line 382 at System.Reflection.MethodInvoker.InterpretedInvoke(Object obj, Span`1 args, BindingFlags invokeAttr) ]]>
</stack-trace>
</failure>
</test>
<test name="System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests.BuildEmptyRsaPss(hashName: \"SHA384\")" type="System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests" method="BuildEmptyRsaPss" time="0.0057438" result="Fail">
<failure exception-type="Xunit.Sdk.TrueException">
<message>
<![CDATA[ Certificate's public key verifies the signature\nExpected: True\nActual: False ]]>
</message>
<stack-trace>
<![CDATA[ at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests.<>c__DisplayClass17_0.<BuildEmptyRsaPss>b__0(X509Certificate2 cert, DateTimeOffset now) in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/CertificateCreation/CrlBuilderTests.cs:line 407 at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests.BuildRsaCertificateAndRun(IEnumerable`1 extensions, Action`2 action, Boolean addSubjectKeyIdentifier, String callerName) in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/CertificateCreation/CrlBuilderTests.cs:line 1490 at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests.BuildEmptyRsaPss(String hashName) in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/CertificateCreation/CrlBuilderTests.cs:line 382 at System.Reflection.MethodInvoker.InterpretedInvoke(Object obj, Span`1 args, BindingFlags invokeAttr) ]]>
</stack-trace>
</failure>
</test>
<test name="System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests.BuildEmptyRsaPss(hashName: \"SHA512\")" type="System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests" method="BuildEmptyRsaPss" time="0.0070822" result="Fail">
<failure exception-type="Xunit.Sdk.TrueException">
<message>
<![CDATA[ Certificate's public key verifies the signature\nExpected: True\nActual: False ]]>
</message>
<stack-trace>
<![CDATA[ at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests.<>c__DisplayClass17_0.<BuildEmptyRsaPss>b__0(X509Certificate2 cert, DateTimeOffset now) in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/CertificateCreation/CrlBuilderTests.cs:line 407 at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests.BuildRsaCertificateAndRun(IEnumerable`1 extensions, Action`2 action, Boolean addSubjectKeyIdentifier, String callerName) in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/CertificateCreation/CrlBuilderTests.cs:line 1490 at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CrlBuilderTests.BuildEmptyRsaPss(String hashName) in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/CertificateCreation/CrlBuilderTests.cs:line 382 at System.Reflection.MethodInvoker.InterpretedInvoke(Object obj, Span`1 args, BindingFlags invokeAttr) ]]>
</stack-trace>
</failure>
</test>

The only failures are in the CrlBuilderTests.BuildEmptyRsaPss test, and that test is passing everywhere except Android.

[bartonjs]

Newer tests show that even RSASSA-PKCS1 is failing with big-exponent keys; it looks like RsaVerificationPrimitive is not working how we'd expect. That makes big-exponent encryption also suspect.