Failures in SSL Stream tests: There are no more endpoints available in the endpoint mapper.

[carlossanlop]

Occurrences from Runfo based on last 30 days and on Test Monitor history:

Day	Run	Failures	Details
9/8	Rolling run 9900 (7.0)	2x entries	net7.0-windows-Release-x86-CoreCLR_release-Windows.10.Amd64.Server2022.ES.Open - 2x System.Reflection.PortableExecutable.Tests.PEBuilderTests (BasicValidationSigned & Checksum)
9/7	Rolling run 6928	360x entries	net7.0-windows-Release-x86-CoreCLR_release-Windows.10.Amd64.Server2022.ES.Open
8/24	PR 1964645 - PR #62863	1,016x failures in Kusto	net7.0-windows-Debug-x64-CoreCLR_release-Windows.10.Amd64.Server2022.ES.Open
~8/11	PR 1936509	? (data not available in Runfo) / PR not listed in Kusto	net7.0-windows-Debug-x64-CoreCLR_release-Windows.10.Amd64.Server2022.ES.Open
8/6	Rolling run 1927354	828x failures / 1,269x failures in Kusto	net7.0-windows-Release-x86-CoreCLR_release-Windows.10.Amd64.Server2022.ES.Open & net7.0-windows-Release-x64-CoreCLR_release-Windows.10.Amd64.Server2022.ES.Open
8/4	PR #73200	2,566x failures in Kusto	Windows.10.Amd64.Server2022.ES.Open - x86 & x64
7/29	Rolling run	112x failures in Kusto	Windows.10.Amd64.Server2022.ES.Open - x64
7/29	PR #73061	1,619x failures in Kusto	Windows.10.Amd64.Server2022.ES.Open - x64 & x86
7/29	PR #73057	293x failures in Kusto	Windows.10.Amd64.Server2022.ES.Open - x64 & x86
7/29	PR #73055	789x failures in Kusto	Windows.10.Amd64.Server2022.ES.Open - x64
7/27	PR #72882	789x failures in Kusto	Windows.10.Amd64.Server2022.ES.Open - x64
7/26	PR #72886	1,239x failures in Kusto	Windows.10.Amd64.Server2022.ES.Open - x86 & x64
7/26	PR #72869	882x failures in Kusto	Windows.10.Amd64.Server2022.ES.Open - x64
7/25	PR #72814	1,082x failures in Kusto	Windows.10.Amd64.Server2022.ES.Open - x86 & x64
6/30	PR #71473	787x failures in Kusto	Windows.10.Amd64.Server2022.ES.Open - x64
6/30	PR #71405	329x failures in Kusto	Windows.10.Amd64.Server2022.ES.Open - x86 & x64

This happened in the release/7.0 branch. Can you please confirm if this will require a backported fix?

PR: #74808
Queue: Libraries Test Run release coreclr windows x86 Release
Job: https://dev.azure.com/dnceng/public/_build/results?buildId=1976097&view=logs&j=457f7e88-dfa2-5bd9-f871-fdf124c2477d&t=bfe52dfb-2099-5c7f-ee52-70a1d81c544e
Log: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-74808-merge-0a29e7160d114b13be/System.Net.Security.Tests/3/console.38a99609.log?helixlogtype=result

Failure message:

CryptographicException : No hay más extremos disponibles desde el asignador de extremos.

Which translates to:

CryptographicException : There are no more endpoints available in the endpoint mapper.

System.Net.Security tests with this failure:

System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_SecondNegotiateClientCertificateAsync_Throws
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_UntrustedCaWithCustomTrust_OK
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_NegotiateClientCertificateAsync_IncompleteIncomingTlsFrame_Throws
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_NegotiateClientCertificateAsync_ClientWriteData
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_NegotiateClientCertificateAsyncNoRenego_Succeeds
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_NestedAuth_Throws
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_NegotiateClientCertificateAsyncConcurrentIO_Throws
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_NegotiateClientCertificateAsync_PendingDecryptedData_Throws
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_NegotiateClientCertificateAsyncTls13_Succeeds
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_NegotiateClientCertificateAsync_Succeeds
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_TargetHostName_Succeeds
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_RandomSizeWrites_OK
System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback
System.Net.Security.Tests.SslStreamMutualAuthenticationTest.SslStream_RequireClientCert_IsMutuallyAuthenticated_ReturnsTrue
System.Net.Security.Tests.SslStreamCredentialCacheTest.SslStream_SameCertUsedForClientAndServer_Ok

System.Net.Http tests with this failure:

System.Net.Http.Functional.Tests.PlatformHandler_HttpClientHandler_ServerCertificates_Http2_Test.UseCallback_SelfSignedCertificate_ExpectedPolicyErrors
System.Net.Http.WinHttpHandlerFunctional.Tests.ClientCertificateTest.UseClientCertOnHttp2_DowngradedToHttp1MutualAuth_Success

Callstack example:

    System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_SecondNegotiateClientCertificateAsync_Throws(sendClientCertificate: True) [FAIL]
      System.AggregateException : One or more errors occurred. (No hay m�s extremos disponibles desde el asignador de extremos.) (The following constructor parameters did not have matching fixture data: CertificateSetup setup)
      ---- System.Security.Cryptography.CryptographicException : No hay m�s extremos disponibles desde el asignador de extremos.
      ---- The following constructor parameters did not have matching fixture data: CertificateSetup setup
      Stack Trace:
        
        ----- Inner Stack Trace #1 (System.Security.Cryptography.CryptographicException) -----
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/CngHelpers.cs(33,0): at System.Security.Cryptography.CngHelpers.OpenStorageProvider(CngProvider provider)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/CngKey.Create.cs(41,0): at System.Security.Cryptography.CngKey.Create(CngAlgorithm algorithm, String keyName, CngKeyCreationParameters creationParameters)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/CngAlgorithmCore.cs(73,0): at System.Security.Cryptography.CngAlgorithmCore.GetOrGenerateKey(Int32 keySize, CngAlgorithm algorithm)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSACng.Key.cs(24,0): at System.Security.Cryptography.RSACng.get_Key()
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSACng.ImportExport.cs(50,0): at System.Security.Cryptography.RSACng.ExportKeyBlob(Boolean includePrivateParameters)
        /_/src/libraries/Common/src/System/Security/Cryptography/RSACng.ImportExport.cs(282,0): at System.Security.Cryptography.RSACng.ExportParameters(Boolean includePrivateParameters)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSA.cs(811,0): at System.Security.Cryptography.RSA.WritePkcs1PublicKey()
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSA.cs(675,0): at System.Security.Cryptography.RSA.ExportRSAPublicKey()
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSAWrapper.cs(141,0): at System.Security.Cryptography.RSAWrapper.ExportRSAPublicKey()
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/RSAPkcs1X509SignatureGenerator.cs(37,0): at System.Security.Cryptography.X509Certificates.RSAPkcs1X509SignatureGenerator.BuildPublicKey(RSA rsa)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/RSAPkcs1X509SignatureGenerator.cs(27,0): at System.Security.Cryptography.X509Certificates.RSAPkcs1X509SignatureGenerator.BuildPublicKey()
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/X509SignatureGenerator.cs(10,0): at System.Security.Cryptography.X509Certificates.X509SignatureGenerator.get_PublicKey()
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/CertificateRequest.cs(141,0): at System.Security.Cryptography.X509Certificates.CertificateRequest..ctor(String subjectName, RSA key, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding)
        /_/src/libraries/Common/tests/System/Security/Cryptography/X509Certificates/CertificateAuthority.cs(820,0): at System.Security.Cryptography.X509Certificates.Tests.Common.CertificateAuthority.BuildPrivatePki(PkiOptions pkiOptions, RevocationResponder& responder, CertificateAuthority& rootAuthority, CertificateAuthority[]& intermediateAuthorities, X509Certificate2& endEntityCert, Int32 intermediateAuthorityCount, String testName, Boolean registerAuthorities, Boolean pkiOptionsInSubject, String subjectName, Int32 keySize, X509ExtensionCollection extensions)
        /_/src/libraries/System.Net.Security/tests/FunctionalTests/TestHelper.cs(168,0): at System.Net.Security.Tests.TestHelper.GenerateCertificates(String targetName, String testName, Boolean longChain, Boolean serverCertificate)
        /_/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamNetworkStreamTest.cs(30,0): at System.Net.Security.Tests.CertificateSetup..ctor()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
        /_/src/libraries/System.Private.CoreLib/src/System/Reflection/ConstructorInvoker.cs(86,0): at System.Reflection.ConstructorInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

Other exceptions:

System.Security.Cryptography.CryptographicException : Datos incorrectos.

   at System.Security.Cryptography.X509Certificates.CertificatePal.FilterPFXStore(ReadOnlySpan`1 rawData, SafePasswordHandle password, PfxCertStoreFlags pfxCertStoreFlags) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/CertificatePal.Windows.Import.cs:line 167
   at System.Security.Cryptography.X509Certificates.CertificatePal.FromBlobOrFile(ReadOnlySpan`1 rawData, String fileName, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/CertificatePal.Windows.Import.cs:line 88
   at System.Security.Cryptography.X509Certificates.CertificatePal.FromBlob(ReadOnlySpan`1 rawData, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/CertificatePal.Windows.Import.cs:line 16
   at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(Byte[] rawData, String password, X509KeyStorageFlags keyStorageFlags) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/X509Certificate.cs:line 95
   at System.Security.Cryptography.X509Certificates.X509Certificate2..ctor(Byte[] rawData, String password, X509KeyStorageFlags keyStorageFlags) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/X509Certificate2.cs:line 72
   at Test.Cryptography.CertLoader.TryGetCertificateWithPrivateKey(Boolean exportable) in /_/src/libraries/System.Security.Cryptography.Pkcs/tests/CertLoader.cs:line 84
   at System.Security.Cryptography.Pkcs.Tests.SignedCmsTests.SignWithImplicitSubjectKeyIdentifier() in /_/src/libraries/System.Security.Cryptography.Pkcs/tests/SignedCms/SignedCmsTests.cs:line 1225
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr) in /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodInvoker.cs:line 64

System.Security.Cryptography.CryptographicException : Algoritmo especificado no es válido.

   at System.Security.Cryptography.CapiHelper.CreateHashHandle(SafeProvHandle hProv, Byte[] hash, Int32 calgHash, Boolean throwOnSizeError) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/CapiHelper.Windows.cs:line 1385
   at System.Security.Cryptography.CapiHelper.CreateHashHandle(SafeProvHandle hProv, Byte[] hash, Int32 calgHash) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/CapiHelper.Windows.cs:line 1373
   at System.Security.Cryptography.CapiHelper.SignValue(SafeProvHandle hProv, SafeCapiKeyHandle hKey, Int32 keyNumber, Int32 calgKey, Int32 calgHash, Byte[] hash) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/CapiHelper.Windows.cs:line 1161
   at System.Security.Cryptography.RSACryptoServiceProvider.SignHash(Byte[] rgbHash, Int32 calgHash) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSACryptoServiceProvider.Windows.cs:line 504
   at System.Security.Cryptography.RSACryptoServiceProvider.SignHash(Byte[] hash, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSACryptoServiceProvider.Windows.cs:line 633
   at System.Security.Cryptography.RSA.SignData(Byte[] data, Int32 offset, Int32 count, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSA.cs:line 189
   at System.Security.Cryptography.RSA.SignData(Byte[] data, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSA.cs:line 170
   at System.Security.Cryptography.X509Certificates.Tests.CollectionTests.SerializedCertDisposeDoesNotRemoveKeyFile() in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/CollectionTests.cs:line 1437
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr) in /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodInvoker.cs:line 64

System.TypeInitializationException : The type initializer for 'System.Security.Cryptography.CngKeyLite' threw an exception.
---- Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException : No hay más extremos disponibles desde el asignador de extremos.

let failedTests = (testNameSubstring : string, methodName : string, messageSubstr: string, includePR : bool, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
    | where TestName contains testNameSubstring
    | where includePassedOnRerun or (Outcome == 'Failed')
    | extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
    | extend Method = substring(TestName, startOfTestName)
    | extend Type = substring(TestName, 0, startOfTestName - 1)
    | project-away startOfTestName
    | where (methodName == '') or (Method == methodName)
    //| where Message contains messageSubstr
    | where (Message contains 'System.Security.Cryptography.CryptographicException : No hay más extremos disponibles desde el asignador de extremos.' or Message contains 'System.Security.Cryptography.CryptographicException : Datos incorrectos.' or Message contains 'System.Security.Cryptography.CryptographicException : Algoritmo especificado no es válido.')
    | distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
        //| where Branch == 'refs/pull//merge'
        //| where Branch == 'refs/pull//merge'
        | where Branch != 'refs/pull/71473/merge'
        | where Branch != 'refs/pull/73057/merge'
        | where JobId != '20283731' // 7/29
        | where JobId != '20342414' // 8/6 ... 488x
        | where JobId != '20342402' // 8/6 ... 781x
        | where Branch != 'refs/pull/71405/merge'
        | where Branch != 'refs/pull/72869/merge'
        | where Branch != 'refs/pull/72814/merge'
        | where Branch != 'refs/pull/72886/merge'
        | where Branch != 'refs/pull/72882/merge'
        | where Branch != 'refs/pull/73055/merge'
        | where Branch != 'refs/pull/73200/merge'
        | where Branch != 'refs/pull/62863/merge'
        | where Branch != 'refs/pull/73061/merge'
        | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")))
        | where Type startswith "test/functional/cli/"
            and not(Properties contains "runtime-staging")
        | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
        | project-rename JobType = Type) on JobId
    | extend PropertiesJson = parse_json(Properties)
    | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
    | where OS contains "ES"
    | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
    | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
    | extend Architecture = PropertiesJson.architecture
    | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
    //| extend DefinitionName = PropertiesJson.DefinitionName
    | project-away PropertiesJson
};
failedTests(
    'System.Security.Cryptography', //testNameSubstring
    '', //methodName
    '',//ignored
    true,  //includePR
    true); //includePassedOnRerun

Known Issue Error Message

Fill the error message using known issues guidance.

{
    "ErrorPattern": ".*No hay.*extremos disponibles desde el asignador de extremos.*",
    "BuildRetry": false,
    "ExcludeConsoleLog": false
}

Report

Build	Definition	Test	Pull Request
165740	dotnet/runtime	System.Net.Http.Functional.Tests.PlatformHandler_HttpClientHandler_ServerCertificates_Http2_Test.UseCallback_SelfSignedCertificate_ExpectedPolicyErrors	#81831
165735	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ClientCertificates_Test.Manual_CertificateOnlySentWhenValid_Success	#81728
164105	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ClientCertificates_Test.Manual_CertificateOnlySentWhenValid_Success	#78852
164104	dotnet/runtime	System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_NetworkStream_Renegotiation_Succeeds	#81822
163901	dotnet/runtime	System.Formats.Cbor.Tests.CborReaderTests.CoseKeyHelpers_ECDsaParseCosePublicKey_HappyPath	#80382

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	5

Tagging subscribers to this area: @dotnet/ncl, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

This happened in the release/7.0 branch. Can you please confirm if this will require a backported fix?

PR: #74808
Queue: Libraries Test Run release coreclr windows x86 Release
Job: https://dev.azure.com/dnceng/public/_build/results?buildId=1976097&view=logs&j=457f7e88-dfa2-5bd9-f871-fdf124c2477d&t=bfe52dfb-2099-5c7f-ee52-70a1d81c544e
Log: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-74808-merge-0a29e7160d114b13be/System.Net.Security.Tests/3/console.38a99609.log?helixlogtype=result

Failure message:

CryptographicException : No hay más extremos disponibles desde el asignador de extremos.

Which translates to:

CryptographicException : There are no more endpoints available in the endpoint mapper.

Tests with a similar failure:

System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_SecondNegotiateClientCertificateAsync_Throws
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_UntrustedCaWithCustomTrust_OK
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_NegotiateClientCertificateAsync_IncompleteIncomingTlsFrame_Throws
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_NegotiateClientCertificateAsync_ClientWriteData
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_NegotiateClientCertificateAsyncNoRenego_Succeeds
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_NestedAuth_Throws
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_NegotiateClientCertificateAsyncConcurrentIO_Throws
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_NegotiateClientCertificateAsync_PendingDecryptedData_Throws
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_NegotiateClientCertificateAsyncTls13_Succeeds
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_NegotiateClientCertificateAsync_Succeeds
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_TargetHostName_Succeeds
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_RandomSizeWrites_OK
System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback
System.Net.Security.Tests.SslStreamMutualAuthenticationTest.SslStream_RequireClientCert_IsMutuallyAuthenticated_ReturnsTrue
System.Net.Security.Tests.SslStreamCredentialCacheTest.SslStream_SameCertUsedForClientAndServer_Ok

Callstack example:

    System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_SecondNegotiateClientCertificateAsync_Throws(sendClientCertificate: True) [FAIL]
      System.AggregateException : One or more errors occurred. (No hay m�s extremos disponibles desde el asignador de extremos.) (The following constructor parameters did not have matching fixture data: CertificateSetup setup)
      ---- System.Security.Cryptography.CryptographicException : No hay m�s extremos disponibles desde el asignador de extremos.
      ---- The following constructor parameters did not have matching fixture data: CertificateSetup setup
      Stack Trace:
        
        ----- Inner Stack Trace #1 (System.Security.Cryptography.CryptographicException) -----
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/CngHelpers.cs(33,0): at System.Security.Cryptography.CngHelpers.OpenStorageProvider(CngProvider provider)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/CngKey.Create.cs(41,0): at System.Security.Cryptography.CngKey.Create(CngAlgorithm algorithm, String keyName, CngKeyCreationParameters creationParameters)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/CngAlgorithmCore.cs(73,0): at System.Security.Cryptography.CngAlgorithmCore.GetOrGenerateKey(Int32 keySize, CngAlgorithm algorithm)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSACng.Key.cs(24,0): at System.Security.Cryptography.RSACng.get_Key()
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSACng.ImportExport.cs(50,0): at System.Security.Cryptography.RSACng.ExportKeyBlob(Boolean includePrivateParameters)
        /_/src/libraries/Common/src/System/Security/Cryptography/RSACng.ImportExport.cs(282,0): at System.Security.Cryptography.RSACng.ExportParameters(Boolean includePrivateParameters)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSA.cs(811,0): at System.Security.Cryptography.RSA.WritePkcs1PublicKey()
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSA.cs(675,0): at System.Security.Cryptography.RSA.ExportRSAPublicKey()
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSAWrapper.cs(141,0): at System.Security.Cryptography.RSAWrapper.ExportRSAPublicKey()
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/RSAPkcs1X509SignatureGenerator.cs(37,0): at System.Security.Cryptography.X509Certificates.RSAPkcs1X509SignatureGenerator.BuildPublicKey(RSA rsa)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/RSAPkcs1X509SignatureGenerator.cs(27,0): at System.Security.Cryptography.X509Certificates.RSAPkcs1X509SignatureGenerator.BuildPublicKey()
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/X509SignatureGenerator.cs(10,0): at System.Security.Cryptography.X509Certificates.X509SignatureGenerator.get_PublicKey()
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/CertificateRequest.cs(141,0): at System.Security.Cryptography.X509Certificates.CertificateRequest..ctor(String subjectName, RSA key, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding)
        /_/src/libraries/Common/tests/System/Security/Cryptography/X509Certificates/CertificateAuthority.cs(820,0): at System.Security.Cryptography.X509Certificates.Tests.Common.CertificateAuthority.BuildPrivatePki(PkiOptions pkiOptions, RevocationResponder& responder, CertificateAuthority& rootAuthority, CertificateAuthority[]& intermediateAuthorities, X509Certificate2& endEntityCert, Int32 intermediateAuthorityCount, String testName, Boolean registerAuthorities, Boolean pkiOptionsInSubject, String subjectName, Int32 keySize, X509ExtensionCollection extensions)
        /_/src/libraries/System.Net.Security/tests/FunctionalTests/TestHelper.cs(168,0): at System.Net.Security.Tests.TestHelper.GenerateCertificates(String targetName, String testName, Boolean longChain, Boolean serverCertificate)
        /_/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamNetworkStreamTest.cs(30,0): at System.Net.Security.Tests.CertificateSetup..ctor()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
        /_/src/libraries/System.Private.CoreLib/src/System/Reflection/ConstructorInvoker.cs(86,0): at System.Reflection.ConstructorInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

Author:	carlossanlop
Assignees:	-
Labels:	area-System.Net.Security
Milestone:	-

[wfurt]

looks like some environmental problem. cc @bartonjs in case he has some insight to the Cng....

[bartonjs]

We've seen it before on the es-ES test machine(s)... something about the RPC system is broken and that breaks CNG which breaks .NET Crypto and also SChannel.

It happened for a while, then stopped... and I guess is happening again.

#70320 (comment)

[carlossanlop]

@dotnet/ncl This issue is a bit old. Should we close it in favor of a newer one? I think people will keep assigning their failures to this, but the root cause is probably going to be different.

[wfurt]

It seems like SslStream is victim of #70320 as @bartonjs mentioned.
This looks like broken environment to me. Maybe @MattGal would have some insight why the queue is any different.

We may add some checks into test contractor and blow whole suite if crypto is not functional but that would probably not help with build stability.

[MattGal]

why the queue is any different.

Happy to investigate, but could you share the queue and ideally the specific log you're asking about so I don't have to trawl through all of them?

[wfurt]

This seems to exclusively happen on Windows.10.Amd64.Server2022.ES.Open (and its predecessors)
I have no clue why Spanish locale would break crypto.

[carlossanlop]

Seeing this happening again. Specifically, I found it in at least two unrelated release/8.0-preview1 backport PRs. It is affecting several DSA, EcDSA and RSA tests. cc @bartonjs @vcsjones

• Affected unrelated PRs: [release/8.0-preview1] [QUIC] Fix conditions to bin place msquic.dll for Windows builds #81492 and [release/8.0-preview1] Update dependencies from dotnet/emsdk #81491
• Queue: Libraries Test Run release coreclr windows x64 Debug
• Job results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=155894&view=logs&j=cb131f87-d5af-5ecc-6e5f-6dc86994859c&t=76503e0a-8c41-542c-43da-63a4758a95b0&l=104
• Log file: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-81492-merge-87d3855728224aee8c/System.Security.Cryptography.Cng.Tests/1/console.e58b6d67.log?helixlogtype=result
• Affected tests: System.Security.Cryptography.Dsa*, System.Security.Cryptography.Rsa*, System.Security.Cryptography.EcDsa*`
• Failure example:

System.Security.Cryptography.Dsa.Tests.DSASignVerify_Array.VerifyKnown_2048_SHA384 [FAIL]
      System.Security.Cryptography.CryptographicException : No hay mas extremos disponibles desde el asignador de extremos.
      Stack Trace:
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/CngHelpers.cs(36,0): at System.Security.Cryptography.CngHelpers.OpenStorageProvider(CngProvider provider)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/CngKey.Import.cs(125,0): at System.Security.Cryptography.CngKey.Import(ReadOnlySpan`1 keyBlob, String curveName, CngKeyBlobFormat format, CngProvider provider)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/CngKey.Import.cs(113,0): at System.Security.Cryptography.CngKey.Import(Byte[] keyBlob, String curveName, CngKeyBlobFormat format, CngProvider provider)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/CngKey.Import.cs(41,0): at System.Security.Cryptography.CngKey.Import(Byte[] keyBlob, CngKeyBlobFormat format, CngProvider provider)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/CngKey.Import.cs(30,0): at System.Security.Cryptography.CngKey.Import(Byte[] keyBlob, CngKeyBlobFormat format)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/DSACng.ImportExport.cs(25,0): at System.Security.Cryptography.DSACng.ImportKeyBlob(Byte[] dsaBlob, Boolean includePrivate)
        /_/src/libraries/Common/src/System/Security/Cryptography/DSACng.ImportExport.cs(55,0): at System.Security.Cryptography.DSACng.ImportParameters(DSAParameters parameters)
        /_/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/DSA/DSASignVerify.cs(305,0): at System.Security.Cryptography.Dsa.Tests.DSASignVerify.VerifyKnown_2048_SHA384()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
        /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodInvoker.cs(59,0): at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

Also failing in another unrelated PR, affecting the System.Net tests using SSL: #81457

[bartonjs]

It is affecting several DSA, EcDSA and RSA tests.

The OS is broken, not .NET. There's nothing we can do about it... this is an infrastructure problem (something, somehow, has messed up the es-es machine(s)).