x509 Certificate validation failure when using OpenSSL and host is an IP Address #581

x509 SSL certificates fail to verify when connecting as a client to a host that's declared as an IP Address and OpenSSL is the TLS encryption provider. We believe this is due to the exclusive use of X509_VERIFY_PARAM_set1_host in the IXSocketOpenSSL.cpp SocketOpenSSL::connect function.

Our fix was to use the existing IXNetSystem::inet_pton functionality to detect whether the host was a DNS name or IP Address by adding the following to the OpenSSL hostname validation section:

struct in_addr _addr;
if (inet_pton(AF_INET, host.c_str(), &_addr.S_un.S_addr) == 1) {
   //Host is an IPV4 Address
   X509_VERIFY_PARAM_set1_ip_asc(param,  host.c_str());

}else if(inet_pton(AF_INET6, host.c_str(), &_addr.S_un.S_addr) == 1){
   //Host is an IPV6 Address
   X509_VERIFY_PARAM_set1_ip_asc(param,  host.c_str());

} else {
  //Host is a DNS Name
  X509_VERIFY_PARAM_set1_host(param, host.c_str(), host.size());
}

This was tested using Self Signed CA, and SSL certificates where the SSL certificates SAN was established using the TextExtension "2.5.29.17={text}DNS=localhost&IPAddress=127.0.0.1" with both wss://127.0.0.1:44300/Channel/ and wss://localhost:44300/Channel/ as host addresses.