SocketsHttpHandler should use first available connection (don't wait on new connection creation)

[geoffkizer]

AB#1254012
Currently, when we have no idle HTTP/1.1 connections (and are not at the connection limit) and a new request comes in, we will create a new connection and then use it for the new request.

The connect itself may take a while, especially for HTTPS connections. During this time, an existing connection may complete its current request and become available. If so, we should just use this idle connection instead of waiting for the new connection to finish its connect.

This should help request latency, and probably reduce the number of connections created during burst usage as well (see #43764).

More details:

When this happens, the new connection can still be put in the pool as idle, even if we don't have a request for it at the moment. If another new request comes in, we will use it like any other idle connection.

Note also that we should avoid creating new connections when we already have enough pending connections to (eventually) handle all the pending requests. Consider this scenario, starting with no connections:

• Request A arrives, we initiate a connect for Connection 1, wait for it, and start processing A on it
• Request B arrives, we initiate a connect for Connection 2. Before the connect completes, A finishes and Connection 1 becomes available. We start processing B on Connection 1.
• Request C arrives. There's no established connection for it to use, but there is a pending connection (Connection 2). So there's no need for us to initiate another connection right now.
• Request D arrives. Now, we have 0 available connections, 1 pending connection (Connection 2), and 2 pending requests (C and D). So we should go ahead and initiate a new connect for Connection 3.

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Details

Issue Details

Description:	Currently, when we have no idle HTTP/1.1 connections (and are not at the connection limit) and a new request comes in, we will create a new connection and then use it for the new request. The connect itself may take a while, especially for HTTPS connections. During this time, an existing connection may complete its current request and become available. If so, we should just use this idle connection instead of waiting for the new connection to finish its connect. This should help request latency, and probably reduce the number of connections created during burst usage as well (see #43764). More details: When this happens, the new connection can still be put in the pool as idle, even if we don't have a request for it at the moment. If another new request comes in, we will use it like any other idle connection. Note also that we should avoid creating new connections when we already have enough pending connections to (eventually) handle all the pending requests. Consider this scenario, starting with no connections: Request A arrives, we initiate a connect for Connection 1, wait for it, and start processing A on it Request B arrives, we initiate a connect for Connection 2. Before the connect completes, A finishes and Connection 1 becomes available. We start processing B on Connection 1. Request C arrives. There's no established connection for it to use, but there is a pending connection (Connection 2). So there's no need for us to initiate another connection right now. Request D arrives. Now, we have 0 available connections, 1 pending connection (Connection 2), and 2 pending requests (C and D). So we should go ahead and initiate a new connect for Connection 3.
Author:	geoffkizer
Assignees:	-
Labels:	area-System.Net.Http, tenet-performance
Milestone:	[object Object]

[Tratcher]

Could the same approach be used for HTTP/2 when EnableMultipleHttp2Connections is enabled?

[geoffkizer]

Could the same approach be used for HTTP/2 when EnableMultipleHttp2Connections is enabled?

Potentially, yeah. It's a different set of logic though. The way EnableMultipleHttp2Connections works is a bit wonky today, so we should probably consider this as part of a general cleanup here.

[stephentoub]

Seems like a reasonable thing to do, albeit likely adding some implementation trickiness (in particular when it intersects with a set MaxConnectionsPerServer) to what is largely a straightforward process today.

[gmorris007]

I have been following this subject for a while, and wanted to add a comment about my experience with it for the record. My interest stems from the fact that occasionally I will get “An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full”. We (the company & I) require timely polling of data so that is bad news, obviously. I did not have this problem with the previous iteration which used HttpWebRequest/HttpWebResponse objects. From what I've read, HttpClient is supposed to be the better option, so our firm has invested time in researching it and switching over.
I am using HttpClient directly, not the factory. The Windows Service application I am working on polls embedded systems, and makes several requests per IP, sequentially (awaited), to get different bits of data. The URI is always the same, with different POST data to tell the device what to return. I can see using netstat that several connections are open to the same IP & port, most sitting at TIME_WAIT status. There are multiple instances of this Windows Service running, each using different groups of IP addresses, so are essentially running like multiple threads. This can add up to a lot of simultaneous requests.
I have tried using these suggestions:
httpClient.DefaultRequestHeaders.ConnectionClose = true;
(and leaving it unassigned)
httpClientHandler.MaxConnectionsPerServer = 1;
(and leaving it unassigned)
These have had no effect. My supposition is that the sockets won't get reused until the TIME_WAIT expires, but I am confused. If ConnectionClose is not used, why is it not reusing the HttpClientHandler and the last socket it had open?
When Connection: Keep-Alive is in the request header (ConnectionClose is false or null), I am seeing the application side close the connection after each request (FIN-ACK, etc.). Shouldn't the application side be waiting for further requests? I think I see the problem: the remote device has "Connection: close" in its header. For the record, the device manufacturer says that keep-alive should NOT be used, and it appears that they enforce this on their end if the host side doesn't. My guess is that this forces the socket into the TIME_WAIT state. It seems that, after the connection is torn down, TIME_WAIT should be aborted, so the socket can be reused.
Note that I am not an HTTP expert, so I could be making bad assumptions here. Thanks for your forbearance in advance.

Here is a tail section of "netstat -p tcp -n" output. I have removed lines that are not related to our app.
TCP 192.168.163.8:58179 10.239.128.67:80 TIME_WAIT
TCP 192.168.163.8:58207 10.239.128.67:80 TIME_WAIT
TCP 192.168.163.8:58259 10.239.128.67:80 TIME_WAIT
TCP 192.168.163.8:58314 10.239.128.85:80 TIME_WAIT
TCP 192.168.163.8:58328 10.239.128.85:80 TIME_WAIT
TCP 192.168.163.8:58368 10.239.128.86:80 TIME_WAIT
TCP 192.168.163.8:58387 10.239.128.86:80 TIME_WAIT
TCP 192.168.163.8:58448 10.234.72.67:80 TIME_WAIT
TCP 192.168.163.8:58498 10.234.72.85:80 TIME_WAIT
TCP 192.168.163.8:58564 10.234.72.86:80 TIME_WAIT
TCP 192.168.163.8:58612 10.234.72.87:80 TIME_WAIT
TCP 192.168.163.8:58631 10.234.72.87:80 TIME_WAIT
TCP 192.168.163.8:58651 10.235.44.67:80 TIME_WAIT
TCP 192.168.163.8:58661 10.235.44.67:80 TIME_WAIT
TCP 192.168.163.8:58682 10.235.44.67:80 FIN_WAIT_1
TCP 192.168.163.8:58700 10.212.196.67:80 LAST_ACK
TCP 192.168.163.8:58706 10.212.196.67:80 SYN_SENT
TCP 192.168.163.8:58708 10.235.44.85:80 SYN_SENT

[wfurt]

The default is to re-use connection for HTTP1/1 and above. Client can indicate it explicitly or it can state that that it prefers to close the connection. However, server can choose to close as well regardless and it seems like this is happening in your case. AFAIK there is really nothing you can do aside from fixing or replacing the server. (or frontload it with load balancer so clients can keep persistent connection to it while the balancer handles short connections to the server it self)
Generally all OSes have setting to control duration of TIME_WAIT state. You can crank it down to enforce falser expiration but that is unrelated to HttpClient it self.
Also note this all above is unrelated to this issue.

[gmorris007]

@wfurt Would I be better off using HttpWebRequest/HttpWebResponse objects for this application, since the server wants a new connection for every request? Perhaps this is not an ideal scenario for use of HttpClient. We do not have access to the servers to add load balancers. They are all over the United States.