Repository navigation
HTTP Error 500 with substatus 1011/1013 and win32 error 109 (ERROR_BROKEN_PIPE) #57
Description
On some of our servers our users are experiencing occasional internal server errors which we are unable to explain. According to our logs some requests fail with a broken pipe, but these requests are no different from other requests which work fine. We have now implemented a retry-mechanism, and so far requests that failed do work on the first retry. This seems to confirm that there is nothing 'wrong' with the requests themselves.
We have tried to create a minimal example (see files below), and performed some tests:
| Node version | IISNode version | NODE_PENDING_PIPE_INSTANCES | Result |
|---|---|---|---|
| 7.3.0 | 0.2.18.0 | Undefined | Works (in practice). There are no errors, at least long as there are not "too many" concurrent requests. However, when it fails, it fails with error 500.1003 (which is a different error, which we have not seen, "The service is unavailable"). The number of concurrent requests needed to cause this is sufficiently large that we don't expect to have hit this in practice. |
| 9.3.0 | 0.2.18.0 | Undefined | Error 500.1011 / 500.1013 on some requests. |
| 10.7.0 | 0.2.18.0 | Undefined | Error 500.1011 / 500.1013 on some requests. |
| 7.3.0 | 0.2.21.0 | 5000 | Works perfectly. I get errors on my PC (regarding fork limit) before the server starts to give error 500s. |
| 9.3.0 | 0.2.21.0 | 5000 | Error 500.1011 / 500.1013 on some requests. |
| 10.7.0 | 0.2.21.0 | 5000 | Error 500.1011 / 500.1013 on some requests. |
I used the following minimal example, on an up-to-date Windows Server 2016 Version 1607 (OS Build 14393.2248) installation with IIS (Version 10.0.014393.0). The below files were placed in C:\Error1013. In this directory I placed 3 node executables with different versions (see above table). On a different PC we
run the 'poll.sh' script to test concurrent requests.
The example web server is very simple, and consists only of a short delay (1/8th of a second) in order to simulate processing time otherwise taken by our service.
It is our understanding that with NODE_PENDING_PIPE_INSTANCES set to the default value of 5000, we should not run into resource limits regarding concurrent requests until we reach 5000 concurrent requests. However, are getting error 500.1013 / 500.1011 with as few as 50 concurrent requests. Is this a known issue (e.g. #50)? In our case it does not seem to be related to the processing time of a request, as each simulated request takes only 1/8th of a second.
web.config
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<system.webServer>
<iisnode nodeProcessCommandLine="C:\Error1013\node_10.7.0.exe"
loggingEnabled="true"
devErrorsEnabled="false"
enableXFF="true"
promoteServerVars="HTTP_URL"
logDirectory=".\logs"
/>
<handlers>
<add name="iisnode" path="hello.js" verb="*" modules="iisnode" />
</handlers>
<rewrite>
<rules>
<rule name="nodejs">
<match url=".*" />
<action type="Rewrite" url="hello.js" />
</rule>
</rules>
</rewrite>
</system.webServer>
</configuration>package.json
{
"name": "my-awesome-package"
, "version": "1.0.0"
, "main": "hello.js"
, "dependencies": {
"http": "*"
}
}hello.js
var http = require('http');
const util = require('util');
var seconds = 0.125;
var startUp = new Date();
http.createServer((req, res) => {
res.writeHead(200, {'Content-Type': 'text/plain'});
var waitTill = new Date(new Date().getTime() + seconds * 1000);
while(waitTill > new Date()){}
res.end(`${startUp} process.env.NODE_PENDING_PIPE_INSTANCES: ${process.env.NODE_PENDING_PIPE_INSTANCES}\n`);
}).listen(process.env.PORT);poll.sh
#!/bin/sh
server=192.168.22.254:8080
# This breaks sometimes, but not very often
concurrent_requests=30
inter_request_delay=0.05
# This breaks almost every time
concurrent_requests=50
inter_request_delay=0.01
# # We've gotta push it harder!
# concurrent_requests=250 # 500 gives fork error on my pc
# inter_request_delay=
while true ; do
echo "Press <enter> to send ${concurrent_requests} requests (delay: ${inter_request_delay:-none})..."
read l
i=0; while [ ${i} -lt ${concurrent_requests} ] ; do
curl "${server}" &
i=$((i+1))
[ -n "${inter_request_delay}" ] && sleep ${inter_request_delay}
done
# concurrent_requests=$((concurrent_requests+10))
echo "Waiting for requests to finish..."
wait
echo "Done"
echo
done
Activity
MBDeveloper commentedon Sep 20, 2018
Did you find the solution or workaround to this issue?
Cellcote commentedon Nov 20, 2018
We are also running into this problem. Any idea for a fix?
MBDeveloper commentedon Nov 20, 2018
Try change the setting for your node server:
server.keepAliveTimeout = 0; //default = 5000
see nodejs/node#13391
nodejs/node#13655
PG-Devs commentedon Dec 4, 2018
We've tried quickly tried this workaround, and it does seem to help.
To confirm that this is the same problem; when we set the keepAliveTimeout to a very small value, such as 50, the problem triggers quite often, but not as often as we'd expect. I.e. in the test script, there is a processing delay of 1/8s. 50ms is less than that, so we'd expect every request to cause an error. However, most requests still work fine.
Is this expected behaviour?
Also, we are a bit hesitant to set the keepAliveTimeout to 0 (infinite). Will the problem still be solved if we set the keepAliveTimeout to a large value, such as 1 minute, or does it really need to be set to 0?
MBDeveloper commentedon Dec 4, 2018
see node documentation for this property (value of 0 just disable this new feature):
https://nodejs.org/api/http.html#http_server_keepalivetimeout
tony-gutierrez commentedon May 10, 2019
PG-Devs What did you decide to do? Disable server keep alive? I am in a similar situation.
MBDeveloper commentedon May 12, 2019
yes
PG-Devs commentedon May 13, 2019
Yeah, same here (disabled the keep alive).
tony-gutierrez commentedon May 13, 2019
I wasn't on a node version that had the 5 second bug, but disabling keep alive seemed to have solved the problem anyway (at least in the last 24 hrs). I wonder if this is a problem in iisnode or what?
nandowalter commentedon Oct 27, 2021
Can confirm that this workaround fixed mine app service HTTP Error 500 too. Our app runs on top of node.js 14.x and it's an Angular 12.x app.