System.Net.Sockets.Tests crashing in CI with Assertion failed: (numEvents != 0)

[ViktorHofer]

https://dev.azure.com/dnceng/public/_build/results?buildId=644902&view=ms.vss-test-web.build-test-results-tab&runId=20019118&resultId=179234&paneView=attachments

core.28801:
https://helix.dot.net/api/2019-06-17/jobs/2170593e-910f-4178-93ef-eef8b273f366/workitems/System.Net.Sockets.Tests/files/core.28801

console.25c1d1b8.log:
https://helix.dot.net/api/2019-06-17/jobs/2170593e-910f-4178-93ef-eef8b273f366/workitems/System.Net.Sockets.Tests/files/console.25c1d1b8.log

Configuration: net5.0-OSX-Debug-x64-CoreCLR_release-OSX.1013.Amd64.Open

  Discovering: System.Net.Sockets.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Sockets.Tests (found 801 of 1201 test cases)
  Starting:    System.Net.Sockets.Tests (parallel test collections = on, max threads = 4)
Assertion failed: (numEvents != 0), function WaitForSocketEventsInner, file /Users/runner/runners/2.168.2/work/1/s/src/libraries/Native/Unix/System.Native/pal_networking.c, line 2845.
./RunTests.sh: line 161: 28801 Abort trap: 6           (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Sockets.Tests.runtimeconfig.json --depsfile System.Net.Sockets.Tests.deps.json xunit.console.dll System.Net.Sockets.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/private/tmp/helix/working/9DF008FB/w/CACB0A25/e
----- end Thu May 14 08:31:14 PDT 2020 ----- exit code 134 ----------------------------------------------------------
exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.

cc @dotnet/ncl

Tagging subscribers to this area: @dotnet/ncl
Notify danmosemsft if you want to be subscribed.

[wfurt]

I've seen this long time ago so probably it is not regression.

[wfurt]

The code looks like

    while ((numEvents = kevent(port, NULL, 0, events, GetKeventNchanges(*count), NULL)) < 0 && errno == EINTR);
    if (numEvents == -1)
    {
        *count = -1;
        return SystemNative_ConvertErrorPlatformToPal(errno);
    }

    // We should never see 0 events. Given an infinite timeout, kevent will never return
    // 0 events even if there are no file descriptors registered with the kqueue fd. In
    // that case, the wait will block until a file descriptor is added and an event occurs
    // on the added file descriptor.
    assert(numEvents != 0);
    assert(numEvents <= *count);

I look at the Darwin kernel code:
https://github.com/apple/darwin-xnu/blob/a449c6a3b8014d9406c2ddbdc81795da24aa7443/bsd/kern/kern_event.c

It is curious that the kevent_internal() ends like:

static int
kevent_internal(struct proc *p,
		kqueue_id_t id, kqueue_id_t *id_out,
		user_addr_t changelist, int nchanges,
		user_addr_t ueventlist, int nevents,
		user_addr_t data_out, uint64_t data_available,
		unsigned int flags,
		user_addr_t utimeout,
		kqueue_continue_t continuation,
		int32_t *retval)
{

....
....
...

	/* don't restart after signals... */
	if (error == ERESTART)
		error = EINTR;
	else if (error == EWOULDBLOCK)
		error = 0;
	if (error == 0)
		*retval = noutputs;
	return (error);
}

there is section to check timeout

				if (now >= deadline) {
					/* non-blocking call */
					error = EWOULDBLOCK;
					break; /* lock still held */
				}

but also

	case THREAD_TIMED_OUT:
		error = EWOULDBLOCK;
		break;
	case THREAD_INTERRUPTED:
		error = EINTR;
		break;

There may be some other reason but it seems like if EWOULDBLOCK could be caused by something else then timeout value, we may get unexpected results. (and the behavior may not be properly documented)

Could/should we relax the assert or do you have any other thoughts @stephentoub?
I think this one may be very difficult to reproduce but the outcome is quite severe and I don't think this is test bug.

[stephentoub]

If we remove the assert and thus proceed execution with numEvents == 0, what will happen? Seems like we'd then hit this assert:

runtime/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs

Line 312 in 2e1bc9d

Debug.Assert(numEvents > 0, $"Unexpected numEvents: {numEvents}");

and we'll then loop around and call WaitForSocketEvents again. Will this loop end up being a spin loop, constantly calling WaitForSocketEvents, immediately getting back 0, and repeating? Or is it possible we're getting back 0 when, for example, the poll fd has been closed somehow?

[wfurt]

I could see two possible strategies:

• try it again, perhaps with limited iterations.
• treat it as "normal" error and fail the operation instead of Assert can crashing while app.