Windows service infrequently results in error 1067 on stop operation

[zryska]

Description

.NET 6 published application sometimes crashes on service stop operation. I created a web application from the template and only added the Microsoft.Extensions.Hosting.WindowsServices (version 6.0.0) package and called UseWindowsService() to make sure the application listens for service controller requests.

Here is the code:

using Microsoft.Extensions.Hosting.WindowsServices;

var options = new WebApplicationOptions
{
    Args = args,
    ContentRootPath = WindowsServiceHelpers.IsWindowsService() ? AppContext.BaseDirectory : default
};

var builder = WebApplication.CreateBuilder(options);

// Add services to the container.
builder.Services.AddRazorPages();
builder.Host.UseWindowsService();

var app = builder.Build();

// Configure the HTTP request pipeline.
if (!app.Environment.IsDevelopment())
{
    app.UseExceptionHandler("/Error");
}

app.UseStaticFiles();

app.UseRouting();

app.UseAuthorization();

app.MapRazorPages();

app.Run();

Reproduction Steps

Repository with code, script and repro steps: https://github.com/zryska/net6-repro

1. Create a new Web application using a template
2. Make the changes to add UseWindowsService() call as shown above
3. Publish the application as SFD or FDD for win-x64 runtime
4. Create a new service using sc create command
5. Start the service and verify it's running
6. Stop the service
7. Start the service
8. Repeat steps 4 and 5 until the issue is reproduced

It can be done through the services UI or using a script. On my system it usually happens 1 time out of 10 graceful shutdowns - but it isn't deterministic.

When done through services UI the following will be displayed:

Even log => Windows Logs -> System will contain the following message (when done through script or UI):

The .NET6 test service service terminated unexpectedly.  It has done this 26 time(s).

Expected behavior

Service stops gracefully every time.

Actual behavior

Non-graceful shutdowns are happening.

Regression?

No response

Known Workarounds

Explicitly assigning value to ServiceName fixes the problem.

builder.Host.UseWindowsService(o => o.ServiceName = "AnyServiceName");

Configuration

.NET version: .NET 6.0.100
OS version: Windows 10 pro build 19044.1348
Architecture: x64

Other information

No response

Tagging subscribers to this area: @dotnet/area-extensions-hosting
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

.NET 6 published application sometimes crashes on service stop operation. I created a web application from the template and only added the Microsoft.Extensions.Hosting.WindowsServices (version 6.0.0) package and called UseWindowsService() to make sure the application listens for service controller requests.

Here is the code:

using Microsoft.Extensions.Hosting.WindowsServices;

var options = new WebApplicationOptions
{
    Args = args,
    ContentRootPath = WindowsServiceHelpers.IsWindowsService() ? AppContext.BaseDirectory : default
};

var builder = WebApplication.CreateBuilder(options);

// Add services to the container.
builder.Services.AddRazorPages();
builder.Host.UseWindowsService();

var app = builder.Build();

// Configure the HTTP request pipeline.
if (!app.Environment.IsDevelopment())
{
    app.UseExceptionHandler("/Error");
}

app.UseStaticFiles();

app.UseRouting();

app.UseAuthorization();

app.MapRazorPages();

app.Run();

Reproduction Steps

1. Publish the application as SFD or FDD for win-x64 runtime
2. Create a new service using sc create command
3. Start the service and verify it's running
4. Stop the service
5. Start the service
6. Repeat steps 4 and 5 until the issue is reproduced

It can be done through the services UI or using a script. On my system it usually happens 1 time out of 10 graceful shutdowns - but it isn't deterministic.

When done through services UI the following will be displayed:

Even log => Windows Logs -> System will contain the following message (when done through script or UI):

The .NET6 test service service terminated unexpectedly.  It has done this 26 time(s).

Expected behavior

Service stops gracefully every time.

Actual behavior

Non-graceful shutdowns are happening.

Regression?

No response

Known Workarounds

No response

Configuration

.NET version: .NET 6.0.100
OS version: Windows 10 pro build 19044.1348
Architecture: x64

Other information

No response

Author:	zryska
Assignees:	-
Labels:	untriaged, area-Extensions-Hosting
Milestone:	-

[eerhardt]

Any chance you can get more information here to narrow down where the problem is? Maybe try getting a crash dump.

https://github.com/dotnet/diagnostics/blob/main/documentation/dotnet-dump-instructions.md

Or adding a try-catch somewhere in your project to try to catch the exception.

Can you attach a Debugger before stopping the service? If you do, does it still crash?

[zryska]

I wasn't able to get a crash dump generated for some reason. Also when I included try-catch block around the main method it didn't catch anything. The application seems to be crashing after it leaves the main method. When I run the same code on Linux just with systemd package instead of WindowsServices the same code works just fine.

[eerhardt]

Your best bet here is to get more information - what is the stacktrace / message / etc that is causing the process to crash.

[zryska]

The issue is not surfacing when I have debugger attached. Would you be able to give it a try on your end by following the repro steps or is there something else I could provide in terms of repro to make it easier?

[eerhardt]

I wasn't able to get a crash dump generated for some reason

Was dotnet-dump not working for you? What was the reason?

[zryska]

My understanding was that dotnet-dump allows only to generate a dump at time of executing the collect command. I was using SysInternals ProcDump but it didn't create any crash dump even when service manager reported crash.

This is procdump output from event when service controller reported ungraceful shutdown:

PS C:\Users\zryska\Tools\Procdump> .\procdump64.exe -ma -e WebApplication1.exe

ProcDump v10.11 - Sysinternals process dump utility
Copyright (C) 2009-2021 Mark Russinovich and Andrew Richards
Sysinternals - www.sysinternals.com

Process:               WebApplication1.exe (41180)
Process image:         C:\Program Files\Test\WebApplication1.exe
CPU threshold:         n/a
Performance counter:   n/a
Commit threshold:      n/a
Threshold seconds:     n/a
Hung window check:     Disabled
Log debug strings:     Disabled
Exception monitor:     Unhandled
Exception filter:      [Includes]
                       *
                       [Excludes]
Terminate monitor:     Disabled
Cloning type:          Disabled
Concurrent limit:      n/a
Avoid outage:          n/a
Number of dumps:       1
Dump folder:           C:\Users\zryska\Downloads\Procdump\
Dump filename/mask:    PROCESSNAME_YYMMDD_HHMMSS
Queue to WER:          Disabled
Kill after dump:       Disabled


Press Ctrl-C to end monitoring without terminating the process.

[22:27:46] Exception: E0434352.CLR
[22:27:46] Exception: E0434352.CLR
[22:27:46] Exception: E0434352.CLR
[22:27:46] Exception: E0434352.CLR
[22:27:46] Exception: 00000006
[22:27:46] Exception: E0434352.CLR
[22:27:46] Exception: E0434352.CLR
[22:27:47] The process has exited.
[22:27:47] Dump count not reached.

[eerhardt]

C:\Users\zryska\Downloads\Procdump\

There's no dump files in that folder?

Can you try https://docs.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dumps? Or possibly https://docs.microsoft.com/en-us/visualstudio/debugger/debug-using-the-just-in-time-debugger?view=vs-2022#BKMK_Enabling?

[zryska]

No. No dumps were created. I should also mention that the main method apparently exists gracefully since in case I add a line of code to create a file before exiting main the file gets created even when the service control manager reports that the application crashed. I was trying the registry approach in the past when I was suspecting that something is wrong in our code application without results but I can give it a try again with the empty WebApplication project.

[davidfowl]

Look at the event log to se if there's more information.

[zryska]

Hi David, unfortunately the event log doesn't contain anything actionable. The alleged crash event is logged in the system log and there is nothing in the Application log around that time.

Here is the raw event (removed computer name) in case there is something I missed:

 - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" /> 
  <EventID Qualifiers="49152">7034</EventID> 
  <Version>0</Version> 
  <Level>2</Level> 
  <Task>0</Task> 
  <Opcode>0</Opcode> 
  <Keywords>0x8080000000000000</Keywords> 
  <TimeCreated SystemTime="2021-12-10T08:15:28.6705360Z" /> 
  <EventRecordID>105858</EventRecordID> 
  <Correlation /> 
  <Execution ProcessID="1632" ThreadID="29084" /> 
  <Channel>System</Channel> 
  <Computer>ComputerName</Computer> 
  <Security /> 
  </System>
- <EventData>
  <Data Name="param1">.NET6 test service</Data> 
  <Data Name="param2">7</Data> 
  <Binary>2E004E0045005400360020007400650073007400200073006500720076006900630065000000</Binary> 
  </EventData>
  </Event>

[zryska]

I tried to replicate it on another machine with lower HW specs:
CPU: 4 cores
RAM: 8 GB
OS: Windows 10 Enterprise build: 19042.1348

It still happened but much less frequently. I let the script run for a while and out of 1000 restarts there were just two "crash" events reported in the event log.

For comparison my development machine HW specs:
CPU: 16 cores
RAM: 32 GB
OS: Windows 10 Pro build: 19044.1348

I moved the repro code into a separate repository if it helps: https://github.com/zryska/net6-repro

[OwnageIsMagic]

@zryska use procdump64.exe -mp -t -e 1 -x . "WebApplication1.exe"