Http server performance regression in net6 preview 7

[hanabi1224]

Description

I have a small benchmark program that runs fast with netcore3.1, net5, net6 preview6 and prior (take 1~5 seconds), but the same program runs super slowly in net6 preview 7 (several minutes ~ infinite)

Configuration

Regression?

I can confirm it works fine on netcore3.1, net5, net6 preview 6 and prior

Data

It takes 1~5 seconds normally but several minutes ~ infinite with net6 preview 7

Analysis

N/A

Code

using System;
using System.Collections.Generic;
using System.IO;
using System.Net;
using System.Net.Http;
using System.Reflection;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;

static class Program
{
    private const string MimeType = "application/json";
    private static readonly HttpClient s_client = new HttpClient() { Timeout = TimeSpan.FromSeconds(15) };

    public static async Task Main(string[] args)
    {
        int n;
        if (args.Length < 1 || !int.TryParse(args[0], out n))
        {
            n = 2000;
        }

        var port = 30000 + new Random().Next(10000);
        var server = CreateWebHostBuilder(port).Build();
        using var cts = new CancellationTokenSource();
        _ = Task.Factory.StartNew(() => server.Start(), TaskCreationOptions.LongRunning);
        var sum = 0;
        var api = $"http://localhost:{port}/";
        var tasks = new List<Task<int>>(n);
        for (var i = 1; i <= n; i++)
        {
            tasks.Add(SendAsync(api, i));
        }
        foreach (var task in tasks)
        {
            sum += await task.ConfigureAwait(false);
        }
        Console.WriteLine(sum);
        System.Environment.Exit(0);
    }

    private static async Task<int> SendAsync(string api, int value)
    {
        var payload = JsonConvert.SerializeObject(new Payload { Value = value });
        while (true)
        {
            try
            {
                var content = new StringContent(payload, Encoding.UTF8, MimeType);
                var response = await s_client.PostAsync(api, content).ConfigureAwait(false);
                return int.Parse(await response.Content.ReadAsStringAsync().ConfigureAwait(false));
            }
            catch { }
        }
    }

    private static IWebHostBuilder CreateWebHostBuilder(int port)
    {
        return WebHost.CreateDefaultBuilder()
            .SuppressStatusMessages(true)
            .ConfigureLogging((context, logging) =>
            {
                logging.ClearProviders();
            })
            .UseKestrel(options =>
            {
                options.Limits.MaxRequestBodySize = null;
                options.ListenLocalhost(port);
            })
            .UseStartup<Startup>();
    }
}

public sealed class MyController : Controller
{
    [Route("/")]
    public async Task<int> PostAsync()
    {
        using var sr = new StreamReader(Request.Body);
        var bodyText = await sr.ReadToEndAsync().ConfigureAwait(false);
        var payload = JsonConvert.DeserializeObject<Payload>(bodyText);
        return payload.Value;
    }
}

public class Payload
{
    [JsonProperty("value")]
    public int Value { get; set; }
}

public sealed class Startup
{
    public Startup(IConfiguration configuration)
    {
        Configuration = configuration;
    }

    public IConfiguration Configuration { get; }

    public void ConfigureServices(IServiceCollection services)
    {
        services.AddMvcCore().AddApplicationPart(Assembly.GetExecutingAssembly());
    }

    public void Configure(
        IApplicationBuilder app,
        IWebHostEnvironment env)
    {
        app.UseRouting();
        app.UseEndpoints(endpoints =>
        {
            endpoints.MapControllers();
        });
    }
}

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[JanEggers]

while (true) this will always run forever ?!

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I have a small benchmark program that runs fast with netcore3.1, net5, net6 preview6 and prior (take 1~5 seconds), but the same program runs super slowly in net6 preview 7 (several minutes ~ infinite)

Configuration

Regression?

I can confirm it works fine on netcore3.1, net5, net6 preview 6 and prior

Data

It takes 1~5 seconds normally but several minutes ~ infinite with net6 preview 7

Analysis

N/A

Code

using System;
using System.Collections.Generic;
using System.IO;
using System.Net;
using System.Net.Http;
using System.Reflection;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;

static class Program
{
    private const string MimeType = "application/json";
    private static readonly HttpClient s_client = new HttpClient() { Timeout = TimeSpan.FromSeconds(15) };

    public static async Task Main(string[] args)
    {
        int n;
        if (args.Length < 1 || !int.TryParse(args[0], out n))
        {
            n = 2000;
        }

        var port = 30000 + new Random().Next(10000);
        var server = CreateWebHostBuilder(port).Build();
        using var cts = new CancellationTokenSource();
        _ = Task.Factory.StartNew(() => server.Start(), TaskCreationOptions.LongRunning);
        var sum = 0;
        var api = $"http://localhost:{port}/";
        var tasks = new List<Task<int>>(n);
        for (var i = 1; i <= n; i++)
        {
            tasks.Add(SendAsync(api, i));
        }
        foreach (var task in tasks)
        {
            sum += await task.ConfigureAwait(false);
        }
        Console.WriteLine(sum);
        System.Environment.Exit(0);
    }

    private static async Task<int> SendAsync(string api, int value)
    {
        var payload = JsonConvert.SerializeObject(new Payload { Value = value });
        while (true)
        {
            try
            {
                var content = new StringContent(payload, Encoding.UTF8, MimeType);
                var response = await s_client.PostAsync(api, content).ConfigureAwait(false);
                return int.Parse(await response.Content.ReadAsStringAsync().ConfigureAwait(false));
            }
            catch { }
        }
    }

    private static IWebHostBuilder CreateWebHostBuilder(int port)
    {
        return WebHost.CreateDefaultBuilder()
            .SuppressStatusMessages(true)
            .ConfigureLogging((context, logging) =>
            {
                logging.ClearProviders();
            })
            .UseKestrel(options =>
            {
                options.Limits.MaxRequestBodySize = null;
                options.ListenLocalhost(port);
            })
            .UseStartup<Startup>();
    }
}

public sealed class MyController : Controller
{
    [Route("/")]
    public async Task<int> PostAsync()
    {
        using var sr = new StreamReader(Request.Body);
        var bodyText = await sr.ReadToEndAsync().ConfigureAwait(false);
        var payload = JsonConvert.DeserializeObject<Payload>(bodyText);
        return payload.Value;
    }
}

public class Payload
{
    [JsonProperty("value")]
    public int Value { get; set; }
}

public sealed class Startup
{
    public Startup(IConfiguration configuration)
    {
        Configuration = configuration;
    }

    public IConfiguration Configuration { get; }

    public void ConfigureServices(IServiceCollection services)
    {
        services.AddMvcCore().AddApplicationPart(Assembly.GetExecutingAssembly());
    }

    public void Configure(
        IApplicationBuilder app,
        IWebHostEnvironment env)
    {
        app.UseRouting();
        app.UseEndpoints(endpoints =>
        {
            endpoints.MapControllers();
        });
    }
}

Author:	hanabi1224
Assignees:	-
Labels:	area-System.Net.Http, tenet-performance, untriaged
Milestone:	-

[ManickaP]

Hi @hanabi1224, in your while loop, you have an empty catch block. Could you change it to something like:

catch (Exception ex)
{
    Console.WriteLine(ex);
}

and report back whether some errors are happening there?

[hanabi1224]

@ManickaP There're tons of errors below, if the input parameter is smaller, like 20 or 200, it works just fine, but with 2000, it never finishes (only in net6 preview 7 tho).

   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in System.Net.Http.dll:token 0x600044c+0x1723
   at System.Net.Http.HttpConnectionPool.SendUsingHttp11Async(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in System.Net.Http.dll:token 0x6000714+0x12a
   at System.Net.Http.HttpConnectionPool.DetermineVersionAndSendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in System.Net.Http.dll:token 0x6000715+0x224
   at System.Net.Http.HttpConnectionPool.SendAndProcessAltSvcAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in System.Net.Http.dll:token 0x6000716+0x8e
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in System.Net.Http.dll:token 0x6000717+0x99
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in System.Net.Http.dll:token 0x60007c1+0x94
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in System.Net.Http.dll:token 0x6000235+0xae
   at Program.SendAsync(String api, Int32 value) in /tmp/c/app.cs:line 71
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: The response ended prematurely.
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in System.Net.Http.dll:token 0x600044c+0xf0e
   --- End of inner exception stack trace ---

[hanabi1224]

while (true) this will always run forever ?!

@JanEggers There's a return inside the loop, it just keeps retrying when error happens

[JanEggers]

didnt catch that sry

[ManickaP]

but with 2000, it never finishes (only in net6 preview 7 tho).

I'm able to reproduce it locally with 6.0.0-rc.1.21411.2.

I also tested the same thing with the current main (~rc2) and it didn't not reproduce.
So this might be something we've fixed in the meantime, or it's Debug/Release related (my build is Debug).

Interestingly, Wireshark shows requests going out, but server never replies (the POST handler is never hit). And since I replaced only runtime libs (not ASP.NET ones) this is suspicious.

In the hanging case I see this as last log message on Kestrel (all connections get accepted, only one gets started and that's the end):

dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[1]
      Connection id "0HMB3721RTE2S" started.

cc @Tratcher can you thing of anything that could cause this?

[hanabi1224]

or it's Debug/Release related

@ManickaP FYI I always run this test in Release mode

[Tratcher]

@ManickaP can you share the wireshark trace and kestrel logs from your repro?