HTTP header encoding

[samsp-msft]

The HTTP spec is weak on what encoding is used for headers. Some treat it as UTF-8 others as ASCII/Latin1. There should be a way to control what encoding is used when converting headers to/from string.

Tagging subscribers to this area: @dotnet/ncl
Notify danmosemsft if you want to be subscribed.

[samsp-msft]

Should consider in relation to #35126.
@Tratcher - what is ASP.NET doing for this?

[Tratcher]

See dotnet/aspnetcore#17400

[karelz]

This used to work on .NET Framework -- from compat perspective we will have to enable that - maybe as an opt-in config, or API, maybe as default.

[karelz]

@MihaZupan can you please look at a repro? I think we need to investigate these things on .NET Framework 4.8 and .NET Core 3.1/5.0, how they behave today to make decision how we want them to behave in 5.0:

1. What happens if we pass byte ("Latin1"-encoded character) casted to char in the range of 0x80-0xff in request headers. What do we send over the wire on .NET Framework? (Wireshark/Fiddler may be needed) + Confirm .NET Core throws.
2. What happens if we get these bytes over the wire from server in response headers? How do we present them in our APIs? As byte casted to char, or do we try to do some decoding?
3. What happens if full UTF16 character is passed to request headers? How does it look over the wire?
4. What happens if full UTF16 character is received from server in response headers? What if 2nd byte is end-of-line terminator for headers?
5. What happens if multi-byte UTF8 character is received from server in response headers? What if 2nd/3rd/4th bytes are end-of-line terminator for headers?

(I might have missed some interesting cases)

We will need repro for test case in .NET 5.
Looking at .NET Framework code is bonus, but not necessity at this point.

All this info will help us decide how sane the .NET Framework behavior is and how to expose it in .NET Core -- as default behavior, as context-switch opt-in behavior, or as new API.