Differences in response body encoding on JRuby

[janko]

When I run the following:

require "http"

response = HTTP.get("http://httpbin.org/image/png")
response.body.each do |chunk|
  puts chunk.encoding
end

MRI outputs ASCII-8BIT, whereas JRuby outputs UTF-8 encodings. The same thing happens when I swap the endpoint with http://httpbin.org/encoding/utf8.

I'm note sure what should be the correct general behaviour, whether chunks should always be in binary encoding, or only when charset isn't specified. But it seems that me that in the latter case the encoding should always be binary, both in MRI and JRuby.

I'm not sure yet whether this cross-Ruby inconsistency comes from HTTP.rb or http_parser.rb, but just wanted to report it here. I think a good solution would be to call force_encoding on the result of Response::Body#readpartial (just like we're doing in Response::Body#to_s).

[ixti]

It's definitely on our end (not http_parser.rb, as it's only responsible for headers parsing). And I agree that chunks must be either in encoding specified in headers, or binary (I believe we had that somewhere).