cookie header is not supporting utf-8 encoding

[MathieuB1]

Bug report

Hi everyone,

When passing headers that contain utf-8 characters (in my case a Cookie), it failed to encode the correct charset because stuck on latin-1 encoding.

File "/usr/lib/python3.9/http/client.py at line 1228
values[i] = one_value.encode('latin-1')

Can you please fix this one and set the default encoding as utf-8? I don't see why the 'latin1' charset have been set. It should be related to this change 8f36af7

Many thanks in advance

Your environment

python3.9

[michaelfm1211]

I'm not so sure about this. First of all, RFC 9112 explicitly warns about using Unicode characters as a potential security risk (however, I don't know enough about Python's implementation to know if it really would, but it's generally better to be safe). Additionally this Stack Overflow answer (although ten years old) says latin-1 (aka: ISO-8859-1) is the largest encoding that's ever been supported, so all of UTF-8 is definitely not supported by the standard.

If we really did want to allow Unicode characters, then I think it'd be better to automatically MIME encode/decode (RFC 2047) with UTF-8 (here's a quick summary from Wikipedia if you're not familiar with this).

In your case of using cookies, you could also alternatively just URL encode the cookies and get rid of this error (using urllib.parse.quote or urllib.parse.urlencode). By doing some research on Stack Overflow, this looks like the general consensus on how to solve the issue you have. I'm not aware of any language/framework that URL encodes cookies for you, so I'm not sure it'd be particularly useful for Python to do it either.

[MathieuB1]

Thanks for the investigation, after reading (RFC 2047) you mentionned,
I see that it could be possible to allow utf-8 encoding inside of headers
by using =?utf-8? (encoded-words) syntax. But I can't see this implementation inside of http/client.py as its a thing dedicated to mail.

In my opinion the default encoding set at iso-8859-1 (Latin-1 see RFC 5997 ) should be reviewed as
there is not only European caracters that are used by people in headers. For a security perspective yes it allows more characters to be decoded, but on Devs side we will have this error when trying to be full utf-8 (headers included) on a request:

UnicodeEncodeError: 'latin-1' codec can't encode character '\ufffd' in position 1582: ordinal not in range(256)

In fact its not really what is expected when you have your OS, Server and your Proxy all set to UTF-8 charset.
The cookie seems more like an exception for supporting only ISO-8859-1 and ASCII.
Also I would be curious to see the behavior with Java and C++ servers handling headers in UTF-8.

[michaelfm1211]

I agree, it would make sense to have the http module support Unicode via encoded-word, considering Unicode is the default throughout most of the standard library. I actually realized http.client doesn't handle encoded-word at all, even if it's receiving those headers from the server. Getting encoded-word in a header might be surprising and confusing to a beginner, so at the very least, we should handle it when parsing headers in responses.

Yesterday I opened a separate issue on this encoded-word bug alone (gh-105530), and wrote this PR to attempt to fix the issue. It ended up failing some tests beyond the http module (not sure why, but I'll try to fix it up later today), but that's one possible resolution to this issue.

Lastly, I agree that gauging how other languages/servers handle Unicode in servers and clients would be helpful to determine the best course of action here in Python.