Hacker News Reader

Comment by geofft | Hacker News Reader

Oh, I see. But if it was UTF-8 it would have worked... I guess the problem is that JetBrains is generating the file in (e.g.) Windows-1252, and Python needs to be told that?

Does it work if you set the environment variable PYTHONENCODING to cp1252?

(I suppose I should either contact the author, or try it myself...)

View on HN

JetBrains is generating an invalid YAML file, which are UTF-8. If they were using a decent YAML library, it would have crashed at that point. And firmly pointed the finger at the real bug, reading raw bytes from the environment or a .properties file parser and assuming it is valid UTF-8.

And this is why you always validate your data when you slurp it in, or else you pass crap down several layers where it crashes or mostly works with the potential for security holes or catastrophic behavior, and a pain in the arse to track down since the actual bug is nowhere near where you are looking.

I'd expect a decent YAML library to have functions taking UTF-8 and not wasting time verifying that the data passed is actually UTF-8 in release builds.

You generally don't verify on output, because you verified on input (especially with languages where text strings are Unicode or UTF-8 byte strings like Python3 or Rust). But it would also be a premature optimization when it does make sense to check. For expected YAML use cases I doubt it would be a measurable difference in runtime. And it has to inspect the strings in any case to correctly quote things and deal with indentation if there are newlines.

Normally I'd agree, a windows-1252 misencode should be one's default guess when mojibake is afoot. Unfortunately, the errant byte in the error is 0xb3, which is "³" in windows-1252.

If you Google, "Mikołaj",

> Mikołaj is the Polish cognate of given name Nicholas

Then Google, "windows character encoding polish"

> Windows-1250 - Wikipedia

And 0xb3 is "ł" in that encoding.¹

> Does it work if you set the environment variable PYTHONENCODING (sic) to cp1252?

I don't know if setting PYTHONIOENCODING would work here; I don't think it should affect this. Really, fixing the YAML file is the fix. (And fixing the thing that generated it.)

¹it is queries like this that really make me love the search engines of today. This would have been hell in the days of Alta Vista.

Reply on news.ycombinator.com