All My Homies Hate Text Encoding

kurtisknodel.com

7 pointsArch4854 comments

I suppose the problem is not perl being lied to, but storing unicode (utf-8?) in a field designated as latin-1. That’s bound to cause issues down the line.

Well, the database was lying to Perl, perhaps.

But I'm not sure the author's diagnosis is correct. The mangled text has only one "?", Which means that whatever software did the mangling knew that two-byte sequence represented a single character. Maybe whatever was reading from the database was(I'm assuming MySQL since 1990s vintage) using a non-Unicode-capable connection?

Oh absolutely! Unfortunately we can't just change our databases text encoding... Luckily the database doesn't change any of the data, so it should be okay with some fixing.

If I had a dollar for every page on the internet that advertises UTF-8 and provides Latin-1, I could buy Microsoft.

Reply on news.ycombinator.com