Or maybe I've just been fortunate enough to be in an environment where an occasional goof of this caliber doesn't have any serious consequences.
Or maybe I've just been fortunate enough to be in an environment where an occasional goof of this caliber doesn't have any serious consequences.
Just last week, my co-worker had to waste two days debugging a two-year-old Sphinx setup (the person who implemented it no longer works with us) because a Japanese user of our blogging service wasn't seeing the post he was looking for in our search feature. The problem was that the conduit feeding the Sphinx indexer was handling Unicode incorrectly (to be specific, it was deleting certain bytes wholesale because this guy believed them never to be valid, and this broke multi-byte sequences horribly). Those two days would have been much better spent working on his current project.
Additionally, not handling Unicode correctly can leave you open to certain types of security holes!
The biggest of these is probably that Unicode means _a string is not an array of bytes_, so naïve allocators for languages with byte-array strings (read: C and its brethren) are susceptible to buffer overruns when handed multi-byte Unicode sequences when they're expecting ASCII.
Here's another one: the Unicode character space contains many, many glyphs that look almost (or in some cases exactly) like ASCII characters. RFC 3492 defines a framework for internationalized domain names (IDN) that encodes non-ASCII code points using ASCII characters, and the most common implementations of this transparently go between the two. This means that you could register "bаnkofamerica.com" (actually xn--bnkofamerica-x9j.com) and put a phishing site there, and people would happily click on the identical-looking URL and give you their bank account. This was pointed out several years ago and most modern browsers have mechanisms in place to defend against it, but your custom application might not unless you're careful to check what you're doing.
There are plenty of blog posts and articles out there designed to tell you how to be safe when dealing with Unicode (and you should assume that you will be). I highly suggest you go read one.
Primarily, this means you don't have to support internationalization - which is hardly a bad thing, especially if you work on a startup, where worldwide distribution should be the last thing on your mind. When your product is rendered in over 80 scripts, including right-to-left languages, you can't afford to figure encoding will sort itself later.
So, if the two bodies have differing encodings (charsets), then the HTML body will look wrong. Unless you force Outlook to always use UTF-8 for encoding emails (which is a setting, but not the default) then you'll end up sending emails that will look garbled to your recipient.
This "differing charset" scenario actually happens pretty frequently, because of the following scenario:
a) You write an email (or reply to an existing email - actually it happens most with replies).
b) Outlook's text editor decides to insert a non-breaking space (codepoint U+00A0). Perhaps it generates HTML with but before transmission this eventually turns into the single UTF-8 character 0xC2 0xA0.
c) When generating the text-body, Outlook decides to just use a plain old space, so the text body is plain ASCII.
d) Outlook, in its cleverness, then says "ooh, I can 'conserve' encoding-ness and use plain old iso-8859-1 for the text body, but I need to use UTF-8 for the HTML body because of that non-ascii character"
e) Outlook generates this email (please excuse formatting woes due to HN).
Content-Type: multipart/alternative; boundary="0016e64dbd929784310488b2b082"
This is a multi-part message in MIME format.
--0016e64dbd929784310488b2b082
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="ISO-8859-1"
yo yo
--0016e64dbd929784310488b2b082
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<html> <body>
yo=C2=A0yo
</body> <html>
--0016e64dbd929784310488b2b082--
When you view the above email in Outlook, you see "yo yo" instead of "yo yo"
Such mistakes were excusable in the 90s. They're the sign of an amateur programmer today.