Comment by Animats | Hacker News Reader

Comment by Animats | Hacker News Reader

Something they should have mentioned: put a

    <meta charset="utf-8" />

in all your HTML documents that are in UTF-8. Note that this has to be in the first 1024 bytes of the document. Otherwise, the browser has to invoke the "encoding guesser"[1], which will sometimes guess wrong. (W3C: "The user agent may attempt to autodetect the character encoding from applying frequency analysis or other algorithms to the data stream.") The result will be occasional users seeing random pages in the wrong encoding, depending on browser, browser version, platform, and page content.

I recently saw the front page of the New York Times misconverted because they didn't specify an encoding, and the only UTF-8 sequence near the beginning of the document was in

   <link rel="apple-touch-icon-precomposed" sizes="144×144" ...

The "×" there is not the letter x, it's the Unicode multiplication symbol. This confused an encoding guesser. Don't go there.

[1] http://www.w3.org/TR/html5/syntax.html#determining-the-chara...

You can also add "; charset=utf-8" to your Content-Type header instead.

Yes, but then it's separated from the document. If someone saves the file, they lose the charset info.

How reliable are CDNs and caching servers about preserving Content-Type headers?

I have to admit I didn't consider the use case of saving the HTML page.

Regarding CDNs, I know that Cloudflare passes them as-is, I don't know about the others.

While this is true, I find the meta tag to be a horrible pain in the ass.

If you have to parse some HTML that you get over an HTTP connection - you're writing a crawler, say, or you want to extract RDFa metadata, you have to deal with the following, surprisingly common case: both the header and the html document contain encoding information, and they disagree. The RFC states that you should trust the header, but in practice, that's certainly not always the case - nor even, in my experience, is it the case the majority of the time.

If you decide to use the meta tag, that means that you have already strarted decoding your byte stream, get the encoding, then need to re-interpret the bytes you've already read. I have seen a lot of pages that declared their encoding after the title tag.

What's worse, you can't know whether you have a meta tag until you've parsed the whole head, which can be huge with hundreds of kilobytes of inlined javascript and css.

The argument that you should just read the first 1024 bytes and assume utf-8 if nothing is found is just not satisfactory - I want the encoding of the documents I'm parsing to be correct all the time, not when the remote host follows the rules. If I'm writing a crawler, the remote host cares nothing about my needs and I'm the one suffering from my unwilingness to be flexible.

So, yeah. Don't use the meta encoding tag, and trust your user agent to save the html code in a sane (utf-8) encoding. There is no reason to store encoding information in an html file, just like I doubt your source code always starts with a preprocessor instruction declaring the encoding that the compiler should use.

> There is no reason to store encoding information in an html file.

That's simply wrong. If you use libraries like D3.js, that contain non-ASCII characters in the source code, and you do local development with a server that sends no encoding headers or even without using a server at all, your code won't work.

That is, as far as I know, not true.

If your file is in html5, then all browsers will assume a default encoding of utf-8, which is what you should be using anyway - unless you have a very good reason, such as your file contains a majority of kanji or kana.

If you're using something older, the official defaut is iso latin 1, but I believe all modern browsers will try utf-8 first - this is not something I've verified for myself, it works on my setup but I also configured my os to use a default utf-8 encoding, so I can't tell for sure.

I'm also unclear how this relates to d3.js - that's a javascript import, not an html one. Or do you mean inline javascript that uses d3.js?

Hang on isn't this argument basically: "lots of people don't follow the standards now so no one should bother?"

Well, not exactly. If people truly followed the standards, there would be no need for the meta charset element: the RFC clearly states that encoding should either be specified in the header or default to iso latin 1. I can't recall whether it makes provisions for media type specific default charsets, but either way, if you follow the HTTP standard, you should not specify content encoding in your text document (this of course does not apply to binary formats that migh encode text).

So, to be a bit pedantic about it, my argument is that you should follow the standards and ignore / work around the hacks used to make life easier for people that don't know / don't care about encoding.

Note that I do not mean that as condescending - at some point, a lot of designers were writing HTML manually, and I don't expect them to know about encoding, just the same as they hopefuly don't expect me to know about... design stuff I'm really terrible at.

Fair enough. Not condescending. I don't know much about HTML tbh.

However, if I were faced with your situation I would try to use whatever logic is used by firefox or chromium to work out encoding. After all designers are going to (should) test if things work on one/both of these right?

Reply on news.ycombinator.com