Comment by blauwbilgorgel | Hacker News Reader

Comment by blauwbilgorgel | Hacker News Reader

I do use UTF-8 for my entire web stacks, but I encode characters in their special characters:

Iñtërnâtiônàlizætiøn

Becomes:

Iñtërnâtiônàlizætiøn

When outputted to the browser.

Can anyone tell me if that is correct and sane?

Translating characters like & and € to & and € saves me a lot of hassle with validation: http://validator.w3.org/check?verbose=1&uri=http%3A%2F%2...

  Line 265, Column 190:
  non SGML character number 157
  <code>â€œIÃ±tÃ«rnÃ¢tiÃ´nÃ lizÃ¦tiÃ¸nâ€</code>

  You have used an illegal character in your text. HTML 
  uses the standard UNICODE Consortium character 
  repertoire, and it leaves undefined (among others) 65 
  character codes (0 to 31 inclusive and 127 to 159 
  inclusive) that are sometimes used for typographical 
  quote marks and similar in proprietary character sets. 
  The validator has found one of these undefined characters 
  in your document. The character may appear on your 
  browser as a curly quote, or a trademark symbol, or some 
  other fancy glyph; on a different computer, however, it 
  will likely appear as a completely different character, 
  or nothing at all.

  Your best bet is to replace the character with the 
  nearest equivalent ASCII character, or to use an 
  appropriate character entity. For more information on 
  Character Encoding on the web, see Alan Flavell's 
  excellent HTML Character Set Issues reference.

  This error can also be triggered by formatting characters 
  embedded in documents by some word processors. If you use 
  a word processor to edit your HTML documents, be sure to 
  use the "Save as ASCII" or similar command to save the 
  document without formatting information.


  Line 344, Column 79: 
  cannot generate system identifier for general entity "src"
  An entity reference was found in the document, but there 
  is no reference by that name defined. Often this is 
  caused by misspelling the reference name, unencoded 
  ampersands, or by leaving off the trailing semicolon (;). 
  The most common cause of this error is unencoded 
  ampersands in URLs as described by the WDG in "Ampersands 
  in URLs".

Sure that works for some things but

1. You know there are no entity names for most unicode characters? Eg Chinese. You may as well use the numeric entity codes 2. In XML the entity names other than lt gt amp and quot are not defined unless you have a dtd, so you should not use them across an xml api, eg for an Atom feed, or across an xml web service, unless it defines a dtd including them which is unlikely. 3. If you get those errors, it is because you have something set up wrong. Those things are fixable. Fixing them will help you understand whats going on better. As the article says, get out your hex...

Thank you very much!

If I understand correctly, you are saying to drop the entity names and start using the numeric entity codes. This shouldn't be much of a problem.

I really did bump into problems with an RSS feed and entity names, so another great point. I solved that by wrapping it in <![CDATA[ ]] and using the numeric entity codes (ë becomes ë), so now I am wondering why I am even mixing entity names and numeric entity codes in the first place.

Break your habit of converting them to HTML entities. Put them in your source as actual UTF-8 characters.

If your page is really being delivered as UTF-8 then it will pass all validation, just using the real characters. (You still have to escape & as & of course.)

Here, I put together a little example for you: http://50pop.com/i18n.html

View source to verify. Click the validate link.

Hope that helps.

The error does not come from “Iñtërnâtiônàlizætiøn”, but from â€œIÃ±tÃ«rnÃ¢tiÃ´nÃ lizÃ¦tiÃ¸nâ€, which contains U+009D character (correctly encoded and delivered as UTF-8). Apparently that character is not allowed in HTML documents. So & is not the only character you need to escape, and seeing that even you didn't know this detail, I don't think it's a bad habit to play on the safe side and just escape all non-ascii (printable) characters.

edit: I read the HTML5 spec, it says: "Text must not contain control characters other than space characters". So a reasonable solution would be to pass all printable characters as UTF8 and encode control characters. But as I said, I'd prefer to err on the side of caution, in this case encode more than necessary if I'm not sure exactly which characters need encoding and which do not.

No, your problem is that the UTF-8 encoding of U+009D isn't 9d, it's c2 9d. So if you're encoding it as 9d, you're not writing out UTF-8, you're writing out latin-1, which of course leads to displaying random characters. Serve your page as utf-8 and encode it properly.

Who cares what validators say. You should use UTF. We're long past the point where web browsers don't support it.

I care what validators say or imply. The reason I started using UTF-8 in the first place, was because it is a W3C accessibility guideline to use UTF-8 over the more common (in that time) iso-8859-1.

If you want to create accessible websites, one of the first requirements is validated code.

If we ignore validators and the W3C, who is there to officially tell us what we _should_ do?

We are long past the point, sure, so much so, that W3C recommends it too.

And about browser support: If you want to guarantee that most browsers understand and support your code, your best bet is to adhere to the W3C that wrote the standard.

Get a better validator: http://validator.nu/

The W3C validator works fine. Please see this example: http://50pop.com/i18n.html

Maybe your webserver is not serving your page as UTF-8?

Reply on news.ycombinator.com