Comment by lmm | Hacker News Reader

Comment by lmm | Hacker News Reader

Parent Full thread

> While I believe we should have a standard, I feel that having UTF-8 in source code brings more risks than benefits. [0]

> Next to those additional risks it also limits discoverability if `fiancee` now is written as `fiancée` (yea I had to c/p that from somewhere). Searching for the former does not discover the latter.

If your search function can't handle different characters with the same meaning then it's already broken (e.g. usually you want search to be case insensitive). Likewise if similar-looking characters are an exploitable problem in your codebase then that problem already exists with e.g. l/I.

> Lastly, there is the issue of Intellisense, or whatever it is called in many languages. I have seen codebases in English with the aforementioned `é` in a function name. The only way for me to select that function name was with the arrow keys / mouse. I couldn't type it on my QWERTY.

In any decent system you can type it with compose-'-e like it looks like.

I'm all for making programmers dogfood the use of non-ascii characters (actually I'd like to see more use of non-unicode encodings, unless and until unicode stops screwing over Japanese). Maybe it'll give them more empathy for those of us who need to type things like this daily, and stop them doing things like flashy new packaging systems that break IME.

> unless and until unicode stops screwing over Japanese

Can you say more about this? Are there still reasons to prefer shiftjis over unicode for japanese characters?

They might be alluding to issues from CJK unification. But I don’t believe that many practitioners working with Japanese text consider these concerns more important than the value that Unicode brings to the table.

I think that CJK unification leads to awkwardness where some fonts will basically render some kanji the “Chinese way” or the “Japanese way”, so you can’t really have a good font that covers both at once. There’s a good aesthetic argument about not using the same font for both languages, but Unicode outright precludes it I think

Virtually every Japanese organisation disagrees, you only have to look at a Japanese website to know as much (or are those somehow not "practitioners working with Japanese text"?). Most will use a non-Unicode encoding and/or resort to rendering important text as images, since it's impossible to rely on Unicode text rendering well.

I wonder if this can ever be fixed. If basically every Japanese organization disagrees with the Unicode consortium to the point that its undermining the use of unicode in entire countries, it might be worth unwinding CJK normalization.

This isn't the case obviously. Japanese organisations are involved in the Unicode process (something evidenced in the input for the several CJK Unified Ideograph Extensions). Thank to Unicode, many historical characters can be represented in digital text without resorting to placeholders or hacks.

Anecdotally, I've visited a group enthusiastically exploring the possibilities of the then-new CJK Unified Ideograph Extension B in Tokyo over a decade ago. The CJK issue mentioned here was a bigger problem then than now (most developers know how to handle it properly, e.g. with language declarations in HTML etc.).

People will use legacy encodings. That's just a fact of life. Sometimes out of inertia (i.e., it just works, so why bother changing it?), sometimes out of technical limitations of their pipeline, sometimes because of a lack of understanding, rarely out of principle. I'm stuck fixing an issue with some numb skull sending us SAML requests declared as UTF-8 containing some obsolete encoding instead, and this is for Dutch text! This just happens.

There is also nothing wrong with text in Shift-JIS, if that encoding encompasses all characters used.

From here [0], 94.1% of .jp websites use UTF-8 and 6.5% use Shift-JIS. You can find a list of popular sites still using Shift-JIS here [1].

If you change your PC or your phone locale to Japanese and access mostly Japanese website, the font display on most major OS works well. It's probably fair to assume that's a very common setup for Japanese users, and probably for Chinese and Korean users as well.

I think the CJK "problem" is more apparent for international users who don't have the locale setup or dealing with multiple languages plain text at the same time.

[0] https://w3techs.com/technologies/segmentation/tld-jp-/charac...

[1] https://w3techs.com/technologies/details/en-shiftjis

Your own link shows that both Shift-JIS and EUC-JP are on an upward trend, which isn't what we'd expect if Unicode was succeeding. I can't see it listing examples of .jp sites using UTF-8; would be interesting to see how many of them are relying on text-as-images.

Yeah what lmm says is not what happend/thought in Japan. Japanese still uses Shift_JIS because they love (/s) legacy software. Using Shift_JIS isn't a practical solution.

Random sample of some websites: - Rakuten Bank -> Shift JIS - NHK, the Ministry of Health and Labor Welfare, yahoo japan -> UTF-8

I don't really really know, but this random yahoo answers [0] page seems to agree with me that really you're gonna use UTF-8 for new stuff in general

One point that answer brings up that I hadn't thought of: older flipphones would have good shift-JIS support but not necessarily good unicode support

My guess is that "text as images" might partly be that, partly what you are saying in a legacy sense, and also partly for the same reason old sites in the US would do that: to make something look the way you want and you already have a photoshop render [0]: https://detail.chiebukuro.yahoo.co.jp/qa/question_detail/q14...

Ironically my company sends letters fully rendered as raster images, ugh.

> Can you say more about this? Are there still reasons to prefer shiftjis over unicode for japanese characters?

Yes. Unicode uses codepoints that are primarily for Chinese characters to represent Japanese characters that they consider equivalent, even when those Japanese characters have different appearences; as a result, Japanese text in unicode looks bad (readable, but ugly) unless displayed in a specifically Japanese font (in which case you'd have the converse problem of Chinese text looking wrong). The unicode consortium suggests various vaporware approaches to combat this, but the thing that actually works is keeping Japanese text in Japanese encodings and Chinese text in Chinese encodings. (Of course this means that you need to be able to display text from multiple encodings in the same page if you want to display both languages on the same page, but all of the unicode consortium vaporware fixes require you to build something equivalently complex, and you wouldn't even be able to test it by using strings from two western encodings since it would be specific to Japanese and Chinese)

What's wrong with just wrapping UTF-8 text in lang spans? Browsers and Electron will just do the right thing, no?

Even assuming that works, getting it to make its whole way through your whole tech stack is no easier (and more HTML-specific) than having spans with their own encoding.

Since when can HTML pages have multiple encodings at the same time?

This isn't about multiple encodings, this is about the one encoding, UTF-8, being able to represent multiple languages. Which is mostly can, except for han characters.

In my reply here, I can type in english and russian at once. Привет, мир.

Yet, if I try to type chinese on one line, and japanese on the next, I cannot do it. Hacker news does not let me enter "lang" tags, so I can only type either the chinese or the japanese variant of a kanji.

Parent post says

> spans with their own encoding.

so yes it is about pages with multiple encodings. A span's smaller than a page!

The Unicode answer is "variation selectors" which are used for some historical variant kanji, but not for whole language switching. I suppose they could be used for that too though.

I don't know about HTML specifically; I meant spans as a general concept rather than a literal <span> tag. If the web stack has actually implemented mixing languages on the same page to the point where you can use it in a "normal" application then that's very cool (and if they've done it with their lang tags rather than by allowing mixed encodings, well, fine), but I've yet to see a site that actually has that up and running.

I can't read it, but there will be hundreds of articles on Japanese Wikipedia covering Chinese literature etc that have text in both languages, all in Unicode.

Had a look, they have a macro for it. Cool!

Also,

  <html lang="ja">
    Japanese text ... 
    <span lang="zh">
      Chinese quote
    </span>
    ... 
  </html>

is much easier than mixing encodings. With the above entirely in Unicode, it will be handled reliably by anything that can handle Unicode, and is still reasonably readable even if the Chinese text is shown in a Japanese font. Reading just the fourth line without the third will still show something 'OK'.

Mixing Shift_JIS and Big5 sounds like a recipe for corruption, but something similar was done in an old Russian and Japanese encoding: https://en.wikipedia.org/wiki/Shift_Out_and_Shift_In_charact...

RFC 2482 was a Unicode adaptation of this, but it was deprecated 12 years ago: https://www.rfc-editor.org/rfc/rfc6082.html

There's no difference between encoding a page in Shift_JIS and tagging the page as Japanese language (assuming your browser switches fonts correctly based on language tags). Chinese text in Shift_JIS isn't going to look correct either.

Sure, but every application handles encoding (or at least, every application did up until the recent UTF-8-only movement), whereas language tagging is web/html-specific.

If you're really hardcore processing multilingual text you need an equivalent to language tags anyway, because you need a dictionary for word-wrapping, date formatting, quote marks etc along with changing fonts (or glyph selectors in the same font.) TeX has them too.

But usually people only care about their language, so it goes by the system UI language and it works out.

> If you're really hardcore processing multilingual text you need an equivalent to language tags anyway, because you need a dictionary for word-wrapping, date formatting, quote marks etc along with changing fonts (or glyph selectors in the same font.) TeX has them too.

Sure. And once you're doing that you don't gain a lot of benefit from unicode AFAICS, because you have to track these spans of locale-specific text.

Lets you copy characters that are the same (like people's names) between text though.

People's names are the thing you should be most careful to not do that with! Most Japanese people will generally put up with seeing the Chinese rendering of a character in ordinary text, but they really don't want you to do that to their names!

I think the suggested alternative though is having UTF-8 take care of this.

UTF-8 already has different codepoints for every other language, like I can type russian (День), greek (Ημέρα), english, even egyption hieroglyphs (𓀃) all of them in this one text field, and they all render right for both of us if we have a suitable font. If they don't render, it's unambiguous.

It's only chinese and japanese where I can type some characters, and depending on if you have a chinese or japanese font _first_ in your system, it might render wrong, and either way it's ambiguous to the computer without further metadata. The computer doesn't need metadata to know that "α" is "α", why does it need metadata to know if 直 is the chinese (http://www.hanzi5.com/bishun/76f4.html) or japanese (https://kanjivg.tagaini.net/viewer.html?kanji=%E7%9B%B4) character, two different looking characters that share the same codepoint.

Unicode does not have different codepoints for every language. Leaving aside scripts that aren't encoded at all, all languages using the Latin script get a single set of glyphs, and if you want Comic Sans MS or Carolingian miniscule or fine-grained control over whether "a" has a hook at the top or not, you need to specify a font that renders it the way you want, just like with 直.

Because Unicode is about the meaning of characters, not their appearance. If there were a context where the two different shapes of 直 have different meaning, Unicode would add a new codepoint to distinguish them. In fact "ɑ" without hook does have a separate codepoint because it is used in linguistics to mark a vowel different from "a".

If CYRILLIC SMALL LETTER A deserves a code point despite being so similar to LATIN SMALL LETTER A, why doesn't a Japanese character that actually looks different from a Chinese one get a codepoint? And the idea that something can have the "same meaning" across two languages is very wooly.

Why don't Japanese characters in seal script or handwriting get different codepoints from mincho font characters? They look different too.

Sometimes looking visually different doesn't matter because if you know how to write them by stroke order, you'll still be able to read it (which is how handwriting works, I think; I'm pretty bad at reading that…)

Hanzi simplification in the Mainland also complicated things, since I doubt they wanted to make all of those into different characters.

> flashy new packaging systems that break IME.

...what?

Snap/Flatpak/etc. often mean you can't use IME (input method extensions) with programs packaged with them unless the packager has jumped through certain hoops, which they usually haven't.

Ew, that's... extremely unfortunate. I'm sure there's some internal reason, but that feels like something the system should be able to add support for unilaterally, at least by default, without needing special packager effort:(

You can use the character map however, as character map copies the output to clipboard.

Reply on news.ycombinator.com