[flagged]
Bonus - as soon as the Unicode consortium do find a way, my software should be able to handle it with no further changes. Well, it might need a recompile against a newer `libicu` as I don't think they maintain ABI backcompat between versions. But there's not much I can do about that.
And it indeed facilitates, as in practice it works better than encoding in utf-8 that lacks an in-band way to communicate that, and out-of-band often fails/is ignored/doesn't exist.
In theory yes. In practice it doesn't.
> just like it would have with a Japanese-centric encoding.
The difference is that encoding-aware systems naturally use Japanese fonts for Japanese encodings and other fonts for other encodings.
Systems like what?
Switching your entire encoding system to set a different font is by far the stupidest way to do it.
Encoding-aware programs, UI toolkits, etc..
> Switching your entire encoding system to set a different font is by far the stupidest way to do it.
If it's stupid and it works, it's not stupid. I wish there were other reliable ways to have international programs display Japanese correctly, but there aren't.
Name just one please.
To be clear, we're talking about a program/library that can handle both unicode and shift-JIS, and it will render a character that unicode considers identical in different ways depending on what encoding you loaded the character from, right?
Firefox (not the best example for a number of reasons, if you're going to follow up then I'll talk about a different one, but if you really want just one then it's the program I have to hand right now).
> To be clear, we're talking about a program/library that can handle both unicode and shift-JIS, and it will render a character that unicode considers identical in different ways depending on what encoding you loaded the character from, right?
Yes.
Huh, today I learned.
Though if you're making any attempt to use valid tags you'll have <html lang="ja">, and that solves the problem for the web context at least as far as my testing goes.
Right, which is why that's a bad example. But it's the same for most "normal" applications (even e.g. text editors, or I remember hitting this kind of thing in WinRAR), and a lot of the time there isn't a standard way of indicating the language/locale in the file. Even within firefox, there are (admittedly rare) cases where you're viewing something that isn't HTML and doesn't have HTTP headers so using the encoding or manually setting the page language is still the only way to make it work - and applications that have a manual "file language" setting are the exception rather than the rule.
Realistically --- if someone hands me well marked Shift JIS content, I'm just going to reencode it as UTF-8 anyway... And if they hand me unmarked Shift JIS content, I'll try to see if I can decode it as UTF-8 and throw it away as invalid if not.
Trivia: There are some variants of ShiftJIS where this isn't entirely true. The traditional yuugen gaisha symbol, for example, which is analogous to U+32CF (LTD), is not supported. The /VENDORS/APPLE/JAPANESE.TXT file uses a PUA designator and then a sequence of four Unicode code points to convert it.
You'd think that, but in practice I've found the opposite. Applications that use encodings managed to display things properly. Applications that hardcode UTF-8 don't.
If you implement only unicode then you put yourself in a situation where Japanese is uniquely different from every other language, and your program will not work properly for Japanese unless you tested Japanese specifically.
Only among people who don't care. Implementing encoding support means your app supports a way to display Japanese properly. If you want to add more ways to display Japanese properly, go ahead, but that's supererogatory, whereas UTF8-only apps don't have a way to display Japanese properly at all.
> to support Japanese, you have to do some extra work to get and use extra info about a language, which is also possible within UTF8
It's not. There's some imaginary theorycrafted way in which it might notionally be possible within UTF-8, but not one UTF-8-only app has ever actually implemented support for displaying Japanese properly. The only approach to displaying Japanese properly that has ever actually been implemented in reality is to support multiple encodings (or to support only a Japanese encoding, but that has obvious downsides), and if you make your app encoding-aware then that's enough, you don't have to do anything else to be able to display Japanese properly for people who care about displaying Japanese properly (it's always possible for an app maker to go above and beyond, but I'm sure you'll agree the big difference is between an app that has a way to display proper Japanese at all and one that does not).
I know there's a weird Chinese/Japanese encoding problem where characters that kind-of look alike have the same character id, and the font file is responsible for disambiguation (terrible for multi-language content and we should really add more characters to create versions for each, but still the best we have).
[1] section 1.2 in https://www.fontucom.com/pdf/AFSARIBR-Map.pdf
For example, if I want to talk about the fairly rare japanese name '刃一' (jinichi), there's a chance your computer displays it correctly, but there's also a chance your computer displays the chinese variant of the first character, making it look wrong. It's basically up to font-choice of your computer.
The "correct" way to fix that would be for me to be able to tag it with 'lang=ja', but Hacker News doesn't let me include html tags or some other 'language selector' in my comment, so I'm unable to indicate whether that's supposed to be the chinese or japanese variant of the character.
Most unicode text files don't have extra metadata indicating if a certain bit of text is japanese or chinese, so displaying it correctly by adding the correct 'lang' tag is impossible, especially since it's perfectly possible for one utf-8 text to mix both chinese and japanese.
See https://heistak.github.io/your-code-displays-japanese-wrong/
However, a unicode codepoint which acted as a "language hint" switch would solve this, and wouldn't require doubling the number of han codepoints.
There already are unicode variant selectors, but iiuc they only apply to a single character at a time, and no one actually uses them, so they're not very useful.
Only if you have some unicodebrained mentality where you consider a Chinese character that looks sort of similar to a Japanese character to be "the same". If you think of them as two different characters then they're just different characters, which may or may not be present in particular encodings (which is completely normal if you make a program that handles multiple encodings: not every character exists in every encoding)
> In which case, why is it unreasonable to ask for the same for UTF-8, except using some more clearly specified way to indicate this (like lang="ja" etc), rather than encoding it all into separate characters?
Firstly, you have to "draw the rest of the fucking owl" and actually implement that language selection mechanism. Secondly, if you implement some clever extension mechanism on top of UTF-8 that's only needed for Japanese and can only really be tested by people who read Chinese or Japanese, realistically even if you implement it perfectly, most app makers won't use it or will use it wrong. Whereas if you implement encoding-awareness in the standard way that we've been doing for decades and test with even one non-default encoding, your application will most likely work fine for Japanese, Chinese and every other language even if you never test it with Japanese.
Of course, there is no correct solution (people in other comments seem to believe it exists a bit too quickly). Dynamic HTML page can load pieces of text written by people all across the world, it can't rely on any “main” language. Those people can't be automatically classified (based on some browser settings or location). Then there are always people who know both Chinese and Japanese (while using some other system locale, to make things more complex). It is wrong to assume that they should not be able to use Chinese forms and Japanese forms at once, even inside a single paragraph or phrase.
I wonder why Unicode has not simply introduced some “combining language form character” to make user choice stick. After all, there's a whole subsystem for emoji modification, and those things were once “weird Japanese texting customs”. As for complexity of handling Unicode text, it asymptotically reaches its maximum anyway.
Wait a second, there are “variation selectors” and some “Ideographic Variation Database”. Is it the solution? Can IMEs and converters simply stamp each variable character with invisible mark based on current input language? I suppose there's some catch…
It's the opposite of vaporware; there's a whole bunch of well-known software that handles encodings correctly.
> new code that doesn't care about handling some lang metadata is not going to care about Shift JIS either.
If you support encodings the way almost every programming language tells you too, you'll handle Shift-JIS just fine. If you use the "legacy" encodings approach and test even one non-English language you'll handle Japanese fine.
> Can IMEs and converters simply stamp each variable character with invisible mark based on current input language? I suppose there's some catch…
They're officially deprecated, IMEs and converters don't use them, naïve search implementations (which is to say most search implementations, because people do not test for an obscure edge case) break, ....
As for official stance, maybe it's time for some group to agree on certain non-conflicting sequences, and implement them in some library/stack. A particularly evil solution would choose arbitrary existing combining characters that can't be declared “illegal” retroactively.
By the way, is it possible to make proper quotes from other languages as described above when using Shift JIS?
I assure you there are thousands if not millions of apps that implement search the naive way, by comparing bytes.
> It won't handle emoji, normalization, and thousands other things.
Yeah. But it works well enough for Americans, so the maintainers don't care enough to fix it.
> As for official stance, maybe it's time for some group to agree on certain non-conflicting sequences, and implement them in some library/stack.
And then what? What do you do with documents in Japanese that have already been converted without using your new codepoints? What do you do with programs that ignore your new standard? What do you do about search being broken in most apps with no prospect of fixing it? There are already some PUA codepoints for a proper implementation of Japanese, but most app authors don't even understand the problem, never mind being willing to support something that's "non-standard". Asking them to support traditional non-unicode encodings, which is something that's at least relatively well-known and standardised, and something they can test without knowing Japanese, is much easier.
> By the way, is it possible to make proper quotes from other languages as described above when using Shift JIS?
No, not for arbitrary languages. If you want to mix arbitrary languages, you need some structure that's a rope of strings with an encoding per segment. But that's exactly the same thing that the unicode advocates claim is easy ("just have a sequence of spans with different lang tags"), and at least if you use traditional encodings then you don't have to do any such fancy business for the much more common case of a file that's entirely in Japanese (perhaps with some ASCII keywords).
We are already doing GSUB/GDEF tables and a lot of horrible stuff to display pretty glyphs on the screen. Hell, there are literal instructions for VM in TTF files to help with pretty rasterization on low-res screens.
Making font rendering library is hard, Nintendo hard.
That's just a nature of the beast. Fonts are messy and if we want one standard to deal with it once and for all, that means some compromises must be done. CJK is not some obscure alphabet. In this case, perfect is enemy of good.
I am just glad there really is only one (non-niche) standard, obligatory https://xkcd.com/927/
Right. One would think that misrendering a major world language used by over 100 million people would be an issue that warrants some attention. But too many of the HN crowd don't care.
> In this case, perfect is enemy of good.
It's not though. Unicode-only is not just imperfect, it's an outright regression for Japanese. Meanwhile traditional encoding-aware programs render Japanese just fine.
Unicode has variant selectors which can deal with region variants (e.g. https://imgur.com/a/syMcWNO) to deal with the issue.
Granted, it's not very widely used, but Unicode provided a solution. The onus is now on application developers and font providers.
Choosing to limit number of characters to 2 bytes was a good technical choice back in 1992, non-unified CJK wouldn't fit (that was a time of 2-4MB RAM and 100-200MB HDD). A solution to problem was later provided.
Using Shift-JIS is like doing on-IP, non-UDP protocol on the internet.
Sure, you can do it, but it's just non-perspective choice. A better invested time is on improving standard protocol, which is Unicode (e.g. conversion software).
They're officially deprecated and cause issues with breaking search etc.. So they're still below feature parity with using a traditional encoding. Traditional encoding support is also easier to test since documents with traditional encodings are more widespread and exist for many world languages, not just Japanese.
And even back in the mid 2000s, the only real problems I saw otherwise, were things like malformed html pages that assumed a specific encoding that they wouldn't tell you, or an MP3 file with an ID3 tag with CP932 shoved into it against the (v1) spec.
I also disagree with the author that Shift-JIS can be "good enough" hueristically detected due to its use of both 7 and 8-bit characters in both the high and low bytes to mean different things depending on what character is actually intended. Even string searching requires a complex custom-made version just for Shift-JIS handling.