Hacker News Reader

Comment by hinkley | Hacker News Reader

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters.

Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us we would be better off being able to a assert that a piece of text was German, or sanskrit, not a jumble of both. It's been living rent free in my head for almost two decades and I can't agree with it, nor can I laugh it off.

It might have been better if the 'code pages' idea was refined instead of eliminated (that is, the string uses one or more code pages, not the process). I don't know what the right answer is, but I know Every X is a Y almost always gets us into trouble.

View on HN

> but for most of us we would be better off

That's simple - it is provably wrong. While relatively uncommon there are plenty of examples that would contradict this statement. And it's not about being able to encode the Rosetta Stone - non-scientists mix languages all the time, from Carmina Burana to Blinkenlights. They even make meaningful portmanteau words and write them with characters from multiple unrelated writing systems, like "заshitано" (see - Latin and Cyrillic scripts in the same single word!)

I'm intrigued, what's заshitано?

A portmanteau of Russian "засчитано" ("credited", "taken into account", "check!") and English "shit".

The word is a joke and there is no well-defined meaning. I've seen it used as both "it counts but it's shitty" and as a way to give credit for a failure.

Surely that's not the best example, but it's a word I've remembered.

You miss the point. The basic unit of ASCII v2 (aka 'Unicode') should have been the codepage, not the codepoint. Having a stateful stream of codepage-symbol pairs is not a problem - in practice, all Unicode encodings ended up being stateful anyways, except in a shitty way that doesn't help to encode any semantic information.

Aside from all of the other issues mentioned, for some languages it's not clear what language something is purely based on codepoints.

For languages that have Latin-derived writing systems, it's not uncommon to use English letters (without diacritics) to write the language -- how would that be handled? In addition, thanks to Han unification (though this would've been a problem anyway -- loads of characters would've been unified anyway) all similar CJK Hanzi/Kanji/漢字 characters are mapped to the same codepoint regardless of language. This means that for some sentences it is entirely possible for you to not know whether a sentence fragment is Chinese or Japanese without more context and a native-like understanding of the language.

Also in many languages English words are written verbatim meaning that you can have sentences like (my Japanese is not perfect, this is just an example):

> あの芸人のYouTube動画を見たの？面白すぎるww

And (at least in Japanese) there are loads of other usage of Latin characters aside from loan words that would be too unwieldy to write in katakana -- "w" is like "lol", and fair few acronyms (BGM = Background Music, CM = Advertisement, TKG = 卵かけご飯 = (Raw) Egg on Rice). There are other languages that have similar "issues", but unfortunately I can't give any more examples because the only other language I speak (Serbian) writes everything (even people's names) phonetically.

As an aside -- if anyone ever has to support CJK languages (in subtitles for instance), please make sure you use the right fonts. While Unicode has encoded Han characters with the same codepoint, in different languages the characters are often drawn differently and you need to use the right font for the corresponding language (and area -- Mandarin speakers in different areas write some characters differently -- 返 is different in every CJK locale). Many media players and websites do not handle this correctly and it is fairly frustrating -- the net result is that Japanese is often displayed using Chinese fonts which makes it uncomfortable to read (it's still obvious what the character is, it's just off-putting).

Yeah, for the same reason that CJK designers often use absolutely abhorrent fonts for English words on packaging and printed media, is the same reason Westerners use terrible fonts for CJK. They are working with a language and culture they have no knowledge of, and think that if the characters look sorta right, then they must be readable and look good.

I've made so many horrible localization errors in the past because I had to translate things into 30 different languages and I can only barely read a handful of them, so I just copy and paste whatever the translators give me.

Incidently, I saw Japanese text recently that was quoting both English AND Arabic in the same sentence. And this was in a block of vertical text. That is literally a worst-case scenario I think. You have RTL-vertical text also containing RTL- and LTR-horizontal text. And unlike English, which when placed into vertical Japanese text, you can essentially choose whether you want the characters going vertically or sideways - you can't do that with Arabic as the letters must be joined together, I don't believe you can break them down and stack them on top of each other.

Why can't we all just convert to Esperanto? ;)

> Incidently, I saw Japanese text recently that was quoting both English AND Arabic in the same sentence. And this was in a block of vertical text. That is literally a worst-case scenario I think.

Look up Mongolian script and you might change your mind :)

Oh Lord. I just looked it up on Wikipedia. It's a vertical-only language for a start. How do you even discuss a vertical-only language in a language which is horizontal? Wikipedia has to write all the script horizontally, which would be like a page talking about English stacking all the letters vertically - it's weird.

From Wikipedia: "Computer operating systems have been slow to adopt support for the Mongolian script, and almost all have incomplete support or other text rendering difficulties."

In our Jenkins system, we have remote build nodes return data back to the primary node via environment variable-style formatted files (e.g. FOO=bar), so when I had to send back a bunch of arbitrary multi-line textual data, I decided to base64 encode it. Simple enough.

On *nix systems, I ran this through the base64 command; the data was UTF8, which meant that in practice it was ASCII (because we didn't have any special characters in our commit messages).

On Windows systems... oh god. The system treats all text as UTF-16 with whatever byte order, and it took me ages to figure out how to get it to convert the data to UTF-8 before encoding it. Eventually it started working, and it worked for a while until it didn't for whatever reason. I ended up tearing out all the code and just encoding the UTF-16 in base64 and then processing that into UTF-8 on the master where I had access to much saner tools.

Generally speaking, "Unicode" works great in most cases, but when you're dealing with systems with weird or unusual encoding habits, like Windows using UTF-16 or MySQL's "utf8" being limited to three bytes per unicode character instead of four, everything goes out the window and it's the wild west all over again.

Funnily base64 suffers from a related issue that the likes of base58 correct : l and I or O and 0 looking similar or even identical depending on the font !

Why does that matter? When would a human need to read and comprehend base64 encoded data?

Base58 is used for example for Bitcoin addresses. Being able to type an address from one system to another is a nice property, and it's much less error-prone if you don't have to worry about look-alike characters.

We had to ETL .csv data that must have originated in SQLServer.

The utf-16 fact about Windows was apparently unknown to my predecessor.

Who wrote some nasty c-language binary to copy the data, knock the upper byte off of each character ahead, and save the now ASCII text to a new file of the mysql load.

The encoding='utf-16' argument was all that was needed.

For want of a nail. . .

I've had to fix this before. A co-worker working with data from a 3rd party supplier had gone "Oh this input data is mangled with stray zero bytes, I'll fix that" and of course that destroys any non-ASCII inputs, eventually I'm told that sometimes the import fails, I investigate, and I realise the "mangled" input is just UTF-16 encoded, conditionally remove the "strip zero bytes" hack and tell the decoder it's UTF-16 and it just works correctly.

The "maybe strip null bytes" code lived for years "just in case" after I fixed that because people couldn't believe that's all that was ever "wrong" with the data.

The perils of valuing backwards compatibility above all else… imagine having to use UTF16 in this day and age. Happy 2022!

UTF-16 is a simple encoding. It should take a few dozens of LoC to convert to UTF-8. At least if you don’t need extreme performance with AVX, etc.

> UTF-16 is a simple encoding

This isn't true, for starters it's two encodings UTF-16LE and UTF-16BE.

My experience is exactly like what you say. Until it isn't. UTF16 seems like such a neat idea until it meets reality.

Mostly because of weird interactions of different libraries, languages and operating systems.

You can already map Unicode ranges to "code pages" of sorts, so how would that help?

Thing is, people who are not linguists do want to mix languages. It's very common in some cultures to intersperse the native language with English. But even if not, if the language in question uses a non-Latin alphabet, there are often bits and pieces of data that have to be written down in Latin. So that "most of us" perspective is really "most of us in US and Western Europe", at best.

For domains and such, what I think is really needed is a new definition of string equality that boils down to "are people likely to consider these two the same?". So that would e.g. treat similarly-shaped Latin/Greek/Cyrillic letters the same.

Oh, you can do far more than "code pages of sorts". Unicode has a variety of metadata available about each codepoint. The things that are "code pages of sorts" are maybe "block" (for ö "Latin-1 Supplement"), and "plane" (for ö it's "Basic Multilingual Plane"), but those are really mostly administrative and probably not what want.

But you also have "Script" (for ö "Latin). Some characters belong to more than one script though. Unicode will tell you that.

Unicode also has a variety of algorithms available already written. One of the most relevant ones here is... normalization. To compare two strings in the broadest semantic sense of "are people likely to consider these the same", you want want a "compatibility" normalization. NFKC or NFKD. They will for instance make `1` and `¹`[superscript] the same, which is definitely one kind of "consider these the same" -- very useful for, say, a search index.

That won't be iron-clad, but that will be better than trying to role your own algorithm involving looking at character metadata yourself! But it won't get you past intentional attacks using "look-alike" characters that are actually different semantically but look similar/indistinguishable depending on font. The trick is "consider these the same" really, it turns out, depends on context and purpose, it's not always the same.

Unicode also has a variety of useful guides as part of the standard, including the guide to normalization https://unicode.org/reports/tr15/ and some guides related to security (such as https://unicode.org/reports/tr36/ and http://unicode.org/reports/tr39/), all of which are relevant to this concern, and suggest approaches and algorithms.

Unicode has a LOT of very clever stuff in it to handle the inherently complicated problem of dealing with the entire universe of global languages that Unicode makes possible. It pays to spend some time with em.

Yeah, Greek alphabet is used a lot in sciences. It's really annoying that we're only starting to get proper support now. (Including on keyboards : http://norme-azerty.fr/en/ )

By far the most common use case these days is when an URL has to be mentioned in an otherwise non-Latin text.

I am trying to formalise this with Cosmopolitan Identifiers (https://obua.com/publications/cosmo-id/3/). These identifiers consist of words and symbols. Symbols are normalised based on how they look like, and so Latin / Cyrillic / Greek symbols that look alike are mapped to the same symbol. Words are normalised differently, so that "Tree" and "tree" map to the same normal form. As a symbol, "T" and "t" are obviously different. I am not totally happy with the concept yet, I have implemented a fourth, simpler iteration of that concept as a Typescript package: https://www.npmjs.com/package/cosmo-id .

One of the problems is, how do you distinguish symbols and words? A simple way to do this is to classify something as a symbol if it is just a single character, and as a word otherwise. For example, "α-β" would consist of two symbols, separated by a hyphen, but "αβ" is a word and normalised to "av" based on some convention on how to "latinise" greek words.

You don't need a new definition, you just need to follow the official Unicode security guidelines.

I recommend the Moderate Restrictive Security profile for identifiers for mixed scripts. TR39. Plus allow Greek with Latin. This way you can identify Cyrillic, Greek, CFK or any recommended script, but are not allowed to mix Cyrillic with Greek. And you still can write math with Greek Symbols.

What we don't have are a standard string library to compare or find strings. wcscmp does not do normalization. There is no wcsfc (foldcase) for case insensitivity. There's no wcsnorm or wcsnfc. I'm maintaining such a library.

coreutils, diff, grep, patch, sed and friends all cannot find Unicode strings, they have no string support. They can only mimic filesystems, finding binary garbage. Strings are so rthi g different than pure ASCII or BINARY garbage. Strings have an encoding and are Unicode.

Filesystems are even worse because they need to treat filenames as identifiers, but do not. Nobody cares about TR31, TR39, TR36 and so on.

Here is an overview of the sad state of Unicode unsafeties in programming languages: https://github.com/rurban/libu8ident/blob/master/c11.md

Sprinkling English with foreign words is really, really common. I'm in New Zealand and people do it all the time. And even in the states, right? Don't want two different strings because someone writes an English sentence about how much they love jalapeño.

It’s very common in the online messaging I’ve seen in English, Spanish, Chinese, and Russian.

I'm not a linguist and that will probably be readily apparent. The word jalapeño leaves me wondering how distinct a boundary a language can possess or how one can sort out which language an individual word belongs to outside the context of the rest of the text or speech.

In English, jalapeño is correctly spelled with or without the eñe (and AFAIK the letter doesn't have a name in English, you have to use the Spanish name). So, there's an English word that doesn't use the letters assigned to the English alphabet. How do we place the word? Well, obviously English borrowed the word from Spanish, so it's a Spanish word. Well, no, it's only the Spanish adjectivization of Nahautl words used to name the place called Xalapa...

Words like angst or ersatz are English words borrowed from German. The German words are written identically (except capitalisation), but the meaning of the English word is much more specific than the German "original". Meanwhile the word "Blitz" has completely district meanings in English and German. In English it's a sudden concerted effort, in German it's lightning. Despite the English word originating from German, they don't share a meaning at all

Isn't the etymology of 'blitz' in English due more to 'blitzkrieg'?

German has the word blitzschnell which means really fast (as fast a bolt). So in a way the english meaning of blitz still fits your description.

Think of just something simple like writing an immigrants name inside a sentence. It's kinda funny that people in SV, full of immigrants, never seem to think of putting their own or coworkers name in a String.

My grandfather's thesis was auf Deutsch and is sprinkled with French and Latin words.

What's a word? (A quick test - how many words were in the previous sentence, maybe 3 or 4 depending on whether the 's is part of a word; so can we talk about Jóhannesson's foreign policy?).

It's hard enough to know what a letter is in unicode. Breaking things into words is just another massive headache.

> Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us we would be better off being able to a assert that a piece of text was German, or sanskrit, not a jumble of both.

Presumably the person who wrote it speaks a single language.

Just because something is not useful to them, it doesn't mean it is not useful in general. There are millions of polyglots as well as documents that include words and names in multiple scripts.

I think in that case the idea would either be that you should then have an array of strings, each of which may have its own language set, or that the string should be labelled as "containing Latin and Cyrillic", but still not able to include arbitrary other characters from Unicode. And multi-lingual text still generally breaks on words... Kilobytes of Latin text with a single Cyrillic character in the middle of a word is very suspicious, in a way that kilobytes of Latin text with a single Cyrillic word isn't.

Of course you'd always need an "unrestricted" string (to speak to the rest of the system if necessary), but there are very few natural strings out there in the world that consist of half-a-dozen languages just mishmashed together. Those exceptions can be treated as exceptions.

How would that look like to an end-user? Do I need to tell my browser which scripts are contained in my emails? What would happen if I start typing in a different script?

> there are very few natural strings out there in the world that consist of half-a-dozen languages just mishmashed together

...in your experience. What is an almost-absurd exception to you, is every day life to others.

> Presumably the person who wrote it speaks a single language.

Presumably the person who wrote it speaks English.

Of course living in denial makes it easy to ignore harsh realities. Unfortunately for them, humans don't work that way. Things aren't gonna spontaneously change just to make their life easier. The software adapts to us, not the other way around. People complain about the complexities of dates and times but they still make every effort to get it right because it matters.

If a programming language allows text processing but can't even properly compare unicode text, it is buggy and needs to be fixed. If an operating system can't deal with unicode, it's buggy and needs to be fixed.

Reminds me of the good old days with EUC-KR, KSC 5601, and all those different encoding schemes I've successfully repressed in my memory for years. Yes, you could probably assert that a piece of string was either Korean or English but never anything else... because the system was incapable of representing it.

I'm not exactly sure how a code page is supposed to help us here. Developers have trouble supporting multiple languages when they're all in the Unicode Standard. Supporting code pages for languages they've never heard of? Not a chance.

I'd guess a standardized codepage marker like a "start of CP[932]” ｉｓｇｏｉｎｇｔｏｂｅｎｅｃｅｓｓａｒｙ CP[1252] at each CP switches but it might be just a necessity. Han unification is a well known problem to Far Eastern but Unicode normalization problem is basically the same as that.

Heh, funny, I'm implementing this exact thing at the moment, oddly enough -- rather, implementing a security check that provides that same guarantee you mention, Mixed Script protections.

In Unicode spec terms, 'UTS 39 (Security)' contains the description of how to do this, mostly in section 5, and it relies on 'UTX 24 (Scripts)'.

It's more nuanced than your example but only slightly. If you replace "German" with "Japanese" you're talking about multiple scripts in the same 'writing system', but the spec provides files with the lists of 'sets of scripts' each character belongs to.

The way that the spec tells us to ensure that the word 'microsoft' isn't made up of fishy characters is that we just keep the intersection of each character's augmented script sets. If at the end, that intersection is empty, that's often fishy -- ie, there's no intersection between '{Latin}, {Cyrillic}'.

However, the spec allows the legit uses of writing systems that use more than one script; the lookup procedure outlined in the spec could give script sets like '{Jpan, Kore, Hani, Hanb}, {Jpan, Kana}' for two characters, and that intersection isn't empty; it'd give us the answer "Okay, this word is contained within the Japanese writing system".

Where I work and communicate, mixing 2, 3, and sometimes 4 writing systems is pretty normal; I have 3 keyboard layouts on my phone (Latin that covers English and occasional Spanish, Cyrillic, and Japanese).

In any case, there are emoji which are expected to be a part of text.

On one hand, it would be great to separate areas of different encodings inside a string. But character codes are already such separators.

Two things need to go though: the assumption of linear-time index-based access to characters in a string, and the custom to compare strings as byte arrays.

The first us already gone from several advanced string implementations. The second is harder: e.g. Linux filesystems support Unicode by being encoding-agnostic and handling names as byte sequences. Reworking that would be hard if practical at all.

> and the custom to compare strings as byte arrays

I think more generally, the idea that a langauge std lib can provide string equality for human langauge strings is just silly. String equality is an extremely context dependent, fuzzy operation, and should be handled by each context differently. For example for Unicode hostname to certificate mapping, hostname equality should be handled by rendering the hostname in several common web fonts and checking if the resulting bitmaps are similar. If they are, then assigning different certificates to these equal hostanmes should not be done.

Of course, in other contexts, there are different rules. For example, if looking up song names, the strings "jalapeno" and "jalapeño" should be considered equal, in English text at least.

That doesn’t make sense to me. Even disregarding cases where people mix languages (how do you write a dictionary? If the answer is “just create a data structure combining multiple strings”, shouldn’t we standardize how to do that?), all languages share thousands of symbols such as currency symbols, mathematical symbols, Greek and Hebrew alphabets (to be used in math books written in the language), etc. So, even languages such as Greek and English share way more symbols than that they have unique ones.

> being able to mix arbitrary languages into a single String object

Unless I missed something that is impossible with Unicode. Mixing multiple languages would require a way to specify the language used for case conversion, sorting and font rendering settings mid string and I don't think that Unicode has that. For example try to write a program that correctly uppercase a single string containing both an English i and a Turkish i in your favorite Unicode supporting language, the code point is the same for both, and you generally only get to specify one language globally or per function call.

You can write a string with words from multiple languages, you just can't easily modify it with operations like case conversion. But sorting shouldn't depend on the origin language anyway, it depends on the language of the reader. All words in an English dictionary are sorted in "English" order

Displaying is also questionable. If you want japanese/chinese/etc. rendered correctly in your browser you have to mark corresponding sections with a language tag, they have different rules on how several abstract graphemes shared between them should be rendered (amount and shape of strokes).

I mix languages all the time.

> It might have been better if the 'code pages' idea was refined instead of eliminated

Obviously yes, it would have been better.

But Unicode was designed by the same people who designed ASCII - monolingual Americans who never had to deal on a daily basis with anything that doesn't fit into the 26 letters of the English alphabet. So here we are.

> But Unicode was designed by the same people who designed ASCII - monolingual Americans who never had to deal on a daily basis with anything that doesn't fit into the 26 letters of the English alphabet.

This is not even remotely true.

Yes it is. The people who made Unicode went out of their way to make life hard and offensive for CJK, Cyrillic, etc., users.

Since you could watch this all unfold in real time (and can go back and read mailing list archives back ab initio) you can easily see the participants and their positions and arguments. The “people who made Unicode” came from and continue to come from all over the world.

Also, do you remember what (polyglot computing) life was like before Unicode?

Reply on news.ycombinator.com