String Lengths in Unicode
hsivonen.fi
hsivonen.fi
Under that quote three rationale are offered.
1. "to allocate memory". Fine, like an integer your very low level code needs to actually store strings in RAM and in this sense both integers and strings have a size. I don't think I'd name this size "length" though and certainly if length(5) is an error in your language it seems as though length("word") is also an error by this definition.
2. "quota limit". Just pick anything, there is definitely no reason to name the value you're using for this arbitrary quota "length". If you have a _reason_ for the quota, then you need to measure the reason, e.g. bytes on disk.
3. "display space". This one is a length! But it's in pixels (or milimeters, or some other type of distance unit) and it requires a LOT of additional parameters to calculate. And you'll notice that at last there is actually a string height too for this case, which likewise requires many additional parameters to calculate.
Treat code that cares about string "length" as code smell. If it's in your low-level fundamentals (whether in the memory allocator or the text renderer) it might, like a lot of other things which smell down there, have a good reason to be there, make sure it's seriously unit tested and clearly documented as to why it's necessary. If it's in your "brilliant" new social media web site backend, it's almost certainly wrong and figuring out what you ought to be doing instead of asking the "length" of a string will probably fix a bug you may not know you have.
The problems only arise when taking about "a string of length 20" which indeed has a strong smell
It has no smell at all. We interface with all kinds of systems with limited display space, external forms, POS displays, databases with varchar columns, and so on all the time.
Whether certain strings should have a specific length is a business decision, not a code smell...
Databases with varchar columns exist, but varchar(20) sounds generally suspect unless it's a hash or something else that's fundamentally limited in length.
It is notoriously easy when the display is a LED display, a banking terminal, a form-based monospaced POS, something that goes to a printed out receipt (like a airline check-in or a luggage tag), a product / cargo label maker, and tons of other systems billions depend upon everyday, where one visible glyph = 1 length unit, and type design doesn't come into much play...
The mere assumption that "one glyph" is a meaningful well-defined concept that works across languages is the problem here.
For example, I have a phone app that abbreviates various things at 300 whatevers because I know that on one hand 300 is more than the phone will display, but on the other it's little enough that it'll cause no processing problems. If I process 300 whatevers and then the display can only show 120, I haven't burned much of the battery. I overstimated by a factor of 2.5, burned a minuscule part of the battery, and I gained simplicity.
I strongly agree that numbers as low as 20 are a code smell.
There are fancy Unicode corner cases, like the flags (cutting a flag emoji in half doesn't get you half a flag it gets you one half of the flag's country code identifier) but there are plenty of corner cases already in ASCII.
Abbreviating "Give all the money to Samantha's parents to hold in trust" as "Give all the money to Sam" doesn't involve any scary modern encoding, just the problem that this isn't how human languages work.
Consider an unbounded input string (generally less than 1k, but sometimes more than 100k), a phone display large enough to display, say, 80-120 glyphs, and a mechanical abbreviation to a few hundred codepoints soon after input, before all expensive processing. What are the nasty corner cases?
Embedded systems care deeply and know exactly display sizes.
It's subtle, which is what makes it a smell, rather than a "DO NOT WANT".
That one is likely to get you into trouble when you allow 0 width characters.
Whether the metric is called "length", or "size", or whatever, is irrelevant, and length for strings and arrays and lists is so well entrenched and understood that the objections don't make sense.
What's actually the problem is not that "string length" is not "really" a length in the way that a leg or a wall has length, but that for unicode strings it's difficult to calculate, and confusing, unless one understands several unicode implementation mechanics...
The name length itself, for example, or the measuring of said length, was never an issue with ASCII strings (the issue there was to remember to add/subtract the NUL at the end).
It’s like measuring “the size of a box.” I want volume, you want maximum linear dimension, UPS wants length plus width plus height. The problem with “the size of a box” isn’t that it’s hard to measure, it’s that it doesn’t exist. Imagine your favorite language has a Box type with a “size” property. What does it return. How likely is it that the thing it measures is the thing you want?
Of course it was never a problem for ASCII, because ASCII is structured to make most of the measurements people care about be the same value. I want bytes, you want code points, he wants “characters,” doesn’t matter, they’re all the same number.
ASCII is also incapable of representing real world text with good fidelity. It’s inherently impossible to remedy that while maintaining a singular definition of “length.” If you value length measurement over fidelity, you can keep using ASCII.
That's already covered in my comment: ("[the problem] stems from there being multiple, similar, metrics one wants").
Whether we call all of them length or a specialized name is not the real problem.
The real problem is you need to know what you want of each, and some of them (e.g. regarding normalization, decomposition, and so on) can be hard to grasp.
In 99% of cases people either want to know "how many bytes", or "how many discreet character glyphs of final output" (even if they have combining diacritics etc).
I suspect you mean the number of grapheme clusters, which is Unicode's attempt to define something that lines up with the intuitive notion of "a character." This is basically a unit that your cursor moves over when you press an arrow key.
However, it's pretty uncommon to want to know the number of grapheme clusters too. Lots of people think they want to know it, but I struggle to come up with a use case where it's actually appropriate. An intentionally arbitrary limit like Tweet length is the best I can think of.
"How many bytes" is ambiguous. Do you mean UTF-8, UTF-16, UTF-32, or something else?
There are a lot of different ways to answer the question, "how long is this string?"
You did mention similar metrics, but you then went on to say that the objections don't make sense and that the actual problem is that length for a unicode string is difficult to calculate.
My point is that the difficulty of calculating a length is not the problem. It's annoying, but people have written the code to do it and there's rarely any reason to write it yourself. Just call into whatever library and have it do the work. The problem is that you have to know what kind of question to ask so you can make the call that will actually give you the answer that you need. And that is not the sort of thing that can be wrapped up in a nice little API.
TCP packet size is a very real thing for me.
Also in forms it helps to know, for example, that no one has entered more than 2 characters for a USA state abbreviation.
Definitely open to bigger ways about thinking of these things though.
As for state abbreviations, is $7 a valid abbreviation? Is XQ? You’d probably want to validate those against a list of known good ones.
That link doesn't seem to explain the why, but my understanding is that it's just a decent heuristic for the general difficulty of handling packages as they go through the system. Volume wouldn't be appropriate, because a really long, skinny box is harder to handle than a cube of the same volume.
(essentially, iOS was trying to truncate strings in notifications to make them fit available space, except it did that in a very naive way, and would crash on any case where removing codepoints from a string made it display _longer_. Pretty easy to trigger with letters like ي)
But in Arabic, letters change form based on where they are in the word. All three things with two dots on them are the same letter in تتت, but the middle one is much smaller. I can imagine something like يم being truncated to ي and the end result being longer visually, depending on the font.
This is nonsense. Pretty much anywhere that you're accepting user input on the web you want to have some sort of length limit. Even if you're Hacker News rather than Twitter, you probably don't want to let anybody post a 10GB comment. You just need to pick an appropriate length metric. That's part of the post's point, and it's also apparently your point a few sentences after you rubbish it; why are you insulting the post and then suddenly agreeing with it?
For almost every task, an iterator that iterates graphemes (extended grapheme clusters) is the right thing to do.
I do think it's annoying that Unicode ended up with such a complex extended grapheme cluster scheme though.
2. This is reasonable.
3. What about fixed-width fonts like a terminal screen. In this case, I care about how many distinct (non-combined) code points exist, and a count of code points is often a acceptable approximation.
That being said, when talking about string length, I mostly care when iterating and allocating memory, and those are two different usages of the term "length". I like how Go handles it: len(string) gives number of Unicode code points, len([]byte(string)) gives number of bytes.
Everything else is a special case IMO that shouldn't use length, except perhaps as a conservative estimate (e.g. fixed width rendering).
Keep in mind that even Annex 29 is, on some level, just a starting point. Its segmentation rules don’t work “out of the box” for me with Persian and Arabic text. I’m not totally on-board with its treatment of the zero-width non-joiner (U+200C), and it doesn’t deal with manual tatweel/kashida (U+0640) at all. So you make the necessary adjustments for your use case. The rabbit hole is deep.
It's not bad, but Swift still does better:
* String.Index is an opaque type (ideally it would be linked to a specific string too)
* it returns grapheme cluster counts by default, that's probably the least surprising / useless though not necessarily super useful either
* other lengths go through explicit lazy views e.g. string.utf8.count will provide the UTF8 code units count, string.unicodeScalars.count will provide the USVs count
> If you switch to s.chars().count(), you’ll get UTF-8 scalar values
You get unicode scalar values (which I assume is what you meant).
Swift is primarily geared towards UI, and in that case using EGCs is the right design choice.
Perhaps string length should be more hidden though, or parametrized with a unit to count. Perhaps it should be
let nu = s._smelly_length(UTF8CodeUnits);
let nc = s._smelly_length(ExtendedGraphemeClusters);
The author's substantive criticism of Swift's String/Character/etc. types seems to be complications arising from the dependency on ICU (such as not necessarily being able to persist indices across icu4c versions).
Rust put the whole thing front and center and made it incredibly clear why they made that choice. The design choices around encodings also have been more well reflected than those in Python, it never becomes a “hairy” mess, but something you can rationally manage.
Go named their char rune. That's a distinction without a difference.
Yes. It's also the original intended meaning of java char, just didn't survive the transition to 21 bit USVs.
So distinction without a difference.
For example, if you want to encode emojis some non-Unicode abstract description goes in and a sequence of characters go out, and if you want to figure out the boundaries of letters clustered with the respective combining diacritical marks they are subsequences in a sequence of characters.
Be very careful here; when talking Unicode, there is no such single thing as a "character", and people being insufficiently precise here has been the source of much confusion.
That is exactly what "rune" is intended for, though. In the several years I've been using Go, I believe I've used it once, precisely in a parsing situation.
For the most part, Go's answer to Unicode is to just treat them as bytes, casually assume they're UTF-8 unless you really go out of your way to ensure they are something else, and to not try to do anything clever to them. As long as you avoid writing code that might accidentally cut things in half... and I mean that as a serious possible problem... it mostly just works.
To be honest, if you're trying to tear apart this example emoticon at all, you're almost certainly doing something wrong already. The vast bulk of day-to-day [1] code most people are going to write should take the Unicode characters in one side, and just spit them out somewhere else without trying to understand or mangle them. A significant amount of the code that is trying to understand them should still not be writing their own code but using something else that has already implemented things like word tokenization or something, or performing operations using Unicode-aware libraries (e.g., implementing a find-and-replace function can basically do a traditional bytestring-based replacement; code to replace "hello" with "goodbye" will work even if it has this emoticon in the target text if you do the naive thing). What's left is specialized enough to expect people to obtain extra knowledge and use specialized libraries if necessary.
A lot of what's going wrong here is putting all these things in front of everybody, which just confuses people and tempts then into doing inadvisable things that half work. In a lot of ways the best answer is to stop making this so available and lock it behind another library, and guide most programmers into an API that treats the strings more opaquely, and doesn't provide a lot of options that get you into trouble.
It isn't necessarily a perfect solution, but it's a 99% solution at worst. I write a lot of networking code that has to work in a Unicode environment, so it's not like I'm in a domain far from the problem when I say this. It's just, 99% of the time, the answer is, don't get clever. Leave it to the input and rendering algorithms.
[1]: I say this to contrast things like font rendering, parsing code, and other things that, while they are vitally important parts of the programming ecosystem and execute all the time, aren't written that often.
He is working for Mozilla and I guess he needs to actually know all those nitty-gritty details. I could not imagine myself to even bring up the patience for analyzing it.
In some way I am really scared about Unicode. I don't care if the programming language simply allows input and output of Unicode in text-fields and configuration files. But where it gets tough is, if you actually need to know the rendered size of a string or convert encodings, if the output format requires it. There are so many places where stuff can go awry and there's only a small passage in the blog post which mentions fonts. There are multiple dragons abound, like font encoding, kernings and what not.
Imagine writing a game. Everything works nice and dandy with your ASCII format and now your boss approaches you and wants to distribute the game for the asian market! Oh, dear... My nightmares are made out of this!
How do you handle this? Do you use a programming language which does everything you need? (Which one?) Do you use some special libraries? What about font rendering? Any recommendations?
You just make sure all translated text is in UTF-8, and use Google Noto fonts for those languages. All game engines I know render UTF-8 text without problems if you supply a font that has the needed glyphs.
Source: I'm and indie game developer and have recently localized my game to Chinese. The game is a mix of RPG and roguelike, so it has a lot of text (over 10000 words). I used SDL_TTF to render text. Precisely: TTF_RenderUTF8_Blended() function. The only issue I had is with multiline/wrapped text. SDL_TTF doesn't break lines on Chinese punctuation characters (.,;:!?) so I would search+replace strings at runtime to add a regular space characters after those.
Those aren't Chinese punctuation characters. Chinese punctuation characters are full-width, including the spacing that should follow them (or in the case of "(", precede) within the glyph itself: （。，；：！？）. (You may also notice that the period is radically different.) Chinese text should almost never include space characters.
Chinese applications seem happy to break lines anywhere including in the middle of a word, but punctuation seems like an especially good place for a line break, so I'm confused why SDL_TTF would go out of its way to avoid breaking there.
I wonder if SDL_TTF works with Unicode Zero-Width Space (U+200B). If so, that would probably be the right choice.
I know, I meant the actual ones you wrote above.
> I'm confused why SDL_TTF would go out of its way to avoid breaking there.
SDL_TTF doesn't break at all. If you have a long Chinese text which uses proper punctuation characters, it would never break, because it only breaks on ASCII whitespace.
I wanted to avoid breaking lines in the middle of a word, so I added extra "regular" space characters to force breaking the line.
The game could have used a custom engine (like tons of games do), or the requirement could include e.g. Arabic or some such RTL text, further messing up the display...
Also disclaimer: I've been working on games which were translated to >20 languages including some exotic ones, but I'm in no way an UNICODE expert.
- most important: consider using UTF-8 as text encoding everywhere, and only encode and decode from and to other text encodings when needed (for instance when talking to APIs which don't understand UTF-8, like Windows)
- be very careful in all places where users can enter strings, and with filesystem paths, this is where most bugs happen (one of the most popular bugs is when a user has Unicode characters in his login name, and the game can't access the user's "home directory", happens to the big ones too: https://what.thedailywtf.com/topic/15579/grand-theft-auto-v-...)
- get familiar with how UTF-8 encoding works and how it is "backward compatible" with 7-bit ASCII, there are good chances you don't need to change much of your old string processing code
- rendering is where it gets interesting, and here it makes sense to only do what's needed:
(1) The easiest case is American and European languages, these are all left-to-right, have fairly small alphabets and don't have complicated 'text transformation' rules
(2) East-Asian languages with huge alphabets can be a problem if you need to pre-render all font textures.
(3) The next step is languages which render from right-to-left, the interesting point is that substrings may still need to be rendered left-to-right (for instance numbers, or "foreign" strings)
(4) And finally there are languages like Arabic which rely heavily on modifying the shape of 'characters' based on where they are positioned in words or in relation to other characters, you need some sort of language-specific preprocessing of strings before you forward them to the renderer. HarfBuzz is a general solution for this problem, but it's also a lot of code to integrate (we created a specialized transformation only for Arabic).
(5) For actual rendering, all text rendering engines which can use TTF fonts are usually ready for rendering UNICODE text
So basically, the problem becomes a lot easier if you only need to support a specific set of languages.
By "sanitize", I mean it should take as input any string for a filename, and clean it up so it looks the same and can be safely stored on any platform/filesystem. This would allow people to, for instance, download or exchange files over the Internet and know deterministically how the filename shall look on the receiving side, instead of relying on the receiving side to clean up the filename.
The length of Unicode strings was definitely one of the pain points. Back then I only knew about NFC vs NFD.
Now that I have read this article, I realise that my algorithm to determine where to truncate a filename is probably still wrong, and that I need to dig both much deeper and much wider.
If anybody wants to help dig some serious rabbit holes, you're most welcome to fork and make PRs.
In the absence of support in the stdlib, is https://github.com/alvinlindstam/grapheme the best to use?
What you want in your UI is the number of glyphs that will be presented to the user. Which means it is correct for it to change across OS or library versions, as new emoji will result in a different number of rendered glyphs.
There are very few places in program UI that want anything at all other than the actual glyph count - emoji aren’t even the first example of this, there are numerous “characters” the are made up of letters+accents that are not single code units/points/whatever (I can never recall), for which grapheme count is what you want.
The only time you care about the underlying codepoints is really if you yourself are rendering the characters, or you’re implementing complex text entry (input managers for many non-English characters). The latter of which you would pretty much never want to do yourself - I say this having done exactly that, and making it work correctly (especially on windows) was an utter nightmare.
Then once you get past code point/units to the level of bytes your developer set divides neatly in two:
* people who think the world is fixed # of bytes per character
* people who know to use the API function to get the number of bytes
But for any actual use of a string in a UI you want the number of glyphs that will actually be displayed.
My assumption is that that logic is why Swift gives you the count it does.
In every other case you are 100% correct: you want either the graphical bounds or the number of bytes. The other measures seem of questionable utility to anyone outside of the Unicode library implementation :)
My point is generally that the only measurements a developer should ever care about are the visible glyph count and the byte/storage size.
This is probably the biggest mystery to me of the Python 3 migration. If they were going to break backcompat, why on Earth didn't they fix Unicode handling all the way? They didn't have to go completely crazy with new syntax like Perl 6 did, but most languages shift too much of the burden of handling unicode correctly onto the programmer.
The codepoint-based string representation used by Python 3 may be "the worst" (I'm not totally sure I agree) but it's fine. The article's main beef is about the somewhat complex nature of the internal storage and the obfuscation of the underlying lengths.
I didn't seek to mention every programming language for everything. E.g. I didn't mention C#, since UTF-16 was already illustrated using JavaScript.
No rationale for this seems to be included in the article.
The article also explains how internally this is an optimization the Python interpreter actually does.
"Reacting to surrogate pairs by wishing to use UTF-32 instead is a bad idea, because if you want to write correct software, you still need to deal with variable-width extended grapheme clusters.
The choice of UTF-32 arises from wanting the wrong thing."
Whether the use of Unicode code points instead of some Unicode encoding is a good thing or not, that I don't know.
The article says "Python 3 strings have (guaranteed-valid) UTF-32 semantics" and later argues that the fact that there's a distinction between the semantics and actual storage is a data point against UTF-32.
> they are sequences of Unicode code points, not code units in some encoding
They are sequences of _scalar values_ (all scalar values are code points but surrogate code points are not scalar values). Exposing the scalar value length and exposing indexability by scalar value index is the same as "(guaranteed-valid) UTF-32 semantics".
Note that e.g. Rust strings are conceptually sequences of scalar values, and you can iterate over them as such, but they don't provide indexing by scalar value or expose the scalar value length without iteration.
JavaScript strings, on the other hand, are conceptually sequences of code points.
> I don't remember how they're stored internally
The article say how they are stored...
Sorry. I'm shocked that I tested wrong when researching the article. Python 3 indeed has code point semantics and not scalar value semantics. I've added a note to the article that I've edited in corrections accordingly.
Python 3 is even more messed up than I thought!
Length of a string in bytes is useful, though most code shouldn't need to operate at that level of abstraction.
Length of a string in glyphs is useful if you are formatting something to a fixed-width display, though that is kind of a niche use-case.
Length of a string when rendered is frequently useful, though impossible to calculate from just the string's contents.
Length of a string in code points can't be used correctly for anything.
CPython since 3.3 makes the same idea three-level with UTF-32 semantics: Strings are stored as UTF-32 if at least one character has a non-zero bit in its most-significant half. Else if a string has a non-zero bits in its second-least-significant 8 bits of at least one character, the string is stored as UCS2 (i.e. UTF-16 excluding surrogate pairs). Otherwise, the string is stored as Latin1.
This comes at a cost. Every time you extract a sub-string from a string, you have to scan it first for its codepoints, then convert character positions into byte positions. One way to speed up stuff a bit, is to check if the string is in ASCII (see https://lemire.me/blog/2018/05/16/validating-utf-8-strings-u...) and apply regular operator then.
We implemented many techniques based on "intrinsics" instructions to speed up conversions and search in order to avoid scanning for codepoints.
See https://github.com/naver/tamgu/blob/master/src/conversion.cx... for more information.
If you just rely on code points you risk splitting up a grapheme cluster into (in your example) two graphemes, one in each sub-string, the left representing "black" and the right representing "thumbs up." Further, you actually need to utilize one of the unicode normalization forms to perform meaningful operations like comparison or sorting.
This is one thing Rust's string API gets right, allowing you to iterate over a string as UTF-8 bytes in constant time -- and, by walking, your choice of codepoints and (currently unstable, or in the unicode-segmentation crate) grapheme clusters. Even that though is a partial solution. [2]
Definitely a tough problem!
[1] https://mathias.gaunard.com/unicode/doc/html/unicode/introdu...
[2] https://internals.rust-lang.org/t/support-for-grapheme-clust...
There are some methods to traverse a UTF8 string:
begin(): to initialize the traversal
end() : is true when the string is fully traversed
next(): which goes to the next character and returns the current character.
s.begin();
while (!s.end()) {
u = s.next();
}I wonder if it's a specific app or just something in Android not rendering the font correctly?
The level of ligatures requested in CSS makes sense for the site-supplied font. (I need to regenerate the fonts due to the table at the end of the article using characters that the subsets don't have glyphs for.) If you block the site-supplied font from loading, the requested level of ligatures may be excessive for your fallback font.
> string:length(" ️").
1
> string:to_graphemes(" ️").
[[129318,127996,8205,9794,65039]]
Not just "emoji" either, it bans random codepoints it doesn't consider "textual" enough and rather than tell you they're just removed from the comment when submitting leaving you to find out your comment is completely broken.
It's infuriating.
Array.from('<emoji here>').length
[...'<emoji here>'].length1. If you're using character count to do memory related things, you're introducing bugs. Not every character takes up the same amount of space (see: emoji)
2. If you're using character count to affect layout, you're introducing bugs. Not all characters are the same width. Characters can increase or decrease in size (see: ligatures, Dia critics). Any proper UI library will give you a way to measure the size of text (see JS's measureText API)
3. Even static text changes. Unless you never plan to localize your application, pin it to use one specific font (that you're bundling in your app, because not all versions of the same font are made equal), and bringing your own text renderer (because not all rendering engines support all the same features), you're introducing bugs. The one exception is perhaps the terminal, but your support for unicode in the terminal is probably poor anyway.
Even operations like taking a substring are fraught for human-readable strings. Besides worrying about truncating text in the middle of words or in punctuation (which should be a giant code smell to begin with), slicing a string is not "safe" unless you're considering all of the grammars of all of the languages you'll ever possibly deal with. It's unlikely, even if your string library perfectly supported unicode, that you'd correctly take a substring of a human tradable string. It's better to design your application with this in mind.
This is because the output doesn't have to be perfect, but it does absolutely positively have to have bounded length or various databases start getting real grumpy.
Your database might be grumpy otherwise, but that doesn't make arbitrary truncation correct. This is an issue with your schema, it doesn't mean truncation is the best solution.
Python 3.6.8 (default, Dec 30 2018, 13:01:27)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.5.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: len("FACEPALM_EMOJI_DELETED_BY_HN")
Out[1]: 1Try out the family or flag emoji (which are composite) and you should get a different result.
Your language's length function is probably just returning the number of unicode codepoints in the string. You need to a function that computes the number of 'extended grapheme clusters' if you want to get actually displayed characters. And if that function is out of date, it might not handle ZWJ and variation selectors properly, and still give you a value of 2 instead of 1. Make sure your libraries are up to date.
Also, if you are writing a command line tool, you need to use a library to work out how many 'columns' a string will occupy for stuff like word wrapping, truncation etc. Chinese and Japanese characters take up two columns, many characters take up 0 columns, and all the above (emoji crap) can also affect the column count.
In short the Unicode standard has gotten pretty confusing and messy!
The article didn't say that!
"Number of Unicode code points" in a string is ambiguous, because surrogates and astral characters both are code points, so it's ambiguous if a surrogate pair counts as two code points or one. (It unambiguously counts as two UTF-16 code units and as one Unicode scalar value.)
The article presented four kinds of programming language-reported string lengths:
1. Length is number of UTF-8 code units. 2. Length is number of UTF-16 code units. 3. Length is number of UTF-32 code units, which is the same as the number of Unicode scalar values. 4. Length is number of extended grapheme clusters.
To give you just one little example of how you are not correct:
长久 - cháng jiǔ - long time/long lasting
尝酒 - cháng jiǔ - to taste wine
Japanese uses Kanji (Chinese characters) not for their phonetic value (they’ve got two whole syllabaries for that), but for their meaning.
The "dialectical variations" are really different languages. Mandarin speakers are taught that only pronunciations vary, but a person transcribing Cantonese to Mandarin is doing translation to a degree comparable to Italian -> French. Politically this fact is not allowed in China, but ask anyone who is bilingual in Cantonese and Mandarin. Prepare to be surprised.
And also (less common, but existant) characters that represent different syllables in different contexts. A prominent example would be 觉, pronounced jué in 觉得 but jiào in 睡觉
Hanzi are logographs, not a syllabary.
> and complicated rules about which to use for writing common words
Maybe if you have some need to believe in the bizarre fiction that Chinese characters only provide phonetic information. In the real world, the characters have meaning and history, which simply dictates what characters to use for which word.
This is like saying English is written with logographs, not an alphabet. Kanji are logographs and not a syllabary. Hanzi are closer to being a syllabary than they are to being logographs. The sound is the primary concept.
Obviously, the characters do convey considerably more than just phonetic information. But phonetic information is the first and most important thing they carry.
In English, people believe that "Elements of Style" is full of good advice despite everything good they have ever read violating every rule on every page.
If you consider this from the perspective of Huffman coding, it shouldn't be surprising that the script that gets the most code space in Unicode would carry the most information per symbol on average.
The Chinese (Traditional) translation consists of 2202 characters (UTF-32). The English translation consists of 8555 characters. This amounts to ~3.88 English letters per Chinese character, quite a bit more than "two or three".
None of the actual syllabaries get even close to the density of Chinese. Korean, the closest comparison point (a true syllabary) clocks in at 3856 characters - a far cry from 2202.
...And the OP is wrong.
1) python doesn't count byte sizes, or UTFxxx stuff. python counts the number of codepoints. Do you want byte-lenght? decode to a byte array and count them. 2) javascript doesn't know about bytes, nor characters, knows only about the fact that "a char is a 16 bit chunk", with a UTF16 encoding. there is no such thing a "code unit with UTF-16 semantics". Similar for java.
oh, and by the way, there are byte sequences that are invalid if decoded from utf-8, so I'm non sure about the "guaranteed-valid" utf-8 rust strings.. (if you want an encoding that can map each byte sequence to a character, there are, like Latin1 and such on, but it's a different matter)
UTF32 and codepoints is an identity transformation.
> knows only about the fact that "a char is a 16 bit chunk", with a UTF16 encoding. there is no such thing a "code unit with UTF-16 semantics". Similar for java.
A UTF-16 code unit is 16 bits. The difference between "UTF16 encoding" and "UTF16 code units" is the latter makes no guarantee that the sequence of code units is actually validly encoded. Which is very much an issue in both Java and Javascript (and most languages which started from UCS2 and back-defined that as UTF-16): both languages expose and allow manipulation of raw code units and allow unpaired surrogates, and thus don't actually use UTF16 strings, however these strings are generally assumed to be and interpreted as UTF16.
Which I expect is what TFA means by "UTF-16 semantics".
> oh, and by the way, there are byte sequences that are invalid if decoded from utf-8, so I'm non sure about the "guaranteed-valid" utf-8 rust strings..
Your comment makes no sense. There are byte sequences which are not valid UTF-8. They are also not valid as part of a Rust string. Creating a non-UTF8 rust string is UB.
The article says "(potentially-invalid) UTF-16 semantics". The "potentially-invalid" part means that the JavaScript programmer can materialize unpaired surrogates. The "semantics" part means that the JavaScript programmer sees the strings acting as if they were potentially-invalid UTF-16 even when the storage format in RAM is actually Latin1 in SpiderMonkey and V8.
So how does rust deal with filenames under Linux? Use somethinge other than strings?
This is also used to deal with filenames on Windows, as they're a different flavour of "not unicode", IIRC that's the original use case for WTF8.
It also has a separate pseudo-string type to deal with C strings: https://doc.rust-lang.org/std/ffi/struct.CString.html
[1] https://doc.rust-lang.org/std/path/struct.PathBuf.html
[2] https://doc.rust-lang.org/std/path/struct.PathBuf.html#metho...
[3] https://doc.rust-lang.org/std/path/struct.PathBuf.html#metho...