String length functions for single emoji characters evaluate to greater than 1
hsivonen.fi
hsivonen.fi
my \emoji = "\c[FACE PALM]\c[EMOJI MODIFIER FITZPATRICK TYPE-3]\c[ZERO WIDTH JOINER]\c[MALE SIGN]\c[VARIATION SELECTOR-16]";
#one character
say emoji.chars; # 1
#Five code points
say emoji.codes; # 5
#If I want to know how many bytes that takes up in various encodings...
say emoji.encode('UTF8').bytes; # 17 bytes
say emoji.encode('UTF16').bytes; # 14 bytes
$a = "\N{FACE PALM}\N{EMOJI MODIFIER FITZPATRICK TYPE-3}\N{ZERO WIDTH JOINER}\N{MALE SIGN}\N{VARIATION SELECTOR-16}";No such method 'length' for invocant of type 'Str'. Did you mean any of these: 'codes', 'chars'?
Because as the article points out, the "length" of a string is an ambiguous concept these days.
* Number of native (preferably UTF-8) code units
* Number of extended grapheme clusters
The first is useful for basic string operations. The second is good for telling you what a user would consider a "character".
~Also, DOM Strings are not UTF-16, they're UCS-16.~
EDIT: UCS-2, not UCS-16. Also, I'm confusing the DOM with EcmaScript, and even that hasn't been true in a while.
Hm, according to the spec they should be interpreted as UTF-16 but this isn't enforced by the language so it can contain unpaired surrogates:
From https://heycam.github.io/webidl/#idl-DOMString
> Such sequences are commonly interpreted as UTF-16 encoded strings [RFC2781] although this is not required... Nothing in this specification requires a DOMString value to be a valid UTF-16 string.
From https://262.ecma-international.org/11.0/#sec-ecmascript-lang...
> The String type is the set of all ordered sequences of zero or more 16-bit unsigned integer values (“elements”) up to a maximum length of 253 - 1 elements. The String type is generally used to represent textual data in a running ECMAScript program, in which case each element in the String is treated as a UTF-16 code unit value... Operations that do interpret String values treat each element as a single UTF-16 code unit. However, ECMAScript does not restrict the value of or relationships between these code units, so operations that further interpret String contents as sequences of Unicode code points encoded in UTF-16 must account for ill-formed subsequences.
RE the UTF-16 vs UCS-2 stuff, that’s probably a distinction which has technical meaning but will collapse at some point because no one actually cares, much like the distinction between URI and URL.
Meaning the software will deal much less well when it's wrong.
> and provides a nice intuitive estimation of how long text actually is
Not really due to combining codepoints, which make it not useful.
> as well as providing a fair encoding for most commonly used languages.
Which we know effectively doesn't matter: it's essentially only a gain for pure CJK text being stored at rest, because otherwise the waste on ASCII will more than compensate for the gain.
UTF-8 is the encoding you should generally always reach for when designing new systems, or when implementing a network protocol. Rust, Go and other newer languages all use UTF-8 internally because it’s better. Well, and in Go’s case because it’s author, Rob Pike also had a hand in inventing UTF-8.
Ironically C and UNIX, which (mostly) stubbornly stuck with single byte character encodings generally works better with UTF-8 than a lot of newer languages.
To summarize: the "codepoint" is a broken metric for what a grapheme "is" in basically any context. Edge cases would be truncating with a guarantee of a valid encoding on the substring? But really you want to truncate at extended grapheme cluster boundaries. Truncating a country flag between two of the regional indicator symbols might not throw errors in your code, but no user is going to consider that a valid string. The same is true of all manner of composed characters, and there are a lot of them.
So the only advantage of using the very-often-longer UTF-16 encoding is that it's an attractive nuisance! This makes it easier to write code which will do the wrong thing, constantly, but at a low enough rate that developers will put off fixing it.
Unicode is variable width, and what width you need is application-specific. That's the whole point of the article! UTF-8 doesn't try to hide any of this from you.
Reversing a string is what I would consider basic string operations, but I also expect it not to break emoji and other grapheme clusters.
Nothing is easy.
Tbh I can't think of a time when I've actually needed to do that.
find ... | rev | sort | rev > sorted-filelist
By reversing the lines before sorting them (and un-reversing them) they come out sorted and grouped by filename.
Recently I rewrote the codepoint reverse function to make it faster. The trick is to not write codepoints, but copy the bytes.
One the first attempt I introduced two bugs.
I wonder: several comments are saying it hardly makes any sense to reverse a string but... Certainly there are useful algorithms out there which do work by, at some point, reversing strings no!? I mean: not just for the sake of reversing it but for lookup/parsing or I don't know what.
I have never once wanted to actually do this in real code, with real strings.
Furthermore, the few times I've tried to do something like this by being cute and encoding data in a string, I never get outside the ASCII character set.
> The first is useful for basic string operations
Can you expand on this? I don't see why knowing the number of code units would be useful except when calculating the total size of the string to allocate memory. Basic string operations, such as converting to uppercase, would operate on codepoints, regardless of how many code units are used to encode that codepoint.
Converting 'Á' to 'á', for example, is an operation on one codepoint but multiple code units.
In some languages this is 90% of everything you do with strings. In other languages it's still 90% of everything done to strings, but done automatically.
I’ve used this for collaborative editing. If you want to send a change saying “insert A at position 10”, the question is: what units should you use for “position 10”?
- If you use byte offsets then you have to enforce an encoding on all machines, even when that doesn’t make sense. And you’re allowing the encoding to become corrupted by edits in invalid locations. (Which goes against the principle of making invalid state impossible to represent).
- If you use grapheme clusters, the positions aren’t portable between systems or library versions. What today is position 10 in a string might tomorrow be position 9 due to new additions to the Unicode spec.
The cleanest answer I’ve found is to count using Unicode codepoints. This approach is encoding-agnostic, portable, simple, well defined and stable across time and between platforms.
The only thing it's useful for is sizing up storage. It does nothing for "basic string operations" unless "basic string operations" are solely 7-bit ascii manipulations.
Yes you locate the specific indexes by using extended grapheme clusters, but you use the retrieved byte indexes to actually perform the basic operations. These indexes can also be cached so you don't have to recalculate their byte position every time (so long as the string isn't modified).
That has little to do with the string length, those indices can be completely opaque for all you care.
> Yes you locate the specific indexes by using extended grapheme clusters
Do you? I'd expect that the indices are mostly located using some sort of pattern matching.
* Passing strings around
* Reading and writing strings from files / sockets
* Concatenation
Anything else should reckon with extended grapheme clusters, whether it does so or not. Even proper upcasing is impossible without knowing, for one example, whether or not the string is in Turkish.
I'd quibble it's not the basic unit of measure so much as how changesets are represented. The user edits based on grapheme clusters. The final edit is then encoded using codepoints, which makes sense because a changeset amounts to a collection of basic string operations (splitting, concatenating, etc). As you note, it would be undesirable for changesets to be aware of higher level string representation details.
I can see why it would happen to be a codepoint, this might be ergonomic for the language, but it seems to me that, like clustering codepoints together in graphemes, clustering bytes into codepoints is something the runtime takes care of, such that a changeset will be a valid example of all three.
Using byte offsets also makes it possible to express a change which corrupts the encoding - like inserting in the middle of a multi byte codepoint. That goes against the principle of “make invalid data unrepresentable”. Your code is simpler if you don’t have to guard against this sort of thing. And you don’t have to worry about that if these invalid changes are impossible to represent in the patch format.
dz - \u0064\u007a, 2 basic latin block codepoints
DZ - \u0044\u005a
Dz - \u0044\u007a
dz - \u01f3, lowercase, single codepoint
DZ - \u01f1, uppercase
Dz - \u01f2, TITLECASE!
You can use
dż - \u0064\u017c - d followed by 'LATIN SMALL LETTER Z WITH DOT ABOVE'
dż - \u0064\u007a\u0307 - d followed by z, followed by combining diacritical dot above
dż - \u01f3\u0307 - dz with combining diacritical dot above
multiplied by uppercase and titlecase forms
Medieval ß is a letter/ligature that was created from ſʒ - that is a long s and a tailed z. In other words it is a form of 'sz' digraph. Contemporarily it is used only in german orthography.
How long is ß?
By some rules uppercasing ß yields SS or SZ. Should uppercasing or titlecasing operations change length of a string?
> 'But It’s Better that "[emoji]".len() == 17 and Rather Useless that len("[emoji]") == 5'
It sounds like it's just whether you're counting UTF-8/-16/-32 units. Does the article explain why one is worse and one is "rather useless?"
My two cents is that string length should always be the number of Unicode codepoints in the string, regardless of encoding. If you want the byte length, I'm sure there is a sizeof equivalent for your language.
When we call len() on an array, we want the number of objects in the array. When we iterate over an array, we want to deal with one object from the array at a time. We don't care how big an object is. A large object shouldn't count for more than 1 when calculating the array length.
Similarly, a unicode codepoint is the fundamental object in a string. The byte-size of a codepoint does not affect the length of the string. It makes no sense to iterate over each byte in a unicode string, because a byte on its own is completely meaningless. len() should be the number of objects we can iterate over, just like in arrays.
Should the letter plus a combining character count as one (I think so), or two characters? Should you normalize before counting length? And so on.
If you had a list of 5 Dom Element objects, and one Dom Attr object, the length of that list is 6. Its nonsensical to say "The Attr object modifies an Element object, so its not really in the list".
Which is the most useful for dealing with strings in practice though? Are either interpretations useful at all?
Codepoint is the smallest useful unit of a unicode string. It is a character, and you can do all the character things with it.
If you wanted to implement a toUpper() function for example, you would want to iterate over all the codepoints.
Nope. In order to deal with special casings you will have to span multiple codepoints, at which point it's no more work with whatever the code units are.
This is incredibly arbitrary - it depends entirely on what "length" means for a particular usecase. From the user's perspective there might only be a single character on the screen.
Any nontrivial string operation must be based around grapheme clusters, otherwise it is fundamentally broken. Codepoints are a useful encoding agnostic way to handle basic (split & concatenate) operations, but the method by which those offsets are determined needs to be grapheme cluster aware. Raw byte offsets are encoding specific and only really useful for allocating the underlying storage.
You count the width. And there are Unicode rules how you count the width. Which do change every year.
If you ask the user, it's an array of characters (aka extended grapheme clusters in Unicode speak).
If you ask the machine it's an array of integers (how many bytes make up an integer depends on the encoding used).
Nothing really considers them an array of code points. Code points are only useful as intermediary values when converting between encodings or interpreting an encoded string as grapheme clusters.
Not sure what you mean by this. A string is an array of bytes, in the way that literally every array is an array of bytes, but its not "implemented" with integers. Its a UTC-encoded array of bytes.
And what is the information that is encoded in those bytes? Codepoints. That's what UTF does, it lets us store unicode codepoints as bytes. There is a reasonable argument that the machine, or at least the developer, considers a string as an array of codepoints.
UTF-16 is an array of 16 bit integers.
UTF-32 is an array of 32 bit integers.
The machine doesn't know anything about code points. If you want to index into the array you'll need to know the integer offset.
The machine doesn't know anything about Colors either. But if I defined a Color object, I would be able to put Color objects into an array and count how many Color objects I had. You're being needlessly reductive.
> UTF-8 is an array of bytes (8 bit integers)
UTF-8 encodes a codepoint with 1-4 single-byte code units. The reason UTF-8 exists is to provide a way for machines and developers to interact with unicode codepoints.
Is a huffman code an array of bits? Or is it a list of symbols encoded using bits?
The concrete thing we're encoding is human readable text. The atomic unit of which is the user perceived character.
I'm curious, what use is knowing the number of code points in a string? It doesn't tell the user anything. It doesn't even tell the programmer anything actionable.
The way MoarVM does it is that it does NFG, which is sort of like NFC except that it stores grapheme clusters as if they were negative codepoints.
If a string is ASCII it uses an 8bit storage format, otherwise it uses a 32bit one.
It also creates a tree of immutable string objects.
If you do a substring operation it creates a substring object that points at an existing string object.
If you combine two strings it creates a string concatenation object. Which is useful for combining an 8bit string with a 32bit one.
All of that is completely opaque at the Raku level of course.
my $str = "\c[FACE PALM, EMOJI MODIFIER FITZPATRICK TYPE-3, ZWJ, MALE SIGN, VARIATION SELECTOR-16]";
say $str.chars; # 1
say $str.codes; # 5
say $str.encode('utf16').elems; # 7
say $str.encode('utf16').bytes; # 14
say $str.encode.elems; # 17
say $str.encode.bytes; # 17
say $str.codes * 4; # 20
#(utf32 encode/decode isn't implemented in MoarVM yet)
.say for $str.uninames;
# FACE PALM
# EMOJI MODIFIER FITZPATRICK TYPE-3
# ZERO WIDTH JOINER
# MALE SIGN
# VARIATION SELECTOR-16
So, given that the property is required to be present, some semantic or the other had to be chosen. I am sure there were debates when they were writing `String` about which one was proper.
> So, given that the property is required to be present, some semantic or the other had to be chosen.
Sounds like an invented solution to an invented problem. The programmer's speciality.
This conforms exactly to our intuition about what a "collection" is. Some (exact) number of items which share some meaningful property in common such that we can consider them "the same" for purposes of enumeration and iteration.
In the real world, we also have to decide what our collection is a collection of! Let's say we a pallet of candy bars, each of which is in a display box. If we want to ask "how many X" are on the pallet, we have to decide whether we're asking about the candy bars or the boxes. Clearly we should be able to answer questions about both; just as clearly, operations on the pallet should work with boxes. Because we don't want to open them, and even if we do want to open them, we have to open them, we can't just ignore their existence!
I assert that the extended grapheme cluster is the "box" around bytes in a string. Even if you do care about the contents (very often you do not!) you have to know where the boundaries are to do it! Because a Fitzpatric skin tone modifier all on its own has different semantics from one found within an emoji modifier sequence.
So it makes perfect sense for Swift to provide one blessed way to iterate over strings, and provide other options for when you're interested in some other level of aggregation. Which is what Swift does.
The alternative to a collection would be an iterator or other public method returning a collection-like accessor, which would be a good compromise.
Though if you were to choose a canonical collection quanta, the it'd probably be the grapheme cluster, yeah.
Unfortunately OOP can never be based; only functional or procedural programming can attain such standards.
Unfortunately, only Common Lisp and Lua do it that way.
Actor models are pretty based as well, and have a better historical claim to the title "object oriented programming" than class ontologies do, but that ship has sailed.
100% on the actor model. My visual programming platform is basically based on actors, but the core data model is cybernetic (persistence is done via self referentiality). Alan Kay got shafted by the creation of C++, OO in the original conception was very based.
I don't understand why the same is not the obvious solution for strings. len("🇨🇦") should be 1, because we as humans read one emoji as one character, regardless of the memory layout behind. Most if not all methods operate on characters (reversing, concatenating, indexing).
And then, if you need access to the low level data, the String.memory would contain the actual bytes... which would be different depending on the actual text encoding.
To be fair, some systems distinguish between size and length (with size expected to be O(1) and length allowed to be up to O(n)). For those systems proceed as parent.
To draw an absurd parallel "[emoji]".len() == 17 is equivalent to [1,2].len() == 8 (2 32 bit integers)
In my opinion the most useful result in the case the article describes is 5. There should of course be a way to get 1 (the number of extended graphemes), but it should not be the strings "length".
It'd be great to have a function for that in other scripting languages like Python, Ruby, etc.
There was an interesting sub-thread here on HN a while ago about how a string-of-codepoints is just as bad as a string-of-bytes (subject to truncation, incorrect iteration, incorrect length calculation) and that we should just have string-of-bytes if we can't have string-of-graphemes. I don't agree, but some people felt very strongly about it.
Nobody ever cares about unicode codepoints. You either want the number of bytes or the width of the string on screen.
UTF-32 codepoints waste space and give you neither.
>>> len("🇨🇦")
2I'll bet there are some systems in the PRC that don't include 🇹🇼.
But I believe this can be handled explicitly and well and I'm trying to do that in my fuzzy string matching library based on fuzzywuzzy.
I still haven't run into a situation where I need to count the logical number of glyphs in a string. Any system that we need to push a string into will be limiting things on the basis of byte counts, so a .Length check is still exactly what we need.
Does this cause trouble for anyone else?
Edit: confirmed, it was on HN just half a day prior. Probably why this article arrived too. Just surprised nobody reference this other one in this thread :)
Coming from python2, for me it was the first time I saw a language handle unicode so gracefully.
In C land it is called ucwidth (libunistring). Length is too arbitrary. Byte? Unicode points? Utf-8 length?
It’s asking why a skin toned (not yellow) facepalm emoji’s length is 7 when the user perceives it as a single character.
Tangent: Emojis are an interesting topic in regards to programming. They challenged the “rule” held by programmers that every character is a single codepoint (of 8 or 16 bits). So, str[1234] would get me the 1235th character, but it’s actually the 1235th byte. UTF-8 threw a wrench in that, but many programmers went along ignoring reality.
Sadly, preexisting languages such as Arabic weren’t warning enough in regards to line breaking. As in: an Arabic “character”[a] can change its width depending on if there’s a “character” before or after it (which gives it its cursive-like look). So, a naive line breaking routine could cause bugs if it tried to break in the middle of a word. Tom Scott has a nice video on it that was filmed when the “effective power” crash was going around.[0]
[0]: https://youtu.be/hJLMSllzoLA
[a]: Arabic script isn’t technically an alphabet like Latin characters are. It’s either an abugida or abjad (depending on who you ask). See Wikipedia: https://en.wikipedia.org/wiki/Arabic_script
Of course, we're taught to parse that as multiple discrete letters from an early age, so we don't get confused :)
Maybe it's some kind of bias?
With that mindset, emojis (and emoticon) can actually add context to interactions, considering how much context is lost when communicating over text. A simple smiley face at the end of a sentence can go a long way in my experience :)
HN however handles all of unicode just fine, it just chose to specifically exclude emojis (and a bunch of other symbols)
However, I don't like the rabbit hole it started to go into with gendered and colored emojis. There's never going to be enough. I wish we had stuck to color-neutral and gender-neutral, like original smileys :)
I find it also conveys too much meaning. I am generally not interested in knowing a person's gender or ethnic group when discussing over text... but I digress.
Eventually, Unicode will allow you to combine flag modifiers with human emojis. So you can have a black South Korean man facepalming.
This will trigger a war which ends industrial civilization, and is the leading candidate for the Great Filter.
What’s left is things like the max length for a title (not too problematic to count that in code points or bytes, I think)
The big risk, I think, is in users (mis)using rendering direction, multiple levels of accents, zero-width characters and weird code points to mess up the resulting page. Some Unicode symbols and emojis typically look a lot larger and heavier than ‘normal’ characters, switching writing direction may change visual indentation of a comment, etc.
Worse, Unicode rendering being hard, text rendering bugs that crash browsers or even entire devices are discovered every now and then. If somebody entered one in a HN message, that could crash millions of devices.
You can play mahjong, domino or cards (🀕, 🁓, 🃅) but not chess, you can show some random glyphs like ⏱ or ⤪ but not others, you can use box-drawing characters (┐) but not block elements, you can use byzantine musical notation (𝈙) but only western musical symbols (𝅗𝅥) and the notes are banned, you can Z̸̠̽a̷͍̟̱͔͛͘̚ĺ̸͎̌̄̌g̷͓͈̗̓͌̏̉o̴̢̺̹̕ just fine.
˙ǝsnqɐ ƃuıpıoʌɐ ʇɐ qoɾ pooƃ ɐ ɥɔns sǝop sɹǝʇɔɐɹɐɥɔ ɟo ɥɔunq ɐ ƃuıuuɐq ʎʃıɹɐɹʇıqɹɐ ǝsnɐɔǝ𐐒
It isn't the only arbitrary convention enforced here, and the sum of those conventions are what attracts the audience.
Edits:
⭕ that post was https://news.ycombinator.com/item?id=25237688
⭕ The rules seem pretty arbitrary. Recycling symbol U+2672 ♲ is allowed but recycled paper symbol U+267C is not. Chess kings ♔ are allowed but checkers kings aren't.
⭕ is allowed (for now).
⭕ I think the right thing to do would be to strip anything with emoji presentation http://unicode.org/reports/tr51/#Presentation_Style