It’s not wrong that "🤦🏼♂️".length == 7 (2019)
hsivonen.fi
hsivonen.fi
Did I miss the part where he explains this take? It's made up of 5 valid unicode code units. For a language where you're not supposed to need to know the byte size semantics, the correct length should be 5. What am I missing?
The close second being 17, because length in bytes. Is another fine way to represent this data, e.g. what a successful write of some sort would look like. Network or file.
I guess I'm basing this all on the idea that it's almost always a mistake to confuse how a program manages some data, vs how a drawing lib might. Your language shouldn't concern it self with how many glyphs it needs to draw... until you actually try to draw them.
You can’t use that information to know how much actual space it takes (in storage) as nobody sane stores UTF-32, you can’t use it to know much much logical space it takes (aka the user’s interpretation), you can’t use it to know how much visual space it takes (not that you can ever get that), and you can’t use it to segment or process the text.
A length in codepoints gives you nothing that’s really actionable, at least not that you’d need outside of a context where you could easily obtain it otherwise.
That’s… not useful?
I can’t say I remember ever caring knowing how many items I would be getting during an iteration[0]. If I want to set an iteration limit I can just… do that, using `islice` or some such.
[0] in python anyway, in lower level language there can be a utility in order to pre-allocate an output collection
Not often. There’s almost nothing useful you can correctly do with a sequence of code points.
Maybe it should just return None because the only useful thing is probably how much "space" it occupies on screen in a fixed-width font, but that's too difficult to know.
A string should be viewed as an opaque data type with views into it depending on what you're trying to do. You can have its length in the context of storage/retrieval/transmission (UTF-8 byte count), its length in the context of parsing (code points), its length in the context of editing (grapheme clusters) or length in the context of display (a bounding box in points when used in conjunction with a specific font and paragraph style attributes).
Claiming to provide an out-of-context length is strictly wrong because there's no such thing. This is where people get confused.
The attribute shouldn't be 'length' it should be something like 'countOfCodePoints' or exposed via a `CodePoints` type view.
It's particularly bad because so often (esp. for western programmers) 'countOfCodePoints' == 'countOfBytesInUTF8' == 'countOfGraphemeClusters' == """length""" so it's hella easy to accidentally write buggy software. Especially for people who don't know the above about unicode, which let's face it, most people don't. Not until they have to explain to their designer why they can't limit a label to '10 characters.' ("What do you mean there's no such thing as a character, and what am I trying to do?").
This is basically the tl;dr of the article but it's also my personal opinion.
All of this isn't about 'wrong' so much as 'imprecise and overloaded terminology making it easy to write buggy software through poor abstractions.'
If python explained which length you were getting, then this article wouldn't exist.
In JS, for(i in emoji) will iterate twice, but for(i of emoji) will iterate once.
;)
Remember, one of the big accomplishments (breaking changes) of python 3 is that all strings are Unicode, not byte arrays. If you want to view a dtring as bytes, you need to convert the string to bytes. But note the number of bytes depends on the ancoding u use (utf8, …).
The in memory size of a rune was just an implementation detail, and while it could be important for the programmer that the size of a rune was 2 bytes, this didn't mean the length of an array of 2 runes was 4.
I always liked the rune unit, and while my memory is hazy I think it was just code points.
I think part of the issue is programmers and apis mixing bit units for in memory representation of a conceptual value mapping (unicode), conceptual characters, stored size when encoded and so on ... without firming up those abstractions with interfaces. It gets lossy.
>>> import sys
>>> s = 'A' * 1000
>>> len(s)
1000
>>> sys.getsizeof(s)
1049
>>> s = '\N{SNOWMAN WITHOUT SNOW}' * 1000
>>> len(s)
1000
>>> sys.getsizeof(s)
2074
>>> s = '\N{MUSICAL SYMBOL G CLEF}' * 1000
>>> len(s)
1000
>>> sys.getsizeof(s)
4076
That's because you defined it first as "the number of elements in the array".
It is ambiguous however because that's not how people understand it when it comes to strings, and there are several counter-intuitive ways they expect it to behave.
Not to mention there might not be any "array". A string (whatever the encoding / representation) is a chunk of memory, not an array. That you can often use a method to traverse it doesn't mean it's in an array.
It doesn't really matter how some people think "how people understand" something, the documentation matters. Any string in any language is some ordered sequence of atomic text-like objects, so python's approach isn't unreasonable or unexpected, either.
So OP’s definition still does not apply to python’s definition of a string in an unambiguous manner, which was the claim they were making.
In fact, using the OP’s “unambiguous” definition leads to the conclusion that strings shouldn’t have a length function at all since it’s not an array.
Well, weren't we talking about things being "ambiguous"?
In Python we call what you describe a list. An array is something different. And people would expect something like the C (or the Java) data structure. In Python that would match the "array" lib package.
And that's just discussing the meaning of array - before we even get to whether a string is an array, and what this means.
>It doesn't really matter how some people think "how people understand" something, the documentation matters
In what universe? In practical use, clarity and non-ambiguous, least surprise names and semantics matter.
"But we clarify it in page 2000 of the documentation" is not an excuse. Nor is invoking moral or professional failings of those not reading the documentation. A good library design doesn't offload clearing ambiguity to the documentation.
>Any string in any language is some ordered sequence of atomic text-like objects
You'd be surprised. Especially since this isn't 1985 where strings were a bunch of 8-bit ascii characters, or even 1995, when widechar 16-bit arrays were "good enough" for Windows and Java, but we have not just non-ascii strings, but even variable length (e.g. utf-8) internal strings in mainstream languages.
Btw, python my try to overload the meaning of the words array and list, but the word “array” has a generic meaning in this branch of math called computer science (an ordered sequence of elements indexible in O(1)), which is how I used it here.
It can't be used to determine length in bytes (important for storage or network transmission), it can't be used to determine number of displayed characters, it can't be used to safely split a string at some position.
The only reason it has caught on is that it is easy to encode into UTF-8 and UTF-16, and that anything more interesting generally requires a language context and even a font.
I hope that future languages will get rid of this single string abstraction, and instead offer two completely separate types:
- symbol strings, which would only be usable for programming purposes and should probably be limited to ASCII
- text strings, which would be intended for human display purposes, with full Unicode support, and have APIs which answer things like "in the specified Culture, what is the length of human-recognizable characters of this string" or "what is the seventh human-recognizable characters in this string in the specified culture"
There's no reason to pay the conceptual cost of Unicode for representing field names or enums (and yes, I don't believe supporting Unicode identifiers is a good idea for a programming language; and note that I am not a native English speaker, and while I do use an alphabet, ASCII is missing some of the letters&symbols I use in my native Romanian). And there's no reason to settle for the misleading safety of Unicode code points when trying to process human displayable text.
If you care about the number of bytes, or to operate on individual bytes, then convert to utf-8,16 or 32, and operate on the bytes object. If you wish to operate on grapheme clusters, then you could probably find some 3rd party Python library that allows you to represent and operate on strings in terms of grapheme clusters.
There are basically only two things that can be done with a Unicode codepoint: encode it in bytes for storage, or transform it to a glyph in a particular font or culture.
You can't even compare two sequences of Unicode codepoints for equality in many cases, since there are different ways to represent the same text with Unicode. For example the strings "thá" and "thá" are different in terms of codepoints, but most people would expect to find the second when typing in the first. Even worse, there are codepoints which are supposed to represent different characters, depending on the font being used / the locale of the display (the same Unicode codepoints are used to represent related Chinese, Japanese, or Korean characters, even when these characters are not identical between the three cultures).
The answer is of course both, depending on context. You certainly don't want a fuzzy match when, say, decoding a protobuf, but you also don't want a codepoint match when looking up user input.
What most modern languages have settled on is having a Unicode codepoint array type, typically called string or text, and an array of bytes type. However, common string operations are often only provided for the text type, and not the bytes type - which becomes very annoying when doing low level work and using bytes for text, and hoping for simple text operations.
> Strings are immutable sequences of Unicode code points
(from https://docs.python.org/3/library/stdtypes.html#text-sequenc...)
> The Length property of a string represents the number of Char objects it contains, not the number of Unicode characters.
https://learn.microsoft.com/en-us/dotnet/csharp/programming-...
https://learn.microsoft.com/en-us/dotnet/api/system.text.run...
> The attribute shouldn't be 'length' it should be something like 'countOfCodePoints' or exposed via a `CodePoints` type view.
That’s a great way to explain it and something I’ll keep with me. Thanks
so do 'www' and 'iii' (though less dramatically), that't not a foreign concept to designers, not sure they'd want to bound 'www' to the width of 'iii'
If the user can freely change a font it is impossible to send a string of 3 graphene clusters because you won't know if it actually will show up as 3 to client or a different number.
Thank you. Have my upvote.
And by western you mean american, right? You can't even use ASCII in the UK --- '£'.
Example: https://www.llyw.cymru/ (Welsh government homepage).
Bits of https://www.highland.gov.uk/press (Scottish Highlands council).
(I'll admit most people who don't speak these languages won't bother with the diacritics, unless it's as easy as typing English.)
Or in a single font. It's impossible to render any mixed combination of simplified Chinese, traditional Chinese and Japanese with a single font (Korean might be also involved, but not sure about that). Even in Unicode, characters might share the the same space which don't have anything common in their looks, nor in their meaning. That applies to the shared CJK space as well. Btw. Japanese has halfwidth and fullwidth characters.
Well, that could be phrased better. Many such mixed combinations would encounter no problems. There is "Han Unification" in Unicode, in which certain graphical forms are declared equivalent and the intent is that they display as Japanese characters if you print them in a Japanese font, but as Chinese characters if you print them in a Chinese font. 直 is a good example of how that looks; try viewing it in different fonts.
But nobody likes unification and explicit fixed forms are constantly being defined so that it's possible to talk about them. Imagine if I wanted to write "in Old English, the word for dog was hund"... except that your font automatically replaced the sequence hund with a special ligature that looks exactly like dog.
So we have separate unicode points for ⻘ (modern, CJK RADICAL BLUE) and ⾭ (old, KANGXI RADICAL BLUE), and for ⿓ (traditional Chinese, KANGXI RADICAL DRAGON), ⻰ (simplified Chinese, CJK RADICAL C-SIMPLIFIED DRAGON), and ⻯ (Japanese, CJK RADICAL J-SIMPLIFIED DRAGON). Interestingly, the dragon characters are all considered different according to the original "Han Unified" specification, where they are CJK UNIFIED IDEOGRAPH 9F8D, CJK UNIFIED IDEOGRAPH 9F99, and CJK UNIFIED IDEOGRAPH 7ADC. In contrast, there is only the one "unified" form of 直, CJK UNIFIED IDEOGRAPH 76F4, but you can refer to its Chinese form explicitly with CJK COMPATIBILITY IDEOGRAPH FAA8 and to its Japanese form with CJK COMPATIBILITY IDEOGRAPH 2F940. (My browser font fails to render either of those.)
It was never possible to rely entirely on the font to handle dealing with simplified vs traditional characters for you, for the obvious reason that their mapping is not one-to-one. In simplified Chinese, 后 means "after"† or "behind" and it also means "empress". In traditional Chinese, "after" and "behind" would be 後. And "empress" would be... 后. This means there can be no way for a traditional Chinese font to determine what it should display if you write 后.
Ultratraditional Korean hanja participate in the same variation of forms that we see between Chinese and Japanese. But it isn't normal to write Korean in hanja outside of very specific contexts. Hangul are radically different and belong to a separate part of unicode entirely.
† "After" in time. "After" in sequence is 下, "below".
1. UTF-8 byte length
2. Code point count
3. Extended grapheme cluster count
#3 makes sense for users but it doesn’t make sense for programs which often need to work at the code point level.
I expect programming language string length to obey the law:
len(a ++ b) = len(a) + len(b)
Code point length is the most useful for people who are actually writing string algorithms based upon Unicode.
UTF-8 length is useful for people who are treating strings as opaque byte sequences, but in that case they should be using a bytes/buffer object and not a string object, except in very low-level languages that don’t want to pay an encoding/decoding cost.
Extended grapheme cluster count is useful for people who are constructing certain kinds of user interfaces, where the number of characters is limited for a policy rather than memory or width reason.
i.e. when length limits are imposed by human policy, grapheme cluster count is the way to go. Length limits for memory reasons should rather be in UTF-8 bytes. If you need a limit for visual width reasons then you need to go measure the string in pixels, otherwise I’m going to put a U+FDFD in there and ruin your day.
UTF-16 length can GTFO.
What algorithms would you be writing against code points?
The string will be fine if you only move to, split and concatenate at indices outside those grapheme clusters. But that is also true when indexing by bytes or UTF-16 code units.
So in some senses, indexing by bytes is just as good as indexing by code points, but faster. Either way to avoid mangling strings you need to restrict the indices of whatever type to meaningful character boundaries.
If you have decided to avoid string indices inside grapheme clusters, there comes the awkard question of what should you do when editing text in an environment rendered with font ligatures like "->" rendered as → (rightward arrow). From one perspective, that's just a font. From another, the user sees a single character yet there are valid positions (such as from cursor movement and character search) that land mid-way through the character, and editing at those positions changes the character. Neither is clearly best for all situations.
For example, let’s say you want to define a “natural sort” order that sorts e.g. “A2” < “A10”. To do that you divide the string at boundaries between code points in ranges of each numeral type that you are supporting (e.g. western numerals, Arabic numerals, Chinese numerals).
We could use grapheme clusters, but the grapheme cluster boundary points change as unicode evolves, and not all systems update at the same time. Separating strings based on grapheme cluster boundaries also requires a big lookup table to be embedded in every app. Unicode codepoints are obvious, stable, and easy to work with. And they're encoding-agnostic, so there's no weird UCS2-to-UTF8-bytes conversion needed in javascript, C#, etc.
Going UTF-8 is fairly clearly superior: it will be the wire format, even if it’s not the language’s string format, so now environments that use UTF-8 strings never need any conversions (apart from decoding escape sequences, most likely).
Much as I hate UTF-16, I would even be inclined to argue that UTF-16 was a better choice than code points, as it will reduce the amount of extra work UTF-16 environments have to do, without changing how much UTF-8 environments have to do at all; but it also has the disadvantage that validation is vanishingly rare in UTF-16, so you’re sure to end up with lone surrogate trouble at some point, whereas UTF-8 tooling has a much stronger culture of validation, so you’re much less likely to encounter it directly and can much more comfortably just declare “valid Unicode only”.
Yes, code points is a purer concept to use. I don’t care: it’s less efficient than choosing UTF-8, which adds negative-to-negligible complexity. Please, just abandon code point indexing and embrace the UTF-8.
The problem with utf8 byte offsets is that it creates a data validation problem. In diamond types I’m using document positions / offsets in my wire format. With utf8 byte offsets, you can receive changes from remote peers which name invalid insertion positions. (Ie an insert inside a character, or deleting half of a codepoint). Validating remote changes received like this is a nightmare, because you need to reconstruct the whole document state to be able to tell if the edit is valid. Using Unicode codepoints makes invalid state unrepresentable. So the validation problem goes away. (You might still need to check that an insert isn’t past the end of the document, but that’s a much easier check).
Almost all application programming languages use utf16 anyway, (javascript, c#, swift, Java) so you still need to convert positions anyway. Even in rust it’s common to see line/col positions from text editors.
Using utf8 byte offsets just doesn’t really give you any benefits in exchange for making validation much harder.
> Almost all application programming languages use utf16 anyway, (javascript, c#, swift, Java)
Swift 5 switched to UTF-8: https://www.swift.org/blog/utf8-string/. I’m hopeful that other UTF-16 environments might eventually manage to switch to UTF-8 internally despite retaining some UTF-16 code unit semantics for compatibility; two projects have already demonstrated you can very practically do this sort of thing: Servo from fairly early on (WTF-8 despite the web’s UTF-16 code unit semantics), and PyPy since 7.1 (UTF-8 despite code point semantics, not sure what they do about surrogate code points). I know the web has largely backed away from UTF-16 and uses code point semantics (well, scalar values plus loose surrogates) on almost all new stuff, with good UTF-8 support too.
Is that true? Maybe it's not "invalid", but you might very well slice through the middle of a grapheme cluster.
But then again, I know that some CRDTs struggle with interleaving, and maybe grapheme-awareness could help things out in some way or other. I dunno.
And yes, using unicode codepoints, buggy clients might insert extra unicode characters in the middle of a grapheme cluster. But ... Eh. Fine. I'm not super bothered by that from a data validation perspective.
Why don't I have the same attitude toward invalid UTF8? I mean, the CRDT could syncronize arbitrary arrays of bytes that by agreement contain valid UTF8, and treat it as user error in the same way if that happens? Two reasons. First, because some languages (eg rust) strictly enforce that all strings must contain valid UTF8. So you can't even make a document into a String if it has invalid UTF8. We'd need a try_ codepath, which makes the API worse. Secondly, languages like javascript which store strings using UTF16 don't have an equivalent encoding for invalid UTF8 bytes at all. Javascript would have to store the document internally in a byte array or something, and decode it to a string at the frontend. And thats complex and inefficient. That all sounds much worse to me than just treating the document as a sequence of arbitrary unicode codepoints - doing which guarantees correctness and we don't need any of that mess.
I’m not sure if there’s any spec that defines any sequence of Unicode scalar values as “invalid” (though there’s certainly a lot that’s obviously wrong, like some forms of script-mixing). Grapheme cluster segmentation doesn’t concern itself with meaningfulness, but just doing something with what it has; so if you inject something into the middle of what it decided was a cluster, it’ll just split it differently.
> treating the document as a sequence of arbitrary unicode codepoints
Though I hope you validate that they are Unicode scalar values and not code points. Surrogates are terribad.
Yes, my mistake. I do mean scalar values. I am constantly confused about the terminology for unicode. (Unicode code point? Scalar values? (Character?) Surrogate pair? Is there a term for half of a surrogate pair?)
UTF-16 length is only useful if you are moving UTF-16, perhaps for interop with other software that chose UTF-16. Remember to pass on your condolences and look forward to a day when we don't do that any more.
Java/the JVM says hi!
Arguably, "how many bytes does this string occupy in memory/on disk (e.g. in class files)" is a pretty useful thing to be able to ask.
Sure. On disk those strings are (modified) UTF-8 of course, is that what you meant ?
This is still relevant for many databases as well, as far as I remember.
I get what you're intending, but it's kinda funny because that string actually is 7
One question I have is which of these are fixed across Unicode standards.
(a) Byte count for storage and (b) code point count are definitely fixed, Unicode provides an explicit guarantee of the latter: https://www.unicode.org/policies/stability_policy.html
(c) Visual width will change depending on the system.
But what about (d) grapheme count? If I make a microblogging site which limits post length to 144 graphemes, can my database invariants break when I upgrade my version of Unicode?
[0] > "\c[FACE PALM]".chars
1
[1] > "\c[FACE PALM]".codes
1
[2] > "\c[FACE PALM]".encode.bytes
4
[3] > "\c[FACE PALM]".encode
utf8:0x<F0 9F A4 A6>
[4] > "\c[FACE PALM]".encode('utf-16')
utf16:0x<D83E DD26>
Then programmers will pick a random view and assume its length equals the number of characters and bytes. Also the grapheme view will introduce an OS-dependent bug.
An area this makes sense is, what do you expect to get if you do something like:
emoji = " "
print(emoji[:3])
Modern strings are complex objects that have evolved a bit past char[] or byte[].
Why should it not? You’re literally breaking the content.
Though in reality, indexing strings is a broken operation. That you’re using it at all is the core issue.
> Modern strings are complex objects that have evolved a bit past char[] or byte[].
And yet that’s exactly what you’re advocating, just with 21 bit chars.
Strings are just an array of unicode codepoints rather than "characters", so all I'm doing is asking for the first three of those codepoints.
> Though in reality, indexing strings is a broken operation. That you’re using it at all is the core issue.
Substring is a broken operation? What's the justification for that idea?
"Ice trays are just a pile of molecules rather than "cubes", so all I'm doing is separating those molecules", he states as he activates the igniter.
> Substring is a broken operation? What's the justification for that idea?
You take a thing and you mangle beyond recognition without regards for its purpose or meaning. That's like considering the jaws of life a normal part of opening a door to take a piss at work.
If the user gives you what, as far as they're considered, is a glyph. And you return a completely different glyph. You've mangled their data.
Absolutely.
Whether this is a good idea on the whole is debatable, there's even a full PEP talking about the security concerns around doing it this way[1].
However, given this is how it works, the behaviour displayed makes complete sense to me and is the best of the bad choices presented by needing multi-byte strings.
Well, why not? There are a lot of things that people would want to call string.length for — drawing little equals signs under text in a terminal, for a frivolous example — where that’s the whole reason they’re making the call. Off the top of my head I’m not really sure how you solve that with variable-width characters if there’s no way to separate out or count them.
Separation of concerns?
The context is "I guess I'm basing this all on the idea that it's almost always a mistake to confuse how a program manages some data, vs how a drawing lib might. Your language shouldn't concern it self with how many glyphs it needs to draw... until you actually try to draw them."
This means that shouldn't be some generic "length" method, but appropriate separate-concerns methods (plural), some of which (e.g. regarding character width in pixels when rendered) even belong to a drawing lib and not the language at all.
The parent's point is that length (bytes), characters (count), and glyphs (size, shape) are different concerns. The latter would concern a drawing lib or a renderer, but not be a core string method (which should concern itself with the abstract notion of characters and the concrete notion of bytes).
In the words of the article: “The choice of UTF-32 (or Python 3-style code point sequences) arises from wanting the wrong thing.”
“Not needing to know the byte size semantics” seems reasonable, but it simply isn’t a useful goal. The things it makes easier or faster (knowing how many code points there are, and O(1) indexing by code point) are things you shouldn’t be doing—and when you have to interact with the rest of the world, you now have a more expensive encoding step that is always needed, rather than just sometimes if you’d chosen UTF-8 or even UTF-16.
But the overall tone of the article is still bashful. Caring about the internal representation of strings and bashing UTF-32 feels lame and angry. (Especially if I'm right about the multi-rep nature of Python 3: their choice is good for most text, and they could add a UTF-8 internal rep in the future, although that would probably break enough code that expect the UTF-32 value for len() that it is not worth it.)
Since you mention the varying internal representation of strings: that’s PEP 393 <https://peps.python.org/pep-0393/>, which landed in CPython 3.3, and it generally made things slower by introducing a lot of branching and reallocating and such, though it does speed up some cases due to having to touch less memory, and some methods due to being able to quickly rule out possibilities (e.g. str.isascii can immediately return False for a canonical UCS-2 or UCS-4 string, since if they were ASCII they’d have been of the Latin-1 kind).
PEP 393 was done because people were complaining about how much memory their UCS-4 encoding had been using.
Note also how PEP 393 retains code point semantics: Latin-1 (Unicode values 0–255), UCS-2 or UCS-4; all fixed-width encodings of code point sequences. PEP 393 does also allow a string to cache UTF-8 representation (see PyCompactUnicodeObject.{utf8, utf8_length}), choosing “UTF-8 as the recommended way of exposing strings to C code”, but I gather this isn’t used very much.
(Related: PyPy 7.1 shifted to using UTF-8 exclusively internally, and according to https://www.pypy.org/posts/2019/03/pypy-v71-released-now-use... got a “nice speed bump” out of it.)
Almost. 17 is the number of bytes it occupies in memory. But you don't generally dump memory directly to disk or network. It happens to make sense (and it's convenient) for utf8 strings. But it's better to be explicit about that. Python is better. If you care about bytes, say you care about bytes:
len(bytes(" ", "utf8")) == 17
len(bytes(" ", "utf-16-be")) == 14
len(bytes(" ", "utf-32-be")) == 201. It's five "Unicode scalars," that's the name for the top-level logical unit. The term "code points" technically refers to a lower-level concept, one that varies across encodings, just not as much as the number of bytes. I didn't know that, and it's the helpful thing I learned from this article. UPDATE: And it's also not true, sorry. "code units" are the lower-level concept from the article, "code points" are a more expansive category at the same level: https://www.unicode.org/versions/Unicode10.0.0/ch03.pdf#G740...
2. The author takes it as an unstated assumption that top-level logical structure is useless because any specific usage either ignores all structure or has a point at which low-level structure comes into play. (That assumption is false: Top-level structure is useful for keeping track of what you are doing and as a sort of "common currency" for translating between different low level representations. For example, see the very first table in the article.)
In this case, the tidbit is: "grapheme clusters exist and they are useful".
The misleading part is that the article draws a false equivalence between what the author calls "UTF-32 code units" and UTF-16 code units.
UTF-32 code units are Unicode code points. This is a general Unicode concept that exists in all Unicode encodings. UTF-16 code units, on the other hand, are an implementation detail of UTF-16. It is wrong to present them as equally arbitrary concepts.
They’re not. UTF-32 code units have a 1:1 mapping to USVs, surrogates are not valid.
> It is wrong to present them as equally arbitrary concepts.
Is it? It’s not like they’re any more useful. Arguably less so, UTF-16 is at least a somewhat common storage medium.
This is true, although very pedantic and irrelevant to the point of my comment. The distinction only matters when you're dealing with ill-formed strings.
BTW, Python strings can store surrogates.
>Is it? It’s not like they’re any more useful. Arguably less so, UTF-16 is at least a somewhat common storage medium.
If you aren't directly dealing with UTF-16, UTF-16 code units aren't useful at all.
Code points/USVs, OTOH, are the building blocks of Unicode strings and various Unicode algorithms operate on them. They're low-level, but not useless.
Would that it were only so. Instead, UTF-16 ruined Unicode for everyone with the abomination that is surrogates, and almost nothing that deals with UTF-16 actually asserts well-formedness, and ill-formed UTF-16 cannot be represented in UTF-8 or UTF-32.
UTF-32 and UTF-8 code units are truly implementation details of their encodings, as other encodings don’t need to know about them in any way. UTF-32’s code units are a trivial mapping between scalar values and 32-bit values (not four-byte values, given the big- and little-endian variants), but that still causes UTF-32 code units to be semantically distinct from Unicode scalar values. U+12345 is a Unicode scalar value and doesn’t have any “size”: it’s an abstract value. 0x00012345 is a UTF-32 code unit, a 32-bit value.
If you’re talking about encoding of Unicode scalar values, you talk about code units. Even when talking about UTF-32, the code unit/scalar value semantic distinction is worth maintaining.
What?
Why should we expect invalid utf-16 strings to be representable in utf-8 or 32? I don't see anyone trying to represent invalid utf-8 in utf-16 or 32.
We shouldn't care. UTF-16 should just be an encoding and its internal details shouldn't leak into Unicode code points. There's just no good reason to exclude code points U+D800–U+DFFF merely because 0xD800–0xDFFF happen to be used specially in UTF-16 encoding, just like U+0080–U+00FF aren't excluded merely because (most of) 0x80–0xFF are used in UTF-8 encoding.
It does occupy 1/32 of the BMP, displaying a couple thousand potential actual characters (making them take an extra byte in UTF-8, and an extra two in UTF-16).
So: the problem is that the Unicode standard was compromised for the sake of a buggy encoding (they should instead have written UCS-2 off as a failed experiment), and every implementation that uses that buggy encoding is itself buggy, and that bugginess has made it into many other standards (e.g. ECMAScript).
The problem comes when trying to convert A to utf-8. Is this the leak you are talking about?
The other is… well, much the same really, but when it makes it into specs that others have to care about. The web platform demonstrates this clearly: just about everything is defined with strings being sequences of UTF-16 code units (though increasingly new stuff uses UTF-8), so then other things wanting to integrate have to decide how to handle that, if their view of strings is different: whether to be lossy (decode/encode using REPLACEMENT CHARACTER substitution on error), or inconvenient (use a different, non-native string type). Rust has certainly been afflicted by this in a number of cases and ways, generally favouring correctness.
UTF-16 is a hack. Unicode originally thought 65,535 values should be enough to represent all human languages and so 16-bit fixed size characters would work. However, that proved incorrect. UTF-16 was a hack to try retrofit this onto systems that had already adopted this 16 bit character (Java, Windows NT, etc).
An entertaining summary of the situation is here:
But it’s not. That style is about tone and the article doesn’t exude that kind of tone.
Do you see the author scolding programmers for being ignorant Americans, for having unknown unknowns, or for not being “professionals”? Well, me neither.
https://ruby-doc.org/3.2.2/String.html#class-String-label-Me...
https://hexdocs.pm/elixir/1.12/String.html#module-code-point...
`codeunits(str)` lets you access the underlying code units, which is bytes for the default UTF-8 encoding. (External packages implement UTF-16 and others, and there `codeunits` could return non-bytes, for eg. 16-bit values for UTF-16.)
The Unicode stdlib provides `graphemes(str)`, the equivalent of `each_grapheme_cluster`.
5 is the number of code points, and 17 is the number of bytes. Both are reasonable answers.
7 is the number of code units for utf-16. Seems like the least useful option.
For 5, the idea is that while you might want to iterate code points, the total number of code points is less useful than either grapheme count or byte count. I think that argument makes sense.
It's not a real measure of bytes though. It's the count of bytes in an encoding scheme that is (probably) neither what you use to communicate with the outside world nor what your language runtime uses. (And certainly it's no better than 5, since that's also a measure of bytes in a particular encoding).
I don't think just about anything communicates in UTF-32. 5 is basically just a codepoint count, and as such I don't think its usefulness rating should be between the byte counts.
> and BOTH of those are insane for sticking to it
They don’t really have much of a choice because they exposed those semantics as part of the string interface (or for Windows the interaction is slow low level it can’t be hidden), they have performance guarantees and behaviours which matches that.
It’s also why Python uses UTF-32, and went through the entire PEP-393 / FS complication to try and stop blowing up memory left and right: the core team considered that switching strings to UTF8 was a bridge too far.
There are approximate solutions, but they come with their own costs and complications (e.g. pypy uses UTF8 strings with lazily constructed indices to emulate UTF-32 strings).
Windows and Java (and Javascript) adopted unicode at a time when it was thought that 64k code points would be enough for everyone. Then they prioritized backwards compatibility over anything else. Most of us have benefited from their insistence on backwards compatibility in some form or the other, so I'm really not in a position to complain about it :-/
That said, IMHO any "length" property (as opposed to `codepoints` or `bytes`) on a UTF-16 string should definitely be deprecated.
Not my experience at all. The article points out that even languages that are committed to an UTF-16 interface prefer to use other internal storage representations, and I can't remember the last time I saw it used in a transfer format.
len(bytes(" ", "utf8")) == 17
void main() {
String x = "(that emoji here)";
System.out.println("Chars: " + x.length());
System.out.println("Codepoints: " + x.codePointCount(0, x.length()));
System.out.println("As stream of chars (= UTF16-esque with surrogate pairs):");
x.chars().forEach(System.out::println);
System.out.println("As a stream of codepoints:");
x.codePoints().forEach(System.out::println);
}
Chars: 7
Codepoints: 5
As stream of chars (= UTF16-esque with surrogate pairs):
55358
56614
55356
57340
8205
9794
65039
As a stream of codepoints:
129318
127996
8205
9794
65039
The fact that the codepoint counter is a very awkward `codePointCount` call has the dubious benefit of highlighting this method loops through and therefore would be quite slow on very large strings.
Did not you still need the `java --source 11 ${filename_without_java_extension_because_JEP_330}` to use it? And you still need a wrapper class with a static method main in it.
One of the preview features he's using is JEP 445[1] that allows you to omit the wrapper class, as well as the arguments to main and the public and static modifiers.
The country flags are each made of two Unicode code points, which Unicode calls Regional Indicator Symbols. There are twenty six, one for each of the Latin capital letters A through Z. These are used to encode a flag by writing the ISO two letter country code from ISO-3166-1 e.g. F + R is France, you get a French flag.
Given your black flag experience, and the fact this is an English language forum, I'd guess maybe you wanted a flag for some entity that isn't a UN member state or some sort of recognised similar entity (e.g. the European flag EU symbolising the continent of Europe) and thus doesn't have an ISO two letter code, such as California or Wales. Those are built from a waving black flag plus their long ISO-3166-2 region code
Especially because the article says that Python's take is the worst.
They are less of an implementation detail.
Grapheme > Code point > Encoding > Endianness > Media
It's all "implementations" but some are lower then others
If you have to care about the visual representation of text then you probably need to be familiar with other concepts as well.
Does it? Do you have a link?
[edit] I looked up the spec and here is what it says.
> The Unicode Standard does not define what is and is not a text element in different processes; instead, it defines elements called encoded characters. An encoded character is represented by a number from 0 to 10FFFF_16, called a code point. A text element, in turn, is represented by a sequence of one or more encoded characters. [1]
The definition of 'text' in the context of Unicode seems to explicitly not be defined as a sequence of code points, but rather a more nebulous sequence of aggregations of code points. It's probably closest to a grapheme cluster but they seem to want to avoid pinning it down.
[1] https://www.unicode.org/versions/Unicode15.0.0/UnicodeStanda... p. 7 (1.3 - Text Handling), PDF page 33.
Text elements are an abstract concept whose definition depends upon what is being processed. It might be a grapheme, it might be word, etc...
I'm open to being wrong but I would be very surprised if they defined text as a "series of code units" the count of which can vary by encoding even for the same character. IMO in this context 'character codes' would likely be far more consistent with 'code points' and they're just trying to differentiate between styled and un-styled text. Whereas the 1.3 definition appears to be trying to make an authoritative definition of 'text.'
If we read 2.2's "character codes" as code points, then that can be multiple code points as referenced in 1.3
[edit] I originally flipped 'units' and 'codes' - cleaned it up.
Unicode is one of the most poorly understood topics. I think the confusion stems from 1. most programming languages getting the abstraction wrong, and 2. programmers trying to reconcile their non-technical interpretation of what "character" means.
But 7 is not the number of code points/USVs - that's the number of UTF-16 code units. The string is 5 USVs. If UTF-whatever is an implementation detail, wouldn't the correct answer to length be 5?
What am I missing haha.
They aimed to work with a nice, clean, abstract concept, untrammelled by encoding squabbles. They failed badly by choosing code units rather than scalar values (Unicode strings are sequences of scalar values, not code points—'\udead' is a valid Python string, but you can’t encode it into any UTF-* format since [U+DEAD] is not a valid Unicode string).
Then they also neglected to observe that they were optimising for something that you should practically never be doing, so that now everyone has to pay the costs. As the article summarises it part-way through: “The choice of UTF-32 (or Python 3-style code point sequences) arises from wanting the wrong thing.”
Seriously, Python 3’s approach is almost the worst of all available worlds. I loathe UTF-16 with such fiery passion that I can’t quite bring myself to say Python 3’s approach is worse than weak UTF-16, but it’s of similar badness in practical terms. The decisions were very clearly made by people that were not expert in the domain and who were caught up in a Concept of Mathematical Purity. They’ve since walked some of it back as far as they could, and I think did recognise it all as a mistake (no citation, just a vague memory of seeing such an admission), but they can’t fix it all properly without a breaking change.
However such an API would be pretty cumbersome because for all non-edge cases (read: a western language and a reasonable encoding that language - which when looking at world demographics is a very narrow way of saying non-edge case) we just want to ignore all that fancy stuff and assume it's latin-1/ascii and use "Length" and get on with it, usually accepting that it doesn't work for many scripts or emoji.
So almost every api I have encountered has both the dangerous or ambiguous "length" and any number of the more specific counts. Good? No. But good enough, I guess.
A much worse related API that exists every where is that for parsing and formatting numbers to and from text. How that's done "depends" but most languages I have seen - unfortunately - offers a "default way". In the worst examples - looking at you .NET - this default uses the system env and assumes formatting and parsing numbers should us the OS locale. Horrible horrible idea when used in conjunction with automatic type conversions. WriteLine($"The size is {3.5}"); shouldn't print "3.5" in the US and "3,5" somewhere else.
Actually why?
A great example how bad it is would be that the C# compiler repo for a very long time had tests that failed for everyone with non-US formatting.
Apparently the thing to use is a library with a very strange name, which does glyph placement. I’ll go look for it.
EDIT: harfbuzz https://harfbuzz.github.io/why-do-i-need-a-shaping-engine.ht...
The “length of an emoji” depends on the data type used to represent it. Its that simple and that correct.
I had to learn what pointers were, but it was OK, haha.
That said, maybe people who don't have "The Knack" would be better off learning a slightly harder language first..
So... Java/C#/Kotlin?
https://unicode-x-ray.com/?t=%F0%9F%A4%A6%F0%9F%8F%BC%E2%80%... (sorry if link looks scary, that's just the URL encoding of this emoji)
A codepoint is the "smallest useful addressable unit" when dealing with Unicode text, so it makes sense that's the default.
It's also comparatively expensive to address grapheme clusters.
I can see that iterating through by codepoint could be useful for some of those cases, but I still can't see why you'd ever want to index by codepoint?
> =LEFT(F280,2) & LEFT(F281,2) & LEFT(F282,2) & LEFT(F283,2)
Since the Emojis are actually 2 bytes.
(It also has `length` which equals codepoints.size)
Same goes for .each_byte.size, but for that you have the faster .bytesize method that avoids the intermediate Enumerator.
" ".length
[..." "].length
[...new Intl.Segmenter('en-US', { granularity: "grapheme" }).segment(" ")].length
Mind you, this is inefficient due to unnecessarily constructing an array. Here’s a more efficient version, though the difference will normally be fairly slight:
function codePointLength(str) {
let len = 0;
for (const c of str) {
len++;
}
return len;
}
> [...new Intl.Segmenter('en-US', { granularity: "grapheme" }).segment(" ")].length
Caution: Intl.Segmenter may not be available, so be sure to have a fallback if you want to use it. Chromium shipped it 2½ years ago, Safari 2 years ago, and Firefox hasn’t shipped it yet. (No idea why and I haven’t looked. It’s not always the case: I know of other Intl things that Firefox has shipped first.)
The Swift approach can also asymptotically reach perfection. The other approach will be broken forever.
These are categorical improvements in design
That's why in the article you see Swift running on Ubuntu 14.04 returning len==2 while the same code on Ubuntu 18.04 returns len==1 for the same emoji string.
IMO that's a big philosophical question here: do we accept that "string length" means something you can't compute for arbitrary strings unless your code is receiving annual updates containing the latest Unicode interpretation instructions?
And you can compute it, you can pin a Unicode version and ship it in the language if those platform differences are unbearable (so, you can actually isolate it and simply ignore the future :))
The bigger philosophical question: how much longer do we accept that "string length" does not measure the most intuitive measure of string length and call a byte a char?
Then upon opening the post I was 100% ready to believe that js has three different string length functions that all handle Unicode differently.
Even for limiting input field sizes byte count is much better, as otherwise you are opening up yourself for unicode denial of service. I think the game Minecraft has such an exploit where you can fit in absurd amounts of utf-8 data (to the point of data corruption in multiplayer games) since it's limited by visual length.
My personal favorite dealing with UTF-8: pretend it's ascii and assume everything above 128 is an alphabetic character. It just works. For 99% of use cases it doesn't matter if the content is emojis, families of emojis, or ancient sumerian scripts. You can parse JSON and most other formats this way without caring about code points at all. The trend of unicodizing everything was a mistake, just treat strings as bytes and parse them as utf-8 only when you really need it (like when building a text editor or a browser engine from scratch).
utf-8 is a variable width encoding and if you treat it like ascii your software just isn't going to work outside of the ascii range.
If a code point is outside the ascii range (0-127 inclusive), then it's utf-8 encoding is also guaranteed to not contain any ascii bytes. So as long as you treat anything inside 128-255 as "some unknown character", the utf-8 code points will be preserved and eventually displayed when the byte sequence is parsed as utf-8 by your terminal/browser/whatever.
mysql> WITH chars AS (SELECT ' ' c)
-> SELECT LENGTH(c), CHAR_LENGTH(c) FROM chars;
+-----------+----------------+
| LENGTH(c) | CHAR_LENGTH(c) |
+-----------+----------------+
| 17 | 5 |
+-----------+----------------+
1 row in set (0.01 sec)
This should be easier to reproduce:
mysql> WITH chars AS (SELECT 0xF09FA4A6F09F8FBCE2808DE29982EFB88F c)
-> SELECT CONVERT(c USING utf8mb4), LENGTH(c), CHAR_LENGTH(c) FROM chars;
+--------------------------+-----------+----------------+
| CONVERT(c USING utf8mb4) | LENGTH(c) | CHAR_LENGTH(c) |
+--------------------------+-----------+----------------+
| | 17 | 17 |
+--------------------------+-----------+----------------+
1 row in set (0.00 sec) perl -e 'use utf8; print length(""). "\n";'
1 iex(3)> String.length(" ")
1
> length(' ')
Which is: [129318, 127996, 8205, 9794, 65039]
For all intents and purposes, a user will count it as one character. Truncating the string without including the whole cluster would change the meaning of it, and is not an operation anyone would do as a general purpose thing any more than someone would want to randomly replace the last character with random letters.
It looks like one character. I'd rather APIs let us continue pretending it is one character.
GSM-7 uses 7 bits per character and a correspondingly limited character set that allows for 1120/7 = 160 characters.
UCS-2/UTF-16 uses 16 bits per character for a total of 1120/16 = 70 characters per message.
String = List ( Char )
Chars don’t have a length, like a number doesn’t have a length - unless you talk about number of bits. If you are working with strings stick with strings. The string of a single character should be “1”. Just enforce proper typing. Anything else is not consistent.
A "character" is not a well defined term in Unicode, rather the "base" that does not vary across implementations is code points, which is what Python measures when you get the length of a string.
If you put unicode characters consisting of multiple data points into a String, it necessarily increases the amount of chars. There's no way around that. Because there is no such thing as a UnicodeChar type in Java. You can't actually assign multi data point unicode characters to a char.
Essentially all the workarounds for a 'correct' unicode character count in a String would either end up using a different and probably way more expensive data structure (e.g. a list of a list of chars or bytes where each list is a unicode character) or implementing some expensive logic for counting characters that is O(n) instead O(1). Most languages ranging from extremely strictly typed to weakly typed don't do that for cost reasons. The tradeoff is simply not worth the price it takes.
This stackoverflow post provides a few suggestions for how you could count 'correctly'. https://stackoverflow.com/questions/15947992/java-unicode-st... that illustrates the point nicely.
If you want to do Unicode string manipulation and length counting, then use specific functions for that - but the base internal .length function should just output bytes.
That is not how UTF-32 works.
>but the base internal .length function should just output bytes.
Do you think the length of an `int64_t[3]` array should be 3 or 24?
There should be functions to do both: sizeof(int64_t[3]) * sizeof(int64_t) for example to get bytes.
In this example, the base function should do bytes, and there should be a unicode function to count it in other ways.
I could be sizing to fit in a database, or send over the wire, or I might want visible space on the screen, or I might want to know how to move the cursor.
Each of those types of length should be supported.
That's not what that does, the answer to the code you wrote is 192 because you're multiplying the same size factor twice.
The first one has size 3, because there are 3 elements in the array. The second one has size 8 because an int64 is 8 bytes.
You don't have to like it
I should write instead: sizeof(int64_t[3]) / sizeof(int64_t) for length of array.
This does help my argument that Javascript length on a Unicode string should return bytes though :)
You can't just "iterate over a string" because that's not a thing. You can get an iterator over the bytes in the string, with "foo".bytes() or you can get an iterator over the Unicode scalar values in the string with "foo".chars(), or you can iterate over a UTF-16 encoding of the string with "foo".encode_utf16()
You can index into Rust's strings, but you need to specify slice indexes, you can't just treat this like it's a array because that's not what it is. If you wanted a slice of bytes you can have one cheaply, it's as_bytes() which is a [u8] and you can index directly into that slice as with any array of bytes, but you can't mutate that and those aren't characters, they're just bytes.
In the words of the article: “The choice of UTF-32 (or Python 3-style code point sequences) arises from wanting the wrong thing.”
I’ll agree with you that strings-are-sequences-of-bytes is bad. That’s painful compiler-flag, codepage, &c. territory.
But what’s not bad is strings-are-sequences-of-code-units. That’s what Rust has, for example. Rust strings aren’t sequences of bytes, but of UTF-8 code units, and the two are semantically very different.
if the language default was anything other than this, THAT WOULD BE WRONG and unexpected. I would prefer the default to be the dumb, fast thing. then if I want the slow, fancy thing, I can import some first or third party package.
(and it's not expected that a character's length is>1 unless you've been conditioned to excpect it)
And bytes is the only thing that matter for a specific type of string, conveniently named, sequence of bytes
If you want something else than utf-8 you can use another data type, like a vector of bytes.
Rust gives you the freedom to specify what you mean.
I don't remember, but I think the size hint is set on the Chars iterator, so it can see it has 17 bytes of data, it knows that can't encode more than 17 Unicode scalar values, nor can it encode fewer than five. But since we ask for an exact count that hint is unused, the actual decoding will take place.
Rust doesn't take sides here. It exposes all the different ways you might want to calculate the "length" of a string, and lets you pick which one you mean. The non-zero-cost choices involve a multi-step specification (like `.chars().count()`), which states explicitly the calculation involved.
In contrast chars().count() creates an iterator and runs the iterator to completion counting steps, that's O(N) for a string of length N, and is in practice very expensive, you should definitely cache this value if you will need it repeatedly. It is possible the compiler can see what you're doing and cache it, but I am very far from certain so you should do so explicitly.
This is important in contrast to say, C, where strlen(str) is O(N) because it doesn't have fat pointers and so it has no idea how long the string is in any sense.
That feels a bit "pit of despair" design[2], the default thing is unhelpful and doing more than that requires the programmer to climb up out of it.
[1] https://go.dev/blog/strings
[2] https://blog.codinghorror.com/falling-into-the-pit-of-succes...
I've been quite happy that popular emojis were introduced in supplementary planes, because my language has quite a few common words (eg. 𨋢 [lift/escalator]) that ended up on plane 2.
Proper software support for those characters used to be terrible, but things got much better after emojis became popular. So, thanks and sorry everyone :)
I think that the industry could have instead adopted the old web-forum convention of colon-word-encoding, originating from ASCII art. Example: ":facepalm:". When the sequence is not supported as an emoji, it degrades gracefully into text that can be understood by anyone reading it instead of into a sequence of empty squares or diamonds with question marks in them. Text also provides a more efficient input method than having to browse for an icon in a list.
I think the first big mistake was using postfix/infix operators (combining characters, modifiers, variant selectors, joiners, etc.) rather than prefix, preferably in blocks by arity. That would have simplified processing (in particular a keyboard dead key could have been identical to a combining character) and made broken sequences detectable.
The latest big mistake, I think, was retroactively changing some non-emoji characters to have “emoji presentation”, which means that some text has to be edited to preserve its original appearance.
And they're still complaining about the handful of cases that were missed: https://news.ycombinator.com/item?id=29022906
----
Another mistake IMHO was that they accepted too many "dictionary characters", i.e. the ones only seen once or twice in some obscure dictionary -- they often had explanations like "an obscure form of [common character]".
[..." "].length === 5 // in JSLength = 5
Size: depends on the encoding
Width = 1
Also please make sure to read the first heading after the title, which summarizes the whole point of this essay.
( '\u200b'.repeat(7) )
Edit: HN does not, in fact, handle it.
Awww dang they updated the filter, booo
But if I was selling you a drop-in comment widget and boasted "it handles all of Unicode", but really I was just running s/[^ -~]+/ /g, wouldn't you feel a bit let down?
Zero width joiner emoji sequences https://emojipedia.org/emoji-zwj-sequence/
Edit: there are always exceptions - https://news.ycombinator.com/item?id=34460417