Hacker News Reader

Comment by grayhatter | Hacker News Reader

> Python 3’s approach is unambiguously the worst one, though.

Did I miss the part where he explains this take? It's made up of 5 valid unicode code units. For a language where you're not supposed to need to know the byte size semantics, the correct length should be 5. What am I missing?

The close second being 17, because length in bytes. Is another fine way to represent this data, e.g. what a successful write of some sort would look like. Network or file.

I guess I'm basing this all on the idea that it's almost always a mistake to confuse how a program manages some data, vs how a drawing lib might. Your language shouldn't concern it self with how many glyphs it needs to draw... until you actually try to draw them.

View on HN

It is wrong that "{emoji}".length == 7 -- but it's wrong because there's no such thing as the 'length' of a string out of context.

A string should be viewed as an opaque data type with views into it depending on what you're trying to do. You can have its length in the context of storage/retrieval/transmission (UTF-8 byte count), its length in the context of parsing (code points), its length in the context of editing (grapheme clusters) or length in the context of display (a bounding box in points when used in conjunction with a specific font and paragraph style attributes).

Claiming to provide an out-of-context length is strictly wrong because there's no such thing. This is where people get confused.

The attribute shouldn't be 'length' it should be something like 'countOfCodePoints' or exposed via a `CodePoints` type view.

It's particularly bad because so often (esp. for western programmers) 'countOfCodePoints' == 'countOfBytesInUTF8' == 'countOfGraphemeClusters' == """length""" so it's hella easy to accidentally write buggy software. Especially for people who don't know the above about unicode, which let's face it, most people don't. Not until they have to explain to their designer why they can't limit a label to '10 characters.' ("What do you mean there's no such thing as a character, and what am I trying to do?").

This is basically the tl;dr of the article but it's also my personal opinion.

All of this isn't about 'wrong' so much as 'imprecise and overloaded terminology making it easy to write buggy software through poor abstractions.'

If python explained which length you were getting, then this article wouldn't exist.

I agree, "length" is an ambiguous function name. It should probably not exist and instead you have functions with units in the name: .sizeBytes, .widthCharacters, .widthResAdjPixels, and so on. Back when the world was ASCII you could get away with just .length because the numbers would always be the same, but with Unicode and all of the other complications of the modern world it isn't sufficient.

Seeing a .length on something makes me think it’s an iterable without looking more deeply.

In JS, for(i in emoji) will iterate twice, but for(i of emoji) will iterate once.

;)

Yeah; I've recently noticed that almost every time I use string.length in javascript, its wrong and going to break something as soon as emoji appears. In my code, I always want to deal with either the number of codepoints or the number of UTF8 bytes. String.length gives you neither, but unfortunately it looks correct until you test with non-ASCII strings.

At least the “of” loop returns what is selectable via a cursor in the browser. I think…

Yeah; its really confusing but javascript - for legacy reasons - treats strings as "arrays of UCS2 items". But javascript also implements iterator on strings which iterate through strings in unicode codepoints. Thats why "of" loops work differently from "in" loops. (for-of in javascript uses Symbol.iterator). That also means you can pull a string apart into an array of unicode codepoints using [...somestring].

length is not ambiguous at all. Its the number of elements in the array. A string in python3 is an array of unicode code points, so the length of a string is the number of unicode code points. If you want the number of bytes, you need to encode the string in a unicode format (utf8, utf16 or utf32) to get a bytes object, which is an array of bytes. Then you can get the length of that.

Remember, one of the big accomplishments (breaking changes) of python 3 is that all strings are Unicode, not byte arrays. If you want to view a dtring as bytes, you need to convert the string to bytes. But note the number of bytes depends on the ancoding u use (utf8, …).

Exactly this. People conflate unicode with encoding quite a bit. I think it was plan9 and early Go that used "runes" as a unit, where one or more runes formed a character and an array of runes could be encoded into bytes using a given encoding.

The in memory size of a rune was just an implementation detail, and while it could be important for the programmer that the size of a rune was 2 bytes, this didn't mean the length of an array of 2 runes was 4.

I always liked the rune unit, and while my memory is hazy I think it was just code points.

I think part of the issue is programmers and apis mixing bit units for in memory representation of a conceptual value mapping (unicode), conceptual characters, stored size when encoded and so on ... without firming up those abstractions with interfaces. It gets lossy.

FWIW, CPython uses one of several Unicode string implementation representations, depending on the code points involved:

  >>> import sys
  >>> s = 'A' * 1000
  >>> len(s)
  1000
  >>> sys.getsizeof(s)
  1049
  >>> s = '\N{SNOWMAN WITHOUT SNOW}' * 1000
  >>> len(s)
  1000
  >>> sys.getsizeof(s)
  2074
  >>> s = '\N{MUSICAL SYMBOL G CLEF}' * 1000
  >>> len(s)
  1000
  >>> sys.getsizeof(s)
  4076

See https://peps.python.org/pep-0393/ . Mentioned in the linked-to article with "CPython since 3.3 makes the same idea three-level with code point semantics".

>length is not ambiguous at all. Its the number of elements in the array

That's because you defined it first as "the number of elements in the array".

It is ambiguous however because that's not how people understand it when it comes to strings, and there are several counter-intuitive ways they expect it to behave.

Not to mention there might not be any "array". A string (whatever the encoding / representation) is a chunk of memory, not an array. That you can often use a method to traverse it doesn't mean it's in an array.

The python doc says "str" are immutable sequences of unicode code points. Since it implements __getitem__, its fair to call it an array (it has a length, and allows indexing). I couldn't find out in the documentation whether the __getitem__ is O(1), which I consider a deficiency -- this should definitely be well documented.

It doesn't really matter how some people think "how people understand" something, the documentation matters. Any string in any language is some ordered sequence of atomic text-like objects, so python's approach isn't unreasonable or unexpected, either.

Except a sequence is not an array.

So OP’s definition still does not apply to python’s definition of a string in an unambiguous manner, which was the claim they were making.

In fact, using the OP’s “unambiguous” definition leads to the conclusion that strings shouldn’t have a length function at all since it’s not an array.

I didnt say say a sequence is an array. I said its fair to call str objects in python arrays: https://en.m.wikipedia.org/wiki/Array_(data_type)

>Since it implements __getitem__, its fair to call it an array (it has a length, and allows indexing)

Well, weren't we talking about things being "ambiguous"?

In Python we call what you describe a list. An array is something different. And people would expect something like the C (or the Java) data structure. In Python that would match the "array" lib package.

And that's just discussing the meaning of array - before we even get to whether a string is an array, and what this means.

>It doesn't really matter how some people think "how people understand" something, the documentation matters

In what universe? In practical use, clarity and non-ambiguous, least surprise names and semantics matter.

"But we clarify it in page 2000 of the documentation" is not an excuse. Nor is invoking moral or professional failings of those not reading the documentation. A good library design doesn't offload clearing ambiguity to the documentation.

>Any string in any language is some ordered sequence of atomic text-like objects

You'd be surprised. Especially since this isn't 1985 where strings were a bunch of 8-bit ascii characters, or even 1995, when widechar 16-bit arrays were "good enough" for Windows and Java, but we have not just non-ascii strings, but even variable length (e.g. utf-8) internal strings in mainstream languages.

Tell me a language where a string isnt an ordered sequence of elements of some atomic text-like data type. Those may have different types - like utf8 bytes, bytes, unicode code points, grapheme clusters etc. But these are all some sort of representation of text at some level. Which one a programming language uses depends on the language, and should be checked in the documentation. Its not like some obscure “check page 2000” of the doc type small print, implying that you need to read 18 tomes of language doc before u can work with the language —- no, but if u want to work with strings in any programming language, u should know what type the elements consist of.

Btw, python my try to overload the meaning of the words array and list, but the word “array” has a generic meaning in this branch of math called computer science (an ordered sequence of elements indexible in O(1)), which is how I used it here.

Interestingly, the number of Unicode codepoints is probably the only measure of a string that is unlikely to ever be relevant to anyone in practice except when it happens to coincide with a different measure.

It can't be used to determine length in bytes (important for storage or network transmission), it can't be used to determine number of displayed characters, it can't be used to safely split a string at some position.

The only reason it has caught on is that it is easy to encode into UTF-8 and UTF-16, and that anything more interesting generally requires a language context and even a font.

I hope that future languages will get rid of this single string abstraction, and instead offer two completely separate types:

- symbol strings, which would only be usable for programming purposes and should probably be limited to ASCII

- text strings, which would be intended for human display purposes, with full Unicode support, and have APIs which answer things like "in the specified Culture, what is the length of human-recognizable characters of this string" or "what is the seventh human-recognizable characters in this string in the specified culture"

There's no reason to pay the conceptual cost of Unicode for representing field names or enums (and yes, I don't believe supporting Unicode identifiers is a good idea for a programming language; and note that I am not a native English speaker, and while I do use an alphabet, ASCII is missing some of the letters&symbols I use in my native Romanian). And there's no reason to settle for the misleading safety of Unicode code points when trying to process human displayable text.

The length of an array should correspond to the number of elements. Since each element is a code point, it's the most relevant number if you intend to operate on individual elements. That is, the maximum index corresponds to the length of the array.

If you care about the number of bytes, or to operate on individual bytes, then convert to utf-8,16 or 32, and operate on the bytes object. If you wish to operate on grapheme clusters, then you could probably find some 3rd party Python library that allows you to represent and operate on strings in terms of grapheme clusters.

A string is not an array, it is a chunk of text, for the vast majority of uses of strings. Exactly how that chunk of text is represented in memory and what API it should expose is the discussion we're having. My point is that it shouldn't be exposed as an array of codepoints, since array operations (lengths, indexing, taking a range) are not a very useful way of manipulating text; and even if we did expose them as an array, Unicode code points are definitely not a useful data structure for almost any purpose.

There are basically only two things that can be done with a Unicode codepoint: encode it in bytes for storage, or transform it to a glyph in a particular font or culture.

You can't even compare two sequences of Unicode codepoints for equality in many cases, since there are different ways to represent the same text with Unicode. For example the strings "thá" and "thá" are different in terms of codepoints, but most people would expect to find the second when typing in the first. Even worse, there are codepoints which are supposed to represent different characters, depending on the font being used / the locale of the display (the same Unicode codepoints are used to represent related Chinese, Japanese, or Korean characters, even when these characters are not identical between the three cultures).

Splitting into ASCII-only and Unicode would be more of a regression than a progression. And yes, the “I’m not a native speaker” is a typical pre-emptive reply, as if it matters (neither am I—doesn’t mean anything by itself).

Let me give an example of why I don't think a single unified string API works. When doing (stringA == stringB) , what do you expect to get as a result? Do you expect it to tell if you the two strings represent the exact same codepoints, or do you expect it to tell you whether they represent the same Unicode grapheme clusters, as Unicode recommends?

The answer is of course both, depending on context. You certainly don't want a fuzzy match when, say, decoding a protobuf, but you also don't want a codepoint match when looking up user input.

What most modern languages have settled on is having a Unicode codepoint array type, typically called string or text, and an array of bytes type. However, common string operations are often only provided for the text type, and not the bytes type - which becomes very annoying when doing low level work and using bytes for text, and hoping for simple text operations.

It's ambiguous because it's not clear what elements go into separate cells of the array.

Why is it ambiguous? The Python documentation is pretty clear about what type of elements a string contains:

> Strings are immutable sequences of Unicode code points

(from https://docs.python.org/3/library/stdtypes.html#text-sequenc...)

According to python. Every language can have a different definition. C#, for example, defines the blocks to be Char objects, which is based on UTF 16:

> The Length property of a string represents the number of Char objects it contains, not the number of Unicode characters.

https://learn.microsoft.com/en-us/dotnet/csharp/programming-...

Though C# also recommends the Rune APIs for more modern/better code point handling. The Rune APIs have a bit more in common with Python 3's unicode treatment than the classic (and sometimes wrong) UTF-16 approach.

https://learn.microsoft.com/en-us/dotnet/api/system.text.run...

Did you not read the article?

> A string should be viewed as an opaque data type with views into it depending on what you're trying to do.

> The attribute shouldn't be 'length' it should be something like 'countOfCodePoints' or exposed via a `CodePoints` type view.

That’s a great way to explain it and something I’ll keep with me. Thanks

What do you mean there is no such thing as a character when grapheme cluster is exactly that? This is also the out-of-context , and people get confused because instead of this human context attribute they've been forced to use all the other alternatives that require more knowledge

Characters in context are printable or non-printable/formatting marks right? I agree they probably meant grapheme clusters, but grapheme clusters can vary dramatically in width so the point of the conversation was to explain why a bounding box was a better approximation of their goals.

They do very in width, but with a proportional font that’s true even with ASCII text. What grapheme clusters tells you is how many times you have to press the arrow key/backspace to get to the beginning of the string.

Only if the text editor made some bad assumptions. You're forgetting about non-printable characters, such as the LTR mark. These are not part of grapheme clusters (or are their own grapheme cluster), but the cursor shouldn't probably stop at them.

You know it's been a long time since this conversation but I think, reflecting, it has to do with grapheme clusters not being particularly consistent across operating systems and over time. The article even has an example where one Unicode spec encodes the same 5 USVs and either 1 or 2 graphemes.

> grapheme clusters can vary dramatically

so do 'www' and 'iii' (though less dramatically), that't not a foreign concept to designers, not sure they'd want to bound 'www' to the width of 'iii'

Counting graphene clusters is a hard problem because it depends on the font that is being used. It only exists at render time in the context of a specific client.

If the user can freely change a font it is impossible to send a string of 3 graphene clusters because you won't know if it actually will show up as 3 to client or a different number.

I wanted to brainfart about that length in the typical assumed usage should be 1 ignoring the inner encoding of Unicode of emoji ... But your comment was spot on and showed me my own assumption would fall into exactly this view scheme.

Thank you. Have my upvote.

> for western programmers

And by western you mean american, right? You can't even use ASCII in the UK --- '£'.

Yes I was using western as synonymous with ASCII users, and it’s not. My bad.

£ is still just a byte

And yet it is reasonably common to see "Â£" when the UTF-8 is misinterpreted.

When using latin-1/latin-15/iso-8859-1/iso-8859-15/cp1252 that statement is true. With utf-8 it is two bytes (c2 a3), if a software uses utf-16, ucs-2, etc. it may be more.

Not in any modern encoding and certainly not in ASCII either. Having the highest order bit set makes that kind of problematic.

Nope, says UTF-8.

'u32_pound & 0xff == u32_pound' happens to be true, ye. It doesn't make it a byte. You need the leading 0s.

My mistake

For the UK, you also need to represent the Celtic languages. You'll need at least these letters: â, ê, î, ô, û, ŵ, ŷ, à, è, ì, ò, ù, ẁ (maybe ỳ?), á, é, í, ó, ú, ẃ (maybe ý?), ï (maybe more ¨), ...

Example: https://www.llyw.cymru/ (Welsh government homepage).

Bits of https://www.highland.gov.uk/press (Scottish Highlands council).

(I'll admit most people who don't speak these languages won't bother with the diacritics, unless it's as easy as typing English.)

> Not until they have to explain to their designer why they can't limit a label to '10 characters.'

Or in a single font. It's impossible to render any mixed combination of simplified Chinese, traditional Chinese and Japanese with a single font (Korean might be also involved, but not sure about that). Even in Unicode, characters might share the the same space which don't have anything common in their looks, nor in their meaning. That applies to the shared CJK space as well. Btw. Japanese has halfwidth and fullwidth characters.

> It's impossible to render any mixed combination of simplified Chinese, traditional Chinese and Japanese with a single font (Korean might be also involved, but not sure about that)

Well, that could be phrased better. Many such mixed combinations would encounter no problems. There is "Han Unification" in Unicode, in which certain graphical forms are declared equivalent and the intent is that they display as Japanese characters if you print them in a Japanese font, but as Chinese characters if you print them in a Chinese font. 直 is a good example of how that looks; try viewing it in different fonts.

But nobody likes unification and explicit fixed forms are constantly being defined so that it's possible to talk about them. Imagine if I wanted to write "in Old English, the word for dog was hund"... except that your font automatically replaced the sequence hund with a special ligature that looks exactly like dog.

So we have separate unicode points for ⻘ (modern, CJK RADICAL BLUE) and ⾭ (old, KANGXI RADICAL BLUE), and for ⿓ (traditional Chinese, KANGXI RADICAL DRAGON), ⻰ (simplified Chinese, CJK RADICAL C-SIMPLIFIED DRAGON), and ⻯ (Japanese, CJK RADICAL J-SIMPLIFIED DRAGON). Interestingly, the dragon characters are all considered different according to the original "Han Unified" specification, where they are CJK UNIFIED IDEOGRAPH 9F8D, CJK UNIFIED IDEOGRAPH 9F99, and CJK UNIFIED IDEOGRAPH 7ADC. In contrast, there is only the one "unified" form of 直, CJK UNIFIED IDEOGRAPH 76F4, but you can refer to its Chinese form explicitly with CJK COMPATIBILITY IDEOGRAPH FAA8 and to its Japanese form with CJK COMPATIBILITY IDEOGRAPH 2F940. (My browser font fails to render either of those.)

It was never possible to rely entirely on the font to handle dealing with simplified vs traditional characters for you, for the obvious reason that their mapping is not one-to-one. In simplified Chinese, 后 means "after"† or "behind" and it also means "empress". In traditional Chinese, "after" and "behind" would be 後. And "empress" would be... 后. This means there can be no way for a traditional Chinese font to determine what it should display if you write 后.

Ultratraditional Korean hanja participate in the same variation of forms that we see between Chinese and Japanese. But it isn't normal to write Korean in hanja outside of very specific contexts. Hangul are radically different and belong to a separate part of unicode entirely.

† "After" in time. "After" in sequence is 下, "below".

There are three notions of length that make sense:

1. UTF-8 byte length

2. Code point count

3. Extended grapheme cluster count

#3 makes sense for users but it doesn’t make sense for programs which often need to work at the code point level.

I expect programming language string length to obey the law:

  len(a ++ b) = len(a) + len(b)

For example, if I concatenate a two strings, one containing an “e” and one containing a combining acute accent, then I expect the length to be longer than a string containing a precomposed ‘é’ character. It’s in fact useful if strings that look the same but have different code points have different lengths, because it tells you that they’re not the same (and maybe you forgot to normalize something etc).

Code point length is the most useful for people who are actually writing string algorithms based upon Unicode.

UTF-8 length is useful for people who are treating strings as opaque byte sequences, but in that case they should be using a bytes/buffer object and not a string object, except in very low-level languages that don’t want to pay an encoding/decoding cost.

Extended grapheme cluster count is useful for people who are constructing certain kinds of user interfaces, where the number of characters is limited for a policy rather than memory or width reason.

i.e. when length limits are imposed by human policy, grapheme cluster count is the way to go. Length limits for memory reasons should rather be in UTF-8 bytes. If you need a limit for visual width reasons then you need to go measure the string in pixels, otherwise I’m going to put a U+FDFD in there and ruin your day.

UTF-16 length can GTFO.

> Code point length is the most useful for people who are actually writing string algorithms based upon Unicode.

What algorithms would you be writing against code points?

I suspect primarily substring, where if you index by bytes you'll mangle the string, but if you index by codepoints everything works out.

You'll mangle the string if you index and search by code points too, when the string contains the emoji this article is about, or for that matter an "e" following by combining acute accent.

The string will be fine if you only move to, split and concatenate at indices outside those grapheme clusters. But that is also true when indexing by bytes or UTF-16 code units.

So in some senses, indexing by bytes is just as good as indexing by code points, but faster. Either way to avoid mangling strings you need to restrict the indices of whatever type to meaningful character boundaries.

If you have decided to avoid string indices inside grapheme clusters, there comes the awkard question of what should you do when editing text in an environment rendered with font ligatures like "->" rendered as → (rightward arrow). From one perspective, that's just a font. From another, the user sees a single character yet there are valid positions (such as from cursor movement and character search) that land mid-way through the character, and editing at those positions changes the character. Neither is clearly best for all situations.

Isn't that the level emoji sequences work at?

Everything that you can do to a Unicode string, except concatenation, is defined in terms of code points. Normalization, case transformations, collation, regexes, layout and rendering and encoding.

For example, let’s say you want to define a “natural sort” order that sorts e.g. “A2” < “A10”. To do that you divide the string at boundaries between code points in ranges of each numeral type that you are supporting (e.g. western numerals, Arabic numerals, Chinese numerals).

Wouldn't layout/rendering etc. be done in terms of grapheme clusters?

Codepoints is best for collaborative text editing / CRDTs (diamond types, automerge, etc). We generally model the document as a big list of unicode codepoints.

We could use grapheme clusters, but the grapheme cluster boundary points change as unicode evolves, and not all systems update at the same time. Separating strings based on grapheme cluster boundaries also requires a big lookup table to be embedded in every app. Unicode codepoints are obvious, stable, and easy to work with. And they're encoding-agnostic, so there's no weird UCS2-to-UTF8-bytes conversion needed in javascript, C#, etc.

Using code points (or scalar values, I hope) just means that it’s inefficient for everyone, because now everyone has to convert indexes (well, except Python, but it has other problems), instead of only half the people.

Going UTF-8 is fairly clearly superior: it will be the wire format, even if it’s not the language’s string format, so now environments that use UTF-8 strings never need any conversions (apart from decoding escape sequences, most likely).

Much as I hate UTF-16, I would even be inclined to argue that UTF-16 was a better choice than code points, as it will reduce the amount of extra work UTF-16 environments have to do, without changing how much UTF-8 environments have to do at all; but it also has the disadvantage that validation is vanishingly rare in UTF-16, so you’re sure to end up with lone surrogate trouble at some point, whereas UTF-8 tooling has a much stronger culture of validation, so you’re much less likely to encounter it directly and can much more comfortably just declare “valid Unicode only”.

Yes, code points is a purer concept to use. I don’t care: it’s less efficient than choosing UTF-8, which adds negative-to-negligible complexity. Please, just abandon code point indexing and embrace the UTF-8.

No, I don’t agree.

The problem with utf8 byte offsets is that it creates a data validation problem. In diamond types I’m using document positions / offsets in my wire format. With utf8 byte offsets, you can receive changes from remote peers which name invalid insertion positions. (Ie an insert inside a character, or deleting half of a codepoint). Validating remote changes received like this is a nightmare, because you need to reconstruct the whole document state to be able to tell if the edit is valid. Using Unicode codepoints makes invalid state unrepresentable. So the validation problem goes away. (You might still need to check that an insert isn’t past the end of the document, but that’s a much easier check).

Almost all application programming languages use utf16 anyway, (javascript, c#, swift, Java) so you still need to convert positions anyway. Even in rust it’s common to see line/col positions from text editors.

Using utf8 byte offsets just doesn’t really give you any benefits in exchange for making validation much harder.

The data validation concern seems fair enough.

> Almost all application programming languages use utf16 anyway, (javascript, c#, swift, Java)

Swift 5 switched to UTF-8: https://www.swift.org/blog/utf8-string/. I’m hopeful that other UTF-16 environments might eventually manage to switch to UTF-8 internally despite retaining some UTF-16 code unit semantics for compatibility; two projects have already demonstrated you can very practically do this sort of thing: Servo from fairly early on (WTF-8 despite the web’s UTF-16 code unit semantics), and PyPy since 7.1 (UTF-8 despite code point semantics, not sure what they do about surrogate code points). I know the web has largely backed away from UTF-16 and uses code point semantics (well, scalar values plus loose surrogates) on almost all new stuff, with good UTF-8 support too.

> Using Unicode codepoints makes invalid state unrepresentable.

Is that true? Maybe it's not "invalid", but you might very well slice through the middle of a grapheme cluster.

I contemplated querying that myself, but decided that for CRDT editing purposes, it’s probably never practical to think about grapheme clusters. Given text in a language with syllabic grapheme clusters (e.g. Indic languages), if you start with “ba” (one syllable/grapheme cluster) and edit it to “boca” (two syllables/grapheme clusters), you could say “replaced ‘ba’ with ‘boca’”, but I’d be surprised if any CRDT did it that way if it could instead handle it as “inserted ‘oc’”, even though “oc” mightn’t make sense by itself, linguistically. But Unicode doesn’t do much in the way of defining what makes sense or not, and I don’t think there’s any coherent “bad grapheme clustering” detection algorithm. (Aside: on reflection, my Indic languages example is messier still, since -a probably represents the inherent vowel, so in “ba” → “boca” those “a”s are probably actually represented by the absence of a vowel sign code point—and if you wanted to suppress the inherent vowel, you’d need to add a virama sign. Fun stuff.)

But then again, I know that some CRDTs struggle with interleaving, and maybe grapheme-awareness could help things out in some way or other. I dunno.

Yeah I agree. I think its inevitable that collaboratively edited documents sometimes end up with grapheme clusters that are considered invalid by some peers, simply because different peers might be using different versions of unicode. If my phone supports the polar bear emoji and yours doesn't, you'll see weird stuff instead of a polar bear. There's no getting around that.

And yes, using unicode codepoints, buggy clients might insert extra unicode characters in the middle of a grapheme cluster. But ... Eh. Fine. I'm not super bothered by that from a data validation perspective.

Why don't I have the same attitude toward invalid UTF8? I mean, the CRDT could syncronize arbitrary arrays of bytes that by agreement contain valid UTF8, and treat it as user error in the same way if that happens? Two reasons. First, because some languages (eg rust) strictly enforce that all strings must contain valid UTF8. So you can't even make a document into a String if it has invalid UTF8. We'd need a try_ codepath, which makes the API worse. Secondly, languages like javascript which store strings using UTF16 don't have an equivalent encoding for invalid UTF8 bytes at all. Javascript would have to store the document internally in a byte array or something, and decode it to a string at the frontend. And thats complex and inefficient. That all sounds much worse to me than just treating the document as a sequence of arbitrary unicode codepoints - doing which guarantees correctness and we don't need any of that mess.

> grapheme clusters that are considered invalid by some peers

I’m not sure if there’s any spec that defines any sequence of Unicode scalar values as “invalid” (though there’s certainly a lot that’s obviously wrong, like some forms of script-mixing). Grapheme cluster segmentation doesn’t concern itself with meaningfulness, but just doing something with what it has; so if you inject something into the middle of what it decided was a cluster, it’ll just split it differently.

> treating the document as a sequence of arbitrary unicode codepoints

Though I hope you validate that they are Unicode scalar values and not code points. Surrogates are terribad.

> Though I hope you validate that they are Unicode scalar values and not code points. Surrogates are terribad.

Yes, my mistake. I do mean scalar values. I am constantly confused about the terminology for unicode. (Unicode code point? Scalar values? (Character?) Surrogate pair? Is there a term for half of a surrogate pair?)

UCS-2 length is also something that is still required today.

UCS-2 is dead. You can't express Unicode in UCS-2. If you have old UCS-2 data you can just treat it as UTF-16, maybe check for encoding irregularities but if it was really UCS-2 correct Unicode it'll be fine.

UTF-16 length is only useful if you are moving UTF-16, perhaps for interop with other software that chose UTF-16. Remember to pass on your condolences and look forward to a day when we don't do that any more.

UCS-2 support is shipped with every modern x86 firmware. UEFI system partitions use it for VFAT name encoding.

> UTF-16 length is only useful if you are moving UTF-16 [...] Remember to pass on your condolences and look forward to a day when we don't do that any more.

Java/the JVM says hi!

Arguably, "how many bytes does this string occupy in memory/on disk (e.g. in class files)" is a pretty useful thing to be able to ask.

> Arguably, "how many bytes does this string occupy in memory/on disk (e.g. in class files)" is a pretty useful thing to be able to ask.

Sure. On disk those strings are (modified) UTF-8 of course, is that what you meant ?

On disk yes, at runtime it's still UTF-16, I believe.

This is still relevant for many databases as well, as far as I remember.

> It is wrong that "{emoji}".length == 7

I get what you're intending, but it's kinda funny because that string actually is 7

The underlying size is 7, the length isn't 7.

Also, the length of the literal string “{emoji}” is 7.

Fantastic comment, thank you.

One question I have is which of these are fixed across Unicode standards.

(a) Byte count for storage and (b) code point count are definitely fixed, Unicode provides an explicit guarantee of the latter: https://www.unicode.org/policies/stability_policy.html

But what about (d) grapheme count? If I make a microblogging site which limits post length to 144 graphemes, can my database invariants break when I upgrade my version of Unicode?

This is exactly what Raku does. Neither strings nor arrays have a `.length` method because it's arbitrary.

    [0] > "\c[FACE PALM]".chars
    1
    [1] > "\c[FACE PALM]".codes
    1
    [2] > "\c[FACE PALM]".encode.bytes
    4
    [3] > "\c[FACE PALM]".encode
    utf8:0x<F0 9F A4 A6>
    [4] > "\c[FACE PALM]".encode('utf-16')
    utf16:0x<D83E DD26>

FWIW, to get the "length" of an array, the method is `.elems`.

>It's particularly bad because so often (esp. for western programmers) 'countOfCodePoints' == 'countOfBytesInUTF8' == 'countOfGraphemeClusters' == """length""" so it's hella easy to accidentally write buggy software.

Then programmers will pick a random view and assume its length equals the number of characters and bytes. Also the grapheme view will introduce an OS-dependent bug.

5 makes perfect sense to me; the author's complaints seem kinda silly.

An area this makes sense is, what do you expect to get if you do something like:

    emoji = " "
    print(emoji[:3])

Should this throw an error because there's only one displayed "character"? Should it return only a partial codepoint by returning only the byte data for the first 3 bytes?

Modern strings are complex objects that have evolved a bit past char[] or byte[].

> Should this throw an error because there's only one displayed "character"?

Why should it not? You’re literally breaking the content.

Though in reality, indexing strings is a broken operation. That you’re using it at all is the core issue.

> Modern strings are complex objects that have evolved a bit past char[] or byte[].

And yet that’s exactly what you’re advocating, just with 21 bit chars.

> Why should it not? You’re literally breaking the content.

Strings are just an array of unicode codepoints rather than "characters", so all I'm doing is asking for the first three of those codepoints.

> Though in reality, indexing strings is a broken operation. That you’re using it at all is the core issue.

Substring is a broken operation? What's the justification for that idea?

> Strings are just an array of unicode codepoints rather than "characters", so all I'm doing is asking for the first three of those codepoints.

"Ice trays are just a pile of molecules rather than "cubes", so all I'm doing is separating those molecules", he states as he activates the igniter.

> Substring is a broken operation? What's the justification for that idea?

You take a thing and you mangle beyond recognition without regards for its purpose or meaning. That's like considering the jaws of life a normal part of opening a door to take a piss at work.

One of the first things the author of the article does is breaks it down into the 5 code points and explains their individual meanings.

Your point being, what exactly?

If the user gives you what, as far as they're considered, is a glyph. And you return a completely different glyph. You've mangled their data.

>Should this throw an error because there's only one displayed "character"?

Absolutely.

I think this is where the misunderstanding comes in. Python doesn't treat strings as char[] but as essentially unicode_codepoint[].

Whether this is a good idea on the whole is debatable, there's even a full PEP talking about the security concerns around doing it this way[1].

However, given this is how it works, the behaviour displayed makes complete sense to me and is the best of the bad choices presented by needing multi-byte strings.

[1]: https://peps.python.org/pep-0672/

Well, an index into a string is not necessarily another string, nor a character.

> For a language where you're not supposed to need to know the byte size semantics, the correct length should be 5. What am I missing?

In the words of the article: “The choice of UTF-32 (or Python 3-style code point sequences) arises from wanting the wrong thing.”

“Not needing to know the byte size semantics” seems reasonable, but it simply isn’t a useful goal. The things it makes easier or faster (knowing how many code points there are, and O(1) indexing by code point) are things you shouldn’t be doing—and when you have to interact with the rest of the world, you now have a more expensive encoding step that is always needed, rather than just sometimes if you’d chosen UTF-8 or even UTF-16.

... but maybe it simplifies and speed-up the internal processing? I haven't looked at Python 3 C implementation of strings, but that is a guess. Also, IIRC, Python 3 has the ability to keep different internal representations of strings and uses the most compact one. If all character are 7-bit ASCII, it uses bytes representation. That's what I remember from Python dev discussions long ago.

But the overall tone of the article is still bashful. Caring about the internal representation of strings and bashing UTF-32 feels lame and angry. (Especially if I'm right about the multi-rep nature of Python 3: their choice is good for most text, and they could add a UTF-8 internal rep in the future, although that would probably break enough code that expect the UTF-32 value for len() that it is not worth it.)

No. Python internally would be made much faster by working on pure UTF-8. Absolutely nothing internal to the language uses the operations that code point semantics speeds up.

Since you mention the varying internal representation of strings: that’s PEP 393 <https://peps.python.org/pep-0393/>, which landed in CPython 3.3, and it generally made things slower by introducing a lot of branching and reallocating and such, though it does speed up some cases due to having to touch less memory, and some methods due to being able to quickly rule out possibilities (e.g. str.isascii can immediately return False for a canonical UCS-2 or UCS-4 string, since if they were ASCII they’d have been of the Latin-1 kind).

PEP 393 was done because people were complaining about how much memory their UCS-4 encoding had been using.

Note also how PEP 393 retains code point semantics: Latin-1 (Unicode values 0–255), UCS-2 or UCS-4; all fixed-width encodings of code point sequences. PEP 393 does also allow a string to cache UTF-8 representation (see PyCompactUnicodeObject.{utf8, utf8_length}), choosing “UTF-8 as the recommended way of exposing strings to C code”, but I gather this isn’t used very much.

(Related: PyPy 7.1 shifted to using UTF-8 exclusively internally, and according to https://www.pypy.org/posts/2019/03/pypy-v71-released-now-use... got a “nice speed bump” out of it.)

> I'm basing this all on the idea that it's almost always a mistake to confuse how a program manages some data, vs how a drawing lib might. Your language shouldn't concern it self with how many glyphs it needs to draw... until you actually try to draw them.

Well, why not? There are a lot of things that people would want to call string.length for — drawing little equals signs under text in a terminal, for a frivolous example — where that’s the whole reason they’re making the call. Off the top of my head I’m not really sure how you solve that with variable-width characters if there’s no way to separate out or count them.

>Well, why not?

Separation of concerns?

I’m not sure what separation of concerns is served by not providing a method to show how many glyphs are in a given string.

Who said there shouldn't be one? The point is there should be more than one, and that not all are a language/string library-level concern.

The context is "I guess I'm basing this all on the idea that it's almost always a mistake to confuse how a program manages some data, vs how a drawing lib might. Your language shouldn't concern it self with how many glyphs it needs to draw... until you actually try to draw them."

This means that shouldn't be some generic "length" method, but appropriate separate-concerns methods (plural), some of which (e.g. regarding character width in pixels when rendered) even belong to a drawing lib and not the language at all.

The parent's point is that length (bytes), characters (count), and glyphs (size, shape) are different concerns. The latter would concern a drawing lib or a renderer, but not be a core string method (which should concern itself with the abstract notion of characters and the concrete notion of bytes).

As far as I can tell, you're only missing two things:

1. It's five "Unicode scalars," that's the name for the top-level logical unit. The term "code points" technically refers to a lower-level concept, one that varies across encodings, just not as much as the number of bytes. I didn't know that, and it's the helpful thing I learned from this article. UPDATE: And it's also not true, sorry. "code units" are the lower-level concept from the article, "code points" are a more expansive category at the same level: https://www.unicode.org/versions/Unicode10.0.0/ch03.pdf#G740...

2. The author takes it as an unstated assumption that top-level logical structure is useless because any specific usage either ignores all structure or has a point at which low-level structure comes into play. (That assumption is false: Top-level structure is useful for keeping track of what you are doing and as a sort of "common currency" for translating between different low level representations. For example, see the very first table in the article.)

> The close second being 17, because length in bytes. Is another fine way to represent this data, e.g. what a successful write of some sort would look like. Network or file.

Almost. 17 is the number of bytes it occupies in memory. But you don't generally dump memory directly to disk or network. It happens to make sense (and it's convenient) for utf8 strings. But it's better to be explicit about that. Python is better. If you care about bytes, say you care about bytes:

    len(bytes(" ", "utf8")) == 17
    len(bytes(" ", "utf-16-be")) == 14
    len(bytes(" ", "utf-32-be")) == 20

I think he meant to bring out defensiveness with that quip. He never says that it's a big deal, just that it's the worst way to get the length of a string containing emoji, presumably of the mainstream languages.

Why is it the worst way though?

> Note about Python 3 added on 2019-09-09: Originally this article claimed that Python 3 guaranteed UTF-32 validity. This was in error. Python 3 guarantees that the units of the string stay within the Unicode code point range but does not guarantee the absence of surrogates. It not only allows unpaired surrogates, which might be explained by wishing to be compatible with the value space of potentially-invalid UTF-16, but Python 3 allows materializing even surrogate pairs, which is a truly bizarre design. The previous conclusions stand with the added conclusion that Python 3 is even more messed up than I thought!

Perhaps the part about it needing lookups of the unicode database and being dependent on the version of the database used?

That's not true though. It just counts the number of code units, that's not version dependent. It's certainly no worse than counting the number of UTF-16 points (I'd argue it's better since it's less arbitrary - whether something is a unicode scalar is a design decision, whether something is in the BMP or not is mostly an accident of implementation).

Because it’s never actually useful.

You can’t use that information to know how much actual space it takes (in storage) as nobody sane stores UTF-32, you can’t use it to know much much logical space it takes (aka the user’s interpretation), you can’t use it to know how much visual space it takes (not that you can ever get that), and you can’t use it to segment or process the text.

A length in codepoints gives you nothing that’s really actionable, at least not that you’d need outside of a context where you could easily obtain it otherwise.

It is useful: When iterating over a string in Python (which I hope you agree _is_ useful?), you get that many parts.

> It is useful: When iterating over a string in Python (which I hope you agree _is_ useful?), you get that many parts.

That’s… not useful?

I can’t say I remember ever caring knowing how many items I would be getting during an iteration[0]. If I want to set an iteration limit I can just… do that, using `islice` or some such.

[0] in python anyway, in lower level language there can be a utility in order to pre-allocate an output collection

> When iterating over a string in Python (which I hope you agree _is_ useful?)

Not often. There’s almost nothing useful you can correctly do with a sequence of code points.

But 100% of all those complaints also apply to JS, except that UTF-16 is just much more stupid than UTF-32?

What is useful? What should it be? Don't say bytes because there is already an idiomatic way to get bytes: `len(bytes(s, enc))` which is both more correct and explicit.

Maybe it should just return None because the only useful thing is probably how much "space" it occupies on screen in a fixed-width font, but that's too difficult to know.

Reply on news.ycombinator.com