Grapheme clusters and no. Let’s see if anyone has a different interpretation or unsupported use-case.
Codepoints is easy, as are bytes. Graphemes much harder.
So, what's the length of a sole non-breaking joiner codepoint? (U+2060, yes it's allowed)
Looking here the answer looks like “0” for all contexts to me.
And well, I don’t think anyone’s arguing that a “length” function needs to be exposed by the standard library of any language (I saw that Swift does btw).
Besides from byte count it’s the only interpretation of “length” that I think makes sense for users (like I mentioned above, validating that a name has at least length 1, for example)
that's literally the core premise of the original post
This won't matter for most applications but it's something you may easily overlook when assuming grapheme clusters are enough for everything.
> Grapheme clusters and no. Let’s see if anyone has a different interpretation or unsupported use-case.
And how about this: שַלוֹם
Notice that the former is lacking the diacritics of the latter. Is that more characters?
Maybe a better example: ال vs ﻻ. Those are both Alif-Lamm, and no that is not a ligature. The former is often normalized to the latter, but could be stored separately. And does adding Hamza ء add a character? Does it depend if it's at the end of the word?
Naive heuristic: look at the font rendering on my Mac:
ء : 1 (but it sounds like it can be 0 depending on context)
ﻻ : 1
ال : 2
שלום : 4
שַלוֹם : 4
Do you see any issues?
In Japanese, which I am more familiar with, maybe it’s similar to ゙, which as you can see here has width 1 but is mostly used as a voicing sound mark as in turning か (ka) into が (ga), thus 0 (or more properly, as part of a cluster with length 1)
The use-case I’m getting at is answering the question “does this render anything visible (including white space)”? I think it’s a reasonable answer on the whole falsehood-programmers-believe-about-names validation deal (after trimming leading and trailing whitespace).
It’s the most intuitive and practically useful interpretation of “length” for arbitrary Unicode as text. The fact that it’s hard, complex, inelegant and arguably inconsistent does not change that.
Unicode has a fully spelled out and testable algorithm that covers this. There's no need to make anything up on the spot.
You don't need cultural context. It's not trivial. The algorithm is six printed pages.
No individual part of it is particularly hard, but every language has its own weird stuff to add.
I feel strongly that programmers should implement Unicode case folding with tests at least once, the way they should implement sorting and network connections and so on.
.
> I think it’s a reasonable answer on the whole falsehood-programmers-believe-about-names validation deal
All these "false things programmers believe about X" documents? Go through Unicode just one time, you know 80% of all of them.
I get that you like the framing you just made up, but there's an official one that the world's language nerds have been arguing over for 20 years, including having most of the population actually using it.
I suggest you consider that it might be pretty well developed by now.
.
> The use-case I’m getting at is answering the question “does this render anything visible (including white space)”?
This is not the appropriate approach, as white space can be rendered by non-characters, and characters can render nothing and still be characters under Unicode rules.
.
> It’s the most intuitive and practically useful interpretation of “length” for arbitrary Unicode as text.
It also fails for Mongol, math symbols, music symbols, the zero-width space, the conjoiner system, et cetera.
.
> The fact that it’s hard, complex, inelegant and arguably inconsistent
Fortunately, the real approach is easy, complex, inelegant, and wholly consistant. It's also well documented, and works reliably between programming languages that aren't Perl.
At any rate, you're off in another thread instructing me that the linguists are wrong with their centuries of focus because of things you thought up on the spot, and that Unicode is wrong because it counts spaces when counting characters, so, I think I'm going to disconnect now.
Thanks for understanding. Have a good day.
> Do you see any issues?
Regarding שַלוֹם I personally would consider it to be four letters, but the culture associated with that script (my culture) enjoys dissecting even the shape of the lines of the letters to look for meaning, I'm willing to bet that I'll find someone who will argue it to be six for the context he needs it to be. That's why I mentioned it.
> In Japanese, which I am more familiar with, maybe it’s similar to ゙,
Programmers should understand that it doesn't matter if they think the thousands of years of language rules are irrational; Unicode either handles them or it's wrong.
Unicode doesn't exist to standardize the world's languages to a representation that programmers enjoy. It exists to encode all the world's languages *as they are*.
Whether you are sympathetic to the rules of foreign languages isn't super relevant in practice.
> Unicode doesn't exist to standardize the world's languages to a
> representationthat programmers enjoy. It exists to encode all
> the world's languages *as they are*.
If you give a Japanese writer 140 characters, they can already encode double the amount of information an English writer can. Non-storage “character”-based lengths have always been a poor estimation of information encoding so worrying about some missing graphemes feels like you’re missing the bigger problem.