Hacker News Reader

I'd say all of the above objections don't fix any of the actual issues (which revolve stem from there being multiple, similar, metrics one wants regarding the size in bytes, characters, normalized characters, etc of a unicode string), while introducing extra semantic bike-shedding!

Whether the metric is called "length", or "size", or whatever, is irrelevant, and length for strings and arrays and lists is so well entrenched and understood that the objections don't make sense.

What's actually the problem is not that "string length" is not "really" a length in the way that a leg or a wall has length, but that for unicode strings it's difficult to calculate, and confusing, unless one understands several unicode implementation mechanics...

The name length itself, for example, or the measuring of said length, was never an issue with ASCII strings (the issue there was to remember to add/subtract the NUL at the end).

It doesn’t matter how well you understand Unicode, it is impossible to compute “the length of a string” because there is no single metric that means that.

It’s like measuring “the size of a box.” I want volume, you want maximum linear dimension, UPS wants length plus width plus height. The problem with “the size of a box” isn’t that it’s hard to measure, it’s that it doesn’t exist. Imagine your favorite language has a Box type with a “size” property. What does it return. How likely is it that the thing it measures is the thing you want?

Of course it was never a problem for ASCII, because ASCII is structured to make most of the measurements people care about be the same value. I want bytes, you want code points, he wants “characters,” doesn’t matter, they’re all the same number.

ASCII is also incapable of representing real world text with good fidelity. It’s inherently impossible to remedy that while maintaining a singular definition of “length.” If you value length measurement over fidelity, you can keep using ASCII.

>It doesn’t matter how well you understand Unicode, it is impossible to compute “the length of a string” because there is no single metric that means that.

That's already covered in my comment: ("[the problem] stems from there being multiple, similar, metrics one wants").

Whether we call all of them length or a specialized name is not the real problem.

The real problem is you need to know what you want of each, and some of them (e.g. regarding normalization, decomposition, and so on) can be hard to grasp.

In 99% of cases people either want to know "how many bytes", or "how many discreet character glyphs of final output" (even if they have combining diacritics etc).

For which use case is "how many character glyphs" the measure you you want to know?

It's really rare to care about the number of glyphs. That's not something you can answer for a string in isolation, anyway; it depends on the font being used to render it. The only code that would care about this would be something like a text rendering engine allocating a buffer to hold glyph info.

I suspect you mean the number of grapheme clusters, which is Unicode's attempt to define something that lines up with the intuitive notion of "a character." This is basically a unit that your cursor moves over when you press an arrow key.

However, it's pretty uncommon to want to know the number of grapheme clusters too. Lots of people think they want to know it, but I struggle to come up with a use case where it's actually appropriate. An intentionally arbitrary limit like Tweet length is the best I can think of.

"How many bytes" is ambiguous. Do you mean UTF-8, UTF-16, UTF-32, or something else?

There are a lot of different ways to answer the question, "how long is this string?"

You did mention similar metrics, but you then went on to say that the objections don't make sense and that the actual problem is that length for a unicode string is difficult to calculate.

My point is that the difficulty of calculating a length is not the problem. It's annoying, but people have written the code to do it and there's rarely any reason to write it yourself. Just call into whatever library and have it do the work. The problem is that you have to know what kind of question to ask so you can make the call that will actually give you the answer that you need. And that is not the sort of thing that can be wrapped up in a nice little API.

> I struggle to come up with a use case where it's actually appropriate.

TCP packet size is a very real thing for me.

Also in forms it helps to know, for example, that no one has entered more than 2 characters for a USA state abbreviation.

Definitely open to bigger ways about thinking of these things though.

How is calculating the number of grapheme clusters relevant to TCP packet size?

As for state abbreviations, is $7 a valid abbreviation? Is XQ? You’d probably want to validate those against a list of known good ones.

Super curious about why UPS wants l+w+h. Any detail you can link?

https://www.ups.com/us/en/help-center/packaging-and-supplies...

Apparently I misremembered slightly: it's actually width + 2 x length + 2 x height.

That link doesn't seem to explain the why, but my understanding is that it's just a decent heuristic for the general difficulty of handling packages as they go through the system. Volume wouldn't be appropriate, because a really long, skinny box is harder to handle than a cube of the same volume.

How do you decide which dimension is the width versus the length? I assume height is often significant for packages containing things that shouldn't be turned upside down, but length versus width seems pretty arbitrary. Is width just assumed to be the shorter of the two dimensions?

Length is defined as the longest side. The other two sides are interchangeable so pick what you like. This measure doesn’t appear to account for packages that require a certain orientation.

Fun fact: the iOS bug where you could brick a device by sending some strings in Arabic to the device was essentially due to assuming that there is a monotonic relationship between some measure of "string length" and "display space".

(essentially, iOS was trying to truncate strings in notifications to make them fit available space, except it did that in a very naive way, and would crash on any case where removing codepoints from a string made it display _longer_. Pretty easy to trigger with letters like ي)

Can you go into a little more detail how that works? I know almost nothing about Arabic.

The same letter can look different depending on whether it's at the beginning, in the middle of, or at the end of a word. So if you remove a letter with a fairly short final form, which is preceeded by a letter with a fairly short medial form but a fairly long final form, it's possible that the new rendering of this letter would be longer than the previous two.

This problem isn't exactly Arabic-specific, it's just super easy to trigger in Arabic. You've got kerning in the Latin script as well, and you could theoretically have a font that makes `the` look smaller than `th`.

But in Arabic, letters change form based on where they are in the word. All three things with two dots on them are the same letter in تتت, but the middle one is much smaller. I can imagine something like يم being truncated to ي and the end result being longer visually, depending on the font.

Interestingly, in many use cases all these "lengths" agree. For example an empty string is 0 in all these measures, and fuzzy metrics like "unreasonably long password" work equally well with "500 normalized characters" "500 bytes" "1000px at font size 8" etc.

The problems only arise when taking about "a string of length 20" which indeed has a strong smell

"unreasonably long password" must go into the area of "Content-Length is too long for this HTTP request to login/change password".

>The problems only arise when taking about "a string of length 20" which indeed has a strong smell

It has no smell at all. We interface with all kinds of systems with limited display space, external forms, POS displays, databases with varchar columns, and so on all the time.

Whether certain strings should have a specific length is a business decision, not a code smell...

limited display space is a case for "string length in px", which is notoriously hard to calculate and has poor library support. Just because 20 "x" fit doesn't mean 20 "w" will fit. Fixed space fonts are an exception, but they have problems with Chinese.

Databases with varchar columns exist, but varchar(20) sounds generally suspect unless it's a hash or something else that's fundamentally limited in length.

>limited display space is a case for "string length in px", which is notoriously hard to calculate and has poor library support.

It is notoriously easy when the display is a LED display, a banking terminal, a form-based monospaced POS, something that goes to a printed out receipt (like a airline check-in or a luggage tag), a product / cargo label maker, and tons of other systems billions depend upon everyday, where one visible glyph = 1 length unit, and type design doesn't come into much play...

It's only easy if the system forbids everything that would make calculating visible length hard, which I think constitutes extremely poor library support. I want to see the monospaced system that can correctly print Mongolian: ᠮᠣᠩᠭᠣᠯ ᠪᠢᠴᠢᠭ If properly implemented, it should join the characters and display them vertically. But your browser is probably showing them horizontally right now, because support for vertical writing is seriously broken: https://en.wikipedia.org/wiki/Mongolian_script#Font_issues

Most of those systems are terrible at handling non-latin text because they get all these things wrong. Of course it's "easy" to handle length in these cases, they've selected out any kind of text that makes handling length hard.

The mere assumption that "one glyph" is a meaningful well-defined concept that works across languages is the problem here.

I was at one time responsible for code to populate preprinted forms. I only had to deal with ascii but the best solution was still to just do the layout and then check that the bounding box wasn't too big.

I quite agree... mostly... but I tend to abbreviating input from untrusted sources using the String.length() anyway, because it's so simple, and a single-digit overestimate is often safe.

For example, I have a phone app that abbreviates various things at 300 whatevers because I know that on one hand 300 is more than the phone will display, but on the other it's little enough that it'll cause no processing problems. If I process 300 whatevers and then the display can only show 120, I haven't burned much of the battery. I overstimated by a factor of 2.5, burned a minuscule part of the battery, and I gained simplicity.

I strongly agree that numbers as low as 20 are a code smell.

Mechanically abbreviating strings is likely to have nasty corner cases. Avoid if at all possible.

There are fancy Unicode corner cases, like the flags (cutting a flag emoji in half doesn't get you half a flag it gets you one half of the flag's country code identifier) but there are plenty of corner cases already in ASCII.

Abbreviating "Give all the money to Samantha's parents to hold in trust" as "Give all the money to Sam" doesn't involve any scary modern encoding, just the problem that this isn't how human languages work.

Can you suggest a relevant nasty corner case?

Consider an unbounded input string (generally less than 1k, but sometimes more than 100k), a phone display large enough to display, say, 80-120 glyphs, and a mechanical abbreviation to a few hundred codepoints soon after input, before all expensive processing. What are the nasty corner cases?

You are not thinking outside of your web development box.

Embedded systems care deeply and know exactly display sizes.

Embedded systems increasingly have screens with real pixels and support for weird languages. At least they are low-level that "give me the width and height of that string" is actually an achievable task.

The ambiguity of which length is being talked about is what smells. It is a bad odor, not bad enough that I'd not use a library that had 'length' as a property, but it is absolutely a smell because I can't be sure that every single last usage of 'length', inside the library nor in my code, is using the same definition for 'length' as is meant.

It's subtle, which is what makes it a smell, rather than a "DO NOT WANT".

> "1000px at font size 8"

That one is likely to get you into trouble when you allow 0 width characters.

1. You don't iterate over bytes in an integer, so you don't need to know the size of an int since all uses are handled by the compiler. If you do care, you can always find out with a sizeof check (or preferably specify the size directly). The compiler can't handle Unicode for you since it's typically dynamic data.

2. This is reasonable.

3. What about fixed-width fonts like a terminal screen. In this case, I care about how many distinct (non-combined) code points exist, and a count of code points is often a acceptable approximation.

That being said, when talking about string length, I mostly care when iterating and allocating memory, and those are two different usages of the term "length". I like how Go handles it: len(string) gives number of Unicode code points, len([]byte(string)) gives number of bytes.

Everything else is a special case IMO that shouldn't use length, except perhaps as a conservative estimate (e.g. fixed width rendering).

If you're not using a modern iterator, you'll want string length and string indexing. Otherwise you don't need string lengths except for the reasons given in TFA and above, but (1) (to allocate memory) can be hidden in most cases, (2) is mostly not necessary (account for encoded bytes where possible and be done), and there's very little code that needs to figure out (3).

For almost every task, an iterator that iterates graphemes (extended grapheme clusters) is the right thing to do.

I do think it's annoying that Unicode ended up with such a complex extended grapheme cluster scheme though.

> Treat code that cares about string "length" as code smell

This is nonsense. Pretty much anywhere that you're accepting user input on the web you want to have some sort of length limit. Even if you're Hacker News rather than Twitter, you probably don't want to let anybody post a 10GB comment. You just need to pick an appropriate length metric. That's part of the post's point, and it's also apparently your point a few sentences after you rubbish it; why are you insulting the post and then suddenly agreeing with it?

for what is worth, I have used the string byte length (i.e. the utf-8 code unit count) as a (weak) component for word weight in a clustering algorithms.