Hacker News Reader

Comment by dwheeler | Hacker News Reader

This guide omits an IMPORTANT issue: handling characters outside the Basic Multilingual Plane (BMP). JavaScript, like some other languages, suffers from the UCS-2 curse - that is, it assumes that all characters fit inside 16 bits, even though that is no longer true.

For example, the cited text says: "You can return the UTF-16 code for a character in a string using charCodeAt()...".

Not true. This only works if the UTF-16 code fits in 16 bits; if it's more than 16 bits, charCodeAt will only return a part of the character.

There are lots of discussions about this, here's one: https://stackoverflow.com/questions/3744721/javascript-strin...

JavaScript can handle characters outside the BMP, but you sometimes have to aware of the problem & carefully code around it when such characters are possible.

View on HN

Exactly, and emoji are outside the BMP, so it's not exactly an edge case, but the norm where two code units (UTF-16 double-bytes) are used to make one code point (Unicode character) [1].

And it gets even worse, when you consider that for many purposes you're not even interested in code points but in graphemes which can be sequences of code points -- e.g. a single visible emoji might actually be a sequence of 5 code points, represented by 8 UTF-16 code units, taking up 16 bytes [2]. Similarly a single accented character will often be two code points (letter plus combining diacritic).

If you want to split a string by graphemes -- e.g. to count its visible length, or delete its last visible character -- you can either use a library for it [3], or the relatively new Intl.Segmenter [4] which is in Chrome and Safari, but hasn't made it to Firefox [5].

Kind of amazing it's 2021 and you still can't calculate the number of visible characters (graphemes) in a string using native functions across all modern browsers.

[1] https://blog.jonnew.com/posts/poo-dot-length-equals-two

[2] https://www.contentful.com/blog/2016/12/06/unicode-javascrip...

[3] https://github.com/orling/grapheme-splitter

[4] https://github.com/tc39/proposal-intl-segmenter

[5] https://bugzilla.mozilla.org/show_bug.cgi?id=1423593

Thank you for these comments. I didn't know about this at all!

I'll read up on it until I understand it, and then add something to the article that covers it.

This might also help you:

https://stackoverflow.com/questions/4547609/how-to-get-chara...

e.g. use Array.from() to at least process code points rather than code units, though that's still not graphemes.

You're right about the graphemes. If you need it, you need it, but I recommend writing code that does not need to count graphemes if you can avoid them.

In many cases strings are best considered units that can be concatenated at will, but it's best if you avoid splitting them, and if you must split them, generally only split them on ASCII character boundaries. Don't consider "lengths" as something that has a meaning to humans (it doesn't), and don't assume that a "character" is a single JavaScript character (it isn't). If you normally just work strings as opaque sequences of "characters" that can be later displayed, you can avoid many complications (though obviously NOT all of them).

Reply on news.ycombinator.com