Hacker News Reader

Comment by Manishearth | Hacker News Reader

Note that JS isn't UTF-16, it's slightly different. In JS unpaired surrogates are allowed. String indexing is by code unit, so non-BMP code points are considered to be two "characters".

IIRC Java also allows unpaired surrogates? But does consider non-BMP codepoints to be single characters. I think. Idk.

So from JSs point of view it is a fixed-width encoding, one where all byte values are valid. `"\uD83D\uDC68".match(/\ud83d/)` works (U+D83D is a high surrogate, and "\uD83D\uDC68" is U+1F468 MAN), so the regex engine operates on code units, not code points.

I think for Java the regex engine would have to deal with it as a variable-width encoding, though.

Edit: Hacker News removes emoji from comments.

View on HN

Java strings are sequences of 16-bit units and don't guarantee UTF-16 validity. There are newer standard library APIs that perform astral-aware operations, though.

So does regex matching break up paired surrogates? It seems like Java Characters can be from the higher planes. JS treats surrogates as their own kind of character, but I think Java only does this for unpaired surrogates, so the encoding is still multibyte. ICBW.

Java, like NT, was originally UCS-2 and moved to UTF-16 when it became clear that UCS-2 won't suffice. APIs that operate on units (anything involving "char"s [the type], CharSequences and so on) are legacy and, when used, are almost certainly used incorrectly. Note that newer APIs on String also use the word "char", but the type "int", these are actually operating on code points and not CUs, ie. these APIs are correct.

(Note that java.lang.Character is a char, so wrong. I think there is no CodePoint type or similar that wraps int.)

In Java regexps, the dot matches a code point or an unpaired surrogate, so if you have a proper surrogate pair, the dot matches the whole pair without splitting it. It does split grapheme clusters.

It's unclear to me what you mean by "Java Characters". A Java char is a UTF-16 code unit, and the java.lang.Character boxes a char (16 bits unsigned). However, since Java 5, java.lang.Character has static methods for astral-aware UTF-16 operations where a code point is a Java int (32 bits signed).

Yeah, by Character I meant java.lang.Character.

> the dot matches the whole pair without splitting it

Yeah, so UTF16 in Java is being used as a variable-width encoding. Nice to know.

Reply on news.ycombinator.com