JEP 400: UTF-8 by Default
openjdk.org
openjdk.org
The "UTF-8 by Default" refers to the default character set used for decoding and encoding, when Java Strings are read and written. Prior to JEP 400, the default character set was "platform specific." It was often UTF-8, but sometimes it was not, in which case hijinks ensued.
As noted elsewhere the internal representation of Java Strings is either UTF-16-like or is Latin-1, if all the characters of the String can be encoded in Latin-1. We think about using UTF-8 as an internal representation constantly. Doing so would potentially save space and reduce decoding/encoding overhead.
Unfortunately doing this is difficult. The key issue is that tons of code out there treats a String as a UTF-16 array (since that's what's in the API) so there are coding patterns like this:
for (int i = start; i < end; i++) {
char ch = str.charAt(i);
// do something with ch
}
16 bits is not enough space to store all unicode characters. Does that mean that java's charAt won't join chars for codepoints over U+FFFF?
Edit: Yeah, that's exactly what happens[1]. That's not a very nice implementation at all .
> Because 16-bit encoding supports 2^16 (65,536) characters, which is insufficient to define all characters in use throughout the world, the Unicode standard was extended to 0x10FFFF, which supports over one million characters. The definition of a character in the Java programming language could not be changed from 16 bits to 32 bits without causing millions of Java applications to no longer run properly. To correct the definition, a scheme was developed to handle characters that could not be encoded in 16 bits.
> The characters with values that are outside of the 16-bit range, and within the range from 0x10000 to 0x10FFFF, are called supplementary characters and are defined as a pair of char values.
[1] https://docs.oracle.com/javase/tutorial/i18n/text/unicode.ht...
So kudos to every language that axes platform dependent encodings. Honestly platform dependent anything is so annoying to deal with. I would love if languages make it painfully explicit when you depend on platform defaults.
I'm at least glad we had some guarantees from being Python 3.
It’s Java, right? Everything is variable latency.
... other than short and byte, what are you missing? nibble?
https://docs.oracle.com/javase/tutorial/java/nutsandbolts/da...
Also, Valhalla is said to be a 7 PhD combined-difficulty feat to pull off in a backwards compatible way, so it takes time. Java actually cares about that, instead of repeatedly pulling the plug on the whole platform like Microsoft likes to do.
Java's got byte (8 bits) and short (16 bits)? (Still signed, though, of course.)
Unless those are actually padded to 32 bits in the VM or something.
They are.
And since Java doesn't expose pointer guts, struct padding is purely a runtime optimization concern, not a language concern.
The `fast` typedefs are effectively programmer-opted-in to promotion to register size or something. But compiler optimizations often do the same thing on a per-operation level (signed overflow is UB, unsigned overflow masking can be deferred for addition/subtraction/multiplication and bitwise ops (though some can change the mask)).
[I mean,make the strings use utf-8 instead of utf-16, which is a bit different than TFA]
But language wars are almost always meaningless, both have their places and they definitely don’t occupy the same niche.
Ah yes, that feature which means we must have 2x memory overhead and/or dynamic dispatch, yet is also totally useless in practice.
For the most basic case you wouldn't even need JIT support - just keep a cache field of the last index accessed and where it ended and decode based on that.
You can also ask to get the "codepoint", which returns you an int.
But, yup, the Java char primitive is 16 bits.
It's a SNAFU hard to understate. SNAFU doesn't even being to describe it. But to be honest Unicode in itself is probably the biggest SNAFU of our entire field.
Please stop taking dumps on Unicode. It does nothing for actually getting it implemented and implemented well.
Oh please, Unicode didn't start out as an encoding covering human scripts - it was too late for that. It started out as an encoding to end all already existing encodings and that was a shit show with all kinds of weird and non printable symbols to begin with. It also didn't get better when Unicode started to add all kinds of shit like combining emojis into the mix.
> It started out as an encoding to end all already existing encodings
And those other encodings were encodings for a variety of human scripts.
Re: emojis, well, they're not that different from Kanjis, not really -- both are pictographs after all, one just has much more modern functionality (e.g., color).
Explain.
EDIT: Oh! I construed this as a negative sentiment about Unicode, but perhaps it was just: "Yeah, it's complicated because language is complicated". Thanks, responders!
Case for example, why the heck would there be exactly two versions of A, one of which usually only appears at the start of words, and not always? Other people's squiggles don't do that, but notably several European writing systems (Latin and Cyrillic) do. People who use those systems think ABCDE and abcde are somehow "similar" even though they don't seem very similar at all. Then they're surprised other people don't do that.
The current state of emoji, unfortunately, makes this no longer true. (That being said, it's also designed in a way that's mostly ignorable if you don't need it. But only mostly.)
I understand why emoji is as complicated as it is. I don't think it was a good idea to bake that complication into Unicode.
It does not, and if you think it does it's obvious you didn't read UAX 29. GB11 is the most complex rule and only applies to emoji (well, and other Extended_Pictographics if you don't want to call the legacy pictographics emoji, but no Chinese).
> Specify UTF-8 as the default charset without providing any means to change it — The compatibility impact of this change would be too high.
Nah, nobody is getting this correct right now. You should've swung for the fences.
Better late than never.
I'm guessing it's not worth it, even aside from legal costs.
* If you're not really serious about using strings, why would you care?
* If you're really serious about actually doing things with Strings, you are gonna need to reimplement International Components for Unicode (or something similar in scope), and you have a free, libre and already working one for the "UTF-16 String Java" already. You don't have a working one for your custom fork with custom String handling.
What would that even mean? A forked Java's strings could insist their implementation is UTF-8 encoded bytes, but that's strings, you're talking about char. Do you want char just to be a byte, like in C ? But Java already has a byte type.
https://manishearth.github.io/blog/2017/01/14/stop-ascribing... and the followup post https://manishearth.github.io/blog/2017/01/15/breaking-our-l...
for _, c := range str {
}
https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundarie...
You are correct that there are lots of things that you can't do without grouping codepoints into grapheme clusters. And it is unfortunate that the Go standard library doesn't have utilities for grouping codepoints into grapheme clusters; the Go team's response to that is largely "don't do those things; even if we had utilities for grouping codepoints into grapheme clusters, those things would be hard to do and you are likely to get it wrong; human-level text is complicated, so treat it as opaque as much as you can."
But also, there are still lots of things you can do with just codepoints. You can tokenize. You can trim whitespace. You can do case-conversions and comparisons. You can transcode to a different encoding (whether that be a different Unicode encoding, or something like a JSON where some codepoints will need to be \uXXXX-encoded).
So yeah, codepoints aren't always what you want, but they're what you want more often than Java's "usually codepoints, but sometimes surrogate pairs". Especially because I suspect that very few Java programmers bother to test their code with surrogate pairs.
Plus, I'd bet that most (but of course, not all) code that iterates over a Java String using UTF-16 chars is broken; failing to properly handle surrogate pairs, and should instead use those iterate-over-them-as-codepoints/UTF-32 methods.