It will be relegated to the computing dustbin like non-8-bit bytes and EBCDIC.
Main-core computing is vastly more homogenous than when I was born almost 50 years ago. I guess that's a natural progression for technology.
I wish the same applied to written numbers in LTR scripts. Arithmetic operations would be a lot easier to do that way on paper or even mentally. I also wish that the world would settle on a sane date-time format like the ISO 8601 or RFC 3339 (both of which would reverse if my first wish is also granted).
> It will be relegated to the computing dustbin like non-8-bit bytes and EBCDIC.
I never really understood those non-8-bit bytes, especially the 7 bit byte. If you consider the multiplexer and demux/decoder circuits that are used heavily in CPUs, FPGAs and custom digital circuits, the only number that really makes sense is 8. It's what you get for a 3 bit selector code. The other nearby values being 4 and 16. Why did they go for 7 bits instead of 8? I assume that it was a design choice made long before I was even born. Does anybody know the rationale?
IIRC, in most countries the native format is D-M-Y (with varying separators), but some Asian countries use Y-M-D. Since those formats are easy to distinguish, that's no problem. That's why Y-M-D is spreading in Europe for official or technical documents.
There's mainly one country which messes things up...
If I'm writing a document for human consumption then why would I expect the dates to be sortable by a naive string sorting algorithm?
On the other hand, if it's data for computer consumption then just skip the complicated serialisation completely and dump the Unix timestamp as a decimal. Any modern data format would include the ability to label that as a timestamp data type. If you really want to be able to "read" the data file then just include another column with a human-formatted timestamp, but I can't imagine why in 2025 I would be manually reading through a data file like some ancient mathematician using a printed table of logarithms.
If you're naming a document for human consumption, having the files sorted by date easily without relying on modification date (which is changed by fixing a typo/etc...) is pretty neat
Its a serialization and machine communication format. And that makes me sad. Because YYYY-MM-DD is a great format, without a good name.
> NOTE: ISO 8601 defines date and time separated by "T". Applications using this syntax may choose, for the sake of readability, to specify a full-date and full-time separated by (say) a space character.
For the representation of text of an alphabetic language, you need to hit 6 bits if your script doesn't have case and 7 bits if it does have case. ASCII ended up encoding English into 7 bits and EBCDIC chose 8 bits (as it's based on a binary-coded decimal scheme which packs a decimal digit into 4 bits). Early machines did choose to use the unused high bit of an ASCII character stored in 8 bits as a parity bit, but most machines have instead opted to extend the character repertoire in a variety of incompatible ways, which eventually led to Unicode.
I wouldn’t be surprised if other machines had something like this in hardware.
Only if you assume a 1:1 mapping. But e.g. the original Baudot code was 5-bit, with codes reserved to switch between letters and "everything else". When ASCII was designed, some people wanted to keep the same arrangement.
[1] Yes, I remember you could bit-bang a UART in software, but still the parity bit didn't escape the serial decoding routine.
When your RAM is vacuum tubes or magnetic core memory, you don't want 25% of it to go unused, just to round your word size up a power of two.
wasnt this more to do with cost? they could do arbitrary precision code even back then. its not like they were only calculating numbers less than 65537, ignoring anything larger
Doing numbers little-endian does make more sense. It's weird that we switch to RTL when doing arithmetic. Amusingly the Wikipedia page for Hindu-Arabic numeral system claims that their RTL scripts switch to LTR for numbers. Nope... the inventors of our numeral system used little-endian and we forgot to reverse it for our LTR scripts...
Edit: I had to pull out Knuth here (vol. 2). So apparently the original Hindu scripts were LTR, like Latin, and Arabic is RTL. According to Knuth the earliest known Hindu manuscripts have the numbers "backwards", meaning most significant digit at the right, but soon switched to most significant at the left. So I read that as starting in little-endian but switching to big-endian.
These were later translated to Arabic (RTL), but the order of writing numbers remained the same, so became little-endian ("backwards").
Later still the numerals were introduced into Latin but, again, the order remained the same, so becoming big-endian again.
And as for numbers, perhaps it isn't too late to set it right once and for all. The French did that with the SI system after all.
> So apparently the original Hindu scripts were LTR
I can confirm. All Indian scripts are LTR (Though there are quite a few of them. I'm not aware of any exceptions). All of them seem to have evolved from an ancient and now extinct script named Brahmi. That one was LTR. It's unlikely to have switched direction any time during subsequent evolution into modern scripts.
But why? The brilliance of 8601/3339 is that string sorting is also correct datetime sorting.
To get the little-endian ordering. The place values of digits increase from left to right - in the same direction as how we write literature (assuming LTR scripts), allowing us to do arithmetic operations (addition, multiplication, etc) in the same direction.
> The brilliance of 8601/3339 is that string sorting is also correct datetime sorting.
I hadn't thought about that. But it does reveal something interesting. In literature, we assign the highest significance to the left-most (first) letter - in the direction opposite to how we write. This needs a bit more contemplation.
Yes, we do that with everything, which is why little-endian numbers would be really inconsistent for humans.
YYYY-MM-DD to me always feels like a timestamp, while when I want to write a date, I think of a name, (for me DD. MM. YYYY).
For better or worse, PowerPC is still quite entrenched in the industrial embedded space.
So it's far more pervasive than people think, and will likely be in the picture for decades to come.
Of course they chose to integrate JavaScript so that's less likely now.
[1] Ok I admit, not trivially when it comes to unpaired surrogates, BOMs, endian detection, and probably a dozen other edge and corner cases I don't even know about. But you can offload the work to pretty well-understood and trouble-free library calls.
Most Unix syscalls use C-style strings, which are a string of 8-bit bytes terminated with a zero byte. With many (most?) character encodings you can continue to present string data to syscalls in the same way, since they often also reserved a byte value of zero for the same purpose. Even some multi-byte encodings would work if they chose to avoid using 0-value bytes for this reason.
UTF-16LE/BE (and UTF-32 for that matter) chose not to allow for this, and the result is that if you want UTF-16 support in your existing C-string-based syscalls you need to make a second copy of every syscall which supports strings in your UTF-16 type of choice.
That's completely wrong. If a syscall (or a function) expects text in encoding A, you should not be sending it in encoding B because it would be interpreted incorrectly, or even worse, this would become a vulnerability.
For every function, encoding must be specified as are specified the types of arguments, constraints and ownership rules. Sadly many open source libraries do not do it. How are you supposed to call a function when you don't know the expected encoding?
Also, it is better to send a pointer and a length of the string rather than potentially infinitely search for a zero byte.
> and the result is that if you want UTF-16 support in your existing C-string-based syscalls
There is no need to support multiple encodings, it only makes things complicated. The simplest solution would be to use UTF-8 for all kernel facilities as a standard.
For example, it would be better if open() syscall required valid UTF-8 string for a file name. This would leave no possibility for displaying file names as question marks.
Unfortunately some text encodings (UTF-16 among them) use nuls for codepoints other than U+00. In fact UTF-16 will use nuls for every character before U+100, in other words all of ASCII and Latin-1. Therefore you can't just support _all_ text encodings for filenames on these OSes, unless the OS provides a second syscall for it (this is what Windows did since they wanted to use UTF-16LE across the board).
I've only mentioned syscalls in this, in truth it extends all through the C stdlib which everything ends up using in some way as well.
It is good for an implementation to enforce this at some level, sure. MacOS has proved features like case insensitivity and unicode normalization can be integrated with Unix filename APIs.
And you should be using one specified encoding for file names if you want them to be displayed correctly in all applications. It would be inconvenient if different applications stored file names in different encodings.
For the same reason, encoding should be specified in libraries documentation for all functions accepting or returning strings.
UTF-16 is annoying, but it's far from the biggest design failure in Unicode.
UTF-32 is the worst of all worlds. UTF-16 has the teeny tiny advantage that pure Chinese text takes a bit less space in UTF-16 than UTF-8 (typically irrelevant because that advantage is outweighed by the fact that the markup surrounding the text takes more space). UTF-8 is the best option for pretty much everything.
As a consequence, never use UTF-32, only use UTF-16 where necessary due to backwards compatibility, always use UTF-8 where possible.
There's also the problem that grapheme cluster boundaries change over time. Unicode has become a true mess.
Not really. Unicode is still fundamentally based off of the codepoints, which go from 0 to 2^16 + 2^20, and all of the algorithms of Unicode properties operate on these codepoints. It's just that Unicode has left open a gap of codepoints so that the upper 2^20 codepoints can be encoded in UTF-16 without risk of confusion of other UCS-2 text.
The expansion of Unicode beyond the BMP was designed to facilitate an upgrade compatibility path from UCS-2 systems, but it is extremely incorrect to somehow equate Unicode with UTF-16.
Then there is also the issue that technically there is no such thing as UTF-16, instead you need to distinguish UTF-16LE and UTF-16BE. Even though approximately no one uses the latter we still can't ignore it and have to prepend documents and strings with byte order markers (another wasted pair of code points for the sake of an encoding issue) which mean you can't even trivially concatenate them anymore.
Meanwhile UTF-8 is backwards compatible with ASCII, byte order independent, has tons of useful properties and didn't require any Unicode code point assignments to achieve that.
The only reason we have UTF-16 is because early adopters of Unicode bet on UCS-2 and were too cheap to correct their mistake properly when it became clear that two bytes wasn't going to be enough. It's a dirty hack to cover up a mistake that should have never existed.
That's a strange way to characterize years of backwards compatibility to deal with
https://devblogs.microsoft.com/oldnewthing/20190830-00/?p=10...
I disagree you could just "easily" shove it into the "A" version of functions. Functions that accept UTF-8 could accept ASCII, but you can't just change the semantics of existing functions that emit text because it would blow up backwards compatibility. In a sense it is covariant but not contravariant.
And now, after you've gone through all of this effort: what was the actual payoff? And at what cost if maintaining compatibility with the other representations?
Also ISO 8601 (YYYY-MM-DD) should be the default date format.
I have a relatively large array of uint16_t with highly repetitive (low entropy) data. I want to serialize that to disk, without wasting a lot of space. I run compress2 from zlib on the data when serializinsg it, and decompress it when deserializing. However, these files make sense to use between machines, so I have defined the file format to use compressed little endian 16-bit unsigned ints. Therefore, if you ever want to run this code on a big-endian machine, you need to add some code to first flip the bytes around before compressing, then flipping them back after decompressing.
You're right that when your code is iterating through data byte for byte, you can write it in an endian-agnostic way and let the optimizer take care of recognizing that your shifts and ORs can be replaced with a memcpy on little-endian systems. But it's not always that simple.
Hey, you! You're supposed to be dead!