Cuneicode, and the Future of Text in C
thephd.dev
thephd.dev
#include <stdio.h>
#include <locale.h>
#include <stdlib.h>
#include <windows.h>
#include <io.h>
int main() {
if (!setlocale(LC_ALL, "Japanese_Japan.932")) {
puts("setlocale failed");
exit(1);
}
// U+00A7 SECTION SIGN encoded in CP932
char cp932_string[] = "\x81\x98\n";
// this converts the encoding:
printf("%s", cp932_string);
// this too:
fwrite(cp932_string, 1, sizeof(cp932_string) - 1, stdout);
// this too:
_write(1, cp932_string, sizeof(cp932_string) - 1);
// this outputs raw bytes:
DWORD bytes_written;
WriteFile(
GetStdHandle(STD_OUTPUT_HANDLE),
cp932_string, sizeof(cp932_string) - 1, &bytes_written, NULL
);
return 0;
}
>chcp
Active code page: 852
>.\a.exe
§
§
§
üś> When stream is opened in Unicode translation mode—for example, if stream is opened by calling fopen and using a mode parameter that includes ccs=UNICODE, ccs=UTF-16LE, or ccs=UTF-8, or if the mode is changed to a Unicode translation mode by using _setmode and a mode parameter that includes _O_WTEXT, _O_U16TEXT, or _O_U8TEXT—buffer is interpreted as a pointer to an array of wchar_t that contains UTF-16 data. An attempt to write an odd number of bytes in this mode causes a parameter validation error.
I presume stdout is open in that mode, but can't verify it.
Try adding:
_setmode(_fileno(stdout), _O_BINARY);
But you are right, that's not mentioned in the linked-to examples.
Does setlocale() have a side-effect of modifying the mode?
_write(), which is used internally by stdio, performs encoding translation if all of the following are true:
- The current locale isn't "C".
- The _O_TEXT flag is set.
- The fd points to the console.
When you redirect the stdout (e.g. with > in a shell), it's no longer considered a console handle, so the translation won't be applied.
Personally, I think it was a weird design decision to include encoding and CRLF translation in what Microsoft calls "low-level I/O".
> The execution character set is the encoding used for the text of your program that is input to the compilation phase after all preprocessing steps. This character set is used for the internal representation of any string or character literals in the compiled code.
But the author seems to be using "execution encoding" to mean "the current locale of the running program". These are completely different things.
See the following links for the test program that's outputting "Execution Encoding":
https://github.com/soasis/cuneicode/blob/018d284088fce910ac8...
https://github.com/soasis/idk/blob/ada0f119da3ccce0f1f58abc9...
1: https://learn.microsoft.com/en-us/cpp/build/reference/execut...
First it defines the base execution character set. That includes: upper and lower case A to Z, digits 0 to 9, these symbols !"#%&'*+,-./:;<=>?[\]^_{|}~ , space, horizontal tab, vertical tab, form feed, alert, backspace, CR, LF. These will all have single byte encoding.
The full execution character set is further defined as the basic execution characters (single byte), plus additional locale-specific characters (either single or multi-byte):
> The source character set may contain multibyte characters, used to represent members of the extended character set. The execution character set may also contain multibyte characters, which need not have the same encoding as for the source character set. For both character sets, the following shall hold:
> - The basic character set shall be present and each character shall be encoded as a single byte.
> - The presence, meaning, and representation of any additional members is locale-specific. [ED: emphasis added]
> - [allows for shift-dependent encodings]
> - A byte with all bits zero shall be interpreted as a null character independent of shift state. Such a byte shall not occur as part of any other multibyte character.
---
Which is to say, that the basic characters must have identical encodings in all supported char* strings, and everything else is dependent on the locale, which means setting the locale effectively modifies the execution character set. If you want to correctly interpret any string literal that exceeds the basic character set, you need to know what locale the compiler used when converting to the execution character set, and set the locale to match.
Any encoding that does not have the base characters having the same byte values as are not technically valid as a string, but could be represented as an arbitrary byte array, even if you choose to spell that byte array as `char*`. This also implies that no locale may legally use such as encoding.
What? No, UTF-8 won for a reason, and that reason is not just that C has a deficiency in this area but, rather, that UTF-8 is: a) simple and much saner than UTF-16, b) self-synchronizing in both directions, c) as -or even more- space efficient than UTF-16 on average even for non-Latin text, d) even UTF-32 doesn't make it possible to turn logical character string indices into 32-bit word string indices. (d) is the real killer.
One just cannot assume that a character or glyph requires just one codepoint to express, therefore one can't assume that a character or glyph will require some fixed number of code units to express, therefore one might as well use UTF-8 because it's saner and more efficient than the other UTFs, therefore... <drumroll/> a "constant clinging to and dependency upon the “multibyte” encoding" is NOT a deficiency of C but an advantage of C.
C does have problems here though, namely all the usual problems it has:
- C strings (NUL-terminated) suck
- C doesn't have a first class string type,
only pointer to char
- C doesn't have a first class string type
that indicates what encoding the string
uses
So it really is a deficiency in C. It's nearly useless to have a "multibyte" or "wide character" encoding when those can mean anything. Having conversion between UTF-8 and UTF-32 is useful. Having conversion between "implemetation and platform dependent 'multibyte'" and "implementation and platform dependent 'wide character'" strings is nearly useless.
I mentioned that. TFA didn't say that specifically.
> But, rather than using them and needing to praying to the heaven’s the internal Multibyte C Encoding is UTF-8 (like with the aforementioned wcrtomb -> mbrtoc8/16/32 style of conversions), we’ll just provide a direction conversion routine and cut out the wchar_t encoding/multibyte encoding middle man.
Not sure why it wasn't mentioned up top. When trying to convert between UTF-8 and UTF-16 without doing it myself or pulling in external dependencies, this was the most annoying thing that slapped me in the face. This is the problem that makes reliable charset conversions between specific encodings actually impossible using just the stdlib functions.
Basically, non-Unicode needs to always be at the edge, while in the middle everything needs to be Unicode.
From an application perspective it's easy: document that it only works in UTF-8 locales. Really, that is my position for my software. Anything else is ETOOHARD.
I found it ridiculous that there was no way to just convert UTF-16 to UTF-8 without either reinventing that wheel, pulling in an external dependency, or changing global state and having the right system locales installed (as well as knowing the name of at least one of those locales, and guessing a language along with it), despite having the latest C and C++ compilers at my disposal.
And better, write a proper string API. It's still missing.
I had no clue!
I'm just a happy little Python developer for Unix, and I think I'll stay that way.
I assume you haven’t supported a twenty-year-old website entirely encoded in KOI8-R—the ability to occasionally just set LC_ALL=ru_RU.KOI8-R can be invaluable, and the same caveats would apply there.
Granted, I don’t think modern Python versions will even run properly by default under such conditions. (ETA: Ah, no, I was thinking about PEP 528 and 529, and those only apply to Windows. So dunno.)
Verifying your "Ah, no" on a Mac:
% python3.11
>>> print("\N{CYRILLIC CAPITAL LETTER SHORT I}")
Й
>>> ^D
% env LC_ALL=ru_RU.KOI8-R python3.11
>>> print("\N{CYRILLIC CAPITAL LETTER SHORT I}")
�
>>> ^D
luit -encoding koi8-r env LC_ALL=ru_RU.KOI8-R python3.11
𝔉𝔬𝔯 𝔊𝔬𝔡 ſ𝔬 𝔩𝔬𝔳𝔢𝔡 𝔶ͤ 𝔴𝔬𝔯𝔩𝔡,𝔱𝔥𝔞𝔱
𝔥𝔢 𝔤𝔞𝔲𝔢 𝔥𝔦𝔰 𝔬𝔫𝔩𝔶 𝔟𝔢𝔤𝔬𝔱𝔱𝔢𝔫 𝔖𝔬𝔫𝔫𝔢: 𝔱𝔥𝔞𝔱
𝔴𝔥𝔬ſ𝔬𝔢𝔲𝔢𝔯 𝔟𝔢𝔩𝔢𝔢𝔲𝔢𝔱𝔥 𝔦𝔫 𝔥𝔦𝔪 , ſ𝔥𝔬𝔲𝔩𝔡
𝔫𝔬𝔱 𝔭𝔢𝔯𝔦ſ𝔥, 𝔟𝔲𝔱 𝔥𝔞𝔳𝔢 𝔢𝔲𝔢𝔯𝔩𝔞𝔰𝔱𝔦𝔫𝔤 𝔩𝔦𝔣𝔢.
But I have only skimmed the article so maybe I missed something
edit: I note that HN filters out utf8 emojis..
These days one of the systems level programming languages (Rust) is natively UTF-8.
This would have been a huge source of drama in say, 1993 (when UTF-8 is basically brand new) and maybe even in 2003 (by which point it's clear Unicode is a success, but still conceivable that UTF-16 "wins" in some sense) but in 2023 people just shrug - obviously it's UTF-8, why not?
C in principle allows to abstract over the system encoding.
> Until recently, Windows has emphasized "Unicode" -W variants over -A APIs. However, recent releases have used the ANSI code page and -A APIs as a means to introduce UTF-8 support to apps. If the ANSI code page is configured for UTF-8, -A APIs typically operate in UTF-8. This model has the benefit of supporting existing code built with -A APIs without any code changes.
This requires Windows 10 version 1903 or newer. As the version number suggests, this feature is only four years old, and using them prevents your program from working on older versions.
[0]: https://learn.microsoft.com/en-us/windows/apps/design/global...
There's no reason all the rest of your software needs to pay for Microsoft's mistake.
> and converting to/from UTF-8 on every Windows API call is not only a pessimization both in runtime and memory usage, but also introduces complications in having to handle possible conversion failures (involving unpaired surrogate characters, in particular).
If the text isn't actually Unicode, then all you're demonstrating by not handling that in another language is that you didn't care about correctness. There's no magic here, unpaired surrogates aren't somehow actually valid Unicode so long as you stick with C.
Traditionally, strings in C are encoding-agnostic. By default, the encoding of the current locale is assumed, and applications are expected to convert to other encodings when specifically needed. Forcing everything into “there can only be one encoding we can work with” isn’t a great solution. One can wish that everything was UTF-8 from the beginning, but that’s not the world we live in.
How is it a mistake when UTF was not even a thing when the NT kernel was developed?
Note that Rust has OsString to handle both of these cases, but it’s somewhat hard to use.
Except that the geniuses who maintain the extant Unix systems still couldn't bring themselves to at least start transitioning pathnames to utf8, so now we're forever stuck with the terrible OsString cruft and wasting brain and CPU cycles converting from and to OsString in some unsatisfactory manner.
At least ZFS (which also incidentally is pretty much the only non-terrible filesystem) has an utf8only flag.
As far as POSIX and Unix `open(2)` and friends go, pathnames are opaque binary with only two special byte values: NUL (0x00, because it's C strings, which are NUL-terminated) and '/' (because that's the path component separator). Any codeset and form that is compatible with that will "work" -- for some value of "work" where if the codeset isn't Unicode in UTF-8 then you'll be sad.
Nothing keeps users / sites from declaring that thou shall use UTF-8 on the filesystem, or, even better, that thou shall use UTF-8 locales only, as the latter is the only simple way to [mostly] get the former.
> At least ZFS (which also incidentally is pretty much the only non-terrible filesystem) has an utf8only flag.
Note that ZFS can't tell if the strings from user-land are in UTF-8. ZFS can only tell if they're not valid UTF-8.
And there is no serious backwards compatibility problem. You just introduce a mount option to enforce utf-8, flip it to default after a say a decade, and people who then still have file systems with pathnames in latin-1 (or other craziness) can then flip it off and get a few extra decades to migrate their stuff. In the meantime the rest of the world just writes software under the assumption that it will be deployed to non-crazy installs, and software becomes magically more reliable and quicker to write.
And it's not like even 1% of software now would handle non-utf-8 filenames robustly anyway (I mean even if you are aware of the problem and diligent about it, there is absolutely no good way to deal with non-utf8 filenames if you need to output them as text for consumption by humans or other programs, which is close to 100% of cases, because at the very least you will need to do so in error messages if there is an IO problem).
Uh, not exactly, not quite. UTF-8 was invented by Unix people who were fed up with all the alternative proposals at the time. Those people did not then go and "fix Unix" in any way regarding this. Those people were none others than Ken Thompson and Rob Pike, who were among the creators of Unix -- you could only have gotten closer to "those people" being the creators of Unix by having had Denis Ritchie at that diner table on that fateful day.
And even if you settle for utf-8 you'll still have to deal with differences with filesystems automatically canonicalizing names or being case insensitive.
Exactly what you would want to happen (this is the actual error message you will get, BTW)?
> (cd /mnt/funkyfilesystem && touch $'\370' && tar cf my-bad-file.tar $'\370')
> tar xf /mnt/funkyfilesystem/my-bad-file.tar
tar: \370: Cannot open: Invalid or incomplete multibyte or wide character
tar: Exiting with failure status due to previous errors
> And even if you settle for utf-8 you'll still have to deal with differences with filesystems automatically canonicalizing names or being case insensitive.
(Case-preserving you mean? I think truly case-insensitive died with 8.3 DOS filenames). These are also annoying, but a much more minor issue, e.g. you don't need to create a new weird string type just because of them.
But hey, if enough tools and applications stop supporting non-utf-8 names, it is possible that in a a few decades you might get what you want as de-facto non-utf8 files will just disappear.
Let's say you get an EILSEQ on accessing the dirent with readdir and now have to debug how you got some corrupted filename you clearly didn't want in the first place, remount and fix it. How is that worse than having the corrupted filename and not knowing about it before something more insidious happens?
Another good one is being completely honest about your height (mostly just filters out American expats but still).
Anyway, the only reason I used emoticons was because I use this keyboard:
http://www.exideas.com/ME/index.php
... but it apparently had some side-effects.
That a and ´ and so on are related is a feature of the Latin scripts, and the need/desire to have a coding system that can express that relatedness would exist even if you object to that desire.
Almost everything about Unicode that people love to hate comes from humans who have nothing to do with Unicode:
- composition (thus multi-codepoint glyphs)
- weird collation rules
- weird equivalence rules
- the need for normalization and/or
normalization-insensitive string hashing
and comparison
The only complexity that could have been elided would have been pre-compositions and NFC and NFKC, and maybe the K mappings so that NFKD also could be dispensed with. Everything else would still necessarily be a part of Unicode or any competitor to Unicode.
And no, lots of codesets, one per-script, does not work as an alternative -- we tried it and it sucks, and that's why Unicode exists.
To complain about Unicode's complexity is to tilt at windmills.
... but it's not Unicode's fault!
> To complain about Unicode's complexity is to tilt at windmills.
A specification that people don't want to fully support is a bad specification;
Complaining about programs and tools that don't implement everything there is to Unicode is what is tilting at windmills.
At the end of the day, the programs win; they determine the functionality you have, not the specification.
A specification that people don't want to fully embrace is a bad specification.
> ... but it's not Unicode's fault!
Right, it's not.
> [the rest]
You say that as if current support for Unicode will not improve, as if somehow it has never improved (but then, even what we have now wouldn't exist). That's clearly not the case.
- Combining characters, joiners, modifiers, etc. should have been prefix, not suffix/infix, and in blocks by arity. This would have allowed detecting incomplete sequences, allowed code to skip combinations without knowing their meaning, and made dead-key input trivial.
- A standard normalized form should have been specified up front.
- Once blocks overflowed, subsequent blocks should have been made large enough that more that one would not be necessary. Unicode has ten different Latin blocks. You've got ‘Kana Extended-B’, ‘Kana Supplement’, ‘Kana Extended-A’ (in that order!), and ‘Small Kana Extension’. (If there's an explanation for ‘Supplement’ vs ‘Extended’ vs ‘Extension’, I've never seen it.)
- Character ordering is inconsistent even within scripts, e.g. ÀÁÂ… àáâ but ĀāĂ㥹.
- Assigning code points for the UTF-16 encoding is ridiculous.
- The contentious cases of CJK unification should have been disambiguated promptly, before use of the originals became entrenched. (I know where to find 2K code points they could have used…)
- They recently changed some characters to have ‘emoji presentation’ by default, meaning that existing text needs to be edited to preserve its original appearance.
Yes, this is a very good suggestion. UTF-8 is a multi-byte encoding that is self-synchronizing in either direction. Multi-codepoint characters/glyphs should have been similarly self-synchronizing in either direction.
> - A standard normalized form should have been specified up front.
There are four. The only reasonable simplifications here would have been to get rid of NFC and NFKC, and maybe NFKD, leaving only NFD, and then maybe also remove all precompositions (but this would have made take-up of Unicode take much longer because it would have required more complexity in implementations earlier).
> - Once blocks overflowed, subsequent blocks should have been made large enough that more that one would not be necessary. Unicode has ten different Latin blocks. You've got ‘Kana Extended-B’, ‘Kana Supplement’, ‘Kana Extended-A’ (in that order!), and ‘Small Kana Extension’.
That's a function of how allocations were made, especially before UTF-8 and UTF-16, when we were stuck with just the BMP.
Well, anyways, we'd need a time machine to get any of these things fixed now.
> - Character ordering is inconsistent even within scripts, e.g. ÀÁÂ… àáâ but ĀāĂ㥹.
Do you mean codepoint ordering? Hardly matters: the actual ordering will vary by locale anyways.
> - Assigning code points for the UTF-16 encoding is ridiculous.
That's not a thing.
> - The contentious cases of CJK unification should have been disambiguated promptly. (I know where to find 2K code points they could have used…)
That would have required getting representatives from those countries to sit down at the UC table sooner.
> - They recently changed some characters to have ‘emoji presentation’ by default, meaning that existing text needs to be edited to preserve its original appearance.
Blech. Occasionally the UC has to make backwards-incompatible changes, but they should try really hard to keep those to the absolute bare minimum -- zero would be ideal.
I was referring to U+D800–U+DFFF being reserved just because UTF-16 happens to use 0xD800–0xDFFF for its encoding scheme. Contrast valid code points U+0080–U+00FF, while (most of) 0x80–0xFF are used by UTF-8.
I have an implementation in which codes U+DC00 through U+DCFF indicate "this was a bad byte in the original UTF-8". When UTF-8 is generated, these values reproduce the original invalid bytes. Additionally one of those values, U+DC00, is particularly useful since it encodes a null byte that occurred in the UTF-8 stream. While that is not strictly necessary as far as UTF-8 goes, the encoding provides a way to embed the null character in null-terminated strings.
* https://jdebp.uk/Softwares/nosh/guide/commands/console-fb-re...
One character = one code point.
I would use 32 bits for a code point. The lower 16 bits would be code; the upper 16 bits contain some flags and fields for classification. This would allow simple bit operations to inquire about important properties. There would be a 4-bit flexible type code, leaving 12 bits, some of which would have a code-specific meaning, others fixed. Or something like that.
The goal would be to have a code that programs can work with, without requiring megabytes and megabytes of meta-data about the code.
Every symbol in Morse? Easy. Every symbol in ASCII? Easy. Add accents, and now you need to ask: are these (1) unique letters with their own sort order, or (2) modifiers that go "on top of" other letters? Answers may vary by language. Add ligatures, and now you're also forced to care about character length even if you would rather not, e.g. ﷽ (U+FDFD). Emoji? Simple ones are fine… but the general case has ways to combine characters to make different icons, making them a systematic synthlang in the same style as Chinese and Japanese[0].
What even is the right sort order of "4", "四", and "៤"? Or "a" vs. "A"?
What about writing directions? LTR, RTL, the ancient Greek thing whose name I forget with alternating directions on each line, the Egyptian hieroglyphics where it can be either depending on which way the heads face, or vertical?
What about alphabets like Arabic where glyphs are not in general separate, and where they can take different forms depending on if they're the first in a word, the last in a word, in the middle of a word, or isolated?
What about archaic letter forms? e.g. ſ, þ, and ð in old English.
[0] where I will probably embarrass myself by suggesting 犭(dog) + 瓜 (melon) = 狐 (fox), which, though cute, feels like as much a false etymology as suggesting that in English "congress" is the opposite of "progress". Or perhaps Japanese foxes really are melon dogs — I don't know, I can barely count to 4 in that specific writing system.
I don't know Japanese but with what I know about classical Chinese character construction, I'd expect that melon acts as a phonetic hint and dog hints at the meaning (e.g. the word this character represents sounds like "Melon" but is related to "Dog").
Edit: I was curious and looked it up, it's exactly this https://en.wiktionary.org/wiki/%E7%8B%90#Chinese
I believe the word you're looking for is "boustrophedon"
And Java and JavaScript followed Windows, and Python is constrained by it
The surrogate pairs of UTF-16 even infected JSON and thus implementations in all languages, but funny enough encoded JSON is specified to be UTF-8, which is better but a bit confusing
Newer, sane languages like Go and Rust are more Unix-like and use UTF-8 natively
It’s basically a Windows vs Unix problem
UTF-8 was publicly presented at USENIX at the beginning of '93. Not sure when Unicode incorporated it.
It is unlikely that Windows would have been changed at the last minute to use it, especially as the variable encoding of UTF-8 was significantly more complicated than the fixed size UCS-2.
If only UTF-8 had been invented a little earlier, we could have avoided so much pain :-(
The idea of global varables like LANG= and LC_TYPE= in C is utterly incoherent.
Python's notion of "default file system encoding" is likewise incoherent.
You can obviously process strings with two different encodings in the same program !!! Encodings are metadata, and metadata should be attached to data. Encodings shouldn't be global variables!
Python 3 made things worse in many ways, largely due to adherence to Windows legacy, and then finally introduced UTF-8 mode:
https://vstinner.github.io/painful-history-python-filesystem...
So, you can't, because Unicode processing can be (though I'm not sure how much is) locale dependent, and that that metadata is NOT attached to data. Unicode Consortium had been messing up non-Latin languages multiple times, causing hacks and new standards to build on top of UTF-8. Han Unification immediately comes to mind[1], but there are others as the Korean Mess[2], Cambodian Khmer problem[3], to name a few. I don't quite understand why it's always has to be like that.
1: Sets of characters from zh-Hans(zh-CN), zh-Hant(zh-TW), kr-KR, ja-JP that were deemed "same" were merqed lnto same code points, in an attempt to keep commonly used UTF-8 in nice 2 bytes
2: Korean Hangul characters were literally relocated between Unicode 1.1 to Unicode 2.0, causing affected characters written in 1.1 displayed in just unrelated characters
3: Reportedly the Consortium simply did not have a Cambodian linguist(???) (partly due to unrest and genocide that took place during 60s-80s)
Then a decent library design would let you process those in different threads in the same program
A global variable like LANG= inhibits that
So if you have metadata, it should be attached to the DATA, and not the CODE
---
Same thing with a file system. You can obviously have 2 different files on the same disk with different encodings. So Python's global FS encoding and global encoding doesn't make any sense.
They are basically "punting" on the problem of where the metadata is, and the programmer often has NO WAY to solve that problem!
---
The issues you mention are interesting but I think independent of what I'm saying
If French or Greek or Russian or Japanese had happened to be the lingua franca of the Internet, the features those writing systems require would be first class, and the implementation of English would be the one that felt like a bit more of a kludge. Imagine if computers were designed around unicase Arabic languages [1], and then years later there was a "Han Unificiation"-type summit where it was decided that capitalization was to be handled with extra metadata, since an upper- lower-case latin letter are really just the same thing and why bother wasting double the codepoints on that.
(I don't have a link handy, but I believe I read somewhere that you can see some of this effect in NES-era RPGs from Japan— that some of the narrative had to be simplified down for the English translations because the text boxes were originally built for a small number of complex characters rather than the larger number of simpler ones that would be required to capture the same meaning.)
Unicode has round trip compatibility with the old encodings, too, without the use of any of that extra metadata, which is only used for incredibly minor character variations which are pretty much never semantically meaningful the way capitalization is. A human copyist might change one for another in copying a text by hand, just as when copying English, you do not consider it semantically meaningful whether lowercase “a” is drawn as a circle and a line, or a circle and a hook.
https://www.sljfaq.org/afaq/encodings.html
What I will point out though, that you can easily verify with a simple search, is that Shift-JIS can be represented losslessly in Unicode, and that this has been true for as long as Unicode CJK has existed.
The continued popularity of Shift-JIS is worth noting, but it is also important to note that its continued use is not a stable thing, and has been declining for two decades. The most popular websites in Japan no longer use it, and among smaller websites, the percentage that use it gets smaller each year. Secondly, there is absolutely no benefit to the user in terms of that can be encoded and distinguished, because it will get converted into Unicode at some stage in the pipeline anyway. Any modern text renderer used in a GUI toolkit will use Unicode internally. “Weird byte sequences to indicate Unicode characters” is really all that legacy encodings are from the perspective of new software. They are essentially just incompatible ways of specifying a subset of Unicode characters.
As for why it has held on so long, there are a few reasons. The Japanese tech world can be quite conservative and skeptical about new things. That is one factor. But I think another is that Japan was really a computing pioneer in the 1980s, and local standards ruled supreme. Compatibility was not a big concern, and even the mighty IBM PC and its clones barely made an impact there for a long time, as it was completely eclipsed by Japanese alternatives. Now, everyone is forced by our increasingly interconnected world to work on international standards, and I can’t help but feel that there is some resentment at not being about to just “do their own thing” anymore. Every time a new encoding extension is proposed, they have to present it to an organization that includes China, Korea, Taiwan, and Vietnam, who will scrutinize it. A few years ago JIS (the Japanese standards organization) actually proposed that each East Asian country should just get their own blocks in Unicode and they should be able to encode whatever they want with no input from others. Of course, none of the other East Asian countries took their proposal seriously. I wish I could find the proposal, because I hate saying stuff like this without sources, but I can tell you that it is buried somewhere among all of the proposal documents that you can find on the website of the Ideographic Research Group, which is the organization that East Asian countries participate in which is responsible for CJK encoding in Unicode. You might find it here with enough scrolling. I have to get on the subway though, so I have to end this comment here.
This all goes back to typewriters, where this is how one typically wrote accented characters.
And most of this got lost. Though we still use overstrike for underline and bold in the terminal.
English used to be cursive-only, like something like Arabic. But the language evolved to fit into a modern world, and I think other written languages would benefit from a simplified "print" version as well.
Computers merely got it wrong several times, mostly for silly reasons.
More like English uses a relatively simple writing system which has the drawback of being unable to properly represent all the sounds of the language.
Like a lot of things in life it's a question of tradeoffs, not being right or wrong
English's lack of diacritical marks is why romanized Korean is so difficult to read, with digraphs used for monophthongs.