Unicode is harder than you think
mcilloni.ovh
mcilloni.ovh
I'd take "we don't do unicode at all" or "we only support BMP" or "we don't support composite characters" any day over pretend-support (but then inevitably breaking when the program wasn't tested with anything non-ASCII)
(ninjaedit: to see how prevalent it is, even gigantic message apps such as discord make this mistake. There are users on discord who you can't add as friends because the friend input field is limited to 32.... something - probably bytes, yet elsewhere the program allows the name to be taken. This is easy to do with combining characters)
The only thing anyone should actually care about is either the number of bytes it takes to store the string, or the number of pixels wide it is when rendered. Both of which are only loosely related to how many ‘characters’ or ‘grapheme clusters’ they contain, and which are themselves only vaguely correlated.
𒈙 (CUNEIFORM SIGN LUGAL OPPOSING LUGAL) is four bytes of UTF8 and as wide as about 9 Latin characters. (an England flag emoji) is about a character wide, but takes 28 bytes to store in UTF8.
Failing that, ask an AI.
There's no way to take an arbitrary utf-8 string and shorten it to <= 4000 bytes without losing something. Heck, even ASCII has \r\n which can leave you in a bad place if you cut it off in the middle.
Even if you do the Unicode stuff right and make sure you don't break a grapheme cluster, you're still cutting words in half, which, if you're doing something like pulling AP headlines and truncating them to fit on a screen, can have embarrassing consequences for one like 'FDA Chief Exposes Butter Industry Corruption'. Even if you make sure to break on word boundaries you're still at risk of turning 'US Navy Fires Nuclear Weapons Program Chief' into 'US Navy Fires Nuclear Weapons' on your news ticker.
Strings are language. Language is hard.
And what "various things" do you mean that are on top the char split?
Does it matter if it's Hangul?
The easiest way to support Unicode is to avoid supportimg it. Leave text editing to UI widgets. Leave truncation to web browsers. Avoid fancy marquee effect on the terminal. Your program may look less fancy but it will automatically support unicode, even future extensions.
Most software is not an "app," and even if it is, you at least need to be able to tell the difference between, e.g., ISO 8859-1, ISO 8859-15, and UTF-8. Otherwise you'll get text talking about cutting doorways 8œ feet tall or submitting a résumé.
Any software that needs to process information effectively needs to understand character sets, including Unicode.
And if you do deal with old data, still prefer to use UTF-8, and make sure you only translate character sets if user asks / they are explicitly specified. And for the love of god, _please_ don't just hardcode two common encodings (like 8859-1 and 8859-15)... I used to deal with encodings a lot, and the only time I actually had data loss from encoding problems was from misguided apps which assume 8859-1 (and tried to convert to ASCII by stripping accents). Please either assume UTF-8 or assume nothing and keep data as-is.
And that's it, _most_ software does not need to know more. Let's look at some examples:
- For file archiver, assume utf-8 everywhere and don't worry about encoding at all (unless you planning to deal with very old archives, in this case you'd need encoding support. But for something made from scratch, don't bother.) _Especially_ don't add any encoding support for "extract to console" function -- if I need a different encoding I can pipe into iconv myself, thank you very much.
- For API client, you don't need to worry about encoding either.. A very old HTTP server might return data in something other than utf-8 but (1) your HTTP library likely handles it already (2) how many such servers are left, anyway?
- For XML analyzer, use proper XML library, they will handle encodings for you.
- For web dashboard or a database, keep everything in utf-8 and things will just work.
- For ETL-like applications, many modern data sources are already in UTF-8.. and if they are not, convert everything to utf-8 ASAP.
In other words: while it's useful to know that encodings exist, unless you are working with very old data or legacy systems, utf-8 is the only thing you need.
Back in the day, I worked quite a bit various encodings (my language had 2 primary one and 2 secondary one, and it was a guess which one the text was), and the data loss usually happened from programs that tried to support encodings.
When program would not touch encodings, there might be some mojibake and unreadable text, but you could generally fix things.. I had to write some scripts which changed encoding of filenames and fixup random database or five, but there were no data loss.
It's the programs which were "able to tell the difference between, e.g., ISO 8859-1, ISO 8859-15, and UTF-8" which caused data loss. There were so many cases when I ended up with directory full of "?????????" or "aoeoaoao" files and that was it, there is no way to recover.
So please don't add Unicode support unless you have to, it is not ideology but rather the results of hard practical experience.
> I had to write some scripts which changed encoding of filenames and fixup random database or five, but there were no data loss.
If you do the first, it is impossible to do the second.
Are you just disagreeing for the sake of being disagreeable? Why would you do that?
Hmm, I could imagine that you have a precomposed character with two diacritical marks; if there were no precomposed character with the remaining diacritical you'd need to replace the precomposed character with the base character and diacritical. However I would imagine the better UX would be for backspace to remove the entire character (whether diacriticals were composed or not) because that's how people think of their characters.
Is there a real world case? I'd be excited to learn it.
... Hangul.
What happens when the user presses backspace when the carat is positioned just past the graphical representation of U+AC01? It should by default result in U+AC01 being replaced by U+1100 and U+1161. (In some IME implementations the user might have to configure it, e.g., "Delete by jaso unit" in Windows 7.)
Imagine if hanzi had been encoded as radicals…
Placing a fuzzy tester like "hypothesis.strategies.characters" into the CI may also be revealing.
truly an appropriate name
In practice, UTF-16 validation is rare.
https://learn.microsoft.com/en-us/windows/win32/intl/surroga...
It just happens that most Windows programs will treat it as UTF-16 and it'll mostly be fine. Until you run into the edge case where you do have file names that aren't UTF-16.
This is basically a result of Windows's file API being fixed in 1993, a good few years before Unicode decided that 16 bits wasn't enough for everyone, and hence a good few years before UTF-16 existed.
A file path is just a sequence of bytes. Don't try to validate it in any sort of encoding, it's just bytes that the OS uses to identify a file.
"That's how unix does it as well" is the exact same thing. Most people these days expect file names in UTF-8, but not all file systems restrict the valid character sequences to be UTF-8 compliant. If you treat everything as if it must be UTF-8, your application may just break when it encounters a name that isn't UTF-8. (This detail gets messy because there are configurations of, eg, ext4 and ZFS that limit valid file names to be valid UTF-8.)
Beyond that, before the mid-2000s, it was common to use non-UTF-8 locales on Linux. So I'm sure I still have ISO-8859-1/—15 encoded file names somewhere, especially in archived data. They're not always trivial to rename either, because there might be references to them by name. (Or, in odd cases, you can't convert the name to UTF-8 because you hit a filename length limit, since UTF-8 is more bytes).
I believe wanting to access data from 20 years ago is a perfectly reasonable use case.
It's not so bad if a program can't display the file name right, as long as it doesn't crash with an exception or refuse to open the file. Unix file names have been defined as arbitrary sequences of octects except / and NUL for 30+ years.
On Linux, it’s not as rare as you might expect to end up with a few non-UTF–8 file names, though they’ll normally be on things you won’t often touch directly, so things like desktop software can reasonably only support UTF-8 paths.
I'm getting strong XKCD "Workflow"[0] vibes out of the idea that someone would have unintelligible paths on their system
I have quite a number of times. There's no one "thing"; it's usually some legacy app that write these names. And when you get a program that refuses to open any non-UTF16 compliant file name, you'll have a time of grief.
That works fine until it becomes literally impossible when your language/library/framework forces you into picking an encoding.
Even Microsoft's own APIs that add "UTF-8 support" choke when they encounter file names with invalid UTF-16 sequences.
You can't just dismiss the whole operating system API space just because NTFS doesn't enforce filename encoding. Yes, you might encounter file names with invalid UTF sequences because some app might be allowing them to be created. That can also happen in Linux too. Then add Linux, and probably others to the list. Then, what's the use for this list?
NTFS, on the other hand, has no dictated format for filename encoding. It just stores raw bytes as filenames, so anything can be a filename on NTFS, including invalid strings if the caller decides to do so. That's not on Windows either, otherwise, we should add Linux to the list too as ext4 and most other file systems also don't care about filename encoding.
Language is a very sensitive topic - in Central Asian countries, using Latin, Cyrillic or Perso-Arabic script for instance has very strong political connotations, same in the Balkans. The world is just like that
Besides, why do you care what funny symbols people use in discussions that don’t involve you?
You think they get it, but surprise happens when a database load fails when loading Chinese Character "string" into a field sized calculated based upon 2 bytes per character.
(And emoji are such a fine example - once they ate on the table, you need support for combining characters, characters outside of BMP, ligatures.. a large part of Unicode spec)
Like Apple trying to region-lock emojis, or whoever decided merging CJK together was a good idea, or that plan9 dev who scribbled out UTF8 on a napkin.
[1]: https://www.swift.org/blog/utf8-string/
There were many discussions on how to handle strings in the forums too. Remarkably, it is not possible to access "abc"[1] as it has unpredictable performance; instead one has to build the index and making this should make the developer realize the operation is costly. All in all most beginners in Swift hate working with strings because it’s not intuitive at first, but to be fair in almost all languages strings handling is not done properly.
Some of the users in Hong Kong discovered that you could import CSVs with Unicode text (e.g. index compositions with Chinese company names) and they'd display in the pivot table widgets and even be exportable to reports. But only most names. Some names were truncated or turned into garbage, and I was called upon to help debug this.
My first reaction was frank amazement that this "worked" at all: apparently, the path from the dumb CSV import code through to the Unicode-aware pivot table was sufficiently clean that much of the encoded text made it through OK. I can't remember the precise details now but I think the problem turned out to be embedded nulls from UTF-16 encoding and so was completely insoluble without a gut renovation of the platform.
So if you just move strings around and concatenate them, you don't even need to be aware of Unicode, even things like templating can work by just processing bytes. Of course, when you get to editing and rendering, you get all the complexity.
My favorite, similar to the issue of case-folding, is the idea of something being a "letter" and "lowercase" but NOT "lowercase letter."
Also a reason why I'm against Unicode identifier in source code. You got some source with characters like 'ö' in one variant and now your editor decides to use a different variant (e.g. compatibility character vs base character + combining decoration). And from my observation programming languages which claim to support Unicode tend to not make the effort to resolve those equivalents. At least for older versions of Go the specification clearly stated that they only compare Unicode code points. (don't know the current state)
Looks like the entire idea is to avoid using any ASCII/Extended ASCII control character bytes, except to represent those characters.
Not sure what the advantage of that was.
https://en.wikipedia.org/wiki/Spread_of_the_Latin_script#21s...
Any Indians here? I heard that most Hindi texting is done with a weird Latin character slang these days?
A reviewer for a scientific paper I wrote asked if I had considered the impact of UTF-9 on what I was doing. To this day I don't know if they were trolling me.
To start with, alternative representations for the same visibly identical character should have been excluded. In the base standard all supported characters should be precomposed - no modifiers. The article points out the difficulties in something as simple (and as security critical) as determining whether two strings refer to the same visibly identical characters.
A character set that does not make that trivial is not suitable for general use in system programming or for security critical identifiers (unfortunately). It massively complicates programming in many programming languages as well, and even UTF-32 is not sufficient to remedy the problem, as the article well notes.
The inability to handle and process any arbitrary string of bytes in a universal character set is a serious problem as well. The world is complicated, and the inability to pass through incorrectly coded data, alternatively coded data, and general binary data is a major limitation with serious consequences. System code such as device drivers or filesystems typically can't deal with inefficiencies or limitations like that.
In addition, there probably should be two types of uppercase and lowercase conversions, a simple, predictable one for system programming, and a more complex one that deals with considerations in languages that do not follow the normal rules.
String collation should be done at two levels as well, a simple code point level suitable for such things as prefix matches on database indexes, and a more complex variation for applications where linguistically sensitive collation is critical.
In general trying to solve all the higher end use cases in a large body of software that do not need to deal with them, should not need to deal with them, or cannot deal with them is impractical and has exacerbated the common string processing issues we see today. A lightweight standard - perhaps a subset or profile of Unicode that supported arbitrary binary data as opaque codepoints - could be helpful in a lot of contexts.
I was rather disappointed when I noticed my pretty Unicode icons would sometimes end up cut in half :-(
My stdlib will provide a (byte) Buffer class with basic low-level methods but I feel like iterating through it in fancy ways should be the concern of the user or 3rd-party libraries.
I fail to see this as part of a programming language.
Am I wrong here ?
Then you can have a nice, user-friendly string class for basic UTF-8 text, which is pretty easy. Ignore sorting and grapheme clusters (those probably belong in libraries, and they require fairly large tables). Consider providing a library function to iterate over UTF-8 "characters" (as unpacked 32 bit Unicode code points).
This is one of the sweet spots in language design, and it provides enough structure for third-party libraries to work well together, without everyone reinventing their own string type.
For another good alternative to this approach, see Ruby.
Yes.
Current headache: I have spent the last few days writing a parser for a data representation with far too much embedding. The innermost layer is JSON, representing glTF graphics data. That's UTF-8. That's encapsulated in an obsolete format called "Notation", which is sort of a pre-JSON kind of JSON. It's a byte stream, and has byte counted forms such as
s(5)Hello
Rust has the right concept. Strings are UTF-8, strictly enforced, so you can iterate over them. Byte arrays are arrays of 8-bit unsigned integers. Conversions check.
Classic headache: turning SMS texts into ITA-2 for an antique Teletype machine for steampunk demos. Anything outside of the ITA-2 set (uppercase letters, numbers, and some punctuation) was turned into the long text description of that code point. Emoji text were dutifully hammered out as
(SMILING FACE)(RED HEART)
Greetings, Professor Falken.
(SMILING FACE)(RED HEART)That’s how I did it: everything is utf8 (symbols and strings), but you can specifically indicate you want a byte array or nibble or word array etc. And they all work the same transparently.
If I can write "foo".contains('o') or similar in your language, but then you pretend you don't have strings, nobody will believe you and they're quite right to complain when, inevitably, lots of trickier stuff doesn't work because you didn't bother.
WUFFS doesn't have strings. But WUFFS isn't a general purpose language. Fortran has strings. C has strings (they're terrible but they're clearly strings).
By your logic, couldn't I pretend then to have strings insomuch as C when "foo".contains('o') can be transpiled by my lang as
strchr("foo", 'o') != NULL > handle at least some encodings > locale-sensitive casing and collating, and grapheme-cluster access (note that this depends on the font!)
Imagine how you would feel about a language that supports Arabic by default and needs special foo to work with American English?
You need to start thinking of characters as a type. Characters do not fit in bytes unless you’re American. And even if you’re American, people will still throw emdashes and Unicode quote symbols and emojis into your text. I’d suggest that if a language doesn’t have a proper character type, it’s not a useful language for the majority of the world.
UTF-8 is a reasonable compromise though.
My Buffer class lets you allocate bytes and it's up to your code to interpret those bytes.
I get your concern though and realize parsing
let foo = "<unicode>"
Let’s just say, no one in Greece, Iceland, or Korea is sitting around wondering if their new programming language should support their alphabet natively.
Imagine being from a country where a language’s built in Strcmp doesn’t work?
> Imagine being from a country where a language’s built in Strcmp doesn’t work?
Problem is it's seemingly a massive headache. Citing the article, "Unicode handling is always best left to a library".
I could bundle the recommended lib (ICU) into my lang's stdlib but it seems quite big...
EDIT: and most languages made before 1990 too
Try working with raw UTF-8 in JS and find yourself in a world of pain. Mathias Bynens talks about these gotchas here: