Hacker News Reader

Comment by josteink | Hacker News Reader

> One of the big ironies of all this is how well C has stood the test of time. C strings interoperate beautifully with UTF8

I wouldn't say that.

I'd rather claim that UTF-8 has been engineered to be ASCII pass-through compatible because of the lacking Unicode support in most C-codebases out there.

I mean... If you compare Unicode support in platforms and OSes where this was clearly a thought (Windows, C#, Java) to platforms with the more naive approach (Linux, C, PHP, etc), you will see a very clear picture of which side has the most unicode bugs and encoding errors.

And I say that as a Linux-guy.

C is terrible for text-processing. UTF-8 was designed as it was because of that.

Trying to paint C as the best choice here because someone found a working solution they could also apply to terrible C-code is clearly seeing things backwards.

View on HN

> If you compare Unicode support in platforms and OSes where this was clearly a thought (Windows, C#, Java)

Their insistence on UCS2 however gave us the worst outcome of all. We have forever lost all north of 23 bits for characters due to UTF16. The result of this misdesign creates bigger problem than C’s char type.

UCS-2 was all there was at the time, though. Blame Unicode for thinking 65,536 code points would be enough.

That's not a very good excuse though: UTF-8 was invented, by Thompson and Pike, specifically because existing solutions were not deemed adequate for their needs.

"Reasonable people adapt themselves to the world. Unreasonable people attempt to adapt the world to themselves. All progress, therefore, depends on unreasonable people"

When Java and friends were being designed, it looked like 16 bits would be adequate. They didn’t insist on UCS-2 over something better, they chose what looked like the best option at the time. It later turned out not to be. UTF-16 was created to fix that problem without having to completely redo their strings from scratch.

Likewise, C was created in a time when it looked like 7 or 8 bits would suffice. That turned out to be wrong. UTF-8 was invented to solve that problem without redoing C strings from scratch.

Windows is the platform where I see most of my Unicode problems. Here's an example of one of them:

hello.py:

    print("こ")

Ubuntu:

    $ python3 --version
    Python 3.5.2

    $ python3 hello.py
    こ

    $ python3 hello.py > out.txt
    $ cat out.txt
    こ

Windows:

    C:\Users\u\tmp>python --version
    Python 3.7.1

    C:\Users\u\tmp>python hello.py
    ?

    C:\Users\u\tmp>python hello.py > out.txt
    Traceback (most recent call last):
      File "hello.py", line 1, in <module>
        print("?")
      File "C:\Users\u\AppData\Local\Programs\Python\Python37-32\lib\encodings\
    cp1252.py", line 19, in encode
        return codecs.charmap_encode(input,self.errors,encoding_table)[0]
    UnicodeEncodeError: 'charmap' codec can't encode character '\u3053' in position
    0: character maps to <undefined>

The cmd console is terrible. It has a limited set of unicode code point, and worse, an even more limited set of fonts support.

The python unidecode pypi package is your best friend if you have to deal with it.

> The cmd console is terrible.

Agreed. And it's not really as much a native Windows app, as it is an emulator for the "good old" times of MS-DOS, which has received the necessary adjustments to keep chugging along, but little else.

Given that you probably have what is the single weakest link unicode-wise you can find anywhere in the entirety of the Windows universe.

It's really a completely obsoleted, abandoned mess. The fact that we had to wait until Windows 10 before you could actually resize the window freely should tell you at what level you should put your expectations.

If you want to live your life in the command-line on Windows (for python or whatever else), do yourself a favour and get any other terminal.

There's really quite a few to choose from. CMDer, ConEmu and MinTTY springs to mind.

What about this then? https://stackoverflow.com/a/47843552

Claims unicode support is decent in Windows 8 and up.

Your post clearly state that not only you have to manually setup your console on each session for each machine to get anything remotly workable, but you need to limit yourself to a subset of unicode.

Cmder is very decent.

This is because Python 3.5 and earlier used the old non-Unicode Windows API to access the console. This was fixed in Python 3.6:

https://docs.python.org/3/whatsnew/3.6.html#whatsnew36-pep52...

But he's using Python 3.7?

You are right that UTF-8 was created to pass safely through C code that isn't Unicode aware. But that turned out to be the right design everywhere. Every system that went with UTF-16 or UCS-2 is either carefully switching to UTF-8, or suffering from various encoding bugs and inefficiencies.

It is not true that UTF8 is the right design "everywhere". Perhaps it is the right design when you have strings that are stored and later read by various programs, but it's not necessarily right for in-memory strings.

Python 3 internally uses ASCII, UCS2 or UCS4 for its strings depending on which is most space efficient but still capable of representing the string. It can do that because Python strings are immutable* and because it is impossible for user code to see what the encoding is (to see the bytes of a string you must explicitly convert it, at which point you specify the destination encoding and Python ensures the translation is correct). If you join a UCS2 string with a UCS4 string then the result is automatically UCS4 and there's no way to tell from user code (except by memory usage)!

There's a good reason for this: indexing into a UTF8 string takes O(n) time because you must parse all the bytes before that point in order to count the number of characters. Iterating over a string by index would either take O(n^2) time or you would have to use some sort of awkward string iterator. If you do not provide any way to index by character (rather than byte position) then I would argue that your data structure is not really a "string". I suppose an alternative implementation would be to use UTF-8 but have an auxiliary data structure that maps between character indices and byte positions.

* Immutability matters because if you had a mutable string in ASCII and you changed one character to be a code point >127 then you'd need to copy the string before making the modification, which if possible at all would be O(n) for what you would expect to be O(1).

I agree with much of what you said here, but 'character', and therefore indexing by character, isn't a well behaved concept in the unicode world. Normalization helps somewhat, though it's another expense at string creation time. However, even after normalization there are plenty of graphemes that don't have a precomposed characters.

Our fundamental ideas about how a string should work and what kinds of things we might want to do with one are deeply influenced by the languages spoken by the pioneers of the computer age. Short of someone from a very different language tradition not being exposed to these implicit assumptions in their formative programming years I don't think we are ever going to know what a different path would look like.

> Normalization helps somewhat, though it's another expense at string creation time

And needs to be handled with care...there are edge cases where it can bite you. For example, if Unicode is being used internally to process data in a legacy CJK encoding, normalisation may lose distinctions that are needed for accurate round-trip conversion.

Another surprise "gotcha" is that simply concatenating two already-normalised strings may give you a result that is not normalised.

> Python 3 internally uses ASCII, UCS2 or UCS4 for its strings depending on which is most space efficient but still capable of representing the string.

UTF8 is as efficient as ASCII in the ASCII range, and more than or as efficient as UCS4 beyond the BMP.

The only iffy point is the U+0800 ~ U+FFFF range: Samaritan script to the end of the BMP, which mostly affects extremely contents-dense (mostly text, little to no markup) asian, and native scripts, as well as a few african ones.

> There's a good reason for this: indexing into a UTF8 string takes O(n) time

Here's the thing: You simply don't do that, ever. It's a meaningless operation. Most you would do is iterate over a string.

Bookmarks into a string are one such example of the need for this. No need to O(n) iterate over the string again when the bookmarking logic has already done so.

Computed indicies into strings are a no-go but that is far from the only use case.

But you can index UTF-8 strings with precomputed bookmark indices (byte offsets) just fine. Point is, to precompute them, you would surely have iterated the string before.

What I mean is that "extract the 42th codepoint/glyph/whatever from this UTF-8 string" is a pointless operation for free-form strings because in a free-form string the character position is meaningless.

(Basically all non-ASCII UTF-8 is pretty much a black box. You can't do serious computation with general Unicode because it's so complex and, as a consequence, ill-defined in practice).

This is an advanced trick, but I've successfully done this as part of a binary search to find the glyph at the visual mid-point of the string. UTF-8 is self-synchronizing so you can detect if you cut a code point in half. You still need to handle things like ligatures which even UTF-32 won't speak to directly.

Doesn't work well with BIDI languages, or Mongolian. Seriously if you haven't tested with Mongolian your code is probably wrong.

> Bookmarks into a string are one such example of the need for this. No need to O(n) iterate over the string again when the bookmarking logic has already done so.

You can do that just fine with UTF8, both Swift and Rust allow it.

While I agree that it's not really a measure of C standing the test of time, I kind of doubt that an alternate universe where 'C interop' / 'ASCII compatibility' weren't a design consideration, we'd have ended up with something much better (or TBH much different) than UTF-8.

> I mean... If you compare Unicode support in platforms and OSes where this was clearly a thought (Windows, C#, Java) to platforms with the more naive approach (Linux, C, PHP, etc), you will see a very clear picture of which side has the most unicode bugs and encoding errors.

so... windows ? unicode has always worked fine for me on linux, and has always been a complete pain in windows - as a software user and even worse as a developer. The only problem until ~ 6-7 years ago was libre unicode fonts with complete coverage but they are now pretty good.

Reply on news.ycombinator.com