Hacker News Reader

Comment by pcr910303 | Hacker News Reader

As there are a lot of C-masters lurking in this thread:

How can one process unicode (UTF-8) properly in C? As a CJK person, I wish there was a robust solution. Are there any standardized ways or proposals? (Using wchar doesn't count.)

View on HN

Ignore all character support in the standard library and handle UTF-8 as opaque binary buffers. If you need complex string algorithms, decode into UCS-4 (UTF-32). You'll find short encoding and decoding functions on StackOverflow. For case-insensitive comparisons and sorting, use an external library that knows the latest Unicode standard.

Except that not all binary data is valid UTF-8 so you also need functions that check if a binary buffer is valid UTF-8.

The decoding phase will do that, if needed. Also note that in many cases you must process it as opaque binary, even though it should be valid UTF-8. This is in particular with filenames on POSIX systems because otherwise you could not access any files that happen to have invalid UTF-8 in their names.

UTF-8 encoding works "as is" based on byte strings (char[]). The latest versions of the draft standard provide somewhat more support.

I recommend heading toward a future where only UTF-8 encoding is used for multibyte characters and UCS-2 or similar for wchar_t. There is no need to support several different encodings.

Aaron Ballman even got a u8 character prefix added to C2x:

N2198 2018/01/02 Ballman, Adding the u8 character prefix

http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2198.pdf

UCS-2 is a bad choice -- it fails to represent most unicode characters. If you meant UTF-16, that's also a bad choice, because UTF-16 is also a variable width encoding, forcing programmers to use a some for of "extra-wide char".

I'm of the opinion that wchar_t should become an alias for char32_t.

Yes, I meant the 31-bit code point value (more than 16, anyway). It is the most useful width for doing things with wide characters.

UTF-32 is also a variable-width encoding; eg 00000044 00000308 aka "D̈".

I thought it was strictly one character per 32-bit code. Anyway, whatever it is called it is what wchar_t should be.

There are no fixed width encodings with range of encodable characters anywhere near that of Unicode.

It's too bad Unicode wasn't designed around the concept of easily-recognizable grapheme clusters and "write-only" [non-round-trip] forms that are normalized in various ways. A text layout engine shouldn't have to have detailed knowledge of rules that are constantly subject to change, but if there were a standard representation for a Unicode string where all grapheme clusters are marked and everything is listed in left-to-right order, and an OS function was available to convert a Unicode string into such a form, a text-layout using that OS routine would be able to accommodate future additions to the character set and and glyph-joining rules without having to know anything about them.

You can't do that without commiting to not supporting pathological text, otherwise you're stuck adding new special cases to the layout engine every update anyway.

I do have some ideas for a better encoding (like, I assume, anyone competent with sufficient free time and interest in text encoding), but there's a lot of reluctance to put effort into something that's already completely eclipsed by a technically inferior but not completely unusable alternative, so I've had it mostly shelved.

Check this: http://utf8everywhere.org/

Basically store the text as char arrays, and convert them when needed. Meanwhile, you could use this single file header: https://github.com/RandyGaul/cute_headers/blob/master/cute_u...

As a reviewer for Robert's upcoming C book “Effective C”, I thought that this aspect was better covered than in existing manuals for learning C.

However, the book only describes the available standard functions, so even doing better than other manuals, everything it has to say on this subject fits in one chapter and feel underpowered.

Your best bet is probably to use a library like ICU.

Here are examples of working with unicode in C: https://begriffs.com/posts/2019-05-23-unicode-icu.html

https://bitbucket.org/knight666/utf8rewind/src/default/

What sort of processing do you want to do?

Reply on news.ycombinator.com