Comment by ff0066mote | Hacker News Reader

Comment by ff0066mote | Hacker News Reader

Parent Full thread

This article presents some interesting consequences of using a unified map from numbers to glyphs for all the plethora of human languages.

Perhaps instead of Unicode there could have been an established way to change character encoding mid-string? Regarding and treating control characters as normal characters, as this article demonstrates we do with the mirror character at least, is nonsensical. Maybe a string could have a primary encoding, and tags could be placed around sections which use an alternate.

Under such a scheme, in an english string any cryllic sections would be surrounded by "tag" bits. The viewing program could then handle the cryllic part according to its capability and known environment:

- An english terminal might render as nonsense english characters.

- A capable viewing program could render proper cryllic glyphs according to the cryllic encoding.

- A good browser could render the cryllic glyphs but highlight them, display them in a different color, or even remove them.

Likewise, right-to-left languages would be implicitly rendered as such according to the capability of the viewing program.

But this has no-doubt already been suggested and rejected by somebody...

It doesn't really buy you anything. Everything you described is already possible with UTF-8, even the first bit about misinterpreting UTF-8 as an 8-bit ASCII. You've still described a single mapping of characters to numbers, you've just created a much more inconvenient encoding.

One could argue that a filesystem which contains two files, one in ASCII and one in EBCDIC, is on the whole just single mapping of characters to numbers. Your point isn't well argued.

What I'm suggesting differs in that it would use consistent tags to indicate deviations from the intended primary language of a string.

Surround sections of a string which should be interpreted differently with tags-bits indicating such. These tags should not be treated as characters as the mirror character described in the article obviously is. And finally, display routines would easily be able to indicate what's what; any sections of the string with tags indicating an alternate encoding could be displayed with bold, highlight, or in a different color. (ie. As in "ti<cryllic>a</cryllic>n<cryllic>a</cryllic>nmen square", except the tags would simply be unused bit-patterns.)

To accomplish the same thing with a Unicode string, one would have to store a list of ranges of code-points which correspond to the string's locale. Then, anything which isn't in one of those ranges could be displayed in the alternate color. (ie. A character outside of the range 32-126 would be considered "not EN-US". For other languages, the code-point ranges might be more complex.)

The scheme I suggested isn't great, but it would avoids having to compare each character to see if it falls somewhere in a list of ranges for the current locale.

Reply on news.ycombinator.com