The revenge of Unicode | Hacker News Reader

The revenge of Unicode

eclecticlight.co

22 pointschmaynard18 comments

The author thinks that by "fooling" search engines and llms, using obfuscated codepoints is "Unicode's" revenge! Unicode is a truly gift to humanity, as it has so far encoded the majority of the worlds scripts, ancient or modern, brought order from the chaos of earlier encodings. The author forgets that humans write to communicate with other humans. If search engines are confused by such antics let it be.

Exactly. And to go back to Hungarians:[1] the alternative to having one large standard for all text is to have multiple encodings for different languages and peoples. And that’s what we did for decades before Unicode which was apparently a mess.

And of course there certainly are Hungarians that also write in Thai. So what about mixed Hungarian/Thai text? Should that be a multiple (interleaved) encoding text? I mean as an alternative to “149,813 ‘characters’” that “any one human is [not] likely to use [most of]”.

[1] https://news.ycombinator.com/item?id=39066135

I remember this for the Japanese language back in the early 2000s when I started learning the language. Nowadays Japanese websites mostly use UTF-8, but back in 2000 there was a mixture of JIS, Shift-JIS, and EUC content, and sometimes browsers had difficulties automatically determining the encoding, resulting in rendering “mojibake” until I had to manually specify the encoding.

Unicode is a great thing and has made computing more accessible to those who read and write in languages that don’t use the Latin alphabet.

Here is an example of Hungarian and Thai https://hu.wikipedia.org/wiki/Thai_%C3%ADr%C3%A1s

> Let me explain how you can get Unicode revenge without harming a soul.

Except, of course, anyone relying on a screen reader or other accessibility tool.

https://kence.org/2020/10/02/accessibility-of-unicode-as-sty...

If that's a concern, Unicode `Default_Ignorable_Code_Point` property [1] represents a set of characters that are suitable for steganography because they can be more or less safely ignored without affecting semantics. (And most of them even do not affect visuals, like higher variation selectors.)

[1] https://util.unicode.org/UnicodeJsps/list-unicodeset.jsp?a=[...:]

That style (using Unicode bold text and such) is so ugly too. Among my peers we’ve just used it as a gag.

10-15 years ago Unicode obfuscation was used by corrupt Russian bureaucrats to start online procurement auctions on a government platform, so that only an affiliated supplier could find them. E.g. if you sell cars, you monitor the platform for the specific keywords, but you find nothing, because the description contains the keyword with latin A instead of cyrillic А.

I wrote a small python library for searching obfuscated Unicode text by creating regex expressions; our company’s chat was being overrun by spammers.

https://github.com/wanderingstan/Confusables

From the readme:

E.g. "℮1೦" would match "Hello"

"Hello" gets turned into the following regex of character classes:

  [HＨℋℌℍΗⲎНᎻᕼꓧ𐋏ⱧҢĦӉӇ]  
  [e℮ｅℯⅇꬲеҽɇҿ]
   …

Edit: interesting that some versions of the letters are already being filtered by the commenting system here.

https://en.wikipedia.org/wiki/IDN_homograph_attack , https://en.wikipedia.org/wiki/Duplicate_characters_in_Unicod...

Kind of seems silly. How much of this is fixed by running the text through NFKD? That probably takes care of a good portion. There are many characters it probably doesn't but if people ever seriously used this, someone would easily make a DB of homoglyph characters. There probably already is one.

Not only such a DB exists, but is provided by Unicode itself. [1] There are also additional rules for internationalized domain names, because well, IDN would be an instant no-go without such measures.

[1] https://util.unicode.org/UnicodeJsps/confusables.jsp

> I sometimes wonder how many of its 149,813 ‘characters’ any one human is likely to use, and suspect for most that’s in the low hundreds or less.

Chinese sends it's regards.

But also text interchange is useful even if most people only use a fraction of it most of the time. How else is an English document supposed to quote Chinese characters for instance (this happens all the time on wikipedia pages which share the original name of something next to the translation).

The days before unicode were dark days, nowadays besides the ocassional oddity text pretty much just works without users thinking about it -- thanks to unicode.

> I sometimes wonder how many of its 149,813 ‘characters’ any one human is likely to use, and suspect for most that’s in the low hundreds or less.

In other news. How many Hungarians are ever going to be writing in Thai?

EDIT: Used the wrong quote here. Fixed.

> All those ‘characters’ enable deliberate misuse, where visual similarities are exploited to spoof people over identity or worse.

Communities like mathematicians are more to blame for that than Unicode.

Unless maybe you want a Latin script/Cyrilic/Greek etc. unification (for the relevant characters).

> We still do a great deal in life using text that can be searched rapidly and readily. Sometimes it pays to obfuscate that so that only humans reading it will understand what it says.

I guess that would work in the same way that rot13 encoding (like spoilers) does; not worth to decode since only a few specially interested people do it. But it’s not like it’s difficult if anyone puts some effort in. But yeah, such nerdisms can definitely be useful as long as they remain a subcultural thing (presumably always will be).

Now why that statement about mathematicians?

Maybe my history is rusty. Some of the weird bold (and even font like Gothic) came from mathematics because mathematicians use them in a semantically significant way (like Gothic G is a different signifier than just G; you can’t “plaintext” the former to G and still have it mean the same thing).[1] The general preference of Unicode is to relegate style-looking things to markup. That’s certainly their stance on superscript digits (the ones in Unicode are just for compatibility). But for some reason (again rusty history) these format-looking characters had to be included as separate code points.

The point is that Unicode has to deal with how text has been used by various communities in order to be the text encoding standard to rule them all.

[1] Unlike in prose where bold and such is just for emphasis. And at worst you can use informal markup (similar to MarkDown) and convey the same thing.

You can think of mathematical symbols as representing math objects. They are fairly new in unicode and yes different alphabets can be used. Like the symbol for infinity is from the Hebrew alphabet. Unicode gives them different codepoints so that screen readers and eventually one day renderer engines can typeset them correctly. Of course if you try to understand some of the more esoteric math subjects, is now mostly diagrammatic. I dont understand them to be honest!

  > Like the symbol for infinity is from the Hebrew alphabet.

Are you perhaps not confusing infinity with the symbol used to denote the cardinality of infinite sets? If the infinity symbol really comes from the Hebrew alphabet I'd love to hear more.

Reply on news.ycombinator.com