Unicode Utilities: Confusables
util.unicode.org
util.unicode.org
https://github.com/wanderingstan/Confusables
For example, "𝓗℮𝐥1೦" matches "Hello".
"Hello" gets turned into the following regex of character classes:
[H\Ｈ\ℋ\ℌ\ℍ\𝐇\𝐻\𝑯\𝓗\𝕳\𝖧\𝗛\𝘏\𝙃\𝙷\Η\𝚮\𝛨\𝜢\𝝜\𝞖\Ⲏ\Н\Ꮋ\ᕼ\ꓧ\𐋏\Ⱨ\Ң\Ħ\Ӊ\Ӈ] [e\℮\ｅ\ℯ\ⅇ\𝐞\𝑒\𝒆\𝓮\𝔢\𝕖\𝖊\𝖾\𝗲\𝘦\𝙚\𝚎\ꬲ\е\ҽ\ɇ\ҿ] [l\\|\∣\⏽\￨1\\۱\𐌠\\𝟏\𝟙\𝟣\𝟭\𝟷I\Ｉ\Ⅰ\ℐ\ℑ\𝐈\𝐼\𝑰\𝓘\𝕀\𝕴\𝖨\𝗜\𝘐\𝙄\𝙸\Ɩ\ｌ\ⅼ\ℓ\𝐥\𝑙\𝒍\𝓁\𝓵\𝔩\𝕝\𝖑\𝗅\𝗹\𝘭\𝙡\𝚕\ǀ\Ι\𝚰\𝛪\𝜤\𝝞\𝞘\Ⲓ\І\Ӏ\\\\\\\\\ⵏ\ᛁ\ꓲ\𖼨\𐊊\𐌉\\\ł\ɭ\Ɨ\ƚ\ɫ\\\\\ŀ\Ŀ\ᒷ\🄂\⒈\\⒓\㏫\㋋\㍤\⒔\㏬\㍥\⒕\㏭\㍦\⒖\㏮\㍧\⒗\㏯\㍨\⒘\㏰\㍩\⒙\㏱\㍪\⒚\㏲\㍫\lj\IJ\‖\∥\Ⅱ\ǁ\\𐆙\⒒\Ⅲ\𐆘\㏪\㋊\㍣\Ю\⒑\㏩\㋉\㍢\ʪ\₶\Ⅳ\Ⅸ\ɮ\ʫ\㏠\㋀\㍙] [l\\|\∣\⏽\￨1\\۱\𐌠\\𝟏\𝟙\𝟣\𝟭\𝟷I\Ｉ\Ⅰ\ℐ\ℑ\𝐈\𝐼\𝑰\𝓘\𝕀\𝕴\𝖨\𝗜\𝘐\𝙄\𝙸\Ɩ\ｌ\ⅼ\ℓ\𝐥\𝑙\𝒍\𝓁\𝓵\𝔩\𝕝\𝖑\𝗅\𝗹\𝘭\𝙡\𝚕\ǀ\Ι\𝚰\𝛪\𝜤\𝝞\𝞘\Ⲓ\І\Ӏ\\\\\\\\\ⵏ\ᛁ\ꓲ\𖼨\𐊊\𐌉\\\ł\ɭ\Ɨ\ƚ\ɫ\\\\\ŀ\Ŀ\ᒷ\🄂\⒈\\⒓\㏫\㋋\㍤\⒔\㏬\㍥\⒕\㏭\㍦\⒖\㏮\㍧\⒗\㏯\㍨\⒘\㏰\㍩\⒙\㏱\㍪\⒚\㏲\㍫\lj\IJ\‖\∥\Ⅱ\ǁ\\𐆙\⒒\Ⅲ\𐆘\㏪\㋊\㍣\Ю\⒑\㏩\㋉\㍢\ʪ\₶\Ⅳ\Ⅸ\ɮ\ʫ\㏠\㋀\㍙] [o\ం\ಂ\ം\ං\०\੦\૦\௦\౦\೦\൦\๐\໐\၀\\۵\ｏ\ℴ\𝐨\𝑜\𝒐\𝓸\𝔬\𝕠\𝖔\𝗈\𝗼\𝘰\𝙤\𝚘\ᴏ\ᴑ\ꬽ\ο\𝛐\𝜊\𝝄\𝝾\𝞸\σ\𝛔\𝜎\𝝈\𝞂\𝞼\ⲟ\о\ჿ\օ\\\\\\\\\\\\\\\\\\\\\ഠ\ဝ\𐓪\𑣈\𑣗\𐐬\\ø\ꬾ\ɵ\ꝋ\ө\ѳ\ꮎ\ꮻ\ꭴ\\ơ\œ\ɶ\∞\ꝏ\ꚙ\ൟ\တ]
Commit: https://github.com/wanderingstan/Confusables/commit/c4121428...
I've been developing a normalization library for Ethereum Name Service: https://adraffy.github.io/ens-normalize.js/test/resolver.htm...
Seriously, it's 2022, we have better solutions than ASCII by now. And for what it's worthful, even in ASCII some chars like l, I and 1 are quite confusable.
> Internationalized domain names are stored in the Domain Name System (DNS) as ASCII strings using Punycode transcription.
Source: https://github.com/gc/confusables Demo: https://confusables.gc.codes/
https://unicode-org.github.io/icu-docs/apidoc/released/icu4j...
https://unicode-org.github.io/icu-docs/apidoc/released/icu4c...
PyICU (≥ 1.9) has bindings for this (class icu.SpoofChecker).
So you scroll through source code and hit this weirdly spaced line with a big "h" character and go-- oh-- that's some buggy crap in there.
Then maybe have a map for ranges that you want to include in your allowable set, and we're all good to go. :)
Whilst I was aware of the problem I wasn't aware that it's as big as it is. Those of us who use Latin scripts are reasonably familiar with the common ones such as o, O and 0 [l/c, u/c alpha & zero], but there's some tricky ones even in these Latin scripts that many of us get wrong.
As I discovered a while back - but I can't remember how - most of us get the World War Two abbreviation WWII wrong (myself included). The 'II' is not two alpha characters as we almost inevitability use but rather it should be the Unicode characters for Roman numerals. Even then I cannot remember if the correct transliteration for Arabic numeral 2 is supposed to to be Roman numeral '1' used twice/repeated or if Roman numeral '2' actually has its own Roman numerical glyph (I suspect the latter is correct). There are many more instances like this too, the dash, minus sign, en and em dash for instance.
Yes, I could look them up but it's a nuisance to do so on-the-fly and that's the whole point/trouble.
It seems to me we need much better proofing tools that would flag errors or potential errors. I reckon we've been very poorly served in this regard in that there no simple software tools available of the quality we need.
It's not only symbols or characters we need to correct but also typos that don't show up on spelling checkers such as for and fro and the big troublemakers it's and its. Spelling and grammar checkers should automatically highlight or flag such words whether their usage is correct or not.
FWIW, this example is really, really far down on the list of things you'll be going to typographic and/or semantic hell for. It doesn't impede communication, and I'd guess that using the Roman numeral glyphs would actually break stuff that assumes the usual/normal Latin representation.
From https://en.wikipedia.org/wiki/Roman_numerals#Unicode: "The Unicode standard, however, includes special Roman numeral code points for compatibility only, stating that '[f]or most purposes, it is preferable to compose the Roman numerals from sequences of the appropriate Latin letters.'"
> Potentially this is a big problem.
I wonder how far context can go to mitigate this in OCR applications?
> Unicode Utilities: Confusables
Totally agree, it's just that on occasions I've seen others raise the matter as an excellent example of how complex Unicode can be. Using a well-known, well used example clearly illustrates the point.
However, the matter of data integrity and machine learning is another matter altogether as meanings can easily be misinterpreted if wrong characters are used. One of the biggest problems of course is the transliterations of scripts both ancient and modern. This is a serious issue for historians and archivists and it's the reason for why Unicode makes such distinctions.
Considering these serious limitations, it's little wonder why OCR is not used as much as it ought to be. Take any text PDF on the Internet Archive that's derived from an image scan of a book and you'll find the error rate to be enormous. Clearly, these errors are why the IA offers books in both text and 'raw' image forms. Similarly, many newspaper publishers have usergroups of readers who correct scans of historical newspapers.
Obviously, this is very time consuming and somewhat defeats the purpose of OCR, it still however is a more accurate and much faster process than retyping the text.
The only real longterm solution is AI/machine learning and even then it won't be perfect. Statistics (hit/error rates for any given text) will come into play. Texts that are converted with, say, a deemed accuracy of 99.99% [my guesstimate] would be exempt from further correction but still marked with '99.99%' in the attached metadata - here the metadata is everything. Others will be flagged for correction with the potential errors flagged.
This needs new conversion format rules for OCR, as I see it, both metadata and the original scans should always be embedded within the 'converted' file. As far as I'm aware whilst work has progressed along these lines we're still a long way from a practical working solution.
But I doubt that the Romans had any clearer of a sense than you of whether "II" was one letter or two. It probably varied across eras and regions, and most people probably didn't think one way or another about it. It's probably not such a big deal to write II when you meant U+2161.
However, I definitely do agree that we are badly lacking in good software tools to help us enrich the set of letters that we can easily use. Imagine if you could type "II" and your text editor could offer to replace it with "Ⅱ".
On the other hand, font support is a limiting factor. For most purposes, II should looks similar to Ⅱ. But in our current world, you are at the mercy of whatever fonts that happen to implement Ⅱ, which might be totally different from how II looks.
FYI, I've somewhat expanded on my earlier comment in reply to CharlesW.
MS Word has had this for decades.
We know the Romans were influenced by the Greek, and the Greek by the Minoan.
Both the Greek and Minoan had their own numerals:
https://en.wikipedia.org/wiki/Aegean_numerals https://en.wikipedia.org/wiki/Greek_numerals
Before the Romans, the Greeks were already using phonetic letters as numerals. They did saw Iota as either a number or a phoneme depending on the context.
Looking at these older sytems, it looks like the romans just mixed both approaches, either on purpose or not. (I have absolutely no evidence to support this, it's a guess).
Maybe the problem is really older than we think?
An example would be the “Roman numerals” set that are discussed elsewhere in this HN discussion. Others include “ghost characters” and obsolete keitai emojis from the 90s that only appear in obsolete character tables. Mostly these should be normalised to modern / canonical versions.
Too many raw items to process.
Any good guesses what this is about?
EDIT: Got it, the number of strings confusable with the input.