Hacker News Reader

Comment by m_mueller | Hacker News Reader

The way I understand it from their examples, it's rather latin-written-languages-centric, no? Could you give an example where it doesn't work with a romanized language? If not, then I'd hardly call that English-centric and practically useless worldwide.

View on HN

Luminoso cofounder here (we make ftfy, among other things). Our use case is fairly specific: a customer uploads text documents, often as a spreadsheet originally exported from someone else's tool, but doesn't think hard about the encodings involved, so in order to serve them well we have to fix whatever happened. Accordingly, we put the most effort into solving problems that happen for our English-centric, US-centric customer base; they're the most important problems for us (though Russian has also gotten some love, as you can see from the commit history). On top of that, making Asian language exports from other tools work correctly usually requires enough encoding-awareness to mitigate a bunch of the problem[1], so we see those problems less frequently.

That said, if you have examples where ftfy fails in any language, please submit them! We want this tool to work well, because anything that we can't fix will cause us to have egg on our faces with a customer someday...

[1]No peeking: how many format options in Excel's "Save As" dialog, excluding Excel formats, produce a document from which Unicode can be recreated accurately?

from __init__.guess_bytes(): "This is not a magic bullet. If the bytes are coming from some MySQL database with the "character set" set to ISO Elbonian, this won't figure it out. Perhaps more relevantly, this currently doesn't try East Asian encodings."

The world is a very large place, there are many codepages in use besides latin-1 and "ISO Elbonian". All central european contries use latin-2 (1250), or cyrillic codepage (1251). Since they are all single-byte codepages, they cannot be detected by try: convert() catch: try_another_codepage() and must be distinguished statistically. LTR/RTL language and asian encoding detection is even worse. https://en.wikipedia.org/wiki/Code_page https://en.wikipedia.org/wiki/Windows_code_pages

Another python Unicode conversion module which is slightly less US/English-centric: https://github.com/buriy/python-readability

I would like to emphasize that "guess_bytes" is not what ftfy is about. It's there for convenience, and because the command-line version can't work without it. But the main "fix_text" function isn't about guessing the encodings of bytes.

Not all text arrives in the form of unmarked bytes. HTTP gives you bytes marked with an encoding. JSON straight up gives you Unicode. Once you have that Unicode, you might notice problems with it like mojibake, and ftfy is designed for fixing that.

Like you say, encoding detection has to be done statistically. That's a great goal for another project (man, I wish chardet could do it), but once statistics get in there, it would be completely impossible to get a false positive rate as low as ftfy has.

It's not even latin-written-languages-centric. Something like a quarter of mojibake I find in the wild is in Cyrillic. ftfy does Cyrillic.

    >>> ftfy.fix_text("РґРѕСЂРѕРіРµ РР·-РїРѕРґ #С„СѓС‚Р±РѕР»")
    'дороге Из-под #футбол'

I've even fixed a bug that occurred in Ukrainian, based on automatic testing.

Reply on news.ycombinator.com