Hacker News Reader

Comment by unicodedammit | Hacker News Reader

Very common problem in web scraping, for example forum site might contain a mix of MacRoman, Windows codepage and various european codepages in a single page (yes, even in 2014!). Seems like a more advanced version of UnicodeDammit module ( http://www.crummy.com/software/BeautifulSoup/bs4/doc/#unicod... ).

Note: this module, like UnicodeDammit, is very US/English-centric, and is practically useless for worldwide web scraping. For non-english pages, it is necessary to statistically estimate the codepage and language of each page segment, and then try to normalize each segment to unicode.

View on HN

Oh cool. I've used BeautifulSoup and UnicodeDammit, but I didn't know the "detwingle" function was in there. I'll take a look at if there's anything I can learn from its heuristic.

You should perhaps take a look at the "sloppy-windows-1252" codec in ftfy, and it may help "detwingle" handle some messier cases. (For example, Python will say 0x81 isn't a valid byte in Windows-1252. It's technically right. But there it is anyway.)

feedparser and the often-paired-with chardet library also approach this problem. The feedparser library has a series of fallbacks it uses to attempt to figure out the character encoding of a feed (finally bailing out and assuming windows-1252 if nothing else works). The chardet library is also quite good at guessing the intended encoding of a chunk of text, and will report its best guesses and confidence in them.

When you give chardet text in a 1-byte encoding, it sometimes ends up >99% confident that it's in ISO-8859-2.

Empirically, it's not in ISO-8859-2.

I think the problem here is that chardet is built on the assumption that "encoding detection is language detection" (from its docs). This assumption is necessary, and basically correct, when distinguishing Japanese encodings from Chinese encodings. It's even pretty much taken as a given that you can't have Japanese and Chinese text in the same document without contortions that most developers are unwilling to go through.

But European languages and encodings are much more intermixed than that. One document may contain multiple European languages, and these languages may be written outside of their traditional encoding.

I wouldn't know how to fix the European languages without damaging chardet's clear success at distinguishing East Asian encodings.

The way I understand it from their examples, it's rather latin-written-languages-centric, no? Could you give an example where it doesn't work with a romanized language? If not, then I'd hardly call that English-centric and practically useless worldwide.

from __init__.guess_bytes(): "This is not a magic bullet. If the bytes are coming from some MySQL database with the "character set" set to ISO Elbonian, this won't figure it out. Perhaps more relevantly, this currently doesn't try East Asian encodings."

The world is a very large place, there are many codepages in use besides latin-1 and "ISO Elbonian". All central european contries use latin-2 (1250), or cyrillic codepage (1251). Since they are all single-byte codepages, they cannot be detected by try: convert() catch: try_another_codepage() and must be distinguished statistically. LTR/RTL language and asian encoding detection is even worse. https://en.wikipedia.org/wiki/Code_page https://en.wikipedia.org/wiki/Windows_code_pages

Another python Unicode conversion module which is slightly less US/English-centric: https://github.com/buriy/python-readability

I would like to emphasize that "guess_bytes" is not what ftfy is about. It's there for convenience, and because the command-line version can't work without it. But the main "fix_text" function isn't about guessing the encodings of bytes.

Not all text arrives in the form of unmarked bytes. HTTP gives you bytes marked with an encoding. JSON straight up gives you Unicode. Once you have that Unicode, you might notice problems with it like mojibake, and ftfy is designed for fixing that.

Like you say, encoding detection has to be done statistically. That's a great goal for another project (man, I wish chardet could do it), but once statistics get in there, it would be completely impossible to get a false positive rate as low as ftfy has.

Luminoso cofounder here (we make ftfy, among other things). Our use case is fairly specific: a customer uploads text documents, often as a spreadsheet originally exported from someone else's tool, but doesn't think hard about the encodings involved, so in order to serve them well we have to fix whatever happened. Accordingly, we put the most effort into solving problems that happen for our English-centric, US-centric customer base; they're the most important problems for us (though Russian has also gotten some love, as you can see from the commit history). On top of that, making Asian language exports from other tools work correctly usually requires enough encoding-awareness to mitigate a bunch of the problem[1], so we see those problems less frequently.

That said, if you have examples where ftfy fails in any language, please submit them! We want this tool to work well, because anything that we can't fix will cause us to have egg on our faces with a customer someday...

[1]No peeking: how many format options in Excel's "Save As" dialog, excluding Excel formats, produce a document from which Unicode can be recreated accurately?

It's not even latin-written-languages-centric. Something like a quarter of mojibake I find in the wild is in Cyrillic. ftfy does Cyrillic.

    >>> ftfy.fix_text("РґРѕСЂРѕРіРµ РР·-РїРѕРґ #С„СѓС‚Р±РѕР»")
    'дороге Из-под #футбол'

I've even fixed a bug that occurred in Ukrainian, based on automatic testing.

Is that really necessary? Web browsers do no statistically analysis, and yet users seem to have no problem with them? Okay, certainly they have menus which allow you to override them, but they are relatively infrequently used. The most common issue with web scrapers that I've seen is they don't detect the character encoding the same way as web browsers do — which is what web authors expect.

http://www.whatwg.org/C#determining-the-character-encoding is what the spec defines, and though the eventual fallback is implementation-defined (Firefox, for example, combines locale-specific defaults with TLD-based defaults).

Reply on news.ycombinator.com