Hacker News Reader

Comment by ianamartin | Hacker News Reader

I say this as someone who uses the latest version of Python available in every new project or script.

Text encoding issues are absolute garbage in Python 3.x

I fucking hate the way that csv module works with text encodings.

As soon as I can figure out a reliable way to take latin-1 and save it as UTF-8 without breaking everything, I will try to shoehorn in a PR.

Right now, it's fucking awful. My ETL pipeline hates it, I hate it, my boss hates it, and my internal constituents hate it. Because it sucks.

A file I can read in one encoding and write as another should be readable with the encoding I wrote it in. That is not currently the case with the latest version of Python.

And it makes me hate the world.

View on HN

    with open('some latin-1 file', 'rb) as f:
      text = f.read().decode('latin-1')
    with open('some utf8 file', 'wb') as f:
      f.write(text.encode('utf-8'))

Python 3's string encoding support is super good. I've said it before and I'll say it again: if you use bytes as a string you are Doing It Wrong.

If you use bytes as a string you are Doing It Wrong.

And what happens when you don't get a choice about what strings you are digesting?

I don't follow. What do you mean?

To be perfectly clear: bytes (b'') is not a string. Again: bytes is NOT a string. It is an array of octets, aka bytes, aka unsigned 8 bit integers. NOT characters. NOT a string.

If you are dealing with bytes that are encoded representations of a string, then you have to know what encoding they use to decode them and treat them as strings.

I'm not sure what you mean. If you don't know what the encoding of the input file is you have a problem. As far as I know there are libraries to guess the encoding, but it cannot be determined completely accurate.

Allow me to rephrase.

I do that operation on a file I get from an API. I know for a fact that the encoding I'm receiving is latin-1.

I run exactly that operation on the file that you wrote out in code.

When I try to read that file back in as UTF-8, I get encoding errors. That does not make for "super good." That makes me want to scream.

I do not have this problem when I use Python 2.7.x

You're definitely making a mistake somewhere, because I just tested it for myself and it worked perfectly fine. I made a latin-1 file, applied the above code with it, and got a correct utf-8 file out. Are you reading the final file back as latin-1? You have to read it as utf-8 of course.

I don't see the problem; reading and writing different encoding works fine. The CSV module makes no problems either:

  in_file = open("in.csv", 'r', encoding='latin-1')
  in_csv = csv.reader(in_file)
  out_file = open("out.csv", 'w', encoding='utf-8')
  out_csv = csv.writer(out_file)
  out_csv.writerows(row for row in in_csv)

I have a persistent problem that does almost exactly that.

The resulting file is not readable, and it makes me want to kick puppies and punch kittens.

Reply on news.ycombinator.com