Hacker News Reader

Comment by RantyDave | Hacker News Reader

Oh I grok it just fine, it just sucks horribly.

I may be missing something "pythonic" but casting between byte strings and ascii strings - a null operation - is by far my greatest cause of runtime bugs.

To make it worse, the obvious cast is broken beyond all belief. What do we get from str(b'hello world')? Literally "b'hello world'" - a __repr__ of the object. Of course, adding an encoding changes the actual functionality of the cast ... str(b'hello world', 'ascii') gives 'hello world'. So, so broken.

View on HN

You are doing it wrong. Don't mix bytes and str. "Cast" is C lingo -- you don't "cast" completely distinct types like str and bytes.

(Note: Personally I would love to have str always represented internally in UTF-8. If it were like this, I would say "cast" was acceptable vocabulary. But it's not like this).

> str(b'hello world', 'ascii') gives 'hello world'. So, so broken.

https://docs.python.org/3/library/stdtypes.html#str

It's all documented and actually quite simple.

By the way, I never needed this. str() is just a convenience function, it's not meant for reliability. As you surely know there is complex interplay with __str__ and __repr__. I use str() for debugging, error messages etc. But not for anything that I expect to be able to process further.

How do you decode a bytes object to str object? Simple and intuitive, use .decode('utf-8')

> (Note: Personally I would love to have str always represented internally in UTF-8. If it were like this, I would say "cast" was acceptable vocabulary. But it's not like this).

UTF-8 is a good choice for transmission and storage, but since it's somewhat difficult to decode and isn't easily seekable it's not necessarily the best in-memory representation of a string.

Fortunately, that's, from a Python PoV, an implementation detail (even at the C API level).

> By the way, I never needed this. str() is just a convenience function, it's not meant for reliability. As you surely know there is complex interplay with __str__ and __repr__. I use str() for debugging, error message etc. But not for anything that I expect to be able to process further.

Or log messages. Broadly speaking str() means "tell the object to describe itself", while repr() means "tell the object to describe itself, possibly in a way that I can eval() it"

To add, I think UTF-8 is actually very easy to decode. It's true that indexing is expensive, so ideally different or additional data structures are needed for different use cases.

UTF-32 isn't easily seekable either because Unicode text has combining marks that shouldn't be separated from their base character. Almost every language has a "string" type that doesn't handle this correctly.

You don't grok it. Bytes are not ASCII strings. They are NUMBERS between 0 and 255. Strings are strings. DECODING bytes with ASCII encoding into TEXT is not "casting", it's DECODING.

A thing I've found to help people about that is to override __repr__ on bytestrings in python3 and bytes in python3.

If you make it not print strings as "ascii" but as hex, people grasp the difference easier.

I wouldn't say they are not ASCII strings, because they are if you squint.

Better say "bytes are not Unicode strings".

No, they are NOT strings, they are not any kind of strings. They are bytes:

    >>> b'asd'[0]
    97
    >>> b'asd'[1]
    115
    >>> b'asd'[2]
    100

They happen to contain an ASCII DECODABLE string in this case, but to get the text back, you NEED TO decode it first:

    >>> text = b'asd'.decode()
    >>> text[0]
    'a'
    >>> text[1]
    's'
    >>> text[2]
    'd'

Technically true. I'd inferred a different context on this one: By "strings" I meant the abstract concept in this case, not the python str which is the one meant.

Mate... the article was written for you. You don't 'grok' the difference between byte arrays and strings. Take a piece of paper and figure it out.

No, really, I know it. It's a pile of bytes until it has an encoding - and I agree with this whole heartedly. I think that getting involved with 'str' is the (my) problem here.

Reply on news.ycombinator.com