Hacker News Reader

Comment by capitainenemo | Hacker News Reader

Article claims python 3 uses UTF-8.

https://stackoverflow.com/questions/1838170/ "In Python 3.3 and above, the internal representation of the string will depend on the string, and can be any of latin-1, UCS-2 or UCS-4, as described in PEP 393."

Article also says PHP has immutable strings. They are mutable, although often copied.

Article also claims majority of popular languages have immutable strings. As well as the ones listed there is also PHP and Rust (and C, but they did say C++ - and obviously Ruby since that's the subject of the article).

I'm also a bit surprised by the last sentence. "However, if you do measure a negative performance impact, there is no doubt you are measuring incorrectly." There must surely be programs doing a lot of string building or in-place modification that would benefit from non-frozen.

View on HN

> There must surely be programs doing a lot of string building or in-place modification that would benefit from non-frozen.

The point is that the magic comment (or the --enable-frozen-string-literal) only applies to literals. If you have some code using mutable strings to iteratively append to it, flipping that switch doesn't change that. It just means you'll have to explicitly create a mutable string. So it doesn't change the performance profile.

> can be any of latin-1, UCS-2 or UCS-4, as described in PEP 393

My bad, I haven't seriously used Python for over 15 years now, so I stand corrected (and will clarify the post).

My main point stands though, Python strings have an internal representation, but it's not exposed to the user like Ruby strings.

> Article also says PHP has immutable strings. They are mutable, although often copied.

Same. Thank you for the correction, I'll update the post.

Cool, although I feel if on one side you have Java, JavaScript, Python, Go and on the other Perl, PHP, C/C++, Ruby, Rust it's hard to say overwhelming majority in either direction.

Also someone below claims python byte arrays can be considered mutable strings, although I have no idea of the stringy ergonomics of that and whether it would be convenient to do - I try to avoid python too.

... and honestly, since java has both stringbuffer and string I feel it's really in the "has mutable" camp too

Python strings aren’t even proper Unicode strings. They’re sequences of code points rather than scalar values, meaning they can contain surrogates. This is incompatible with basically everything: UTF-* as used by sensible things, and unvalidated UTF-16 as used in the likes of JavaScript, Windows wide strings and Qt.

But isn't 'surrogateescape' supposed to address this? (no expert)

https://vstinner.github.io/pep-383.html

surrogateescape is something else altogether. It’s a hack to allow non-Unicode file names/environment variables/command line arguments in an otherwise-Unicode environment, by smuggling them through a part of the surrogate range (0x80 to 0xFF → U+DC80 to U+DCFF) which otherwise can’t occur (since it’s invalid Unicode). It’s a cunning hack that makes a lot of sense: they used a design error in one place (Python string representation) to cancel out a design error in another place (POSIX being late to the game on Unicode)!

It's not taking advantage of the weird way python strings work. You can put that hack on top of any string format that converts back and forth with unicode.

No you can’t: it only works if your string type is something other than a sequence of Unicode scalar values. In Rust, for example, strings must be valid UTF-8, so this hack is not possible.

Python strings are normally Unicode, but they are augmented with this mechanism to to smuggle other data as invalid surrogates.

Rust strings are normally Unicode, but Windows OSStrings are augmented with a similar mechanism to smuggle other data as invalid surrogates. (Rust smuggles 16 bit values as WTF-8 but it could do 8 bit smuggling instead with barely any change.)

Rust chooses not to make that the main string type, but it could. Any system based on Unicode can implement a hack like this to become a superset of Unicode.

Why do you think it can't? Rust would have to admit that the type is no longer exactly Unicode, just like python did. That's the opposite of a disqualifier, it's a pattern to follow.

Maybe you're unaware that [generalized] UTF-8 has a way to encode lone surrogates? They encode into 3 bytes just fine, either ED A_ __ or ED B_ __

With regards to what rust team is admitting or not... https://wtf-8.codeberg.page/#the-wtf-8-encoding "It is identical to generalized UTF-8, with the additional well-formedness constraint that a surrogate pair byte sequence is ill-formed. It is a strict subset of generalized UTF-8 and a strict superset of UTF-8."

https://wtf-8.codeberg.page/#intended-audience "WTF-8 is a hack intended to be used internally in self-contained systems with components that need to support potentially ill-formed UTF-16 for legacy reasons.

Any WTF-8 data must be converted to a Unicode encoding at the system’s boundary before being emitted. UTF-8 is recommended. WTF-8 must not be used to represent text in a file format or for transmission over the Internet."

They seem very transparent, and certainly are not proposing it as a general type.

> With regards to what rust team is admitting or not...

That wasn't an accusation. They admit things just fine. It was a hypothetical about using it as the main string type.

> and certainly are not proposing it as a general type.

1. Python's hack isn't used in file formats or transmissions either, as far as I know. It's also internal-only.

2. What they propose it for has zero relevance to my argument. It's merely proof that a hack like this can be added to ordinary Unicode representations. Python's goofy string representation is not enabling its surrogate hack.

In C, C++, and Rust, the question of "are strings in this language mutable or immutable?" isn't applicable, because those languages have transitive mutability qualifiers. So they only need a single string type, and whether you can mutate it or not depends on context. (C++ and Rust have multiple string types, but the differences among them aren't about mutability.) In languages without this feature, a given value is either always mutable or never mutable, and so it's necessary to pick one or the other for string literals.

Sure, that doesn't change the point that mutable strings are a thing in those languages. And I don't think C's const is really a "mutability qualifier" - certainly not a very effective one at any rate.

For the records, mutable strings, eh, bytearray objects, are also a thing in Python: https://docs.python.org/3/library/stdtypes.html#bytearray-ob...

Reply on news.ycombinator.com