Indeed, the Python 3 Unicode string object is fascinatingly clever. Code worth reading:
https://github.com/python/cpython/blob/master/Objects/unicod...
https://github.com/python/cpython/blob/master/Objects/unicod...
IMNSHO, most modern languages should be storing strings as UTF-8 and give up on random access by characters. You almost never need it; in the most frequent case where you do (using indexOf or equivalent to search for a substring, and then breaking on it), you can solve the problem by returning a type-safe iterator or index object that contains a byte offset under the hood, and then slicing on that. Go, Rust, and Swift have all gone this route.
That's not true as of 3.3. It will store the compact representation, which in most cases is ASCII, which is UTF-8 compatible. https://www.python.org/dev/peps/pep-0393/
So unless you go outside of UCS-1, you don't need to reencode anything.
This has nothing to do with (inefficient) encodings, and their bad historic namings, probably derived from perl.
Of course any name like byte2utf8 or just to_byte or b2u8 would have been better than encode/decode.
And of course cache size matters nowadays much more than immediate substring access for utf8, so nobody should use ucs-2 or even ucs-4 internally. This is easily benchmarkable.
Clever is relative. The Python 3 unicode string objects makes very little sense given real world situations. It wastes enormous amount of memory in many common scenarios for no reason at all other than to facilitate O(1) access to charpoints which is completely inappropriate for text processing anyways.
Worse than that: it can blow up by another 50% the moment someone encodings it to utf-8 as the encoded version is cached around.
The object is too clever for its own good.
>>> import sys
>>> def charsize(x):
... return sys.getsizeof(x + x) - sys.getsizeof(x)
...
>>> charsize(b'A')
1
>>> charsize('A')
1
>>> charsize('\xff')
1
>>> charsize('\uffff')
2
>>> charsize('\U0010FFFF')
4Except it does not help. When you render out a template it starts out as latin1, then the first time it hits a unicode char the whole thing re-encodes into ucs2 and then when it sees an emoji it re-encodes to ucs4. In practical terms you will see UCS2 and UCS4 all the time.
Only a few standard member functions of string (find, rfind, index, rindex) return integer subscripts into a string. Those could return an opaque index object which has an index and a link to the string. Such objects could be used for string operations which need a string index. If an index object is forcibly converted to an integer, the array of indices has to be generated, but otherwise, it is unnecessary. You could even allow adding or subtracting integers from an index object, which would walk the UTF-8 string forwards or backwards as indicated.
This would make UTF-8 strings usable with subscript and slice notation without exploding them to wide characters.