Why Python 3 Exists
snarky.ca
snarky.ca
I think there would have been much less of a problem if encode and decode were far more obvious, unambiguous and intuitive to use. Probably without there being two functions.
Still a problem of course today.
* unicode -> encode, u -> e (vowels)
* str -> decode, s -> d (consonants)
There is one obvious way: strings are encoded to bytes, bytes are decoded to strings.
I think that reveals that the names really do have a problem. The problem is that "encode" sounds like "make this Unicode" to people who aren't familiar with Unicode.
Bytes hold a bunch of data in some encoding. It could be an image, UTF-8 or LZMA compressed ASCII. Once you know the encoding, to reconstruct the data you decode into a semantically meaningful form.
To put it another way, imagine the terms were "serialize" and "deserialize". Of course one serializes to and deserializes from binary data. Just replace "{,de}serialize" with "{en,de}code" and you're done.
Veedrac had a good analogy, think of text as something abstract, for example imagine text is an image or sound, if you want to store it in bytes you need to encode it, and to read back you decode it.
As to_bytes/from_bytes, actually python provides it too:
to_bytes -> bytes(<text>)
from_bytes -> str(<bytes>)
EDIT: This last line of the article pretty much sums it up: "We structured the transition thinking the community would come along with us in leaving Python 2 behind, but that turned out not to be the case and instead we have taken some more time and are using a Python 2/3 compatible subset of the language to manage the transition."
I wonder if python 3.6 or 3.7 will acquiesce to this and give an option for something like "from __past__ import str" like we have for __future__ in 2.7 to make these backward incompatibilities easier to deal with.
For example
old_function(bytes(myarg))
if it needs to be a literal:
old_function(b"literal")
That's what I do, and works for me so far.
As the past did not contain the __past__ that is not going to work.
Here's how you remember it: "Unicode" is not an encoding. It never was, it never will be. Of course, the data must be encoded in memory somehow, but in Python 3, you cannot be sure what encoding that is because it's not really exposed to the user. From what I understand, there are different encodings that string objects will use, transparently, in order to save memory!
You always "encode" something into bytes, and "decode" bytes back into something. There should be exactly two functions, because the functions have different types: "encode" is str -> bytes, "decode" is bytes -> str. Explicit is better than implicit.
output = input.decode(self.coding)
In [1]: len("नि")
Out[1]: 2
In [2]: for c in "नि":
...: print("Character {}".format(c))
...:
Character न
Character ि[1] Sadly surrogate escapes break this. Rust's division into String:OsString:Vec<u8> makes that aspect a lot cleaner.
Strings have to be arrays. It is inevitable. We are on Von Neumann machines with L1 cache lines that are something like 64 bytes wide, so making strings into something other than arrays is a complete non-starter. Because strings are arrays, we use array indexes to slice strings. The indexes are integers, because that just makes sense on a Von Neumann machine.
So your complaint is that you can slice a string in "bad" ways. What makes it bad? You're trying to prevent people from slicing up grapheme clusters, which is a noble goal. But in practice, we get the indexes from methods like str.find() or re.match(), and we treat them as more or less opaque, so we don't end up slicing grapheme clusters very often in practice. Grapheme cluster slicing requires a big Unicode table in memory anyway, so in the rare case that you need it, you can pay the high performance cost for using it. In the meantime, formats like JSON and XML are defined in terms of code points, so using code point arrays eliminates a class of bugs where you could accidentally make malformed XML or JSON, which would then get completely REJECTED by the receiving side, causing your Jabber client to quit or your web browser to show a bunch of mojibake.
And let me ask you this: what do I get when I write:
x = "a"
y = "\u0301"
Please, tell me what you actually think is correct behavior. It is far, far more useful than pointing out that something is "wrong" on purely semantic arguments.
https://www.mikeash.com/pyblog/friday-qa-2015-11-06-why-is-s...
What you can drop is the idea of the index being an integer with any particular semantic meaning. Rust uses byte indexes with an assertion that the index lies on a codepoint boundary, for instance.
Outside of the ASCII range? I doubt it. That most likely means you are doing something fundamentally wrong.
EDIT: Or a JSON parser, etc.
It seems like this whole mess comes from conflating bytes and characters. They are not the same, any more than integers and booleans are the same just because you can jnz in most assembly languages.
Why on earth not? Any slice of a string (with a non-pathological internal encoding) with ends on codepoint boundaries is itself a valid string. Losing that information sounds like a whole lot of pointless hassle and potential for harm. It also forces the internal string encoding to be part of the interface, since taking a slice of a string and getting a byte array requires knowing how the string is encoded to use it, but getting another string means that such details are private.
It's true that they're byte offsets, but that doesn't matter. The parser doesn't care whether it's on the 10543th grapheme or the 10597th. It just wants to know where its stuff is.
I mean, if one has a hashmap
hash = {"foo": 1, "bar": 2}
There are plenty of uses for getting the length of a string where you will not be running into combining characters, clusters or any other things which would require special handling.
(remember that on the internet, all the data you receive is initially in string form, and very often we want to receive data representing fixed-length values such as phone numbers or credit card numbers, so "you're wrong for wanting to know the length" is both a non-starter and, well, just plain wrong)
How do you know that? Any user supplied text may contain those things.
>remember that on the internet, all the data you receive is initially in string form
It's not. It's bytes.
>very often we want to receive data representing fixed-length values such as phone numbers or credit card numbers
This is a good example of why strings shouldn't have a length property - people don't understand what length actually does in most languages. If you think that a len() function can verify that the contents of a text field is in the format of a CC number then you don't understand the len function. It can't do what you are asking. See the examples given in the GP (len("\u0301") for example).
The fact that the people who advocate for length functions want to use them for things that they are broken for is exactly why length functions shouldn't exist.
Which is why, in addition to things like length checks, we use other checks. But length is a quick and easy check for many types of values, and should be supported.
I've been down in the depths of Unicode behavior a time or two. I know what lurks down there, and I know from experience that your "strings should never allow these operations" stance is just as dumb as the more-common "I don't know what Unicode really is" stance. Can length checks and slicing and other operations run into trouble in some corners of Unicode? Sure. But if "this might cause trouble" were grounds for forbidding programmers ever to do something, we wouldn't have computers, any useful software at all, the internet or most of the other things we like and take for granted despite the fact that we occasionally have to fix bugs in it.
Now, get down off your snobby horse and join the rest of the real world, where we know that something might not be guaranteed reliable in all logically-possible cases, but find ways to work with it anyway because in 99.99% (or more) of the cases our code deals with it's good enough.
You say "wrong" as if that's a quote from me. But it's not. Yes, strings have to be arrays, but how they work depends on the semantics of your language.
I'm not entirely happy with Python 3 because I don't think they really took their time to improve on the warts. I doubt there's a one-size-fits-all solution.
But that's a choice and not the only one. Sure you're likely to implement your Unicode string as an array of some type. But that doesn't mean that the only sensible approach is to expose that array directly to the user. Or more specifically as the primary interface.
For my money, Swift has probably the most comprehensive Unicode string API I've seen [0]. What they do is, essentially, have an opaque String type but support various "views" as properties. The main view is called .characters and represents a collection of grapheme clusters. They also have properties that present Unicode scalars, utf8 encoding etc.
Their API is complex, no question. But then proper handling of Unicode is complex. But it does show that there are other options than simply exposing the Unicode scalars.
BTW to your point on len(x) + len(y). The answer is 2 if you define len() on Unicode scalars but 1 if you define it on characters. Why? because len("\u301") should be 0. It's not a character, it's a base modifier. It is, of course, true that getting this right is likely to be substantially more expensive than getting it wrong but that doesn't mean it can't be done.
[0] https://developer.apple.com/library/ios/documentation/Swift/...
You learnt ASCII. Then you learnt about escaping to get around limitations in ASCII. Then you learnt that Unicode got around non-Latin issues with ASCII.
Each step has added cognitive load. It seems surprising to me that the next step wouldn't.
I'm not saying that the string API in Python is perfection, but it had a specific role to fill in the larger Python ecosystem, and it does that very well.
Anyway... I would say that len(x) == 0 should be the same as saying that x is empty. According to the Unicode standard, the number of grapheme clusters in "\u0301" is 1, anyway! And on most systems, the string will display as an acute accent in its own space, and you can place the cursor before or after it.
Not like the set of valid "cursor positions" in a string isn't a locale-dependent value, anyway. Grapheme clusters can be thought of to be locale-independent but they are NOT the same as cursor positions.
If I recall correctly the modifier on it's own isn't correct. There is a base that should go before it to produce a valid character. I don't recall the base unfortunately and I don't have the spec to hand.
I mean this from the standard: "Nonspacing combining marks used by the Unicode Standard may be exhibited in apparent isolation by applying them to U+00A0 no-break space. "
You can have valid Unicode that doesn't do this. But it's not intended to represent the character as some kind of textual thing. It's intended so that you can build up Unicode strings sensibly. Same idea as allowing mis-matched surrogates.
Perl 6 has long specified an interesting and rich approach to Unicode. Have you explored it?
(An initial implementation for an initial release of the Rakudo compiler was worked on over the last few years and declared done, modulo bugs, a few weeks ago. A recent blog post provides a friendly overview.[1].)
One notable difference is that Swift adopts an iteration-only view of strings for dealing with characters whereas Perl 6 does not. One consequence is that indexing operations in Perl 6 are O(1) (time) at the expense of potentially (and for the current implementation, typically) greater use of RAM.
> BTW to your point on len(x) + len(y). The answer is 2 if you define len() on Unicode scalars but 1 if you define it on characters. Why? because len("\u301") should be 0.
Are you 100% sure of this of your last sentence ("should be 0")? I get that it's intuitively reasonable that an isolated base modifier be length 0 but I thought the Unicode standard specified (or at least recommended) that it be 1, and I note that Perl 6 treats isolated base modifiers (or at least this particular one) as length 1:
> ("a").chars 1
> (0x301.chr).chars 1
> ("a" ~ 0x301.chr).chars 1
So both Swift and Perl 6 correctly report the length of "a" as 1 and an "a" concatenated with "\u301" as 1 but disagree on what to do with an isolated base modifier.
> It is, of course, true that getting this right is likely to be substantially more expensive than getting it wrong but that doesn't mean it can't be done.
Indeed.
[1] https://perl6advent.wordpress.com/2015/12/07/day-7-unicode-p...
String searching & matching return an opaque String.Index type. I assume that under the hood they do Boyer-Moore on bytes and String.Index is a byte offset, but the important thing is that String.Index values are not convertible to integers, and so you never run into the case where a user passes in a byte offset that would slice a grapheme in half. Instead, String.Index has properties for the next and previous index and a method to advance by an arbitrary amount, so you'd access the 7th character after a dash in a string as myString[myString.rangeOfString("-").advanceBy(7)]
Swift gives up on the idea that len(x+y) = len(x) + len(y). This is just something you have to remember; in UI programming, however, it makes a lot of sense because adding an accent to an existing string isn't going to make it take up more width in the text box. (x + y).utf8.count == x.utf8.count + y.utf8.count, however.
https://developer.apple.com/library/ios/documentation/Swift/...
If I were to adapt this to a domain like Python's, where string processing is pretty important, I'd allow indexing & slicing by integers (including negative integers), but I'd define them in terms of String.Index types, which iterate over extended graphemes under the hood:
str[2] == str[str.start.advanceBy(2)]
str[-3] == str[str.end.advanceBy(-3)]
str[str.find('foo') + 3] == str[str.find('foo').advanceBy(3)]
str[0:-3] == str.substring(str.start, str.end.advanceBy(-3))
del str[0:3] == str.removeRange(str.start, str.start.advanceBy(3))So do fullwidth characters have a length of 2?
If not, how is this useful for finding the size of text in a text box anyway?
(0) Dependent types: The type of valid indices into a string depends, well, on the string. It's this type, not `uint`, that should be used by functions like `str.find()` and `re.match()`. This makes it a compile error to, say, slice a string with an index to another string. Internally, or course, all indices are represented as `uint`s, but your program has no business knowing this. It's an abstract type. IMO, this solution is particularly useful for systems languages, where the increased flexibility and fine-grained control are worth the increase in complexity.
(1) Coalgebras: `str.find()` shouldn't return an index. Instead, it should return in O(1) time, and allocating O(1) additional space, the left and right slices of the original string, split at the desired index. (If the string can be split in multiple places, `str.find()` should return an iterator whose element type is such left-right pairs.) For this to work, it's fundamental that the left and right slices not be allocated in their own buffers, but instead share the original string's buffer. IMO, this solution is best for high-level languages, where simplifying the API is well worth a little loss in flexibility.
Neither of these APIs allows the programmer to split strings at the wrong places.
Perhaps this is because Unicode is an exhaustive standard. :)
Seriously, I hear ya.
> every argument had the same hole that I see in your post: What, exactly, is a "character"? Please answer me that
I'd say the answer varies depending on who's talking, what they're talking about, and who's listening.
If it's a programmer interested in listening to the Unicode consortium and interested in what Unicode.org documents define for "what a user thinks of as a character" then the answer is, according to those documents, a "grapheme".
> then you can write a proposal for how the API should work, exactly
Right.
It can get, and has indeed gotten, better than that; one can write specifications, build reference implementations, and try things out in battle for a few years. Several text processing systems (including some programming languages) have gone this route and their results should be taken in to account. (ICU is perhaps the go to reference implementation.)
> and why this works with Indic and German and Korean and everything else.
For designs and implementations of text processing systems that claim an aspiration of progressing toward fully following the Unicode specification, the simplest answer to "why does it work?" (or "why it is expected to work") with Indic and German and Korean and everything else is of course the Unicode standard itself.
For those of us who aren't blessed with the patience of a saint and thus haven't read the entire spec from start to finish a few times, one has to rely to a degree on those who do, even if it's really hard to follow what the heck they're saying.
> You're trying to prevent people from slicing up grapheme clusters, which is a noble goal.
Given that grapheme clusters correspond to what the Unicode standard specifies for identifying and preserving the integrity of "what a user thinks of as a character", it seems less a noble goal than a fundamental eventual goal for any text processing system that explicitly claims to aspire to support the world's human languages via Unicode compliance.
> we don't end up slicing grapheme clusters very often in practice.
Which "we" are you referring to?
The western world has dominated text processing to date does. This must not mean that "we", by which I mean all humans, would best continue practices that marginalize those with native languages not well served. (By ignoring the central importance of "what a user thinks of as a character"; again, I'm quoting Unicode consortium documentation.)
Or, more parochially, what about the "we" that's just programmers who want to be able to increase their confidence that they are properly processing Unicode compliant text?
> In the meantime, formats like JSON and XML are defined in terms of code points ...
Code points are the middle level of Unicode, sandwiched between byte encodings at the bottom level and graphemes at the top. There's nothing wrong with a text processing system stopping at the middle level provided folk don't lose sight of the fact that it stops at the middle level.
Thinking that code points are "what a user thinks of as a character" mostly works out OK when processing text that's English or the like, but is definitively not Unicode compliant to the degree it's used too broadly.
> Can you tell me what the correct behavior is?
I would suggest that the most appropriate authority is the Unicode consortium and its Unicode standard.
> Might you end up with bugs in programs because len(x + y) != len(x) + len(y)?
Oh for sure. But it's part of the Unicode standard. It's one of several apparently strange discontinuities that the Unicode effort introduced last century in an attempt to deal with the combination of human language reality and computer system realities. They may have chosen the wrong sweetspot. But now Unicode is a significant part of reality. I, for one, don't think it's wise or even realistic to abandon the Unicode standard as it is...
https://github.com/python/cpython/blob/master/Objects/unicod...
Clever is relative. The Python 3 unicode string objects makes very little sense given real world situations. It wastes enormous amount of memory in many common scenarios for no reason at all other than to facilitate O(1) access to charpoints which is completely inappropriate for text processing anyways.
Worse than that: it can blow up by another 50% the moment someone encodings it to utf-8 as the encoded version is cached around.
The object is too clever for its own good.
>>> import sys
>>> def charsize(x):
... return sys.getsizeof(x + x) - sys.getsizeof(x)
...
>>> charsize(b'A')
1
>>> charsize('A')
1
>>> charsize('\xff')
1
>>> charsize('\uffff')
2
>>> charsize('\U0010FFFF')
4Except it does not help. When you render out a template it starts out as latin1, then the first time it hits a unicode char the whole thing re-encodes into ucs2 and then when it sees an emoji it re-encodes to ucs4. In practical terms you will see UCS2 and UCS4 all the time.
IMNSHO, most modern languages should be storing strings as UTF-8 and give up on random access by characters. You almost never need it; in the most frequent case where you do (using indexOf or equivalent to search for a substring, and then breaking on it), you can solve the problem by returning a type-safe iterator or index object that contains a byte offset under the hood, and then slicing on that. Go, Rust, and Swift have all gone this route.
This has nothing to do with (inefficient) encodings, and their bad historic namings, probably derived from perl.
Of course any name like byte2utf8 or just to_byte or b2u8 would have been better than encode/decode.
And of course cache size matters nowadays much more than immediate substring access for utf8, so nobody should use ucs-2 or even ucs-4 internally. This is easily benchmarkable.
That's not true as of 3.3. It will store the compact representation, which in most cases is ASCII, which is UTF-8 compatible. https://www.python.org/dev/peps/pep-0393/
So unless you go outside of UCS-1, you don't need to reencode anything.
Only a few standard member functions of string (find, rfind, index, rindex) return integer subscripts into a string. Those could return an opaque index object which has an index and a link to the string. Such objects could be used for string operations which need a string index. If an index object is forcibly converted to an integer, the array of indices has to be generated, but otherwise, it is unnecessary. You could even allow adding or subtracting integers from an index object, which would walk the UTF-8 string forwards or backwards as indicated.
This would make UTF-8 strings usable with subscript and slice notation without exploding them to wide characters.
I'm not sure "encoding" tells us that the output is bytes. Surely information can be encoded in other ways. Names like .to_bytes() and .from_bytes() might have saved a lot of trips to the documentation.
Information - yes. Strings... I'm not sure. What other representations are you thinking about?
When save some "things" I read from utf16 file and want to save them to Postgresql who's client encoding is set to utf8, what do I do?
It's important to point out that "Unicode" in Windows is an encoding. It means specifically UTF-16 LE, many people like me learned the concept "Unicode" on Windows got confused to hell when learning Python.
They are not. What is unintuitive is the default encoding in Python where an encode can trigger an implicit encode and the other way round. The `encode()`/`decode()` availability of strings was never a problem has you have many bytes -> bytes and str -> str codecs;
But then function would do something completely different. "\x01\x02".encode('zlib') for instance is a bytes to bytes operation. The problem is that "foo".encode('utf-8') does not give you an exception. If the coercion would not be enabled you would get an error:
>>> reload(sys).setdefaultencoding('undefined')
>>> 'foo'.encode('utf-8')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/Cellar/python/.../encodings/undefined.py", line 22, in decode
raise UnicodeError("undefined encoding")
UnicodeError: undefined encoding
>>> b'foo'.encode('utf-8')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'bytes' object has no attribute 'encode'I think the inclusion of things like zlib (or rot13 or whatever) was a conceptual error that just fosters confusion.
We should not optimize languages for idiots. There is nothing confusing about such an operation for anyone who can use their brain. Python 3 still contains those operations but instead of x.encode(y) you now do encode(x, y).
Of course this is a real thing and it's very frequently used. Yes I am in favour of it as the codec system in Python is precisely the place where things like this should live.
>>> "foo".encode('zlib')
LookupError: 'zlib' is not a text encoding; use codecs.encode() to handle arbitrary codecsConversely, you can decode the byte message into something readable, ie. your Unicode string.
I'll take an occasional 2 seconds in the REPL over the headache of debugging codec issues any day of the week.
The confusions is around people viewing byte strings implicitly as ascii codepoints, and not fully understanding what the things they're looking at actually are.
output = ''.join(str(x) for x in some_list)
That's maybe because JSON is by the standard supposed to be utf-8, always.
The point of the comment is that everyone knows that you json encode native entities to the utf8 representation, and json decode from utf8 to native entities.
The same is true of unicode. It's not ambiguous at all.
Only utf-8 is a very specific encoding, so converting to it is straightforward, whereas unicode is an umbrella term and the most ambiguous thing you can imagine.
Even saying "the same is true of unicode" shows confusion, as utf-8 is itself unicode. As are numerous different encodings, and then you have normalization and other subtleties.
If some is "in Unicode" it's a list of codepoints - that is the "thing" to be encoded. I don't see that ambiguity. It's a list of numbers, it's not fluffy kittens in the network or something. UTF-8 is a way of transcribing that list of numbers as 1s and 0s. That is the encoding process. Once it's encoded to 1s and 0s, it's not anything. It's just 1s and 0s.
One person's utf8 encoded 1s and 0s may be another person's audio signal. It doesn't really make sense to talk about the encoded data as being utf8. It's the encoded / decoding process that's utf8. Sure, it might be utf8 decodable.
In terms of json, it's the same deal. You have the "thing", simple nested objects/lists/numbers in memory. It only makes sense to talk about json when you're talking about the process of transcribing those.
In both cases you have a "thing" (I think of it as "real", but in a way it's an abstract concept - how does Python represent a dict, I have no idea), you can encode those things into a tangible list of 1s and 0s.
That's why they're the same. And people aren't generally confused about the json encoding decoding process.
First you need to think of text and binary (bytes) as two distinct types.
The text most of the times is how you interact with user, so things that you will display to the user, or what user enters to you.
Now, the text is stored in unicode (how, it's not our concern, Python abstracts it from us), the bytes is representation how it's stored in files, send over a network etc.
Now if you need to store text in a file, or send over network you need to encode it (most of the time as UTF-8), if you receive data which supposed to be text you decode it. It might be helpful to think of a text as a sound and bytes as mp3. You encode sound as mp3 to store it in a file, and when you want to play it back you decode it.
There are some things can cause a confusion. For example if you write text to a file, python will automatically apply the conversion. Things like that are there so you don't have to do the conversion every single time.
Encoding and decoding are pretty well defined (though, I've never thought about the formal definition before). When you have an entity in its native form, it needs to be _encoded_ for the purposes of communication (in a broad sense). The encoded message can then be to be _decoded_ back to the natural form. There is no ambiguity.
Really, the reason people get them mixed up in Python is because Python 2 totally stacked it by adding str.encode and unicode.decode.
In Python 2, you can _decode_ unicode to unicode – which it does by silently _encoding_ as ascii first. This operation is total madness.
>>> u = u'\xe9'
>>> u.decode('utf8')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 0: ordinal not in range(128)
And similarly, you can encode str to str (where str is really bytes in Python 2, another issue that adds to the confusion).
Don't get me wrong. I feel your pain. When I was using Python 2 I also got confused about what form things were in and where I needed to encode / decode.
Honestly, once I switched to Python 3, that cognitive overhead just totally vanished. str is the natural form of text, and if I need to store / communicate it I _encode_ it to bytes (utf8, generally). When I'm loading a stored/transmitted message, I _decode_ it to its natural text form.
There are edge cases that make certain situations more complex, but in terms of general usage, I feel Python 3 really got this stuff right.
You can use bytes(string,encoding) to replace encode(). Unfortunately it doesn't have a default encoding, which makes it a pain to use. And str(bstring) isn't symmetric, it can't replace decode().
By Python 2.7, there were types "unicode", "str", and "bytes". That made sense. "str" and "bytes" were still the same thing, for backwards compatibility, but it was clear where things were going. The next step seemed to be a hard break between "str" and "bytes", where "str" would be limited to 0..127 ASCII values. Binary I/O would then return "bytes", which could be decoded into "unicode" or "str" when required. So there was a clear migration path forward.
Python 3 dumped in a whole bunch of incompatible changes that had nothing to do with Unicode, which is why there's still more Python 2 running than Python 3. It was Python's Perl 6 moment.
From the article: "Obviously it will take decades to see if Python 3 code in the world outstrips Python 2 code in terms of lines of code." Right. Seven years in, Python 2.x still has far more use than Python 3. About a year ago, I converted a moderately large system from Python 2 to Python 3, and it took about a month of pain. Not because of the language changes, but because the third-party packages for Python 3 were so buggy. I should not have been the one to discover that the Python connector for MySQL/MariaDB could not do a "LOAD DATA LOCAL" of a large data set. Clearly, no one had ever used that code in production.
One of the big problems with Python and its developers is that the core developers take the position that the quality of third party packages is someone else's problem. Python doesn't even have a third party package repository - PyPI is a link farm of links to packages elsewhere. You can't file a bug report or submit a patch through it. Perl's CPAN is a repository with quality control, bug reporting, and Q/A. Go has good libraries for most server-side tasks, mostly written at Google or used at Google, so you know they've been exercised on lots of data.
That "build it and they will convert" attitude and the growth of alternatives to Python is what killed Python 3.
Well said.
I'm guessing it's not a coincidence that string encoding was also behind the Great Sadness of Moving From Ruby 1.8 to 1.9. How have other mainstream languages made this jump, if it was needed, and were they able to do it in a non-breaking way?
That said, the Unicode features were a major thing planned for PHP 6, and (afaict) one of the reasons there has never been a PHP 6, but rather they went straight from 5 to 7.
I'm not aware of any graceful transitions.
The key is UTF-8: UTF-8 is a superset of 7-bit ASCII, so as long as you only convert to/from other encodings at the boundaries of your system, unicode can be introduced to the internal components in a gradual and mostly-compatible way. You only get in trouble when you decide that you need separate "byte string" and "character string" data types (which is generally a mistake: due to the existince of combining characters, graphemes are variable-width even if you're using strings composed of unicode code points, so you don't gain much by using UCS-4 character strings instead of UTF-8 byte strings).
My theory is that the python 3 transition would have gone much smoother and still accomplished its goals if they had left the implicit str/bytes conversion in place but just made it use UTF-8 instead of ASCII (although in environments like Windows where UTF-16 is important this may not have worked out as well).
You're putting the cart before the horse here in terms of the Python 3 transition. The str/unicode fix was one of the driving factors for Python 3 to exist in the first place, and if you removed it, then what's the point?
Again, look at Go or Rust. Both of them have separate types for strings and bytes, even though they have the same representation in memory, and as a result we don't have that kind of bug in our program.
The reason python 3 exists is that most python 2 code had latent unicode-related bugs that would only manifest when they were exposed to non-ascii data. The backwards-incompatible barrier between str and bytes was the solution the python 3 team chose for this problem; adopting utf-8 as the standard encoding would have been another solution which I claim would have been more backwards-compatible (essentially moving to the go/rust model, which prove that you don't necessarily need separate byte and character string types for correct unicode handling).
But the thing with unicode-everywhere is that it's more beginner friendly: you can have unicode in your variable names, and strings without giving it another thought.
$ cat u.py
from __future__ import print_function
å="æ"
print(å)
$ python2 u.py
File "u.py", line 2
SyntaxError: Non-ASCII character '\xc3'
in file u.py on line 2, but no
encoding declared;
see http://python.org/dev/peps/pep-0263/
for details
$ python3 u.py
æ
That said, Perl 5 does not have different types for strings and bytes, which is definitely a source of bugs. Since Unicode support is essentially opt-in for a library (you have to explicitly decode/encode data somehow) it's easily possible for a library you use to break your data. Most of the major libraries (database, HTTP, etc.) have long since been patched to respect Unicode in appropriate ways, so the state of Unicode in Perl 5 is good overall.
I am not competent to say whether this is spot on or rubbish or somewhere in between [2], but it seemed interesting at least.
[1] http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/
[2] Almost all of my Python 2 experience is in homework assignments in MOOCs for problems where there was no need to care about whether strings were ASCII, UTF-8, binary, or something else. My Python 3 experience is a handful of small scripts in environments where everything was ASCII.
Yes.
> So you want cat to error out if one of the files that was passed has a different encoding.
No. I expect it to read bits from stream A until it is exhausted and then read bits from stream B until it is exhausted. All the time just writing the ones and zeros read to the output stream (of bits). And no, a byte does not have to be "8 bit" (http://www.lispworks.com/documentation/HyperSpec/Body/f_by_b...).
API-wise, in my humble experience, it's hell to deal with operations that are supposed to work on bit-streams but try to be smart and ask me for encodings of those - this is information I might not even have when building on those basic operations. The "oddities", how you call them, are the result of not over-abstracting the code to handle yet-unknown problems.
You want to concatenate text files with different encodings? Convert them. Expecting a basic tool to do this for you (or carp on "problems") quintessentially leads to cat converting image formats for you and demanding to know how to combine those images: (addition, subtraction, append to left/top/right/bottom of first image etc).
That's how C ended up as the security nightmare that it is. There are a lot of things you can do in C that you can't do in Python - you can reinterpret strings or floats as integers, you can subtract pointers from integers, you can write to random memory addresses.... Sometimes these things are useful, but most of the time they just lead to your program breaking in an unhelpful, nonobvious way.
Python is not that kind of language; it will go to some pains to help you not make mistakes. If you want to do bit-twiddling in Python there are APIs for it, and you could implement a "bit-level cat" using them, but it's never going to be the primary use case. Arguably there should be better support for accessing stdin/stdout in binary mode, but that would make it very easy to accidentally interleave text and binary output which would again result in (probably silent) corruption. (Writing a "binary cat" that concatenates two files of bytes would not lead to any of the problems in the linked article - it's only trying to use stdin/stdout that's causing the trouble in the link).
And that's how I ended up completely wasted after a friend had to throw a party after a GNU version of a common unix tool was able to accept his first name as valid parameters.
And no, C's problem are not based on encoding issues. Those are not even a first-class symptom.
> Python is not that kind of language
Maybe. I don't care much, even though I do speak Python fluently. Though, I do care about minimal functional units whose documentation I can grasp in minutes, not hours.
> [Python] will go to some pains to help you not make mistakes.
This is not specific to python, this is specific to [language] developers. If all you do is text processing, you will think in characters and their encoding. I don't. A lot of programmers don't - because they deal with real world data that is almost never most efficiently encoded in text.
To reiterate: We are all dealing with bit-streams. Semantics of those are specific to their context. If your context is "human readable text" - deal with it. But please don't make me jump through hoops if I actually just want to deal with bit-streams. If you need magic to make your specific use-case easier, wrap the basic ops in a library and use it.
Last but not least: This all is completely off-topic when the question is about "why python3" - it's great for your use-case, but from an abstract point of view, python3 was just the rational continuation of python2, cleaning up a lot of inherited debt. Though it might fit your world-view, it's not necessarily what it was about.
The only glitch is with stdin/stdout. They're opened on your behalf before your program even starts, and the assumption is that you'll be reading and writing text in the default OS encoding. This doesn't mesh well with the Unix pipe paradigm.
Fixing the unicode mess is nice too of course, but you can get most of the benefits in Python2 as well, by simply putting this at the top of all of your source files:
from __future__ import unicode_literals
Also make sure to decode all data from the outside as early as possible and only encode it again when it goes back to disk or the network etc.
I never really understood the "rather stay with py2.7" thing. I get it with big old monolithic applications. You don't "just" rewrite those, but _every new python project_ should be done with the latest stable release.
Is anyone starting their PHP projects on PHP4? Any new node projects in 0.10? Of course not, that would be moronic.
I mostly avoid writing Python 3 nowadays, because I don't want to rewrite my project or find painstakingly an alternative solution when I could have just imported a module that runs fine under Python 2.
1) It was easier than porting to CP3.
2) It gave me a tangible benefit by removing all CPU performance worries once and for all. Added "performance" as a feature for Python. Worth the testing involved.
3) It removed the GIL. If you use PyPy4 STM, which is currently a separate JIT. Which will be at some point merged back into PyPy4.
So for me, Python3 can't possibly compete, and likely never will with PyPy4 once you consider the performance and existing code that runs with it. PyPy3 is old, beta, not production-ready, based on 3.2 and Py3 is moving so fast I don't think PyPy3 would be able to keep up if they tried.
Python3 is dead to me. There's not enough value for a new language. I'm not worried about library support because Py2 is still bigger than 3 and 2.7 will be supported by 3rd party libraries for a very long time else choose irrelevance (Python3 was released in 2008, and still struggling to justify its existence...). My views on the language changes themselves are stated much better by Mark Lutz[0]. I'm more likely to leave Python entirely for a new platform than I am to migrate to Python3.
PyPy is the future of Python. If the PyPy team announces within the next 5 years they're taking the mantle of Python2, that would be the nail in the coffin. All they have to do is sit back and backport whatever features the Python2/PyPy4 community wants into PyPy4 from CPython3 as those guys run off with their experiments bloating their language. I believe it's all desperation, throwing any feature against the wall. Yet doing irreparable harm bloating the language, making the famous "beginner friendly" language the exact opposite.
I already consider myself a PyPy4 programmer, so I hope they make it an official language to match the implementation. There's also Pyston to keep an eye on which is also effectively 2.x only at this time.
[0]http://learning-python.com/books/python-changes-2014-plus.ht...
So far, solving CPU performance in Python and removing the GIL is pretty much the holy grail. Python2 would have to completely collapse (no signs of this) for CPython3's ecosystem to outweigh the long-tail of libraries only on 2.x and the truly next-level dynamic language CPU performance.
While I am enjoying "performance as a feature" and GIL-free Python at the moment, I can still migrate to Python3 at any time if I lose my safe bet with PyPy4/Py2.
To me it looked and still looks like a no brainer.
Well, that's kinda true, although there is somewhat of a learning curve doing 2-to-3 migrations. Depending on your situation, this may or may not matter, but if you need to "move fast" on this at some point, it'd at least be beneficial to know what's involved (even though it's not that onerous IMO).
In my attempts to test Python3 releases though I've ran across bugs and performance regressions from 2.x. After I continually ran into this in a few releases I eventually threw my hands up. The last release I ran tests on was 3.4 and I'm no longer interested in later releases until something as substantial as PyPy's CPU performance and removal of the GIL (PyPySTM) shows up to beat what I have now.
It's slick to have the entire Python2 ecosystem, major performance boost to your code, and still leave the door open for Python3 if they ever stop bloating the language.
Other than being dramatically slower than PyPy4, Python3 is also feature soup. I strongly dislike technical churn rather than true technical innovation (which is what PyPy represents). I'm more in line philosophically with Go, I'd prefer to remove features until you're down to a very concise and stable core. Python3 has many negatives, but from my perspective they keep piling on more.
You need to use all of the __future__ imports and be mindful of constructs that 2to3 will handle wrong. Better yet is to bite the bullet and make your code work with automated tests running 2to3 if all of your library dependencies have py3 support. Then you can continue to write in 2.7 and anything that runs afoul of py3 will be caught early.
I used to use the future imports and the things you're suggesting but since my job is Python2.x I stopped. I prefer to just keep my head there all the time. If I were to migrate to CPython3 instead of PyPy4 as I've done, it would be a wholesale move. Employer moves, I move. The underlying incentives aren't there though at the moment for either of us.
I'm certainly not going to push my employer into something that isn't in their best-interests (or mine, work for nothing). If anything, they need to move to PyPy for the same reasons I did.
Py3 needs a bigger ecosystem than 2.x, needs an answer to PyPy4's CPU performance and removal of the GIL, and convince employers it's better than CPython2 and/or PyPy4. Those are 4 potentially insurmountable tasks.
The best thing to do is start removing features rather than adding them, such as this 4th string formatting method in 3.6. It (Python3) is just very unappealing.
Industry players who want to remove the Python CPU bottleneck and willing to stick with Python rather than moving on to Go or JS on Node, are donating to the 2.7 based PyPy4.
So I'll be sticking with 2.7 for now.
This should be under penalty ;)
Anyone to divide it into few simpler sentences?
UPDATE: And another one from our connected sentences loving author: "We assumed that more code would be written in Python 3 than in Python 2 over a long-enough time frame assuming we didn't botch Python 3 as it would last longer than Python 2 and be used more once Python 2.7 was only used for legacy projects and not new ones."
It's a simple IF <condition> THEN <result>.
You can argue that he's overly verbose, but breaking the IF/THEN into multiple sentences reduces their connection and the ability to understand.
"If we were actually creating bug free code in Python 2 then porting code to Python 3 would be seamless, which it is not".
"I consistently hear from basically every person who ports their project to Python 3 that they found latent bugs in their code (regarding encoding/decoding of text and binary data). This would not be the case if people's hopes of coding bug free code had actually panned out."
That's a little better, but still not great.
then they would have bug free code
thus when they port their code to python 3, the unicode changes would not reveal latent (existing) bugs
thus they would not blog about or tell people about said bugs, hence the author of that sentence would not have heard about such bugs
moral of the story: Python 2 hides certain kind of unicode related bugs that are not exposed until you port to Python 3.
> If people's hopes of coding bug-free code in Python 2 actually panned out
Python2 developers wanted to write bug-free code.
code = for the purpose of processing text and binary data
> then I wouldn't consistently hear from basically every person
Python2 developers could not write bug free code. So they complained.
complained = complained about their algorithms having bugs when they rewrote those algorithms in Python3
> that they found latent bugs in their code regarding encoding and decoding of text and binary data.
Python2 code written by the same developers had bugs that they did not know about.
When the same developers rewrote their code in Python3, they found the bugs.
(If Python3 did not exist, then it would be very hard to write bug-free code in Python2.)
The second one:
> We assumed that more code would be written in Python 3 than in Python 2 over a long-enough time frame assuming we didn't botch Python 3 as it would last longer than Python 2 and be used more once Python 2.7 was only used for legacy projects and not new ones.
If we designed Python 3 correctly, then we expect Python 3 to live longer than Python 2. We also expect more code to be written in Python 3 for the same reason. We also expect only old projects will be written in Python 2.7.
Because making one of those those is a huge amount of work, may introduce more serious backwards incompatibilities (like C extensions), and not everyone has the knowledge of how to do it so they'd need to either learn from scratch themselves or find people interested in doing it for them.
A scripting language that support multi-threading is possible, right? I think TCL does it.
It's also often forgotten that one of the easiest paths to parallelism is to just run multiple processes (and for many problems you only need to split the input data, the processes don't need to even communicate), and this solution naturally uses multiple CPUs without any GIL worries.
I think the actual use cases that require GIL removal are very little (something like a server, perhaps), and if you actually need to do that, I think you're better served with some JVM language or Go.
People always proclaim that you can just use multiple processes for parallelism. It's nice when it suits your (embarrassingly parallel) problem but when you need to share a large amount of data between processes it's a major hassle.
Eh, isn't that because of the GIL?
In fact, as I understand it, you can actually call multi-threaded applications from Python. You can only execute Python code (in the same interpreter) single-threaded.
Although, an embedded scripting language (say, in a game) could in theory benefit. But isn't Python already too big for that kind of applications?
I think their attitude is generally, "if it's performance-critical, write it in C", which was a good approach 15 years ago. However, Python's competition now comes from languages like Go and Rust that have both good performance and are user-friendly and productive (with features such as expressive syntax and a comprehensive standard library).
It's trivial to remove the GIL - probably a week of mechanical work. Just don't depend on global variables in the interpreter. No global state, no problem. But the Python C API has to be changed to always store interpreter state into a struct, and a pointer to that interpreter state struct has to be passed as the first argument in all C API calls. Not rocket science; this is what Lua does.
It's a political decision to keep the GIL, not a technical one. As for preserving C API backwards compatibility, it's a straw man argument - the Python API broke from 2.x to 3.x anyway. There's no such thing as "lesser breakage" - only breakage.
What people usually mean when they talk about removing the GIL is having multi-threaded code make use of multiple cpu cores (as it does in Jython.) This would involved splitting the GIL into more fine-grained locks. Unfortunately experiments taking this approach have so far shown significant performance impacts for single threaded code.
It is better to have an interpreter instance per thread, each having their own separate variable pool, and pass messages between the different interpreter instances. This model scales well with multicore systems and is faster than the multi-process equivalent.
It's also a simpler model to implement and maintain. Python's current multithreading model is too complicated from an implementation point of view.
If they had to make Python 3 anyway, I think the main thing they were missing is that they should have added a JIT. That makes upgrading to Python 3 a much easier argument. If the only point of the JIT was to add a selling point to Python 3, that probably would have been worth it.
There are a lot of other subtle changes that makes the transition harder: comparison changes and keys() being an iterator for example. These are good long term changes, but I wish they weren't bundled in with the bytes/unicode changes.
(and yes, Unicode in Py2 is a mess ...)
They just broke to many things (unnecessarily!) internally. Particularly they changed many C APIs for enhancement modules, so that all of them had to be ported, before they could be used with Python 3. They did not even consider a portability layer ... why not??
Some (not all) of the bad decisions (like the u"..." strings) they did change afterwards, but than it was a little late.
So many modules are still not ported to Python 3 -- so the hurdle is a little to high -- for small to nil benefits!
So, the problem (from my side) is not Unicode at all ... just the lack of reasonable support from the deciders side.
---
Maybe, some time later, when I have to much spare time.
"So expect Python 4 to not do anything more drastic than to remove maybe deprecated modules from the standard library."
But why break all the existing libraries that use those modules, even if there's now "better" ways. In every comparison I've ever seen on performance, robustness, etc python always loses to the other big languages. Except in one area: the availability of user-land submitted packages and extensions. So why break them for a little perf boost?
I hope, that they really think hard, which modules really must be removed.
I would only consider security reasons to really remove a library module. Or at least mark it deprecated for a long while until it is used only really rarely.
To use anything newer, I'd have to ask users to install a different interpreter, or bundle a particular version that adds bloat. There's no point. The most I've done is to import a few things from __future__; otherwise, my interest in Python 3 begins when Apple installs it.
in practical terms, the transition is about to be over, since now the Linux distros are all-in on converting to Python 3 for their current or next releases and that will forcibly move the unported libraries in the bin of obsolescence.
What I don't get is: why has Python 3 adoption been so slow? Is it just backward compatibility, or are there deeper problems with it that I'm not aware of?
'%s %s' % ('one', 'two')
'%s %s'.format('one', 'two')
'{} {}'.format('one', 'two')
'{} {}'.format('one', 'two')
I love love love printf() and it's ilk, so switching to something else (no matter how well designed) seemed asinine at first.
But .format() has really started to grow on me and did uncover some subtle bugs in old code.
'Coordinates: {latitude}, {longitude}'.format(latitude='37.24N', longitude='-115.81W') f'{one} {two}'https://docs.python.org/2/library/string.html#string.Templat...
Really this topic is coming to an end. Most libraries support Python 3 and if they don't there's better alternatives.
For new users and new projects there's no reason now except personal preference to choose Python 2 and in fact beginners who start with Python 2 are just instantly incurring a learning debt upon themselves to be paid down the track when they have to move to Python 3.
The community has some extremely vocal Python2 diehards but their arguments no longer hold water.
>>why has Python 3 adoption been so slow? Whatever, it's just history now.
import pymysql as MySQLdb
Another reason is that the advantage of switching is just not that big, if you already have everything working in Python2.
It took a few versions of 3 to hit a sweet spot (in some cases features that were removed in the initial version of 3 have slowly been re-added in subsequent versions). There were a lot of crucial libs that needed to be ported. Just general inertia.
Perhaps you want to parse an old obscure file format, and the only code you find for it is from a usenet post in 1996. That code isn't being updated, and no one has ported it. That means you need to do the work to update it, and that can be hard when you aren't familiar with what it's supposed to do.
The other place I've seen people sticking with Py2 is when they've got a huge chunk of internal code. Some companies have been writing python for 20 years, and the original authors have long since left the company. It can be hard to write a business case for having someone spend several weeks updating all the old code, particularly if it's purely internal, and doesn't touch the internet.
The list I've gone off of is https://python3wos.appspot.com/. I use python a lot, and only now in late 2015 I might finally use python3 if starting a new project. When I started a new project last year, we used python2.
The biggest hold-out for me was gevent, which was only released 5 months ago. Gunicorn with gevent workers is my preferred stack for running python apps.
If you use protobufs or thrift, those both aren't yet on python3.
The wall of shame currently lists requests as not working on python3, though I think that might be a fluke.
These ones might not be a deal-breaker since you can have a separate environment for infrastructure & app code, but for some reason a lot of the infrastructure tools still haven't updated to python 3 (supervisor, ansible, fabric, graphite).
All together, it adds up to a not-insignificant number of things that aren't yet on python3. And even if nothing you use when you first start a project is python2 only, you have no idea what libraries you might need or want in the future and if those might be python2 only.
If you're willing to do a bit more than "pip install x", well I guess it doesn't work, back to py2, you can use almost everything on py3. (and yes requests is ported too)
Gevent said it looks like they're supporting Python3 as of the 1.1 release - http://www.gevent.org/whatsnew_1_1.html
I've used requests with Python3 quite a bit.. I have no idea why it's not listed as supported on the WoS, but their page shows it as working since 2012. https://pypi.python.org/pypi/requests/
Protobugs looks like it's now Py3 compatible, per the devs- https://github.com/google/protobuf/issues/646
ThriftPy has supported Python3 for a while (although ThriftPy is slower than the official lib). Apache Thrift has recently begun working with Python3 as well, however (https://issues.apache.org/jira/browse/THRIFT-1857)
I run Ansible/etc in their own virtualenvs, so they aren't part of system python for me, but you're right - I did have to write an Ansible module recently, and I recall that I did have to use Py2.
The only major package I recall having a problem with was PIL - Eventually I moved to Pillow as a drop-in replacement.
[1]: http://docs.python-requests.org/en/latest/
[2]: http://docs.python-requests.org/en/latest/#feature-support
Moreover, in machine learning Python 2.7 is still considered the default version of Python. (E.g.: some part of OpenCV need Python 2.7; until recently Spark supported only Python 2.7.)
Within the last six months we've moved to writing all new code in Python 3 and migrating a fair bit of legacy code as well. Been fairly smooth on Linux -- a bit rockier on Windows.
This wasn't just down to the fact that Python 3 didn't support u"" (it was 3.3 that was added, for reference), but also down to the fact that much of the eco-system still supported RHEL5's default of Python 2.4 which meant `from __future__ import unicode_literals` wasn't an option (it is, almost certainly, a less good option, but it's in many ways good enough).
Following their glacial release schedule, maybe we'll see Python 3 by 2019 in RHEL 8.
(It of course limits software that is to be shipped with the distro itself)
> I'm also a fan of map and filter returning generators rather than being hard-coded to a list implementation
I find myself often having to wrap expressions in list(...) to force the lists. (which is annoying)
Generators make things much more complicated. They are basically a way to make (interacting, by means of side-effects) coroutines, which are difficult to control. In most use cases (scripting) lists are much easier to use (no interleaving of side effects) and there is plenty of memory available to force them.
Generators also go against the "explicit is better than implicit" mantra. It's hard to tell them apart from lists. And often it's just not clear if the code at hand works for lists, or generators, or both.
So IMHO generators by default is a bad choice.
> stream fusion is a good thing
I don't think generators qualify for "stream fusion". I think stream fusion is a notion from compiled languages like Haskell where multiple individual per-item actions can be compiled and optimized to a combined action. Python instead, I guess, just interleaves actions, which might even be less efficient for complicated actions.
Out of curiosity, why do you need to force the lists?
> Generators make things much more complicated. They are basically a way to make (interacting, by means of side-effects) coroutines
Huh? Generators are a way to not make expensive computations until you have to--as well as to not use memory that you don't need. Basically, if all you're doing with a collection of items is iterating over it (which covers a lot of use cases--but perhaps not yours), you should use a generator, not a list--your code will run faster and use less memory.
> In most use cases (scripting) lists are much easier to use (no interleaving of side effects) and there is plenty of memory available to force them.
Generators don't have to have side effects. And there are plenty of use cases for which you do not have "plenty of memory available" (again, perhaps not yours).
> IMHO generators by default is a bad choice.
I think lists by default was a bad choice, because it forces everyone to incur the memory and performance overhead of constructing a list whether they need to or not. The default should be the leaner of the two alternatives; people who need or prefer the extra overhead can then get it by using list() (or a list comprehension instead of a generator expression, which is just a matter of typing brackets instead of parentheses).
for i in range(large_number):
...
i.e., where you don't bind the generator to a variable, but instead immediately consume and reading from it as an iterator has no other side-effects.
And that's the only usage of generators that in my usage is both common and practical. Other use has always quickly become a mess for the above named reasons.
I don't disagree generators can be occasionally useful (or often, for your special applications). But mostly it's a pain that they are the only thing many APIs return and that they are not visually distinctive (in usage) from lists and iterators. For example
c = sqlite3.connect(path)
rows = c.execute('select blablabla')
do_some_calculation(rows)
print_table(rows)
So by far the common (= no billions of rows) sane thing to do is
rows = list(c.execute('select blablabla'))
Or, to put it another way, since there are two possibilities--realize the list or don't--one of the two is going to have to have a more verbose spelling. The obvious general rule in such cases is that the possibility with less overhead is the one that gets the shortest spelling, i.e., the default.
Also, if you've already realized a list, it's too late to go back and un-realize it, so there can't be any function like make_generator(list) that saves the overhead of a list when you don't need it. So there's no way to make the list alternative have the shorter spelling and still make the generator alternative possible at all.
As far as your sqlite3 example is concerned, why can't do_some_calculation(rows) be a generator itself? Then print_some_calculation would just take do_some_calculation(rows) as its argument. Does the whole list really have to be realized in order to print the table? Why can't you just print one row at a time as it's generated?
Basically, the only time you need to realize a list is if you need to do repeated operations that require multiple rows all at the same time. But such cases are, at least in my experience, rare. Most of the time you just need one row at a time, and for that common case, generators are better than lists--they run faster and use less memory.
If you need to do repeated operations on each row, you just chain the generators that do each one (similar to a shell pipeline in Unix), as in the example above. This also makes your code simpler, since each generator is just focused on the one operation it computes; you don't have to keep track of which row you're working on or how many there are or what stage of the operation you're at, the chaining of the generators automatically does all that bookkeeping for you.
I can tell you about our situation.
We are an animation studio with decades of legacy Python 2 code. We sponser pycon and are one of the poster children for python.
We have absolutely no plans to switch to Python3.
Here are the various reasons:
- Performance is a big deal, and moving to a version of python that is slower is a no-go off the bat.
- Python3 has no compelling features that matter to us. The GIL was the one thing that should have been tackled in Python3.
- Since the GIL is here to stay, our long-term plan will likely involve removing more python from the pipeline rather than putting a huge effort into a python3 port.
- We have dependencies on 3rd-party applications (Houdini, Maya, Nuke) that do not support Python3
- We have no desire to port code "just because". Each production has the choice of either spending effort on Real Features that get pixels on the screen, or on porting code for No Observable Benefit. Real Features always win.
- Python3 has a Windows-centric "everything is unicode" view of the world that we do not care about. In our use case, the original behavior where "everything is a byte" is closer to UNIX. A lot of the motivation behind Python3 was to fix its Windows implementation. We are a Linux house, and we do not care about Windows.
- Armin's discussion about unicode in Python3 hits many of the points spot on.
Why has adoption been so slow? Simply because we have no desire to adopt the new version whatsoever. We'll be using Python2.7 for _at least_ the next 5 years, if not more.
It's far more likely that we'll adopt Lua as a scripting language before adopting Python3.
- We have dependencies on 3rd-party applications (Houdini, Maya, Nuke) that do not support Python3
This is our reason. It's a bit of a conundrum meets catch-22 situation. On one hand we would start using python3 if there would be a support for it, on the other hand no one wants to because of porting legacy code seems bothersome if everything works as it should. To be honest though, all of our python code is to augment those 3rd party applications. Everything we have that's not tied to those applications is C(99 more or less).
And you're welcome to do that.
But "we want a language frozen in time forever so we never have to maintain code" -- which seems to be what you're aiming for -- is not a goal you can achieve short of developing your own in-house language and never letting it make contact with the public (since as soon as it goes public it will change).
Meanwhile, the libraries are moving on and sooner or later they'll either move to Python 3, or be replaced by equivalent libraries with active maintenance, and the distros are winding down their support for Python 2. Switching to something else, and probably just rolling an in-house language you can control forever, is likely your only option if this is your genuine technical position.
The real feature needed is to eliminate the GIL. That would be worth breaking compatibility over.
Python 3.0 was slower than 2.7 due to several key bits being implemented in pure Python in 3.0. Since the 3.0 release (remember, Python's on 3.5 now) things that needed it have been rewritten in C, and as of Python 3.3 the speed difference is one. Also, on Python 3.3+ strings use anywhere from one-half to one-fourth the memory they used to.
As for "no compelling features", well...
* New, better-organized standard library modules for quite a few things including networking
* Extended iterable unpacking
* concurrent.futures
* Improved generators and coroutines with 'yield from'
* asyncio and async/await support in the language itself
* The matrix-multiplication operator supported at the language level (kinda important for all the math/science stacks using Python)
* Exception chaining and 'raise from'
* The simplified, Python-accessible rewritten import system
etc., etc., etc.
Given how many people and projects suddenly said "yeah, actually, we want to be on Python 3 now" after seeing the new stuff in 3.4 and 3.5, I think you're overestimating the number of people who don't care about these features.
1. Python 3.4 was released 7 years after 3.0. That's a very long time.
2. Assuming you're referring to the async features (none of the other stuff is really momentous), I can understand that for framework devs. It's really not a big deal for most users though, and because of the GIL, it's not as if they'll suddenly reap the benefits of parallelism. All it really means is nicer algorithm expressions and event loops.
3. There's no technical reason the new stuff couldn't have been added to Python 2. The roadblocks are manpower and politics.
4. Even if we stipulate async/await/asyncio are huge, busted Unicode support, 7 years of development, and breaking compatibility with everything is just a terrible tradeoff.
There's just no way this was a good idea, and Python devs could earn a lot of credibility back if they just said "oops". But there's no chance of that.
I wonder:
1. Would this have changed if there was a Python 2.8 with some new features (but still a GIL) that ran most 2.7 code?
2. Is your experience representative, i.e. are teams just not starting new projects in any version of Python, even though so many did in the last decade?
I am curious - if there was a GIL-removed version of Python 2, would you change your assessment that your "long-term plan will likely involve removing more python from the pipeline"? i.e. is the GIL the primary (or even sole) factor in that?
I don't think the lesson is "never break compatibility". The lesson is "don't compete with yourself by releasing a product that is actively worse than your current version"
Print is used either for:
1) Writing to stdout/stderr 2) Debugging the hacky way
The print function is better for the former (though more often than not, I use Armin Ronacher's click.echo for compability)[0], but I fastly prefer the print statement for (2). Not dealing with parentheses is always a plus; I can add and remove print statements far more quickly.
I find it to be an increase in friction, and I don't see any real downside in leaving it in.
For 2), though, I have always used auto-inserting parens in my vim, so I never really experienced any pain w.r.t. parentheses. I can see how that would be a pain, though.
Though it does serve as a neat diagnostic "this program wasn't built with python3 in mind" message when you encounter the syntax error, so maybe it balances out...
>>> print "hello world"
It defined the ethos of the language, was the first thing people came across when considering Python for the first time, and probably punched well above its weight in bringing people in.
And that's so much harder to grok?
I feel that simplicity is a driving factor in python's 'virality' from one programmer to the next, and in that context your case is very compelling.
I'm fine with leaving it out, but I would like it back in. The reason is that in my mind Python has been the language of simplicity and elegance, where you are constantly pleasantly surprised by the ease of use. Python, to me, is the language of childish glee. The parentheses on the print function is Python putting on a suit and going to work.
Two primary exceptions: byte/string encoding in Py2, and print. So much of the Py2 syntax works without change in Py3, and while I wholeheartedly agree with "print('foo')" as the proper way to print something, "print 'foo'" is littered all over the random Py2 scripts I've found in my travels.
I am, in fact, converted to print(), for the record. The first time (in Py2) I tried to extrapolate what I learned about (asterisk)args, I attempted to do this:
print *df.columns
print(*df.columns)
(edit again: Can't seem to get that asterisk to work without a full-on space)
The best quip I've seen about this is:
Python 3 broke "hello, world".
The article claimed that Guido started Python back in 1989. Wiki tells me that Python 3 was released in 2008. So, if after nearly 20 years of existence, you break the quintessential program of your (or rather any) computer language, then, ipso facto, you've just abandoned your current language and switched to a new one.
Others have mentioned that the print statement could have been retained, while also allowing a function. I would have preferred that. Not "the Python way" of having one way to do something, but there's also backward compatibility to consider.
Setting up an abbrev or a snippet also helps. I use these a lot:
p<tab> -> print(|)
pp<tab> -> import pprint; pprint.pprint(|)Yes, AWK specifically.
The 'there should be one – and preferably only one – obvious way to do it' rule sounds like another reason. It's like being asked to choose between a perfect general use knife or a Swiss army knife.
You're missing the obvious word. It's not about removing options, it's about having good and clear defaults.