Python 3.3.0 beta 1 released
mail.python.org
mail.python.org
I mean: I do almost all of my "business logic" coding in Python these days, and yet when I hear of a new version of Python it carries as much weight to me as a new version of COBOL (which, honestly, probably has more use cases, and almost certainly more users, than Python 3.x). This is the first time I have simply totally felt stranded by a development environment since Microsoft abandoned Visual Basic and replaced it with "Visual Fred" (aka, Visual Basic .NET): and in that case I had already jumped ship to C++ years earlier.
At this point, the cost of trying to get all the libraries I rely on set up on 3.x--especially when a ton of them have decided either 1) to slightly change the semantics of the API "as everything is breaking anyway" or 2) that they are obsolete and that you should use some other newer library if you want 3.x--is sufficiently high that one may as well switch to an entirely different language (in my case, probably Clojure).
FWIW, and while I disagree with you wrt long-term (though short term I'm still on Python 2 due to working on a non-ported 10 years old code-base with dependencies on non-yet-ported libraries), 3.3 has added some concessions back for ease of porting with the re-introduction of the unicode literal prefix `u` (see PEP 414).
> and in some cases for the obvious worse, such as how Unicode is now less correctly handled when iterating filenames in directories
Either it's handled or it's not, Python 3 provides both options for you to pick, linux filesystems are broken either way.
> in my case, probably Clojure
From my playing with it so far, even if I switch "big" developments to that Python will have to remain part of my toolbox due to how horrendously slow Clojure and clojure-based tools are to start up. And I have issues with the utterly terrible error reporting as well.
Just how "long term" are you thinking here? Another four years? Does it really seem likely that this will change? At some point the long term horizon is so far in the future that it becomes meaningless.
As an example, will the entire set of libraries you are using even have maintainers anymore? I will argue by the time Python 3 even has a hope of satisfying this bootstrapping problem, they will have moved on.
> Python will have to remain part of my toolbox due to how horrendously slow Clojure and clojure-based tools are to start up
Yes: that is currently the thing that is most bothering me about Clojure. That said, I have some plans for how I can fix that, and those plans, as ludicrously complex as they are, honestly sound easier to me than the Python 3 porting problem.
(In fact, modifying Python 3 to run Python 2 code sounds easier to me than the transitive-to-all-libraries porting problem, and has therefore been on my todo list. However, I have the problem that I need to get work done immediately...)
Regardless, I have little choice here: the Python 2 ecosystem is dying by fiat, while the Python 3 ecosystem has no real impetus behind it. The result is that the entire Python ecosystem seems to be stagnating: it is harder and harder to find good libraries for new things.
While I agree that tools which take a long time to start up can be annoying I don't believe that an IDE which takes 30 seconds to start up, but is left open for days at a time should be an issue. Are you referring to the IDE or the interpreter?
Interpreter and command-line tools (e.g. leiningen)
Useless for you :). Personally, I made the switch to Python 3 some time ago, and couldn't be happier. My dependencies are largely sorted these days, availability on the systems I need to deploy on is no longer an issue either, and I find it to be a cleaner, leaner language makes me more productive overall.
Sure, there was a time when I read these 3.x announcements with a sort of weary, frustrated feeling, but now I'm back to feeling like a kid in a candy store and excited at what the new release is going to bring.
I realize not everybody has made it to this side of the transition yet, but don't fret it, the day will come, and it's a nice place to be in.
I am curious, however, how many libraries you actually rely on, given the ease you seem to indicate you were able to move with. Most of my code is just gluing together libraries, from wrappers for libapt to PDF parsing and generation (for which I actually have to use two different libraries) to database bindings to advanced math. I have pages of import statements in my codebase...
Near as I can tell, were I to start fresh and use Python 3, even if all the libraries I currently use were ported, I'd just come across a new use case that causes me to need a new library, and as that library will likely not be ported, I'll just be cursing my decision to have switched. Seriously: switching from Java to Clojure has less potential future regret than switching from Python 2 to 3. Hell: Java to C++ has better backwards compatibility (JNI is actually reasonable, and is easily abstractable).
Not that many, admittedly. I'm fairly wary of dependencies and prefer to stick to the standard lib unless something is just orders of magnitude better at the job, to cut down on deployment overhead. That attitude has of course benefited me tremendously with the Python 3 transition.
That said, I currently use lxml and requests a lot, a long with a bunch of smaller things like python-dateutil and mock (the latter of which was just added to the standard lib in 3.3, actually). Sum total I have maybe 12-15 external libs across my codebases that I didn't write myself, and I've had only one case of an abandoned lib that I needed to port myself.
As for my own code, when I started out transitioning I had already cut compatibility with anything older than 2.6, and making things run both on 2.6 and 3.1 turned out to be relatively easy for the most part (certainly with some pain points too however, like needing to abandon syntactic sugar for metaclasses). As the next step I went to 2.7/3.2, which made things even easier. Most of my code now still runs on 2.7 in addition to 3.2, but I'm now starting to abandon the notion of backwards compatibility and introducing code that relies on Python 3.2 and am slowly removing the hacks once required for cross-compat as I find need to touch the respective areas again.
Now, I realize that I'm not the norm and that a lot of Python folks out there have a much harder time transitioning. But with the bleakness of your post, I felt compelled to point out that it's not been hell for all of us.
I can't see myself dropping 2.x support anytime soon.
If you're updating just to update, then the problem isn't with Python...
One man's stable environment is another man's technical debt I guess...
Many, many high profile projects are already on Python and many will follow, now porting has been made even easier.
Django tries to support Python 3 with 1.5, Pyramid already does support it, Twisted has a GSoC student that tries to bring it to Py3… And funnily enough, “normal” code that doesn’t screw around too much with advanced features is rather trivial to port. I have written a Python 3.2 program myself only to find out later that it also works on Python 2.7 without changes. :)
With every bigger change you lose some – and I’m sorry if it’s inevitably you – but you also win some. Because all in all, Python 3 is great.
Everything on the install CD is planned to be Python 3 by this fall. I'm already working to port what's necessary for Ubuntu One.
In my comments, I describe a specific phenomena that are leading to the problems that I experience: that there is now a fractured community of libraries for what are really two unrelated languages.
As some even-more-specific actual-examples:
1) key libraries such as PIL are not ported, despite your claims regarding the ease
2) of those that are, such as psycopg2, there are often awkward (and I will often claim "broken") semantic changes to the behavior of the API
3) not all libraries actually have maintainers to port the code at all (such as the iPhone PNG transcoder I rely on)
4) finally, there are libraries, even entire frameworks (mod_python), that rely on functionality that is actively being removed with prejudice (in the case of mod_python, for reasons I continue to not understand, despite having read copious e-mail exchanges on the Python mailing list regarding the issues)
(A quick aside, for anyone about to pull a "lol mod_python" on me: mod_python is fundamentally more powerful than the various replacements, something even the primary maintainer of mod_wsgi admits; there is a reason why, despite it having been retired from Apache due to a lack of maintainer, it was still used by Google engineers as the basis of their pywebsocket implementation.)
I will therefore challenge you to put up something behind your insults, and explain what you feel has changed, and how it affects the argument I have made. While it is of academic interest that a few web frameworks now support Python 3, real programs, even real websites, are going to rely on libraries to accomplish their underlying goals.
I will even further challenge you to defend your accusation that my experience is "unique"; I personally find this idea incredible, given that I ask every Python programmer I meet "2 or 3", and not a single person has ever said "3". Yes: if you call out Python 3 in public on a Python 3 release post, a number of people will come out of the woodwork, but in a random sample under more realistic conditions (non-Python developer conferences) the reality seems closer to 0%.
In my comments, I describe a specific phenomena that are leading to the problems that I experience: that there is now a fractured community of libraries for what are really two unrelated languages.
As some even-more-specific actual-examples:
1) key libraries such as PIL are not ported, despite your claims regarding the ease
2) of those that are, such as psycopg2, there are often awkward (and I will often claim "broken") semantic changes to the behavior of the API
3) not all libraries actually have maintainers to port the code at all (such as the iPhone PNG transcoder I rely on)
4) finally, there are libraries, even entire frameworks (mod_python), that rely on functionality that is actively being removed with prejudice (in the case of mod_python, for reasons I continue to not understand, despite having read copious e-mail exchanges on the Python mailing list regarding the issues)
(A quick aside, for anyone about to pull a "lol mod_python" on me: mod_python is fundamentally more powerful than the various replacements, something even the primary maintainer of mod_wsgi admits; there is a reason why, despite it having been retired from Apache due to a lack of maintainer, it was still used by Google engineers as the basis of their pywebsocket implementation.)
I will therefore challenge you to put up something behind your insults, and explain what you feel has changed, and how it affects the argument I have made. While it is of academic interest that a few web frameworks now support Python 3, real programs, even real websites, are going to rely on libraries to accomplish their underlying goals.
I will even further challenge you to defend your accusation that my experience is "unique"; I personally find this idea incredible, given that I ask every Python programmer I meet "2 or 3", and not a single person has ever said "3". Yes: if you call out Python 3 in public on a Python 3 release post, a number of people will come out of the woodwork, but in a more random sample under more realistic conditions (non-Python developer conferences) the reality seems closer to 0%. Hell, even the comments I've made on this thread have been upvoted quite a bit (+22 on my root argument).
And think about how the 3.X changes totally screwed the Jython and PyPy (and Cython, and IronPython, and...) teams. We were headed towards much higher-performance implementations of the language, with additional powerful features like sandboxing and JIT, and it was set back years by the 3.X schism. Furthermore, the continued complication of the language makes it ever less likely that core problems like the GIL will ever be fixed. (Yes, I know Jython and IronPython are GIL-free.)
My startup relies heavily on Python, and the only reason I don't seriously regret choosing Python as a core technology is that there was nothing else better at the time. The Python overlords have reduced the language to the best of bad options; it didn't have to be this way. FAIL.
I still hope against hope that there will be a Python 4 that brings backward compatibility and GIL removal. But since none of the overlords seem to care about either of these goals, and since both get harder with every new Python 3 release, it's probably just my own sad little fantasy.
I don't see how the 3.x changes have set back PyPy "by years". PyPy was focussing exclusively on 2.x until very recently, and made its last big steps towards production-readyness entirely undisturbed by the 3.x transition. As of a couple of months ago they now have a small team working on 3.x support, which has made huge progress in a short time frame and doesn't seem to have detracted from progress in the core and further 2.x release at all as far as I can see.
> Furthermore, the continued complication of the language makes it ever less likely that core problems like the GIL will ever be fixed.
Ironically, specifically the oh-so-handicapped PyPy is currently working on a new approach to address the GIL with their transactional memory work. And the language has gotten less complicated with 3.x, not more.
> I still hope against hope that there will be a Python 4 that brings backward compatibility and GIL removal. But since none of the overlords seem to care about either of these goals [...]
Right, that's why 3.3 just added back support for u'' for increased backwards compat and ease of porting.
You're making a lot claims that simply seem to have no basis in reality.
"PyPy is a fast, compliant alternative implementation of the Python language (2.7.2)."
Why not 3.3? Oh, right, because when the PyPy team started, Python 2.X was the Python language. Now it's not. This has been the case (PyPy has existed and supported 2.X code) since 3.0 was released. I.e., for years.
As for PyPy STM, you trot that out like it's a real solution. It is not. It's 5 times slower and doesn't support Python 3.X. Which, of course, is the whole point.
But, yes! Thank goodness I can type u'...' again! That will save me years of work! FAIL.
This sort of comment doesn't really make me feel like you're qualified to comment on the subtleties of killing the GIL, to be honest.
When it comes to removing the GIL, any form of "it's slower" is usually going to refer to single-threaded workloads, and avoiding to be slower for those is going to be nigh on impossible. The idea is that it's an investment to be recouped for various workloads due to the improved ability to exploit concurrency over a GIL-ful interpreter.
And really, it's at the proof of concept stage right now, hardly optimized. And there's new CPU hardware with hardware support for transactional memory coming down the pipe.
This idea has been rejected by the core team though, getting rid of the gil with a major performance hit is not considered acceptable when most existing Python code is not significantly affected by the GIL.
I think the PyPy folks want to get to a place where you can run PyPy in modes tuned for either single-threaded or concurrent workloads.
- First and foremost, your frequent use of ridicule and hyperbole is escalating the tone of the debate.
- You prefer to move goal posts instead of addressing arguments (e.g. you still haven't actually gotten around to explaining how the 3.x transition has supposedly set back PyPy by years).
- You're not beneath skewing things to fit your narrative, even if it doesn't really make any sense (e.g. pointing out that PyPy's STM tinkering doesn't yet support Python 3.x yet - yeah, duh, given that both are WIP branches that aren't merged into PyPy proper yet, and nevermind that the STM work doesn't really impact the same area of the codebase that adding Python 3.x support does, so there is no cause to anticipate problems with eventually integrating both).
- You make bold claims and leave them unsubstantiated (e.g. "It should require neither new hardware nor Herculean effort to realize acceptable multicore performance on CPU-bound tasks." just now - if you've got it all figured out, where's your proposal to eliminate the GIL?).
- Those bold claims even miss their mark in the context of the debate (I pointed out that "it's slower" refers to single-threaded workloads; when it comes to multicore performance the STM proof of concept is already showing some benefits).
And so on and so on. Simply put, your posts are low quality, that's why I'm hitting that downward arrow.
I stand by my claim that any reasonable general-purpose programming language should be able to run simple CPU-bound threaded tasks properly. I didn't claim arbitrary parallel programming should be easy; just that the GIL causes multi-threaded programs written in standard style in Python to not work properly in CPU-bound contexts, because the GIL prevents these programs from using all the cores available, and, furthermore, because even in single-core contexts it is prone to performance failure modes such as convoying and starvation that induce worse performance when there are more cores available. (cf. very extensive discussions in bugs.python.org on GIL problems, along with David Beazley's, Antoine Pitrou's, and others' attempts to address them).
It is a standard, uncontroversial expectation of general purpose languages that offer threading that the threads run in parallel, using the available cores. I don't believe this needs substantiation any more than the claim that general purpose programming languages should run serial programs deterministically.
I believe the better course for Python post 2.X would have been to fix the core problems with the language's reference implementation, while maintaining backward compatibility with the very large existing code base, rather than adding features of dubious value, doing nothing to address the GIL problem, and breaking compatibility with the vast majority of 2.X code. While this is certainly my opinion, and yours obviously differs, you can't badger away the limitations of the GIL, and you can't just browbeat people into accepting the 3.X series as a good thing.
Look, I don't hate the language -- I use it every day and like it a lot. But don't pretend that there aren't real problems with Python when there clearly are -- the discussions on bugs.python.org are mainly driven by strong proponents of the language and are hardly politicized -- and don't pretend that Python 3 wasn't problematic; it's created years -- yes, years -- of problems and confusion with 2 vs 3 incompatibility, has expanded the footprint of the language and changed its semantics in substantive ways (cf. the many, many accepted PEPs post 2.X) that make implementing the language harder. Every language bump (whether 2.5 to 2.6 or 2.7 to 3.0) has meant additional work for the developers of alternative implementations; this is obvious on the face of it; you need only read these projects' blogs and look at the elapsed calendar time.
I also claimed that the additional features added to Python 3.X have meant that those who have already fixed the GIL problem (Jython/IronPython) and those who might still (PyPy) have been forced by the 3.X change to chase the taillights of CPython, making it essentially impossible for these implementations to replace CPython as the canonical version.
(I do take for granted that the canonical implementation of Python should have multithreading primitives that can take advantage of all the cores in machine; that has been debated above already so I won't repeat it here.)
The fact that Python 3.3 has a lot of more features than Python 2.7 is is due to the fact that 2.7 was the last release that got Python 3 features back ported (two years ago) and is in bug-fixing mode since then. (and people in these comments still complain about Python moving too fast)
There also have been efforts to ease the pain for PyPy et al by requiring new modules to have also a Python based implementation and by – put simply – giving them money from PSF funds. Jython’s problem was rather lack of interest and developing funds as they were stuck at 2.5 forever until one of their developers got funding to bring it to 2.7.
It’s not all as bleak as it seems to you and chances are that your porting effort will be negligible as soon as the dependencies are met.
And the GIL…just leave it be. There are really no plans or efforts to remove it for many, many reasons I won’t reiterate here.
You're saying it's not bleak, but then immediately say the GIL is a permanent fixture of the language. The GIL not being fixed is in fact what makes it so bleak (to me, at least).
Python isn't just for scripts anymore; we have 150k lines of Python code doing nontrivial stuff and relying on many third party libraries -- ultimately probably a million lines of code or so if you trace the dependencies all the way down.
This is in most ways really great: the Python code is very clear and concise, and comes along with all the documentation and unit testing conventions established by the community over the years. All those high-quality third party libraries mean we don't even have to write lots of code in the first place. It's easily a 10x productivity improvement over C/C++ for us.
But to make this code base take proper advantage of multiple cores, we must run under Jython or IronPython. (And at least Jython has its own problems; start up times are long because the Java class loader is slow.) We can't move to 3.X even if we wanted to, because no GIL-free implementation supports 3.X. We can't rely on the processing module, because we can't afford the overhead of an entire process of memory usage per "thread" at scale.
It just sounds like the Python devs have decided that fixing the GIL is too hard, so people should just live without using multiple cores. Do they view our use case as exotic or somehow unsupported? Do they not see that it's important for a general purpose programming to support real threading? I'm not trying to be obnoxious; it's a serious question.
We are likely to ultimately abandon the language because of this core issue, which is tragic given the other great things we get from using it, and the lack of a genuinely equivalent alternative. "So what, no one cares about you -- you are nobody," you may say. For the sake of argument, I will stipulate that. But I don't imagine our constraints are unique, and this bodes ill for the Python ecosystem. Our abandoning it is one thing; increasingly many developers abandoning it as multicore use becomes ever more important is another. Cores are getting a little bit faster with each tick and tock at Intel. But the rate at which we're getting additional cores looks to far outpace the per-core improvements for quite some time. This makes real threading increasingly important as time goes on.
I know STM is the solution du jour, and it's an admirable piece of work. If that solves the problem in the long run, great, but it's far from obvious that it's going to win without dedicated hardware support, and in any case isn't part of the reference implementation of the language (which seems to always be ahead of the other implementations).
I know you're saying it's not a big deal, don't worry about it, etc. but it just doesn't seem that way to me. :(
FWIW, my personal opinion is that the future of threading in Python lies clearly in PyPy and STM. Their performance is nuts and the contributors are smarter than both of us. At some point the STM overhead simply won’t matter.
Would it have been a better call to work on problems like the GIL instead? Perhaps. Certainly many, yourself included, would consider that a more significant problem to solve than the comparatively minor touchups that Python 3 did to the language. However, I do tend to disagree there: I think that the slow accretion of cruft that bogs down a language over time is a more significant problem and more likely to cause a language to fall out of favor. And I think the timing works out in such a way that Python can recover the lost momentum from the transition in time for projects like PyPy to come up with a good answer to the GIL.
So, yeah, we disagree ultimately, but thank you for explaining your POV better. Upvoted.
> Every language bump (whether 2.5 to 2.6 or 2.7 to 3.0) has meant additional work for the developers of alternative implementations [...]
This is true, but I'd say the core Python developers haven't been entirely blind to this concern, cf. the 2 year moratorium.
And one still cannot write true multithreaded programs in Python using the language's standard thread library. To use all the cores, you have to resort to the processing module, which relies on processes rather than threads; this vastly increases the memory consumption, and makes the resulting programs less idiomatic as well. (You have to pass only pickle-able stuff to your child processes, for example.)
As for the GIL, it's such a dead horse. Threads are a very problematic parallel programming model and parallel threads would bring in an enormous amount of implementation complexity as well, if it even could be pulled off.
Each time we've checked out 3.x, we're like, "meh".
* Reworking of OS and I/O exceptions
* Built-in virtualenv
* yield from (proxy generators)
* Old Unicode literals allowed (but no-op)
python, python27, python3, python31, python32, python33.
That and half of pypi only works on < python3.
Even the JDK isn't that bad.
You didn't notice the moratorium on language changes which was in effect for the last couple of years?
This doesn't match my experience at all. Most distributions I can think of had no problem replacing 2.6 with 2.7, and have offered fresh Python 3 packages alongside them as well for years now. My distro of choice (Fedora) has dozens of Python 3 libs packaged.
Ubuntu is aiming for "no Python 2 in main" by 2014 now, and Arch even switched to Python 3 by default already.
I'd go as far as claiming that the distros are actually at the forefront of and partly driving the transition right now.
Fedora, Arch and Ubuntu are orders of magnitude too unstable for me to go near and I'm sure the same is true for a lot of other people.
More conservative distributions tread very carefully.
Well, yes, that's what conservative means. You're using stable distros, with older versions of Python, and it sounds from your original post like you're not interested in using newer versions right now. So... what's the problem exactly?
Python appears to be a playground for features versus a stable, reliable well tested and well thought out language these days.
And as for the standard lib, a lot of the additions e.g. in 3.3 is simply stuff that's been around and proven for years: virtualenv, mock ...
I mean, I respect that you feel that way, but I just don't see the same sort of cause for concern I guess.
Python has an extensive set of comparability tests and the system is quite modular. It also follows the maxim that complex systems that work are built from simple systems that work.
You don't have to use the new features. You don't even need to use Python3. Sounds more like "Damn kids and their loud rock and roll."
Fine example here (there are loads more like this):
That may be well and good for startups, or smaller companies, but larger companies with infrastructure running on CentOS/RedHat (and old-ish versions of them at that) still have trouble with Python 3. In CentOS 5 if you change the default python distribution to 3.0 > yum completely dies. Not to mention various other little bits and pieces.
Also we have many CentOS 5 servers but won’t let dictate our Python based software development by their Python 2.4. So we just rolled our own RPMs and put our python binary outside PATH. We deploy only via virtualenvs, so it’s not problem.
Sometimes the best solution to your problem is economically untenable, sometimes you currently don't have the engineering talent to allocate, etc etc.
You speak about economical factors, but doing broken deployments and using ancient Python versions (I’m talking 2.4 vs. 2.7 here, not Python 3) costs you much more money on the long run than building a new RPM every few months.
Nevertheless 2.7 (which was released two years ago) still gets bug fixes and no new features. So I really don’t get what this comment thread is about.
So it should be as fast as liblzma itself.
CPython developers quite often expressed their opinion against having JIT or more sophisticated GC because such changes would make Python interpreter too complex. (See Guido's keynote from last PyCon.)
Unladen swallow was supposed to be in CPython at one point.
(This assumes that your specific use case is covered, e.g. PyPy doesn't support Python 3 yet.)
[0]: http://www.cython.org/ [1]: http://pypy.org/
The biggest of these changes is likely the new adaptive system for the internal representation of strings which tries to pick a representation most suited to each respective string instead of using a "one size fits all" representation, to minimize the memory footprint and improve cache efficiency. Additional work that happened along side this should also greatly speed up encoding into UTF-8 and -16 and some string operations.
Another biggie is the new dict implementation, which should also significantly reduce memory footprint and improve cache efficiency (and keep in mind that object attribute namespaces are dicts, so that also affects all objects).
So, I for one always go with "Always use the right tool for the problem at hand.", which means I use Python for most things and something like C for the few things that have to be faster.
To answer your question: Python is fast enough for most things. It depends on how and what you are using it for though.
Am I the only one worries that Python is slowly turning into Java?
Instead of simple text/string for everything, now Class for everything, with nauty weird ill-defined Exceptions.
Hint: there’s more than /24 networks and IPv6 makes it even more complicated.
>>> str(ipaddress.ip_address('127.0.0.1'))
'127.0.0.1'
>>> str(ipaddress.ip_address('::1'))
'::1'
>>> str(ipaddress.ip_network('192.168.42.0/24'))
'192.168.42.0/24'
>>> ipaddress.ip_address('192.168.42.23') in ipaddress.ip_network('192.168.42.0/24')
True
>>> ipaddress.ip_address('192.168.41.23') in ipaddress.ip_network('192.168.42.0/24')
False
for addr in ipaddress.ip_network('192.168.42.0/24'):
print(adds)
edit fixed code markup
The example your are pointing out is for the module ipaddress which is for inspecting and manipulating ip addresses. It's not that Python now requires you to use ipaddress objects in place of strings for sockets.
In that case it's good to hear.