Django's future, and Python 3
djangoproject.com
djangoproject.com
1. http://scipy.github.com/faq.html#do-numpy-and-scipy-support-...
By any means, if you feel the urge, you really should help yourself and scratch that itch.
I understand that for someone new to python (hence lacking the knowledge to get what's different between 2 and 3) you might feel uneasy about it, but really you should not.
Just learn py2.7 (+ django if that's you want) now, and bulletproof your code by using u"", b"", .encode and .decode at the edge when needed, also 123L in your code, together with from __future__ import division, print_function, and whatnot. The remainder is mostly syntax (like except) or namespace (like http.server) changes and will get caught and thus easy to fix when you try to run in py3 (contrary to if you don't use u""/b"" as values can "move" through code and blow up in a completely unrelated area). I might be forgetting something, but that's the gist of it.
As for learning, really you will have zero trouble acclimating yourself to any breaking change py3 introduces. The trouble only lies in existing, 'legacy' code, not for the one you write from scratch straight in py2.7, and especially if you aim for compatibility with the provided forward-compatibility features of python.
For the most part Python 3 removes syntax and functionality that was long-since depreciated or at least strongly advised against.
b'foo'
The list you've given looks like a good start - is there a definitive list you'd recommend anywhere?
[0] http://docs.python.org/release/3.2.1/whatsnew/3.0.html
[1] http://docs.python.org/release/3.2.1/whatsnew/3.1.html
[2] http://docs.python.org/release/3.2.1/whatsnew/3.2.html
[3] http://www.ibm.com/developerworks/linux/library/l-python3-1/
[4] http://www.ibm.com/developerworks/linux/library/l-python3-2/
It's been a long, drawn-out process in part because everyone uses Unicode strings (and if you're not, you should be). Python 2.x let you be fairly lax about unicode vs. legacy/byte strings. Python 3 forces you to clean up your act and be explicit. Other than that the two languages are -basically- the same. This isn't like C vs. C++ (which share an unhealthy fetish for curly braces but are quite different under the surface), but rather more like K&R C vs ANSI C, or C++03 vs C++0x.
In other words if you're waiting for Django/SQLAlchemy/whatever to transition, don't bother. Learn Python 2.7 today, use it, love it, and transition to Python 3 only when all your dependencies have made the switch. You'll then have to change a few minor things like "raise Exception, reason" to "raise Exception(reason)", but I'm sure you'll adapt just fine.
It wasn't until a month later when I went to deploy that I discovered Python 3 wasn't what everyone was using. It was a small app, so it only took me a a day to port it to 2.6.
My point is go ahead and learn Python 3. The hurdles of back-porting are pretty minor when you don't have any version specific habits.
GUI toolkits like wxPython and PyQT/PySide supporting Python3 will also be important.
I'm not sure how outdated this is, but there's a list of most wanted Py3k packages available here: http://www.python.org/3kpoll
https://github.com/acdha/NativeImaging
There's also a very new project started here at PyCon to build a completely native Python imaging library:
https://github.com/ojii/pymaging
Obviously it's nowhere near as fast, although it'll be interesting to see how PyPy performs
This seems like a silly thing to do. It would make for much more readable code to use numpy arrays. But regardless, the pixel data should be one single array of data.
It looks to me like the guy working on this hasn’t spent tremendously much time thinking about good image APIs. Is there any document that describes the design goals and process?
I suppose it doesn’t really matter. This library is also missing all the other flexibility you’d want in a general imaging toolkit – e.g. the assumption of 8 bit/channel RGB is baked into everything, with no notion of color space or gamma encoding (and the JPEG decoder doesn’t seem to have any mention of color spaces, meaning that many JPEG images will be wrongly interpreted). It’s not usable for any but the most basic image processing tasks.
The JPEG decoder might make an okay benchmark for pypy performance, though it’s a straight port from C++, so it’s probably not the style of python code that pypy is designed to be fast on.
I'd be reluctant to condemn a project which is a few days old. Having spent time wrestling with optimizing the horror which is the PIL code, I'd definitely suggest that having a pure Python implementation will make it much easier for people to experiment with API and performance improvements.
So only now they are moving from depending on 8 years old technology to depending on 6 years old technology.
The reason for this is probably the proliferation of really old versions in the default installation of still supported OSes like RHEL. Worse: Due to the fact that these distros largely depend on the Python installation they came with, it's likely that admins are reluctant to touch anything python related on the system.
Compare this to ruby/rails where the common practice is to treat system ruby as a hands-off thing: Usually, the deployment payload just includes its own ruby (using rvm or rbenv). This allows for much less conservative minimum requirements.
Being forced to constantly target way outdated software during development is really annoying and kills motivation as you are always reminded that if only you could update, you would be spared re-implementing feature x or you could use much more expressive code while implementing feature y.
And don't get me started on PH-4.x is good enough-P.
I am not trying to say Django is a better framework better for making this decision, but it is a decision that makes sense in the Python context, which is careful about breaking backwards compatibility. There's no reason to stop a developer who has that choice to work with Python 2.7 though (this is actually recommended in the FAQ). There is no need to touch the system's Python to do that.
Usual practice is to use virtualenv which allows for an up to date python on OSes with an ancient system python.
Virtualenv is now on the ascendancy but it wasn't always like this and there were many people who had workflows based on different tools including zc.buildout (for example). Virtualenv has gone from an up-and-comer to a top-dog over the last 4.5 years.
Django core devs have tried to be hands-off on dev-op issues perhaps to a fault.
Rails team: "New Ruby alpha? Deprecate everything!"
For the uninitiated, Ruby's tiny version is the equivalent of Python's minor version (Ruby 1.9.2=>1.9.3 ~ Python 3.2=>3.3).
Rails 2.3 supported almost the entire Ruby 1.8 series (except for one release, 1.8.3 I think, that had a serious bug). Rails 2.3 had experimental support for Ruby 1.9.1 when it shipped, but most people used Ruby 1.8.6 or 1.8.7, as Ruby 1.9 was still somewhat unstable at that point.
Rails 3.0 (which also shipped in August 2010) dropped support for Ruby 1.8 versions before Ruby 1.8.7, and shipped with stable support for Ruby 1.9.2. Ruby 1.8.7 shipped in May 2008, and was very compatible with Ruby 1.8.6. We (the Rails team) continued full support for Rails 2.3, which supported Ruby 1.8.6, until August 2011, when Rails 3.1 was released.
We have announced that Rails 4.0, which we still have not shipped, will drop support for Ruby 1.8.7. This means that we have dropped support for one version of Ruby (we will still support 1.9.2) in more than 2 years. That hardly qualifies as "New Ruby alpha? Deprecate everything!"
Short timeline:
- August 2006: Ruby 1.8.5
- March 2007: Ruby 1.8.6
- December 2007: Ruby 1.9.0 / Rails 2.0
- May 2008: Ruby 1.8.7
- June 2008: Rails 2.1
- November 2008: Rails 2.2
- January 2009: Ruby 1.9.1
- March 2009: Rails 2.3
- August 2010: Ruby 1.9.2 / Rails 3.0
- August 2011: Rails 3.1
- October 2011: Ruby 1.9.3
- January 2012: Rails 3.2
Releases of Rails 2.x all supported Ruby 1.8.2 and above. Releases of Rails 3 have supported Ruby 1.8.7 and Ruby 1.9.2 and above. The only change in Ruby support since Rails 2.0 was requiring at least Ruby 1.8.7 in Rails 3.0. There is an announced change for Rails 4.0, but Rails 4 has not been released yet, even in betas.
Rails is moving more quickly at adopting Ruby 1.9 than Django is at adopting Python 3, but that is because it was possible, from the get-go to support both Ruby 1.8 and Ruby 1.9 in a single codebase. That was not possible with the move from Python 2 to Python 3.
The typical argument I've heard about the difference is that Ruby 1.8 to 1.9 was a relatively smaller release. I think this underestimates the work that the Ruby team put into compatibility while developing Ruby 1.9, especially given the large encoding-related change made to Strings in Ruby 1.9 (Ruby 1.9 Strings are encoding aware, while Ruby 1.8 Strings are semantically simple byte Arrays).
How can the Core team claim to support 1.9.2 in Rails 4.0 when the minimum version of Ruby required to run the test suite is 1.9.3?
Again, Rails 4.0 is not out yet, even in betas. We're currently in the planning stages, with a lot of movement on master, so master is in a state of flux at the moment that doesn't really reflect what a release will look like.
"The master git branch for Rails is now targeting Rails 4.0, which will require Ruby 1.9.3 and above."
All Rails core team members have running apps on the latest version of Rails, and Ruby 1.9.3 was the first version that everyone felt comfortable shipping to production with. So Rails 4 removes support for Ruby 1.8.7, and requires the first "stable" version of Ruby 1.9, based on our experience bringing apps to production.
It'd be nice if things could move a little faster. But the alternative is wild-wild-west.
Scala/Play! I was fed up of waiting.
Django, from my point of view, seems to be approaching this from the wrong direction. They are waiting for the community to shift to Python 3, but at the same time the community is waiting for Django to shift.
When Django 1.0 was released, we supported minimum Python 2.3. Dropping 2.3 and then 2.4 were major logistical hurdles; they were incredibly widely deployed, and were the stock Python for many platforms with long-term support. So it was going to take a while, no matter what, for us to sensibly deprecate those Python versions without just throwing a bunch of our users under a bus. And, of course, deprecating everything older than 2.6 is a prerequisite for starting to actually adopt 3.x.
Meanwhile, the first few releases in the Python 3 series weren't really meant for widespread adoption, ever. They were there so people could start getting a taste of Python 3 and working out a migration process while shaking out remaining issues for Python itself.
We are now at a confluence of two very happy things:
1. We've dropped our support for the older Python 2.x releases, without just screwing over a bunch of people, and by year's end we'll be a minimum-2.6 project, and
2. Python 3.2 (the current 3.x) is really quite a nice release, and Python 3.3 (later this year) looks like it'll really put the spit and polish on the Python 3.x series, meaning it's a very attractive time to be targeting Python 3.
This is the first time we've realistically been in a position to actually make the jump, and we're taking that opportunity.
I'll hold back on commenting now because my insight into the Python community is limited to what I hear from friends who develop on Python. Though something I've noticed is that many of them are making the switch to Node.
The real problem is that Django isn't the only problem. I've started cataloging the packages that are supporting Python 3 in my project, and there are still several that are critical to it that haven't made the transition. My web framework is the least of my concerns in that regard (considering it already supports Python 3).
developers have choice. when they're starting new app / new project they can opt for the older version or if they're feeling lucky adopt the new stack.
the important thing is developers should have a choice, because i firmly believe more choice leads to more innovation.
its not like when rails 4.0 come out they're going to force everyone to upgrade, you can upgrade at your own pace.
Use what seems best to You. Rails always tend to have faster movement with Ruby support, but to be honest having at least 1 year is enough to redeploy app with new Rails & Ruby. If not then you're doing it wrong.
All major projects seem to be adding support up to the latest Python 3 version, not the 4 year old 3.0 one.
This is a sincere question by the way, just tempered by post-PyCon weariness.
Sorry to bore you. Back to your regularly scheduled hacking.
Get custom T-shirts printed? ;)
Not that I care that much for environments stuck in 2.4 land, its last update was 7 years ago.
Why get comfy with Python 3 when you must remain attuned to the conventions and quirks of 2.x in order to use Python's most popular web framework? Folks keep talking about Python 3 because they really want to use it, but have reservations about adopting 3 when 3 isn't supported by useful frameworks.
Once Django supports Python 3, that reservation vanishes. Python 3 will be adopted by more people and get support from more frameworks.
tl:dr; Django's popularity holds dev community captive to Python 2.x, slows adoption of Python 3. Django team announces Python 3 support not in distant future but near future. Folks get excited. "About damn time."
You're severely underestimating the impact that the scientific computing community has over Python 3's slow adoption.
Other libraries I use are smaller, can be replaced, or I could rewrite them.
I am quite sure that I am not in a minority.
Maybe my standards are too low since I am using Django as a vehicle to learn programming and web-development as a hobby right now, but as far as I can tell the Python/Django environment seems to work exceptionally well meeting, and even exceeding, its intended objective (The web framework for perfectionists with deadlines)
It seems a lot of people have this child-like concept of time where a few months is a "long time" and upgrading to the latest and/or greatest thing is so much of a priority that they are regarding it as a "fix". But as far as I can tell nothing is broken by having Django run on 2.x. If a bicycle works really, really well; is replacing the chain a top priority?
... and it always will be.
I don't understand what makes phraseology like this useful to anyone but a corporate lawyer: "the Django project is and will remain committed to achieving Python 3 compatibility"
Yes, I know, I'm not supposed to talk like this about work being done by volunteers free of charge. It's of course their right to do whatever they like. My anger is more about the Python 3 migration going off the rails in a way that poses an existential threat to my favorite programming language. Django isn't helping at all to say the least.
In fact, that's exactly what we're doing. Django's going to support Python 2 for quite some time yet; this move will simply let those who want to switch to 3 do so.
> I don't understand what makes phraseology like this useful to anyone but a corporate lawyer: "the Django project is and will remain committed to achieving Python 3 compatibility"
Because we weren't clear that Python 3 support was in the pipeline some folks in our community started to wonder if we planned to move the Python 3 at all. That's pretty much the point of the post: to clarify that we do indeed plan to port to Python 3 (and when). How would you have worded it differently?
> Yes, I know, I'm not supposed to talk like this about work being done by volunteers free of charge.
That's not true at all: if you have substantive critiques to make, then you should make them. We try incredibly hard to listen to our critics and take away what we can. However, your criticism here is pretty thin and rather personal -- seriously, accusing a bunch of volunteers of being like Oracle or SAP? Low blow, man, low blow. It's hard for me to figure out exactly why you felt compelled to post this criticism, so it's hard to figure out what the action you'd like us to take.
> My anger is more about the Python 3 migration going off the rails in a way that poses an existential threat to my favorite programming language. Django isn't helping at all to say the least.
OK, so here we go -- there is something you'd like to see us doing differently. But what?
I can totally understand that you don't like my Oracle/SAP comparison and I do apologize for the tone, but there is actually some substance behind it. Oracle and SAP provide core transactional capabilities to very large corporations. These corporations make money based on their scale, not because they are particularly nimble on the IT side.
They're not startups. They rarely do any real greenfield stuff. They don't hire developers who want to be on the bleeding edge. Their IT culture is one of mediocrity and that's why it doesn't matter how many generations/years the software they use lags behind the latest platform version.
I would have thought and hoped that Django is not just for that type of corporation.
But judging by the reactions here you seem to be in sync with the majority of your users, so the problem isn't yours so much as it is mine. I should probably not get agitated about it and just move on.
* Database drivers (psycopg2, MySQLdb, etc.) weren't updated for Python 3 until... last year some time, I think? Django supporting Python 3 is somewhat academic if you can't use a database with it, don't you think?
* The WSGI spec wasn't updated for Python 3 until summer 2011 (July, IIRC). Again, Django supporting Python 3 is pretty academic if you can't run it under a web server, right?
* The previous suggested approach for dual Python 2 and 3 support (using 2to3) proved to be untenable for Django [1], and it wasn't until this last year (or so) that a better approach (single shared source) emerged.
So to my mind, "as soon as possible" was something like July 2011. We had unmet dependencies before then; moving would have been... well, not impossible, but incredibly painful and nearly useless.
I will, however, take full responsibility for not getting Python 3 support shipped between then and how. Our current release workflow [2] got in the way, and about 7-8 months because of it. That's my fault, and if you want to rake me over the coals for those months go right ahead: I deserve it.
[1] Details omitted to be concise, though I'm happy to share them if anyone's interested.
[2] Which we're changing, mostly because of this silly workflow-induced delay.
I'm sure you could have found a way to talk to web servers without waiting years for the WSGI spec to be finalized. And if Django had supported Python 3 using py-postgresql, which was released in 2009 (I know Python 3 only), others would have worked a little faster to make their database systems available on Python 3 as well. Upgrading psycopg can't be such an incredibly difficult task either.
The reality is, everyone has basically been waiting for everyone else to make a move and it has resulted in deadlock.
How exactly does one create a fact? That would be a neat trick.
> I'm sure you could have found a way to talk to web servers without waiting years for the WSGI spec to be finalized.
Indeed, we could have, and it would have benefited Django at the expense of diversity in the Python web world. It probably would have helped our "market share". But that's not how we roll. I'd rather work on something that the whole community can benefit from, and that means standards and specifications.
> And if Django had supported Python 3 using py-postgresql, which was released in 2009 [...]
That's true - I'd forgotten about that library.
> [...] others would have worked a little faster to make their database systems available on Python 3 as well.
That... I'm not so sure about. Volunteer community, limited by free time, etc. I'm fairly sure most people have worked on this stuff just about as fast as they can.
You do realize, of course, that the criticism here applies to you as well. If you feel this strongly you could have added a py-postgresql backend in 2009, you could have come up with a way to run Django under Apache and Python 3, etc. I'm happy to take a lot of the blame for Django's slow movement, but you're also responsible for not channelling your frustration into action.
> The reality is, everyone has basically been waiting for everyone else to make a move and it has resulted in deadlock.
I don't see a deadlock. I see steady movement -- slow, too slow, but movement nonetheless -- towards Python 3. First Python 3.0 came out, and it was horrifically slow, taking away any impetus to use it. So people fixed that. Then, developers ported the low-level libraries (database drivers, network protocol implementations, etc). Then we started moving to higher level libraries and tooling (numpy, wsgi, ...). Now the highest-level libraries (web frameworks, scipy, matplotlib, ...) are porting. Next, and finally, you'll get to port your apps. It's a classic dependency upgrade chain. You can't upgrade layer N until layer N-1 is done.
I share your frustration about the slowness, but that's frankly a characteristic of the Python community. We're relatively conservative, tending to prioritize stability and backwards compatibility over rapid movement. Everyone pretty much expected this process to take a while, and turns out they're right.
As I said, by implementing something instead of waiting for the spec. I totally disagree that it would have helped Django at the expense of everyone else, unless you patented it. Any implementation, even lots of fragmented ones, would have been better than years of stalemate. I guess we'll just have to agree to disagree on that one.
You do realize, of course, that the criticism here applies to you as well
Yes, absolutely, that's a very fair point. I just didn't expect things to move _that_ slowly. With the benefit of hindsight I want to punch myself in the face for not doing it myself.