Python 2, Python 3, Debian and Porting
lists.debian.org
lists.debian.org
I've been disappointed by the "Python 3 Wall of Superpowers" and the fact that they've basically never updated their package list. (They update the Python 3 readiness, but you'd never find out if a popular package created in the last 4 years was incompatible, or if packages on the list have become unpopular.)
Today I found py3readiness.org, which is much more relevant and up-to-date. It has a better call to action, too.
Thanks for noticing py3readiness.org
Yes, it's up-to-date and takes advantage of https://github.com/brettcannon/caniusepython3
Also, you can help to keep it updated https://github.com/chhantyal/py3readiness
* Python-MySQL is abandoned, but it's been superseded by python-mysql-connector.
* oauth2 is kind of superseded by oauthlib.
* pathtools can probably be replaced with Python 3's pathlib.
* ipaddr can be replaced with iptools, which has the additional advantage that you can contribute to it without approval from Google's lawyers.
In above list, these are not drop-in replacements and it will require lots of changes in your application code if you want to use them. One idea is to show alternative packages https://github.com/chhantyal/py3readiness/issues/9
[1] https://news.ycombinator.com/threads?id=Animats&next=9379262
If you're interested in this effort, please email me. This is a really good new
contributor task, so if anyone's asked you how they could get involved with
Debian, you should send them to us!Yes, it's hilarious that version of Mailmain needs to be ported to Python 3.
Is there a list of projects that need porting? Would be good to see what sort of size they were.
Major things that would be useful to port for Debian's sake, because they're system utilities: supervisor, Fabric, carbon
Major things that would be useful to port because Python projects depend on them: Twisted, thrift, protobuf
Don't bother with these because replacements already exist: MySQL-python, oauth2
For example: I'm looking at ./main/f/flask-wtf/flask-wtf_0.10.2-1.json and I'm not even sure what I'm looking at. The contents of that file are:
{"has_python3_module": false, "canidate": true, "trove_python3": true, "has_python2_module": true}
What's all that mean?On my machine `apt-cache search flask-wtf` finds a package called `python-flaskext-wtf`. Digging deeper with `apt-cache showpkg` I find the homepage http://packages.python.org/Flask-WTF/, which in turn gets me to the code https://github.com/lepture/flask-wtf. I haven't updated the code to Python 3, but let's assume for a second that I have: now what? How do I tie this all together? Do I just send you a patch for this code? Is this even the right code?
If you point me in the right direction I'll be happy to write up a tutorial to help other folks contribute.
And as a side note: I really like Hy.
has_python2_module: is a python2 package
has_python3_module: is a python3 package
trove_python3: Is is declared as python3 ready in Pypi ?
canidate: Python 3 module already there and Python 3 compatible on Pypi
I published it as a csv to better understand the situation: https://github.com/mdamien/mdamien.github.io/blob/master/sta...
Sorry, I didn't document this since i'm rushing stuff out as soon as I get it together -- I was looking at the Trove classifiers, and setting candidate to true iff it says it's Python 3 compatable but no Python 3 package :)
I see for example, openpyxl_1.7.0 is listed as needing attention. Looking around it seems that openpyxl supports python3 now, so I guess whatever has that dependency needs to be checked?
Binary packages are what's found by apt-cache search and installed by apt-get install.
You can install a Debian source package using:
apt-get source flask-wtf
(This just downloads and unpacks it into your current directory, so you don't have to be root - it's not actually modifying the installed package database).Information on the source package is available here:
I don't expect Unicode trouble to disappear with Python3, quite the opposite, it's trivially to insert ticking Unicode time bombs into Python3 code, especially in things surrounding shell scripting. Something as trivial as this:
print(os.listdir())
Will explode when it runs into filenames that aren't encodable by Unicode (i.e. perfectly legal filenames in Linux). Linux allows raw binary in a lot of places (filenames, arguments, environment variables) where Python expects Unicode.However, there are ways to do this safely. At the boundaries of your code you have to deal with the unicode/bytes question, in any language out there. The good thing about Python 3 is that once that boundary is crossed, unicode and bytes are nicely separated.
The problem is that `print(some_string)` is now incorrect, even so it looks perfectly harmless. So far I haven't seen any "best practice" guide on how to fix that. `sys.stdout.write(os.fsdecode(some_string)` works, but is pretty ugly. Handling all filenames as bytes instead of strings works as well, but again, that gets ugly quick. Cleanest way seems to be:
PYTHONIOENCODING=utf-8:surrogateescape python3 somescript.py
But should I depend on users setting `PYTHONIOENCODING`? Should I do that in my scripts? It feels all very rough and causes a lot of issues that you really shouldn't have to deal with in the first place.> The good thing about Python 3 is that once that boundary is crossed, unicode and bytes are nicely separated.
Not exactly "nicely separated", the unencodable bytes get squished as surrogates into the Unicode strings waiting to cause trouble later on.
From my experience most people don't care anyways. In some of my libraries the Python 3 version just refuses to work unless the encoding is utf-8 to avoid accidental failures down the line which nobody can debug.
Arguably this was always a problem, because there were values in your program that you weren't sure whether they were a byte sequence or a string. But in practice, under python 2 you could regard a filename as an opaque "handle" of type "string" (that would in fact be a string on some platforms and a byte sequence on others), and as long as you didn't need to manipulate it directly, everything would work.
Python 3 can't do that, so it desperately needs a better cross-platform way of handling filenames.
Why are you worried about python 3.0?
There was a recent LWN.net submission about the other carrots:
If you need to print byte strings, you use
print(repr(os.listdir()))
That's worked since Python 2.6.Filesystem encoding is somewhat of an exception, but there's no good solution to this. In Python 2, the above would print binary characters, which is generally not what you want, potentially confusing the terminal emulator. In Python 3, you have to explicitly specify whether you want to output the original bytes or replace any bytes which aren't valid Unicode:
>>> os.listdir()[0].encode(errors='surrogateescape')
b'\xfe\xb1\xb7\x95[\xc9t\xc3\x18y'
>>> os.listdir()[0].encode(errors='replace')
b'????[?t?\x18y'
(the latter matches the behaviour of "ls")The equivalent to print(os.listdir()[0]) would be the following:
sys.stdout.buffer.write(os.listdir()[0].encode(errors='surrogateescape'))
This is more verbose, but also much more explicit, which is a good thing. If you want this as the default behaviour, you can set the environment variable PYTHONIOENCODING=utf-8:surrogateescape (or alternatively "replace"), and the following now works, just as with Python 2: >>> sys.stdout.write(os.listdir()[0])
(binary output)
>>> print(os.listdir()[0])
(binary output)
Alternatively, you can do the following: >>> sys.stdout = io.TextIOWrapper(sys.stdout.buffer, errors="surrogateescape")
>>> print(os.listdir()[0])
(binary output)
Note that print(os.listdir()) would not blow up since it's a list and Python would just call repr() on it: >>> print(os.listdir())
['\udcfe\udcb1\udcb7\udc95[\udcc9t\udcc3\x18y']
Judging from the responses in this thread, it's either not properly documented, or people aren't reading the docs.There is some discussion about making "surrogateescape" the default sys.std{out,in,err} encoding error handler in Python 3.5 in order to avoid surprises.
Otherwise we're going to end up with a similar situation as BASH and other utils in a few years time, where OSX comes with Python2.7.x, and everything else in the world is Python3...
(Although, by 2020, apple will probably have deprecated OSX in favour of iOS, and have javascript as the only blessed scripting language...)
Apple have invested quite a bit into JS recently, with their new LLVM JIT for Safari, I believe? (OK, by Apple standards, not much investment, but the fact that they did it at all says quite a lot...).
Since they did all that work, and are committed to maintaining it now, would it not (in some way) make sense to then use the same pluggable scripting language VM into other parts of the OS, and deprecate applescript, etc?
$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
$ brew install python3 pypy3
(to be less pithy, it's super easy to write code that will run on both 2.7 and 3.x -- avoid print-as-statement and use format strings instead of '%' for interpolation and that will get you 95% of the way there--probably 99-100% of the way for simple automation scripts)To be clear, %-formatting has not gone away in Python 3: https://docs.python.org/3/library/stdtypes.html#printf-style...
However, I don't really want (if I can help it) to install homebrew on all the workstations of everyone on the team (mostly video-editing), and for personal laptops, I don't really want to add that as a requirement. I certainly don't want to turn my < 100kb of scripts into hundreds of megabytes of requirements.
I'm already having to bundle an up-to-date static compiled standard rsync rather than the apple one (to cope with extended attributes across SMB mounts...).
It would be really nice to not have to think the whole time "Is this compatible python2 and python3", but to just write pure python3.
A lot of our scripts are to do with managing and synching all the files needed for video-editing, and other media-management utilities. I already dumped BASH (my initial attempt), as it failed or became ludicrously verbose when dealing with things like folder names starting with '-' or '*', newlines, quotation marks, tabs, or other "special" characters in file names. I don't know HOW they managed to get newlines into filenames, but our editors are creative people. I wouldn't be surprised if they managed to embed fonts and colours next.
So it does involve quite a lot of string processing, our automation scripts. Our tape archival system is CentOS, our main Server is a Synology RAID, and all the main edit stations are OSX, with the possibility of adding Windows boxen in the future. And since not all our projects are in English, everything is UTF-8, and mostly 'just works'. But I don't think the transition to Python3 is going to be 100% seamless. Hopefully unittests help, but I'm sure there are cases I'll miss.
Am I missing something? Did Debian decide to extend their release cycles?
Jessie is due to be released next week, which is 2 years since Wheezy (May 2013). In fact, the last several years have all been 2-year cycles, not 3. The only three-year cycle was Woody → Sarge (2002-2005), and AFAIK, that delay was one of the reasons that Ubuntu had room to enter the market. Since this put pressure on Debian to make their development more rapid, I'd be surprised if they decided to move in the opposite direction.
Also, our releases aren't timed, they're based on bug count, so the timing of the releases wasn't even the main point (and doesn't really matter) :)
I also typoed Python 3 as Python 2 in one of the lines, so, uh, just switch those two numbers. I'm pretty sure it works like that.
Either way, getting Buster off Python 2 means the transition will be more bug free, since we'll have a bit of slack time.
Currently we are waiting for a last few packages to be ported (anaconda is already running on python3 and ready for Fedora 23 release). Ironically the hardest part isn't porting (we've sent a huge amount of patches to upstream projects) but getting upstreams to accept the python3 as supported interpreter ("what is this python3 thing?" yes, this is a real quote).
I am happy to see a new distros joinning a party, gl!
See http://conda.io
Disclosure: conda is open-source, but I work for company that sponsors most of the development.
https://virtualenv.pypa.io/en/latest/ https://virtualenvwrapper.readthedocs.org/en/latest/
I use Anaconda on Win7, like it a lot, but I haven't figured out how to use, say, the python 2 that comes with anaconda alongside the python 2 that comes from python.org. I've looked around, haven't really understood what I should be looking at for that. But whatever it is, it isn't virtualenv if you use Anaconda.
True, but conda has its own virtual environments that work quite nicely. You can make a new one with conda create and switch between them with activate/deactivate. You even get the same sort of interaction with pip (for the packages you want that aren't in conda or binstar), and I've been able to use e.g. Emacs packages that expect to work with virtualenv and have (mostly) had success using conda's envs instead.
How would you use conda's environments to switch between Andconda's python 2 and python.org's python 2?
"Why would you want to ..." answers are generally pretty frustrating. It's a big universe with a very large possible set of individual circumstances.
Can this be done?
I'm not sure "whatever was built" refers to in this case. Some kind of app? Of course you can use anaconda for normal library or application development, since there's functionally no difference between Continuum Analytics' python builds and the python.org python build.
C:> python -c "print 'hello'"
helloI cannot get used to typing print()
Every time I go to debug a line, I get a syntax error, and have to go back and change it. And I type a lot of debug lines!
I need to find a vim extension to handle this, but so far it just makes me frustrated whenever I try to type Python 3.
For a vim extension - I use snipMate http://www.vim.org/scripts/script.php?script_id=2540 It's a bit on the dead side but works great as-is. ifmain followed by the expander key (shift-tab in my case) expands to the if __name__ ... idiom. You could add your own "p" snippet that expanded to print() in python code.
The reasoning probably was "Hey, the rest of the Python community would probably leave us in the dust, if we don't..."
I've been meaning to check out Python 3 for a couple of years now, but so far I did not have a compelling reason. Maybe it's time to start anyway.
Instead, at the time, Wikipedia claimed that Python 3 ran somewhat slower on CPython than Python 2. :(
Maybe all of those minor changes taken together constitute a big improvement, once I give it a try. The Unicode handling is said to be much better, and Unicode handling does suck really, really hard on Python 2. So there's that.
Blender has been on Py3K since 2008 and is an underrated software platform.
Asyncio is a revolutionary concurrency framework based on coroutines. The "yield from" feature is useful both for coroutines and for other generator functions.
Python 3 is happening, and that's the reality.
Unfortunately, I think what this means is definitely up for debate. Python3, "happening" may be nothing more than a permanent split, not the obsolescence of 2. In fact, that's what it looks more and more like everyday.
What IS happening, is that libraries are not, will not, and cannot drop 2.7 support. The opposite isn't true and may never be.
Again, its the trade off between a relatively simple port and adopting a new language with different standards, technical difficulties and culture.
It would be better for everybody if we'd treat Python 2 and Python 3 like separate languages. Then we can reason about the sanity of spending development time to port existing and working code from one language to another.
Python 2 -> Python 3 is a trivial port compared to adopting a new language and the technical/cultural baggage that come with it.
Perhaps Debian should avoid the war entirely and switch to another scripting language.
It's like the jump between Ruby 1.8 and 2.0 - there is no reason to be doing anything in 1.8 in 2015.