Hacker News Reader

Comment by antileet | Hacker News Reader

Hi Zed,

Can you please elaborate with some details about how "broken" python is on systems, and the situation with python 2.4? According to the article, "recent Ubuntu releases" had broken/deprecated python installations. A quick search reveals that python 2.5 has been shipping as the default "python" meta-package(?)

http://packages.ubuntu.com/search?keywords=python

Can you also please elaborate any "features" that the install was missing - are these specific modules ?(I think python's sqlite3 is missing by default on Ubuntu. I'm not too sure.) I'm not questioning you or anything, I'm just slightly concerned as we had plans to ship a Python application, and when someone experienced enough has a valid complaint, it makes sense to understand the basis behind that.

View on HN

It was a fairly small chunk of Ubunutus that had problems with Python versions. I made sure m2sh-py worked with 2.5 (since I had an OSX around with 2.5) and I ran into a couple people with what was a recent Ubuntu, but with a 2.4 Python. Boggled the mind, but then again people do really weird stuff with their systems.

If your application is going to depend on Python then you definitely need to evaluate who's using it. If it's like mongrel2 where sysadmins will need to install it on various Linux distros they might not control, then you're screwed. You'll need some kind of installer that hooks up the right kind of python in a safe area.

If it's more like a Desktop application, then just get it to work on the latest of the desktop Linux variants, and then have it bomb out if they don't have the right python. People on Desktop systems are used to having to upgrade to use software.

I'd very much like to know why it wasn't easier for him to simply write a Python code that doesn't use anything not existing in 2.4.

Just because he downloaded to his computer whatever later version he likes he can't expect from all distros and all users to have exactly the same version across of all the currently running computers, it's more than unrealistic, it's simply impossible and always will be.

I'd very much like to know why it wasn't easier for him to simply write a Python code that doesn't use anything not existing in 2.4.

This isn't an "oh, a brand-new version just now came out and it's not in the distros yet" problem. Python 2.4 is six years out of date at this point. Python 2.3, which Red Hat will support until 2012, is even older (and for the longest time Red Hat didn't even use a 2.x Python at all -- they sat on 1.5 forever).

That's simply unacceptable, and creates endless headaches for people who want to distribute software written in Python.

Why is it "simply unacceptable" to produce the code to the least common denominator?

Anything that is successful must have more versions in the installed base. You can target a single version of something only if it's not used at all. It seems too much developers live in "Everybody must have that what I have" world.

I haven't tried myself, but I can't believe Python changed that much that you can't write the plain Python code to the least common denominator of versions >= 2.4. If you have the specific examples why you can't I'd like to know them.

Edit: I've rechecked, he actually mentions 2.2 as the lowest version he saw, still I believe it doesn't change too much.

You can write code that runs on 2.4, yes. But you miss quite a few useful features of more recent Pythons by doing so (context managers, for example, are a big deal, as are several of the newer standard-library modules). The same thing used to happen with Python 2.3; you could write 2.3-compatible code, but it meant no decorators, no generators, etc.

Python has these useful features and has had them for years; why is it acceptable to say that it will be 5-10 years before we can reliably use anything new?

In his particular case, he certainly was not missing the newer Python modules as he was able to write a "small" C-only replacement. That what he wrote in C would certainly be easier to write in Python 2.4 (or 2.2)

Did you understand that he wrote the big C program just to process a small file with the syntax as in the following example (taken from his own file):

    main = Server(
        chroot="./",
        hosts = [mongrel2]
    )
    settings = {"zeromq.threads": 1}
    mimetypes = {".txt": "text/plain"}
    servers = [main]

I'm concerned with the general case of writing Python software. Zed's example perhaps could get by, but offers some minor annoyances.

Distributing a framework like Django, on the other hand, when distros lag six years behind current Python, is a royal pain in the ass.

Django is in a very different situation because Python is central to all of its users, therefore it is reasonable to expect that they maintain a somewhat modern (released in the last 3 or 5 years, say) environment. When Python merely performs a subsidiary role, and your users may never use it themselves, it is far more important to support old versions.

I mentioned Django as a counterexample; the conversation seemed to be heading toward using the single specific example of Mongrel to demonstrate that there either is not a problem here or that it's not particularly serious.

Meanwhile, anyone who depends on Python for much more than a config-file parser knows what a genuine and large issue this is, and that "just write to the lowest common denominator" is not really a reasonable solution.

My point is that the lowest common denominator is higher for users of something like Django because Python is a central component of what they do, thus it's reasonable to assume that they are not using an 8-year old Python environment.

Tell that to the many, many people I know who are using Django on RHEL. There's a reason why we just (as of Django 1.2) got around to dropping Python 2.3 support...

We can't drop Python 2.3 until 2012 (when RHEL4 reaches end-of-life).

However your replies started with your response "That's simply unacceptable" to my "I'd very much like to know why it wasn't easier for him to simply write in Python 2.4" You mentioned (in this context actually OT) Django much later.

And I still don't know if he tried adjusting his already existing Python code to simply be backwards compatible -- he never mentions that, neither why that would be a problem in his particular case.

Regarding Django, see the other replies here about side-by-side installations, it's more than reasonable if you have something big which always runs, but not reasonable for a small config script used once.

Sounds like he could've easily embedded python into his app instead.

"Big"? You keep using that word. I do not think it means what you think it means:

  $ rm src/parser.c src/cli.c src/lexer.c src/linenoise.c 
  $ wc -l src/*.c src/*.rl  148 src/ast.c
  646 src/commands.c
  267 src/config_file.c
   93 src/constants.c
   36 src/m2sh.c
   13 src/token.c
  186 src/cli.rl
  156 src/lexer.rl
 1545 total

That's dinky tiny, even with the linenoise and generated files it's only 4061 lines long, which isn't much at all.

My count is 4636 lines, and I can imagine much. much smaller Python 2.3 code:

    ast.c 115
    ast.h 45
    cli.c 482
    cli.h 31
    cli.rl 143
    commands.c 498
    commands.h 5
    config_file.c 193
    config_file.h 27
    constants.c 82
    constants.h 14
    lexer.c 360
    lexer.rl 121
    linenoise.c 433
    linenoise.h 40
    m2sh.c 27
    mimetypes.csql 851
    parser.c 1074
    parser.h 15
    parser.y 69
    token.c 11

No, most of that is generated from the .rl files or just SQL that's common to everything. Also, you'll want to throw in all of Storm and PyREPL if you want a real comparison.

Then again, 4600 lines of anything is tiny. You have a massively skewed view of "Big" and haven't disproven anything by finding 600 lines of cruft in one directory.

I fully agree with you that this is more than tiny for a C project, but it can be big compared to a small script. I also develop/maintain the projects measured in zipped megabytes (where I don't even have a wish to try to wait to count the lines!) I only compared it to some script solution.

You're right, 600 lines more or less are irrelevant. I just thought you're measuring something else as I saw the total of 1000 instead of 4000. Still it is all tiny for a real C program.

I also think that the C solution is more portable than the dependence on any version of Python. I also cross-compiled the code for 32 MB RAM mipsel platforms and I agree that only C dependencies are better than any script language dependencies (not counting shell, when it's carefully written).

But I actively use both Perl and Python so I'd still really like to know what was lacking in Python 2.2 or 2.3 or 2.4, of course only in case you already knew that you were to have any Python on the target computer. But then if you couldn't expect to have any Python, then the fact that distros still use older versions wasn't of much relevance.

What's the point of ever producing newer versions of a language with new features (or, golly, even bug fixes) if no one will ever use them because everyone has a 'lowest common denominator' mentality? More to the point, what is the process of changing the 'lowest common denominator' everyone's using?

Where do you get "no one will ever use them" from? We're talking about the darned config script which the guy first wrote in Python and then discovered that he would have to write the processing of that file in the lower version of Python than he blindly used or to use C. And he decided to use C, I still haven't read anywhere why exactly.

The platforms upgrades have their own dynamic. But you just shouldn't be surprised that somebody who has the server running for years doesn't want to install the latest Python only to process one config script. Not more not less.

In the Git project we target Perl 5.6 for critical code (like git add --interactive) and Perl 5.8 for some other code (like git-svn). Those are 10 and 8 years out of date, respectively. Or the equivalent of targeting Python 1.6 and 2.0.

It can be mildly annoying sometimes to have to use 5.6 features when I usually develop on 5.12 or newer, but Perl's policy of backwards compatibility makes this a lot easier than it would be in Python.

Git also has some Python code that targets 2.4, making it compatible with such an old version was relatively easy (see http://git.kernel.org/?p=git/git.git;a=commit;h=23b093ee087e... for an example). A lot easier anyway than rewriting and maintaining that code in C.

> equivalent of targeting Python 1.6 and 2.0.

I'm less in touch with Perl language development in the past decade, but my impression is that the difference between Perl-5.6 and Perl-5.12 is much less than between these versions of Python and recent releases. This is expected when a language evolves rapidly, but it's rather frustrating to wait 10 years before you can depend on the "new" features being available.

This backwards compatibility is a conscious decision. And it is argued about. To find a bit to read about that, Google e.g. for darkpan. (It is probably one reason why Perl 5 is so extensible.)

Thanks berntb! That's a perfect tip for a new submission:

http://news.ycombinator.com/item?id=1712404

The DarkPAN matters

"There is currently some FUD going about in some parts of the Perl community about why we should break Perl 5 backwards compatibility. A short blog entry, schmarkpan, is a good example of the trend: loud, noisy, but clueless and devoid of any content."

    My impression is that the difference between Perl-5.6
    and Perl-5.12 is much less than between these versions
    of Python and recent releases.

Your impression is correct. Perl is a more mature platform than Python, and backwards compatibility is taken more seriously. The latest Perl 5.12 release can still run most of the test suite for Perl 1 released back in 1987.

That and its wide availability and portability make it a much better target than Python for something like glue in a build system for a project that's mostly in C anyway.

But even though it's backwards compatible with old code it's still somewhat of a pain to write code that works on old releases that don't have the modules / core features you want.

Backwards compatibility also comes at a price. Many of the changes that broke old Python code broke it because old warts were being fixed in Python. There's a lot of equivalent old warts in Perl that haven't been fixed, and there's no plan for doing so (other than Perl 6).

  There's a lot of equivalent old warts in Perl that haven't been fixed

This is the crux of the issue. Serious language users (especially the ones writing the new libraries that add significant value to your platform) want the warts fixed to make the environment pleasant to work in. Meanwhile, most casual users are desperate for backward compatibility because it makes distribution easier. I actually prefer Python's choice, but pain is inevitable either way.

I should have said: They haven't been fixed by default.

Perl is also moving forward, it's just doing so differently than Python. Python has major releases like 2.6 and 3.0 where they explicitly break old code in the default installation.

Perl however doesn't break any of the old warts by default, but developers can easily do so by importing modules like Moose and perl5i in their own code. See http://search.cpan.org/perldoc?Moose and http://search.cpan.org/perldoc?perl5i

Perl's model means that you can usually upgrade any code base from say 5.6 to 5.12 without major headaches. But if you try to do the equivalent with Python 1.6 to 3.0 you'll run into trouble.

There are advantages to both models, but with Perl's you rarely see users with big legacy code bases staying behind on some legacy 10 year old release of Perl that may have bugs and security issues. Users usually just upgrade along with their OS without any major pains.

Which code was explicitly broken by Python 2.6?

iirc nothing was, but it made available some features that would be used in 3.0 and 2.7 is essentially a back-port of stuff the 3.0 people found useful. I believe the idea is that in a major version number, everything is back compatible.

> Perl is also moving forward, it's just doing so differently than Python. Python has major releases like 2.6 and 3.0 where they explicitly break old code in the default installation.

3.0 yes, but 2.6 is simply not true. code written for 2.2 will work in 2.6, bugs notwithstanding.

Almost, "with" became a keyword.

> Perl is a more mature platform than Python, and backwards compatibility is taken more seriously.

That's a silly thing to say w/r/t Python 2.x. What makes people’s Python 2.5 code not run on Python 2.2 (or 1.x) is that new features were added in the mean time. New very shiny happy features that make life a lot easier for developers. Almost all python code continues to work from one version to the next, and it’s easy to write code that will work on all the versions from 2.2 to the present (or whatever), as long as you’re willing to avoid all the nifty stuff.

Basically by “backwards compatibility“ you mean “forward compatibility”.

So did Perl 5. The difference seems to be that for the newer Perl features you have to either declare the version of Perl you're depending on or import the feature explicitly.

So after complaining that he could not use new language features that appeared six years ago, he went back and rewrote the whole thing using a different language whose syntax is much much more than six years ago.

I guess he must have found these must-have Python 2.5 language features that are so important and desirable in C.

And by the way, don't use new GCC features, because some ancient GCC versions will be supported by Red Hat until 2020.

For God's sake, support != enforce

It was users complaining, not me. I had no problem with Python, but all of the non-python users (the users of one of the 9 other languages Mongrel2 supports) hated Python for these reasons.

Did you by any chance have the users who haven't had any Python installed?

Maybe he's got a whole shit load of code for this thing? I was able to do a 1,000 line project that was 2.4 compatible just by looking at the 2.7 docs and avoiding anything that said it was introduced in 2.5 or later.

Not fun, mind you, because I really would have rather been doing it in 2.7 or 3.1, but not impossible or even very hard.

That doesn't solve his second issue, though, of user not wanting to touch something that depends on Python. Rewriting in C fixed both.

The config file written in the subset of Python, the only thing users would have to touch, hasn't changed! (edit: see http://news.ycombinator.com/item?id=1712269) He just changed how the file is processed, by implementing in C what would be certainly easier to implement in Python 2.4, (edit: 2.2) the lowest version he mentions.

If he insisted that users install Python 3.1 only to run his config script and they complained, I don't blame them. If he had users that haven't had Python at all, then I fully understand what he did. But then he can't blame distros and users for not having the latest and greatest Python version, it's just about the existence of Python on the target platform.

Just from my personal experience:

Red Hat Enterprise Linux 5.1 ships with Python 2.4.3.

Solaris 10/4 does not ship with python at all. OpenCSW provides Python 2.6.5 but virtualenv doesn't seem to work with it.

Actually, the linked search shows 2.6.5 is the default Python package. The current version is "lucid".

Speaking as someone who knows practically nothing here, I think the issue is existing systems, not brand new systems. The barrier to updating an existing system becomes greater the more and the longer things depend on it.

Reply on news.ycombinator.com