Comment by jerf | Hacker News Reader

Comment by jerf | Hacker News Reader

Parent Full thread

There are many examples of things that people now call "lack of foresight", but I think people quickly gloss over just how long Python has been around. It started in December 1989, when the first Pentium was still three years into the future. If Guido had tried to "foresight" his way to a language with an acceptable multicore solution at that point (on any hardware he could afford), the only possible result is total failure. I don't think there are many attributes the language currently possesses that were truly that avoidable. Most people's snap suggestions result in incurring costs to the language that it may not be able to recover from when it is young.

Personally, I think the truth is that Python is what it is; a very, very, very good OO language that has reached the mature stage of its life. It probably won't be able to make the leap to "true" multicore, and besides, even if it does it won't be likely to be very successful anyhow because it'll still be a very slow-but-powerful language.

(There's very little point in taking a language very near the bottom of the Shootout and adding 4 or even 8 way parallelism to it, when you could just rewrite the target hotspot in C and get the same performance on one core. The math just doesn't favor trying to add lots of "multicore" to such a slow language. Python's one of my favorite languages, but that doesn't mean I can't see where it is weak.)

PyPy (together with LuaJIT and tracemonkey) has been banned from shootout, partly because we complained about unfair rules. Looking at shootout to asses the relative performance of languages where you cannot compare high-performance implementations of a language is a very very bad idea. Looking at shootout to compare would very likely be a bad idea anyhow, but right now it's just completely useless.

Can you give details and/or a citation for the complaints and the banning? I use the shootout a lot to give me a rough idea of a specific language's properties, so if there are valid criticisms then I'd like to hear them, but your post doesn't really stand on its own to someone who doesn't know the backstory.

There is pypy-dev discussion about the subject that starts here: http://mail.python.org/pipermail/pypy-dev/2011-April/007139....

It's a long and boring read, but here you go.

As in the actual complaints, as far as I remember, but this was a bit ago:

* our versions got rejected because they perform slower on CPython (while being faster on PyPy) and because they're 'unobvious'. At the same time, the C implementation of a lot of things there is very unobvious.

* we were not allowed to use array.array module, despite being in standard library.

* you can use gmp library for C, CPython, but not PyPy via ctypes

* C can use a bad random (that's fast, but also gives a really bad distribution), while PyPy not because Python comes with a random in stdlib

* custom malloc libraries for C are fine, tuning JIT parameters per-benchmark is not.

And various complaints like this. After some discussion, Isaac just kicked us out alltogether. Overall, my personal impression of the experience was that shootout is there to showcase things that are predetermined and we're trying to attack the status quo. This is obviously very personal, I don't have any actual data to back up this claim.

READER BEWARE!

>>our versions got rejected because they perform slower on CPython (while being faster on PyPy)<<

Maciej Fijalkowski knows that I asked him to contribute a PyPy version of n-body because the written-for-CPython version failed with PyPy.

Maciej Fijalkowski knows that his own PyPy version of n-body "performed slower on CPython (while being faster on PyPy)" -- AND WAS NOT REJECTED.

http://mail.python.org/pipermail/pypy-dev/2011-April/007177....

Justify your accusation or take it back!

>> you can use gmp library for C, CPython, but not PyPy via ctypes <<

Joe La Fata's pi-digits code worked first time on x86 and x64, on PyPy and CPython and Python3; and used ctypes to get to GMP -- AND WAS NOT REJECTED

http://anonscm.debian.org/viewvc/shootout/shootout/bench/pid...

Justify your accusation or take it back!

>> C can use a bad random (that's fast, but also gives a really bad distribution), while PyPy not because Python comes with a random in stdlib <<

The fasta task uses random numbers, and every fasta program (including C and Python) implements the same random function.

http://shootout.alioth.debian.org/u32/program.php?test=fasta...

http://shootout.alioth.debian.org/u32/program.php?test=fasta...

Justify your accusation or take it back!

----

Tell Maciej Fijalkowski to justify his accusations!

Cool that you're linking to this mail, this shows yet another problem that I forgot about - that pypy extensions are uncool, while GCC extensions are cool (and say a custom malloc library for GCbench - after all the benchmark does not do anything, an empty malloc would do just fine).

Look Igor -- I don't care how things are now since PyPy got kicked out anyway, so I seriously don't care any more.

gmpy uses C extension, it's not using ctypes. We disagreed over a ctypes version at some point.

random - indeed it got fixed (and it's too late to edit a parent if you wonder)

>> we were not allowed to use array.array module, despite being in standard library. <<

2 April 2011, Joe LaFata contributed a faster for PyPy spectral-norm program. His program worked first time on x86 and x64, on PyPy and CPython and Python 3. His program was measured and published within 24 hours.

    from array     import array

http://shootout.alioth.debian.org/u32/program.php?test=spect...

Justify your accusation or take it back!

>> gmpy uses C extension, it's not using ctypes. <<

    import ctypes
    from ctypes.util import find_library

 	_libgmp = ctypes.CDLL(find_library("gmp"))

 	class mpz_t_struct(ctypes.Structure):
 	_fields_ = [("mp_alloc", ctypes.c_int),
 	("mp_size", ctypes.c_int),
 	("mp_d", ctypes.c_void_p)]

>> random - indeed it got fixed <<

Nonsense! The fasta task has ALWAYS required ALL the programs to implement the same random function.

(Use a different random function and the program output will fail the diff.)

>> so I seriously don't care any more <<

You don't seem to care that your accusations are demonstrably false.

seriously, I don't think this is the correct medium - however you did refuse our solution using ctypes and you did refuse our solution using array at some point. Maybe you changed your mind later. This is one example:

http://mail.python.org/pipermail/pypy-dev/2011-April/007169....

You also kicked PyPy out around the same time the array benchmark was introduced (within days).

>> I don't think this is the correct medium <<

You chose to make false accusations here.

>> however you did refuse our solution using ctypes and you did refuse our solution using array at some point <<

You don't seem to know what program was supposedly refused for using array - so why do you continue to make that accusation?

Presumably by "our solution using ctypes" you mean Alex Gaynor's revcomp program - I'll say something about that where you linked to his blog post.

>> You also kicked PyPy out around the same time the array benchmark was introduced (within days). <<

No more than coincidence -- Joe LaFata contributed excellent PyPy pi-digits, spectral-norm, mandelbrot programs in one week; and they were all accepted.

This is a full writeup: http://alexgaynor.net/2011/apr/03/my-experience-computer-lan...

You're correct, I got my facts wrong. We were allowed to use ctypes for gmpy, but not for something else. We weren't allowed to submit a ctypes-using and array-using benchmark that didn't work on CPython due to a CPython bug.

the end result of us trying to be slightly better was that we ended up being kicked and a few other people ended up being kicked as well, though they weren't really guilty.

Overall the experience was that the results are not "fair" and are guarded by arbitrary rules that are not really written anywhere. One of those rules is that an arbitrary (typically most popular, but not necesarilly so) implementation is only allowed. This makes it much less trustworthy that it can actually be.

>> a full writeup <<

No.

The blog post tell's us that Alex Gaynor confirmed "with some CPython core developers" that his program didn't work because of a bug in CPython.

But the blog post doesn't tell us that Alex Gaynor never said there was any problem with a CPython bug.

Alex Gaynor's blog post tell's us that "It's also not possible to send any messages once your ticket has been marked as closed, meaning to dispute a decision you basically need to pray the maintainer reopens it for some reason."

But that's completely untrue! You can send messages when the ticket is marked closed! And you can open topics in the public forum! And you can click on a username and send email in 2 clicks.

There just wouldn't be any story to blog about, if Alex Gaynor admitted that he could easily have told me -- the bug is in CPython not in my program, so show my program -- but chose to say nothing.

>> arbitrary rules that are not really written anywhere ... an arbitrary ... implementation is only allowed <<

Have you even read the home page?

"There exist multiple implementations for some programming languages - different C++ compilers, different Java VMs - but those other language implementations are not shown here."

>>a rough idea of a specific language's properties<<

No, but maybe a rough idea of a specific language implementation's properties.

>> if there are valid criticisms <<

There are plenty ;-)

http://shootout.alioth.debian.org/dont-jump-to-conclusions.p...

Readers obviously jump to conclusions based on box plots and summary statistics -- when they really need to look at the individual tasks and look at the program source code, because there's so much variation.

I think there would need to be incredibly tight constraints on minimum program run-time, source code size, and memory use; before summary statistics could be taken at face value.

I was confining myself to CPython on the grounds that it is still what people complain about in terms of adding multicore to it. People don't complain about PyPy not having it since it's still developing, and it is not yet known whether PyPy will be fast enough to be worth worrying about multicore. To be honest I'm skeptical, but open to the possibility. But I rather suspect in the end that Python will forever be a slower language than the competition; all the dynamicness doesn't come for free.

Also the nicer primitives that IMHO the really-useful multicore languages are building on are either impossible (Haskell) or too late (Go/Erlang) for Python to deeply embrace due to massive legacy libraries and code. If you want to make threading "work" in Python, go nuts, it will bring much benefit, but Python will simply never be the first choice for tasks in which multicore performance and safety is the first or second priority, rather than the fifth or sixth. And that's fine. I'd really rather see Python become a better Python than see it become a crappy Go or something.

PyPy (on it's good day) is within 2x mark from the equivalent C. We're planning on closing this gap, but also we're planning on making every day the good day (right now you kind of have to know what you're doing in case you want to hit the sweet spot). This might be an interesting read for you: https://www2.cisl.ucar.edu/sites/default/files/2cameron_spar...

>> partly because we complained about unfair rules <<

That's not true.

I take your word for it - I genuinely don't know, the correlation is however hard to miss.

I also stopped making measurements for Java -Xint, LuaJIT, Tracemonkey, CPython, Iron Python, and Ruby 1.8.7

Post hoc ergo propter hoc is a well known fallacy.

>> I genuinely don't know <<

When you don't know, don't make accusations.

Haskell 1.0 was released in 1990.

The "History of Haskell" paper, a fascinating read in and of itself, is replete with admissions of lack of foresight. The reason Haskell has fared better than Python is probably only that it has a much bigger emphasis on trying new ideas in the reference implementation. The biggest problems with "Python" (GIL, FFI, standard library) are mostly issues with the reference implementation.

http://research.microsoft.com/en-us/um/people/simonpj/papers...

And where Python entered production usage in the late 90s and was building up a huge community and a base of code and continued building that base all the way to the present time, Haskell didn't really begin that until around the mid-200xs. It may be an older language in calendar terms, but in terms of libraries and tooling support it's basically 5 years old or so.

Reply on news.ycombinator.com