Comment by edanm | Hacker News Reader

Comment by edanm | Hacker News Reader

Parent Full thread

I once found myself writing a python program to calculate Poker hands. It was pretty fun to write - it generated, for any given hand you hold, every possible way the game can go forward, and generated statistics about how many times you win (against 1 opponent only, who could be holding anything.)

The program initially ran for 20 minutes on each hand. I worked a day on optimizing it, and got it down to around 1-2 minutes per hand, so I learned a few tricks to make things quicker. Lessons learned:

1. Local lookups are much faster.

2. Anything that you can somehow offload to a c-loop, makes things much faster.

3. Surprisingly, the fastest construct for building a new list was always list comprehension (e.g. [x for x in calc(l) or whatever.) This was far faster than a regular ol' for-loop, of course, but was also faster than using map's and reduce's.

Point 3 is weakened by the fact that map can indeed be faster. http://codepad.org/SzXcubXy -- in fact, point 2 and point 3 are in conflict, because map() does use a C-loop, whereas list comprehensions do not. As it turns out, point 2 was the correct one.

Also, and this is just on a more minor, pedantic note, list comprehensions basically are regular for loops, except with the use of the list-append opcode instead of a getattr-call pair. This produces a minor speedup, but it's nothing to write home about, honest. (using a different pastebin because it has a feature I needed): http://ideone.com/dT1LG . The story is a bit different on 3.x, where list comps are implemented very differently: http://ideone.com/58FGZ

First off, I'm really no expert in Python or optimizations - I'm only reporting on observations.

Having said that, I was pretty surprised that the list comprehensions ran faster than map, exactly because of point 2 - I had assumed that map used a c-loop, whereas list comprehensions did not. Unfortunately, I don't have that code sitting around anywhere so I can't redo the tests - I just remember being very surprised by those results.

In my experience edanm is correct. map/reduce/filter are faster with built-in functions, but list comprehensions are faster with user-defined ones

For eg. using python 2.5.2

In [9]: l = ['10'] * 5000000

In [10]: timeit -n5 map(int, l)

5 loops, best of 3: 1.29 s per loop

In [11]: timeit -n5 [int(i) for i in l]

5 loops, best of 3: 1.64 s per loop

You mean "incorrect". Also, I did post detailed timing results above.

Reply on news.ycombinator.com