Hacker News Reader

Comment by signed0 | Hacker News Reader

It's not brilliant.

This accomplishes the same thing without being hard to understand:

    from itertools import islice

    iterator = iter(array)
    try:
        while True:
           yield list(islice(iterator, 3))
    except StopIteration:
        pass

View on HN

This crashed my computer (it's an infinite loop that needs too much memory, apparently), and there's a mistake (the i is not defined).

Turns out that islice doesn't raise an IterationError, it just returns an empty list.

Fixing the problems, it runs in 237 μsec per loop, around 23 times more than the zip version.

Haha. I feel very silly now. I was just about to respond with the same thing.

    while True:
        result = list(islice(iterator, 3))
        if not result:
            break
        yield result

That's what I did, I edited my post above. It's around 23x slower, although I had to use a function invocation for %timeit as well, so it's probably a bit faster in practice.

One more for kicks:

    n = iter(array).next
    [(n(), n(), n()) for _ in xrange(len(array) / 3)]

Sure, there are many ways to do it, but I think the author was going for speed here.

I'd be surprised if my way is slower. Any time you unpack into a function such as zip() python has to create an intermediary list to store all the results before calling the function.

Exactly, I would be very surprised if the zip version was faster. One of the first steps to optimization in python is moving everything you can to generators and using of itertools.

The OP's question of is this genius or bad is clear in that regard: it is bad, due to not being the proper optimization direction, but it is interesting.

I would be very surprised if it weren't. The zip version has the hot path written in C, the other version has it written in Python with a bunch of exception handling/list assignment code, etc. The original method is just zip(iterator, iterator, iterator), which is probably as fast as anything can be.

That is a common misconception. Moving to iterators adds a function call while list creation in C is quite fast. Every case has to be tested for performance.

I always thought the reason iterators are used in preference to lists was due to the memory advantages, not the performance.

There's also flexibility improvements to do with lazy evaluation, and it makes the Twisted guys not hate you.

The memory advantages are key to performance when you have a non-trivially sized dataset, in my experience.

In my experience, building a non-trivially transformed sequence with the builtin C-backed list and list comprehensions vs iterators and generators, the iterators and generators win.

Maybe I'm doing crazy stuff, though!

You could always use izip:

http://docs.python.org/2/library/itertools.html#itertools.iz...

I was a bit wrong about that. For some reason I imagined that there would be quite a few args being passed to zip, when in fact there are just the three iterators. It does create a temporary list, but it's so small it's negligible. Using izip wouldn't really change anything.

Did you test and time the itertools version?

I did not, but I might now.

EDIT: See update above.

Reply on news.ycombinator.com