Hacker News Reader

Comment by ravenstine | Hacker News Reader

I initially liked the idea of generators, but after years of trying to find ways to apply them, I just haven't found a use case where they were more sensible than using existing logic and looping constructs. They could be useful where they would provide less overhead than constructing arrays for the same purpose, but that doesn't mean the same thing can't be achieved without either of those things. It's good in theory, but hasn't been useful to me in practice.

View on HN

Imagine you have function that returns elements. In order to return each element, you need to do some time-consuming calculation. You don't know how many elements the users of your function will need. Some may need 13 items. Some the first 500. Others might be interested in knowing only the first item.

Let's say that your sequence has a maximum size of 1000. If you were to return an array, you'd need to construct the full array each time, even if the code that calls your function only needs 1 item.

Using a generator, you can write the code once, and it is performant across different use cases.

I could write this against a few other replies but I will write it here. Moreover, I don't think I'm going to say anything that hasn't already been covered by other people, but I am going to attempt to distil down the arguments.

- The benefits of generators are, in a large part, the benefits of using iterators

- What are the benefits of using iterators? As you say, one benefit is that calling `next` on an iterator performs a single unit of work getting the next value. This let's you avoid e.g. allocating intermediate arrays, let's you do infinite streams, etc. Compare that to calling `map` on a list...you have to consume the entire list.

- A second benefit of iterators is that of scope. When I call `next` on an iterator I get the next value in the scope of the caller. This is particularly useful in a language with function colouring, because use of the `next` value can match what is required of the caller. E.g. the caller may want to await a function using some field of the `next` value and this is totally fine. Compare that to calling `map` with a lambda and wanting to `await` inside the lambda...the problem is you are now in the wrong scope and can't `await`.

- So where do generators come in? Well they are just syntactic sugar that will generate the state machine that you would otherwise have to implement by hand in your iterator. In that sense you don't need generators at all...

- BUT, with generators you can do things that would technically be possible with iterators but would be so clumsy to implement (I'm thinking here of coroutine libraries) that having them as a distinct feature makes sense.

There's two places I've used them in the past year where it was a natural fit:

* Querying solr for a massive amount of data using cursors, where the api and cursor use is hidden so I only have to process the result as a simple loop.

* Pulling csv data from disk and grouping by one of the columns, so the main loop is simply looping over groups without having to do that logic itself.

One of they key points in both versions is that the data can be too big to pre-process, and the system would run out of memory if I tried to load it all in at once.

Generators are great for building and consuming lazy iterators, and for building pipelines of lazy iterators. Lazy evaluation in pipelines can interweave work and can be more efficient than immediate evaluation.

Reply on news.ycombinator.com