Iterators in Go: A proposed extension
bitfieldconsulting.com
bitfieldconsulting.com
I love Go because of its simplicity, and because it strictly adheres to the "there's only one way to do it" mantra. Adding syntax sugar muddles this, and introducing another keyword and construct makes the code less readable.
I hope that this proposal is rejected, and that it's not a sign of Go becoming yet another kitchen sink language.
The readability this provides is considerable.
Slower. More complex. Why?!?
Just call your function in a loop.
It's awful, but they do it.
Why would that be slower?
... that's the theory behind it. For more grounded explanations, see [1] & [2].
Thanks for the links!
Lets say the logic is:
for item := range list {
if condition(item) {
break
}
}
items, cancelFunc := apiCall()
for item := range items {
if condition(items) {
cancelFunc()
for _ := range items { }
break
}
}
close(items)
topLoop:
for _, batch := range list {
for _, item := range batch {
if condition(item) {
break topLoop
}
}
}
type Iterator[T any] interface {
HasNext() bool
GetNext() T
}
for i.HasNext() {
current := i.GetNext()
// do things here
}
Iterator design is another area where sum types would help simplify things a lot. Just one method returning Some/None would be so much better.
It would be better, but since go has MRV and zero values
func Next() (hasNext bool, nextValue T)
If you're looking for those particular features, I'd recommend adopting go's `, ok` idiom that appears in several places in the language. A type like:
type Iterator[T any] interface {
Next() (T, bool)
}
for current, ok := i.Next(); ok; current, ok = i.Next() {
// do things here
}
programming is a lazy hobby. it's why as soon as generics were released, I immediately implemented 'Keys', 'Values', 'ToSet', etc. Writing multiple lines to do the equivalent of these functions might be 'pure' but it's not easier to read. Worse: instead of standardizing these common operations, you just have everyone rolling their own implementations of the exact same functions with slightly different naming/parameters/etc.
Am I confused in that there is some merit to this approach beyond dogma? Why do we, as a community, need to accept so few batteries? If you want to only use assembly in your code (the purest!), that's always an option.
Being productive in a language is not just about using every convenience at your disposal, and typing less. It's about producing code that is easy to read and approachable for others or yourself years down the line. The more language features one needs to be aware of when writing or reading code, the less approachable it is.
Having less features also forces the programmer to rethink their solution in a less clever way. In dynamic languages like Python and Ruby it's very convenient to use some fancy trick to write less code. In languages like Rust or Raku, you can (ab)use some obscure feature in a non-intuitive way, and feel good about the "elegance" of the solution, but a fresh pair of eyes might struggle to understand what the code actually does. Considering reading code is much more important than writing it, as we literally write code for humans, not machines, optimizing for simplicity and having less features in a language is a more productive long-term approach.
That's the real value of "simple" languages like go.
You literally don’t have to use iterators if you don’t need them. You can just range over a collection as you do already. But when you want something a bit more performant like looping individually over batches or generating an enormous sequence of records on the fly, they make what would either be tedious or difficult into something trivial.
Go being “simple” just makes application code unnecessarily verbose and makes dealing with even moderately complicated problem domains feel like solving novel computer science problems when it’s be four lines of boring code in better languages.
Complexity is not something you can remove.
Complexity removed from language moves to the application.
The best thing you can do in the language is to make expressing complex concepts in easier, more clear way. Going too simple just produces more boilerplate and makes it harder to read.
I don't see why the author's implimentation is so complicated. I really just want an interface that works with the range operator, and to call it a day.
type Iterable[T any] interface {
Next() (T, bool)
}
for _, el := range someIterable {
}
for el, ok := someIterable.Next(); ok; el, ok := someIterable.Next() {
}
Which is, looking at absolute hulking abomination C++ or Java has become, is the right approach in theory
They just made a massive miss with lack of generics and sum types on the start. They could've avoided nils entirely with that for example. Or have less annoying error handling.
TBF neither generics nor sum types were necessary for either, these could have been builtins or language-level features.
To start with it's not like Go has been shy about those for things they didn't want to formalise (yet), but even in languages where they're library-ish features it's common for them to have language-level support.
"go developers" is not a homogenous group. I write Go and am pretty happy with the language design as it is. I'm also fearful of it becoming a kitchen sink language because of a vocal minority who want Go to be more like language X, instead of just going and writing in language X.
There's plenty of things could be added to make code more concise, but they come with a hidden cost.
It's rare that can is relevant when it comes to language design. You typically can do anything in any language.
Adding dedicated support for iterators would promote it as the way to iterate. That would actually simplify the language - one way to express a common need as opposed to the current mess of ~5 different ways [1]:
"In the standard library alone, we have archive/tar.Reader.Next, bufio.Reader.ReadByte, bufio.Scanner.Scan, container/ring.Ring.Do, database/sql.Rows, expvar.Do, flag.Visit, go/token.FileSet.Iterate, path/filepath.Walk, go/token.FileSet.Iterate, runtime.Frames.Next, and sync.Map.Range, hardly any of which agree on the exact details of iteration. Even the functions that agree on the signature don’t always agree about the semantics."
Go would have been a far simpler language had it originally shipped with generics, iterators, and sum types.
For example, generics were a highly controversial topic, and many people still believe that Go didn't need them. I'm partially in that camp. I've encountered maybe one or two situations where generics would've been convenient since Go 1.18, and in both cases not jumping straight at the opportunity to use them, but being forced to refactor and approach the solution in a different way, has produced simpler and more readable code. To this day, I struggle with Go's generics syntax, and it just looks alien to me.
Iterators are a useful feature, but are they really generally useful to deserve a change in the language spec? Like generics, I've yet to encounter a situation where they're truly required, and when I do, implementing them myself is trivial.
So, sure, language designers can cram every programming construct we've invented in the last 50 years, as Rust and Raku have done, and many programmers will find this very convenient and flexible, but a more focused and conservative approach produces a simpler language that is easy to pickup and read. This is what drew me to Go in the first place, and lately it seems that this is changing.
You have language features which are not needed for the ones using directly (for the lack of better term) the language and but really nice for the people providing building blocks for them.
I am using generics in other languages than Go but not in Go as I am normally not writing libraries in Go or they are already so specific that generics are not needed.
There is one. Return a channel. Just a bit iffy when you want to stop iteration in the middle and throw away the rest.
And in this case there is really no need to spawn goroutines to support iterators
You forgot tuples.
func F() (A, B) { ... }
func G(a A, b B) { ... }
G(F())
func H(ab (A, B)) { ... }
(Before you suggest a Python-like star operator, personally I consider using it for non-homogenous arguments about as hygienic as keeping your toothbrush and dirty socks in the same drawer.)
Which assumes this code matters much, outside of errors for which intermediate processors can just take... a tuple of a value and an error.
Also assumes the magical splatting of MRVs is a good thing in the first place.
> (Before you suggest a Python-like star operator, personally I consider using it for non-homogenous arguments about as hygienic as keeping your toothbrush and dirty socks in the same drawer.)
That is literally what Go currently does for MRVs, except opt-in instead of magical.
There is nothing about such splatting which wouldn't work with proper tuples, if anything it's the opposite. It is, essentially, the value side of `uncurry`.
> Also assumes the magical splatting of MRVs is a good thing in the first place.
IMHO, G(F()) reads nicer than some variant of G(*F()) or G(F()...). Lua does this too, and is (not unlike Go) considered a simple and elegant language.
> Which assumes this code matters much, outside of errors for which intermediate processors can just take... a tuple of a value and an error.
Then you'll sooner or later find yourself writing a G((a, b)), which again, I think is a net loss to readability.
I think my argument would've been much stronger if Go had had generics in 1.0, and the stdlib/ecosystem wasn't already full of MustParse's and MustCompile's. But in that case we could've had a Result type as well.
What's the one way to do iterators now? I thought this proposal came about because there wasn't a default way and everybody had to roll their own.
There is no new syntactic sugar nor there is a new keyword here though. It just changes how range works.
That being said, I'm not a fan of range having two modes of functionality. Perhaps it could just use an interface, now that we have generics it can be implemented on the existing supported types. And compiler can do whatever with it if it would be too slow.
A standard interface feels a cleaner than these nested callback functions.
C# defines an interface IEnumerable<T> with a “next” method, and originally you’d need to implement this interface which was a bit of boiler plate.
They added “yield return” which would create compiler generated syntax sugar to return a generator function like this go proposal, but your code looked identical to a non iterative version.
The go proposal feels like the worst of both worlds and like the team misunderstood the purpose.
I welcome people embracing function closures more, as I feel they're a drastically under-used language tool (particularly in Go where everyone crams everything into single-func interfaces which are almost exclusively far worse). More exposure to it helps people become comfortable with it, and now that we have generics it's finally not a pile of pain and agony at all times. And just using functions makes it very flexible / doesn't repeat the "X for me but not for thee" issues of past Go versions.
I'm not entirely sure how I feel about this proposal though. It kinda feels like it's trying to put a round peg into a square hole. If you try hard enough, everything can go in the square hole! But that doesn't mean it's a good fit.
As is the proposal says "there should be some coro.Pull type thing" which is a bit terse and says nothing about whether the implementation of coro.Pull will be fast or whether it will be built on range + channels
func Pull[V any](push func(yield func(V) bool)) (pull func() (V, bool), stop func()) {
... // snipped because it's big
}Possibly you're interpreting yield as a new keyword? It's just an argument to Pull.
The only not-(currently-)available-in-Go stuff that I can recall from the post is the performance optimization at the very end.
Yes, I know you can build a 100x-too-slow version of this on top of channels, but that's not the proposed extension to the language.
func next[T](i iterfunc[T]) T {
var item T
i(func(it T) bool){
item = it
return true // likely
})
return item
}
Quite a lot more, as it essentially means you can only consume from iterators you create... which sorta defeats the purpose of iterators as a value, and degrades them almost completely to just a `range` helper.
--- edit
from https://github.com/golang/go/issues/61405
>To convert one of these functions to a "next-based" or "pull-based" iterator, we plan to add a function to the standard library that runs the yield-based ("push-based") iterator in a coroutine, along the lines of the coro.Pull function in my recent blog post.
that's... an option I guess. Unless the runtime special-cases this coroutine though it seems like it'll probably have non-trivial performance cost, roughly equivalent to the channel + goroutine equivalents people sometimes build now. Which seems... not great. Bad enough that anyone even slightly performance-sensitive will probably avoid it.
(his coro post also mentions a proposed runtime change to improve the performance, but afaik previous attempts to get this kind of change into the language haven't worked out reliably, and you can't trust callers to be single-threaded)
Depending on scope, I've grown to prefer the single func interface.
For a smaller scope use, say filepath.Walk, absolutely, pass in a closure, good stuff.
For larger stuff, like say a chain of http middleware and handlers, I think it can be very limiting.
you get a big benefit from using the interface over a function: the underlying type _can_ have more methods and implement.
this gives room to implement a parallel interface, for example, to collect all possible routes, or to trace what handlers a request would pass through.
I don't think there's a one-size-fits all approach, use whatever gives the effort:utility tradeoff you're looking for.
But sure, if you want to build a multi-method thing (or upcast to detect optional methods), an interface is the natural choice - it's no longer a single func.
Likewise, (ThingDoerImplemetation{}).DoThing is a valid func reference
Hence I say there's not really a one size fits all, there are a options for different circumstances.
Go's proposed syntax is god awful. Why they are doing that to devs and themselves? Should we expect it to improve in like 15-20 versions, like it happened with any and generic mix/max functions?
https://go.dev/play/p/mrQFLAc11QU
It should be doable to express this somewhat simpler using a (genericized) helper function to take out some of the work of writing the ListItems function.
Since there is already a way to do this I don't think the language should be cluttered with more ways to essentially do the same thing. It is better to push people to develop a vocabulary of idioms that uses fewer features rather than introduce "more stuff".
type Iterator interface{} {
// Returns the next object and true, or nil and false if iteration has ended.
func NextIteration() (interface{}, bool)
}
UPDATE: I misread the article. This isn’t what happens. My bad.
Honestly, I feel like Go focuses so much on superficial simplicity that you wind up with bad/leaky abstractions all over the place as well as bad primitive obsession.
/rant
No, you've misunderstood it.
The iterator function gets a yield function which can be called with a value (the value to be yielded). The yield function returns a boolean indicating whether the process should continue or terminate.
func SomeIterator(yield func(bool) bool) bool {
for whatever {
if !yield(a_calculated_boolean_value) {
return false
}
}
return true
}
I feel this might be the case because language developers are themselves the worst reference for a typical user. They’re experts and no matter how big a language gets, they easily wield it. Even if they recognize that complexity is going up, they can’t truly grok the cost of that.
I think Bob Ross talked about this often. That it can be tricky to know when you’re done. And it can be easy to clutter the canvas.
This iterator function here is really just a `forEach`/`Each` method where the callback can return a boolean. You can't easily use that to split iteration, say iterate the first five elements, then go do something else, and come back to iterate the rest, doing something completely different.
But that also means it doesn't have to worry, as much, about the iterator going stale because the underlying structure changes (`ConcurrentModificationError`), or not knowing when to clean up, because somebody might call `hasNext`at any later time.
So this is basically just a nicer `forEach` method.
```
type Iter[T] interface { More() bool It() T }
type mapOver[T] struct { Iter[T]; func (T) T f }
func (m mapOver[T]) It() T { return m.f(m.Iter.It()) }
func Map[T] (it Iter[T], f func(T) T) Iter[T] { return mapOver{it, f} }
type Range { to, at int }
func (r Range) More() bool { return r.at < r.to }
func (r Range) It() (is int) { is = r.at; r.at += 1 }
func DoEach[T] (it Iter[T], f func(T) T) { for it.More() { f(it.It()) } }
// etc
iter.DoEach(iter.Range(10), fx.Stdout(fns.MultBy(2)))
// or whatever
```
Translating to English:
If you can do it just fine with the syntax you have, why do you need to add it to the core language?
You can even use with without GC if your code has to be fast. I don't see what the fuss is about with other new statically typed languages.
So I oppose almost anything that will make Go less simple. Probably the only exception I would make is enums.
I don't want my brain to have to work any harder reading code.
Edit adding:
the whole point of Go is simplicity and speed. That's why it's called Go in the first place.
Let's keep it that way.
- how do you acknowledge the end of a stack?
- how do you cancel the operation mid iteration?
- how do you make it thread safe?
- how do you make fast?
The proposal here simplifies these concerns and thus makes it easy to write and use iterators. It makes it easier to read iterators too. And it does so without adding hacks to the standard library.
This is one of those instances where the proposal might seem to add complexity but actually it simplifies the code a lot (unlike generics, for example, which complicates the syntax of your code in its effort to dedup lines of code. To be clear, I’m not being critical against generics here. Just using that as an example to show contest).
I think Go lacks the features to make this ergonomic and flexible.
Before adding iterators the language should add tuples so that you won't need to special case 0,1 and 2 artity functions.
And shorter syntax for anonymous functions to make writing this less verbose, I believe this has been proposed before.
Also the special case for "0..N" ranges seems very unnecessary.
I also wonder how defer would be handled. After the last yield? After each yield? Should defer be executed when an iterator is not fully consumed?
This proposal does the opposite. It transforms the loop body into a closure passed into the iterator.
Uh, range over channel ? The author seems to be very bad at go
func Items() (items chan Item)
for a := range Items() {}That is why stdlib map and vector have their own range magic, but as library author this isn't available to you. That is the point of the proposal.
[1] https://syslog.ravelin.com/so-just-how-fast-are-channels-any...
The overhead also compounds when you compose / chain iterators.
Plus the "lag" they induce due to context switching, although I think I saw an optimisation (proposal) where the runtime would directly switch from one routine to the other in some cases of channel interactions? Or am I thinking of an other language entirely?
for k := i.getNext() {
yeet k
}
b) yo can always use for loop in a traditional for way with standard arguments(init, stop, next)
c) yield is poor way of doing channels, i do not see why we should be degrading our code
d) this entire premise is stupid :D