Go proposal: new package to provide generic slice functions | Hacker News Reader

Go proposal: new package to provide generic slice functions

github.com

208 pointstodsacerdoti178 comments

The generic slice functions would certainly reduce boiler plate, but what I miss in comparison to Rust is the concept of iterators. In Rust if you were to iterate over words in a string you'd get borrowed words, so in practice lots of things are non-allocating (unless you choose to collect / take ownership) - in Go etc. instead of windowing over a string you typically fully allocate the []string result (e.g. strings.Split / strings.Fields). Sub-slicing in Go can be used to make your own iterators, but the ergonomics take a hit.

Yes, iterators are typically a hard[1] problem in Go. I think this essay[2] describes perhaps the closest lazy eval iterator pattern in Go, which is much more complex to write than languages like Python.

[1] https://ewencp.org/blog/golang-iterators/index.html [2013]

[2] www.catb.org/~esr/reposurgeon/GoNotes.html [2020]

Clickable: http://www.catb.org/~esr/reposurgeon/GoNotes.html

I don’t understand what’s hard about iterators in Go after reading that article. I’ve written a lot of Python and Go (15 and 10 years respectively) as well as a bit of Rust and while iterators are certainly simpler in Python and Rust since Go lacks generics, most of the “challenges” presented in the article seem petty (complaints about for loops). The easiest way to write an iterator in Go is the closure/generator method and you can use it in a for loop like so:

    for x := next(); x != nil; x = next() { ... }

Not as nice as a for/range but not a real problem. With generics we can have libraries that operate on iterators like map, filter, reduce although I also think people make way too big a deal about a tiny bit of for loop boilerplate (there are legitimate reasons for generics but I don’t think small amounts of boilerplate are among them).

There's nothing hard about writing iterators in go.

Rust iterators are beautiful. If you've ever implemented C++ iterators, you'll be pleasantly surprised at how elegant it is in Rust by comparison (actually you could replace iterators with about any language feature and the above holds true.)

I really like Rust as a language, it's like a lightsaber, an elegant weapon from a more civilized era. Go feels a bit more like a blaster (sorry May 4th was yesterday.)

That said, when it comes to getting things done, the blaster seems more effective. I would chalk that up to garbage collection, goroutines vs async+await, and simplicity of the language in general. That said I use both languages regularly because they have different strengths and weaknesses and one is usually a clearly better option depending on the problem and requirements.

I really admire Rust iterators, especially compared to C++ iterators, but I lament that there is no easy way (that I know of, anyway) to build an iterator from a closure which yields `Option<T>`s. I don’t like that I have to define a type and implement a Trait (in general working with closures is cumbersome in Rust). That said, this isn’t a major problem either and on balance I’m very happy with Rust iterators.

It's probably possible to create a wrapper to do that. But if you need a closure, you can also just use a struct implementing the trait to manage the state. It's not perfect, but it's so much nicer than in C++.

Might you be looking for https://doc.rust-lang.org/std/iter/fn.from_fn.html ?

Oh wow, yeah, I think that’s it! Thanks very much!

Whoa I did not know about this. This would have saved me so much time during Advent of Code…

Thanks for sharing! I wish I'd known about this earlier.

> I lament that there is no easy way (that I know of, anyway) to build an iterator from a closure which yields `Option<T>`s.

Could you expand / explain what you actually mean here? Because I can think of multiple interpretations of this prompt which are straight up part of the stdlib, but that I can interpret this description in multiple ways shows the issue.

Let’s say I have a closure `fn() -> Option<T>`—how do I turn that into an Iterator? Maybe it’s possible with the stdlib and I’m just not familiar?

EDIT: I think this is it: https://news.ycombinator.com/item?id=27050861

Let's say std::iter::from_fn didn't exist. Then you'd need to make something that ends the iterator when the function returns none, you could do it like this:

    std::iter::repeat_with(f).take_while(|elt| elt.is_some()).filter_map(|elt| elt)

Where `.filter_map()` removes the Option layer and `repeat_with` creates an initially infinite iterator from your function `f`.

Long time ago, `repeat_with` might not have existed. Maybe you'd do something like this instead for that in Rust 1.0.0: `std::iter::repeat(()).map(|_| f())...`

You prefer to use std::iter::from_fn now that it exists, of course!

To be honest, making a simple iterator adaptor is quite easy too, so you could have a knack at making one you are missing - if you can't find it in std or in the crate itertools already.

> the blaster seems more effective

And you have to, er, attack the rebels with an army of highly variable talent and training, which contains maybe one or two actual Jedi Ninjas, so maybe it's better if everyone just uses blasters and nobody cuts off their own arm.

I'd love to work in some super elegant language for my own intellectual betterment, but I really hope my next real-world gig uses something like Golang if there are more than a half-dozen devs... ever.

Agreed. I love Rust for my personal projects, but I wouldn’t trust colleagues with it at any place I’ve worked. And it’s not that my colleagues aren’t intelligent or capable, but that Rust’s expressiveness invites bineshedding on the right amount of abstraction or which crate to use or the right way to do errors or how to configure rustfmt or when to just give up a fight with the borrow checker and use clone() or etc.

Rust is a really, really cool language—perhaps my favorite language, but I’ve worked in C++ shops and Python shops and I fully expect the same amount of bike shedding at a Rust shop.

Apparently I’m pretty much alone in thinking that Rust is a clumbsy, messy language. The syntax is all over the place. You get the feeling that there’s no design strategy and than anyone with half an idea is allowed to bolt on features.

Rust’s syntax is definitely complicated and that’s one of my least favorite things about it but it’s extremely consistent. What gives you the “all over the place” feeling?

I don’t really agree that it consistent, but maybe you’re right that it just complicated. There are so many different syntaxes for all sorts of features that you can’t really hold it all in your head.

It’s been a year since I last gave Rust a try, in the mean time I’ve been able to pick Go without much effort. Rust just seem like PHP, it’s fast, featurefull but not thought through.

You're assuming that it was difficult to learn because the features weren't thought through. Have you considered that maybe it was because there were genuinely new concepts not present in other languages?

Yeah, if anything, Rust feels to me like it's incredibly well thought through. All the pieces fit together well precisely as if they were all thought of as a whole. It's "hard" because it forces you to consider an entirely new constraint of your programs upfront. But (almost) every piece composes nicely with every other piece in a way that makes Rust development extremely pleasant.

Golang feels like the opposite, in that none of the parts are actually built to work with one another. Tuple returns exist but there are no language features that allow you to actually work with them. There's a strong type system, but you have to break out of it with interface{} and type punning to solve entire classes of common problems. Type punning exists, but go can't prove exhaustiveness of switch statements so doing this opens further gaping holes. Errors are handled explicitly everywhere, but again there are no tools to handle the repetitive lifting so every program becomes an endless series of error handling occasionally interrupted by bits of program logic. And let's not forget how badly interface pointers interact with nils: comparing a nil interface pointer against nil returns false, so a basic nil-check that works everywhere else fails to actually perform correctly when dealing with interfaces.

> anyone with half an idea is allowed to bolt on features.

I know you're talking about feel here, not about process, but to be clear, while we do have a pretty open and fairly democratic process, and while anyone can propose a feature, only the language team members, which is currently five folks, can decide to say "yes" to a proposal. The team may have been plus or minus a few over the years, but it's never been particularly large.

(We generally feel this process works well to balance all these factors; we want everyone's good ideas, but we also have to maintain coherence. An open suggestion process with a closed selection process has worked well in our eyes, but everyone can feel differently, of course.)

I’m guessing you’ve looked more at Rust code that uses incubating features from Rust nightly.

Most of those features don’t make it to stable—or if they do, their syntax has usually been refined by the time they do.

Inconveniently, a lot of the Rust code examples you’ll run into on HN uses incubating features, because a lot of the people posting about Rust on HN are language researchers talking about how they were finally able to implement new language feature/performance optimization/safety guarantee X using incubating language feature Y; or how they think nascent library Foo’s use of incubating language feature Z is/isn’t a good idea.

All that is inside baseball. It looks nothing like day-to-day Rust code, any more than e.g. .NET CLR static analysis code looks like day-to-day C#.

(Also, there’s the code examples of Rust stdlib code, which is like C++ STL code in that you have to use arcane language features “in anger” to bootstrap the language — e.g. safe concurrent data structures must necessarily at some point rely on unsafe code.)

(Or it’s code for a Rust unikernel / OS device driver / etc, because that’s a thing you can do. But it’s a still-nascent thing you can do, that’s not yet as ergonomic as regular code that can rely on virtual memory to exist. You have to build the safety yourself in such cases, or rely on a library that does; and there are no such libraries that have yet reached high code quality + wide cross-platform applicability.)

Rust stdlib code is a lot nicer to read than C++ stdlib code, which relies on _Mangled names to prevent conflicts, and (ab)uses template metaprogramming, SFINAE, overloading, & and && and forwarding, etc. heavily. Though Rust's trait system results in indirections comparable to (but somewhat more manageable than) C++ SFINAE/overloading/type traits. And some Rust stdlib types delegate to compiler intrinsics rather than being implemented in unsafe Rust (forgot which).

Additionally, third-party Rust data structure/concurrency crates often contain unsafe code comparable to Rust's stdlib in complexity. (And device drivers, as you mentioned, are probably highly unsafe too.)

Is that because Rust is an expression language? As much as I like Rust, I do find that returning variables from conditionals or match expressions can feel a little jarring (especially when you combine the two). In other places though, it really shines and just requires a little discipline.

> If you've ever implemented C++ iterators, you'll be pleasantly surprised at how elegant it is in Rust by comparison

You probably already know this, but if you're willing to take a dependency on Boost.Iterator, then boost::iterator_facade[1] makes it much easier and less error-prone to implement a standards-conforming iterator, and the pre-built iterator adapters[2] cover lots of common cases, so you don't need to directly implement an iterator at all.

Not arguing the relative merits of C++ vs. Rust, just leaving a pointer here for anyone who didn't know about it.

[1] https://www.boost.org/doc/libs/1_66_0/libs/iterator/doc/iter...

[2] https://www.boost.org/doc/libs/1_66_0/libs/iterator/doc/inde...

Rust is a beautiful language. Its only issue is that programming in it is painful for anyone who hasn't followed Modern C++ for a decade and then read everything on Rust, which is just Modern C++ with some niceness from ML thrown on top.

I couldn't even begin to imagine onboarding a Jr Engineer to a complex Rust project. I don't even know if it is possible.

What constitutes "Modern C++"?

Rust iterators are hard to implement without a big compile time hit.

Compile time hit meaning it's difficult to get the types right or something else?

Presumably that it’s inherently slow to compile. “Getting the types right” happens before compile time.

That they rely on a lot of optimisations to be really efficient (= have any chance of approaching a regular for loop).

Without that, every adapter adds a function call to the yielding of an element.

What I miss in comparison to Rust is concise error handling. Rust has shown that it is entirely possible to have clear, type safe, concise error handling without try/catch rigoromol or if(err == nil) {...} pollution. My hope is that the introduction of generics in Go eventually leads to Result[] proliferating throughout golang.

I agree with the goal of the package, but I feel it doesn't do a good enough job of picking a lane between the imperative and functional approach.

If you're going functional, a list is both a functor and a monad, so you definitely need at least map and flatMap for the abstraction to make sense.

On the other hand there are a lot of "in place", "insert", etc, which encourage working with mutation.

You probably need a bit of both, but I'd like a more top-down design better.

I think it's to do with the community and what people who are really into Go are interested in.

The first reply to the thread deems "map" and "filter" borderline-useful, but "reduce" not worth it - that type of misunderstading just shows that the author (and the 60+ people who +1'd it) didn't spend enough time understading functional constructs yet.

Reduce is basically "loop over this list while maintaining state". If you have loops and state built in, it doesn't buy you a whole lot.

I don't think the aim of the package is to move Go towards a more functional style. Rather, it's to provide a set of utility functions that mesh with the style of existing Go code. For example, it would be useful to have a utility to 'pop' an element from a slice, even though that's not a particularly functional idiom.

I definitely wasn't arguing about what should or should not be in the package. I just replied to the parent comment with my idea as to why the discussion around this point is happening in the proposal.

Or maybe they understand them perfectly and have come across enough obtuse code using reduce to not want it as part of the language.

Everyone draws the "code that I like looking at"-line at different places.

Despite our wishes around that, functional constructs do have a hierarchy of abstraction vs power, and skipping a layer is the type of design that backfires in the end, like with Java doubling down on the collection APIs without introducing monads explicitly for users.

> Java doubling down on the collection APIs without introducing monads explicitly for users.

I'm not necessarily disagreeing with you, but how has that backfired for Java? It seems as popular as ever, and I don't hear many people lamenting the lack of monads in Java.

Ah thanks for pointing that out as I didn't make myself clear. I definitely didn't mean Java as a whole, as it is indeed a tremedously successful platform and language.

I meant specifically the lack of flexibility in some regards because of the missing of monads and friends.

For example, we have a few newer languages, TypeScript and Kotlin come to mind, implementing the "null operator":

   thing?.fieldA?.fieldB

While this is quite handy, and people unfamiliar with the Option monad will think that this new syntax is really cool, it poses two limitations: 1. It doesn't compose with other abstractions 2. I can't use this for my own abstractions

If the language designers had chosen to take one step back and implemented it in a more generic way (in some sense, in the same as my other comment in this thread about implementing "map" without "reduce"), the feature could've still been implemented in the language, but without the limitations.

I'm not sure what you meant with this point:

> people lamenting the lack of monads in Java.

Do you mean that, because you don't see people lamenting, it's not a useful feature?

> It doesn't compose with other abstractions

But monads don't compose with each other, do they? I thought you needed something more (monad transformers, algebraic effects, etc.) to get things to compose. Or maybe you meant something else by composition.

> I can't use this for my own abstractions

Yep. But Java doesn't even let you overload addition afaik, so being able to overload ? is pretty much a pipe dream.

> Do you mean that, because you don't see people lamenting, it's not a useful feature?

No, it's just that if something "backfired" for Java, I'd expect to hear people complaining about it.

> But monads don't compose with each other, do they?

They do if they're[0] traversable (cf literally `Data.Traversable` in Haskell), which almost all specific monads are. You'd have something like:

  newtype Comp f1 f2 a = Comp { unComp :: (f1 (f2 a)) }
  instance (Monad f1,Monad f2,Traversable f2) => Monad (Comp f1 f2) where
    (Comp a) >>= k = Comp $ a >>= (map join . traverse (unComp . k))

Edit: 0: actually, only the inner monad needs to be traversable.

Looks like that that's not a valid monad in general.[0] And also, I feel like it's a stretch to say that "almost all specific monads" are traversable.

[0]: https://stackoverflow.com/questions/42284879/is-the-composit...

I'm not convinced (Bool->) is a legitimate Traversable[0], but that's at least highly concerning and makes me wish the typechecker could properly handle "Instances should satisfy the following laws" comments as actual code.

0: in vaguely the same way that Int isn't a legitimate Monoid because you get nonsense by switching back and forth between sum and product (and min, max, xor, and, or, etc)

This is not about misunderstanding. The author argues that heavy map/filter/reduce does not fit with the general style of Go programs. As a heavy Go user, I absolutely agree.

There is nothing preventing people from writing their own reduce() in their own utilities package, and I expect a thriving ecosystem of "even-more-functional-constructs" packages to spring up as soon as generics hit stable.

The discussion in this proposal is not about this. It's only and entirely about what the stdlib should provide, and that is much more a question of practical and idiomatic usage than it is about one's understanding of a theoretical CS syllabus.

If you re-read the comment I was replying to I don't think your comment as a reply to mine makes a lot of sense.

I'm not arguing what's good and bad, nor what should or should not be in the Go language.

I need to update this for the latest generics syntax, but I wrote this a while back: http://www.jerf.org/iri/post/2955 "Why Go Getting Generics Will Not Change Idiomatic Go"

I do not understand the people who think that functional-esque programming is the only and best way to do things who are apparently just sitting and waiting for Go to be able to do it. Are you forced to use Go or something? This is an honest question, btw, and I am legitimately interested in answers. If the only reason or the biggest reason you're not using Go is that you can't import map-centric programming into it, why not use any of the many languages that support that rather than griping about a language where it's still ultimately going to be bodged on? Again, honest question, because there's no shortage of such languages.

My perception is that Go isn't that popular that there's a lot of people forced to use it against their will because it's just The Language.

Anyhow, even after generics drop, it still isn't going to be any fun. I expect to use a map here or there often enough that it'll be nice to have it in the standard library but there's no realistic way that it's going to become "the best way to program in Go". If you're waiting to try it out after the generics drop, I would suggest trying something else instead; Elixir, Nim, Crystal, Pony.

Many problems for which you might end up using something like Kafka streams just lend themselves nicely into the streams/collections -like style

I am happy using Go right now when they don't have map.. but including map and not including reduce/flatten is just odd

"Many problems for which you might end up using something like Kafka streams just lend themselves nicely into the streams/collections -like style"

Have you actually taken the time to write out what it looks like, as I have in that blog post above, though? It's not pretty. Any sane Go programmer is not going to want to use this for very much. (I write that carefully. I anticipate using it a non-zero amount myself. But it's not going to be a lot.)

I understand liking map/filter/reduce, even if it is the beginning of functional programming rather than the end. I don't understand absolutely demanding it in languages where it doesn't fit well and insisting on using it no matter the cost.

it's pretty clear to me. people coming from other languages are used to typing less characters and expressing their ideas like that, and they want to type less characters in go.

for go, a language that idiomizes variable names like `f`, you think that people would be more on board with fewer characters. There are many other areas where go has chosen to avoid tidying up the syntax/making lives easier in the name of simplicity, though I digress.

As for your earlier question of "why don't you pick another language," I think this is a silly question to ask. The vast majority of people working for hire are not picking their language. If you have a few hundred kloc written in go, I doubt very much people are in a position to casually introduce new languages, especially if they're trying to improve the current system.

"it's pretty clear to me. people coming from other languages are used to typing less characters and expressing their ideas like that, and they want to type less characters in go."

It's not. It's more. A lot more sometimes, plus the indentation for trying to chain maps together is ugly.

I think more people who think this is going to be great need to clock some time on the Go Generics playground [1], implement some Map and Filter and MapError functions, and convert some real code to this style in real Go. It's awful. I'm probably spoiled by Haskell and how nice it makes all this, but it's awful even against normal Go code. Try it.

The problem is, the thing that makes "functional-style" programming in Go difficult wasn't "missing generics". It was, missing generics, somewhat heavy-weight anonymous function syntax, indentation rules not optimized for deeply-nested functions, a compiler that almost certainly won't optimize through very many layers of these constructs, the inability to put methods on core types and the fact the core types are missing the useful methods/interfaces like "Traversable", no higher kinded types so no monads or anything else like that, a variety of little syntax issues that make this all slick, and probably some things I'm missing. Just getting some generics (and not having generic methods that can add new generic types hurts this style too) just isn't enough to make it slick. Go would need a suite of changes to make this slick and efficient, not just these generics.

[1]: https://go2goplay.golang.org/

Your blog post was a great read. Completely agree. I’ve seen so much stuff turn to excrement the moment FP is applied.

I’m writing a complex aggregator in go and the abstraction is buffered channels. There is no map or filter in sight. It is not needed.

I dunno, for some domains it feels like 80% of the job openings are using Go.

> Are you forced to use Go or something?

I mean, yes, most companies have a relatively small list of supported languages, whether explicitly or practically. If someone at our company wants to write a program that does some binary data munging that's even more frustrating in Java, and needs to do it faster than Python allows, they have to use Go. We will not let them use C or C++, because most of the team will not understand it and it won't be able to integrate with any of our standardized libraries or CI infrastructure.

Your statements in the blog about performance of that pattern are not accurate to the best of my knowledge.

A program structured as a sequence of short, tight loops over vectors/slices as described is more or less hitting the performance sweet spot of modern microarchitectures.

The general form of a short loop over a vector plays to the strengths of branch prediction and the instruction cache while the temporal and spatial locality of the memory accesses are conducive to caching and pipelining.

Moreover, there's usually no need to allocate or copy the array in that sort of data flow. I mean, unless you're chasing worst case performance to make a point or something. Slice the buffers out of a pool and allow ownership of the data to follow the execution context, then you're free to modify it in place.

The above points cover 3-4 orders of magnitude real world performance.

It is true that there are some issues with the optimizer working in this pattern but I have only seen performance severely degraded in this pattern by compile time type ambiguity -- some but not all interface{} parameters, anonymous functions / closures in scopes with dynamic type. Interface types with a pointer receiver do not typically encounter these specific issues, and I believe the empty interface type has much better performance since ~1.12 or so as well.

"A program structured as a sequence of short, tight loops over vectors/slices as described is more or less hitting the performance sweet spot of modern microarchitectures."

Sure, but even faster is not to loop over intermediate arrays at all, by virtue of never constructing them in the first place when they aren't necessary.

"Moreover, there's usually no need to allocate or copy the array in that sort of data flow. I mean, unless you're chasing worst case performance to make a point or something. Slice the buffers out of a pool and allow ownership of the data to follow the execution context, then you're free to modify it in place."

That starts getting into "I'm sure someone can come up with some solution that meets some of these goals". Whatever map you're talking about here isn't one that is defined as creating a new slice based on mapping a function over the old slice. I mean, it kinda sounds like you're saying "well, if you just write a conventional for loop you can do all this in one pass" to me? Which is my point? I'm not the one pitching for lots of array creation, it's people who insist on using maps and filters in a language that, of all the major languages, just isn't going to put in the optimization time to convert them back into loops under the hood.

> That starts getting into "I'm sure someone can come up with some solution that meets some of these goals".

The sooner we, as engineers, can collectively acknowledge that we're doing things so wrong it's costing us approaching 4 orders of magnitude performance, and agree to stop dismissing knowledge of the hardware as forbidden knowledge, and abusing acceptance of reality as pointless micro-optimization... then I'm sure we'll make progress towards accepting one of the many patterns people have been advocating for upwards of a decade which do solve these problems.

One of the patterns which happens to be able to reclaim most of those missing 4 orders of magnitude performance is functional data flow, which is by unfortunate coincidence essentially the pattern you're denouncing here for its performance. It actually maps almost directly to the ideal implementation because it's ... literally expressing problems in terms of vector operations with no data dependence, which maps perfectly to a parallelized pipeline of the instruction types that achieve near optimal throughput and realized IPC.

I am not trying to be a dick but I am very passionate about this topic and I disagree very strongly with your assessments of the performance here.

I think the pragmatic current realization of a functional data flow pipeline as described in my earlier reply incidentally achieves exactly what you mention here:

> Sure, but even faster is not to loop over intermediate arrays at all, by virtue of never constructing them in the first place when they aren't necessary.

In those examples of this type of pipeline, the vectors you start with are slices of hardware device descriptor queues which reference a DMA region, and by progressive transformation of the receive buffers and composition with additional buffer regions through intermediate decoded states you produce reply buffers.

The hardware is programmed to sample only the packets of interest to this descriptor queue, and copies the packets matching this filter directly to the DMA region referenced in the descriptor it places in the queue.

With sufficiently sophisticated hardware it is possible to offload decode of increasingly large fragments of protocol logic or even entire applications.

Description of the operations in a language of common composable operations over the type of vectors of buffers allows the amount of any given application which is mapped to e.g. general purpose CPU, GPU or other FPGA/ASIC offload feature instructions to vary continuously over time or API surface at runtime pretty neatly, it creates breakpoints in the logic flow that more or less always map exactly to functional hardware boundaries, because everything's pretty much just vectors of buffers all the way down really when you think about it. Your process's entire runtime is just another vector of buffers to the kernel.

This is, in part, why I'm generally not in favour of generics in Go.

Go has a common style across most code bases I've seen. Partly that's because Go is opinionated about style, and doesn't really support doing things any other way. I don't see this as a bad thing, though that may well be because Go's style happens to be similar to my own. I have friends who learned to code on Ruby and JS and who don't like Go at all, because it doesn't really support the way they like to write code.

Adding the ability to write functional-ish code to Go is going to fragment the style. There will be code bases written in functional-ish Go that are difficult to understand for people used to a more imperative style. And vice versa (though I suspect imperative is easier to understand). I don't see this as a good thing. It's kinda "if you want to write functional-ish code, why not pick a language that you can do that in, instead of demanding that Go supports it?". But I realise this is probably a lost cause, and we're going to see that wave of "even-more-functional-constructs" become so common that we end up with JS-like library churn.

I'm not really a Go expert, but I've dabbled a little bit in it and I like it a lot. Why do you believe generics specifically would cause fragmentation? Isn't it more of a standard library issue?

Probably I'm looking at it the wrong way, but it looks to me like it's a relatively harmless language extension with the added bonus of correctly typing stuff normally cast as interface{}.

Probably because generics allow for many different programming paradigms, and different packages will use different paradigms, requiring developers to understand all of them and integrate them. In addition to varying paradigms, packages will also vary in their degrees of abstraction—generics permit very abstract code while idiomatic Go code is not particularly abstract. Note that I’m not arguing that these tradeoffs aren’t worth the while, just that they do in fact exist.

I've been writing Go for 8+ years now, I guess. In all that time I've maybe hit 3 or 4 problems that generics would have helped with. And there were always workarounds (tending to be heavy on the boilerplate). This is fairly common, from the other Gophers I've talked to.

The problem with generics is they're a hammer, and everything is going to look like a nail. Instead of being a rarity, and everyone basically writing idiomatic Go as normal and only pulling out the generics when really needed, this functional-ish style that relies on generics (as in TFA) will become common.

Devs who are used to writing array.map code in JS will start on Go and immediately find the slices package in the standard lib, and create a generic for every single slice they use, because that's what they're used to and how they think it's meant to be used. For...range loops will look as old-skool and primitive as for loops look in modern JS (despite all their advantages).

We already have the "learning path" of newbie Gophers, whose first question is always "which framework should I use?", and it's an uphill battle getting them to accept that they should just use the standard lib. Trying to get them to accept that they basically never need generics is going to be harder.

Old crusty Gophers like me, who are used to and like the simplicity of idiomatic Go, will be left shouting into the void. Again.

edit: typing stuff as interface{} is also one of those things that you learn to get past. I'm working on a ~80Kloc Go code base at the moment, and have not had to resort to interface{} once yet. Interfaces are way more powerful than they appear at first. I'm not saying interface{} doesn't have its place, and it's used heavily in the standard library, but it's not as common as you'd think at first glance.

The biggest problem is the opportunity cost, and you may not be aware of just what you're giving up if you've not done substantial work in other ecosystems over the years.

The two big places this shows up in golang, imo, is the lack of high quality data structure libraries, and the lack of widely used standard library for routine channel management tasks.

The former is a huge opportunity cost. If I'm coding on .net, jvm, c, or c++ I have a variety of state of the art data structures available. Some of these structures are extremely difficult to develop, so the "just write your own boilerplate" approach is completely insufficient. Point to code generation templates all you want, the reality is that this problem is preventing golang from improving much past using mutex protected hash maps and google's interface{} typed btree. Meanwhile, if we look at say .net, you have a very high quality lock free concurrent hash table available that scales to massive heap sizes. This is no niche thing: the bw-tree / cosmosdb storage engine is entirely designed around the capabilities of this lock free map. It's easily responsible for 100's of millions of dollars of value to software in that ecosystem now.

The lack of simple, clear, and useful channel lifetime management functions is another rough edge that bites many go programmers daily. The many abuses of Context make this problem quite clear.

I understand the sentiment of not wanting golang to turn into the Stephanov style C++ generics. But I think that's an imaginary fear. The golang generics design is quite different and far more "gopher" for lack of a better term. The problems golang is facing due to poor type flexibility in this area are not imaginary: they are endemic.

I used to code in C# before Go. This isn't my first rodeo. Or even my second.

And I get that Go lacks in some areas. That's fine. All languages lack in some areas. Go wasn't designed to be all things to all people. It's very good at writing servers in. And there's a few projects written in Go that have proven to be pretty popular and scaled well ;)

It's not really YAGNI, which Go has been accused of (a lot), with some justification. It's more that simplicity is valuable, but it has a cost. Keeping Go simple is good, but the price of that simplicity is sometimes high. As you say, there is an opportunity cost.

I still think that cost is worth paying, and I'm not convinced (yet - I may be in time) that the reduction in simplicity that comes with generics is a price worth paying for the flexibility it brings.

> There is nothing preventing people from writing their own reduce() in their own utilities package, and I expect a thriving ecosystem of "even-more-functional-constructs" packages to spring up as soon as generics hit stable.

I really hope this doesn't happen because then you have random utility libraries that "infect" projects for what is essentially basic flow control. I would say stuff like this is best supported in the standard lib and not home rolled.

> The first reply to the thread deems "map" and "filter" borderline-useful, but "reduce" not worth it - that type of misunderstading just shows that the author (and the 60+ people who +1'd it) didn't spend enough time understading functional constructs yet.

It’s not very charitable to assume that people who disagree with you must not understand your position. In particular, one person could like reduce() because it’s very abstract and can be used for lots of problems, but critics could dislike it because very high levels of abstraction impede readability. Both parties may understand each other, but they are optimizing for different things (abstraction for its own sake and readability, respectively).

I definitely didn't mean it as an insult, as my comment is a reflex of my own journey understanding these concepts. I'm also absolutely not an FP expert. Different people have different levels of understanding about different things, this has nothing to do with intelligence or a general programming skill level.

I do think this is about understanding, as it's not about syntax, or what feels nicer. It's about building blocks and what comes before what. Given the nature of the proposed library, to give the users an arbitrarily less abstract construct is a design mistake as I mentioned in another comment in this thread.

For example, I remember when I needed to flatten a bidimensional array in JavaScript, perhaps back in 2005, and the internet told me that I needed to flatten the array. So I learned then that array.flatMap() served that purpose: to flatten a bidimensional array. This is of course incorrect, but it took me many years until I found the concept again and understood what were the building blocks around flatMap, and what kinds of other abstractions it allowed.

> I do think this is about understanding, as it's not about syntax, or what feels nicer. It's about building blocks and what comes before what. Given the nature of the proposed library, to give the users an arbitrarily less abstract construct is a design mistake as I mentioned in another comment in this thread.

Per my earlier comment, it's only "a design mistake" if you assume that it's always better to trade readability for abstraction. This isn't always the case, and in professional software engineering it's often better to optimize for readability--rarely are high degrees of abstraction beneficial (of course, someone will respond to this with some anecdotes of the instances in which high degrees of abstraction are helpful, which is just an enumeration fallacy).

Sounds about right. Python 3 demoted reduce from builtin to stdlib because it saw very little use after the introduction of the more common use cases as builtins.

Bingo, Go is geared towards junior programmers. Somebody who exits the university has to be able to feel productive within two weeks of using it. That's literally the cornerstone of the entire design and all things follow from it (C-like syntax, attempt to minimize the number of language constructs, and the heavy focus on imperative programming which is what they teach in college).

Oh the unintended irony. Go's syntax may be simple enough to pick up in a few weeks but writing "good" Go code turns out to be much much harder. Idiomatic Go can take a year or two to internalize and introducing non idiomatic Go code into an established code base can really muck it up.

You can see this clearly in job listings looking for Go coders. Almost nobody is looking for junior level Go coders with less than a year of experience. Companies are looking for Go coders with at 2, 3, 5+ years of experience instead.

A simple syntax may provide benefits, but it doesn't seem to be the benefits most companies want in practice after all.

> Almost nobody is looking for junior level Go coders with less than a year of experience.

I have never seen a job posting asking for less than a year of experience with anything.

True, but what other languages or other anythings are doing is at best irrelevant and at worst the entire point.

> The higher-order function style also tends to be much slower, so when writing a loop there's a decision burden every time between saving a line or two and writing fast code. (TIMTOWTDI is not really the Go ethos.) And programmers usually prefer concision, so the net result is often slow, HOF-heavy code.

This is very real. We do it all the time in JavaScript:

Object.entries({...spreadMerge, ...secondSpread}).map().filter().find(x)

Not once do I think about how many things are initialized, iterated over, etc. Rarely would I do something like this in Go.

Whether it matters is a good question, but the point is true: language facilities make you care more or less about performance depending on what it does.

This is a problem of optimization, not language design. There’s no inherent reason a chain of functions like that has to be less performant than a for-loop. You need high-performance iterators in the language, and a good compiler.

One of Go's objectives is to have a simple and fast compiler; emphasizing simple and readable code will ensure that. It's one reason why type arguments were put off for as long as they have been.

`.map().filter().find(x)` is significantly more simple and readable than for-looping for every small thing in Go.

I disagree. Fewer characters != simpler or even more readable. Map/filter/etc are certainly more clever, and I enjoy writing clever code, but I reserve it for my personal projects bprecisely because it isn’t so clear. Note that Python has these first class functions as well as various kinds of comprehensions and still it’s frowned upon to use anything more complex than the simplest comprehensions.

Alright, if it's not too much trouble, can you show me what the equivalent of the following code would be in Go? I think this is extremely clear as-is, but I'd love to see how you can apparently make it even clearer by avoiding iterator combinators.

https://play.rust-lang.org/?version=nightly&mode=release&edi...

Personally I think this is clearer, but it's quite contrived either way: https://play.golang.org/p/FpUWwm5yZ8n

Nice! Out of curiosity, if the iterators weren't of the same type (e.g. one was an iterator over the values of a hashmap and the other was over the items in a list) what would you have used instead of

    for _, m := range maps { ... }?

If you could provide the hypothetical Rust code you're referring to, that would probably make your question more clear here. I know both Rust and Go rather well, and I'm confused about which direction you're wanting to take the code.

Just as with Rust, you'd have to convert both maps to the same type before you could iterate them over together (or chain them, in Rust).

Map and filter are extremely simple, far simpler than a for-loop. They can only do two things: transform items in a collection, and remove items from a collection. After mapping you know for sure that a collection will still have the same number of elements. After filtering you know that all the elements remaining still have the same type. For-loops can do anything. It’s a question of what you’re most familiar with and used to.

I’m not comparing for loops in general with map or filter, but rather the specific for loop boilerplate to implement map and filter. Moreover, the complexity of map/filter isn’t internal, but rather decomposing or breaking down one’s problem into map/filter/etc expressions. Often for loops are clearer, especially since you can frequently collapse a map and filter into a single for loop without thinking in map/filter/etc primitives. This is especially true when you need to abort the chain early (e.g., due to errors)—Rust’s FromIterator provides a clever solution to this problem, but it’s much harder to wrap one’s head around than a simple for loop with an early return branch.

I find that decomposing iteration into map and filter expressions typically clarifies my understanding of the problem and makes the resulting code more general too, and I just disagree that for-loops are clearer. Combining a map and filter into one loop body tends to create specific, imperative code that doesn’t need to be specific or imperative. For one, loops require mutation to have the same effect as filter and map, and as soon as you’re maintaining state, things get confusing. My usage of higher order functions instead of for-loops is an expression of my belief that programmers should always use the least powerful construct they can to solve a problem.

In general though I think the functions used in map and filter should be total. Most languages are not expressive enough to handle errors fluidly in long chains like that, you can do it with monads but apparently everyone hates those except me.

> I find that decomposing iteration into map and filter expressions typically clarifies my understanding of the problem and makes the resulting code more general too, and I just disagree that for-loops are clearer.

Agree to disagree I guess.

> Combining a map and filter into one loop body tends to create specific, imperative code that doesn’t need to be specific or imperative. For one, loops require mutation to have the same effect as filter and map, and as soon as you’re maintaining state, things get confusing.

I agree that immutable, declarative code is easier to reason about in general than mutable, imperative code, but it's not a binary proposition. I posit the cost of managing the tiny bit of very-local mutable state for a for loop is a lower cost than trying to model a problem in terms of map and filter. While I can appreciate the elegance in "immutable always", I think it's too puritanical for real world software development, especially in these cases which are simply expressed as for loops.

Note that languages like Python have strong support for immutable expressions and `for` loops, and it seems very common (even idiomatic) to use imperative `for` loops for anything more complicated than the simplest list comprehensions.

> In general though I think the functions used in map and filter should be total. Most languages are not expressive enough to handle errors fluidly in long chains like that, you can do it with monads but apparently everyone hates those except me.

I think it's just that monads tend to allow for very abstract code at the expense of understandability, and the abstraction that monads facilitate is often well in excess of what a given problem requires. If you're writing code to maximize abstraction (which is a lot of fun), monads (and map/filter/etc) are great, but if you're trying to work with a team to build software to solve real problems, you probably care a lot more about readability than gratuitous abstraction and the monad (/map/filter/reduce/etc) juice frequently isn't worth the squeeze.

Python’s FP capabilities are severely constrained by not supporting multi-line lambdas or composition operators (pipe, sequence, etc). It is not a good language for functional programming at all. The community norms explicitly push you away from using FP in it because of how limited it is. We use it for wrangling large data sets and tend to hate every minute of it.

My workplace uses Elixir exclusively in production for our backend services, so we’re all functional programmers and we do not find that map, filter, and reduce make things harder to read, quite the opposite. Breaking up loops does not introduce cognitive overhead, it makes every step in the chain more explicit. You can call this personal preference but there’s nothing concrete in your belief that for-loops are more readable or easier for teams of programmers to maintain, many people disagree.

I generally agree with you, but it's nice that in Rust you can use a for loop if it makes more sense. I mostly end up doing this when I want to reduce to multiple collections (put type a in here, type b in there) or call functions that return different types of errors.

The other time it's very useful is when translating algorithms specified in an imperative form. I translated a bunch of algorithms specified as for loops to elixir, and it was annoying to have an extra source of complexity when comparing the implementations for correctness.

It’s definitely nice to have the imperative constructs but spending a lot of time with pure languages has changed my perspective on mutability. I think Rust mostly fits it, data is immutable by default in Rust.

> Python’s FP capabilities are severely constrained by not supporting multi-line lambdas or composition operators (pipe, sequence, etc)

Hardly. You can absolutely have multi-line lambdas, just not multi-expression lambdas (the real limitation is that there are no try/except expressions so certain fallible manipulations aren't so easily expressed). Similarly, Python lets you overload operators so you could always override `|` or whatever. Of course, operators are just syntax sugar for function calls so this strikes me as a cop-out. In either case, neither of these excuses are pertinent to the community's rejection of complex comprehensions or higher order functions.

> My workplace uses Elixir exclusively in production for our backend services, so we’re all functional programmers and we do not find that map, filter, and reduce make things harder to read, quite the opposite. Breaking up loops does not introduce cognitive overhead, it makes every step in the chain more explicit. You can call this personal preference but there’s nothing concrete in your belief that for-loops are more readable or easier for teams of programmers to maintain, many people disagree.

You could argue that any two models which evaluate to the same thing are fundamentally equally easy to understand. You could argue that monads are just as easy to understand as a sequence of statements in an imperative language, but experience suggests that people have a harder time understanding the more abstract concept rather than the more concrete concept. This shows up all over--Newtonian physics are more easily understood than the more abstract quantum physics and in mathematics "more advanced" math tends to mean roughly "more abstract".

> In either case, neither of these excuses are pertinent to the community's rejection of complex comprehensions or higher order functions.

I can only assume you haven’t done a lot of functional programming because Python is just not expressive as a functional language. Lambdas are extremely cumbersome. JavaScript by contrast has a huge FP culture and does not even have comprehensions. But it does have syntactically lightweight, multi-expression lambdas.

> You could argue that monads are just as easy to understand as a sequence of statements in an imperative language, but experience suggests that people have a harder time understanding the more abstract concept rather than the more concrete concept.

Except I’m not arguing that, because I don’t believe that monads are easier to understand, I believe that functional programming with pure functions is. This is just another restatement of the same argument you’ve been making the whole time. We onboard people who have never used FP into being productive Elixir programmers in two weeks or less. There are no monads in Elixir, nothing more advanced than recursion and function composition. My experience differs from yours. Our new programmers have very little trouble grokking how FP works. They have much more trouble understanding the Erlang concurrency model and the various primitives available, but those are unique to Erlang and uniquely powerful.

> I can only assume you haven’t done a lot of functional programming because Python is just not expressive as a functional language. Lambdas are extremely cumbersome. JavaScript by contrast has a huge FP culture and does not even have comprehensions. But it does have syntactically lightweight, multi-expression lambdas.

You assume wrong, but more importantly you missed the point of the snippet of mine to which you are replying—namely the expressiveness isn’t the reason the Python community rejected those features, but rather the complexity and cognitive burden.

> Except I’m not arguing that,

I wasn’t arguing that you were arguing that, I was using it as an analogous argument. We’re not going to get anywhere if you aren’t going to respond to what I’m actually saying.

I really like the method style of LINQ in C# - which I would agree are much clearer than the equivalent explicit loops.

This argument doesn't really have any content, because people who advocate for programming with higher order functions would not agree that such code is more clever; in fact, they would probably argue the opposite. What's the point of debating if you assume the people on the other side already agree with you?

I really like chaining array methods but you should also never forget it's incredibly easy to introduce crazy computational complexity with them.

Rust has lazy iterators, so a chain of iterator steps like this has the same complexity as a single for loop over the iterator. There's no reason Go couldn't do that same thing.

In fact, Rust's version compiles down to pretty much the optimal looping form. It will even vectorize your iterations using SIMD.

Rust is seriously cool in this regard, but Go aspires to be different, namely more C-like and less C++-like. This means a dumber, faster compiler that is easier to intuit about, for better or worse. Rust is such an amazing piece of engineering (moreso an amazing process and culture for producing an amazing language), but I think Go’s approach is right for a whole lot of software—lots of things don’t need to be insanely performant (lots of things are chugging along in Python, which runs 2-3 orders of magnitude slower than Go) but we do need to be able to iterate on them quickly in a development context involving dozens or hundreds of other people.

Not if you have solid lazy iterator support in the language. If they’re strict and every link in the chain is adding O(n) to the complexity then yes, this is a problem.

In the case we’d need a little context of what the map, filter, and find operations are, but if map is multiple by 2 and filter is less than 200 you can just replace the filter and map with a single if in the for loop that performs the multiplications and check and return immediately when you find X. That’s pretty obvious, always takes only one run, doesn’t reallocate, and doesn’t add concepts, which is of course adding complexity.

Yes and we could replace all of that with an implementation in assembly which both compiles and runs faster.

But we don't do that because for the entire history of computing, we've come to realize that abstractions enable us to reason about programs at a higher level without having to be bogged down in irrelevant details and that getting those details right once lets us avoid the inevitable bugs that occur while reimplementing them thousands of times.

By absolutely every objective measure we know in software engineering

   let triples = ints.map(|x| x * 3)

is strictly superior to

    var triples = []int
    for value := range ints {
        triples = append(triples, value * 8)
    }

The fact that this even has to be argued is just bonkers to me.

The second one is trivial to introduce bugs into, reading it requires mentally filtering out 90% of the code as irrelevant minutiae, and it's even worse from a performance perspective unless you remember each and every time to allocate a result array of the appropriate length to avoid reallocation. None of the details of allocating a result array or iterating over indices/values are important from the purpose of expressing the problem to be solved, and yet every time someone has to read this solution those things are of greater visibility than the actual business logic.

Worse, repetitive and unnecessary boilerplate like this hides bugs. Did you catch the fact that I'm multiplying by the wrong value in the golang version? Did you catch that I'm accidentally tripling the indices and not the values themselves? If you did, would you at least acknowledge that the first bug is much easier to identify in the first example than the second, and that the second bug is strictly impossible?

As an added bonus, the first one (in Rust at least) is automatically vectorized using SIMD for optimal performance. This happens even as you chain additional operations to arbitrary complexity.

If you still somehow think the second version is "better", then whatever your argument is can trivially be repurposed to say this version is better still:

    int* triples = malloc(sizeof(int) * array_len);
    for (int i = 0; i < array_len; i++) {
        triples[i] = array[i] * 3;
    }

If those details of creating a result array and explicitly enumerating are important, why isn't maintaining your own counter and directly indexing? At least the C version somewhat encourages you to allocate a result array of the correct size.

This is the decade-long argument of "let's sacrifice all our programmers to the devil so that we don't have work as hard on the compiler". I don't buy it, and it has never been demonstrated that the cumbersome style of writing go programs is helping compilation times, yet people keep saying it as if it's a fact.

The cumbersome style seems to have become a strong part of the culture, almost a masochism about the limited features of the language. I find this ironic because I believe Go was intended to be highly pragmatic originally but it seems to have evolved an almost Scheme-like dogmatism about language complexity.

Thank you for naming this. I've always struggled with how to convey this to people.

Do you have specific examples you’re thinking of? Calling it masochism is weirdly insulting. Adding complexity is a one-way street. Languages live for decades. There’s no reason to rush adding features in a half-baked way, and the tradeoffs in terms of compiler complexity are very real.

Virtually every large go program I've read has asymptotically reached 50%+ of its code being

    if res, err := fn(...); err != nil {
        fmt.Errorf("Look ma, I'm a human exception handler: ", err)
    }

This is held up as a positive thing, because handling errors is explicit and good. Somehow, the Rust equivalent of `?` is implicit and bad, despite desugaring to almost literally the exact same thing and being actually impossible to forget or overlook due to the return type of the function enforcing a Result.

In the same breath, devs will assert that the language is "low boilerplate" (despite 50%+ LOC being just one type of boilerplate) because Error Handling is Important. Which seemingly ignores that the important part of error handling is actually ensuring that errors are handled and not the mechanistic act of self-flagellation by wrapping every line of code in nearly-identical error handling ceremony. Which is unfortunate because golang doesn't do so great a job with the first part. It's surprisingly easy to actually forget to do something with the error while carrying on with a return value whose semantics are undefined (unlike Rust's Result, where you can have an error or a value and not both).

I used to hardly think about exception handling on the jvm, mostly let them bubble up and be centrally handled except in special cases. Now I'm always worried that I'll forgot to manually handle the error in Go. Can't tell you how many times I've gotten panics as well, from nil objects.

I use Go a lot and I really like Rust, so maybe I can add some context. Error handling boilerplate isn’t a big deal. Lines of code aren’t a big deal. That’s not where your bugs are coming from and you aren’t bottlenecked on your typing speed. It’s not a real problem, but rather a dogmatic pursuit of terseness for its own sake. Rust is an expression-oriented language—it needs something like ? because without it we would have gratuitous nested braces marching ever-rightward and that actually is laborious to keep track of (deeply nested match expressions). Go could drop some error handling boilerplate with an ? operator, but it wouldn’t benefit as much from it as Rust did.

Lines of code are a very big deal in an organizational context that cares about test coverage. Since all the error handling ceremony is written by hand, so too must a corresponding test case for each instance of the error-handling ceremony be written by hand. Typing out test suites for functions that compose several other error-returning functions is bureaucratic drudgery.

I'll bite. Why not ditch the "% test coverage" metric then, since as you just elaborated, it's not a good metric.

I would prefer my language wasn't designed around corporate bureaucracy personally.

I don't think the coverage metric is necessarily wrong to highlight error return branches: they are program logic you typed, and might have mistyped. I certainly have mistyped them. Checked the wrong error value, or returned a different error from the one I checked. You get to skip tests for default exception handling behavior in e.g. Java because you know the language is going to do; there's no opportunity for human error.

It is possible to have company policies against abstraction and cleverness when working in more expressive languages, and this would achieve almost the same intent as Go, it's just that actually using Go is more convenient. The language is corporate bureaucracy. The whole point is what's not there. What can't go off in a weird directions on a per-project basis. What junior engineers in a high-turnover environment can't screw up. It's not by accident that the native toolchain will give you a test coverage report with untested error return branches in red. Google is just happy to pay people for that drudgery, I guess.

> You get to skip tests for default exception handling behavior in e.g. Java because you know the language is going to do; there's no opportunity for human error.

I fundamentally disagree with that. the only thing you know in java is that it throws an exception that's maybe caught somewhere. That's an untested branch aswell.

You don't have to write a test for "if this underlying call throws an exception, that same exception is propagated up to me," which is about 80% of the test cases I write for Go.

You do have to test cases where exceptions are supposed to be caught and handled/eaten, which I agree is meaningful and relevant in all languages.

You “don’t have to write the test case” in the case of exceptions, but then you’re not getting the same coverage as with go. Just because a branch isn’t represented syntactically doesn’t mean it doesn’t exist. Note also that Rust’s ? operator isn’t going to be saving you from anything here since it just uses fewer characters to express the same branching logic, while languages with exceptions use none.

In general your org’s policies for what to test seems like a recipe for a low-value, high-effort test suite. Reminds me of the unit tests for getters and setters in Java.

You get to take the language for granted; you don't get to take your own code for granted. "String reverse" is an easy function to write and the same instructions end up in my binary either way. But if you write your own, you'd better test it. If you import one from the standard library, no need, the stdlib has already done that. Functionality being in the language does not just save you typing; it also saves you testing. That's my point.

I agree, given that Go is what it is, it's a bad policy.

You aren’t testing whether exceptions propagate correctly, you test that you catch or don’t catch them appropriately just like you test whether or not you return your error appropriately. That said, doing this for every function you write is probably not going to lead to a high value test suite.

“Make sure it doesn’t catch the exception” is like “make sure it doesn’t do a tap dance” - there are quite a few things a function could do, we don’t generally ask unit testing to prove negatives.

If you’re going to check that a function propagates an error in Go it equally makes sense to check that you check to make sure the equivalent Python/etc function propagates the exception. Of course I don’t think testing either of these is a good investment.

Consider these three points:

* it has been actually demonstrated that shorter programs contain less bugs,

* code is read much more frequently than it's written, and to the point code is easier to read than code full of boilerplate and meaninglessness syntactic elements,

* menial tasks and distractions introduce mental fatigue that consumes your limited brain resources that you need to think about the structure of your code that is actually dealing with the problem at hand.

Thanks for this, it's exactly what I had intended to reply with. Every single one of these points is spot-on.

I’m skeptical about the first bullet—is it the fewer characters or the reduced complexity which results in fewer bugs. If the latter, then it undermines your point; if the former the I’m very skeptical as it would suggest we should all be writing minified code.

With respect to the second bullet, I agree that code is read more frequently than it is written and that readability is paramount. I strongly disagree that minimizing the character count is the same as optimizing for readability. In particular, I find very syntactically terse languages like Haskell or Ocaml far harder to read—indeed ReasonML exists precisely because so many people find OCaml so difficult to read.

With respect to the third bullet, you should think about your error control flow. The structure of your code doesn’t change whether you’re using Go’s “if err != nil” or Rust’s ? operator. The difference is a dozen characters. I can’t believe that the real cognitive load is in the actual pressing of keys, but if that is your experience then text editors often have a “snippets” feature or extension that will let you map that boilerplate to a hot key.

I don’t think masochism is inherently negative and I did not mean to insult. For that I apologize. I was trying to characterize in a humorous fashion a difference between what I perceive to be the stated aims of the language and the experience of actually writing it.

It was also an oblique reference to this talk which is one of my favorites: https://youtu.be/mZyvIHYn2zk

The Go compiler is much much faster than C++ and Rust. I don’t think that’s seriously disputed by anyone, is it?

There’s no universally correct answer to the right balance of “smart compiler” vs “simple language”. Having worked with Go on a daily basis for five years now, I really like the simplicity of the language. Compared to the cognitive load imposed by the language when using C++ and Rust, I much prefer the naive straightforwardness of Go. There’s no devilry about it.

Go vs C++ is not a fair comparison, because C++ is a badly designed language from the 1980s. It also serves a different domain. Certain applications of C++ are not serviceable by Go.

Go vs Rust is not a fair comparison, because Rust's type system is much more powerful that Go's and enables applications that Go's doesn't (like embedded systems programming). Also some of the things I said about C++ apply here.

There are also features of Go absent from C++ and Rust (reflection, garbage collection). So it's really puzzling that you'd pick those two.

To have a fair comparison, you'd need to look at something like Kotlin sans the class system. There Kotlin is a clear winner, and with the same engineering resources put behind it, a cut-down version of Kotlin would easily be able to achieve build times in the ballpark of generic-enabled Go.

Despite its claimed simplicity, Go has inconsistent design and bug-prone unwieldy features (try to write code that merges streams of data from two channels without blocking unnecessarily). It also has unnecessary constructs that contradict its claim to include only what's necessary (interfaces are just structs of function pointers, type methods are just functions with a glorified first argument).

It may be enough to write one-off command line applications, but it doesn't give developers the tools to create complex distributed systems in a systematic, clean way, which was supposed to be its main area of application.

> and with the same engineering resources put behind it, a cut-down version of Kotlin would easily be able to achieve build times in the ballpark of generic-enabled Go.

So basically no real example. It is only what would be possible if things were to happen certain way.

> It may be enough to write one-off command line applications, but it doesn't give developers the tools to create complex distributed systems in a systematic, clean way

This is just arrogant opinion. Distributed tools and applications like NATS, Kubernetes, Docker, CockroachDB and so many other exist and written in Go. Developers who can't write distributed systems themselves can ofcourse buy or license tools from vendors. And at that point, honestly, it could have been written in any language.

If your team has to write and review boilerplate by hand, that’s millions of times slower than a computer generating it. The compiler only looks faster when we exclude all that wasted effort.

This seems like the “sufficiently smart compiler” argument. A sufficiently complex compiler could certainly optimize this away, but building and maintaining such a compiler while also keeping compilation times low is an enormous challenge. Rust does an impressive job here, but Rust was built from the ground up to handle these problems and it also trades compile times to achieve those optimizations.

Not really on this case, a common optimization pass on functional programming languages since the 70's is to flatten such kind of calls.

Rust long compile times are mainly caused by LLVM, and the amount of IR that is given to it.

Can you point me to a paper from the 70s that explains how to flatten such calls? The best that I know is "stream fusion, to completeness" which was published in something like 2019!

You can get the book

"Lisp on Small Pieces" from 1994

https://www.amazon.com/exec/obidos/ASIN/0521562473/acmorg-20

It is not 70's, but 26 years should be enough to get hold of such knowledge.

Or "Modern Compiler Implementation" from 1998, a little more young.

https://www.cs.princeton.edu/~appel/modern/

Or rather "The Implementation of Functional Programming Languages", from 1986, now getting closer to 70's.

https://www.microsoft.com/en-us/research/publication/the-imp...

I can still have a look into my SIGPLAN paper collection to dust off some ML papers.

It's easy in functional languages where execution order and side effects aren't an issue. In JS with prototype monkey-patching you can't even know that the call to `.map()`, or anything that it touches, won't suddenly replace the implementation of the `.filter()` method that follows it.

The large amount of IR given to LLVM is closely related to the amount of abstractions that Rust uses. It's especially relevant for iterators where a "simple" `for` loop expands to `into_iter()`, `iter.next()`, `match`, `drop`, and arithmetic expands to calls to overloadable operator methods, all moves are `memcpy` to clean up, and so on.

https://rust.godbolt.org/z/71aEnf9Ta

I remember the days when functional languages wasn't a synonym for like Haskell does it.

Speaking of which just use OCaml bytecode backend or Haskell interpreter to see how slow complex languages are.

This is what is missing to Rust and cranelift might help to deliver, alongside not having to keep compiling the whole world from scratch.

I don't know about JS's particular challenges here and I believe you but a lot of languages with a lot of other features also do this optimization. It's not just the hardcore functional ones though admittedly maybe mostly in the ones most influenced by them.

This is one of the idioms that leads to large amounts of IR!

Avoiding that means doing the flattening earlier, in a higher level IR.

If F#, Haskell, Scheme, Lisp and OCaml are able to do it and still remain fast compilers (there is more than one backend to choose from), I am sure Rust will manage it eventually.

I am not familiar with the compilers for all of these, but I can make a couple of interesting comparisons:

* GHC does exactly what I suggested rustc needs to do- it does the flattening on an earlier, higher-level IR than LLVM's. (It is also not exactly a fast compiler, though!)

* I'm not sure if this applies to F#, but in C#, LINQ compiles faster by not doing as much (if any) flattening in the first place.

> This is a problem of optimization, not language design.

It is also a problem of langage design though: JS’s HoFs are spec-defined as returning Arrays, which makes fusion way more complicated.

Internal / external iteration also have different optimisation profiles e.g. Rust initially used internal iteration, switched to external because that was much more conducive to optimisations, but re-introduced an internal supplement because some combinations / patterns optimisé much better through internal iteration.

This is true, it does depend on how the implementation works.

> switched to external because that was much more conducive to optimisations

Source? I thought conventional wisdom was that internal iteration was much easier for compilers to digest.

https://mail.mozilla.org/pipermail/rust-dev/2013-June/004364... Was probably the main starting point for that change. Though I misremembered performances being the main gist of the switch, ergonomics and flexibility were.

With https://www.reddit.com/r/rust/comments/5ez38g/rusts_iterator... as the… not counterpoint but reintroduction of internal iterators.

Linking to the Reddit thread rather that the article directly because lots of comments from important rust folks.

Rarely does a week go by without me wishing (for a few seconds) those js methods were all lazy.

>The higher-order function style also tends to be much slower,

This shouldn't be the case. A HoF-call should be optimized into a JMP or be inlined directly. The issue of performance is more likely to come from non-mutating functions operating on datastructures which had mutation in mind.

Here's my understanding: to get higher-order function calls to compile to fast code, you need some kind of compile-time code expansion: templates in C++, specialization in Haskell, monomorphization in Rust, etc. But if you make it too easy to have code expand into multiple instances, then you lose the fast compile times that Go's famous for, and that's a hard balance to strike.

Given a call map(a, f) and a loop for _,e := range a { f(e); } both ought to have the same code size. Yes, map(a, f) will need to be inlined, but the for-loop is already inlined.

You're right. To be clearer, the emphasis was on making it too easy to specialize: people don't just stop at maps and filters, they specialize everything, and that can have a significant impact on compilation times. For example, in Rust, people write stuff like [0] for the tiniest bit of syntax sugar, and they often don't bother extract out the non-generic part.

[0]: https://news.ycombinator.com/item?id=24088407

The current idiomatic go style is to write your own for-loops for everything. Isn't that just manual specialization executed by the programmer rather than the compiler?

I can't wait to see what comes out of the channels/goroutine generic packages! Given go's culture it would be really interesting what they deem should make the cut and what shouldn't.

Luckily, with the design of generics as it is now, we'll be able to add the parts that are missing from this proposal ourselves in custom libraries. And even if the Go team doesn't implement generating efficient machine code for higher order functions, it will only matter in the 1% of the codebase that is performance critical, and I will gladly switch that to a loop.

My biggest surprises learning Go were that it didn't have generics, and that it didn't have native map/reduce/filter. I think of many programs in terms of these three constructs and combining them.

It's interesting that some of the comments on this issue are complaining that HOFs are non-Go-like, and that if adopted, they will "take over" for-loops, which would be undesirable.

Yep, different languages have different idiomatic styles, for lots of reasons. When I’m learning a new language that’s very different, I have found that breaking through these mental roadblocks opens my mind to a lot of new possibilities and considerations. No language can be everything to all people and purposes.

I found it funny the first comment seemed to deride the use of function pointers, compare, map, and filter as too different than the for loop centric Go. Even C provides sort with a function pointer for compare...

I've seen this type of argumentation around go and it boggles my mind every time. Saying "we are not sure if we'll need a version of this function that accepts an override to the type's default comparison" only makes sense if you've never used a programming language other than go. Or the function sort.Slice for that matter.

It's always the same thing.

People find a language/library too complicated, so they go on their own to write a simpler version, only to find out that they missed very important cases that prevents it from having the same qualities.

Grudgingly, they then gradually re-introduce what they initially strongly opposed against, but now they also have to work around their current design and bogus abstractions that wont let that happen easily.

That's going to create a lot of cruft, edge cases and API complications, to the point where a bystander might decide it's too complicated and, they too, go on their own write a simpler version.

Go hasn't learned a single thing.

But at least it keeps grad students busy in the programming language departments ;). Imagine that all we had was Haskell and maybe a strict variant of it. These guys would quickly run out of topics that could be explained in less then 5min to a casual developer.

I’m not denying that there are a lot of difficult concepts and million dollar words in advanced functional programming but I will say that I personally took years to understand what was going on with OOP when I was learning programming. Functions made sense, classes and objects and inheritance and subtype polymorphism did not.

Pure functions are incredibly simple thing to understand, much like an algebraic equation you provide inputs and get an output.

There's a lot to love and learn from functional programming.

Times and paradigms change. We need to thank these people or we would still be using Java EE

More importantly the hardware changes and the patterns designed to plug platform shortcomings would have to go away.

I rather use JEE, now JakartaEE, as Go as yet to offer anything that has parity with it.

Two anecdotes why I hate the way they went with //go:generate as workaround.

Borland first attempt to C++ "generics" was the initial release of Borland International Data Structures 1.0 (BIDS) around 1990, where they used the preprocessor to generate multiple copies, something like

    #define LIST_T int
    #define LIST_TYPE MyIntList
    #include <bids/list.h>

Rice and repeat for all required types, when BIDS 2.0 came out this was already replaced by experimental templates support.

Around 2005 it was common to use Eclipse EMF framework, alongside plugins to generate Java code (<= 1.4) that would create type safe subclasses from collections with Object.

So learning is a hard process, then again this was their point of view when C was created,

> Although we entertained occasional thoughts about implementing one of the major languages of the time like Fortran, PL/I, or Algol 68, such a project seemed hopelessly large for our resources: much simpler and smaller tools were called for. All these languages influenced our work, but it was more fun to do things on our own.

> Go hasn't learned a single thing.

How do you explain the success then?

Go hasn't tried to be all things to everyone. There are lots of things out there that need simple solutions and Go provides for that with and nice middle ground of dev and computer performance.

If I want to read from one computer, write to another computer, concurrently, with low footprint, and speed, it's a great tool.

If Go has learned nothing, are Go programmers just picking something more difficult that performs worse? I've found it has dislodged services that were previously written in Node, Ruby, or a whole J2EE stack. Sure there are people out there trying to make it all things to everyone and it fails in some of those things, but it does great at others.

> How do you explain the success then?

A megacorp funding 100 engineers to work on the compiler, libraries and tooling for 10 years

In other words, the same way you explain the success of eg Java.

Java was a safer C++. Go was a safer C. They both ignore a lot of good ideas from other languages, but I find it easier to forgive Java given its age and how much it’s caught up.

Java also had multi-billion dollar marketing campaigns.

Either Go is a horrible language that did not learn anything or it is a language developed by some of the best engineers for a long time. If you are claiming that both things are true, then how do you explain it?

Google engineers are not some demigods or mythical geniuses and people should stop thinking that. They are successful within the framework of what they know, which is mostly C and C++. None of these people have academic background in language design, and it shows.

I suspect that much of Go's popularity stems from one thing that's completely unrelated to its language design woes: its ability to generate standalone binaries with minimal dependencies (and specifically none on Linux, which is prevalent for container use).

Go itself is not particularly simple, actually. It's simplistic, meaning that you have to jump through more hoops to do common things. For a concise example, consider adding an item to a slice in Go vs adding an item to a collection in pretty much any other modern language.

> Even C provides sort with a function pointer for compare...

That's been in non-generic Go for a while (passing a closure for sorting): https://golang.org/pkg/sort/#Slice

The specific argument made there is even stranger. It's essentially saying that if HOFs are adopted, they will inevitably replace for-loops for common cases - and that is a bad thing! In other words, it's trying to block a feature that it specifically acknowledges would be highly popular in the name of simplicity and purity.

Sigh. Yes, tell us more about why the style you've written 3-4 orders of magnitude more code in is inherently simpler/clearer.

Language enhancement proposals should also specify impact and expected benefit.

How much are slices used in other languages? A little? A lot?

We now have a zillion open source projects. Someone could semgrep that mountain, do some analytics. Figure out if adding slices, or whatever, is worth the bother.

Great to hear! This only encourages me to continue writing my book on generic programming with Go (https://gumroad.com/l/generic-golang)

Reply on news.ycombinator.com