Hacker News Reader

Comment by qsort | Hacker News Reader

I agree with the goal of the package, but I feel it doesn't do a good enough job of picking a lane between the imperative and functional approach.

If you're going functional, a list is both a functor and a monad, so you definitely need at least map and flatMap for the abstraction to make sense.

On the other hand there are a lot of "in place", "insert", etc, which encourage working with mutation.

You probably need a bit of both, but I'd like a more top-down design better.

View on HN

> The higher-order function style also tends to be much slower, so when writing a loop there's a decision burden every time between saving a line or two and writing fast code. (TIMTOWTDI is not really the Go ethos.) And programmers usually prefer concision, so the net result is often slow, HOF-heavy code.

This is very real. We do it all the time in JavaScript:

Object.entries({...spreadMerge, ...secondSpread}).map().filter().find(x)

Not once do I think about how many things are initialized, iterated over, etc. Rarely would I do something like this in Go.

Whether it matters is a good question, but the point is true: language facilities make you care more or less about performance depending on what it does.

This is a problem of optimization, not language design. There’s no inherent reason a chain of functions like that has to be less performant than a for-loop. You need high-performance iterators in the language, and a good compiler.

One of Go's objectives is to have a simple and fast compiler; emphasizing simple and readable code will ensure that. It's one reason why type arguments were put off for as long as they have been.

`.map().filter().find(x)` is significantly more simple and readable than for-looping for every small thing in Go.

I disagree. Fewer characters != simpler or even more readable. Map/filter/etc are certainly more clever, and I enjoy writing clever code, but I reserve it for my personal projects bprecisely because it isn’t so clear. Note that Python has these first class functions as well as various kinds of comprehensions and still it’s frowned upon to use anything more complex than the simplest comprehensions.

Alright, if it's not too much trouble, can you show me what the equivalent of the following code would be in Go? I think this is extremely clear as-is, but I'd love to see how you can apparently make it even clearer by avoiding iterator combinators.

https://play.rust-lang.org/?version=nightly&mode=release&edi...

Personally I think this is clearer, but it's quite contrived either way: https://play.golang.org/p/FpUWwm5yZ8n

Nice! Out of curiosity, if the iterators weren't of the same type (e.g. one was an iterator over the values of a hashmap and the other was over the items in a list) what would you have used instead of

    for _, m := range maps { ... }?

If you could provide the hypothetical Rust code you're referring to, that would probably make your question more clear here. I know both Rust and Go rather well, and I'm confused about which direction you're wanting to take the code.

Just as with Rust, you'd have to convert both maps to the same type before you could iterate them over together (or chain them, in Rust).

Map and filter are extremely simple, far simpler than a for-loop. They can only do two things: transform items in a collection, and remove items from a collection. After mapping you know for sure that a collection will still have the same number of elements. After filtering you know that all the elements remaining still have the same type. For-loops can do anything. It’s a question of what you’re most familiar with and used to.

I’m not comparing for loops in general with map or filter, but rather the specific for loop boilerplate to implement map and filter. Moreover, the complexity of map/filter isn’t internal, but rather decomposing or breaking down one’s problem into map/filter/etc expressions. Often for loops are clearer, especially since you can frequently collapse a map and filter into a single for loop without thinking in map/filter/etc primitives. This is especially true when you need to abort the chain early (e.g., due to errors)—Rust’s FromIterator provides a clever solution to this problem, but it’s much harder to wrap one’s head around than a simple for loop with an early return branch.

I find that decomposing iteration into map and filter expressions typically clarifies my understanding of the problem and makes the resulting code more general too, and I just disagree that for-loops are clearer. Combining a map and filter into one loop body tends to create specific, imperative code that doesn’t need to be specific or imperative. For one, loops require mutation to have the same effect as filter and map, and as soon as you’re maintaining state, things get confusing. My usage of higher order functions instead of for-loops is an expression of my belief that programmers should always use the least powerful construct they can to solve a problem.

In general though I think the functions used in map and filter should be total. Most languages are not expressive enough to handle errors fluidly in long chains like that, you can do it with monads but apparently everyone hates those except me.

> I find that decomposing iteration into map and filter expressions typically clarifies my understanding of the problem and makes the resulting code more general too, and I just disagree that for-loops are clearer.

Agree to disagree I guess.

> Combining a map and filter into one loop body tends to create specific, imperative code that doesn’t need to be specific or imperative. For one, loops require mutation to have the same effect as filter and map, and as soon as you’re maintaining state, things get confusing.

I agree that immutable, declarative code is easier to reason about in general than mutable, imperative code, but it's not a binary proposition. I posit the cost of managing the tiny bit of very-local mutable state for a for loop is a lower cost than trying to model a problem in terms of map and filter. While I can appreciate the elegance in "immutable always", I think it's too puritanical for real world software development, especially in these cases which are simply expressed as for loops.

Note that languages like Python have strong support for immutable expressions and `for` loops, and it seems very common (even idiomatic) to use imperative `for` loops for anything more complicated than the simplest list comprehensions.

> In general though I think the functions used in map and filter should be total. Most languages are not expressive enough to handle errors fluidly in long chains like that, you can do it with monads but apparently everyone hates those except me.

I think it's just that monads tend to allow for very abstract code at the expense of understandability, and the abstraction that monads facilitate is often well in excess of what a given problem requires. If you're writing code to maximize abstraction (which is a lot of fun), monads (and map/filter/etc) are great, but if you're trying to work with a team to build software to solve real problems, you probably care a lot more about readability than gratuitous abstraction and the monad (/map/filter/reduce/etc) juice frequently isn't worth the squeeze.

Python’s FP capabilities are severely constrained by not supporting multi-line lambdas or composition operators (pipe, sequence, etc). It is not a good language for functional programming at all. The community norms explicitly push you away from using FP in it because of how limited it is. We use it for wrangling large data sets and tend to hate every minute of it.

My workplace uses Elixir exclusively in production for our backend services, so we’re all functional programmers and we do not find that map, filter, and reduce make things harder to read, quite the opposite. Breaking up loops does not introduce cognitive overhead, it makes every step in the chain more explicit. You can call this personal preference but there’s nothing concrete in your belief that for-loops are more readable or easier for teams of programmers to maintain, many people disagree.

I generally agree with you, but it's nice that in Rust you can use a for loop if it makes more sense. I mostly end up doing this when I want to reduce to multiple collections (put type a in here, type b in there) or call functions that return different types of errors.

The other time it's very useful is when translating algorithms specified in an imperative form. I translated a bunch of algorithms specified as for loops to elixir, and it was annoying to have an extra source of complexity when comparing the implementations for correctness.

It’s definitely nice to have the imperative constructs but spending a lot of time with pure languages has changed my perspective on mutability. I think Rust mostly fits it, data is immutable by default in Rust.

> Python’s FP capabilities are severely constrained by not supporting multi-line lambdas or composition operators (pipe, sequence, etc)

Hardly. You can absolutely have multi-line lambdas, just not multi-expression lambdas (the real limitation is that there are no try/except expressions so certain fallible manipulations aren't so easily expressed). Similarly, Python lets you overload operators so you could always override `|` or whatever. Of course, operators are just syntax sugar for function calls so this strikes me as a cop-out. In either case, neither of these excuses are pertinent to the community's rejection of complex comprehensions or higher order functions.

> My workplace uses Elixir exclusively in production for our backend services, so we’re all functional programmers and we do not find that map, filter, and reduce make things harder to read, quite the opposite. Breaking up loops does not introduce cognitive overhead, it makes every step in the chain more explicit. You can call this personal preference but there’s nothing concrete in your belief that for-loops are more readable or easier for teams of programmers to maintain, many people disagree.

You could argue that any two models which evaluate to the same thing are fundamentally equally easy to understand. You could argue that monads are just as easy to understand as a sequence of statements in an imperative language, but experience suggests that people have a harder time understanding the more abstract concept rather than the more concrete concept. This shows up all over--Newtonian physics are more easily understood than the more abstract quantum physics and in mathematics "more advanced" math tends to mean roughly "more abstract".

> In either case, neither of these excuses are pertinent to the community's rejection of complex comprehensions or higher order functions.

I can only assume you haven’t done a lot of functional programming because Python is just not expressive as a functional language. Lambdas are extremely cumbersome. JavaScript by contrast has a huge FP culture and does not even have comprehensions. But it does have syntactically lightweight, multi-expression lambdas.

> You could argue that monads are just as easy to understand as a sequence of statements in an imperative language, but experience suggests that people have a harder time understanding the more abstract concept rather than the more concrete concept.

Except I’m not arguing that, because I don’t believe that monads are easier to understand, I believe that functional programming with pure functions is. This is just another restatement of the same argument you’ve been making the whole time. We onboard people who have never used FP into being productive Elixir programmers in two weeks or less. There are no monads in Elixir, nothing more advanced than recursion and function composition. My experience differs from yours. Our new programmers have very little trouble grokking how FP works. They have much more trouble understanding the Erlang concurrency model and the various primitives available, but those are unique to Erlang and uniquely powerful.

> I can only assume you haven’t done a lot of functional programming because Python is just not expressive as a functional language. Lambdas are extremely cumbersome. JavaScript by contrast has a huge FP culture and does not even have comprehensions. But it does have syntactically lightweight, multi-expression lambdas.

You assume wrong, but more importantly you missed the point of the snippet of mine to which you are replying—namely the expressiveness isn’t the reason the Python community rejected those features, but rather the complexity and cognitive burden.

> Except I’m not arguing that,

I wasn’t arguing that you were arguing that, I was using it as an analogous argument. We’re not going to get anywhere if you aren’t going to respond to what I’m actually saying.

I really like the method style of LINQ in C# - which I would agree are much clearer than the equivalent explicit loops.

This argument doesn't really have any content, because people who advocate for programming with higher order functions would not agree that such code is more clever; in fact, they would probably argue the opposite. What's the point of debating if you assume the people on the other side already agree with you?

I really like chaining array methods but you should also never forget it's incredibly easy to introduce crazy computational complexity with them.

Rust has lazy iterators, so a chain of iterator steps like this has the same complexity as a single for loop over the iterator. There's no reason Go couldn't do that same thing.

In fact, Rust's version compiles down to pretty much the optimal looping form. It will even vectorize your iterations using SIMD.

Rust is seriously cool in this regard, but Go aspires to be different, namely more C-like and less C++-like. This means a dumber, faster compiler that is easier to intuit about, for better or worse. Rust is such an amazing piece of engineering (moreso an amazing process and culture for producing an amazing language), but I think Go’s approach is right for a whole lot of software—lots of things don’t need to be insanely performant (lots of things are chugging along in Python, which runs 2-3 orders of magnitude slower than Go) but we do need to be able to iterate on them quickly in a development context involving dozens or hundreds of other people.

Not if you have solid lazy iterator support in the language. If they’re strict and every link in the chain is adding O(n) to the complexity then yes, this is a problem.

In the case we’d need a little context of what the map, filter, and find operations are, but if map is multiple by 2 and filter is less than 200 you can just replace the filter and map with a single if in the for loop that performs the multiplications and check and return immediately when you find X. That’s pretty obvious, always takes only one run, doesn’t reallocate, and doesn’t add concepts, which is of course adding complexity.

Yes and we could replace all of that with an implementation in assembly which both compiles and runs faster.

But we don't do that because for the entire history of computing, we've come to realize that abstractions enable us to reason about programs at a higher level without having to be bogged down in irrelevant details and that getting those details right once lets us avoid the inevitable bugs that occur while reimplementing them thousands of times.

By absolutely every objective measure we know in software engineering

   let triples = ints.map(|x| x * 3)

is strictly superior to

    var triples = []int
    for value := range ints {
        triples = append(triples, value * 8)
    }

The fact that this even has to be argued is just bonkers to me.

The second one is trivial to introduce bugs into, reading it requires mentally filtering out 90% of the code as irrelevant minutiae, and it's even worse from a performance perspective unless you remember each and every time to allocate a result array of the appropriate length to avoid reallocation. None of the details of allocating a result array or iterating over indices/values are important from the purpose of expressing the problem to be solved, and yet every time someone has to read this solution those things are of greater visibility than the actual business logic.

Worse, repetitive and unnecessary boilerplate like this hides bugs. Did you catch the fact that I'm multiplying by the wrong value in the golang version? Did you catch that I'm accidentally tripling the indices and not the values themselves? If you did, would you at least acknowledge that the first bug is much easier to identify in the first example than the second, and that the second bug is strictly impossible?

As an added bonus, the first one (in Rust at least) is automatically vectorized using SIMD for optimal performance. This happens even as you chain additional operations to arbitrary complexity.

If you still somehow think the second version is "better", then whatever your argument is can trivially be repurposed to say this version is better still:

    int* triples = malloc(sizeof(int) * array_len);
    for (int i = 0; i < array_len; i++) {
        triples[i] = array[i] * 3;
    }

If those details of creating a result array and explicitly enumerating are important, why isn't maintaining your own counter and directly indexing? At least the C version somewhat encourages you to allocate a result array of the correct size.

This is the decade-long argument of "let's sacrifice all our programmers to the devil so that we don't have work as hard on the compiler". I don't buy it, and it has never been demonstrated that the cumbersome style of writing go programs is helping compilation times, yet people keep saying it as if it's a fact.

The cumbersome style seems to have become a strong part of the culture, almost a masochism about the limited features of the language. I find this ironic because I believe Go was intended to be highly pragmatic originally but it seems to have evolved an almost Scheme-like dogmatism about language complexity.

Thank you for naming this. I've always struggled with how to convey this to people.

Do you have specific examples you’re thinking of? Calling it masochism is weirdly insulting. Adding complexity is a one-way street. Languages live for decades. There’s no reason to rush adding features in a half-baked way, and the tradeoffs in terms of compiler complexity are very real.

Virtually every large go program I've read has asymptotically reached 50%+ of its code being

    if res, err := fn(...); err != nil {
        fmt.Errorf("Look ma, I'm a human exception handler: ", err)
    }

This is held up as a positive thing, because handling errors is explicit and good. Somehow, the Rust equivalent of `?` is implicit and bad, despite desugaring to almost literally the exact same thing and being actually impossible to forget or overlook due to the return type of the function enforcing a Result.

In the same breath, devs will assert that the language is "low boilerplate" (despite 50%+ LOC being just one type of boilerplate) because Error Handling is Important. Which seemingly ignores that the important part of error handling is actually ensuring that errors are handled and not the mechanistic act of self-flagellation by wrapping every line of code in nearly-identical error handling ceremony. Which is unfortunate because golang doesn't do so great a job with the first part. It's surprisingly easy to actually forget to do something with the error while carrying on with a return value whose semantics are undefined (unlike Rust's Result, where you can have an error or a value and not both).

I used to hardly think about exception handling on the jvm, mostly let them bubble up and be centrally handled except in special cases. Now I'm always worried that I'll forgot to manually handle the error in Go. Can't tell you how many times I've gotten panics as well, from nil objects.

I use Go a lot and I really like Rust, so maybe I can add some context. Error handling boilerplate isn’t a big deal. Lines of code aren’t a big deal. That’s not where your bugs are coming from and you aren’t bottlenecked on your typing speed. It’s not a real problem, but rather a dogmatic pursuit of terseness for its own sake. Rust is an expression-oriented language—it needs something like ? because without it we would have gratuitous nested braces marching ever-rightward and that actually is laborious to keep track of (deeply nested match expressions). Go could drop some error handling boilerplate with an ? operator, but it wouldn’t benefit as much from it as Rust did.

Lines of code are a very big deal in an organizational context that cares about test coverage. Since all the error handling ceremony is written by hand, so too must a corresponding test case for each instance of the error-handling ceremony be written by hand. Typing out test suites for functions that compose several other error-returning functions is bureaucratic drudgery.

I'll bite. Why not ditch the "% test coverage" metric then, since as you just elaborated, it's not a good metric.

I would prefer my language wasn't designed around corporate bureaucracy personally.

I don't think the coverage metric is necessarily wrong to highlight error return branches: they are program logic you typed, and might have mistyped. I certainly have mistyped them. Checked the wrong error value, or returned a different error from the one I checked. You get to skip tests for default exception handling behavior in e.g. Java because you know the language is going to do; there's no opportunity for human error.

It is possible to have company policies against abstraction and cleverness when working in more expressive languages, and this would achieve almost the same intent as Go, it's just that actually using Go is more convenient. The language is corporate bureaucracy. The whole point is what's not there. What can't go off in a weird directions on a per-project basis. What junior engineers in a high-turnover environment can't screw up. It's not by accident that the native toolchain will give you a test coverage report with untested error return branches in red. Google is just happy to pay people for that drudgery, I guess.

> You get to skip tests for default exception handling behavior in e.g. Java because you know the language is going to do; there's no opportunity for human error.

I fundamentally disagree with that. the only thing you know in java is that it throws an exception that's maybe caught somewhere. That's an untested branch aswell.

You don't have to write a test for "if this underlying call throws an exception, that same exception is propagated up to me," which is about 80% of the test cases I write for Go.

You do have to test cases where exceptions are supposed to be caught and handled/eaten, which I agree is meaningful and relevant in all languages.

You “don’t have to write the test case” in the case of exceptions, but then you’re not getting the same coverage as with go. Just because a branch isn’t represented syntactically doesn’t mean it doesn’t exist. Note also that Rust’s ? operator isn’t going to be saving you from anything here since it just uses fewer characters to express the same branching logic, while languages with exceptions use none.

In general your org’s policies for what to test seems like a recipe for a low-value, high-effort test suite. Reminds me of the unit tests for getters and setters in Java.

You get to take the language for granted; you don't get to take your own code for granted. "String reverse" is an easy function to write and the same instructions end up in my binary either way. But if you write your own, you'd better test it. If you import one from the standard library, no need, the stdlib has already done that. Functionality being in the language does not just save you typing; it also saves you testing. That's my point.

I agree, given that Go is what it is, it's a bad policy.

You aren’t testing whether exceptions propagate correctly, you test that you catch or don’t catch them appropriately just like you test whether or not you return your error appropriately. That said, doing this for every function you write is probably not going to lead to a high value test suite.

“Make sure it doesn’t catch the exception” is like “make sure it doesn’t do a tap dance” - there are quite a few things a function could do, we don’t generally ask unit testing to prove negatives.

If you’re going to check that a function propagates an error in Go it equally makes sense to check that you check to make sure the equivalent Python/etc function propagates the exception. Of course I don’t think testing either of these is a good investment.

Consider these three points:

* it has been actually demonstrated that shorter programs contain less bugs,

* code is read much more frequently than it's written, and to the point code is easier to read than code full of boilerplate and meaninglessness syntactic elements,

* menial tasks and distractions introduce mental fatigue that consumes your limited brain resources that you need to think about the structure of your code that is actually dealing with the problem at hand.

Thanks for this, it's exactly what I had intended to reply with. Every single one of these points is spot-on.

I’m skeptical about the first bullet—is it the fewer characters or the reduced complexity which results in fewer bugs. If the latter, then it undermines your point; if the former the I’m very skeptical as it would suggest we should all be writing minified code.

With respect to the second bullet, I agree that code is read more frequently than it is written and that readability is paramount. I strongly disagree that minimizing the character count is the same as optimizing for readability. In particular, I find very syntactically terse languages like Haskell or Ocaml far harder to read—indeed ReasonML exists precisely because so many people find OCaml so difficult to read.

With respect to the third bullet, you should think about your error control flow. The structure of your code doesn’t change whether you’re using Go’s “if err != nil” or Rust’s ? operator. The difference is a dozen characters. I can’t believe that the real cognitive load is in the actual pressing of keys, but if that is your experience then text editors often have a “snippets” feature or extension that will let you map that boilerplate to a hot key.

I don’t think masochism is inherently negative and I did not mean to insult. For that I apologize. I was trying to characterize in a humorous fashion a difference between what I perceive to be the stated aims of the language and the experience of actually writing it.

It was also an oblique reference to this talk which is one of my favorites: https://youtu.be/mZyvIHYn2zk

The Go compiler is much much faster than C++ and Rust. I don’t think that’s seriously disputed by anyone, is it?

There’s no universally correct answer to the right balance of “smart compiler” vs “simple language”. Having worked with Go on a daily basis for five years now, I really like the simplicity of the language. Compared to the cognitive load imposed by the language when using C++ and Rust, I much prefer the naive straightforwardness of Go. There’s no devilry about it.

Go vs C++ is not a fair comparison, because C++ is a badly designed language from the 1980s. It also serves a different domain. Certain applications of C++ are not serviceable by Go.

Go vs Rust is not a fair comparison, because Rust's type system is much more powerful that Go's and enables applications that Go's doesn't (like embedded systems programming). Also some of the things I said about C++ apply here.

There are also features of Go absent from C++ and Rust (reflection, garbage collection). So it's really puzzling that you'd pick those two.

To have a fair comparison, you'd need to look at something like Kotlin sans the class system. There Kotlin is a clear winner, and with the same engineering resources put behind it, a cut-down version of Kotlin would easily be able to achieve build times in the ballpark of generic-enabled Go.

Despite its claimed simplicity, Go has inconsistent design and bug-prone unwieldy features (try to write code that merges streams of data from two channels without blocking unnecessarily). It also has unnecessary constructs that contradict its claim to include only what's necessary (interfaces are just structs of function pointers, type methods are just functions with a glorified first argument).

It may be enough to write one-off command line applications, but it doesn't give developers the tools to create complex distributed systems in a systematic, clean way, which was supposed to be its main area of application.

> and with the same engineering resources put behind it, a cut-down version of Kotlin would easily be able to achieve build times in the ballpark of generic-enabled Go.

So basically no real example. It is only what would be possible if things were to happen certain way.

> It may be enough to write one-off command line applications, but it doesn't give developers the tools to create complex distributed systems in a systematic, clean way

This is just arrogant opinion. Distributed tools and applications like NATS, Kubernetes, Docker, CockroachDB and so many other exist and written in Go. Developers who can't write distributed systems themselves can ofcourse buy or license tools from vendors. And at that point, honestly, it could have been written in any language.

If your team has to write and review boilerplate by hand, that’s millions of times slower than a computer generating it. The compiler only looks faster when we exclude all that wasted effort.

This seems like the “sufficiently smart compiler” argument. A sufficiently complex compiler could certainly optimize this away, but building and maintaining such a compiler while also keeping compilation times low is an enormous challenge. Rust does an impressive job here, but Rust was built from the ground up to handle these problems and it also trades compile times to achieve those optimizations.

Not really on this case, a common optimization pass on functional programming languages since the 70's is to flatten such kind of calls.

Rust long compile times are mainly caused by LLVM, and the amount of IR that is given to it.

Can you point me to a paper from the 70s that explains how to flatten such calls? The best that I know is "stream fusion, to completeness" which was published in something like 2019!

You can get the book

"Lisp on Small Pieces" from 1994

https://www.amazon.com/exec/obidos/ASIN/0521562473/acmorg-20

It is not 70's, but 26 years should be enough to get hold of such knowledge.

Or "Modern Compiler Implementation" from 1998, a little more young.

https://www.cs.princeton.edu/~appel/modern/

Or rather "The Implementation of Functional Programming Languages", from 1986, now getting closer to 70's.

https://www.microsoft.com/en-us/research/publication/the-imp...

I can still have a look into my SIGPLAN paper collection to dust off some ML papers.

It's easy in functional languages where execution order and side effects aren't an issue. In JS with prototype monkey-patching you can't even know that the call to `.map()`, or anything that it touches, won't suddenly replace the implementation of the `.filter()` method that follows it.

The large amount of IR given to LLVM is closely related to the amount of abstractions that Rust uses. It's especially relevant for iterators where a "simple" `for` loop expands to `into_iter()`, `iter.next()`, `match`, `drop`, and arithmetic expands to calls to overloadable operator methods, all moves are `memcpy` to clean up, and so on.

https://rust.godbolt.org/z/71aEnf9Ta

I remember the days when functional languages wasn't a synonym for like Haskell does it.

Speaking of which just use OCaml bytecode backend or Haskell interpreter to see how slow complex languages are.

This is what is missing to Rust and cranelift might help to deliver, alongside not having to keep compiling the whole world from scratch.

I don't know about JS's particular challenges here and I believe you but a lot of languages with a lot of other features also do this optimization. It's not just the hardcore functional ones though admittedly maybe mostly in the ones most influenced by them.

This is one of the idioms that leads to large amounts of IR!

Avoiding that means doing the flattening earlier, in a higher level IR.

If F#, Haskell, Scheme, Lisp and OCaml are able to do it and still remain fast compilers (there is more than one backend to choose from), I am sure Rust will manage it eventually.

I am not familiar with the compilers for all of these, but I can make a couple of interesting comparisons:

* GHC does exactly what I suggested rustc needs to do- it does the flattening on an earlier, higher-level IR than LLVM's. (It is also not exactly a fast compiler, though!)

* I'm not sure if this applies to F#, but in C#, LINQ compiles faster by not doing as much (if any) flattening in the first place.

> This is a problem of optimization, not language design.

It is also a problem of langage design though: JS’s HoFs are spec-defined as returning Arrays, which makes fusion way more complicated.

Internal / external iteration also have different optimisation profiles e.g. Rust initially used internal iteration, switched to external because that was much more conducive to optimisations, but re-introduced an internal supplement because some combinations / patterns optimisé much better through internal iteration.

This is true, it does depend on how the implementation works.

> switched to external because that was much more conducive to optimisations

Source? I thought conventional wisdom was that internal iteration was much easier for compilers to digest.

https://mail.mozilla.org/pipermail/rust-dev/2013-June/004364... Was probably the main starting point for that change. Though I misremembered performances being the main gist of the switch, ergonomics and flexibility were.

With https://www.reddit.com/r/rust/comments/5ez38g/rusts_iterator... as the… not counterpoint but reintroduction of internal iterators.

Linking to the Reddit thread rather that the article directly because lots of comments from important rust folks.

Rarely does a week go by without me wishing (for a few seconds) those js methods were all lazy.

I think it's to do with the community and what people who are really into Go are interested in.

The first reply to the thread deems "map" and "filter" borderline-useful, but "reduce" not worth it - that type of misunderstading just shows that the author (and the 60+ people who +1'd it) didn't spend enough time understading functional constructs yet.

Reduce is basically "loop over this list while maintaining state". If you have loops and state built in, it doesn't buy you a whole lot.

I don't think the aim of the package is to move Go towards a more functional style. Rather, it's to provide a set of utility functions that mesh with the style of existing Go code. For example, it would be useful to have a utility to 'pop' an element from a slice, even though that's not a particularly functional idiom.

I definitely wasn't arguing about what should or should not be in the package. I just replied to the parent comment with my idea as to why the discussion around this point is happening in the proposal.

Or maybe they understand them perfectly and have come across enough obtuse code using reduce to not want it as part of the language.

Everyone draws the "code that I like looking at"-line at different places.

Despite our wishes around that, functional constructs do have a hierarchy of abstraction vs power, and skipping a layer is the type of design that backfires in the end, like with Java doubling down on the collection APIs without introducing monads explicitly for users.

> Java doubling down on the collection APIs without introducing monads explicitly for users.

I'm not necessarily disagreeing with you, but how has that backfired for Java? It seems as popular as ever, and I don't hear many people lamenting the lack of monads in Java.

Ah thanks for pointing that out as I didn't make myself clear. I definitely didn't mean Java as a whole, as it is indeed a tremedously successful platform and language.

I meant specifically the lack of flexibility in some regards because of the missing of monads and friends.

For example, we have a few newer languages, TypeScript and Kotlin come to mind, implementing the "null operator":

   thing?.fieldA?.fieldB

While this is quite handy, and people unfamiliar with the Option monad will think that this new syntax is really cool, it poses two limitations: 1. It doesn't compose with other abstractions 2. I can't use this for my own abstractions

If the language designers had chosen to take one step back and implemented it in a more generic way (in some sense, in the same as my other comment in this thread about implementing "map" without "reduce"), the feature could've still been implemented in the language, but without the limitations.

I'm not sure what you meant with this point:

> people lamenting the lack of monads in Java.

Do you mean that, because you don't see people lamenting, it's not a useful feature?

> It doesn't compose with other abstractions

But monads don't compose with each other, do they? I thought you needed something more (monad transformers, algebraic effects, etc.) to get things to compose. Or maybe you meant something else by composition.

> I can't use this for my own abstractions

Yep. But Java doesn't even let you overload addition afaik, so being able to overload ? is pretty much a pipe dream.

> Do you mean that, because you don't see people lamenting, it's not a useful feature?

No, it's just that if something "backfired" for Java, I'd expect to hear people complaining about it.

> But monads don't compose with each other, do they?

They do if they're[0] traversable (cf literally `Data.Traversable` in Haskell), which almost all specific monads are. You'd have something like:

  newtype Comp f1 f2 a = Comp { unComp :: (f1 (f2 a)) }
  instance (Monad f1,Monad f2,Traversable f2) => Monad (Comp f1 f2) where
    (Comp a) >>= k = Comp $ a >>= (map join . traverse (unComp . k))

Edit: 0: actually, only the inner monad needs to be traversable.

Looks like that that's not a valid monad in general.[0] And also, I feel like it's a stretch to say that "almost all specific monads" are traversable.

[0]: https://stackoverflow.com/questions/42284879/is-the-composit...

I'm not convinced (Bool->) is a legitimate Traversable[0], but that's at least highly concerning and makes me wish the typechecker could properly handle "Instances should satisfy the following laws" comments as actual code.

0: in vaguely the same way that Int isn't a legitimate Monoid because you get nonsense by switching back and forth between sum and product (and min, max, xor, and, or, etc)

This is not about misunderstanding. The author argues that heavy map/filter/reduce does not fit with the general style of Go programs. As a heavy Go user, I absolutely agree.

There is nothing preventing people from writing their own reduce() in their own utilities package, and I expect a thriving ecosystem of "even-more-functional-constructs" packages to spring up as soon as generics hit stable.

The discussion in this proposal is not about this. It's only and entirely about what the stdlib should provide, and that is much more a question of practical and idiomatic usage than it is about one's understanding of a theoretical CS syllabus.

If you re-read the comment I was replying to I don't think your comment as a reply to mine makes a lot of sense.

I'm not arguing what's good and bad, nor what should or should not be in the Go language.

I need to update this for the latest generics syntax, but I wrote this a while back: http://www.jerf.org/iri/post/2955 "Why Go Getting Generics Will Not Change Idiomatic Go"

I do not understand the people who think that functional-esque programming is the only and best way to do things who are apparently just sitting and waiting for Go to be able to do it. Are you forced to use Go or something? This is an honest question, btw, and I am legitimately interested in answers. If the only reason or the biggest reason you're not using Go is that you can't import map-centric programming into it, why not use any of the many languages that support that rather than griping about a language where it's still ultimately going to be bodged on? Again, honest question, because there's no shortage of such languages.

My perception is that Go isn't that popular that there's a lot of people forced to use it against their will because it's just The Language.

Anyhow, even after generics drop, it still isn't going to be any fun. I expect to use a map here or there often enough that it'll be nice to have it in the standard library but there's no realistic way that it's going to become "the best way to program in Go". If you're waiting to try it out after the generics drop, I would suggest trying something else instead; Elixir, Nim, Crystal, Pony.

Many problems for which you might end up using something like Kafka streams just lend themselves nicely into the streams/collections -like style

I am happy using Go right now when they don't have map.. but including map and not including reduce/flatten is just odd

"Many problems for which you might end up using something like Kafka streams just lend themselves nicely into the streams/collections -like style"

Have you actually taken the time to write out what it looks like, as I have in that blog post above, though? It's not pretty. Any sane Go programmer is not going to want to use this for very much. (I write that carefully. I anticipate using it a non-zero amount myself. But it's not going to be a lot.)

I understand liking map/filter/reduce, even if it is the beginning of functional programming rather than the end. I don't understand absolutely demanding it in languages where it doesn't fit well and insisting on using it no matter the cost.

it's pretty clear to me. people coming from other languages are used to typing less characters and expressing their ideas like that, and they want to type less characters in go.

for go, a language that idiomizes variable names like `f`, you think that people would be more on board with fewer characters. There are many other areas where go has chosen to avoid tidying up the syntax/making lives easier in the name of simplicity, though I digress.

As for your earlier question of "why don't you pick another language," I think this is a silly question to ask. The vast majority of people working for hire are not picking their language. If you have a few hundred kloc written in go, I doubt very much people are in a position to casually introduce new languages, especially if they're trying to improve the current system.

"it's pretty clear to me. people coming from other languages are used to typing less characters and expressing their ideas like that, and they want to type less characters in go."

It's not. It's more. A lot more sometimes, plus the indentation for trying to chain maps together is ugly.

I think more people who think this is going to be great need to clock some time on the Go Generics playground [1], implement some Map and Filter and MapError functions, and convert some real code to this style in real Go. It's awful. I'm probably spoiled by Haskell and how nice it makes all this, but it's awful even against normal Go code. Try it.

The problem is, the thing that makes "functional-style" programming in Go difficult wasn't "missing generics". It was, missing generics, somewhat heavy-weight anonymous function syntax, indentation rules not optimized for deeply-nested functions, a compiler that almost certainly won't optimize through very many layers of these constructs, the inability to put methods on core types and the fact the core types are missing the useful methods/interfaces like "Traversable", no higher kinded types so no monads or anything else like that, a variety of little syntax issues that make this all slick, and probably some things I'm missing. Just getting some generics (and not having generic methods that can add new generic types hurts this style too) just isn't enough to make it slick. Go would need a suite of changes to make this slick and efficient, not just these generics.

[1]: https://go2goplay.golang.org/

Your blog post was a great read. Completely agree. I’ve seen so much stuff turn to excrement the moment FP is applied.

I’m writing a complex aggregator in go and the abstraction is buffered channels. There is no map or filter in sight. It is not needed.

I dunno, for some domains it feels like 80% of the job openings are using Go.

> Are you forced to use Go or something?

I mean, yes, most companies have a relatively small list of supported languages, whether explicitly or practically. If someone at our company wants to write a program that does some binary data munging that's even more frustrating in Java, and needs to do it faster than Python allows, they have to use Go. We will not let them use C or C++, because most of the team will not understand it and it won't be able to integrate with any of our standardized libraries or CI infrastructure.

Your statements in the blog about performance of that pattern are not accurate to the best of my knowledge.

A program structured as a sequence of short, tight loops over vectors/slices as described is more or less hitting the performance sweet spot of modern microarchitectures.

The general form of a short loop over a vector plays to the strengths of branch prediction and the instruction cache while the temporal and spatial locality of the memory accesses are conducive to caching and pipelining.

Moreover, there's usually no need to allocate or copy the array in that sort of data flow. I mean, unless you're chasing worst case performance to make a point or something. Slice the buffers out of a pool and allow ownership of the data to follow the execution context, then you're free to modify it in place.

The above points cover 3-4 orders of magnitude real world performance.

It is true that there are some issues with the optimizer working in this pattern but I have only seen performance severely degraded in this pattern by compile time type ambiguity -- some but not all interface{} parameters, anonymous functions / closures in scopes with dynamic type. Interface types with a pointer receiver do not typically encounter these specific issues, and I believe the empty interface type has much better performance since ~1.12 or so as well.

"A program structured as a sequence of short, tight loops over vectors/slices as described is more or less hitting the performance sweet spot of modern microarchitectures."

Sure, but even faster is not to loop over intermediate arrays at all, by virtue of never constructing them in the first place when they aren't necessary.

"Moreover, there's usually no need to allocate or copy the array in that sort of data flow. I mean, unless you're chasing worst case performance to make a point or something. Slice the buffers out of a pool and allow ownership of the data to follow the execution context, then you're free to modify it in place."

That starts getting into "I'm sure someone can come up with some solution that meets some of these goals". Whatever map you're talking about here isn't one that is defined as creating a new slice based on mapping a function over the old slice. I mean, it kinda sounds like you're saying "well, if you just write a conventional for loop you can do all this in one pass" to me? Which is my point? I'm not the one pitching for lots of array creation, it's people who insist on using maps and filters in a language that, of all the major languages, just isn't going to put in the optimization time to convert them back into loops under the hood.

> That starts getting into "I'm sure someone can come up with some solution that meets some of these goals".

The sooner we, as engineers, can collectively acknowledge that we're doing things so wrong it's costing us approaching 4 orders of magnitude performance, and agree to stop dismissing knowledge of the hardware as forbidden knowledge, and abusing acceptance of reality as pointless micro-optimization... then I'm sure we'll make progress towards accepting one of the many patterns people have been advocating for upwards of a decade which do solve these problems.

One of the patterns which happens to be able to reclaim most of those missing 4 orders of magnitude performance is functional data flow, which is by unfortunate coincidence essentially the pattern you're denouncing here for its performance. It actually maps almost directly to the ideal implementation because it's ... literally expressing problems in terms of vector operations with no data dependence, which maps perfectly to a parallelized pipeline of the instruction types that achieve near optimal throughput and realized IPC.

I am not trying to be a dick but I am very passionate about this topic and I disagree very strongly with your assessments of the performance here.

I think the pragmatic current realization of a functional data flow pipeline as described in my earlier reply incidentally achieves exactly what you mention here:

> Sure, but even faster is not to loop over intermediate arrays at all, by virtue of never constructing them in the first place when they aren't necessary.

In those examples of this type of pipeline, the vectors you start with are slices of hardware device descriptor queues which reference a DMA region, and by progressive transformation of the receive buffers and composition with additional buffer regions through intermediate decoded states you produce reply buffers.

The hardware is programmed to sample only the packets of interest to this descriptor queue, and copies the packets matching this filter directly to the DMA region referenced in the descriptor it places in the queue.

With sufficiently sophisticated hardware it is possible to offload decode of increasingly large fragments of protocol logic or even entire applications.

Description of the operations in a language of common composable operations over the type of vectors of buffers allows the amount of any given application which is mapped to e.g. general purpose CPU, GPU or other FPGA/ASIC offload feature instructions to vary continuously over time or API surface at runtime pretty neatly, it creates breakpoints in the logic flow that more or less always map exactly to functional hardware boundaries, because everything's pretty much just vectors of buffers all the way down really when you think about it. Your process's entire runtime is just another vector of buffers to the kernel.

This is, in part, why I'm generally not in favour of generics in Go.

Go has a common style across most code bases I've seen. Partly that's because Go is opinionated about style, and doesn't really support doing things any other way. I don't see this as a bad thing, though that may well be because Go's style happens to be similar to my own. I have friends who learned to code on Ruby and JS and who don't like Go at all, because it doesn't really support the way they like to write code.

Adding the ability to write functional-ish code to Go is going to fragment the style. There will be code bases written in functional-ish Go that are difficult to understand for people used to a more imperative style. And vice versa (though I suspect imperative is easier to understand). I don't see this as a good thing. It's kinda "if you want to write functional-ish code, why not pick a language that you can do that in, instead of demanding that Go supports it?". But I realise this is probably a lost cause, and we're going to see that wave of "even-more-functional-constructs" become so common that we end up with JS-like library churn.

I'm not really a Go expert, but I've dabbled a little bit in it and I like it a lot. Why do you believe generics specifically would cause fragmentation? Isn't it more of a standard library issue?

Probably I'm looking at it the wrong way, but it looks to me like it's a relatively harmless language extension with the added bonus of correctly typing stuff normally cast as interface{}.

Probably because generics allow for many different programming paradigms, and different packages will use different paradigms, requiring developers to understand all of them and integrate them. In addition to varying paradigms, packages will also vary in their degrees of abstraction—generics permit very abstract code while idiomatic Go code is not particularly abstract. Note that I’m not arguing that these tradeoffs aren’t worth the while, just that they do in fact exist.

I've been writing Go for 8+ years now, I guess. In all that time I've maybe hit 3 or 4 problems that generics would have helped with. And there were always workarounds (tending to be heavy on the boilerplate). This is fairly common, from the other Gophers I've talked to.

The problem with generics is they're a hammer, and everything is going to look like a nail. Instead of being a rarity, and everyone basically writing idiomatic Go as normal and only pulling out the generics when really needed, this functional-ish style that relies on generics (as in TFA) will become common.

Devs who are used to writing array.map code in JS will start on Go and immediately find the slices package in the standard lib, and create a generic for every single slice they use, because that's what they're used to and how they think it's meant to be used. For...range loops will look as old-skool and primitive as for loops look in modern JS (despite all their advantages).

We already have the "learning path" of newbie Gophers, whose first question is always "which framework should I use?", and it's an uphill battle getting them to accept that they should just use the standard lib. Trying to get them to accept that they basically never need generics is going to be harder.

Old crusty Gophers like me, who are used to and like the simplicity of idiomatic Go, will be left shouting into the void. Again.

edit: typing stuff as interface{} is also one of those things that you learn to get past. I'm working on a ~80Kloc Go code base at the moment, and have not had to resort to interface{} once yet. Interfaces are way more powerful than they appear at first. I'm not saying interface{} doesn't have its place, and it's used heavily in the standard library, but it's not as common as you'd think at first glance.

The biggest problem is the opportunity cost, and you may not be aware of just what you're giving up if you've not done substantial work in other ecosystems over the years.

The two big places this shows up in golang, imo, is the lack of high quality data structure libraries, and the lack of widely used standard library for routine channel management tasks.

The former is a huge opportunity cost. If I'm coding on .net, jvm, c, or c++ I have a variety of state of the art data structures available. Some of these structures are extremely difficult to develop, so the "just write your own boilerplate" approach is completely insufficient. Point to code generation templates all you want, the reality is that this problem is preventing golang from improving much past using mutex protected hash maps and google's interface{} typed btree. Meanwhile, if we look at say .net, you have a very high quality lock free concurrent hash table available that scales to massive heap sizes. This is no niche thing: the bw-tree / cosmosdb storage engine is entirely designed around the capabilities of this lock free map. It's easily responsible for 100's of millions of dollars of value to software in that ecosystem now.

The lack of simple, clear, and useful channel lifetime management functions is another rough edge that bites many go programmers daily. The many abuses of Context make this problem quite clear.

I understand the sentiment of not wanting golang to turn into the Stephanov style C++ generics. But I think that's an imaginary fear. The golang generics design is quite different and far more "gopher" for lack of a better term. The problems golang is facing due to poor type flexibility in this area are not imaginary: they are endemic.

I used to code in C# before Go. This isn't my first rodeo. Or even my second.

And I get that Go lacks in some areas. That's fine. All languages lack in some areas. Go wasn't designed to be all things to all people. It's very good at writing servers in. And there's a few projects written in Go that have proven to be pretty popular and scaled well ;)

It's not really YAGNI, which Go has been accused of (a lot), with some justification. It's more that simplicity is valuable, but it has a cost. Keeping Go simple is good, but the price of that simplicity is sometimes high. As you say, there is an opportunity cost.

I still think that cost is worth paying, and I'm not convinced (yet - I may be in time) that the reduction in simplicity that comes with generics is a price worth paying for the flexibility it brings.

> There is nothing preventing people from writing their own reduce() in their own utilities package, and I expect a thriving ecosystem of "even-more-functional-constructs" packages to spring up as soon as generics hit stable.

I really hope this doesn't happen because then you have random utility libraries that "infect" projects for what is essentially basic flow control. I would say stuff like this is best supported in the standard lib and not home rolled.

> The first reply to the thread deems "map" and "filter" borderline-useful, but "reduce" not worth it - that type of misunderstading just shows that the author (and the 60+ people who +1'd it) didn't spend enough time understading functional constructs yet.

It’s not very charitable to assume that people who disagree with you must not understand your position. In particular, one person could like reduce() because it’s very abstract and can be used for lots of problems, but critics could dislike it because very high levels of abstraction impede readability. Both parties may understand each other, but they are optimizing for different things (abstraction for its own sake and readability, respectively).

I definitely didn't mean it as an insult, as my comment is a reflex of my own journey understanding these concepts. I'm also absolutely not an FP expert. Different people have different levels of understanding about different things, this has nothing to do with intelligence or a general programming skill level.

I do think this is about understanding, as it's not about syntax, or what feels nicer. It's about building blocks and what comes before what. Given the nature of the proposed library, to give the users an arbitrarily less abstract construct is a design mistake as I mentioned in another comment in this thread.

For example, I remember when I needed to flatten a bidimensional array in JavaScript, perhaps back in 2005, and the internet told me that I needed to flatten the array. So I learned then that array.flatMap() served that purpose: to flatten a bidimensional array. This is of course incorrect, but it took me many years until I found the concept again and understood what were the building blocks around flatMap, and what kinds of other abstractions it allowed.

> I do think this is about understanding, as it's not about syntax, or what feels nicer. It's about building blocks and what comes before what. Given the nature of the proposed library, to give the users an arbitrarily less abstract construct is a design mistake as I mentioned in another comment in this thread.

Per my earlier comment, it's only "a design mistake" if you assume that it's always better to trade readability for abstraction. This isn't always the case, and in professional software engineering it's often better to optimize for readability--rarely are high degrees of abstraction beneficial (of course, someone will respond to this with some anecdotes of the instances in which high degrees of abstraction are helpful, which is just an enumeration fallacy).

Sounds about right. Python 3 demoted reduce from builtin to stdlib because it saw very little use after the introduction of the more common use cases as builtins.

Bingo, Go is geared towards junior programmers. Somebody who exits the university has to be able to feel productive within two weeks of using it. That's literally the cornerstone of the entire design and all things follow from it (C-like syntax, attempt to minimize the number of language constructs, and the heavy focus on imperative programming which is what they teach in college).

Oh the unintended irony. Go's syntax may be simple enough to pick up in a few weeks but writing "good" Go code turns out to be much much harder. Idiomatic Go can take a year or two to internalize and introducing non idiomatic Go code into an established code base can really muck it up.

You can see this clearly in job listings looking for Go coders. Almost nobody is looking for junior level Go coders with less than a year of experience. Companies are looking for Go coders with at 2, 3, 5+ years of experience instead.

A simple syntax may provide benefits, but it doesn't seem to be the benefits most companies want in practice after all.

> Almost nobody is looking for junior level Go coders with less than a year of experience.

I have never seen a job posting asking for less than a year of experience with anything.

True, but what other languages or other anythings are doing is at best irrelevant and at worst the entire point.

Reply on news.ycombinator.com