Hacker News Reader

Comment by jadbox | Hacker News Reader

All I want is typesafe efficient higher order list operations like map, filter, reduce, flatmap, zip, takeUntil, etc. I don't care if Go puts them into the stdlib and only allows these operators to use special internal generics to operate. If Go had a robust higher order std library like this, I would stop asking for generics. Again, must be typesafe and zero overhead.

View on HN

One of the main motivations for adding generics is that we could then put typesafe algorithms and data structures in the standard library.

And make the standard library unmagical? One of the things I've always disliked about Go is how stdlib provides magic facilities unavailable to user code.

What's magical about the stdlib?

Maybe they meant the language, which has some generic functions like append or delete - instead of those functions being attached to a data type or accepting any type, they only act on built-in types. I doubt any of that would change though given the stated goal of remaining compatible in most cases.

There are a handful of packages (time and some stuff in os for example) that are tied into the runtime and can't be rewritten outside the standard library. It's not a huge amount, and some of it can probably be fixed in the future.

I agree, but there's not much, and it's usually for decent reasons:

* the "time" package hooks into the Go scheduler.

* the "reflect" package needs type information out of the runtime.

* the "net" package hooks into the Go scheduler a bit. But way less than it used to. I think you might even be able to do this all yourself outside of std nowadays.

What else?

Our rough plan going forward is to actually move a bunch of the stdlib elsewhere but ship copies/caches of tagged versions with Go releases. So maybe "encoding/json" is really an alias for "golang.org/std/encoding/json" and Go 1.13.0 ships with version 1.13.0 of golang.org/std/encoding/json. But if you need a fix, feature, or optimization sooner than 6 months when 1.14.0 comes out, you can update earlier.

My guess that quotemstr meant builtins: append, copy, range "protocol", etc.

There are many examples of things builtins can do which user-code can't:

1. The constructs like "x <- chan" and "x, ok <- chan" (and map indexing and so on) are only available to builtin types.

It's impossible to write a thing like "x, ok <- chan" where it either panics or doesn't depending on if the caller wanted two or one return values.

2. generic types, like 'map' and 'chan'.

3. 'len' can't be implemented for a user type, e.g. "len(myCustomWorkQueue)" doesn't work.

4. range working without going through weird hoops (like returning a channel and keeping a goroutine around to feed it)

5. Reserving large blocks of memory up front (e.g. as 'make' can do for slices and maps) in an efficient manner (e.g. make(myCustomWorkQueue, 0, 20) doesn't work)

You're talking about the language. He's talking about the standard library.

We're both talking about both. The core of my complaint is that the language privileges parts of the standard library.

> privileges parts of the standard library

It really doesn't. Maps and channels are not part of the stdlib. They're part of the language, but not the library.

Similarly, range is part of the language, not the stdlib.

If the stdlib was privileged, "container/*" in it would be generic, but it's not.

All right, but my point stands. Maps and channels should be part of stdlib, and they should rely on primitives available to user code too.

Maps and channels are not part of the standard library. How you know that is, you can replace the entire standard library, using none of it, and still be programming in Go with maps and channels.

Parent wrote "should be" not "are".

Such primitives would bring a lot of complexity which is something Go tried to avoid.

Consider that make, map, chan etc are just language features/specs just like int64, string, func, main etc

Why should string be fundamental?

Nobody said it "should" be. It simply is, in Go, the language we're discussing. If you nerdsnipe an HN thread with an argument that there's no difference between a language, its runtime, and its standard library, people are going to pile on to point out that's not true.

Could, sure. Should? I think you are asserting facts that are not in evidence.

I'm a bit confused, and I'd like to understand your position better. I can see why arrays, structs/tuples, or [tagged] unions are usually primitives – they sort of express different fundamental layouts of data in memory, so they need support from the runtime/compiler. In the case of Go, channels also make sense as a primitive – as I understand it, `select` requires scheduler support.

But higher level data structures like lists, sets, maps etc. are all implemented using those primitives under the hood. So, especially in a "systems language", I would sort of expect them to just be libraries in the stdlib, not language features[1]. But if you think otherwise (or see a gap in my reasoning) I'd love to hear why!

[1] Except maybe some syntactic sugar for list/dict/set literals.

> But higher level data structures like lists, sets, maps etc. are all implemented using those primitives under the hood

In Go you don't have generics as a primitive with which to build those things.

As a result, you don't have the building blocks for anyone to build a usable data structure unless it's baked into the core of the language.

I'm aware of that, and I read the generics proposal linked. I commented here because I wanted to respond to this:

> Should [be just libraries]? I think you are asserting facts that are not in evidence.

I tried to describe why I feel that go's collections should be libraries and not builtins, and hopefully understand why GP seemed like they were questioning that idea.

"Should" implies some normative assignment of value.

> But higher level data structures like lists, sets, maps etc. are all implemented using those primitives under the hood. So, especially in a "systems language", I would sort of expect them to just be libraries in the stdlib, not language features[1].

Maybe! But if those higher order structures are closer to language features than libraries -- and, specifically to this conversation, if they leverage parts of the runtime that aren't available to mere mortals -- is it a strictly worse outcome? I don't know. Some evidence might suggest yes. But I think it's at best arguable.

I think I understand, thanks for explaining.

What changed? Why is the Go team finally considering this now, when it has been requested for these exact reasons by many credible developers since Go's beginning?

EDIT: Yes, something has changed. They always said "no, but maybe later" and now it's "ok how about now". That is a big change and I am just asking what's the reason for it.

Nothing changed. It wasn't off the table to begin with. See: https://golang.org/doc/faq#generics

The question of the parent boils down to "what put it on the menu".

So, yeah, it was always "on the table" in some hazy, we'll see later, way, but not like that.

Likely because they have now figured out implementations of these features which they feel work well with the rest of the language design.

I think a good faith reading of the circumstance would lead one to conclude that the passage of time, and other, higher-priority and lower-cost-of-implementation issues being addressed, have put these issues on the menu.

Because now is later than when they said later?

A year ago we said it was time to move toward Go 2: https://blog.golang.org/toward-go2 . This is another step in that direction.

Then I'm a year late in asking why the Go team is now deciding to say yes to something they said no to for over half a decade.

The blog post explains that.

Adding generics is a major change (e.g. it would restructure the stdlib), and so it would only belong in a major-version bump of the language.

Though the authors have clarified that the "2" in "Go 2" is just for marketing purposes, and that the actual release would still be "Go 1.x", and shouldn't break any existing code (so it won't restructure the stdlib in any major way, unless they want to do a huge mass-deprecation).

So Go is following the exact same path as Java.

Inevitably.

Also inevitably, Java will get value types (structs) at some point.

Just like with Go Generics, people will complain the feature is bolted on. After some time it is just one of the many quirks of a complex language.

I think it was Stroustrup who said: there are two kinds of programming languages, the ones people hate and the ones nobody uses.

And Java had quite a few good examples to take on like Modula-3 and Eiffel, but alas they wanted something for average Joe, kind of sounds familiar.

The difference is that Go could have learned from Java.

> What changed? Why is the Go team finally considering this now, when it has been requested for these exact reasons by many credible developers since Go's beginning?

Simply, Go has probably accumulated all the developers it is going to accumulate without something drastically changing.

At this point, I can't think of any programmers around me who now want to learn Go.

The reason for saying "no, but maybe later" originally was that it was always a desired piece of functionality. I believe the word "inevitable" was even tossed around. The problem was that they didn't like the existing implementations of generics in other languages and also didn't know how they would improve it. I haven't had time to look at the details, so I'm not sure what they actually did, but I think they basically came up with a strategy that they are mostly happy with.

You might ask, why not pick a solution earlier and then improve it over time? However with a popular language it's not that easy. People are writing lots of code. If you start early, you can paint yourself in a corner and end up with a system that you really can't improve very much because it introduces incompatible changes. Note the extremely painful (and drawn out!) incompatible changes with Perl 6 and Python 3. If you make the wrong choice early, you might still take over a decade to find an opportunity to replace it. And since the state of the art in language design moves very slowly, it took a long time before they could see something that they felt they wouldn't regret choosing.

I highly recommend reading https://go.googlesource.com/proposal/+/master/design/go2draf...

This is part of the material linked to from this thread. It answers all your questions. You will see that some of the people who are directly replying to you are very much involved in this effort.

Edit: I tried editing the previous message but HN actually automatically made a reply instead....

I think because it would break backward compatability in v1, and not doing that was the Most Important Thing in all 1.xx releases.

v2 is allowed to Break Stuff. So obviously they're going to consider the thing that people have shouted for most in Go... generics and error handling (and package management but they found a way of doing that earlier).

What if generics were just implemented internally in Go? IMO, there are good use cases, like generic numbers (int32, int64, etc.), list processing, etc. There's already precedence for this with map[A]B, make(), etc.

I basically agree. I still think, while we're at it, making Generics a general feature of the language would probably be easier than adding all kinds of special cases.

But yeah, if Go just added a generic map/filter mechanism (Python's list comprehensions (in turn inherited from Haskell) look very nice), that would cover about %75 of the use cases I want Generics for. Add a generic mechanism to use the range operator with user-defined types, and we're at about 95%.

> Python's list comprehensions (in turn inherited from Haskell) look very nice

Do they? I find them hard to read and hard to chain. I tend to prefer basic map, filter, reduce methods

Mmmh, that is a good point. I never used nested list comprehensions, to be honest, and I have not used Python in a couple of years. For non-nested cases, they are very nice, though.

OTOH, I never used nested map/remove-if constructs in Lisp, because that, too, can get rather annoying to read.

I agree with this sentiment completely. Of course, if the only way to get these wonderful list operations is generics, then I guess I want generics, too.

You might be interested in my project: https://github.com/lukechampine/ply

The idea is that when people say "generics", they usually just mean map/filter/reduce. So ply just bakes those into the language directly instead of adding new generic programming facilities.

Those are the things that I don't want to see in Go, honestly.

I'm curious: why not?

Generally, chaining things in the style that encourages that makes it harder to debug the code, harder to step through it in my head or on paper, and harder to compile to performant code. I'd rather see loops.

On the other hand, you're less likely to need to debug those things because they've been implemented once, correctly. A manually-written filter, map, etc. will pretty much always have a higher probability of containing a bug. And in languages that have these features, I've never found it difficult to debug. What issues have you had in the past?

I've almost never had issues with manually written filters/maps -- at least, not with the iteration itself. On the other hand, having to untangle stack traces of closures of closures that have been passed through closures does slow me down when the function that's being mapped is buggy.

I guess I've never run into issues where a map or filter slowed me down for debugging, most languages will pretty easily show you where the error occurred. And if you're `map`ping and `filter`ing with pure functions, debugging is pretty simple because you just need the input that went into the buggy function. A good compiler will also optimize the closures into something you'd write by hand in Go. Overall I'm glad to see Go get these nice tools for more easily working with collections.

I've just never run into places where that style improved readability, performance, or debuggability. Outside of the Haskell world, I've usually found myself preferring loops for ease of following the code.

Thanks for your answer. I find it very interesting how different people come to different conclusions about these basic approaches to writing programs. Imho, as always, there's no one true answer and understanding where everyone comes from is very important in understanding this divergence of thought.

In my work I don't think I could live without these higher level abstractions anymore. It enables me to write clear and concise code that helps me keeping accidental complexity under control so that I can focus on the essence of what I'm trying to codify. Performance on the other hand is not a primary concern to me, given it stays above some acceptable threshold.

I appreciate how different tasks would require different approaches. I'm interested in which direction go goes (no pun intended).

At that point why not just add generics...

Lisp tends to build abstractions, too, and it also does that via macros, which are even harder to mentally parse unless you have a habit.

I'd say that to attain simplicity and clarity, you have to limit the use of abstraction. OTOH there's no way to avoid abstraction at all; programming is all about abstraction. If the means of abstraction are not powerful enough, copy-paste programming and boilerplate proliferate, making code less maintainable and harder to understand.

In my opinion, implementing these higher order functions with the help of one powerful abstraction (generics) is much simpler than having to deal with a bunch of special cased weaker abstractions (magic map, magic filter, magic reduce, magic...).

So you want a Lisp with other syntax, right?

You say it like it's a bad thing.

I was just curious, because those functionionality wished is in nearly every list processor, no mather what syntax used. And if not you macro that away, right?

That's about right. And gradual types, multi methods, generics and tight integration with C++: [0].

Dylan was too long winded for my taste, as are most modern attempts.

[0] https://github.com/codr4life/snabl

Reply on news.ycombinator.com