Generics facilitators in Go
rakyll.org
rakyll.org
I think they'll be pretty common. It's a simple structure, and simple to manipulate.
Anyone have any more? I'm curious what people come up with for goroutines/channels.
EDIT: Superior for most non-performance sensitive use cases, that is.
For people not familiar with "monadic async", it's the formal way to refer to the function coloring problem [1]. Monads offer you ways to "lift" your functions/values in the monadic world (paint the blue values red), but you can't do the opposite (paint the red functions/values blue).
[1]: http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y...
It is obvious that a channel is not the same thing as a promise. They are quite different in many ways. But the problems that you solve with promises in some languages are solved with channels in Go. There's pros and cons to each, but the cons of channels aren't all that significant in the specific context of Go and there are some compelling pros to the channels in the specific context of Go, so there isn't a very fertile space left over for a promise library. I've already seen half-a-dozen go by (non-generic, but missing generics aren't the problem) and those are just the ones that get posted to reddit.
I advise writing Go in Go, and not Javascript in Go. But it's your codebase.
(Promises are basically an inner platform [1] for functionality not provided by the base language. Go provides the requisite functionality in the base language. I think I'm more hostile to inner platforms every year, but at least when you're adding capabilities to the base language you can't get any other way there's a debate to be had. Adding an inner platform to get functionality that already exists is just a bad plan.)
What you said is accurate in the context of Go though. Having promises in Go will create an "inner platform". If one wants to nitpick, there may be certain workloads that would perform better with async functions and promises, but the design philosophy of Go would happily trade a small amount of performance for simplicity.
That's fair, and I will try to update my internal mental model and stop unconditionally referring to them as such. Thank you.
(My primary objection to them is the way they throw away the stack and throw away structured programming as a result, but they're at least less bad if they aren't also an inner platform. :) )
Structured concurrency is orthogonal to whether you're using a thread-based or monadic API. Goroutines, for example, are both thread-based and unstructured. On the other hand, you can have concurrency that is structured but still thread-based (like Trio), unstructured but monadic (like Scala's built-in Future type), or both structured and monadic (like Rust's Future type).
> Asynchronous and synchronous code cannot always be combined freely. For instance, you can't directly call an async function from a sync function.
Same with JavaScript from [2]:
> The await operator is used to wait for a Promise. It can only be used inside an async function within regular JavaScript code; however it can be used on its own with JavaScript modules.
In JS, the promise world is a different world. Exceptions won't work like they usually do for example.
[1]: https://rust-lang.github.io/async-book/01_getting_started/03...
[2]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
Go will not inline a function that exceeds complexity metrics, and one of those metrics is whether the function contains a range statement. You will get a real heap-allocated closure invocation on each loop.
Go's function calls are not that cheap, and these will be measurably slower than the for-loop equivalent. I see functional idioms as being a huge risk factor for Go.
True, but:
> and one of those metrics is whether the function contains a range statement. You will get a real heap-allocated closure invocation on each loop.
I think this is no longer true in Go 1.18, see https://github.com/golang/go/issues/14768#issuecomment-98175... .
it := GetAnIterator()
for it.Next() {
it.Value()
}
It contains generic implementations of
Any
All
Count
Filter
ForEach
Map
Reduce
ReduceRight
Sum
Chunkhttps://github.com/johan-bolmsjo/gods/tree/v2
It's not much, only a simple list and an AVL tree that I've carried with me for many years. Binary search trees are useful to solve some problems because they are ordered.
I was impressed with how polished the tools where. The go-lsp plugin just worked with the new generic types. I solved all compiler errors in the editor without actually compiling anything. Did not expect that level of polish. The modules introduction broke the editor integration for many release cycles. This seems much smoother.
I assume the limitation on generic methods is just an implementation issue, the vtable for an interface has to have a known number of methods, and they can't insert specializations for each type on the fly into the vtable.
I'd call it a fundamental semantics issue. That discussion is mostly framed in terms of the compile-time problems that arise, but that brief aside about the reflection package is deceptively short, because it means that the entire compile time set of problems is also lifted into runtime, where the problem is even worse. (To translate for someone who isn't a Go expert, the reflection package has enough power that it can dynamically instantiate new types that the compiler did not and can not even in principle have seen at compile time, which may then require new methods to be created. The current Go runtime has no run-time code creation capabilities. [1]) Without the reflection problem, it's annoying but at least solvable in principle, but I don't see any reasonable design (without a fundamental overhaul to specify a runtime-available compiler in the language spec itself, which is a huge step) that solves the reflection issue.
[1] Which personally I consider almost a feature. YMMV. Runtime compilation is very cool in a lot of ways but is a hell of a subsystem to try to secure, just by its very nature. There's a lot to be said for simply not having that capability unless it is integral to some particular program.
Saying "no generic methods" is a nice out.
It's what Swift does if inlining is defeated, the type arguments are passed at runtime as a classical argument on stack.
You don't need full reflection as you suggest, only to be able to materialize the types as structs + an interface at runtime. Yes, it's a little slower but most of the overhead comes from the function not be inlined.
Hopefully Go generics continue to improve.
Another option to circumvent this is to use top-level functions, and accept the receiver as the first argument of the function. This is effectively the same thing, the only thing it's missing is the object.method() syntax that's familiar to OO programmers.
There is hope that they will be added in the future, but solutions to the problems will need to be found first.
That's the problem they should fix.
That's the core problem. Anyone who truly understands generics has shown no willingness to work on Go. The only reason it has progressed as far as it has is because a third-party consultant was temporarily hired to develop what was dubbed Featherweight Go to act as a proof of concept.
It turns out that the intersection of people who care about generics in Go and those those who aren't armchair coders unable to comprehend the domain themselves is limited to very few people. As you know well from your armchair.
There was clearly always interest from the project itself. There may have been some random naysayers on the internet who act like they are opposed, but you'll find people acting every way imaginable on the internet. I really doubt anyone actually refrained from working on the project because an internet troll wrote something silly.
was there any major breakthroughs?
If structs support generic methods but interfaces don't then nothing happens, everything just works as you'd expect (whithin those limitations). A method which isn't part of the interface is not under consideration, so the generic methods have no bearing on the interfaces the object could or would implement.
The reverse would be an issue, but this is not.
Actually it does:
"Or, we could decide that parameterized methods do not, in fact, implement interfaces, but then it's much less clear why we need methods at all. If we disregard interfaces, any parameterized method can be implemented as a parameterized function."
https://go.googlesource.com/proposal/+/refs/heads/master/des...
"And even that traversal is not sufficient in the general case when type reflection gets involved, as reflection might look up methods based on strings input by the user. So in general instantiating parameterized methods in the linker might require instantiating every parameterized method for every possible type argument, which seems untenable."
So methods with type parameters would a) not be usable with interfaces and b) not be usable with reflection.
addCallback(new Callable<Integer>() {
public Integer call() {
return callback();
}
});
addCallback(this::callback);
I claim that there are no original ideas in Go's type system and in the way that Go does generics, and the reason it took so long was because it came out of an environment with actors (including a part of the community) pushing against the feature.
You might disagree, I'm curious what original features of Go you will point out to me.
It seems to me that that it was less about active pushback and more that people who cared about having generics in Go was limited to an exceedingly small group of people who weren't necessarily experts in the subject matter. In fact, the Go team hired an external expert to lay the groundwork for what became the winning proposal.
Also that GP was probably sarcastic when they nominated typed nils.
void *outer = 0; // outer can be nil
int val = 5;
int *inner = &val; // inner can also be nil
outer = (void*)(&inner) // can have an inner value, inner nil or outer niltyped nils are a single value which is both nil and non-nil at the same time.
It's no different than `var x **int` (a variable called "x" whose type is a pointer to a pointer to an int). If you want to safely access the underlying integer data, it's not sufficient to check that the outermost reference is not nil--you must also verify that the inner reference is not nil. We run into this in every language with nullable reference types, including C, Java, Python, JavaScript, etc.
EDIT: `*int` -> `**int`. TIL HN converts two asterisks into one.
None of that is correct. The value-part of the fat pointer is the exact same in both situations: nil. It is a single value which is both nil and non-nil.
> It's no different than `var x **int`
I assume you meant that rather than what you actually wrote.
It is in fact quite different.
> If you want to safely access the underlying integer data, it's not sufficient to check that the outermost reference is not nil--you must also verify that the inner reference is not nil.
Something a typed nil does not let you do, unless you know (or reflect) the underlying concrete type in order to downcast the interface back to a regular pointer.
> We run into this in every language with nullable reference types, including C, Java, Python, JavaScript, etc.
We really don't. In "every language" you have to actually create a double indirection for something within the same heavenly realm (but is not at all the same thing) to occur.
Yes, today I learned HN replaces double-asterisks with single asterisks. Edited my original post.
> None of that is correct. The value-part of the fat pointer is the exact same in both situations: nil. It is a single value which is both nil and non-nil.
I repeat, an interface isn't "both nil and non-nil", but rather the nullity of the interface and the nullity of the underlying data are distinct propositions. For example, there is nothing analogous to "both nil and non-nil" when the runtime type is a value type, such as an integer.
An interface behaves just like a reference. When the underlying data is also a reference, then it behaves like a reference to a reference. This means you can have a nil interface or a non-nil interface which references a nil pointer.
Under the covers, Go implements this behavior by representing interfaces as a tuple where the first field contains a pointer to the runtime type information and the second field contains a pointer to the data. Go encodes a nil interface as (nil, nil) while it encodes an interface backed by a nil int pointer as `(<pointer to `int` type>, nil)`.
> We really don't. In "every language" you have to actually create a double indirection for something within the same heavenly realm (but is not at all the same thing) to occur.
Go interfaces to pointer data are functionally "double indirection". The outer pointer can be either nil or a pointer to the data which itself can be nil or valid.
> Something a typed nil does not let you do, unless you know (or reflect) the underlying concrete type in order to downcast the interface back to a regular pointer.
Well, this would be unsafe as the runtime type of the underlying data isn't guaranteed to be a pointer. So by default Go requires runtime type checks in this case, but you can always opt out of these runtime type checks via `unsafe` but this is usually inadvisable.
EDIT: would really like to know why I'm being downvoted. If you disagree with my comment, then best of luck to you in rebutting it. :)
var i *int = nil
var x interface{} = i
if x != nil {
fmt.Printf("%v was not nil", x)
}
<nil> was not nil
class Ptr:
def __init__(self, x):
self.x = x
x = Ptr(None)
if x != None:
# Go's `%v` print directive unpacks interfaces.
# We'll use `x.x` to emulate the same here
print(f"{x.x} was not None") var i *int = nil
var x interface{} = i
if x != nil {
Thing1 i = null;
Thing2 x = i;
if (x != null)
{
System.Console.WriteLine("x is not null");
}
class Thing1
{
public static implicit operator Thing2(Thing1 c) => new();
}
class Thing2 {}First of all, I’m not asserting that there isn’t a problem with Go. I’m asserting that its interfaces aren’t some three-state beast per the OP.
Secondly, you don’t see the problem with the Python code because you’re not comparing it like for like with Go code. The equivalent Go code would look like this: `var x interface{} = (*int)(nil)` which is at least as conspicuous as the Python example.
> I assigned `nil` to `x` and yet `x` is not nil.
No, you put a nil pointer into an interface, thus the interface is not nil. We have to understand the semantics of the language we’re programming in—it’s not fair to criticize Go because it’s interfaces aren’t my exactly Python variables. I think there’s probably some valid criticism against Go with respect to the error-prone nature of its interfaces, but I don’t think it’s that they lack Python variable semantics.
> Implicit conversions aren't unusual in programming but in Go they stand out a bit.
This isn’t an implicit conversion, we’re explicitly putting an int pointer value (nil in this case) into an explicitly-typed interface{} variable.
error.Error((*CustomErrorType)(nil))
This definitely causes some people confusion and results in bugs, but the question under discussion is whether the original claim ("an interface can be simultaneously nil and not nil") is true. Of course, it's not true as this example illustrates--the interface is not nil but the underlying data (of type `*CustomErrorType`) is nil.
> A great innovation of Go is that it includes a built-in reference type with both inner nils and outer nils.
Which the OP later clarified by saying an interface can be both nil and not nil at the same time. So it’s pretty clear that we’re debating about whether interfaces have three states (nil, valid, and both) i.e., multiple kinds on nullity (inner vs outer).
I don't think I made any such clarification...
> typed nils are a single value which is both nil and non-nil at the same time.
Let's say I have the following code:
func open() (*os.File, *os.PathError) { return os.NewFile(1, "stdout"), nil }
// in main
// (some lines omitted)
fi, err := open()
if err == nil { fmt.Println("no error")
Normally, you would say "no, it does not, 'err' is a nil '*os.PathError' so it does not.
However, I didn't mention what lines were omitted, and those lines actually matter!
// main
_, err := os.Executable()
fi, err := open()
if err == nil { fmt.Println("no error")
Why does this matter? Well, it's extremely common in go (idiomatic in fact) to reuse the "err" variable name, and it's also extremely common to use it in multiple assignment with ":="... where despite using the declaration syntax, you can re-assign to existing variables without their types changing.
I think that's what the parent was pointing out: a combination of type inference and automatically "converting" a struct into an interface for you (both in assignment, as I showed above, and when you return a value, as shown in the parents example) means that you can end up with this issue without doing anything as obviously wrong.
In other languages, returning concrete error types means the caller has more type-system information and is good. In go, returning concrete error types gives the caller more information, but opens them up to accidentally implicitly casting it to an interface and blowing their foot off.
As such, we have almost completely untyped error returns, and any time a gopher thinks "this is dumb, why don't I use the type system for this?" they're actually just loading a shotgun and pointing it at their foot.
> I don't think this is a different (and thus not "worse") case
All the above was just to say, I think it's the same case, but it's pointing out that it's "worse" because it interacts poorly with other parts of the go language spec.
Playground: https://go.dev/play/p/4MYEZr8D5Qv
Any preferred solution (which will never happen due to compatibility guarantees) would be to eliminate zero types altogether and use Rust-like enums instead.
Go's type inference can take:
x, err := 1, someNilStruct
On the other hand, in rust if I type "let err = rhs", that will _always_ introduce a new variable named err that is the same type no matter what lines of code are above me.
The interaction there in go is due to poor interactions between features other than zero types I think, so I don't see dropping all zero types as solving it.
I also don't think zero types are analogous to rust enums, but rather to rust's default trait (which I do vastly prefer over zero types).
You’re mistaken here. `err` will always have exactly the same type as `someNilStruct`. Specifically you’re confusing the variable declaration syntax with the variable assignment syntax. `err := foo` is the same as Rust’s `let mut err = foo` in that `err` will have exactly the same type as `foo` no matter the types of other variables. `err = foo` behaves exactly the same in Go as in Rust in that you’re just reassigning the value of `err` but it’s type doesn’t change.
> Unlike regular variable declarations, a short variable declaration may redeclare variables provided they were originally declared earlier in the same block (or the parameter lists if the block is the function body) with the same type, and at least one of the non-blank variables is new.
This suggests that a redeclaration as part of a short assignment will not change the type of the previously-defined err. is the spec incorrect?
Edit: Here's a demonstration of this sort of thing https://go.dev/play/p/Po_Uy85ORN_L. You can uncomment line 22 to change the meaning of the assignment on line 23 and cause the program to segfault.
Click through to the playground link and add some 'fmt.Printf("%T\n", err)' lines here and there to see what I mean.
It's a defect in the language, plain and simple: if a pointer-to-type implements interface, then converting a nil pointer to type to the interface should result in a nil interface value (that would compare equal to nil) instead of a non-nil interface value that has nil value pointer inside it — which is a second kind of nil in anything but name. You compare it to nil, it says "not nil", then you try to use it and it blows up with panic "I am actually a nil, gotcha sucker".
What does this even mean? If you did it by accident it sounds like a bug in your program, so fix the bug so a nil can’t be passed into the interface. If it’s not a bug but nil is a valid value, then type-assert the interface back to the concrete pointer type and check it against nil.
> It's a defect in the language, plain and simple: if a pointer-to-type implements interface, then converting a nil pointer to type to the interface should result in a nil interface value (that would compare equal to nil) instead of a non-nil interface value that has nil value pointer inside it — which is a second kind of nil in anything but name.
No, this would be a bug because a nil pointer and a nil interface mean different things.
> You compare it to nil, it says "not nil", then you try to use it and it blows up with panic "I am actually a nil, gotcha sucker".
You have the same behavior in any language with nullable references including Java, Python, C, etc. You’re complaining ultimately because you don’t understand references. In this case you have a reference type (the interface) whose value is itself a reference type and you’re upset because checking the outer reference for nil doesn’t also evaluate the inner reference; however, no mainstream language works this way and this would be very surprising behavior to people who actually understand references.
And your explanation does not really hold in actuality:
type DoesAThing interface {
Thing()
}
type ThingDoer struct {
}
func (td ThingDoer) Thing() {
}
func WillPanic() {
var thingDoer *ThingDoer
var someThing DoesAThing = thingDoer
someThing.Thing()
}
This means when you assert that
> the nil is an invalid implementation, not because it is an invalid Go value, but because it is an invalid implementation of the interface.
The design of Go forces that lie on you, you have no way to opt out of it. If you implement interface methods on a value receiver, that lie will be forced onto you with no recourse. If you implement interface methods on a pointer receiver, then that lie will still be forced onto you but give you the option of... defensively ignoring and maintaining garbage state (à la ObjC I guess).
They have different types. This is not splitting hairs.
"The design of Go forces that lie on you, you have no way to opt out of it."
No, it doesn't. I've been programming in Go for about 8 years now, and I do not face this problem, because I do not lie in my code this way. My own experience is constructive proof of my point. That it, it falsifies your point, if this was "forced" on me I'd be encountering this problem. It is not a problem you have to have in Go. I can and have rewritten any code that has this problem when people post their code where they are "forced" to have this problem, and this is the cause every time. It is a result of bad programming decisions. Do not create objects that can not fulfill their contract, and you won't have problems with them propagating through your code.
(I mean, I'm really shocked at how much this angers people. Who is really advocating for "Create whatever garbage you want where ever and get angry at the downstream code when it can't pick up the pieces?" I mean, sure, it's a common style de facto, but who is advocating for this?)
Go is far from the worst case of this, the dynamic typing world affords code at scale that does this so much that it simply becomes a fact of life. It's the biggest reason I've come to loathe writing any code at scale in dynamic languages.
The official golang tutorial explicitly does not say "never use struct receivers at all" even though using them is "lying."
> the reason it took so long was because it came out of an environment with actors (including a part of the community) pushing against the feature.
The reason it took so long is because it simply wasn't the biggest fish to fry. Generics are a massive feature, and there were a lot of other opportunities to deliver more value for considerably less work. Of course, there was a healthy amount of pushback against rushing generics, considering generics must interact with every other feature in the type system even if those features aren't novel.
Frankly, people make too big a deal about generics. It will be a neat quality of life improvement for certain kinds of code, and a lot of additional code will enter the ecosystem which is gratuitously abstract. Builds will probably get a fair bit slower on average while still being fast relative to most other languages. Frankly, no type system feature will ever impact a language as significantly as things like tooling quality, ecosystem, runtime performance, static+native compilation, iteration loop time, learning curve, etc.
The third phrase of the linked essay is:
> Go type system is not a traditional type system
"non-traditional" is generally read as "novel" in some way.
While it's not clear what one might mean by a "traditional" type system, I'm claiming that the solutions to the problem of implementing generics in Go were well-known (not novel), and therefore the difficulty of coming up with them doesn't justify the long time it took to develop the feature.
Different strokes for different folks, but I think generics are a huge deal. They make libraries so much more ergonomic and reduce toil generally. It's a case of dealing with many, many small papercuts over a long period of time rather than a big-bang sort of thing.
> tooling quality
I'm a huge proponent of tools, so much so that I worked on programming language tools for 5 years. My perspective is that tools are there to optimize an existing experience, but reach a local maxima in the overall experience because they're limited by language semantics. When you improve language semantics, tools get to improve experience again. That's why it's so important to do language design in a way that strongly considers the tooling experience, and a big reason why you see so much love for languages like TypeScript and C#. They work hand in hand. And generics are an example of how semantics can push tools to be better.
And then there is the toil reduced by having good performance. I have about 15 years of experience with Python, and almost invariably there's some hot path in the code which needs something between Python's level of performance and Go's level of performance. Unlike Go, Python affords relatively little headroom for optimization--you can't easily parallelize anything or pull some allocation out of a tight loop for a 10x performance gain. Usually this looks like rearchitecting your codebase so that "rewrite the hotpath in C/multiprocessing/etc" is attainable and on top of that you're probably implying significant changes to your build system and/or runtime environment as well as a load of maintainability problems associated with anything that touches the language/process boundary.
By comparison, I think the gains from generics are relatively small.
Ah, that's a different kind of tooling. This matters, but there are loads of developers out there who care just as much (if not more) about IDE tooling and other things that assist in the writing and refactoring of source code itself. Features like generics open up new ways to make this tooling better.
It's also worth pointing out that you can have your cake and eat it too. .NET and Rust are fine examples of having great build/package management tooling, runtime performance, compiler performance (.NET far more than Rust), artifact management, and so on. And great IDE tooling in different environments.
I think there's a lot of confusion brought in from languages where generics are a bigger deal than they are in Go, because when generics came in to those languages it was like turning on a light.
However, if you map out the problem space of "what did generics solve in those languages", rather than obsessing about the solution, what you find is that Go's interfaces, combined with the structural typing ("static duck typing" basically, rather than having to explicitly declare conformance at declaration time) covered quite a bit of the problem space. And that's what matters, whether the problem space is covered, not whether the problem space was covered in exactly the same way as some other language.
People who either have never programmed in Go, or only spent long enough with it to try to program "(Not Go) in Go" (e.g., "Javascript in Go"), and didn't learn how to use the tools Go has to cover the problem space, end up vastly overestimating their utility in Go, and vastly overestimating the sort of problems you can't use Go to solve. They're like "How can you program in a language that doesn't cover the 'generics' problem space?", and the answer is, "You're actually right, I couldn't! But I can use Go, because it actually does cover quite a bit of the problem space, just in a different way than what you're demanding." Interfaces DO NOT cover all the space, I would never dream of claiming otherwise, but they cover a lot of the space. There's even some problems where I've found Go simply outright the best solution because of some particular thing I can use interfaces to solve.
I'll actually kind of miss the "How can I do this without generics in Go?" questions on /r/golang. They're fun, and with the exception of "data structures", generally I didn't just come up with "a" solution, but with a good solution.
I think only insiders on the Go team during that time period can really say for sure what the deal is.
But I will say that retrofitting generics onto a language and standard library is a project of enormous scope and incredible difficulty. So I'm pretty sympathetic to the idea that it took them a while.
If so, it's curious how sparsely mentioned the origin at Bell Labs pre Google is.
I swear there's at least one top-level comment such as yours in every thread, for no reason other than just to discredit people's work. I find it disgraceful.
For example, Go's interface types and its lack of constructors seem different from most languages you might borrow from.
How so, I don't see anything special about Go's type system in the linked article.
var _ y = x{} // syntax might be slightly wrong
After writing this, I found the linked article in the OP link to what the author meant.
https://rakyll.org/typesystem/
There's really a bunch of small things that are different about golang that individually you can find elsewhere but are on the whole unique.
Creates a compile time error if Struct is not a ThingerInterface.
i've enjoyed go for its simplicity. At times it requires more code (or code generation) but that generated code was easier to debug and have tooling operate on.
With generics, Some things will suffer. Complexity of code and tooling increases, tooling is probably going to suffer and become more costly. compile time and performance will probably also be affected.
Generics are likely only great for collections of things, ie []*T, where, you need to do the same thing on many of T.
interface Query {
All ...
Insert ...
Update ...
Delete ...
WhateverElse ...
}
Either the framework has to know how to create a querier for each specific type, drop to using empty interfaces, or use generics.
I really wish people would just use the language as it is. Yes, it does cause for some repetition, but thats not the worst thing in the world. Compared to some Java codebases I have seen, Go is extremely simple already, in terms of "flatness" in the filesystem, and flatness in the type system. I think people should just know when to leave well enough alone. I dont think the complexity of generics is worth it for 99/100 use cases.
Of course they could just use interface{}... But by doing that they lose both type safety and performance.
I guess you think generics is a magic pixie dust that has the exact performance envelope as non-generic code?
And what are we optimizing for anyway? Reading a bunch of code that uses an off-the-shelf library's Stack or Tree or whatever is always going to be easier to read than having to also have to inspect the three hand-rolled Tree implementations, two of which are probably buggy, in addition to the actual client code trying to use the tree to do real work. Or a library that implemented it using interface{} which is a big middle finger to the whole type system, completely discarding the compiler's ability to detect otherwise trivial mistakes.