Functional Programming in Go with Generics
ani.dev
ani.dev
I maintain a few large Go codebases, tens of thousands of LoC each, and not having generics in practice is way, way down on my list of challenges. Equivalent Java codebases that I've maintained are usually harder to maintain because of generics, with over-eager functional programmers itching to show off their skills in abstraction, instead of solving business problems.
I hope I'm wrong, and that this will make Go a better language. But I doubt it.
- We have some utilities (e.g. caching layer) which now either needs to use interface{} (and lose all type safety) or hard-code the type their dealing with (and thus be duplicated).
- Lack of iterators are annoying. We often need to stream data from a database and now we need to introduce a separate iterator-interface for each thing that can be streamed (`Next() (Entry, error)`). Working with these iterators are clunky since we can't write common tooling around it (e.g. "fetch the first entry").
I agree that lack of iterators is one area where there is permanent annoyance. Personally, I think that annoyance is a lot less than the other problems that come with generics (at least in other languages).
I'm not saying there aren't legitimate scenarios where generics simplify things. I'm completely convinced that there are. But holistically I don't think they outweigh all the other baggage that they come with.
At that point aren't you just writing out the generics by hand? Why would you not want a built in language feature for that which saves you the trouble while simultaneously ensuring the code is correct.
What I’m arguing is that for most cases, the implementation and concretization are so easy to write by hand, are safe enough, and perform well enough, that I don’t think the downsides of having the compiler emit the concretizations are worth it. The downsides being the new complex language primitives that have lots of consequences that trickle out to the compiler, runtime, and ecosystem.
Unless I'm mistaken, the "generic type specifications" are just interfaces. Instead of specifying the concrete type of a function parameter, you specify which interface it needs to implement, and then everything else is the same (except you may need to define getters/setters if you need access to struct fields). There's no need to have a new language primitive. That's certainly how Rust does it anyway. And given that Go's interfaces are structurally typed, it'll be even simpler in Go.
I don't really understand the argument that generics are complex. As far as I can see, they're super simple.
In a weird way, with this technique, did Go have "manual generics" all along?
In the exact same way every langage without generics had generics all along: not in any meaningful way.
You could also use codegen hook to instantiate your pseudo-generics after all: https://www.reddit.com/r/rust/comments/5penft/comment/dcsgk7...
To-be-named-at-runtime is simply dynamic typing. It's polymorphism without static typing.
Generics based on type erasure, e.g. Java, wrap that with a type system hoping* to prove safety at compile time, yet compile to a single dynamicly typed implementation, that does not use this type info at runtime.
Generics based on code generation / specialization, e.g. C++, generate distinct run-time implementations with the type hard-coded. (Though you can plug in dynamic typing by making the hard-coded type a pointer.) In C++ this is more than optimization, it's semantics — you can specify distinct behavior for specific types.
JIT compilers can start from a single dynamic-dispatch implementation and generate specialized implemetations for some types, bridging the divide.
But in any case, the common meaning of "generics" is not merely polymorphism, it's polymorphism plus a type system where you _name_ the type you're gonna use.
I’m not sure exactly which qualities/features of Go make it so much easier to read, but I worry that change will lead to the Go losing what makes it unique and special.
Side note: clojure was oddly the second easiest for me to read.
On a deliberate hyperbole, reading assembly, each line is trivial, but it is very very hard to make sense of what does it do compared to a line of a high level language.
In my experience, golang is harder to read. It's overly verbose that it's hard to figure out the underlying logic. What can be written as
items.filter(|n| n.price >= 10)
.groupBy(|n| n.source)
.mapValues(|v| v.sum()/v.length())
The problem is that even if golang added generics, they won't be composable. In their proposal, they proposed adding map/filter/etc functions in a separate package, not as functions on slices as a proper language would do. The same mistake they did with not having any functions on strings, but kept them in a separate package.
Also, they need to improve their anonymous function syntax.
foo("param1", func(a int, b string) bool { return len(b) < a }
foo("param1", |a, b| b.length() < a)
Pretty much every argument about how to write Go programs is hand-wavy.
I'd actually argue (from years of experience at least) that FP discussions of code are pretty much the only ones of actual substance instead of mostly aesthetics and opinion and heuristic.
The best part of writing Haskell professionally is I no longer have to deal with hand-wavy programming folk-isms. We can do better, and it's not that hard.
What's important to me is that the decision is already made, and when me and my team are writing code, we don't have to worry about making those decisions over and over again. They're already made for us. IME the costs/benefits of the different styles are not even worth the discussion, just pick something and move on. This is the aesthetic that I like about Go.
One point of generics is that you have to write your pure functions less often..
Jokes aside, I wish our tools were more "final". When is a programming language done regarding the pain it was meant to solve?
Elixir is often presented as "done" (https://elixirforum.com/t/is-elixir-done/20830/12). For example, the depreciation policy mention that function can only be removed on a major version (1.X to 2.Y for example), and for now it's still in 1.X and no Elixir 2 is planned.
It still has releases (https://elixir-lang.org/blog/2021/05/19/elixir-v1-12-0-relea...) to follow Erlang releases, but Erlang itself is also more conservative than some other languages.
It's hard to imagine that a language can be "done" when the problems being solved with it keep changing and we keep discovering new techniques for solving them.
When is automotive design done? Farming?
Features that you don't use carry a cognitive load, increase the number of choices you have to make, and reduce the usability of other features that they have to be disambiguated with.
10kloc is pretty small these days realistically. Especially in Go land so I get it. But yeah from my experience that simplicity at some point becomes more a burden and less a help.
What I do want is to stop implementing sort.Interface over and over again.
What command-line parsing libraries have “a bunch of unnecessary type parameters”?
Only library I can think of which even has them would be structopt, and that’s not unnecessary, the entire point of the library is to deserialise to a structured type.
And of course rust was designed from the ground up to leverage generics, e.g. by design it does not have reflection / RTTI, interactions with variable or foreign types are thus generics based (often, sometimes trait objects will do though they have their own tradeoffs), or it doesn’t have overloading so while not handing all overloading use cases by a long shot it uses conversion traits (aka generics) for “convenience” overloads e.g. foo(string) and foo(path) become foo<T>(_: T) where T: AsRef<Path>.
> ...by design it does not have reflection / RTTI,..
I don't think of those as comparable features.
Reflection is more or less an alternative way to accomplish many of the same things that you get with macros in Rust. Where you'd see reflection in C# or Go is for something like serializing an object to JSON, and in Rust you'd get some library with a macro to do it for you. The Rust version could still be completely monomorphic, most of the time.
RTTI is just std::any. Rust has it. However, 99% of the time, when I'm using dynamic_cast<T>(x) in C++ or switch x := x.(type) in Go, I would have been using an enum in Rust.
The convenience overloads for Rust are nice, that's not the overuse I'm talking about. I'm talking about the fact that some library authors seem to avoid &dyn like it were poisonous, even if it makes total sense and would make large chunks of your library monomorphic. There are also minor code size issues that come about because some library authors forget that the convenience overload should probably just call an underlying monomorphic function, especially for cases like AsRef arguments.
I am under the assumption here that if you really like generics, you probably like Rust, and vice versa. This is kind of a taste thing, so what is overuse of generics for me (hello, C++'s std::chrono, std::mt19937) might be the bee's knees for you.
I did a fair bit of Haskell and C++ before Rust, and as I got more experience with those languages, I used generics less. Monomorphic types are always easier to understand, and they should be preferred.
You made that claim several time, but still provided literally no evidence, just a lot of handwaving and careful avoidance of any sort of concrete point.
> I don't think of those as comparable features.
> Reflection is more or less an alternative way to accomplish many of the same things that you get with macros in Rust.
They go together e.g. where Go uses reflection, Rust uses Derive or manually implemented traits and generic functions working with those traits. JSON is a pretty obvious example there.
> I'm talking about the fact that some library authors seem to avoid &dyn like it were poisonous, even if it makes total sense and would make large chunks of your library monomorphic.
I would not necessarily disagree with that although I would word it more kindly: since generics always work, they're the default, as a result dynamic dispatch is a fallback when either you must have dynamic dispatch or someone pointed out that the static dispatch was less than optimal (which can be impossible to find out through microbenchmarks).
Plus `&dyn` means you're now restricted to borrow-ability, and `Box<dyn>` incurs allocation costs. Generic function skip both concerns.
I'm not really interested in arguing that point, just because you asked me to give concrete details doesn't obligate me to provide them or engage in the conversation to a level of detail that would satisfy your request. It's okay to share an opinion without defending it.
Part of language ergonomics is that languages provide you with with "escape hatches" to avoid making certain decisions in the design of your system. This might mean that the decisions are made by the callers. Good API design requires figuring out which questions should be answered by your API and which questions should be answered by the caller.
Just as an observation... generics are an escape hatch that let you avoid choosing concrete types, and forces your caller to do that for you, and in general, I think library authors use that escape hatch too often. This is not specific to Rust.
Every language feature can be abused, that's why Go tries to have a small amount
func getTopUsers(posts []Post) []UserLevelPoints {
return posts.GroupBy(func (v Post) string { return v.Level })
.Values()
.Map(getTopUser)
}
x := foo()
y := bar(x)
z := baz(y)
Maybe Haskell and F# can elide this kind of thing, but it's not something the Go compiler is going to do.
The final generics design is probably not expressive enough to do it ergonomically in user code neither (building something where the intermediates are a library type with a last `.something()` call to collect a final []T).
No secret sauce, just having the pleasure of chosing multiple backends, including an interpreter.
Bytecode runtime during workflow, optimized native AOT for release and final production tests.
A much better design is to do it lazily which is possible if you can store closures as fields of a data structure.
Not sure what fast compilation and execution or dependencies has to do with this. Implementing iterators that way is the worst possible solution that results in slow execution that can blow up the stack and crash a program. It would be better not to have iterators at all than to require them to be eagerly evaluated.
You can support functional features without being a Haskell...
Sending data through channels just to implement an iterator seems insane. It's not strictly lazy either...
Why do go developers reject simple patterns that make faster code?
Look, I’ve written lots of scala and erlang so I get your argument. It just does not exist in go. And that’s okay.
Never got caught thinking “gee, need a lazy collection”. Maybe a stream when reading stuff from the ether. In go, channel was enough. 5 minute job. Nothing insane.
for over a channel is in essence lazy. Channel stores the data as a linked list. So what’s the benefit.
Lazy evaluation is the only reasonable way to implement iterators so you don't have to pay the costs of intermediate data structures or sending data over channels. Insertion into a data structure isn't free. Function composition is.
For loops may be unfashionable but they aren’t hard to read.
I do find deeply nested for loops with outer variables that might start uninitialized or need to be mutable much more difficult to read.
I'm often think FP-first (data science) -- here though, I was surprised by how I went back over the imperative alternative.
Avoiding `append()` isnt worth it in a language where this level of annotation is required to do FP. It's less clear.
Using functional programming in Go... just doesn't look right. Generics are useful, but won't change the whole Go ecosystem into FP. Maybe some uses of Map might make sense, but... idiomatic code will look the same.
so Go will change and i think that sucks hardcore. there are already enough languages where folks can go crazy with types, fp, monads etc. My selling point for Go was that it was mega restricted :-)
It's going to be so verbose and unfun in Go that only very stubborn people are going to stick to it anyhow. I'm not saying that because I hate the style, I'm saying that because it's still going to be a lot of boilerplate wrapped around not much payload. It really won't be fun. I like "map (f1 . f2 . f3)"; the equivalent in Go even post-generics will be crazy full of boilerplate and garbage.
It will change some aspects of idiomatic Go. I don't care much at all about the "functional" aspects but I am greatly looking forward to a proliferation of more specialized data structures that are type-safe. I think one of the major weaknesses of Go in what is arguably its wheelhouse, network servers, is that network servers are often precisely where I want some obscure data structure with the performance characteristics I need to scale up some service properly. It's true that the dynamic scripting languages showed us that you can get a long way with arrays & dictionaries, but I'd like some more stuff. I've got a few places a "map" would be slightly convenient, but I've got several places where I've got architectural problems because I had to hack in an immutable data type or tree or something else that I'd much rather be using a bullet-proof, community-tested implementation of a type-safe data structure than some subset I smashed together and lightly covered with unit tests just to do this one particular job. This is the change I'm actually looking forward to. I've also got some channel pipelines that, while they are set up and working, would be easier to understand with some helper functions that could deal with them.
I am also looking forward to either using or writing a very simple concurrent pipeline that specifies a set of transform functions and the number of workers to start up for each and takes care of actually bringing up the network correctly. Or implement structured concurrency: https://vorpus.org/blog/notes-on-structured-concurrency-or-g... Go may never enforce this but I'm tempted to try to enforce it in my programs.
Notably it has: - mature generics - streams - exceptions - rock solid for server apps - some FP style libraries - good timing - probably the widest universe of libraries
Only downsides are, 1) JVM startup times and 2) it's not the newest shiny thing.
I looked at the whole language comparison recently for a greenfields project. While initially optimistic, I was surprised how many capabilities some new alternatives are missing, and by the enthusiasm to write thousands of lines of low value C-style error checking.
An architectural assessment: Above a trivial level of software, the majority of code composes something that can fail. Exceptions are an ideal mechanism for this general truth.
(To be clear, I am aware that Go and Java still have differences beyond that; my claim is that with this change Java would have been "good enough" that Go either never would have been spec'd or it never would have gotten any traction if it was. Java, however, has left a significant window open for Go because it has gained traction.)
The problem with the way Java does it is that it forces a temporal dependency between interface and implementation, all to avoid a programmer error which on average will happen less than once per programmer career: http://www.jerf.org/iri/post/2954 This causes a lot of gyrations to be necessary to use them properly, which starts introducing the desire for heavyweight ways of autogenerating this, etc. and roll around that cycle a few times and that's how you end up with as much XML as code in your codebase.
I think Java and Go are a really interesting case study for people interested in programming languages, because on paper they are virtually identical languages (especially if you ignore channels and select in Go), and it gives a clean demonstration of how even those not-very-large differences end up having a profound effect on how solving problems in those languages are best done.
but i hope i am wrong. :-)
in general i get your point amd agree with it. i have written large go codebases and ran into exactly the same issues. but i wouldn't agree to the trade off :-) then i would have used Rust in the first place I guess
They won't need to start; hit the Go issue tracker to see a whole bunch of them have already been made and rejected. People have been advocating for them for a while. I virtually guarantee that of all languages, the Go team will not change their mind in response to existence of generics for them. Their bar for accepting changes is very, very high. It is not insurmountable, but it is very high.
Remember, Go didn't come out yesterday. Go is now over ten years old. We have a good idea of the velocity of changes it makes. It is low.
I'm curious, because I've used k8s and docker extensively for devops, and aside from templating, which is arguably not so much golang as, well, just a templating engine. And I've thankfully not have had to use golang for anything serious.
As always, using the platform language is the best road to avoid additional development costs dealing with interoperability and tier 2 compatibility issues.
Although some stuff is now happening in Rust as well,
In addition, even if writing infrastructure plugins were to considered necessary as part of DevOps (though I consider this a stretch), writing golang is again not a requirement. The CNFC isn't exclusively filled with golang plugins either. I'd argue most are something else C/C++/Java/web-langs though I haven't made the effort to check. I'd venture a guess at one in four being golang, at most.
I agree however that language evangelism benefits no one, and especially agree on the sentiment of using the platform language to avoid additional development cost. That said, and this is probably a bit more subjective, this does not follow from/to the original statement I took issue with.
but this development kibda makes me sad :-) i use go for so many projects because it finally restrained me and my employees enough to not argue about language features all the same. and all our code bases look the same. i loved that about go. no, with generics, and all potential libs (just wait for go lowdash, go rambda, go fantasyland, go promise etc), that will come to an end :-)
What was pretty straight-forward code, just a bit verbose, becomes a tangle of library functions being composed. It's not shorter, it's not easier to understand, adds dependencies, and most likely will not perform any better. There is nothing gained.
That is wrong.
Interface is a special type in Go. All interfaces in go, including ones you declare, hold 2 pieces of data in memory
1. Pointer to the concrete type
2. Metadata about the concrete type
interface{} is not a special thing, it is just the empty interface, and everything satisfies the empty interface.
This is quite neat. All a type assertion does is follow the pointer and check the type data.
There is also a type switch in Go. https://play.golang.org/p/22aegLocHXI
This makes building handlers around the dynamic type easy. But for the most part: Single function interfaces. Compose those to make make larger interfaces if absolutely needed (see io.Writer, io.Reader and io.ReadWriter)
type meh interface{ foo() error }
type whatever struct {}
func (h *whatever) foo() error
And still pass []whatever{} to func canihaz([]meh) so not sure how much generics really add to that.
I don’t mind generics but when they are there, I use them. Whatever :)
Accepting interface{} is a code smell. I’d accept that only when dealing with wire data. Never in the business logic.
Variadic functions even links to the go example, but classifies them as unsupported in go.
I would not normally say any language "support currying" just by having first-class functions. Python can do several of those things I said Go can't do and I still wouldn't call that "supporting currying". It sure isn't anything like the experience of using curry & uncurry in Haskell: https://hackage.haskell.org/package/base-4.15.0.0/docs/Prelu...
With most Go code today, you can just look at the indentation, or count the for loops.
https://github.com/golang/go/issues/45955#issuecomment-83235...
The fact that functional programming emphasizes "abstraction" so much means that, by definition, you're further away from what's actually happening. So looking at some declarative code it's not immediately obvious what the computer will actually do. The kinds of programs I write require me to care about that deeply. So when I see declarative code all I see is a giant question mark as I wonder what in the world the computer will actually be doing when it executes that code.
Anyway, my point isn't that one is better than the other, it's that it depends on what you're doing. For some of us, the higher levels of abstraction are actually less readable.
But I would argue that the majority of Go programs should not require the programmer to follow the intimate details of a program's execution when trying to understand the business logic. The fact that the language has a garbage collector would suggest that it was designed for programs where some details can be hidden away.
So I agree with your general point, but I am somewhat skeptical about its application to Go.
Also, it’s kind of laughable to think that writing code in a high level language with a runtime, GC, etc, you have any idea what will happen at execution.
1. Both examples contain a bunch of extra logic that has nothing to do with the difference between the two styles. For example, the code that converts users to that `UserLevelPoints` struct takes up a lot of space in both examples and is essentially the same in both. I think it would have been better (for pedagogical purposes) to just return the users (or perhaps a simpler struct containing a level and a user).
2. Both examples still have all the verbosity that is typical of imperative code, since it's still Go after all.
If one were to write the same function in a syntax which was designed for functional programming, the result might look something like this:
topUser = head . sortBy points
topUserPerLevel = topUser . values . groupBy level
In contrast, with the simple functional version I provided, I can read it in seconds and be reasonably confident that it is correct (assuming all the pieces fit together in a valid way, which a type checker can check for me).
If the unfamiliar syntax prevents you from grokking the example I provided, it's reasonable to assume it's just some clever code golf that should be discouraged in production code. But please resist that temptation; this is actually what good code looks like. It's simple, not repetitive, and there aren't many places for bugs to hide.
It could still be better though if you have, for example a number of pipelines for your data. There, the reuse of Sort/Group/Frobnicate/... in close by blocks would actually be nicer.
But yeah - the more extra syntax is required for those pattern, the worse they are in practice. (or, the more things you have to abstract in a small range to make it useful)
In your opinion.
> Generally, functional programs are safer and shorter than their procedural/OOP alternatives
Please provide any evidence that this is true, especially the ‘safe’ part.
Any study I’ve read says there’s almost no effect on language choice at all on bug rate.
> In your opinion.
Correct. But my opinion is informed by years of experience with functional programming, procedural programming, and OOP (each).
> Please provide any evidence that this is true, especially the ‘safe’ part. Any study I’ve read says there’s almost no effect on language choice at all on bug rate.
Sure, here is a paper which provides the evidence you're looking for: "A Large Scale Study of Programming Languages and Code Quality in Github" (https://dl.acm.org/doi/10.1145/2635868.2635922). That paper concludes:
> The data indicates functional languages are better than procedural languages; it suggests that strong typing is better than weak typing; that static typing is better than dynamic; and that managed memory usage is better than unmanaged.
I personally don't trust academic studies on programming language effectiveness, since they often contradict each other (e.g., https://arxiv.org/abs/1901.10220 contradicts the paper I cited above). But if you're looking for a peer-reviewed paper, there you go.
Choosing the right paradigm absolutely has an effect on the bug rate. If you have to write more code, repeat code, or deal with low-level details irrelevant to the problem at hand, you are going to make more mistakes. On top of that, functional languages often have better type systems than procedural/OOP ones, which helps with bug catching that much more. Taking this to extreme, with dependent type systems you can actually verify arbitrary mathematical properties of your programs.
How much experience do you have with functional programming? I have professional and academic experience with both functional programming and OOP, and everyone I've met with that experience agrees about the trade-offs I've been discussing in my comments.
Yes, functional programming exposes you to a lot more concepts that just conditional jumps, but that's fine. Because while a conditional jump might be a simple concept, it's mighty powerful and maybe not constrained enough.
Functional programming gives you more and simpler building blocks to put together. Yes, they have fancy names but most of the time they are almost trivial ("set together with an associative binary operation"). These building blocks interact with each other in very well defined ways that you do not get when trying to compose software in an imperative way.