Go code refactoring: the 23x performance hunt
medium.com
medium.com
This example was really nicely done; it's a classic problem, parsing input data. Some problems, on the other hand, require a different approach. See for example Knuth's wonderful introduction to boolean satisfiability in The Art of Computer Programing, Vol 4B, see [1]. Here the algorithms are really important to performance.
A perfect example of why Go should switch to a generational garbage collector with bump allocation in the nursery, like Java HotSpot, .NET, V8, etc. etc. use. There are consequences to Go's decision to make extreme throughput sacrifices in order to optimize for latency (low pause times) above all else.
> Now let's consider a generational GC. The point of a generational GC relies on the generational hypothesis: that most values allocated in a program are quickly unused, so there is an advantage for the GC to spend more time looking at recently allocated objects. Here Go differs from many garbage collected languages in that many objects are allocated directly on the program stack. The Go compiler uses escape analysis to find objects whose lifetime is known at compile time, and allocates them on the stack rather than in garbage collected memory. So in general, in Go, compared to other languages, a larger percentage of the quickly-unused values that a generational GC looks for are never allocated in GC memory in the first place. So a generational GC would likely bring less advantage to Go than it does for other languages.
> More subtly, the implicit point of most generational GC implementations is to reduce the amount of time that a program pauses for garbage collection. By looking at only the youngest generation during a pause, the pause is kept short. However, Go uses a concurrent garbage collector, and in Go the pause time is independent of the size of the youngest generation, or of any generation. Go is basically assuming that in a multi-threaded program it is better overall to spend slightly more total CPU time on GC, by running GC in parallel on a different core, rather than to minimize GC time but to pause overall program execution for longer.
> All that said, generational GC could perhaps still bring significant value to Go, by reducing the amount of work the GC has to do even in parallel. It's a hypothesis that needs to be tested. Current GC work in Go is actually looking closely at a related but different hypothesis: that Go programs may tend to allocate memory on a per-request basis. This is described at https://docs.google.com/document/d/1gCsFxXamW8RRvOe5hECz98Ft... . This is work in progress and it remains to be seen whether it will be advantageous in reality.
It's just a pointless thing for the Go team to dig in their heels over. They should implement generational GC, like most other languages with optimizing compilers do.
The ObjectStudio VM engineers used to lament over the advantage the VisualWorks GC had: VisualWorks only had to garbage collect one thread. ObjectStudio had gone multi-threaded, and to stop the world, the VM had to signal all of the threads, and wait for all of them to come to an inactive state. The more parallelism in the form of more machine threads, the higher this penalty is likely to be.
treating "request-oriented" GC as something other than generational GC, when it's really just a somewhat limited form of generational GC.
Isn't it more accurate to say that it's a simpler way of getting to use low cost bump allocation and very low cost deallocation? Even generational GC needs some knowledge and skill from programmers to achieve the highest level of efficiency. (Or even some modicum of knowledge to keep it from failing.) The golang request-oriented hypothesis just moves the cost-benefit tradeoff knobs. To get beyond middling performance, you still have to profile and optimize either way. Profiling allocation is usually straightforward.
I'm not sure what this is trying to imply. Modern generational GCs use TLABs to solve this exact problem.
Thread Local Allocation Buffer. Cool. Doesn't the stack in golang serve somewhat the same purpose? I've sometimes wondered about golang going the direction of Erlang, with multiple independent GCs. (Not per goroutine. It would have to be per "Actor" which I find myself implementing again and again in golang.)
Modern generational GCs use TLABs to solve this exact problem.
So that solves the problem on allocation, while the generational hypothesis reduces the frequency of stop the world. Generalizing in this fashion, I don't see why generational GC is inevitably necessary. With golang, the stack provides cheap allocation to local threads, even with concurrency, and throughput isn't as big a deal, with a bit more work needed by the programmer for optimization. (But perhaps my view would change with access to more data or with more experience.)
But remind that almost everybody having to implement a GC (but the Go team) thinks otherwise. Do you believe you know more than langage runtime engineers ? Because that's Go folks seams to believe, and that's anoyingly arrogant…
This is an incredibly technical area of discourse, and sadly, hacker news probably isn't going to be the place to see it happen.
But the whole idea that Go is made of a bunch of strongly-opinionated, insular developers who have no idea what they're doing is hogwash, especially in this area. Go had a sub-par garbage collector and Google put together serious resources to fix it.
For what it's worth, I've also heard that Go is considering moving to a generational GC, in spite of Ian's initial arguments against it. Hopefully they do.
Go team at Google would have enough real usage feedback from running Go systems in their massive data centers. They most likely find it reasonable with all things considered.
This debate about Generational GC is theoretical in nature, claiming Java GC design is universally applicable irrespective of language and its use cases.
There may indirect approval to Go approach by Java when developing value types:
http://openjdk.java.net/jeps/169
key Quote:
"In modern JVMs, object allocation is inexpensive, with a cost comparable to out-of-line procedure calling. But even this cost is often a painful overhead when compared to individual operations on primitive values. Thus, Java programmers face a binary choice between existing primitive types (which avoid allocation) and other types (which allow data abstraction and other benefits of classes). When they need to define small composite values such as complex numbers, pixels, or pairs of return values, neither approach serves. This dilemma often has no good solution, and the workarounds distort Java programs and APIs. "
If we go by claims that Java and its GC supporters always made:
1) RAM is Cheap 2) GC is incredibly fast so as not to worry about allocation.
There should be no need to waste tremendous multi year engineering effort to add value types to Java.
> Go team at Google would have enough real usage feedback from running Go systems in their massive data centers. They most likely find it reasonable with all things considered.
That's not just an appeal to authority, that's a big assumption. As far as I know, no Go maintainer has claimed that they have gathered large amounts of performance data and concluded that a generational GC wouldn't improve performance.
Also, calling generational GC "Java GC design" is disingenuous. Generational GC long precedes the creation of Java, and is used almost universally. Common Lisp (various implementations), .Net, the Lua VM (and the LuaJIT VM), BEAM, SELF, and Haskell (GHC at least) all use generational garbage collection. In fact, the current Go implementation is the only GC implementation I'm aware of that isn't generational.
There was a plan for a generational GC for LuaJIT [0] (not to mention a lot of other cool features and a very clever design) but it was never implemented (and at this point it probably never will be).
GC throughput is one part of the reason why Go code is often suboptimal for CPU-bound workloads. Adding a generational GC won't solve this, but it's bound to help.
Are there any go servers architected according to this? There are libraries for HTTP/HTTPS and JSON which strive to have zero allocations, so we're probably close to this as it is. If all code is allocated on the stack and all requests are served by goroutines, wouldn't this fit the hypothesis perfectly?
It's also pretty bad from an ergonomics (not to mention performance) perspective if the programmer has to manually switch to by-value copying of structs in order to keep escape analysis happy and avoid allocation.
I kind of like that Go's escape analysis is simple enough that I can reason about it reliably without profiling. I also tend to pass and return values unless I have a compelling reason to pass pointers (like mutation or rarely performance in the case of large structs), so I don't think I experience the sort of ergonomic issues you cite (and incidentally, I think this habit of mine makes writing Rust feel quite unergonomic to me since it seems to prefer egregious use of pointers). And I haven't noticed a significant speedup by switching from value copying to pointer dereferencing (but I've only tested this a handful of times).
The beauty of bump allocation in the nursery is that heap allocation is as cheap as stack allocation, at least in the common case.
I'm not sure what you're referring to w.r.t ergonomics. Passing small structs by value should probably be the default choice in Go, and it is not difficult to switch from passing via a pointer to passing by value.
But I don't disagree with what you are saying. It may be that Go would perform better with a generational GC. My claim is not that Go is better off with a non-generational GC (I don't know if it is or it isn't), but simply that it is less in need of one than e.g. Java.
edit: Do you mean that adding pass-object-by-value semantics to Java wouldn't decrease the number of short-lived object allocations because Java already does escape analysis? That may well be true. My claim was that the availability of pass-object-by-value in Go probably reduces the number of short-lived object allocations in Go, compared to what it would be if objects could not be passed by value and everything else remained the same.
I still consider this behavior "passing by reference" since it passes around references/pointers to objects, and does not incur an object copy.
nodes := make([]Node, t.CountNodes())
t.Nodes(nodes)
- No the bottleneck is somewhere else. Even considering this rule, it's not where you thought it to be even after discarding your first two ideas.
- There is no end to optimization. You can just find the problem, and then move the problem to another part of the code. It's less like capture the flag, and more like hot (or slow) potato. I guess you could be done with optimizations if you've built thoroughly optimized ASICs, because then you can blame electrons for wiggling too slow. But at that point I'd be impressed, a lot.
The remaining performance left behind is all in memory allocation and garbage collection - something you could optimize relatively easily if it were written in C. Such as by using a memory pool, so that you wouldn't need allocations or garbage collection at all.
Of course if performance isn't a big issue for your task, then none of this is really important.
This article nicely shows how optimizing your program yields more speed than randomly throwing goroutines at it. Finally it does use goroutines for a good effect, but after proper consideration.
The point of Golang is using them intelligently, not merely throwing at any problem like all you've got is a hammer...
One of the problems I see repeatedly when people try to benchmark things with concurrency is when they don't specify a problem that is CPU-intensive enough, so it ends up blocked on other elements of the machine. For a task like this, I'd expect optimized Go to easily keep up with a conventional hard drive, and with just a bit of work, come within perhaps a factor of 2 or 3 of keeping up with the memory bandwidth on a consumer machine (including the fact that since you're going to read a bit, then write some stuff, you're not going to get the full sequential read performance out of your RAM), not because Go is teh awesomez but because the problem isn't that hard. To get big concurrency wins, you need a problem where the CPU is chewing away at something but isn't constantly hitting RAM or disk or network for it, such that those systems become the bottleneck.
- the benchmark was designed to repeatedly parse an in-memory byte slice (not the hard drive), thus IO contention is unlikely here ;
- concurrency is a big win when IO is a bottleneck : keep processing dozens of things while some of them are waiting for data from network or hdd.
You could still be getting IO contention from the RAM system. RAM is not uniformly fast; certain access patterns are much faster than others.
"concurrency is a big win when IO is a bottleneck : keep processing dozens of things while some of them are waiting for data from network or hdd."
Concurrency is a win when IO is a bottleneck on a single task. Once you've got enough tasks running that all your IO is used up, adding more may not only fail to speed things up, but may slow things down. I'm speaking of situations where you've used up your IO. The tasks you're benchmarking are so easy per-byte that I think there's a good chance you used up your IO, which at this level of optimization, must include a concept of memory as IO.
I think you'd be helped by stepping down another layer from the Go VM and thinking more about how the hardware itself works regardless of what code you are running on it. Go can't make the hardware do anything it couldn't physically do, and I'm getting a sense you deeply understand those limits.
Ideally, in this case I would think one would want to check the number of cores and decide what route to take.
For the point at which the author removed parallelism, and the sequential code was faster, I think this was the case. The computation was too fine-grain. The author successfully took advantage of parallelism by applying it at a coarser granularity; each thread did more work. At this point, the author also does tune the solution for the execution environment, as he uses a fixed set of go-routines to process a bunch of messages rather than one go-routine per message.
FWIW I really mean the "take the numbers with a grain of salt" advice, i.e. "Your mileage may vary". What I'm sharing in this article is not a bunch of hard, strong, exact numbers ; It's a journey and an invitation to apply similar reasoning process to your own use case and hardware.