Hacker News Reader

The memory leak due to channels not being cleaned up was almost certainly because goroutines themselves are not garbage collected when they're forever blocked. In other words, `ch := make(chan int); go func() { ch <- 1 }()` is a memory leak. This is analogous to spawning a thread that deadlocks by attempting to lock a mutex twice, and has nothing to do with garbage collection.

About the GC pauses, check out the latency graphs at https://github.com/ixy-languages/ixy-languages (being careful not to mix up the Javascript and Go lines). Go handily beat every other garbage collected language in latency, and is in the same ballpark as the two non-GC languages (Rust and C). The peak tail latency times are measured in the hundreds of microseconds even at the highest loads tested, so I think the pundits may have a point.

Sounds very similar to "leaking" event listeners in other languages.

The handler and all of the scope it can reach stick around forever even though they should have been replaced or removed.

GC is more than just pauses. Throughput matters too; in fact, it often matters more than latency (for example, when writing batch jobs like compilers).

Sure, but that's not relevant to the discussion here. In this case the context was responding to the the statements that Go pundits claim the garbage collector has low latency but it always has bugs that will be fixed in the next version. I was providing evidence that the garbage collector actually does achieve low latency right now.

Also, there are ways to solve your throughput problems if you have control over your allocations, but there are not ways to solve your latency problems. Indeed, even if you don't have control of your allocations, you can often run multiple copies of your program to increase your throughput, but multiple copies will not help your latency.

Also, tuning your latency does in fact help increase throughput, because if your process spends a significant amount of time in allocations, its latency suffers. It's not a zero-sum knob between the two, especially in the presence of humans that care about tuning the performance of their applications.

You don't have much control over allocation in Go (in fact, according to the spec, you have none at all). Language constructs allocate in ways that are not obvious.

> Also, tuning your latency does in fact help increase throughput, because if your process spends a significant amount of time in allocations, its latency suffers.

I assume you meant to write "throughput" in that last sentence. That's not how throughput is defined. Throughput isn't "how long does it take to allocate", though that influences throughput. It measures how much time is spent in memory management in total during some workload. Optimizing for latency over throughput means you are choosing to spend more time in GC.

> You don't have much control over allocation in Go

Maybe you mean something different by "control over allocation"? Go does allow that, for example "The Journey of Go's Garbage Collector" talk has a good summary,

https://blog.golang.org/ismmkeynote

Specifically the sections about value-oriented programming are exactly that: Go allows the developer to avoid a lot of allocation by embedding structs, passing interior pointers, etc. Compared to Java or C#, it can have a much smaller number of allocations as a result.

> avoid a lot of allocation by embedding structs, passing interior pointers, etc

I can create a struct and take a ref to an interior field of that struct in C#, can I not? And if I wanted to badly enough, I could use unsafe code and take a pointer to the third byte of an interior field of that struct.

> Specifically the sections about value-oriented programming are exactly that: Go allows the developer to avoid a lot of allocation by embedding structs, passing interior pointers, etc.

To elaborate, for example, indirect calls cause parameters to those calls to be judged escaping and unconditionally allocated on the heap. Go style encourages frequent use of interfaces such as io.Reader. So in order to interoperate with common Go code, such as that of the standard library, you will be allocating a lot.

> Compared to Java or C#, it can have a much smaller number of allocations as a result.

C# also allows you to embed structs within other structs and pass interior pointers. Java HotSpot has escape analysis as well, though it's less important for that JVM (and other JVMs), since HotSpot has a generational GC with fast bump allocation in the nursery.

> https://blog.golang.org/ismmkeynote

As I've mentioned before, I also have a problem with the conclusion of this talk: that generational garbage collection isn't the right thing for Go. The problem is that nobody has tested generational GC with the biggest practical benefit of it: bump allocation in the nursery. I'm not surprised that generational GC is a loss without that benefit.

No, I meant what I wrote. The time spent in total in GC includes the time spent performing allocations. If you reduce the time spent in allocations with all else being equal, you reduce both latency and increase throughput. In that way, optimizing for your allocation latency also increases your total throughput.

I'm not disagreeing with you, but the example you're using (the ixy driver) is a pretty poor one to compare garbage-collected languages: Go performs particularly good on this one, because the go implementation doesn't allocate anything on the heap (and then the GC has nothing to collect). Being able to work on the stack only is a feature of the Go language, but it has nothing to do with its garbage collector.

This example is just to show that pauses are not in the same class as any other fully managed languages. I haven’t looked at the code to know how much it stresses the garbage collector, but I agree it does seem unlikely to do so.

That said, being able to work on the stack does influence the design of the garbage collector. Throughput becomes less important because you can choose to allocate less, and pause time is more important because latency is harder to optimize. Latency is affected by your whole stack and can’t be meaningfully reduced by just adding more machines in the same way that throughput can be increased.

> This example is just to show that pauses are not in the same class as any other fully managed languages.

But your choose one of the few examples where there is no pause at all, because go's GC didn't even tick once!

Go could have a 10 seconds stop the world, it would still perform the same way here, that's why I say this examplei is a poor one for the point you're making (which is valid, even though one could argue than 10ms pauses are better than 30% of the CPU usage being the GC running (actual production usage on a Go service at my former work)).

> but any language with GC is a complete nonstarter as a systems programming language.

Check this out: http://www.projectoberon.com/

A full system (not just a language, but also custom CPU on FPGA, OS with GUI, applications, etc) where the language is garbage collected. The system has 1MB of RAM (by default). Note that the garbage collector is implemented entirely in the language itself.

The Oberon language was also a big inspiration for Go.

> The interesting thing about Go is Rob Pike envisioned it as a systems programming language but any language with GC is a complete nonstarter as a systems programming language.

AFAIK, by "systems programming" he meant "non-user-facing programming", not "kernel or embedded programming". So, basically, servers, batch programs and other os utilities.

I'd always assumed he was using "systems language" to just mean "as opposed to a scripting language" like perl or bash.

> any language with GC is a complete nonstarter as a systems programming language

If it was possible to write whole operating systems in garbage-collected LISP in the 80s, then it surely is possible to use a GC'ed language for systems programming thirty years later.

If you dig into those machines, anything vaguely real time including huge chunks of the drivers were written in user writable microcode.

Sure, but that's not really a universal indictment of a system or a language. For example, C gets a lot of justifiable criticism, but the ability to drop down to assembly when it's needed is considered a feature and not a bug.

The highest-performance gc systems (OCaml?) might very well be associated with functional programming languages, but the more relatable system for a lot of people might be the Oberon system, which did influence Go. I'm pretty certain Oberon did not rely on any hand-coded assembly to build a functioning workstation. (although, on the topic of microcode: the newest Wirth Oberon incarnation does have the student program an FPGA to run the system, as an exercise...)

Oberon for sure depends on hand coded assembly. It's just in the compiler rather than .asm or .s files, but it's still a big blob of assembly that the type system and GC have no real knowledge of.

Right now, state of the art for real time GC is that it's a huge trade-off between throughput and determinism. Like orders of magnitude lower throughput to be able to make guarantees that you'd expect out of a desktop system.

> It's just in the compiler

What's your objection to having the compiler emit "hand coded assembly?" It's not clear to me what axe is being ground here. (that's true for me in a big picture way here too. There's a very strong argument against gc languages being made here and I'm not sure if you're saying Oberon doesn't count as a gc language, or what.)

> Like orders of magnitude lower throughput to be able to make guarantees that you'd expect out of a desktop system.

Do you have a current citation for that?

More recent examples to check out were Spin (Modula-3), JX (Java), and House (Haskell).

It was also found in the 80s that any operating system written in LISP was dog slow. They even started making dedicated hardware interpreters of LISP to try and get around this. https://en.wikipedia.org/wiki/Lisp_machine

When you need to start designing your hardware around your computer language you know there's a problem going on.

To be fair those machines made sense at the gate counts we're talking about. They were true von Neumann machines with no caches, so the vast majority of the value add was putting the interpreter in microcode where it doesn't compete with the fetch bandwidth of the rest of the application. That's why CISC machines of the time tend to have mem(cpy/set/cmp) instructions, getting the instruction bandwidth out of the way of data heavy loads in a way that makes sense for the use case.

That's also why these dedicated machines tended to disappear right as instruction caches became more standard; an I$ solves the same problem in a way more general way. As long as the hot path to your interpreter fits in I$, it's six of one, half dozen of another.

It's not just putting the bytecode interpreter into microcode; there is the question of data representations suitable for Lisp, like pointers and numbers with tags.

Caching doesn't really help with this. Even when everything is nicely in I$ and L1, it costs cycles to do the type checking. What helps is compiler techniques: type inference to eliminate some of the checks. But hardware can basically bury the cost of the checks; you can do them all the time on all operands.

> It was also found in the 80s that any operating system written in LISP was dog slow.

Lisp Machine operating systems were written in the mid 70s - developed for a new breed of computers: single user workstations with graphical user interfaces.

They were developed at Xerox PARC and at the MIT AI Lab.

> They even started making dedicated hardware interpreters of LISP to try and get around this.

The machines were built to get around slow Lisp systems on time-shared computers with tiny memory. They wanted to have dedicated machines with memory exclusive memory for that one user.

A bunch of stuff was then invented for those systems, including better automatic memory management like generational garbage collectors.

Around 1980 (!) such machines were extremely expensive and still had tiny hardware: around 1 MB RAM and approaching 1 VAX MIPS of speed. Mid/end 80s they had 20 MB RAM and 5 MIPS...

No wonder: an entirely new class of systems was developed on the slow hardware of the time.

> started making dedicated hardware interpreters of LISP to try and get around this.

That's a misconception; the actual technology of that type provides an instruction set architecture (which isn't Lisp), to which Lisp is compiled.

The issue with Go and garbage collection is basically that Go's GC is just a GC tuned for latency above all else. The GC of Java HotSpot, on the other hand, balances throughput and latency, and is configurable to target one or the other. The latter is typically what applications want, even if it can't advertise the same pause times. For example, allocation is an order of magnitude cheaper in Java HotSpot than it is in Go.

How do you explain the throughput and latency numbers posted by https://github.com/ixy-languages/ixy-languages? Go is beating Java in both latency and throughput, and the author attempted many different Java GC tunings (https://github.com/ixy-languages/ixy-languages/blob/master/J...)

Maybe tuning for latency is the appropriate trade-off given Go's high level of control over allocation and data layout?

Because you are mixing up language with implementation.

https://www.ptc.com/en/products/developer-tools/perc

https://www.aicas.com/cms/en/JamaicaVM

Go's a pretty simple language.

It should be straightforward (not easy, because compilers aren't easy) for somebody to write another implementation, tuned for another use case - I can't imagine it wouldn't be easier than doing it for Java.

You're jumping to conclusions. I understand the distinction and that there are many options for Java. Please be charitable.

To rephrase the question more precisely: can you explain why OpenJDK, which has a multitude of garbage collector implementations tuned for both throughput and latency, performs worse on both throughput and latency than Go on this benchmark which has a garbage collector that, according to the original statement, is "tuned for latency above all else"?

Perhaps OpenJDK is problematic in other ways (the author of the benchmarks suspects it's JIT induced, even though that's attempted to be controlled for), or perhaps this test depends less on the GC for some reason. Or maybe typical Go programs don't require as many allocations making allocation time much less important making tuning for latency the correct engineering decision. Would HotSpot do significantly better as is claimed? That's the sort of interesting technical discussion I'd like to have, instead of pedantry.

Sorry about that.

I guess being a long time part of Java eco-system, kind of gets tiring of having outsiders (in general, not referring to you) always mixing up Java with what comes with their PC, as if C would be defined by GCC.

As for the actual question, naturally having value types helps reduce GC pressure. Which on Java's case could be helped by trying out Graal or other JVMs that do better job at escape analysis than Hotspot. Alternatively, although it kind of is cheating, using the language extensions from either Azul or IBM for value types.

In any case, when inline classes (aka value types) arrive, Java can easily do the same as Go here.

JIT and de-optimizations play a role certainly, and are to blame for some performance impact, which can be further improved if a JVM like J9 gets used, given that it allows for PGO across runs.

Finally, while Hotspot has good defaults, tuning all the knobs is a science, even with help of J/Rockit and VisualVM, which opens the door for performance consulting.

The JVMs I mentioned, are targeted at soft real time deployment scenarios, as such they have APIs for low level control of memory management, while also supporting AOT with PGO compilation, thus allowing for low level fine tuning out of reach for the regular Java developers (pure Java SE implementations).

Thanks.

Since most of the explanations that you gave involve things like controlling value types and data layout, it sounds to me like you might agree with the statement that in languages with better control over allocation and data layout, tuning a garbage collector for latency over throughput can be a good idea because your time spent allocating is less important. Is that fair?

In this case, "tuning for latency over throughput" means "not having a generational GC". Whether this choice makes sense does not depend on how much a program allocates. Rather, it depends on whether the generational hypothesis holds. The generational hypothesis is one of the most powerful and consistent observations in the entire CS field. It certainly holds for .NET, which has a very similar memory model to that of Go, and therefore .NET has a generational garbage collector.

It's not as simple as does the generational hypothesis hold, there are real downsides to having a generational GC - it means you need a copying collector. Go heap values never move, which dramatically simplifies everything else (e.g. other threads don't need to be paused to update heap pointers, reducing STW times).

Some previous discussion https://news.ycombinator.com/item?id=17551012

You don't need stop-the-world pauses for generational GC. As long as your write barrier ensures that pointers to a young object are entirely local to the TLAB that the object is allocated in, you will never need to stop other threads to sweep the nursery.

> How do you explain the throughput and latency numbers posted by https://github.com/ixy-languages/ixy-languages?

Because those graphs measure overall throughput and latency in an I/O setting, not GC throughput and latency specifically. There are many other confounding factors. In particular, the application in question has been tuned to perform as few allocations as possible, so GC throughput will naturally not show up as much as in other apps!

Thanks to its generational GC and TLABs, allocation in HotSpot is like 5 instructions.

> Maybe tuning for latency is the appropriate trade-off given Go's high level of control over allocation and data layout?

Go doesn't give you control over allocation in a meaningful way.

I believe we misunderstood each other. In particular "the application in question has been tuned to perform as few allocations as possible" seems contradictory to "Go doesn't give you control over allocation in a meaningful way."

Allocation being fast isn't very important if you don't spend a significant portion of your time allocating. Isn't it possible that some languages (maybe even Go) by the nature of their semantics spend significantly less time allocating than other languages, and so tuning a garbage collector for latency is the appropriate decision?

Why do you seem to believe that the Go application has been tuned more to avoid allocations than the Java application? If it hadn't been, then you would have to agree that tuning the garbage collector for latency is better because you can invest some effort into your application to have it perform better on both throughput and latency.

Or is the argument that because "[throughput] is typically what applications want", this application is not a good example of most applications (according to what measure of "most")?

You can sometimes use unnatural programming patterns in both Go and Java to reduce allocations in practice. Nontrivial programs will always allocate, though, because various language constructs allocate in ways that are not obvious, and escape analysis is complex and hard to reason about. And the allocation semantics are not part of the language and are subject to change: this is what I mean by it not being meaningfully controllable.

The most salient difference between Go's GC and HotSpot's GC is that the latter is generational. There is no convincing reason I've seen for Go not to have a generational GC, which would dramatically improve throughput by enabling bump allocation in the TLAB. The tiny amount of latency that this could add is by no means worth the cost of making allocations an order of magnitude slower. Allocation in HotSpot is five instructions.

And yet here’s an empirical benchmark providing evidence against both of those claims. What empirical evidence do you have supporting yours? Can you provide them?

Who defines when a construct is “unnatural”? Can you show examples of “unnatural” patterns in the provided benchmark programs supporting your hypothesis that you have to write unnaturally to control allocations? If not, why are you making those claims?

Why is throughput of allocations important for “most” applications? How much time is typically spent on allocations in those programs? What is a typical application? What about typical applications in Go specifically since that’s the only language that Go’s garbage collector matters for?

Also, why did you bring up the point about generational collectors? It seems like a red herring. This discussion has been about tuning for latency, and how in this benchmark, Go’s implementation did better on both throughput and latency than any OpenJDK collector under any tuning, and that tuning for latency is possibly a sound engineering decision.

I’ve also noticed that you’ve made no explicit attempt to answer my more difficult questions, and if you continue to do so I will no longer assume you are acting in good faith.

Again, that benchmark isn't a benchmark of garbage collection; it's a whole-program benchmark.

There's plenty of empirical evidence—a wealth of papers dating back to the 80s—that generational GC provides better throughput than non-generational GC on most applications. I'd be extremely surprised if a properly-implemented generational GC with bump allocation in the nursery wouldn't improve the performance of Go's GC by trading off a small amount of latency for increased throughput. The reason why you won't see a benchmark like that for Go is that nobody has implemented such a collector for Go.

No cited references, no answers about what “most” is, no answers about what “most” is in the specific context of Go, no answers about any “unnatural” patterns existing, no answers about the claim that it was “tuned” to reduce allocations, answering a question that wasn’t asked dodging the question that was (generational having better throughput vs when does throughput matter).

There’s no way I can believe you are acting in good faith. It’s obvious to me now that you’ve just been trolling for years every time you discuss the topic of garbage collectors. I’m no longer going to engage.

I am not sure I'd go that far, but I want to point a thing out:

Back in the 90s, it was a running joke that if you were going backpacking, you should take along a 3' length of fiber optic cable. "If you get lost, just bury the fiber optic cable and ask the backhoe operator for a ride back to town."

In at least three programming-related communities I'm in, I have made a quip about "I think I'm gonna replace the 3' length of fiber optic cable in my backpacking gear with a short post about Go's garbage collector", and people have filled in "so if I get lost, I can just take the post out, and then ask pcwalton for a ride back to town".

I can't say whether or not the argument is made in good faith, but man, there sure is a lot of it.

If you're trying to write high performance Java, that driver was a pretty poor attempt. While not idiomatic in the normal sense, there are a lot of low-latency, high throughput Java apps that never allocate after startup (and maybe some object pool growth before hitting a steady state). That benchmark was even running out of memory when using the non-collecting GC which shows they were allocating quite a bit, and their allocations probably escaped and needed to be moved off the TLAB and become expensive. I'm not sure they really knew how to write performance Java.

I've been a part of systems that process million of events a second and never allocated. These will beat most C++ systems I've seen, and Go doesn't even start a chance. Java and HotSpot can do some amazing things (esp around inlining), but you do have to do them a little differently (and carefully), but at least you can. My experience with Go is that I never had the same level of control, and I don't think it is possible.

> You can sometimes use unnatural programming patterns in both Go and Java to reduce allocations in practice.

This whole "you can't control your allocations in Go" thing is a strawman. You don't have absolute control because of escape analysis, that's true. But there's a pretty wide gap between "oh I'll just move this allocation point out of a loop" and "oh I wrote a bunch of unnatural code". It's pretty idiomatic to manage allocations in Go, just look at all the posts about using pprof.

Besides, the spectrum of "clarity" (or whatever) to performance is present no matter what language you're using. Ex: ripgrep takes on more complexity so it can search a big buffer of text instead of just a line [1]. I wouldn't call that "unnatural" at all, just systems programming.

[1]: https://blog.burntsushi.net/ripgrep/#mechanics

> But there's a pretty wide gap between "oh I'll just move this allocation point out of a loop" and "oh I wrote a bunch of unnatural code".

Here's just one example: The Go compiler judges that parameters to indirect calls such as interface method calls as escaping. So in Go if you don't want to allocate you have no choice other than to avoid interfaces. But Go heavily encourages the use of interfaces, especially in the standard library.

In C, C++, and Rust, on the other hand, you can use indirect calls without allocating, because the language guarantees the escaping behavior. This is a significant difference.

My experience in the Golang side of this is limited to profiling and optimizing Kubernetes, which doesn't have extensive use of interfaces in general, but over the last 5 years I would say 90% of the wins we have have been:

1. optimizing allocations away in serialization

2. optimizing allocations away in api or biz logic in critical paths around that

3. algorithmic improvements on certain naive code paths

4. blocking

... distant gap

5. everything else

Serialization is its own can of worms in Go, and many places we tried to optimize came down to trying to avoid a) stupid (don't use pointers to naive objects) and b) obvious (use value types) and then hit a wall where there weren't many cheap wins.

I probably can count on one hand the number of hot paths which were interface related, so while I've occasionally been annoyed at the forced allocation moving into an interface, it's rarely actual something I've gotten a win from.

That's just one particular experience, but the AMAZING integration with pprof from very early days has saved me far more time in improving perf than other Go annoyances has cost (relative to my experience in Java 2005-2011).

Oh yeah, totally agree that you have way more control in C/C++/Rust, and that's non-negotiable in many important cases. I just don't think that's the same as "you don't have any meaningful control over allocations in Go"; you still have a lot of control. It's a spectrum, and if you don't do enough research you might find you need more control (you used Go) or you might also find you signed up to deal with more low-level memory management than you needed to (C/C++/Rust).

> There is no convincing reason I've seen for Go not to have a generational GC

Well, they explored generational GC (after admitting that their uber-ultimate GC that they marketed as future-proof until 2025 could be better): https://blog.golang.org/ismmkeynote

> It isn't that the generational hypothesis isn't true for Go, it's just that the young objects live and die young on the stack. The result is that generational collection is much less effective than you might find in other managed runtime languages.

> any language with GC is a complete nonstarter as a systems programming language

As someone else hinted, your rant loses everyone familiar with the history of computing right here.

"There were bugs in Go at one time" is not much of an argument against it either.

And in the meantime Java gets its GC finally fixed to not stall. Yes, it's a real problem, ignoring it for many years costed some people lots of productivity.