Hacker News Reader

Comment by defraudbah | Hacker News Reader

looks very promising, one of the biggest issue in golang for me is profiling and constant memory leaks/pressure. Not sure if there is an alternative of what people use now

View on HN

I'd love to hear more! What kind of profiling issues are you running into? I'm assuming the inuse memory profiles are sometimes not good enough to track down leaks since they only show the allocation stack traces? Have you tried goref [1]?. What kind of memory pressure issues are you dealing with?

[1] https://github.com/cloudwego/goref

Disclaimer: I work on continuous profiling for Datadog and contribute to the profiling features in the runtime.

I for one am still mystified how it's possible that a GC language can't expose the GC roots in a memory profile. I've lost so many hours of my life manually trying to figure out what might be keeping some objects live, information the GC figures out every single time it runs...

Do you think the GC roots alone (goroutine stacks with goroutine id, package globals) would be enough?

I think in many cases you'd want the reference chains.

The GC could certainly keep track of those, but at the expense of making things slower. My colleagues Nick and Daniel prototyped this at some point [1].

Alternatively the tracing of reference chains can be done on heap dumps, but it requires maintaining a partial replica of the GC in user space, see goref [2] for that approach.

So it's not entirely trivial, but rest assured that it's definitely being considered by the Go project. You can see some discussions related to it here [3].

Disclaimer: I contribute to the Go runtime as part of my job at Datadog. I can't speak on behalf of the Go team.

[1] https://go-review.googlesource.com/c/go/+/552736

[2] https://github.com/cloudwego/goref/blob/main/docs/principle....

[3] https://github.com/golang/go/issues/57175

no, haven't heard of goref yet but will give it a shot!

usually I go with pprof, like basic stuff and it helps. I would NOT say memory leak is the biggest or most common issue I see, however as time goes and services become more complicated what I often see in the metrics is how RAM gets eaten and does not get freed as time goes, so the app eats more and more memory as time goes and only restart helps.

It's hard to call it memory leak in "original meaning of memory leak" but the memory does not get cleaned up because the choices I made and I want to understand how to make it better.

Thanks for the tool!

Sorry if this is a basic question but are you setting the GOMEMLIMIT?

Also, are you running the code in a container? In K8s?

How are you getting "constant memory leaks" in a GC'd language?

it's not hard. GC lets shit leak until it decided to clean it up...

do you think they will enable Valgrind if there's no leaks?

valgrind finds sooooo many more problems than just memory leaks

uninitialized memory, illegal writes, etc... There's a lot of good stuff that could be discovered.

not to mention cachegrind, callgrind and other things it bundles.

sorry, i guess when i say leaks i mean a bit more broad stuff :'). my own words are a bit leaky hah

still doesnt mean i am wrong. GC doesnt clean up memory when its released but when it wants to, effectively offering opportunities to get that data after a program dont need it anymore. until some point in time u can usually not specify, just hint at.

that in light of things like bad memory ordering between threads etc..can have nasty bugs... (raii has similar bugs but since its more determenistic you can program your way around lot of it more easily and reliably)

Golang has a feature that I love in general but that makes it very easy to keep unintended allocations around. If you have a struct with a simple int field, and you store that somewhere as an *int, the entire struct and anything it points to will be kept alive. This is super useful for short-lived pointers, and super dangerous for long-lived pointers.

Most other widely used GCed languages don’t allow the use of arbitrary interior pointers (though most GCs can actually handle them at the register level).

> If you have a struct with a simple int field, and you store that somewhere as an *int, the entire struct and anything it points to will be kept alive.

While Go allows interior pointers, I don't think what you say is true. runtime.KeepAlive was added exactly to prevent GC from collecting the struct when only a field pointer is stored. Take a look at this blog post, for example: https://victoriametrics.com/blog/go-runtime-finalizer-keepal...

I don’t believe that’s the case based on the example in the blog post. The fd field in that struct was passed into the later function by value (i.e. as an int, not an *int), so there was no interior pointer in play at all.

You are right; I stand corrected

There’s ways, GC isn’t perfect.

A common one I see fairly often is opening a big file, creating a “new slice” on a subset of the file and then using the “new slice” and expecting the old large object to be dropped.

Except, the “new slice” is just a reference into the larger slice, so its never marked unused.

Interesting, I always thought of slices as stand-alone, I wonder if its the same in Python?

No, python slice syntax on lists returns a new list. Of course, the slice syntax on your own classes can do just about anything using operator overloading.

A go slice is a wrapper around a normal array. When you take sub-slices those also point to the original array. There's a possible optimization to avoid this footgun where they could reallocate a smaller array if only subslices are reachable (similar to how they reallocate if a slice grows beyond the size of the underlying array).

Most sublicing is just the 2-arg kind so it would not be safe to truncate the allocation even if the subslice is the only living slice because the capacity still allows indirect access to the trailing elements of the original slice. This optimization would only be truly safe for strings (which have no capacity) or the much less common 3-arg slicing (and Go would need to have a compacting GC).

Of course, the language could also be changed to make 2-arg slice operations trim the capacity by default, which might not be a bad idea anyway.

Yeah I understand why they can't do it for backwards compat reasons. Considering they had the forethought to randomize map iteration order it's a big foot gun they've baked into the language.

there are many ways:

    - you can create deadlocks
    - spawn goroutines while not making sure they have proper exit criteria
    - use slices of large objects in memory and pass them around (e.g. read files in a loop and pass only slice from whole buffer)
    - and so on

A GC only deallocates unreferenced memory, if you keep unused references, that's a leak the GC won't catch, as it has no way to know that you won't need it later.

It can happen when your variables have too long a lifespan, or when you have a cache where the entries are not properly evicted.

But how would Valgrind know more than the GC? Of course a program in a GCed language can leak memory, but it’s not clear to me how Valgrind would detect the kinds of memory leaks that you can create in pure Go code (without calling into C or using unsafe functions to allocate memory directly).

Valgrind can tell you what is still in use when the program exits, along with useful information, like where it comes from. You can then assess to situation and see if it is normal or not.

In addition, Valgrind is actually a complete toolsuite, not just a memory leak detector. Among these tools is "massif", a memory profiler, so you will have a graph of memory use over time and it can tell you from where these allocations come from.

Not, if your language is fully GCed you can have a debug GC that does the job more efficiently than Valgrind on this task, but I don't know if it is the case for Go.

Pprof should cover most of that for pure Go code, though.

This was often a question asked in Java interviews as well.

In Java heap fragmentation is usually considered a separate issue but I understand go has a non-moving garbage collector so you can lose memory due to pathological allocations that overly fragment memory and require constantly allocating new pages. I could be wrong about this since I don't know a lot about go, but heap fragmentation can cause troubles for long running programs with certain types of memory allocation.

Beside that, applications can leak memory by stuffing things into a collection (map or list) and then not cleaning it up despite becoming "stale". The references are live from the perspective of the garbage collector but are dead from the application perspective. Weak references exist to solve this problem when you expose an API that stores something but won't be able to know when something goes out of scope. I wouldn't consider this to be common, but if you are building a framework or any kind of platform code you might need to reach for this at some point. Some crazy folks also intern every string they encounter "for performance reasons" and that can obviously lead to what less crazy folk would consider a memory leak. Other folk stick a cache around every client and might not tune the cache parameters leading to unnecessary memory pressure...

Not going to claim this is all sources, but Go makes it extremely easy to leak goroutines.

Ideally, they would have learnt from other languages, and offered explicit control over what goes into the stack instead of relying into escape analysis alone.

As it is, the only way to currently handle that is with " -gcflags -m=3" or using something like VSCode Go plugin, via "ui.codelenses" and "ui.diagnostic.annotations" configurations.

I sometimes dream of a GCed language with a non-escaping pointer type. However to make it really useful (i.e. let you put it inside other non-escaping structs) you need something on the scale of the Rust borrow checker, which means adding a lot of complexity.

https://oxcaml.org/documentation/stack-allocation/intro/ ?

Which is why several GC languages (the CS meaning of GC), are rather going down the path to keeping their approach to automatic resource management, plus type systems improvements for low level high performance code when needed.

So you only go down into the complexity of affine types, linear types, effects, formal proofs, dependent types, if really needed, after spending time reasoning with a profiler.

Now, this does not need to be so complex, since languages like Interlisp-D and Cedar at Xerox, that many GC languages have offered value types and explicit stack allocation.

That alone is already good enough for most scenarios, provided people actually spend some time thinking about how to design their data structures instead of placing everything into the heap.

> constant memory leaks/pressure

In Go, never launch a goroutine that you don't know exactly how it will be cleaned up.

pprof is pretty good, what do you need?

yes, that's what I use, just wonder if there are alternatives. I am not sure how valgrind compares to it or the goref tool mentioned above, just asking around, does not hurt.

Alternative to solve what problem? pprof is very powerful, it's not missing much.

Pprof doesn't tell you if something was leaked aka still around.

I fixed a leak recently because of misuse of a slice with code like

slice = append(slice[1:], newElement)

I only figured it out by looking at the pprof heap endpoint output and noticed there were multiple duplicate entries.

Reply on news.ycombinator.com