Hacker News Reader

Rabbit holes for you to chase[1][2][3][4][5]

Wow.

So I fully believe that Go will eventually solve this and be even more valuable, because if people are creating so much value with it right now it is likely to get exponentially better as the Golang team considers and takes on things they avoided in the past (ex. Generics).

I've glanced at those holes but I don't want to go down them, I have written a large personal project in go in like 5 years and I probably won't ever again because Rust. Also if I really wanted to be productive I'd pick Typescript.

I do have one for you, in kind[0].

[0]: https://github.com/kubernetes/kubernetes/issues/53533

> I ask because I just ran into this working on a tiny project whose only job was to serve a single file[0] (with few resources) and was immediately bit by the OOM killer (I used docker to artificially limit resources).

Not a fan of Go but unless your memory limit is extremely low there's no reason for that to happen, the GC does work. So clearly there's some sort of memory leak in the user code, either data being appended to a global, or (more likely?) a goroutine leak of some sort.

Maybe check your globals (is there a global slice or map?), and whether all your goroutines will terminate on their own or are guaranteed to only be spawned once? Else it'd be a similar issue in a dependency, looking at kcup.go fasthttp would be the most likely culprit.

edit: looking at their bug tracker, it might also be something like https://github.com/valyala/fasthttp/issues/518: the request is not considered responded to so the goroutines never terminate. I don't know the fasthttp API at all tho, just looked for "leak" on their bug tracker and that stood out

So the thing is I limited the memory to 100MB (because how much memory could you need to serve 1MB, even if every go-routine stack is 2-5kb and the references are all to one piece of data?)

This was an issue, but I found that even with giving it more memory things didn't get much better -- the performance was still much worse. Weirdly enough, net/http did well memory wise (and reasonable performance wise, for Go), but I went to fasthttp since that was what was at the top of benchmarks for go, but the simplest you-would-think-this-would-work code didn't produce the results I would have expected.

[EDIT]

> edit: looking at their bug tracker, it might also be something like https://github.com/valyala/fasthttp/issues/518: the request is not considered responded to so the goroutines never terminate. I don't know the fasthttp API at all tho, just looked for "leak" on their bug tracker and that stood out

So I actually got some help from a reader which was pretty awesome (https://gitlab.com/mrman/kcup-go/-/merge_requests/7), the fasthttp code performed much better after that.

I should also mention that I got some advice that didn't pan out -- in particular using fasthttp.ServeFile (https://gitlab.com/mrman/kcup-go/-/merge_requests/9)

> I ask because I just ran into this working on a tiny project whose only job was to serve a single file[0] (with few resources) and was immediately bit by the OOM killer (I used docker to artificially limit resources).

This looks like some badly-written, memleak-inducing code. You can cause that in pretty much any language.

To Golang's defence - we use InfluxDB and Telegraf on hundreds of production hosts and I've never seen those processes grabbing unhealthy amounts of memory.

> This looks like some badly-written, memleak-inducing code. You can cause that in pretty much any language.

Could you point at the line that is an obvious memory leak? There are three versions right now:

v1 - net/http (no leak, but perf wasn't great in comparison)

v2 - fasthttp

v2.1 - fasthttp (+ []bytes)

v2.2 - fasthttp (mem leak fixed thanks to a contribution)

All of these versions are <50 lines of code. The issue was not the code I wrote, but my misuse of the underlying library (fasthttp in this case) -- but I don't think any part of the code I wrote stands out as particularly memleak-inducing. I'd like to think writing very similar code against net/http and fasthttp producing such different results is a bit of a problem, my golang code aside. The interfaces look the same, act the same, but seem to have wildly different consequences.

> To Golang's defence - we use InfluxDB and Telegraf on hundreds of production hosts and I've never seen those processes grabbing unhealthy amounts of memory.

This is reasonable, I expect InfluxDB to be very well written software, but I'd like to point out:

- the Go devs at InfluxDB are way better than me at writing go

- memory usage and management (and not blowing up your server) is a key feature of InfluxDB

- I was trying to serve... a single file

At this point I'm quite tempted to try this with PyPy + Falcon or NodeJS and see how similarly naive code would perform with 2 cores and 100MB. Those other languages have different characteristics (harder to deploy for example) but it feels like all I'd have to do is make the right library choice (which I tried to do for Go) to get some pretty decent performance (which I expected to get with Go).

You just admitted yourself in a reply to a sibling comment that there was a memory leak. What am I missing here?

> This looks like some badly-written, memleak-inducing code. You can cause that in pretty much any language.

I must have misunderstood you because it read like you were suggesting the code could be determined to be memory leak inducing at a glance — wanted to know which part of it looked like that if so.

> While we're here I'd love to talk about Go's lack of memory pressure control/feedback though. How people are working around it/using Go in production -- is everyone just right/over-provisioning servers? using `ulimit memlock`? Hoping on swap?

none of these in my case. i simply used go’s benchmark tool to figure this out. i kept running it and trying out stuff along the way. at the end of development i ended up with a very performance-focused codebase.

result: i now have a running golang app that’s been up for 2 years without a single restart, indirectly serving millions of users on just 4 cores and 4gb of ram. no memory leaks, no cpu issues. very impressive compared with the standard java solutions that that company usually pursues.

this being said, it really depends on your use case. this golang software was a proxy that was handling an LRU cache. definitely not rocket science.

> none of these in my case. i simply used go’s benchmark tool to figure this out. i kept running it and trying out stuff along the way. at the end of development i ended up with a very performance-focused codebase.

That sounds like "right-provisioning", or am I misunderstanidng you?

> result: i now have a running golang app that’s been up for 2 years without a single restart, indirectly serving millions of users on just 4 cores and 4gb of ram. no memory leaks, no cpu issues. very impressive compared with the standard java solutions that that company usually pursues.

This kind ofexperience is exactly what I was expecting to get out of Golang, very easily. That said, 4 cores sand 4GB of RAM is a lot of resources for a single binary, also there is the magic of swap to save you if you ever get in trouble (the app would theoretically get slower to the point of becoming unresponsive). I'm not sure how complex this application is, and I'm certainly not arguing for Java (because that's both hard to optimize, tune and write in the first place) but I wonder if there's a difference in complexity here. Is it the usual 3 tier app kinda deal?

> this being said, it really depends on your use case. this golang software was a proxy that was handling an LRU cache. definitely not rocket science.

Ahh there it is, thanks for sharing this bit -- Go would function as an excellent LRU cache -- I assume this program is performing a redis like functionality for another app? Are you offering the LRU functionality itself to users? I am actually kind of interested in what it does now...

It sounds like you have a memory leak in your golang app. That shouldn't be happening. Time to bust out pprof.

I'd say though GC is what limits golang as a systems language. If you are doing lots of memory heavy things, I'd say golang may not be the correct language to use.

[1] https://github.com/prometheus/prometheus/issues/6934

Definitely a memory leak, but the code is like < 50 lines -- it's a memory leak (and/or my bone-headed misuse) in the underlying library, fasthttp.

I got some help with fixing it from a reader though:

https://gitlab.com/mrman/kcup-go/-/merge_requests/7

Again, the goal was to serve a single file -- I should point out that v1 was using net/http and it did quite well (memory scaled up and down as necessary), but the combination of resource contraints (on my side, I only gave it 100MB of RAM), and use of fasthttp that wasn't ideal/correct caused the issues. It went something like this:

v1 - net/http

v2 - fasthttp (memory leak but I didn't notice at this point)

v2.1 - fasthttp (use []bytes, still has the leak)

v2.2 - fasthttp (leak fixed thanks to Pavel)

All these solutions performed worse than their rust counterpart though, so I was surprised at how much work Go was making me do more than anything.

GC is certainly what limits golang as a system language, and if I think about it I am certainly standing on top of a lot of abstraction on the Rust side (hyper, tokio, structopt), but this exploration made me more of a rust fan than anything. The ~30 mins I spent figuring out how to efficiently pass the references was painful at the time but the payoff has been hours of not futzing with the code.

I've read your code and there is no reason to use all those libraries, it could have been done in a performant way with std libs. Why would you use fasthttp?

Did you read the post (there’s an attached blog post)? I was worried I wasnt giving golang a fair shake because the performance (under constraint) was so bad in comparison. Fasthttp did end up being faster also, I just had to fall into a bit of a pitfall first.

There's a proposal in github about being able to set an upper memory limit. Right now, you'd have to live with the possibility of being evicted from a k8s cluster however. That being said, if you think your go process shouldn't be consuming "that much memory (user definition here)", it's time to start profiling. I've uncovered my fair share of pebcak memory leaks that way.

> There's a proposal in github about being able to set an upper memory limit. Right now, you'd have to live with the possibility of being evicted from a k8s cluster however.

You've activated my trap card! I actually have another issue with k8s -- their rejection of swap seems like a bad judgement (though I'm sure it helped them ship faster and accomplish other things -- I just made this comment[0].

> That being said, if you think your go process shouldn't be consuming "that much memory (user definition here)", it's time to start profiling. I've uncovered my fair share of pebcak memory leaks that way.

Yeah but I was trying to serve... a single file with contents in memory. I'm pretty biased for Rust, but at some level isn't it a bit ridiculous if I have to break out the profiler for that? I spent like 30 minutes figuring and ultimately searching for the right combination of `move` and `async` to figure out how to give the async closures the right reference, but that code is much easier to read (for me, since I have paid the Rust tax), much more correct, and performed very well without me having to pull out a profiler.

[0]: https://github.com/kubernetes/kubernetes/issues/53533#issuec...

Do you mean the language specification is succinct or that code in the language is succinct - I thought Go favoured the former over the latter?

Edit: No implied criticism of Go intended - while not to my personal taste it seems like a fine language that a lot of people clearly get a lot of value from.

> Do you mean the language specification is succinct or that code in the language is succinct - I thought Go favoured the former over the latter?

Succinct was the wrong word -- for complex things Go is definitely not succinct in that sense you're absolutely right. It's more succinct than Java for doing what I wanted to do (waiting for someone to drop a nice tight Jersey/JAX-RS example in 3...2....1...), but what I should have written was "simple" -- Go code is simple to read and it's much harder to strangle the rest of your team (and your future self) with complexity.

> Edit: No implied criticism of Go intended - while not to my personal taste it seems like a fine language that a lot of people clearly get a lot of value from.

I would disagree on it being a fine language, but I have watched enough of rob pike and go and the team members' public talks to know that this is exactly what they intended and that's great. It's a pragmatic, powerful language with good performance, and people are deriving a TON of value from it.

To be fair I should also note that I am the exact kind of person who Go is not built for -- a yak shaver and amateur type theorist. I'd pick Go over Java in a heartbeat, but I think there are some other good choices out there if you're free to roam.

You describe yourself as a type theorist but would pick go over Java even though it lacks generics?

That doesn't seem to fit.

I’d pick Go over Java for other separate reasons than types.

If you’re on the JVM and you want good types I’d look at Scala and Kotlin and maybe clojure first before Java. Java’s type system is pretty wasteful, the marriage of structure and functions is a big mistake, first class functions didn’t land until in 1.8 IIRC, it’s got nullable types but people rarely use Option<T> as much as they should, the list is long. Go gets protocols and interfaces more right in my opinion as well. Maybe I shouldn’t blame Java for not being able to be up on PL research that wasn’t mainstream yet, but Haskell has been around since the 90s.

Java’s type system is better than Go’s (because Go’s is almost nonexistent) but it’s getting strangled with Spring soup and the unbelievable ceremony around everything in Java that makes me not want to pick it.

If you want good types period, Haskell or Rust.