Hacker News Reader

Comment by jvns | Hacker News Reader

Why do you have to fork to get a memory profile? I don't know anything about memory profiling yet (I collected a bunch of links in https://gist.github.com/jvns/b81eb039f6373bc577d7dbdd978581b... but haven't read their source yet) but I'd like to know more. It seems like it should be possible to get a memory profile in production without a prohibitive amount of overhead though.

Also, I believe forking shouldn't actually double memory usage because of copy on write.

View on HN

Ruby can't tell how much memory it is using from within the process, the only way to is shell out and scrape the output of ps. I would love to be wrong about this, I did a fair amount of research and all the profiling libraries I read used this method. One thing I've done instead is use object allocation, which is available from the builtin benchmarker, as a proxy for memory usage. I don't have a good sense of how accurate that is, but anecdotally it seemed useful and correlated to stuff like page load times.

(Edit: ^happily I am wrong about this, though I believe the copy on write thing below is still correct, see further discussion)

Copy on write does definitely help here, only for Ruby > 2.0.

> the only way to is shell out and scrape the output of ps

Why couldn't you just read /proc/self/status and /proc/self/smaps in process? Would be like an order of magnitude less expensive.

Because I have a shitty knowledge of Unix :)

Best way to get the right answer on the internet is to post the wrong one.

For what it's worth, the /proc filesystem is not on all UNIX /UNIX-like OSes. I believe it originated in Linux, and FreeBSD has a procfs interface as well, although it does not have all the bells and whistles you will find in Linux. OpenBSD on the other hand, does not have procfs at all.

I don't have much experience in ruby profiling, but what you're saying doesn't make sense.

> the only way to is shell out and scrape the output of ps

All `ps` is doing is reading files in proc. The memory info it prints is in `/proc/$pid/status`. Ruby can trivially read `/proc/self/status` to get that very naive view of how much memory it's using.

That isn't a useful amount of memory because, well, it doesn't actually show where object allocations go.

> Copy on write does definitely help here, only for Ruby > 2.0.

I believe she meant the operating system's copy-on-write of memory. Forking a copy of a process is actually very cheap because it's copy-on-write. That has nothing to do with ruby or its version and has been true in linux for as long as linux has existed.

I really don't understand what you're trying to say or why you mention ps at all.

> All `ps` is doing is reading files in proc. The memory info it prints is in `/proc/$pid/status`. Ruby can trivially read `/proc/self/status` to get that very naive view of how much memory it's using.

Yep, great point, I am simply wrong about this.

> I believe she meant the operating system's copy-on-write of memory. Forking a copy of a process is actually very cheap because it's copy-on-write. That has nothing to do with ruby or its version and has been true in linux for as long as linux has existed.

Re: Copy on write the way Ruby implements GC in < 2.0 screws this up:

> The way ruby creates objects, the GC flag or reserved bit is stored in the object itself. So, as you would have guessed by now, when the GC runs in one of the processes, the GC flag would be modified in all the objects even if they are present in the shared pool. Now, the OS would sense this and trigger a copy-on-write making private copies of the objects in each child’s memory space.

https://medium.com/@rcdexta/whats-the-deal-with-ruby-gc-and-...

That is a genuine issue with fork(), but if you do fork() followed immediately by exec() then you won't ever encounter it. (The new process's memory space completely replaces the copied pages of the child.) So it's a real problem for e.g. spinning off worker processes, but not at all for e.g. calling out to a shell.

Ruby 2 was released 5 years ago, this point is kind of moot.

Run the code snippet below to prove to yourself that a Ruby process can get its memory usage.

    print File.readlines('/proc/self/status').select {|line| line =~ /VmRSS/}[0]

This is not what people mean when they talk about memory profiling, but it looked like what you were talking about, so I thought I'd share it.

As an aside, I liked your response to being wrong and would totally work with you.

Hey thanks! In that case, let me shamelessly say I'm actually hiring for iOS and Rails full stack (We are a seed stage startup working on a pretty fun consumer problem, Rails experience not necessary, admitting you are wrong experience helpful) :) Contact info in my profile

Reply on news.ycombinator.com