Measuring the shared RAM usage of a process | Hacker News Reader

Measuring the shared RAM usage of a process

tech.brightbox.com

36 pointswlll11 comments

I wrote some code to try and do this a while ago, 'exmap': https://github.com/jbert/exmap

The basic idea was a linux kernel module which allowed you to get the underlying page id behind each VMA of any process.

Putting this info together (and hoping it wasn't changing too much as you queried different processes) allowed me to calculate "effective resident" and "effective vm size", where I used the rule that a page shared across N processes contributed pagesize/N to each one.

I could also give those same figures for each ELF section and ELF symbol (via some very fragile code which looked at the process mapping and read the ELF details to work out which bit of memory belonged to each section/symbol).

The code has long since rotted and the architecture sucked (I tied the data gathering code into the UI).

Would anyone be interested in reviving it/helping me revive it? I consider doing so from time to time, but I feel that codebase sucks now so avoid touching it.

You can do this just via /proc using "maps" and "pagemap" files on recent Linux kernels.

Nice! /proc/<pid>/pagemap is basically an alternate to my kernel module. So I could do a pure userspace solution now.

Can anyone recommend visualisation tools using pagemap? I'd like to get the exmap functionality (per-process 'effective usage' and also ELF-section/symbol resolution.

If that already exists great - if not maybe I can enhance an existing pagemap tool.

This is a visualisation tool by the guy who implemented pagemap: http://www.selenic.com/smem/

Unlike existing tools, smem can report proportional set size (PSS), which is a more meaningful representation of the amount of memory used by libraries and applications in a virtual memory system.

Because large portions of physical memory are typically shared among multiple applications, the standard measure of memory usage known as resident set size (RSS) will significantly overestimate memory usage. PSS instead measures each application's "fair share" of each shared area to give a realistic measure.

Here's some history behind it: http://lwn.net/Articles/329458/

I was a big user of exmap a few years ago at a previous employer (THALES), where exmap was used to do a lot of things to systems being tested/certified for CENELEC SIL-4 (safety critical) usage. I wrote portions of the online testing systems, and the full system tests for these components used exmap in our test systems to corrupt text segments, copy stuff where it shouldn't be, etc. exmap was essential to testing.

PM me privately your details and we can talk about what you want to do with exmap; at the very least I can explain to you how its being used in an industrial testing context, and there may be people still working there, maintaining exmap code, that would be interested in your situation, who nevertheless do not read n.yc.

As far as I know, there isn't a pm facility on HN and I don't see any contact info in your profile.

My email details are in my profile, if you have time to share some details of how exmap was/is used, I'd be grateful.

Take a look at smem

"mem is a tool that can give numerous reports on memory usage on Linux systems. Unlike existing tools, smem can report proportional set size (PSS), which is a more meaningful representation of the amount of memory used by libraries and applications in a virtual memory system. Because large portions of physical memory are typically shared among multiple applications, the standard measure of memory usage known as resident set size (RSS) will significantly overestimate memory usage. PSS instead measures each application's "fair share" of each shared area to give a realistic measure.

smem has many features:

    system overview listing
    listings by process, mapping, user
    filtering by process, mapping, or user
    configurable columns from multiple data sources
    configurable output units and percentages
    configurable headers and totals
    reading live data from /proc
    reading data snapshots from directory mirrors or compressed tarballs
    lightweight capture tool for embedded systems
    built-in chart generation"

http://www.selenic.com/smem/

Here's a python script to do much the same thing:

http://www.pixelbeat.org/scripts/ps_mem.py

This is great. One of the reasons I've always liked Phusion Passenger is because of its toolchain. They use a similar method (inspecting smaps) to report Private Dirty RSS for Apache processes and Passenger processes:

https://github.com/FooBarWidget/passenger/blob/master/bin/pa...

The Passenger utility is difficult to extracate from their package though, because it has several separate class dependencies. It's good software writing technique for a library, but it doesn't make it terribly useful for a shell script.

KDE's KSysGuard tries to do a detailed calculation much like this if you ask for it (right-click on a process in ksysguard's process list and click "Detailed Memory Status", IIRC). This process list can be pulled directly from KRunner (Alt-F2) as well.

Dianne Hackborn actually has a pretty good description of process memory usage (particularly about Android, but it applies to Linux in general): http://stackoverflow.com/questions/2298208/how-to-discover-m...

Reply on news.ycombinator.com