Linux Performance Measurements using vmstat | Hacker News Reader

Linux Performance Measurements using vmstat

thomas-krenn.com

69 pointssimonreed13 comments

It would be high time for some capable & easy to use IO instrumentation in Linux. This (vmstat) has been the state of the art for ~ 20 years and you often long for something better. Not to mention it's pretty inaccurate sometimes - reports these huge bursts of IO between periods of idleness even though the disk is munching constantly. Especially with write loads.

My ideal tool would let me switch a knob to turn on io profiling and then would let me drill down into the trace, show what processes caused the io, was the io like (contiguous vs seek heavy, fsyncs, etc). A second need is report what kind of io a process incurred during its runtime, a bit like "time".

I guess much like "perf" now but for io.

For all I know all the kernel pieces are there already. We do have some newer tools like iotop/iostat that are small steps forward. But it's pretty quiet.

> show what processes caused the io

pidstat (part of Sysstat, "apt-get install sysstat" on Debian/Ubuntu) does this pretty well.

    # sudo pidstat -d 1

    11:21:54 PM       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
    11:21:55 PM       855      0.00      7.92      0.00  kjournald
    11:21:55 PM      8134      0.00     79.21      0.00  rsyslogd

    11:21:55 PM       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
    11:21:56 PM       829      0.00     28.00      0.00  kjournald
    11:21:56 PM      8134      0.00     60.00      0.00  rsyslogd

    11:21:56 PM       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
    11:21:57 PM      8134      0.00    104.00      0.00  rsyslogd

    11:21:57 PM       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
    11:21:58 PM      8134      0.00     56.00      0.00  rsyslogd

Since 2.6.something, the kernel has had per-process I/O counters.

Still, it's pretty far from DTrace's iosnoop script (on OS X, do "sudo iosnoop" to see it). Something I dearly miss on Linux, but I don't know what the situation is with SystemTap or DTrace these days.

Agreed - it's really frustrating that iotop / vmstat are basically still the top tools for debugging. strace can help, but the impact on performance really hurts. We should be taking a lead from dtrace in this space

You could also use iostat. I monitor my servers' disk i/o using iostat -xNm 1 10. The metrics await* and avgqu* are normally what I am interested in. They give the disk wait timings for serving disk i/o requests.

You also have access to the dtrace probes.

iotop can show you some of that, but doesn't let you drill down quite so far. It's a step in the right direction though.

What about sysstat that provides sar (and iostat). I have this logging every minute in some environments and I can look back and find the root cause of performance issues over multiple devices and networks. kSar can graph this quickly.

vmstat is essential in it's own way, the run queue is critical in threaded environments like weblogic with osb/soa where a default install on a 2 vCPU instance won't perform.

Also; dtrace for linux and osx.

If you're still using sar have a look at atop instead. It's great, it's like top crossed with Time Machine, you can log your system's performance like sar but then look at it like top and rewind until you find the performance problem.

Gnome and KDE have GUI apps that show info.

iotop(8) is pretty useful.

Also have a look at "dstat - versatile tool for generating system resource statistics".

It uses colors, so it's a bit easier to interpret at first sight.

It's a simple "emerge dstat" away in Gentoo.

Nice. I would just add one comment. vmstat is very useful if you run VPS. It can be used to easily detect when your hosting provider is overselling the hardware and other users are killing the I/O (which happens to be the biggest bottleneck in VPS as CPU and RAM can be easily shared and sliced)

I spent years typing vmstat into Solaris machines after reading Cockroft's "Sun Performance and Tuning" book. You would think there would be something better these days, Solaris at least moved on to dtrace. Now I use atop on Linux which isn't as low level as dtrace but is much nicer than vmstat.

On a mac, you can try: vm_stat (with an underscore)

https://developer.apple.com/library/mac/documentation/Darwin...

Reply on news.ycombinator.com