Hacker News Reader

Comment by for_xyz | Hacker News Reader

The problem with htop on Linux is that once there are 200+ processes running on the system htop takes significant portion of CPU time and utilization. This is because htop has to go through each process entry in procfs (open, read, close) every second and parse the text context instead of just calling appropriate syscall like on the OpenBSD and friends to retrieve such information.

It would help if kernel provided process information in binary form instead of serializing it into text. Or even better to provide specific syscalls for it like on macOS, Windows, OpenBSD, Solaris and others.

View on HN

Significant in what way? I created 400 processes + 328 threads on a 10-year-old CPU and htop is not using more than 1.3% CPU on a machine with 800% available CPU power (quad-core, 8-thread)[0]. That means 0.16% total CPU used. While I agree that it is _less_ efficient than some other ways, in what way is that _significant_?

[0] https://i.imgur.com/onNSHQw.png

Try many small or tiny processes, and soon htop overhead becomes significant (up to 5% total utilization in my situation), that's not exactly efficient.

On a 64 core / 512GB RAM server-class machine with 2K tasks, around 20K threads and 12% load, `htop` lags like crazy -- pretty much unusable.

On my system there are 240 tasks running with ~1700 threads. Htop is using 6% of single core with cgroup column disabled and 9% with it enabled. It spends most of the time in kernel space so it's not htops fault.

> While I agree that it is _less_ efficient than some other ways, in what way is that _significant_?

This gets noticeable when every connected user/session runs the htop.

I remember way back in the early years of the Third Age, I had to write a process accounting system that supported all kinds of Unices (HP-UX, AIX, Solaris, FreeBSD, Linux ...). And you're right, there's a plethora of other options other than procfs, although IIRC, Linux wasn't the only one I had to support with that variant. Wasn't Solaris one of them?

I will say, htop is a lot more efficient than GNU top, despite its functionality. I do not use it in the (default?) mode where it lists all the threads, because that is nuts.

Traditional Unix implementation of ps and similar tools work by directly reading the appropriate data structures from kernel (through /dev/kmem or something to that effect). Modern BSD implementations have libkvm.a, that abstracts that somehow, but still directly reads stuff from kernel space.

I don't know, but I don't see doing random reads from kernel memory as particularly sane API to get list of processes and procfs is several orders of magnitude cleaner solution.

Getting a list of numbers from the kernel by serialising them as text in a pretend filesystem is a several orders of magnitude cleaner solution?

> Traditional Unix implementation of ps and similar tools work by directly reading the appropriate data structures from kernel (through /dev/kmem or something to that effect).

This is not correct - /dev/kmem and similar are typically only readable by root. If what you say were correct, ps and friends wouldn't work for unprivileged users (unless they were setuid root, which they're not).

Some version of *BSDs probably fixed that with some sysctl interface, but on older Unixes you would read the kernel memory to get that system information.

You can see a ps.c implementation here: https://searchcode.com/codesearch/view/29853364/

which uses kvm library: https://www.freebsd.org/cgi/man.cgi?query=kvm&sektion=3&apro...

FWIW, in FreeBSD libkvm also uses the sysctl interface, it doesn’t read kernel memory directly.

Its not meant to be used 24/7, its just used as a guide/diagnsostic to how the machine and its processes are performing! I use it for maybe about 10 seconds.

Some people use tmux when connecting to their virtual machines on the server and forget to quit the htop instances. This adds to the total CPU utilization really quick.

Well, you are right (and top has the same problem if not worse), but I'm always surprised what these tools are doing that they take significat CPU time. Parsing 200 virtual text files every second for sure should not.

> Or even better to provide specific syscalls for it like on macOS, Windows, OpenBSD, Solaris and others.

top manages to take 6% CPU on my Macbook.

I have wondered if it made sense to re-write these top style tools using ebpf https://lwn.net/Articles/740157/ but I think ebpf requires root.

Lennart, we know it's you.

Reply on news.ycombinator.com