Hacker News Reader

Comment by apjana | Hacker News Reader

> totally with in their right to challenge you on your claims

Yes, and within the limits of common courtesy.

The other utility does only one thing - reports disk usage so there's not much to compare. The dev did mention that `ncdu's memory usage is definitely its weak point`.

> no other performance metric, no other file system nor device types

because lstat64() is at the core of the performance metric of the feature we are comparing here and with the same number of files on the same storage device the number of accesses are exactly the same. The only metric that differentiates the utilities is memory usage.

> Edit: the “multiple contributors” point you made is also rather dishonest too.

Not really, I prefer to edit the readme myself because I want to keep the documentation clean. You will see features contributed by other devs for which I have written the docs from readme to manpage. Regarding the metrics, sometimes I have taken the data and sometimes I have requested someone else to collect it. Or doesn't that count as contribution?

View on HN

What I actually care most about a file manager is how they perform on mounts with low IOPS and how gracefully they handle time outs and other exceptions.

RAM is cheap and any file manager will be snappy on an SSD. But edge cases are where most file managers fall apart yet are situations where you might need to depend on your tools the most.

However now I understand the point of this project was purely to optimise against memory usage, I can better understand the arguments you were making.

> or doesn't that count as contribution?

Not in this case, no. You published it, so you’re still ultimately accountable for it.

You cannot request figures then play the “nothing to do with me guv’” card when someone queries the benchmarks that you subsequently published. At best it comes across as an unlikely story; at worst you’re still complicit.

>RAM is cheap

This is the wrong mindset. RAM is only cheap if you don't use it. As soon as you go just 1 byte over the maximum RAM it turns into the most precious resource of the entire computer.

If an app uses more memory than another then it is not better because RAM is cheap. It is better because it provides more or higher quality features at a reasonable cost of increased memory usage. But at the same time it is also worse for people who do not need those features.

Here is an example: When I launch the graphical file manager nautlius it consumes roughly 26 MB of RAM showing my home folder but when I go to my "Pictures" folder it suddenly shoots up to 300MB. There is a reason for that and it is not "RAM is cheap", if that were the case it would always use 300MB regardless of what I do with it (electron apps are a major offender of this). Nautlis consumes that much RAM because it has more features like displaying 1000 thumbnails of all those pictures.

Now this feature would get in my way if I set up something like a Raspberry Pi Zero to take a photo every hour. Nautilus will crash because it needs too much memory to display the thumbnails.

I agree from an idealistic point of view (I've often make the same argument myself with regards to non-native applications and self-indulgent GUIs) but you're missing the context of the argument here.

We're not talking about burning through hundreds of megabytes (nor even gigs) on a pretty UI that adds no extra function; we are talking about only a few megabytes to save stressing low bandwidth endpoints.

It isn't 1990 any more, sacrificing a few megabytes of RAM in favour of greater stability is very much a worthwhile gain in my opinion. Hence why I say RAM is cheap - we don't need to cut every corner just to save a few bytes here and there.

In the example we were discussing, the idle RAM usage was higher because it doesn't hammer low bandwidth endpoints with frequent rescans. Caching is actually a pretty usage for spare RAM - your kernel does it too. So we are not talking about anything out of the ordinary here and we're certainly not talking about wasting RAM for the sake of wasting RAM. We're talking about trading some cheap RAM for the sake of potentially expensive (computationally speaking) access times.

However I do feel I need to reiterate a point I've said throughout this discussion: there is no right or wrong way; it's just a matter of compromise. eg the goals for an embedded system would be different to the goals for an CLI tool on a developers laptop.

> then play the “nothing to do with me guv’” card when someone queries the benchmarks that you subsequently published

No, you are cooking things up. I did respond as per my understanding. My statement was very clear - "_Please don't judge me_ by the project page of a utility which is a work of several contributors."

I had problems with the _personal remarks_. And I am not surprised you chose to ignore that and describe it in the light that I have problem with someone challenging the benchmarks.

I am yet to come across figures that can challenge the current one.

I have no idea how people convince themselves to contribute to open source and/or participate regularly in online discourse. It’s basically working really hard for the easiest-to-offend and least-likely-to-appreciate-anything-you-do people on earth, so they can throw shade at everything you do, and then get mad because you did something different than they would have done.

FWIW, I appreciate everything you all are doing, even the stuff I’m not using right now. You all don’t hear it enough, and you’re certainly not getting paid enough for what you’re doing.

I contribute loads to open source too.

It’s actually not that hard to do so without talking trash about other projects. In fact I find those kind of comparisons are often the laziest and least researched ways of promoting a particular project as anyone who’s spent any time dissecting other peoples work in detail (not just running ‘ps’ in another TTY) will usually gain an appreciation for the hard work and design decisions that have gone into competing solutions.

But that’s just the opinion of one guy who has been contributing to the open source community for the last two decades. ;)

> talking trash about other projects

Cleverly fabricated and blown out of proportions. Yes, 2 decades of rich experience sure teaches that!

To clear the context for others, you are talking about a list of performance numbers here, and as I said, I am yet to come across figures that can challenge the current one.

The author of the other project did challenge them though. Hence this entire thread ;)

Yes, and he has received pointers on how he can reduce memory usage in his program (an issue he mentioned exists).

Actually what he said was he knew memory was a problem but it is designed for a different use case. The key part being the bit you’ve conveniently left off.

Your comments come across as very arrogant (particularly considering you’ve not reviewed the other projects source code) and intentionally deceptive too. While I’m normally the first to defend open source developers - because I know first hand how thankless it can be at times - you’re doing exactly to others as you ask not to be done to yourself. Which is not fair at all.

Anyhow, this is all a sorry distraction away from what should have been the real point of this HN submission; and I’m angry at myself for allowing myself to get dragged into this whole sorry affair. so I’m going to log off HN for the rest of the day to avoid any further temptation.

I do sincerely wish you the best with this and your other projects. It should be the code that speaks for itself rather than any other politics and that’s not been the case here for either your project nor the others it has been compared against. But let’s move on and get back to writing code :)

> Actually what he said was he knew memory was a problem but it is designed for a different use case. The key part being the bit you’ve conveniently left off.

No, read the other thread where I said his use case can be satisfied at much less memory usage. There's nothing arrogant in suggesting there's a more efficient way to achieve something.

Yes, let's get back to work. Loads of feature requests to cater to, thanks to this share. :)

> What I actually care most about a file manager is how they perform on mounts with low IOPS

Purely technical questions (let's put `being very snarky` and `dishonesty` and my other irrelevant personal traits aside):

- How does "low IOPS" affect readdir()/scandir() and lstat64() in two C utilities differently?

- What else would be affected?

It’s about how and when they get used. Eg if you’re running on a lower performing mount then you might want to rely on caches more than frequent rescans. What I would often do on performance critical routines running against network mounts was store a tree of inodes and time stamps and only rescan a directory if the parents time stamp changed. It meant I’d miss any subsequent metadata changes to individual files (mtime, size, etc) but I would capture any new files being created. Which is the main thing I cared about so that was the trade off I was willing to make.

There’s no right or wrong answer here though. Just different designs for different problems. Which was also the point the other developer was making when he was talking about his memory usage.

> There’s no right or wrong answer here though.

The processor cache plays an important role which you are ignoring.

External storage devices: most of the time they are write-back and even equipped with read-ahead. Yes, I know there are some exceptions but if you are write-through non-read-ahead you _chose_ to be slow in your feedback already and this discussion doesn't even apply.

Network mounts: cache coherency rules apply to CIFS as well. And again, if you _choose_ to ignore/disable, you are OK to be slow and this discussion does not apply.

If `nnn` take n secs the first time, another utility will take around the same time on the first startup (from a cold boot).

Now the next scans where you go into subdirs would be much faster even in `nnn` due to locality of caching of the information about the files (try it out). The CPU cache already does an excellent job here. And if you go up, both `nnn` and the other utility would rescan.

> point the other developer was making

Yes, he was saying - my memory usage may be 15 times higher because of storing all filenames (in a static snapshot!!!) but you are dishonest if you show the numbers from `top` output without reading my code first for an education of my utility.

I’m not sure what you mean by processor cache here. The processor wouldn’t cache file system meta data. Kernels will, but that is largely dependant on the respective file system driver (eg none of the hobby projects I’ve written in FUSE had any caching).

Different write modes on external hardware also confuses the issue because you still have the slower bus speeds (eg a USB2 for an older memory stick) to and from the external device than you might have with dedicated internal hardware.

> The processor wouldn’t cache file system meta data

What _file system meta data_? The processor doesn't care what data it is! I am talking of the hw control plane and you are still lurching at pure software constructs like the kernel and its drivers.

All the CPU cares is the latest data fetched by a program. The CPU cache exists to store program instructions and _data_ (no matter where it comes from) used repeatedly in the operation of programs or information that the CPU is likely to need next. If the data is still available in the cacheline and isn't invalidated, CPU won't fetch it from an external unit (so bus request is not even made). _Any_ data coming to CPU sits in any Ln cache, source notwithstanding. The external memory is accessed in case of cache misses. However, the metadata these utilities fetch is very very less and the probability is greatly reduced. Moreover, your hypothetical utility also banks on the assumption that this data won't change and it wouldn't have to issue too many rescans to remain performant.

It's the same thing you see when you copy a 2 GB file from a slow storage to SSD and the first time it's slow but the next time it's way faster.

You can see it for yourself. Run `nnn` on any external disk you are having (with a lot of data preferably), navigate to the mounpoint, press `^J` (1. notice the time taken), move to a subdir, come back to mountpoint again (2. notice the time taken). You would see what I mean.

> none of the hobby projects I’ve written in FUSE had any caching

On a side note (and though not much relevant here), all serious drivers (e.g. those from Tuxera, btrfs) maintain buffer cache (https://www.tldp.org/LDP/sag/html/buffer-cache.html). They always boost performance. If our Ln misses, this is where we would get the metadata from and _hopefully_ not from the disk which is the worst case.

Yeah, that’s the kernel caching that (as I described) not some hardware specific thing the CPU is doing. Not disagreeing with you that L1 and L2 cache does exist on the CPU (and L3 in some instances) but it is much too small to hold the kind of data you’re suggesting. It’s really the kernel freeable memory (or file system driver - eg in the case if ZFS) where file system data - inc file contents too - will be cached. The CPU cache is much more valuable for application data to fill with file system meta data (in my personal opinion, you might override that behaviour in nnn but I’d hope not)

However regardless of where that cache is, it’s volatile, it’s freeable. And thus you cannot guarantee it will be there when you need it. Particularly on systems with less system memory (remember that’s one of your target demographics).

If you wanted though, you could easily check if a particular file is cached and if not, perform the rescan. I can’t recall the APIs to do that off hand but it would be platform specific and not all encompassing (eg ZFS cache is separate to the kernels cache on Linux and FreeBSD) so it wouldn’t be particularly reliable nor portable. Plus depending on the syscall used, you might find it’s more overhead than an actual refresh on all but a rare subset of edge cases. As an aside, this is why I would build my own cache. Sure it cost more RAM but it was cheaper in the long run - fewer syscalls, less kernel/user space memory swapping, easier to track what is in cache and what is not, etc. But obviously the cost is I lose precision with regards to when a cache goes stale.

While on the topic of stale caches, I’ve actually ran into issues on SunOS (or was it Linux? I used a lot of platforms back then) where contents on an NFS volume could be updated on one host but various other connected machines wouldn’t see the updated file without doing a ‘touch’ on that file from those machines. So stale caches is something that can affect the host as well.

> but it is much too small to hold the kind of data you’re suggesting

My toy has a 3 MB L1 cache, 8 MB L2. You'll notice from the same figures that the resident memory usage of `nnn` was 3616 KB. And in the default start (not du) right now the figure is:

      PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                
    23034 vaio      20   0   14692   3256   2472 S   0.0  0.0   0:00.00 nnn -c 1 -i

So it required around 350 KB for `du`. Oh yes, it can be cached at will.

> check if a particular file is cached and if not

I am a userland utility. I don't add deps on fs drivers (and maintain redundant code... I discourage code bloat as much as feature bloat) when standard APIs are available to read, write, stat.

> However regardless of where that cache is, it’s volatile, it’s freeable.

Yes!!! The worst case! And the worst case for you is where all dirs have changed. There you issue rescan, `nnn` issues rescan. `nnn` is accurate because it doesn't omit independent files for _personal_ optimization preferences. You do. You show _stale_, _wrong_ info in the worst case.

3MB is not enough to store the kind of data you’re talking about. In your example you were coping files from an SSD to a removable device; what happens if that file is 4MB big? Or 20MB?

Some program executables are multiple megabytes in size these days, your CPU cache is much more valuable storing that than it is storing a random 20MB XLSX file you happened to copy half an hour earlier. ;)

> You do. You show wrong info in the worst case.

Yes, I’d repeatedly said that myself. However I was making a case study rather than giving a lecture on how your tool should be written. Remember when I said “there is no right or wrong answer”? (You have ignored my point that you cannot guarantee the kernels / fs drivers cache isn’t also stale though. That does happen in some rare edge cases).

My point is you shouldn’t be so certain that you’re method is the best because there are plenty of scenarios when accuracy and low memory usage is more costly (eg the fastest file copying tools are those that buffer data - costing more RAM during usage). Plus let’s not forget that syscalls are expensive too. But that’s the trade off you take and that is what is advantageous for the perpose you’re targeting nnn for.

So there really isn’t a right or wrong answer here at all and memory usage is in fact only a very small part of the equation (which is what the other projects author what trying to say).

> Some program executables are multiple megabytes in size these days

I have shared the figures from my system right now in my earlier comment. The memory required for the meta info is extremely cache-able.

And yes, let's call it a day now. In fact, I am happy we had a very informative discussion towards the end. This is what I expected from the other dev and you from the very beginning. I am always ready for a productive non-abusive technical discussion. And if I find someday `ncdu` takes comparable/lesser memory than `nnn` I will update that data verbatim as well. But I wouldn't take uncalled-for shitstorm in public forums from strangers lying low.

But that’s what you’re storing in the system memory, not CPU cache. Ok let’s say the kernel does cache recently accessed memeory in the L2 cache, you’re still competing with everything else - particularly in a multi-user or multi-threaded system. So that is not going to stay in L2 for long.

You’re making so many assumptions about the host platform and they simply don’t stack up for the majority of cases - it’s just that most systems are fast enough to hide your incorrect assumptions.

Also your understanding of how file system caching works is completely off. Granted that’s not something most normal engineers need to worry about, but given the claims you’re making about nnn, I would suggest you spend a little time researching the fundamentals here.

Edit:

> I am happy we had a very informative discussion towards the end. This is what I expected from the other dev

That was exactly what the other dev was doing. You just weren’t ready to listen to him - which is why I started making comments about your “narky” replies. ;)

Well, I shared the numbers, I gave you ways to test it. We are done here.

but You’re not proving that the data you are expecting to be in L2 actually ends up in L2. Let alone persists long enough to be recalled from L2 when you do your rescans. Your test is crude and thus doesn’t prove anything. Which is what we have all been telling you right from the start!!!!

You cannot just throw numbers up and say “look I’m right” if the test of flawed from the outset.

But yes, it’s probably better we do call it a day.

I expected you would run the test. Anyway, here's the data and it aligns with my expectation and figures.

  ## Fired `nnn` in my home dir.
     - all cached, no major page faults
  
  ~$ /usr/bin/time -v nnn
  	Command being timed: "nnn"
  	User time (seconds): 0.00
  	System time (seconds): 0.00
  	Percent of CPU this job got: 0%
  	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:03.77
  	Average shared text size (kbytes): 0
  	Average unshared data size (kbytes): 0
  	Average stack size (kbytes): 0
  	Average total size (kbytes): 0
  	Maximum resident set size (kbytes): 3400
  	Average resident set size (kbytes): 0
  	Major (requiring I/O) page faults: 0
  	Minor (reclaiming a frame) page faults: 274
  	Voluntary context switches: 6
  	Involuntary context switches: 1
  	Swaps: 0
  	File system inputs: 0
  	File system outputs: 0
  	Socket messages sent: 0
  	Socket messages received: 0
  	Signals delivered: 0
  	Page size (bytes): 4096
  	Exit status: 0
  ~$
  
  ## Fired `nnn` on an external disk root in du mode, went into a subdir, came back to root.
     - no major page faults
     - File system inputs: 14680
     - 220 KB cached extra
  
  ~$ /usr/bin/time -v nnn -S /media/vaio/49462fdf-010c-40cc-89db-ef125e7dae99/tmp/
  	Command being timed: "nnn -S /media/vaio/49462fdf-010c-40cc-89db-ef125e7dae99/tmp/"
  	User time (seconds): 0.02
  	System time (seconds): 0.27
  	Percent of CPU this job got: 3%
  	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:08.30
  	Average shared text size (kbytes): 0
  	Average unshared data size (kbytes): 0
  	Average stack size (kbytes): 0
  	Average total size (kbytes): 0
  	Maximum resident set size (kbytes): 3620
  	Average resident set size (kbytes): 0
  	Major (requiring I/O) page faults: 0
  	Minor (reclaiming a frame) page faults: 337
  	Voluntary context switches: 576
  	Involuntary context switches: 2
  	Swaps: 0
  	File system inputs: 14680
  	File system outputs: 0
  	Socket messages sent: 0
  	Socket messages received: 0
  	Signals delivered: 0
  	Page size (bytes): 4096
  	Exit status: 0
  ~$
  
  ## Fired `nnn` again (around 2 mins later, I was formatting the above text) on the external disk root in du mode.
     - File system inputs: 0
  
  ** So even between 2 distinct instances of nnn, _all_ the data fetched in the first instance remained cached.
  
  ~$ /usr/bin/time -v nnn -S /media/vaio/49462fdf-010c-40cc-89db-ef125e7dae99/tmp/
  	Command being timed: "nnn -S /media/vaio/49462fdf-010c-40cc-89db-ef125e7dae99/tmp/"
  	User time (seconds): 0.00
  	System time (seconds): 0.01
  	Percent of CPU this job got: 0%
  	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.31
  	Average shared text size (kbytes): 0
  	Average unshared data size (kbytes): 0
  	Average stack size (kbytes): 0
  	Average total size (kbytes): 0
  	Maximum resident set size (kbytes): 3616
  	Average resident set size (kbytes): 0
  	Major (requiring I/O) page faults: 0
  	Minor (reclaiming a frame) page faults: 329
  	Voluntary context switches: 5
  	Involuntary context switches: 0
  	Swaps: 0
  	File system inputs: 0
  	File system outputs: 0
  	Socket messages sent: 0
  	Socket messages received: 0
  	Signals delivered: 0
  	Page size (bytes): 4096
  	Exit status: 0
  ~$

Yes, I know there is a cache. I went into great detail about it. That wasn’t what I asked you to prove.

What we were arguing about was the L2 cache - which you argued could hold a 2GB file (insanity I call it) and you’re still not proving that point of yours. In fact you’re not even checking if it’s cached in the system memory (there are syscalls to do that), you’re just checking application performance. While it’s pretty obvious that result would have been due to caching it would have been easy to prove that more conclusively by even just flushing the cache (I assume you actually know how to do that? Hint: you can just write to the /proc directory from your shell to do that. It’s easy) and then repeating the test; but You didn’t even do that!

You’re testing is so rudimentary at covering the cases you’re making claims against it’s laughable. you’ve not even begun to address the “processor memory” point you were arguing.

But I honestly think we should just leave it here. You don’t need to be an expert in kernel design nor practical experience in write file systems to write a decent file manager, so all of this arguing is pointless. I just ask that you aren’t so quick to judge other solutions when it’s painfully obvious there are massive gaps in your own understanding.

I think you are referring to where I mentioned a large file copied second time. That is cached in RAM (I have 6GB) and not in L2. I think it's pretty clear from the context that I was emphasizing disk IO is not involved which was your main point of concern.

- which you argued could hold a 2GB file

No I didn't say that! I've always meant the metadata (that nnn needs) cache which I proved above. Clearly there's some misunderstanding.

Reply on news.ycombinator.com