Comment by M95D | Hacker News Reader

Comment by M95D | Hacker News Reader

I noticed this problem when I switched to Linux.

On Windows, when I copied a file, disk writes started immediately. On older systems, like Win98, I had to tweak Total Commander's disk buffer to improve copy speed on the same drive. Total Commander even had separate settings for same disk vs. different disk copy buffer sizes.

When I switched to Linux I was immediately surprised that disk writes did not start until the memory was full, and then it would stop reading while flushing dirty data. This happens even if the copy is between different drives: reads stop, writes only, then reads again with no writes to the other disk, repeat. It basically halves the copy speed.

It even happens when I copy to network mounts: reads 20 GB of data into memory, then reading stops and tries to flush the data over the nfs. Nfs times out, transfer fails. I had to use nfs timouts of 1h just to be able to do a backup.

It drives me crazy. Is there any way to make it write immediately, or at least to put a memory limit on dirty data?

https://docs.kernel.org/admin-guide/sysctl/vm.html#dirty-byt...

https://docs.kernel.org/admin-guide/sysctl/vm.html#dirty-bac...

The values are crazy high by default (on modern hardware anyway): 10% of memory for dirty_background_bytes and 20% for dirty_bytes. I wonder why no distro touches these.

Because people complain their system is "slow" if it blocks on disk I/O.

Another set of people also complain Linux takes too long to safely unplug USB drives.

When the choice is between "slow" file transfers and being unable to do file transfers because of NFS timeouts, the choice should be obvious.

I've lost data as a side effect of a simple file transfer timing out.

Just the small amount of people using NFS suggests this tunable should remain the default. Nothing is stopping sysadmins tuning for their environment.

There's no one-size-fits-all answer, which is why it's a tunable.

This isn't just about NFS timeouts. Try playing a movie from a rotational disk while simultaneously doing high-volume writes. You will get frequent pauses in your video because the write buffer size is so large that a single writeback will cause the video buffer to drain empty.

On my desktop with 32GB ram, I can even get audio to skip when ripping DVD's to disk. That's because practically the entire movie fits into ram before Linux decides to start the writeback process, and that writeback process will hog the disk for almost a minute. Or it used to, until I reduced the buffer size by a full order of magnitude.

This is just another sad example of buffer bloat: the inability to tune data buffers to the capacity of the underlying stream.

It's also because doing real IO scheduling has gotten a lot harder with fast NVME devices and blk-mq.

It's a real problem.

Have you played with "ionice"? Did it help?

That's another thing I can't understand: Why does NFS timeout when the data transfer is still on? Shouldn't it timeout only when the server is no longer ACK-ing packets?

> Shouldn't it timeout only when the server is no longer ACK-ing packets?

That's exactly what happens. The server ACKs data until it fills its write buffer, and then stalls unresponsive until the entire buffer is flushed to disk. If it takes longer to flush the buffer to disk than the client's timeout, it gives up.

I have personally watched this happen via wireshark where the server doesn't ACK for more than 10 minutes.

Holy shit i think you and the above comment, along with this thread, may have finally given me the answer to one of the few problems i was never able to solve.

About 4-5 years ago, i was working on a project, and part of that was copying big amounts of data to a system via nfs. At 30 minutes exactly, nfs would croak, transfer fails.

I think this buffer fill and empty flow was fucking killing it. Its a shame i dont work there anymore, id definitely wanna try tweaking these settings and see if i could solve it

Yeah that does sound like the symptoms of the problem I discovered. If you ever witness it again, the trick is watching /proc/meminfo for the Dirty and Writeback numbers.

And it's the vm.dirty* settings to change to fix it as described here: https://lonesysadmin.net/2013/12/22/better-linux-disk-cachin...

That's not it. I only had this problem on a fast-ethernet connection (because I had to share the cable for two connections). The server could write ~ 50 MB/s, but it still timed out on the 10MB/s upload.

It's possible you were seeing another problem, but this issue is more likely to appear with a faster network connection, because the network transfer happens faster than the disk writes.

You can confirm by watching /proc/meminfo and watching the Dirty and Writeback numbers.

Changing up the vm.dirty* settings can help as described here:

https://lonesysadmin.net/2013/12/22/better-linux-disk-cachin...

> Because people complain their system is "slow" if it blocks on disk I/O.

Yeah, I/O blocks drive me mad every time. They are more noticeable on Windows though. Perhaps that's because Windows doesn't RAM-cache enough and I do a lot of USB I/O (USB NIC, USB drives).

> Another set of people also complain Linux takes too long to safely unplug USB drives.

If only it had an APIs to see how much RAM of a specific device is RAM-cached rigth now and visualize the progress of flushing that cache... Unvisualized long I/O (incl. caching) operations, let alone those freezing the UI, indeed feel bad and are a UX bug.

One of the key reasons I prefer Linux over Windows is Linux is much more rare to freeze, no matter the workload.

> If only it had an APIs to see how much RAM of a specific device is RAM-cached rigth now

This is a typo, should have been "If only it had an APIs to see how much of a specific device is RAM-cached right now".

I've also thought it would be nice if Linux's dirty page handling was more granular. But at the same time, whenever dirty pages are a concern it's usually one large file or one NFS mount or one USB device. The system otherwise doesn't have a great deal of dirty pages to bother reporting on. Also programs have access to selective file flushing with fsync() so there is at least that.

To me this seems great and even humble. As RAM is dirt-cheap this can significantly improve performance (especially when external or a remotes drives are involved and not only NVMe SSDs), also prolong SSDs life (which means saving not just money but also the hassle of replacing them, money as well though - when it's a Mac and you can't just replace the SSD).

I wish I could configure Windows the same way: whenever it can use RAM to avoid an extra disk write/read - it should.

How does it improve performance when it won't read an write at the same time when copying from one disk to another?

In the case you describe it may indeed decrease the performance (or may not, I'm not a disk I/O or caching expert and know the things can be weird) but still may increase it in some scenarios. Copying files from one physical disk drive to another is not the only kind of operation in which RAM cache of disk I/O is involved.

Is there a max yet or per disk limits? IIRC those would start writeback. I always thought wiping a slow USB disk shouldn't consume all available RAM. But it used to. Maybe it still does.

Reply on news.ycombinator.com