Hacker News Reader

Comment by hamandcheese | Hacker News Reader

I don't know if this exists or not, but I'd like to try something like a fuse filesystem which can transparently copy a file to a fast scratch SSD when it is first accessed.

I have a somewhat large zfs array and it makes consistent noise as I stream videos from it. The streaming is basically a steady trickle compared to what the array is capable of. I'd rather incur all the noise up front, as fast as possible, then continue the stream from a silent SSD.

View on HN

> I don't know if this exists or not, but I'd like to try something like a fuse filesystem which can transparently copy a file to a fast scratch SSD when it is first accessed.

You may be interested in checking out bcache[1] or bcachefs[2].

[1] https://www.kernel.org/doc/html/latest/admin-guide/bcache.ht...

[2] https://bcachefs.org/

lvm-cache works as well, if you're already using LVM.

https://github.com/45Drives/autotier is exactly what they were asking for as well

I've done some testing with using ZFS on top of LVM with dm-writecache.

Worked well enough on the small scale, but sadly haven't had the time or hardware to test it in a more production-like environment.

Also, it was starting to feel a bit like a Jenga tower, increasing the chances of bugs and other weird issues to strike.

I wouldn't recommend combining those two. It's only begging for problems.

Yeah that's my worry. Still, got 6x old 3TB disks that still work and a few spare NVMEs, so would be fun to try it for teh lulz.

Rather build a L2ARC or metadata special device hybrid setup using ZFS or skip ZFS and go for lvmcache/ mdadm style RAID with XFS or something.

But L2ARC only helps read speed. The idea with dm-writecache is to improve write speed.

I started thinking about this when considering using a SAN for the disks, so that write speed was limited by the 10GbE network I had. A local NVMe could then absorb write bursts, maintaining performance.

That said, it's not something I'd want to use in production that's for sure.

There was some work being done on writeback caching for ZFS[1], sadly it seems to have remained closed-source.

[1]: https://openzfs.org/wiki/Writeback_Cache

That's what SLOG is for if the writes are synchronous or if you have many small files/ want to optimize the metadata speed look at the metadata special device, which can also store small files of configurable size.

ZFS of course has its limits too. But in my experience I feel much more confident (re)configuring it. You can tune the real world performance well enough especially if you can utilize some of the advanced features of ZFS like snapshots/ bookmarks + zfs-send/recv for backups. Because with LVM/ XFS you can certainly hack something together which will work pretty reliably too but with ZFS it's all integrated and well tested (because it is a common use case).

As I mentioned in my other[1] post, the SLOG isn't really a write-back cache. My workloads are mostly async, so SLOG wouldn't help unless I force sync=always which isn't great either.

I love ZFS overall, it's been rock solid for me in the almost 15 years I've used it. This is just that one area where I feel could do with some improvements.

[1]: https://news.ycombinator.com/item?id=41670945

Yeah, bcache is exactly that

My assumption with bcache is that it operates on blocks rather than entire files. Am I wrong?

Bcachefs.

You can do all sorts of really neat things. You can define pools of drives at different cache levels. You can have a bunch of mechanical drives for deep storage, some for hot storage, SSD to cache recently read files, then write-through from the SSD down to mechanical drives, either immediately or after a delay.

It's pretty much everything I could wish for from a filesystem, though I haven't actually taken the time to try it out yet. AFAIK it's still somewhat experimental, more or less in beta.

Is it fully merged to the kernel? I remember a few weeks ago some drama between Kent Overstreet and Linus about it but I didn't go into details. Has that been resolved?

Edit: or maybe it was some drama over userspace tooling, I can't remember tbh.

Bcachefs is fully merged into the kernel since 6.7. The drama was around Overstreet trying to merge significant code changes to a Release Candidate kernel that should only be receiving minor bug fixes at that stage of the development process. It was developer communication issues, not anything that impacts the user. The changes will just have to wait until the next kernel version.

I think this is what I was recalling.

https://jonathancarter.org/2024/08/29/orphaning-bcachefs-too...

Bcachefs has not proven that it can be trusted to not eat data, and has also recently undergone an on-disk layout change which further demonstrates its lack of maturity.

Not aware of anything that directly matches your description, however all major operating systems do cache filesystem object in RAM. So if you pre-read the file, then it should be read back from cache when you come to stream it.

Additionally, ZFS supports using SSDs to supplement the cache.

Reminded of this -- ZFS on Apple's Fusion Drive

http://jolly.jinx.de/teclog/2012.10.31.02-fusion-drive-loose...

> I don't know if this exists or not, but I'd like to try something like a fuse filesystem which can transparently copy a file to a fast scratch SSD when it is first accessed.

ZFS has caching for writes (SLOG)[0][1] and reads (L2ARC),[2][3] which was introduced many years ago when HDDs were cheap and flash was still very, very expensive:

* https://www.brendangregg.com/blog/2009-10-08/hybrid-storage-...

[0] https://openzfs.github.io/openzfs-docs/man/master/7/zpoolcon...

[1] https://openzfs.github.io/openzfs-docs/man/master/8/zpool-cr...

[2] https://openzfs.github.io/openzfs-docs/man/master/7/zpoolcon...

[3] https://openzfs.github.io/openzfs-docs/man/master/8/zpool-ad...

I should point out that the SLOG only caches synchronous writes, which are written twice with ZFS.

Also, the L2ARC is great, but does still have RAM overhead. There are also useful tunables. I had a workload on a RAM limited machine where directory walking was common, but data reads were fairly random and a L2ARC configured for metadata only speed it up by a large amount.

You can also have a special metadata only/ small files dedicated special VDEV. ZFS can pull of many tricks if you configure it well. Of course, the L2ARC is better, if you don't trust the caching device that much (e.g. only have a single SSD).

You can also configure an L2ARC to only hold metadata, which is safer than a metadata device.

> ZFS has caching for writes

Not really.

It will accumulate synchronous writes into the ZIL, and you put the ZIL on a fast SLOG vdev. But it will only do so for a limited amount of time/space, and is not meant as a proper write-back cache but rather as a means to quickly service synchronous writes.

By default asynchronous writes do not use the ZIL, and hence SLOG vdev at all. You can force it to, but that can also be a bad idea unless you have Optane drives as you're then bottlenecked by the ZIL/SLOG.

"Fast" is meaningless.

A SLOG needs to have: good throughput and low latency at a queue depth of 1.

mpv has --cache-size (or something) that you can set at a few GB. If you run out of ram it should swap to your ssd.

Edit: demuxer-max-bytes=2147483647

It may not be 100% what you're looking for and will probably not make your drives silent while streaming but putting L2ARC on that SSD and tweaking prefetch might get you a good way there.

Another much simpler filesystem-agnostic alternative would be to copy it over to the SSD with a script and commence streaming from there. You'll have to wait for the entire file to copy for the stream to start, though. I think some streaming servers may actually support this natively if you mount /var and/or /var/tmp on the faster drive and configure it to utilize it as a "cache".

Just to test whether your OS setup etc is already up to par, try reading the whole file, eg by calculating the hash with something like `md5` (yes md5 is not secure I know.). This should put the file mostly in the os cache. But with video files being able to hit more than 50GiB in size these days, you need quite a lot of RAM to keep it all in cache. Maybe you can setting the SSD as a scratch disk for swap? I’m not sure how/if you can tweak what it is used for.

As a sibling says, ZFS should support this pretty transparently.

If you've got enough ram, you might be able to tune prefetching to prefetch the whole file? Although, I'm not sure how tunable that actually is.

Yes, I think nowadays you do this with

  blockdev --setra <num_sectors> /dev/sdX

But I feel like there was a sysctl for this too in the past. I used it back in the day to make the HDD in my laptop spin down immediately after a new song started playing in rhythmbox by setting it to 16MB.

This is essentially what macOS's Fusion Drive is/was https://en.wikipedia.org/wiki/Fusion_Drive

I'm unsure if they ship any macs with these anymore. I guess not since the Apple Sillicon iMacs don't have spinning hard drives?

   cat filename > /dev/null

Reads the entire file into the OS buffer.

add this to mpv.conf

   cache=yes                                                                                                                                                                     
   demuxer-max-bytes=5G                                                                                                                                                          
   demuxer-max-back-bytes=5G

You could use overlayfs with the upper layer on the SSD "scratch" and trigger a write operation

Depending on the file-size I wonder if it'd be better to just prefetch the entire file into RAM.

good idea, but that's not his problem. he needs a media player that will sprint through to the end of the file when he first opens it. They don't do that cuz they figure they might be streaming from the net so why tax that part of the sytem.

VLC has all sorts of prefetch settings.

With ZFS that wouldn't necessarily help. I've been using ZFS at work for many years with mostly large files. Even if I repeatedly read the same file as fast as possible, ZFS will not cache the entire thing in memory (there's no shortage of RAM so that's not it). This is unlike most linux filesystems, which are usually pretty aggressive with caching.

Maybe there is some combination of settings that will get it to cache more aggressively; just saying that it's not a given that it will do so.

The ZFS ARC cache tries to keep a balance between most frequently and least recently used data. Also, by default, ZFS ARC only fills out to a maximum of about half the available RAM. You can change that at runtime (by writing the size in bytes to

  /sys/module/zfs/parameters/zfs_arc_max

or setting the module e.g. in

  /etc/modporbe.d/zfs.conf

to something like this

  options zfs zfs_arc_max=<size in bytes>

). But be careful, as the ZFS ARC does not play that nice with the OOM killer.

Thank you, yes this is my problem which most commenters here are missing.

I'm using Jellyfin server and Infuse on Apple TV, so I don't have a great way to force-slurp a file on first read, at least not without patching Jellyfin. And I'm not super eager to learn C#.

Even if I were streaming to mpv, most of my lan is only gigabit which is much less than what my storage array can put out.

i mean you can do that with a simple wrapper

have a ssd ie mounted in /tmp/movies

and create a script in .bin/ (or whatever)

#!/bin/sh

tmp_m="/tmp/movies/$(mktemp -d)"

cp "$@" $tmp_m

mpv $tmp_m

rm $tmp_m

please note i have not tried the script but it probably works

Reply on news.ycombinator.com