ZeroFS vs. Amazon S3 Files
zerofs.net
zerofs.net
It’s been quite a ride building ZeroFS, and I’m happy to answer any questions.
Does running `stat` against a file require pulling the whole file from s3, or can that be handled by the metadata?
Do you know what backup performance is like for something like borg/borgmatic or restic, especially on follow up runs where most files are just checked.
Is there any particular Redis/Valkey config you recommend when using it for `conditional_put`, or just default config?
Is there any chance for NFSv4 support?
stat doesn’t pull the file contents from S3; it only accesses the metadata tree, which is usually cached.
I haven’t benchmarked Borg or Restic specifically. Sequential writes can comfortably reach several Gbit/s. For follow-up runs, if they only stat unchanged files, that should stay entirely in metadata.
The default Redis/Valkey configuration should work fine for conditional_put. NFSv4 is unlikely for now. It would add a lot of surface area, and I’m pretty happy with where the 9P extensions are today.
1. How are hardlinks and duplicate files (same content, different paths) handled?
2. Does deduplication work on a block/chunk level for partially matching files, or does it only look at whole files?
3. Is there any specific integration or handling for Copy-on-Write (CoW)?
There’s no deduplication, either whole-file or block-level. That’s intentional, mostly because of the impact it would have on locality.
If by CoW you mean reflinks, those aren’t currently planned either. They avoid the content matching part of deduplication, but still require sharing extents between files and come with similar locality and complexity tradeoffs. Internally ZeroFS is copy-on-write, with immutable segments and checkpoints, but that isn’t exposed as reflinks.
A) You notably don't write a recovery log (WAL/journal) for things not yet flushed, so data can be lost. Do you have plans to add this? I think it would be pretty crucial.
B) the system is single writer. Do you have plans for adding horizontal scalability so a writer can be dynamically selected and routed to, transparent to the client? (Or with client cooperation, but without forcing sharding on the user)
My usecase is NAS storage of many small files + some big files (think big shared SMB for a non-profit). I need:
- fsync durability (as promised on zerofs homepage), including sqlite durability (so NFS is out of the question)
- inotify support for external script integration (did not find an issue/docs about this)
- support for reading/writing small blocks without hammering the CPU/disks (benchmarked with JuiceFS using torrents and was catastrophic)
Do you think ZeroFS currently makes a good candidate? If not, is the described usecase part of the ZeroFS roadmap?
The reason for me for trying this setup was to provide native S3 features for apps that support it, while still supporting POSIX semantics for other apps (we have a few), while replacing native RAID with a multi-server storage backend to extend available space, and potentially removing the need for backups for most of the non-critical data once it's replicated across different sites.
I never got NFS/SMB to saturate a Gbit/s LAN link, and other solutions like cephfs/glusterfs seemed more complex and did not necessarily have all the features (inotify/fsync).
I never tried rustfs but was particularly interested in garage for the "philosophy" of development: developed by academics with proper R&D, fully FLOSS, with low-end hardware and unstable links/sites in mind, aiming for technological degrowth. From the looks at the rustfs website, it looks like the opposite of that.
You are right i was trying to overengineer things. For now i still run RAID0 (+ backups for important data).
That can literally be the stack, but the same argument could be made about object storage itself: it ultimately writes data to local disks, so why not write files there directly?
The value of object storage isn’t just its API. It’s the durability, distribution, failure handling, and capacity behind that API. ZeroFS lets POSIX applications use those properties. Its server and cache are replaceable and the bucket remains the source of truth.
If one machine and its disks are enough for your data and reliability needs, then yes, a local filesystem is simpler. ZeroFS isn’t intended to improve that case.
zerofs mount has durable fsync and fcntl byte-range locking.
Inotify works for changes made through the local mount. It won’t report changes made through another client, though.
Small writes operate on 32 KiB extents. Partial overwrites are read-modify-write, but the FUSE writeback cache helps combine them, and the resulting extents are packed into segments rather than producing one S3 PUT per write.
I’d be interested in the results if you try it.
CI also runs pjdfstest, xfstests, stress-ng, ZFS scrubs, and Jepsen crash/failover tests: https://github.com/Barre/ZeroFS#testing
ZeroFS is still pretty young as storage projects go, so I completely understand wanting to see it prove itself over time.
Opened with Safari in iOS