Eikon

ZeroFS [0] outperforms JuiceFS on common small file workloads [1] while only requiring S3 and no 3rd party database.

[0] https://github.com/Barre/ZeroFS

[1] https://www.zerofs.net/zerofs-vs-juicefs

ZeroFS author here.

LSMs are “for small burst workloads kept in memory”? That’s just incorrect. “Once compaction hits all bets are off” suggests a misunderstanding of what compaction is for.

“Playing with fire,” “not sure about crash consistency,” “all bets are off”

Based on what exactly? ZeroFS has well defined durability semantics, guarantees which are much stronger than local block devices. If there’s a specific correctness issue, name it.

“ZFS special vdev + ZIL is much safer”

Safer how?

For ZeroFS [0], I went an alternate route with NFS/9P. I am surprised that it’s not more common as this approach has various advantages [1] while being much more workable than fuse.

[0] https://github.com/Barre/ZeroFS

[1] https://github.com/Barre/ZeroFS?tab=readme-ov-file#why-nfs-a...

What I often do on ZeroFS [0] is to convert issues to discussions, it's a one click operation on github and helps reduce the noise for ill-defined issues.

[0] https://github.com/Barre/ZeroFS

Most cloudflare products are very slow / offer very poor performance. I was surprised by this but that’s just how it is. It basically negates any claimed performance advantage.

Durable objects, r2 as well as tunnel have been particularly poor performing in my experience. Workers has not been a great experience either.

R2 in particular has been the slowest / highest latency s3 alternative I ever had experience with, falling behind backblaze b2, wasabi and even hetzner’s object storage.

SlateDB is awesome, that’s ZeroFS [0] storage backend and it’s been great!

[0] https://github.com/Barre/ZeroFS

Unfortunately, this doesn’t support conditional writes through if-match and if-none-match [0] and thus is not compatible with ZeroFS [1].

[0] https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1052

[1] https://github.com/Barre/ZeroFS

> Keep in mind that votes aren't supposed to be about whether you agree or disagree

That’s not what happens in practice.

This has been mostly manual testing for now. ZeroFS currently lacks automatic fault injection and proper crash tests, and it’s an area I plan to focus on.

SlateDB, the lower layer, already does DST as well as fault injection though.

Thank you!

On ZeroFS [0] I am doing around 80 000 minutes a month.

A lot of it is wasted in build time though, due to a lack of appropriate caching facilities with GitHub actions.

[0] https://github.com/Barre/ZeroFS/tree/main/.github/workflows

Is there a great opensource CI system that integrates nicely with github repos?

If you use 9P or NBD it handles fsync as expected. With NFS, it's time based.

https://github.com/Barre/ZeroFS#9p-recommended-for-better-pe...

> Substring doesn't seem like what I'd want in a subdomain search.

Well, if you want only subdomains search for *.ycombinator.com.

https://www.merklemap.com/search?query=*.ycombinator.com&pag...

Glad it was helpful!

Most of merklemap is stored on ZeroFS [0] and thus allows to scale IO ressources quite crazily :)

[0] https://github.com/Barre/ZeroFS

Merklemap offers that: https://www.merklemap.com/documentation/live-tail

Shameless plug :)

https://www.merklemap.com/search?query=ycombinator.com&page=...

Entries are indexed by subdomain instead of by certificate (click an entry to see all certificates for that subdomain).

Also, you can search for any substring (that was quite the journey to implement so it's fast enough across almost 5B entries):

https://www.merklemap.com/search?query=ycombi&page=0

I am working on ZeroFS [0], a POSIX filesystem that works on top of S3.

[0] https://github.com/Barre/ZeroFS

My pleasure, this project has been a lot of fun :).

Perhaps you would find ZeroFS [0] useful. It works great out the box with SQLite [1] and only depends on S3 as an external service.

[0] https://github.com/Barre/ZeroFS

[1] https://github.com/Barre/ZeroFS?#sqlite-performance

I'm mostly interested in being able to interact with the vfs and build a 9p rust client (in addition to the current v9fs) to support my 9p server [0].

Does anyone know what's the current state / what is the current timeline for something like this to be feasible?

[0] https://github.com/Barre/ZeroFS

Shameless plug: https://github.com/Barre/ZeroFS

Mine [0] seems to be very high latency but no 500s. But yes, most cloudflare-proxied websites I tried seems to just return 500s.

[0] https://www.merklemap.com/

> (in the sense that they are not realizing their clients are down?)

Their own website seems down too https://www.cloudflare.com/

--

500 Internal Server Error

cloudflare

> What is the use case for implementing a POSIX filesystem on top of an object store?

The use case is fully stateless infrastructure: your file/database servers become disposable and interchangeable (no "pets"), because all state lives in S3. This dramatically simplifies operations, scaling, and disaster recovery, and it's cheap since S3 (or at least, S3 compatible services) storage costs are very low.

The MinIO article's criticisms don't really apply here because ZeroFS doesn't store files 1:1 to S3. It uses an LSM-tree database backed by S3, which allows it to implement proper POSIX semantics with actual performance.

I've been using Minio in ZeroFS' [0] CI (a POSIX compliant filesystem that works on top of s3). I guess I'll switch to MicroCeph [1].

[0] https://github.com/Barre/ZeroFS

[1] https://canonical-microceph.readthedocs-hosted.com/stable/

You definitely do have to worry about a schema. Except it’s ill defined and scattered across your business logic.

R2 is very high latency with huge variance, definitely lower quality than S3.

In my experience even backblaze b2 performs (way) better.

Their community forums are full of such reports.

KV is so expensive that it’s barely usable, and like R2, is very slow.

Doesn't support if-match.