Wddbfs – Mount a SQLite database as a filesystem

If I want the 50,004,123,121th byte of a file from a disk, that's very fast. If I want the same for a virtual object from an HTTP server, object store, virtual table, etc. it literally involves creating the whole object, and stepping through it byte-by-byte until I get there.

If the next request is doing the same 1k ahead, on a disk, that's probably in cache, and if not, I can get there quickly. If this was a SQL query, I probably need to redo the whole thing.

You get a natural explosion from O(1) to O(n) in many common cases, and for something like a complex SQL query, it can be much, much worse.

FUSE is much, much easier to work with. The protocol (even v4) should also be less chatty, and if performance is your goal there are projects like ExtFuse that can move some caching into the kernel via eBPF which I don't think would be possible with NFS.

If you poke around other user space file systems (sshfs, cephfs, objectivefs, etc) they're all FUSE.

I just can't decide if you're a sci-fi supervillain, a political thriller supervillain, or a Despicable Me supervillain.

;-)

For your model to work, what would be needed is the reverse: a system which allows me to interact with TSV files with SQL.

For 98% of uses of SQLite, performance and storage efficiency don't matter; we're talking about dozens or hundreds of records. It doesn't need to be well-implemented from that perspective. It does need to be robust. For example, databases solve a lot of problems especially around consistency in threaded systems which this would need to handle as well (although a KISS solution, such as handling all access from one thread / process, and simply blocking other requests during transactions, would be fine).

Something like that would likely eat SQLite's lunch.

q seems close. It'd need to support writes, table creation, and all the other stuff, perhaps be more actively maintained, provide documented API (not just command-line) access, and work with things like ORMs.

Yes, s3fs and sshfs can both leave the system in an unstable state. For example, there can be a dead mount which is impossible to unmount, and in severe cases, blocks a clean reboot.

A file system in user space (or in network space) should NEVER break the system, no matter what happens in user space (or in network space). Most network file systems try to respect this (albeit with mixed success). FUSE does not.

I'm not claiming FUSE cannot be made to work. Just that it's very bad since (1) plenty of smart people clearly failed to do so (2) the badness it leaves behind should be more than it's permitted to.

I can point to specific issues, but at the end of the day, that's neither here nor there. At the end of the day, something like:

https://github.com/libfuse/libfuse/blob/master/example/poll....

Is about a hundred times more complicated than it should be. It should not require memsets, pthread mutexes, or flags, and should probably have an implementation in a modern, high-level language. To a large extent, that's the point of moving things out of the kernel.

I'm even perhaps okay with being permitted to do low-level operations for a particularly performance-constrained subsystem, but that's not 95% of the uses of something like FUSE.

Footnote: I actually enjoyed writing low-level code like this a lot, when computers were in the single-digit to triple-digit MHz range, and we didn't need to worry about people breaking in over a ubiquitous worldwide internet, but I left that mindset behind decades ago. Right now, I want code to be stable, simple, auditable, and secure.