Hacker News Reader

Comment by jitl | Hacker News Reader

I’m very interested in this as a backing disk for SQLite/DuckDB/parquet, but I really want my cached reads to come straight from instance-local NVMe storage, and to have a way to “pin” and “unpin” some subdirectories from local cache.

Why local storage? We’re going to have multiple processes reading & writing to the files and need locking & shared memory semantics you can’t get w/ NFS. I could implement pin/unpin myself in user space by copying stuff between /mnt/magic-nfs and /mnt/instance-nvme but at that point I’d just use S3 myself.

Any thoughts about providing a custom file system or how to assemble this out of parts on top of the NFS mount?

View on HN

Hey -- I think this is something that's in-scope for our custom protocol that we're working on. I'd love to chat more about your needs to make sure that we build something that will work great for you. Would you mind shooting an email to hleath [at] regattastorage.com and we can chat more?

We're also interested in SQLite shared by multiple processes on something like Regatta but my concerns are the issues described in the SQLite documentation about NFS [1]. Notably "SQLite relies on exclusive locks for write operations, and those have been known to operate incorrectly for some network filesystems."

[1] https://sqlite.org/useovernet.html

Ah, yes — there are some specific file locking concerns with NFSv3 (notably that locks aren’t built as leases like in NFSv4). Let me do a double click here, but I know we will be able to support locks correctly with our custom protocol when we launch it by the end of the year.

One more question. How does it handle large files that are frequently modified in arbitrary locations (like a SQLite file)? Will it only upload the "diffs" to S3? I'm guessing it doesn't have to scan the whole file to determine what's changed since it can keep track of what's "dirty".

I ask because last time I checked, S3 wouldn't let you "patch" an object. So you'd have to push the diff as separate objects and then "reconstruct" the original file client-side as different chunks are read, right?

That's correct re: the S3 API. What we do is we "merge" multiple write requests together to minimize the cost to you and the number of requests to S3. For example, if you write a file 1,000 times in the span of a minute, we would merge that into a single PutObject request to S3. Of course, we force flush the data every few minutes (even if it's being written frequently) in order to make sure that there's an up-to-date copy in S3.

I would really enjoy hearing why SMBv4 or the hundreds of other protocols are somehow insufficient for your needs. The thought of "how hard can a custom protocol be?!" makes me shudder, to say nothing of the burden -- ours and yours -- of maintaining endpoint implementations for all the bazillions of places one would want to consume a network mount

Ultimately, we're just working on a different problem space than these protocols. That's not to say that all of the existing protocols are bad, I absolutely believe that these protocols are great. Our ultimate goal, though, is to replace block storage, with a file-layer protocol. This sort of requires different semantics than what the existing file protocols support.

I don't at all disagree that it's a hard problem! That's part of what makes it so fun to work on.

Reply on news.ycombinator.com