Hacker News Reader

Comment by huntaub | Hacker News Reader

Ah, yes — there are some specific file locking concerns with NFSv3 (notably that locks aren’t built as leases like in NFSv4). Let me do a double click here, but I know we will be able to support locks correctly with our custom protocol when we launch it by the end of the year.

View on HN

One more question. How does it handle large files that are frequently modified in arbitrary locations (like a SQLite file)? Will it only upload the "diffs" to S3? I'm guessing it doesn't have to scan the whole file to determine what's changed since it can keep track of what's "dirty".

I ask because last time I checked, S3 wouldn't let you "patch" an object. So you'd have to push the diff as separate objects and then "reconstruct" the original file client-side as different chunks are read, right?

That's correct re: the S3 API. What we do is we "merge" multiple write requests together to minimize the cost to you and the number of requests to S3. For example, if you write a file 1,000 times in the span of a minute, we would merge that into a single PutObject request to S3. Of course, we force flush the data every few minutes (even if it's being written frequently) in order to make sure that there's an up-to-date copy in S3.

I would really enjoy hearing why SMBv4 or the hundreds of other protocols are somehow insufficient for your needs. The thought of "how hard can a custom protocol be?!" makes me shudder, to say nothing of the burden -- ours and yours -- of maintaining endpoint implementations for all the bazillions of places one would want to consume a network mount

Ultimately, we're just working on a different problem space than these protocols. That's not to say that all of the existing protocols are bad, I absolutely believe that these protocols are great. Our ultimate goal, though, is to replace block storage, with a file-layer protocol. This sort of requires different semantics than what the existing file protocols support.

I don't at all disagree that it's a hard problem! That's part of what makes it so fun to work on.

Reply on news.ycombinator.com