Hacker News Reader

Comment by garganzol | Hacker News Reader

I used the same approach based on Rclone for a long time. I wondered what makes Regatta Storage different than Rclone. Here is the answer: "When performing mutating operations on the file system (including writes, renames, and directory changes), Regatta first stages this data on its high-speed caching layer to provide strong consistency to other file clients." [0].

Rclone, on the contrary, has no layer that would guarantee consistency among parallel clients.

[0] https://docs.regattastorage.com/details/architecture#overvie...

View on HN

This is exactly right, and something that we think is particularly important for applications that care about data consistency. Often times, we see that customers want to be able to quickly hand off tasks from one instance to another which can be incredibly complex if you don't have guarantees that your new operations will be seen by the second instance!

Might be useful to show the differences with Rclone, s3fs as a table to make it obvious

I agree, I plan to put up a table soon.

The headline seems misleading, then.

rclone can work with AWS' different offerings, some of which at least partially address this: https://aws.amazon.com/blogs/aws/new-amazon-s3-express-one-z...

I'm not totally sure what you mean. I don't think that S3 Express One Zone offers any additional atomic semantics in the file system world.

For the misleading part, I probably should have said confusing because I don't think you intended that, I mean that instead of introducing your caching layer you make it about S3, where the Object Storage provider seems totally interchangeable. Though it seems to work for a lot of your audience, from what I can tell from other comments here.

As for Express One Zone providing consistency, it would make more groups of operations consistent, provided that the clients could access the endpoints with low latency. It wouldn't be a guarantee but it would be practical for some applications. It depends on what the problem is - for instance, do you want someone to never see noticeably stale data? I can definitely see that happening with Express One Zone if it's as described.

Yes, I think this is something that I’m actually struggling with. What’s the most exciting part for users? Is it the fact that we’re building a super fast file system or is it that we have this synchronization to S3? Ultimately, there just isn’t space for it all — but I appreciate the feedback.

I think they both go together. It might take about 10 minutes to give a good high level explanation of it, including how the S3 syncing works - that the S3 lags slightly behind the caching layer for reads, and that you can still write to S3. 2-way sync. I imagine that S3 would be treated sort of like another client if updates came from S3 and the clients at the same time. It would probably be not so great to write to S3 if you aren't writing to somewhere that's being actively edited, but if you want to write to a dormant area of S3 directly, that's fine.

Thanks, this was my thought as well. I use and love rclone and it wasn't immediately clear what this offered above that

I suppose rclone doesn't provide byte range file locking? Running sqlite over rclone would be a disaster.

That would be my expectation, you need something in the middle to actually broker the file locks.

Running sqlite over rclone is not a disaster as long as you run only a single instance working with that database. Rclone provides no support for locking semantics.

Reply on news.ycombinator.com