Hacker News Reader

HNHacker News

Top New Best Ask Show Jobs

Comment by frabbit | Hacker News Reader

Full thread

frabbit·

Buried towards the bottom is the interesting decision to dump CephFS and go with NFS instead.

View on HN

I wonder if Amazon's new managed EFS service would make sense? It's exposed as a NFS mount to the OS. The claims are:

- Up to thousands of Amazon EC2 instances, from multiple AZs, can connect concurrently to a file system.

- Data is stored redundantly across multiple AZs.

- Low, consistent latency.

- Multiple GBs per second.

EFS is pretty great. But for our use case there are a few drawbacks:

1. It doesn't scale to the size we need for all repositories, so we still need sharding.

2. It is expensive, $300/TB-month

3. You're still sending all traffic over the network, we prefer the latency of local storage we can achieve by developing Gitaly https://gitlab.com/gitlab-org/gitaly

Why develop Gitaly instead of using consensus (etcd, zookeeper, etc.) for writes to fast storage like ssds and the normal git binary? As far as I can tell it's adding a lot of complexity by caching high level RPC calls but still doesn't really address multi-master read after write consistency or coordination.

I find that interesting as well. Furthermore, looking into deploying containers with cheap cloud hosters , I was curious how to share data across various nodes by looking at how Google and Amazon do it. Turns out it's still NFS. I've never looked into networked FS before, but I thought they'd move past NFS to something more advanced...and newer I suppose.

Fundamentally not much different, though. N shards holding the data, you have to know which shard is which and maintain connections to it. Split brains could happen if that sharding is dynamic etc.

Reply on news.ycombinator.com