Comment by lsc | Hacker News Reader

Comment by lsc | Hacker News Reader

Parent Full thread

people keep telling me to use ceph... http://ceph.newdream.net/ but it's obviously not ready for prime time. Like the article said, local storage is still the best price/performance/reliability balance. (I use raid 0+1 rather than raid6)

When distributed storage systems like ceph have been around and in production for 5-10 years, I'll re-evaluate. but for now, they present too much risk of data loss in terms of bugs and admin error.

there's also gluster, which is much closer to my own standards in terms of 'time in production' but even so... local disk is simple, and when it fails it fails in a non-spectacular way.

Of course, the other problem with distributed filesystems is that it makes having a good network /much/ more important. Hell, right now I could get away with 100Mbps, so on a gigabit network, I can have some pretty serious network issues before anyone notices anything is wrong. For a widely distributed storage network? even at my current scale, I'd at least need 10G interconnects between the switches, and god help me if there was a network glitch.

Yes I agree with your points. On our 1U boxes we run 8 drives in RAID6. On our 2U boxes we run 22 drives in RAID6 + 2 hot spares. So we use a similar approach in having hot spares on the bigger boxes.

As pointed out in the article, using distributed block storage means you need a pretty high performance storage network and low latency is just as important as high bandwidth.

We already have a physically separated storage network that runs in parallel to our public network. This is used primarily for drive traffic over iSCSI. We have a standard gigabit redundant network for this. As a physically separated network it means we don't get data integrity issues caused by DOS for example.

Moving to distributed block storage will mean we will upgrade our storage back-end to either 10Gbps Ethernet or Infiniband. The advantage of Infiniband is that as a cloud provider we will have essentially a large grid which is what Infiniband is meant for. We can also use 40Gbps per port so with dual networking can go up to 80Gbps relatively cost effectively. Just as important is the super low latency. Suffice to say we are ahead of the software on this one :-)

I think its important to point out that as storage moves to these sorts of systems it will be come increasing difficult to replicate such a setup in a redundant fashion on dedicated hardware. As a cloud provider we spread the cost over many customers.

Best wishes,

Patrick, CEO, CloudSigma

I'm very interested in how infiniband works out for you. I've considered it myself, as used infiniband switches are available for the same price as used 1g ethernet... but while I'm pretty comfortable with fixing ethernet when it breaks... I'd be much less confident in my ability to fix infiniband by swapping out hardware than I would be in my ability to fix gigabit ethernet. As far as I can tell, infiniband is something of a dying technology vs. ethernet, market wise.

On the other hand, infiniband is fucking awesome. DMA over the network, anyone?

Yes and no. The main issue we see with 10Gbps is networking topology. We don't want to roll out a star type networking which has a big single point of failure. The whole point of moving to distributed block storage is to eliminate this. Its difficult to build a grid layout with 10Gbps without going into silly money. With Infiniband it supports very well in grid configuration which is ideal for a distributed storage network. As a technology it has at least a few years left simply because it can offer 40Gbps and Ethernet won't be there at a reasonable cost for a while.

Reply on news.ycombinator.com