Hacker News Reader

Comment by thereallurch | Hacker News Reader

"In 2006 a 320 GB hard drive cost $120. Today (Thailand floods aside) that much money will snag you a 3 TB drive."

Floods or not, the current price isn't $120 dollars. It's 50% higher than that.

http://camelcamelcamel.com/Western-Digital-Caviar-Green-Desk...

Shows one of the cheapest 3tb non enterprise drives. It looks like 3tb was $120 for ~2 weeks. Looking at enterprise drives, 3tb is closer to $300.

This article is basically advocating RAID 5 across many storage providers.

*edit: From the pictures, article is advocating RAID 10. Nonetheless, RAID5 would be just as feasible for additional storage.

View on HN

People always conveniently forget as well that when you store data at S3 or Rackspace they're doing 3 replica's. If you store 3TB's, its not just the cost of a single 3TB drive..its 3x. To do it at home you'd need to buy 3x3TB Drives @300$ each.

That's right. The main point is that these services still benefit from the falling cost of storage.

Suppose the prices of _x_ amount of storage was halved recently. If they stored their data in three replicas, shouldn't their price become (1/2)^3, or 1/8th the original price? That would only serve to prove his point further, or I'm missing something.

provider1=[d1,1][d1,2][d1,3]

provider2=[d2,1][d2,2][d2,3]

provider3=[d3,1][d3,2][d3,3]

Each drive cost $x. You have 9$x = cost to store 1 drive of data, across 3 providers, who each store 3 copies. If drive prices halve, it's still 9$x.

I don't think anyone is proposing 3 providers, who each store 3 copies. Either use one provider who stores 3 copies (Amazon) or three providers who each store one copy.

"People always conveniently forget as well that when you store data at S3 or Rackspace they're doing 3 replica's. If you store 3TB's, its not just the cost of a single 3TB drive..its 3x. To do it at home you'd need to buy 3x3TB Drives @300$ each."

Existing companies store multiple copies transparently...Either way, 1 copy or 3 copies, I dont think the math is wrong. Just change from 9$x to 3$x.

> This article is basically advocating RAID 5 across many storage providers.

That is correct. And to be more precise, I'm advocating RAID 5 across storage providers as a service, so people who just want to store data don't have to manage anything.

>> This article is basically advocating RAID 5 across many storage providers.

> That is correct.

The diagram in the article and text description is RAID 1+0 aka RAID 10.

No, it's RAID 5. Well, specifically, it's RAID 6 but customizable. You can lose any k drives.

If this were built on top of (or in conjunction with) Openstack swift (http://swift.openstack.org), it could be done with a common storage backend. There are already several public cloud storage providers using swift (Rackspace, HP, Softlayer, Internap, KT, and others), and a growing number of private deployments as well (Wikipedia probably being the most recognizable name).

If you'd like to talk more about this, send me an email (address in profile).

Aren't most Swift providers already using 2x-3x duplication? Putting parity on top of that would be like RAID15 (the reason you've never heard of RAID15 is that it would be crazy expensive).

Yes, swift provides multiple redundant copies of the data. It depends on what you are protecting against: failure of a piece of your data or failure of a hosting provider.

If you simply rely on "dumb disks" spread across multiple providers to provide availability, then you may be interested in looking at the nova-volumes part of openstack (it provides block storage attached to nova VMs). As part of openstack, it's an open system that is seeing rapid adoption.

However, one of the most-requested features for swift is to provide support for logical clusters that span a wide geographic area. This could potentially allow multiple providers to collaborate in providing a multi-provider storage system. However, I'd guess that the technical problems are much simpler than the business problems in setting up multi-provider clusters.

Wouldn't the next issue become bandwidth? Sure, you have cheap storage, but you still need to manipulate the data somehow. I know S3 only charges for outbound data. What about the other companies? Is bandwidth free and plenty?

This is a really interesting question.

At scale, this could be managed, I think, through a combination of (a) shipping hard drives around, (b) caching, and (c) peering between storage providers. For example, shipping hard drives around would be expedient if you wanted to switch out a raw storage provider. The optimal strategy also depends on the access patterns and latency requirements.

It seems solvable, but not trivial.

Reply on news.ycombinator.com