Hacker News Reader

Comment by edw519 | Hacker News Reader

I always thought it was best to focus on what I did best (application software) and leave the infostructure to others. Until I saw this:

  Raw Drives      $81,000
  Backblaze      $117,000
  Dell           $826,000
  Sun          $1,000,000
  NetApp       $1,714,000
  Amazon       $2,806,000
  EMC          $2,860,000

I had no idea. Kinda makes one rethink what business they want to be in.

View on HN

  Backblaze      $117,000
  [...]
  Amazon       $2,806,000

I cry foul. Backblaze's "67 TB" pods actually only hold 58.5 TB, so their hardware cost per PB of storage is $134k, not $117k; and that's without any high-level redundancy. Servers fail -- both catastrophically, and by silently corrupting bits -- and Backblaze's $134k / PB doesn't have any protection against that. Datacenters also fail -- power outages, cut fibre, FBI raids, etc. -- and any system which stores all of its data in a single datacenter lacks any protection against that. Store each file on two different servers in each of two different datecenters, and suddenly Backblaze's $134k turns into $536k. The price for Amazon, in contrast, is based on the assumption that their prices remain fixed for the next 3 years -- which seems a rather radical assumption.

Is backblaze's solution cheaper than S3? Absolutely. But they're also twisting the numbers a bit.

Interesting, but Amazon doesn't guarantee most of those things. They guarantee a 99.99% uptime, but that isn't counting a complete datacenter failure. In fact, it sounds to me as if they have a similar set up to the backblaze people.

99.99% uptime means roughly 1 hour per year of downtime. I don't know what the specific failure rates on the components are, but it seems reasonable that A) the data drives are hot-swappable, and will not cause downtime when they are replaced and B) the rest of the components fail once a year (or less) and take ~10 minutes to replace and reboot the system. With 4 main places of failure (PSU, Boot Drive, Motherboard/Ram/CPU, Drive controllers), as long as you have staff constantly on call and they can respond within 5 minutes of a failure, 99.99% uptime seems reasonable.

I don't know where you got the idea that your data was on 4 different servers when using S3. I can't find even the slightest amount of information on that. Yes, that would be nice, and it is cool to think that, but its rather doubtful that they're actually doing that (or they could probably add another 9 to their uptime).

>but that isn't counting a complete datacenter failure

I believe that S3 replicates to multiple physical locations. So, while you might experience downtime, you probably won't experience data loss.

Re. geographical redundancy, see comment http://news.ycombinator.com/item?id=422574 :

"Amazon keeps at least 3 copies of your data (which is what you need for high reliability) in at least 2 different geographical locations. "

I can't find the original source supporting that statement, but I also know it to be true based on direct contact with AWS team. (I'm using AWS since private alpha of EC2 in 2005)

I can't find that anywhere either, and would be interested in seing it. In fact, all I can find is: A bucket can be located in the United States or in Europe. All objects within the bucket will be stored in the bucket’s location, but the objects can be accessed from anywhere. which seem to imply that your data is located in one location, not 2.

In addition, looking at the actual S3 contract, they're really only guaranteeing at 99.9% uptime, which allows for up to 8 hours of downtime a year, more than enough to completely re-build the outer server once a year, as long as they can keep the data intact (which they seem more than capable of with their setup, once again assuming their data center is not completely destroyed).

It really depends on what they ascertained their needs as. Perhaps they don't feel they need the additional protection, or they have another backup scheme in mind. If their solution works for them (which is seems to) then indeed, for their needs at least, the cost comparison is pretty accurate.

It might not add up the same for your needs (assuming you want multi-data center redundancy of all data, etc...).

That's just the cost of hardware. How much more do you think it would cost if they included all their research and development, installation and assembly hours, etc?

That is just the cost to buy all the components they have. If you budgeted that in you'd just have a bunch of boxes at your office that weren't even wired up. On the other hand, from Dell or Sun you at least have all the hardware in the chassis, if not some basic configuration and OS install done for you. Go with Amazon and you've already got it racked up in a data center with dedicated techs and replacement parts on hand.

It isn't a fair comparison.

So you're currently paying $800k+ per box to install Solaris?

Tell you want, since you're a fellow HNer, I'll save you some money and do it for half that. I'll even throw in some basic configuration. You're welcome!

Another thing that they aren't taking into account over S3 is power costs, physical datacenter rent costs, and bandwith hookup. Still its probably a lot cheaper but important to note that they are comparing apples and apple cider.

On their graph it says they subtracted data center costs from the price of S3.

what does s3 charge for transfer? $0.10 per gigabyte? I bet that covers much of the data center costs.

Well, to be fair, Backblaze is using consumer-grade hardware here. No redundant PSUs. No mirrored OS drives. Cheaper components. Etc.

They're going Google's route, which is fine, it's just... a whole different direction than a Sun Thumper. I'd be interested to see how their costs per petabyte stack against Google's.

No, they aren't going Google's route!

Google's operation has at most half the disks-per-rack density of Backblaze, but with 4 or 8 times the server density. Google does almost all their computation on the same nodes where the data is stored. Google is also storing each tablet a minimum of 3 times within each cluster, with most systems having multiple replica clusters.

Backblaze's system is going to have several orders of magnitude less bandwidth:

  way too many drives in one server (some of them on the PCI bus!)
  use of port multipliers (causing 5 drives to share one SATA cable's 300MB/s)
  RAID6 with too many drives per array (15-way XOR is no fun)
  why use JFS?
  only access is via HTTPS, not clear if SSL is done on the pod

I'm guessing that backups are accessed rather infrequently and that someone recovering their data won't mind a few seconds in lag time. This isn't my area of expertise, but their trade off seems sensible. It's just a different scenario than what google is dealing with.

As a sysadmin for organizations of different sizes, I've been acutely aware of this for quite some time.

The more painful corollary is that, even if one measures not by total storage but by performance (a more critical aspect of, e.g., databases), the prices are still an order of magnitude or two apart.

I would argue that the reason "No One Sells Cheap Storage" in the backup/archive sense is that there isn't enough demand. Obviously, this is starting to change.

I am, however, startled that a driveless pod is so expensive ($2467 or $54.82 per drive), considering that external performance is, effectively, limited by gigabit Ethernet.

Two SuperMicro 846E1s would be $2k for 48 drive bays, and one could easily connect 10 of them to a single $300 SAS RAID card. Add another $1k for mobo and cables, and that's $11300 for 240 drives or $47.08 per drive, without having to do a custom case.

Granted, it takes up almost twice the rack space, but it's still a decent power density at 50W/U AC (assuming 6.5W per drive DC).

It takes 52 groups of 13 1.5TB disks (Hmm.. 13 pops up an awful lot in their setup) to make a PB using their pods. 51 groups means 17 pods is .9945TB for $134k ($134.5/PB).

Using my pods, it would be .936TB for $120.3k ($128.5/PB), but what's $6k/PB between friends?

you might also want to check out coraid. it's at a similar price point. http://www.coraid.com/PRODUCTS/SR2421 - now, I've heard mixed experiences with it, and have no personal experience, but if you are talking about wiring up something yourself, it's definitely something to consider.

Personally, I'm working on a system whereby I just fill up otherwise unused slots on my chassis and use that for backup. The system isn't done yet, but I've got 6 disks out there waiting for me to figure it out.

Hmm.. looks suspiciously like exactly the SuperMicro case, for about twice the price of rolling ones own.

from what I understand the coraid systems are plan9-based, and only have a gigabyte of cache ram.

If I only needed one I'd seriously consider it, as my time, for setting up one unit and working out the kinks, would unquestionably be more than that, and they have quite a bit of experience working out the problems by now, even if I made my system redundant, their system would unquestionably be more reliable the first iteration.

However, I plan on needing a bunch more, so it probably makes sense for me to roll my own, work out the kinks, and hopefully end up with something cheaper that has a lot more cache.

Two SuperMicro 846E1s would be $2k for 48 drive bays, and one could easily connect 10 of them to a single $300 SAS RAID card.

I don't think a single SAS card can support more than 128 (or maybe 256 if you're lucky) disks. This doesn't affect your math much, but I wouldn't want someone to try this at home and be disappointed. If you're not willing to build a custom case and don't feel like suffering the vagaries of SATA expanders, LSI SAS cards plus Supermicro JBODs are the way to go.

From experience, I can safely say that luck has nothing to do with it. LSI MegaSASes support more than 240, so it doesn't affect my math at all.

Similarly, from "at home" experience, I can also safely say that luck factors prominently in the vagaries of SATA port multipliers. The reason I even made the attempt was a lower price per port. This is why I'm so started that a solution based on them would end up being more expensive than a SAS-based one.

Of course, if the camel of performance gets its nose into the tent of ones requirements, it has to be the SAS way or the highway.

This is not apples vs apples. Some of those items are living cows, and some are cooked steaks served with a side of mashed potatoes. I'm sure the cow looks cheap (per pound) until you are tasked with turning it into a steak.

Reply on news.ycombinator.com