Backblaze has ordered 100 petabytes of hard drives | Hacker News Reader

Backblaze has ordered 100 petabytes of hard drives

backblaze.com

187 points_jcwu142 comments

I’ve been a customer for years and recently had a catastrophic failure of a computer and it’s direct attached backup drive. I have spent the last four days waiting for backblaze to create a restore a backup for a computer on and last I checked it was at 9%.

I chatted with support and they said this is normal. That progress may jump to completed any time because the restore only tracks file count.

I wish I could say I’m happy about this but I’m four days into trying to pull down my data and I still have no idea when I’ll be able to start the download.

This sucks.

Why would it take any time at all to "create a restore"?

Brian from Backblaze here.

All the data is encrypted and spread throughout our datacenters. So we fetch each file and decrypt it (with the password you provided when kicking off the restore).

If this is less than 1 GByte it is instantaneous. But if it is a TByte of data, it takes a few hours to decrypt and add to the ZIP file.

That makes sense, thanks for the response. (4 days and counting still seems excessive though.)

Definitely seems excessive. We'll look into it.

There are some things that can affect it. We have a "pool" of restore servers that do this task. The oldest ones in the pool are 9 year old computers with slow hard drives. The newest restore servers are built on SSDs and are blazingly fast and have newer processors. So based on which restore server was assigned the task, plus other things like the size of each file can speed or slow the restore.

We are in the middle of a project to speed up all restores. Hilariously we figured out that simply by decommissioning the oldest, saddest restore servers the restores averaged FASTER restore times.

Presumably you mean faster than the non-hardware solution you were working on?

This is confusing and concerning to me as a potential customer. Having used CrashPlan and other proprietary and FOSS backup software, I don't understand this description of how restoration works with Backblaze.

1. Why is the data being decrypted server-side instead of client-side? The server should be sending the encrypted data to the client, and the password should never leave the client machine.

2. Why are zip files being created? The client software should be receiving the data and writing the files directly to the selected restore directory.

e.g. this is how restore works with CrashPlan, and it only takes a few seconds to begin the process. Files are written directly to the selected output directory.

I'll answer these in reverse order:

> 2. Why are zip files being created?

We actually offer two forms of restore: A) Zip File Download, and B) External USB Hard Drive FedEx'ed to your home.

In the case of the Zip file download, we chose the format of zip because both Mac and PC (the most common desktops) natively understand it with no additional software needed. In other words, if you just lost your computer, go to ANY COMPUTER ANYWHERE and you can fetch your files with a web browser. Zip preserves the file hierarchy and the last modified time, so it's "pretty good".

In the case of the USB Hard Drive there is no zip file, we can correctly place each file with all the correct timestamps in the correct heirarchy on the USB Hard Drive (which is actually an encrypted hard drive) and then FedEx the hard drive to you anywhere in the world in a day or two.

> 1. Why is the data being decrypted server-side instead of client-side?

Short Answer: Ease of use.

Longer Answer: To clarify, Backblaze produces four different products/modes for different customers with different needs and requirements. We want customers to choose what is appropriate for them. One size does not fit all:

1) Online Backup ($5/month) where every file is encrypted on your laptop BEFORE being sent to Backblaze and your backup is secured by your username/password - where you can recover your password if you have access to your email account. (We support two-factor auth which provides an additional optional layer of protection.)

2) Online Backup ($5/month) where every file is encrypted on your laptop BEFORE being sent to Backblaze and your backup is secured by your username/password AND your private encryption key is secured by a "passphrase" that is not recoverable in any way, shape, or form. (Two-factor auth is also optional here.)

3) B2 Object storage (half of 1 cent/GByte/month) where you store your file completely unencrypted, and this can be "private" (only accessible by username/password) or "totally public accessible by knowing the URL". A good application of this is serving up a web page to the public - you really WANT people to see all the contents!

4) B2 Object storage (half of 1 cent/GByte/month) where Backblaze has zero knowledge. You cannot browse your file hierarchy because Backblaze doesn't know your filenames. You cannot preview your images. You cannot recover your passwords. There is no other option other than downloading the encrypted blobs and applying whatever decryption algorithm you decided on (we have no ability to know what that is).

Ok, so I think some (many?) people in the security field think that Backblaze should ONLY offer mode #4 (and maybe #3 to serve up public websites). I happen to disagree and I personally feel that products #1 and #2 are useful and appropriate for some customers. But everybody is welcome to their opinion and we want to be completely open as to what exactly is occurring and what we are offering as a service.

Personally I think #2 is an excellent trade off of security vs convenience. Your data is as impervious to attack as a zero knowledge system in #4 for years upon years. Then one day your laptop is stolen or crashes and you want your files back. You want all 4 TBytes back - so you order one of our free (encrypted) USB hard drives to be FedEx'ed to your home with all your data. To kick this process off FOR THE FIRST TIME EVER you tell us your passphrase (up until this very moment it really has been zero knowledge). At this moment you are opening a window of SLIGHTLY lowered security that slams shut after a few hours. For those few hours of preparing your 4 TByte restore, if an undetected hacker had compromised the one restore server in the Backblaze data center that your job was on, that hacker could possibly get access to your files. But then the reduced security window slams shut, we NEVER write your passphrase to any disk so it has now vaporized and we do not remember it, and if a hacker hacks into our system the following day you are STILL completely impervious.

But again, as long as you fully understand the implications of #3 I am COMPLETELY supportive if you instead choose #4 which is our "Zero Knowledge" offering.

> At this moment you are opening a window of SLIGHTLY lowered security that slams shut after a few hours. For those few hours of preparing your 4 TByte restore, if an undetected hacker had compromised the one restore server in the Backblaze data center that your job was on, that hacker could possibly get access to your files. But then the reduced security window slams shut, we NEVER write your passphrase to any disk so it has now vaporized and we do not remember it, and if a hacker hacks into our system the following day you are STILL completely impervious.

This is the part that security people get hung up on.

We have to trust Backblaze [0] to not hang onto the password longer than you say you will, either intentionally (say, "to improve the user experience") or unintentionally ("oh that memory got written to swap and happened to persist on disk for a long time, whoops").

So that's the "malicious" case (maybe some of the above described cases are "benign malice", doesn't matter). There's also the "subverted" case: if the decryption password enters your infrastructure, we have to trust that you have not been popped by an attacker.

Whereas, if Backblaze has zero knowledge, and only serves encrypted bits, the _only_ concerns customers have are with durability and availability, which it sounds like you have pretty well in hand from your description of how bits are distributed around your datacenter(s).

[0] not just Backblaze but 5-years-from-now Backblaze: people and businesses change.

If you really care about the security of your data at this level (or external factors force you to care!), you shouldn't be using a product like BackBlaze.

In a commercial setting, you need to control the infrastructure and security artifacts. Otherwise, old fashioned physical controls are effective, trustworthy and have more legal protections than any of these products at a cost of some convenience.

I would really like #2 to have the option of sending encrypted files on disk or zip, and y'all providing a client that will decrypt them using my passphrase, locally.

Nothing personal, but I really don't want y'all seeing my data, even if I need to restore.

(context: I've been a Backblaze user for years and love the hell out of you. I've never really thought of how the restore would work with my encrypted data, though. Turns out the story is ... not what I had hoped)

Yeah, I don’t see how they would impact usability in any way. Just have the client software perform the decryption process.

This would be the ideal situation. It would require the capability for the desktop application to be able to restore encrypted data from a USB drive, but it would be a good midpoint between #2 and #4 that I think many security-conscious users would find acceptable.

Bonus points for open sourcing a tiny "backblaze decrypter" command line utility that can do this manually. Though maybe that's a stretch.

Yes, +1 for this. This is what I want, #2, but with a restore that transfers the encrypted data and decrypts locally.

This is how CrashPlan worked. (And incidentally, as I'm sure the guys at backblaze are fully aware, CrashPlan home is shutting down and I'm currently struggling to find a replacement that meets my desires. So if backblaze started to offer that #2 + encrypted restore soon they would gain my custom at the least.)

> Nothing personal, but I really don't want y'all seeing my data, even if I need to restore.

We totally understand, especially about the (very real) security issues.

I would reassure you that Zip file restores are COMPLETELY automated and with thousands of them happening every day no Backblaze employee ever sees your files.

The USB restore drives are placed in an envelope by a human, but they are encrypted when they are placed in the envelope so there is still no danger of any person seeing your files.

Plus we take our customer's privacy VERY seriously and it would be a firing offense for any Backblaze employee to ever look at a single customer file (or file name) without that customer's explicit permission.

Thanks for your detailed answer!

> In the case of the Zip file download, we chose the format of zip because both Mac and PC (the most common desktops) natively understand it with no additional software needed. In other words, if you just lost your computer, go to ANY COMPUTER ANYWHERE and you can fetch your files with a web browser. Zip preserves the file hierarchy and the last modified time, so it's "pretty good".

Security issues aside, from a space and time perspective, that's much less convenient for both you and me: it takes time and space for you to prepare the zip file, and space and time for me to save it and extract it. It would be better to install a simple client and restore the files directly into a directory. In the case of a few files, it doesn't matter much, but in the case of gigs of files, it definitely becomes a problem.

Is there any chance of you implementing a more CrashPlan-like restore option in the future?

> Longer Answer: To clarify, Backblaze produces four different products/modes for different customers with different needs and requirements. We want customers to choose what is appropriate for them. One size does not fit all:

I wish this explanation were on the Backblaze web site, because this makes it very clear. :)

Is there any chance of implementing a hybrid of #2 and #4, a client-based, zero-knowledge option? i.e. basically, CrashPlan's client in the private-key mode.

If Backblaze were to implement this, and add a Linux client, I'd seriously consider switching.

Thanks for your time.

> it takes time and space for you to prepare the zip file,

> and space and time for me to save it and extract it

One advantage is that for millions of small files, batching them together into one monolithic ZIP file allows for faster downloading. As opposed to a round trip to fetch each file. But there could be a happy medium where we batch 1,000 encrypted files to download to your computer then split them up, decrypt them, and cleanup any temporary files before fetching the next 1,000 encrypted files, etc.

> Is there any chance of implementing a hybrid of #2 and #4,

> a client-based, zero-knowledge option?

Yeah, we get asked for a "client restore option" quite a bit (see other comments in this thread also). Realistically I don't see it happening in the next 6 months, but eventually we really should get it done.

> One advantage is that for millions of small files, batching them together into one monolithic ZIP file allows for faster downloading. As opposed to a round trip to fetch each file. But there could be a happy medium where we batch 1,000 encrypted files to download to your computer then split them up, decrypt them, and cleanup any temporary files before fetching the next 1,000 encrypted files, etc.

Well, any such software that didn't have pipelining would be poorly designed. ;) Compare to software like rsync, Unison, etc, which transfer large numbers of small files quickly. Seems like a good solution would be to build on librsync.

Makes me wonder how much data/files you are trying to get back from their storage.

Then why not ask?

I had a similar thing happen to me, what I did was pick the files I wanted immediately and got a zip file with those within... 30 minutes? Then got more of the files later

Yev from Backblaze here -> That's definitely not normal, how much data/files do you have (are you doing a .zip or a USB restore)? We're currently QAing a dramatic increase to the restores, but right now only available internally until we are confident in it. In the mean time though you can download mission-critical files in smaller-sized restored, that might help you get up and running faster while the mega-restore completes.

It’s nearly 500GB of data.

Email me Hn <at> strapr.com if you want more details

Which is nothing

It's alot of data when the application is typically write-only, and throws little chunks of data all over the place!

Do you reckon the increase in restores has anything to do with APFS? I also lost my computer and Time Machine this week after upgrading to High Sierra, had to restore from backblaze.

Btw, unlike parent comment, I had a great experience with the restore and couldn't be happier with the decision to go with backblaze.

HE means they're QA testing a new method of restoring that will dramatically increase restore speed, not that there's a sudden uptick in restores. It's unclearly written, I had to read it twice before I got it.

Sorry - yes, you're correct!

I <3 Backblaze, but for giant datasets their policies just don't match up with the reality of doing a full restore. I had ~11-!5TB with them and needed to pull it all down. With the speed of their website, inability to check what I'd restored already and time to get a restore via the mail and over the net, I definitely lost some data since it took longer than 30 days. For Grandma, they are fantastic. For big data sets, I can't trust them anymore.

Note: They recently released a new website and software version that's supposed to address some of these issues but fool me once and all...

Assuming you are paying $50/year to store well over 10TB of data I don't think you are the typical use case. I know they say that it's unlimited, but nothing ever is.

This is why you always want the onsite & offsite backup, because offsite cloud backups will always be fairly slow to fully restore. I use a local NAS to achieve automatic onsite backups myself. It's part of the 3-2-1 backup rule.

I had a Mac Pro Mid-2010 workstation with a RAID 5 Array configured to have a hot spare for redundancy. Then I had a locally attached hard drive for time machine backups + BackBlaze + a hosted repo for the work I do + I use a third party encrypted cloud file sharing service for the most important documents.

The problem started when I decided to upgrade to High Sierra. The upgrade was catastrophic for this configuration of Mac Pro. I have been on the horn with an Apple Senior Advisor within Apple Enterprise Support who has escalated my specific issues to Apple Engineering. I decided to roll back but in the process of restoring the machine from the time machine backup the locally attached drive failed.

Apple has yet to tell me if they will even support the Apple RAID controller even though this version of the Mac Pro is listed as compatible. In the meantime I need to get back up and running. This is why I pay for a cloud backup in addition to having a local backup.

Fortunately, I also have a MacBook Pro laptop that I keep a duplicate copy of some of the data on.

My productivity will not take a hit.

It's just that most people are not me and it seems like it should not take more than four days to create a restore so that I can download my data.

For the life of me, I can't figure out why people upgrade to new operating systems so quickly. I never upgrade my macOS until x.x.4, but I prefer x.x.5. I'm on 10.11.6 because 1) it works perfectly, and 2) Karabiner doesn't work on 10.12.

I routinely help others with operating system upgrades in my day to day so out of necessity I upgrade as soon as a general release becomes available. It's also why I have multiple layers of backups for my data.

That's what VMs are for! VMware Fusion on macOS is a really good product and I heartily recommend people use it.

Karabiner was updated a couple of months back in the form of Karabiner Elements, a rewrite "due to kernel architecture changes in macOS Sierra".

https://github.com/tekezo/Karabiner-Elements

What if you're buying a new device that's already been updated to the latest version? It can be hard to downgrade, especially with Apple devices.

Modern Apple devices I’ve used cannot be downgraded to an older operating system than what they shipped with.

I do twice a day, local NAS backup and it saved my life recently.

A bug in a script I wrote in .net, was supposed to iterate through a list of network directories to delete (all starting with \\SERVERNAME\) and ended up iterating through the characters of the first item in that list ("\").

That's how I learned that in Windows, "\" means the root of the current drive, and that deleting all files from a drive on an SSD only takes seconds...

It sucks that you have to deal with this (I've been there) but I appreciate the added detail. Along with a few other stories I've heard, this has me leery of upgrading to High Sierra in the near future (although my MBP doesn't really have anything of importance on it).

Good luck, hopefully you can get everything back in order before the weekend is over.

That's the thing that always terrifies me about Backblaze. I really want to like it, the pricing is good and they are so open about how their business and technology operates, yet you constantly hear of people on HN getting bitten by corner cases where they lose their data :(

Brian from BackBlaze reached out and provided some feedback:

I checked into it, and we think your restore is slow for several reasons:

1) Lots of tiny files (we are fixing this in a rewrite literally this week)

2) It landed on a slow restore server. (We are planning on decommissioning that server soon because it is 8 years old and slow.)

We have the technology to move your restore to a faster restore server. However, the restore experts here tell me your restore is actually 90% through the part that would speed up, so they recommend just "let it run". I really really hope it will be finished and ready for you in the next few days.

If you need some of your files any faster, I think you should prepare several smaller restores. It turns out that if you prepare four restores, each one 25% as large as the original, the four restores will complete 4x faster because they are prepared by four separate servers.

You can create up to 10 simultaneous ZIP restores, and they are totally free, so feel free to spin up a few more restores even if they overlap with the original large 1.1 TByte restore.

Please reach out to me again if your restore isn't ready by Tuesday (October 10th).

Would it make sense for BackBlaze to offer a tool which automatically split restores into 10 zips if over a certain size?

I love the articles from Backblaze. I enjoy them publishing the failure rates of drives and anything else on the blog.

Two things missing from them in my opinion 1) a Linux client and 2) a second location

I guess I could get away without a Linux client if there was an unRAID plugin for B2.

Regarding 1): rclone does the job. That's what I use with B2.

https://rclone.org/

B2 plus rclone works well, but for many it's potentially significantly more expensive than the standard "all you can eat" $5 a month plan. In my case B2 would workout at roughly 4x the monthly cost and that's before factoring in the fact B2 charges for downloads as well as uploads. An official Linux client would be significantly better value.

As they've said though, the usage of most linux users isn't on par with Windows / Mac users. They simply would crush themselves with people attempting to backup 200TB nas devices for $5.00 a month. If you want any perspective on how crazy the fringes are with data storage and how they impact providers (such as amazon, google, backblaze, etc.) simply read the stories on reddit /r/datahoarder and you'll quickly see why this decision makes sense from a business perspective.

Very small numbers of people already do this on Mac/Windows using tricks like mouting the NAS as an iSCSI device to backup absurd amounts on the $5 plan. The business model is clearly that the legions of small users subsidize the "whales" and Linux desktops are a drop in the ocean vs the Windows/Mac world too. Do you have a source for where Backblaze claimed the usage of most Linux users isn't on par?

> Do you have a source for where Backblaze claimed the usage of most Linux users isn't on par?

One of their employees pretty much said exactly that here on HN recently. I don't have the comment bookmarked or anything like that so I can't cite it for you, but I do remember reading it.

I'm paraphrasing, of course, but s/he basically said "the math doesn't work out" (for the regular service, not B2 (obviously)).

I just started using Duplicacy with B2 and am very pleased. It's a bit more expensive if you have a lot of data (I do), but it's way faster, transparent and usable than their client.

Second on the linux client. I have a linux machine with a zfs volume that I've been backing up to crashplan. Since they've announced they're dumping their consumer market I have no cost-effective place to go. Most likely, I'll be more picky about my backups and use S3 life-cycled to Glacier.

> Since they've announced they're dumping their consumer market

Could you link some source? I'm also backing up my linux machines with them and would prefer to plan ahead.

They have emailed a few times about it.

Some info here: https://www.theregister.co.uk/2017/08/23/crashplan_ends_cons...

time4vps.eu has some nicely price storage servers. They have no backup, since they're designed to be part of a larger backup strategy, but it works for me.

affiliate link: https://billing.time4vps.eu/?affid=1881

Cheaper then backblaze in most situations and you get a full linux host.

I would really like to use rsync.net for my "external third-party" backups specifically because of their "ZFS Platform" [0] but (from my off the top of my head calculations) their pricing seems to come out a little on the high side.

With my luck, the first time I need to be able to quickly restore a full dataset or something, it will probably have been well worth it and I'll likely be kicking myself for not doing it.

[0]: http://www.rsync.net/products/platform.html#zfs

Be careful with rsync.net if you're on a comcast cable modem. They can't seem to handle the "xfinity powerboost" and you end up being throttled to 160kB/sec

In previous comments they've indicated that a Linux client won't be released because they can't afford unlimited storage for all the Linux NAS boxes.

B2 is priced at a sustainable point for them, so that's your best option for other use cases-- it's only $5/TB*mo!

I guess all the crashplan personal users are jumping over there en-masse. Good for business...

something happened to crashplan?

They're shutting down their consumer service to focus on the B2B market.

https://blog.eogn.com/2017/08/22/crashplan-is-shutting-down-...

Yev from Backblaze here -> An influx of Computer Backup customers has certainly played a factor, but also Backblaze B2 is growing quickly!

[Resolved]

I'll respond! Ping me at yevgeniy@backblaze.com :D

I have a hunch the page you're after is https://www.backblaze.com/company/partners.html

Contacting via HN was the last option. I tried there and sales on numerous occasions.

Even before that they were seeing 25PB in-bound per month, so at the previous level this only a 4 month supply.

I've watched Backblaze closely, and I love their detailed reports of drive stats. I've always been curious about their choices of manufacturers though. In my past I found that Hitachi/HGST drives were far more reliable than others. We had a couple people doing hardware and software support on ~120 machines, and hard drive replacements were expensive (in man-hours) for us, so whenever possible we got Hitachi drives.

The Backblaze numbers seem to support this as well, far fewer failures of HGST drives vs Seagate, but they seem to buy mostly Seagate. I guess when you have tens of thousands of drives, you're always going to need to be replacing them. So maybe a 10x reduction in failures isn't worth it? Or maybe they just couldn't get HGST drives in quantity?

I've never had a Seagate drive last any reasonable amount of time

They don't buy direct from manufacturers so it might be partly down to availability numbers from their local distributors.

Also you want to buy from a wide range - if most of the drives come from one place and they have a manufacturing disaster that causes everything made in one month to fail a few months into use then that might take you too close to redundancy requirements for comfort. I would expect the range picked will be based on individual design (platter design, platter layout, controller type, ...) rather than per manufacturer - maybe HGST has fewer designs available at any one time and that tips the balance a bit.

Yev from Backblaze here -> We have relationships with manufacturers but don't directly buy from them (yet - fingers-crossed). It really is a combination of price/reliability. At our scale we can afford slightly higher failure rates if it means less expensive drives, so our #1 data point for purchases is the cost per gigabyte. After that we do look at failure rates, but unless it's something wildly out of "normal" we tend to live with the occasional failures. Thanks to Vaults the back-end is designed in such a way that if a drive fails we don't have to run and replace it - so individual failures tend not to stress us out too much anymore!

If a 10's of petabytes isn't enough scale to talk to manufacturers directly, I wonder what is!

After the order mentioned in the article, I wonder if it will be. Since they mention a mix of 10 an 12 TB hard drives, we can assume the order was for around 9000 drives. At a cost of $0.03/GB (not taking into account shipping or taxes), you're looking at a 3 million dollar order.

It is worth considering that the use-case of Backblaze is different from the average end-user.

No power on/off cycles, no physical impacts to worry about (even most desktops occasionally get bumped into), but non-stop operation and presumably near continuous read/write operations.

You'll likely get different failure patterns as a result.

> but non-stop operation and presumably near continuous read/write operations.

I thought it would be more like: continuous writes until full, and then likely a handful of reads for the rest of that drive's life.

Not if they're in service of B2.

The last I heard, I understood that Backblaze doesn't go around replacing single drives that fail in a pod, rather, their software marks that drive bad and only once a certain percentage of the capacity of the entire pod is gone, do they yank it our and rebuild the pod. That probably helps a lot versus an environment where had to have technicians replacing single drive failures randomly throughout your data center.

they operate at datacenter scale: https://xkcd.com/1737/

Do you keep them powered on at all times? Does a storage pod include a software-switch to turn them off selectively?

Yev from Backblaze here -> Yes, unless the storage pod is in maintenance, it remains on so that backup and B2 data can be uploaded and accessed.

Interesting. Have you ever computed how much power you would save if you turned them off when not needed?

Or is every pod constantly in use? Would it help if smart algorithms repartition the data on storage pods, to optimize power use?

Why wouldn't that be handled already by the drives themselves?

Perhaps partly. But a drive can't repartition data according to frequency of use.

I wonder about this, too. Since access latency is the primary concern, a neat solution would be to create some dynamic "fastup" storage scheme. For large files, loosen the always-on requirement; to fulfill a request the first N bytes are served from the always-on pod, and the tail is stored and served from pods that dynamically power up drives only at access time. The size of N is computed such that the IO time it takes to make it through the fragment stored on the always-on pod is > the power-on time for the pods where the tail is stored.

I don't think latency is an issue for a backup solution.

Except currently it is a concern:

> it remains on so that backup and B2 data can be uploaded and accessed

This is also why Amazon has tiered pricing between S3 and Glacier.

Brian from Backblaze here.

> Have you ever computed how much power you would save if you turned them off when not needed?

Electrical power is a gigantic part of our datacenter bill (like 60%). However, we cannot really ever turn off the hard drives.

If they were powered down, it would take too long to retrieve data. It would be like Amazon Glacier where it can take 15 minutes to get a single image back.

> is every pod constantly in use?

The way we built it, yes. If one user deletes a backup or unsubscribes from the Backblaze service, it frees a little bit of space on pods all over the datacenter. So we allow some other customer to use that space up as soon as we can.

We also use ALL the pods in the datacenter to give us "breadth" to accept data faster. This is particularly helpful on surges of new customers due to a press event, or if a competitor like CrashPlan decides to exit the market.

> If they were powered down, it would take too long to retrieve data. It would be like Amazon Glacier where it can take 15 minutes to get a single image back.

Well, speaking for myself, I don't mind if I have to restore a backup, and have to wait 1 minute extra for a system to come online (that's about the typical boot time of a Linux system, so I suppose for a storage pod it would be about the same).

By the way, it seems possible to power down an individual SATA drive if it is not in use: [1].

[1] https://unix.stackexchange.com/questions/112117/shutdown-my-...

This is what I thought amelius' comment was originally about. You could save power on drives plus it would only take a few seconds to come back online. I imagine it could save a lot of electricity if the additional complication is worth it.

Complications like a write cache drive in case of power failure, or grouping expected-cold data together. Also I wonder if powering the drive up and down occasionally will have a positive or negative impact on drive reliability compared to having it on 100% of the time.

Interesting, your considering that your setup somehow compares to Blackblaze

Also if if he says it takes about 15 Min's for it to come online and be usable in their massive data center I would believe him.

Not really, he was talking about Amazon's solution.

Yev here -> I'll piggy-back a little and say that not all the pods HAVE to be powered on in order for access to remain rapid. Our Vaults system spreads shards of the data out, so if one or two pods go in to maintenance in the same Vault at the same time, it doesn't really impact performance.

If anyone from the Backblaze team reads this: I love your service BUT please provide proper invoices.

The ones provided in the „My Account“ page are an absolute nightmare.

Look at services like Stripe or Digitalocean and make them downloadable as PDF. Best case: send them out via email or auto-upload to specific Dropbox folder.

@atYevP how much does the Backblaze business plan cost?

Or rather, if I have 10TB on a NAS and a Linux machine, what would it cost to back that up to Backblaze as a consumer given that you don't support this and I'd need to purchase the business thing?

$50/month at $0.005/GB/month.

https://www.backblaze.com/b2/cloud-storage-pricing.html

Aha!

I thought B2 was like S3 and figured that Business Backup was what I needed.

TIL.

Brian from Backblaze here.

> I thought B2 was like S3

It is. Backblaze actually has two separate product lines:

1) Online Backup for $5/month/computer where you have to use our client we wrote with all of it's policies. This is widely targeted at "ease of use" type customers. There is a free feature inside of "Online Backup" called "Groups" where one person can pay for many others. You can use this for a business, we absolutely do NOT charge differently just because it is a business. Or you can use "Groups" to pay for a family, or whatever. "Groups" is free, we honestly don't care if you use it or not.

2) Backblaze B2 - this is "object storage for half a penny per GByte per month". There are MANY clients available for you to choose from, you can see a list on this page: https://www.backblaze.com/b2/integrations.html The idea here was many people kept approaching Backblaze saying they needed to store stuff cheaply, but didn't want to use our client, or maybe it wasn't "backup" they needed but to host live images on a website on the internet, stuff like that. So we built an API anybody can program to. https://www.backblaze.com/b2/docs/

Hey there! The business and consumer computer backup service have the same feature-set, so our business service would also not work with a NAS or Linux box. I'd recommend looking in to Backblaze B2. It'll be a bit more expensive as it's $0.005/GB but you'll have more control over your backups and lots of integrators make it easy.

Is Linux backup support anywhere on the radar at all?

I recently trimmed down my Backblaze licenses after switching to Linux installs, just went with s3 as s3cmd had a path of least resistance to storing backups. Would love to switch back if it's ever possible :)

Theyve repeatedly said no -- it breaks their business model to allow Linux machines (often servers) to backup "unlimited" data for $5/month. People with multi-hundred TB NAS machines would end that business model, quickly (see Amazon Cloud Drive, Crashplan, and others).

But if youre using S3, just switch to B2, its per GB pricing, much cheaper than S3 and supports lots of Linux tools. I use rclone.

I took the GP's remark about how he "went with s3 as s3cmd had a path of least resistance to storing backups" to mean that he considered B2 but opted for S3 since the latter offers a backup utility, but the former supports neither a "native" (e.g., git or rsync over SSH) backup interface, nor does it have a Linux client—the approach for backup on B2 from desktop Linux is to roll your own backup solution or use somebody else's client that has already added support for the B2 API.

I'm pretty sure the right Linux backup solution is rsync.net, by the way, which does support a native (SSH) file access interface. For big backups, storage is (slightly less than) twice as expensive as S3, but unlike S3, there are no bandwidth costs.

rsync.net is 18x as expensive as B2. Have you ever used rclone? Its surprisingly good, though might not be a perfect replacement for rsync, depending on your usage.

> rsync.net is 18x as expensive as B2

I'm aware, but B2 is not the reference point. He said he went with S3, so that's what we're comparing to. I.e., if S3's prices are palatable, then rsync.net might be, too.

"For big backups, storage is (slightly less than) twice as expensive as S3, but unlike S3, there are no bandwidth costs."

Just to clarify ... we (rsync.net) support "borg backup" now (with borg deployed on the server side ...) which is the "holy grail" of efficient, zero knowledge, remote backups:

https://www.stavros.io/posts/holy-grail-backups/

... and if you are willing to give up our ZFS snapshots on the account (you manage retention yourself with borg) and you are willing to handle your own technical support of borg, we offer a 3 cents rate. This is about the same price as S3:

http://rsync.net/products/attic.html

Sorry guys the path of least resistance also took into account price. I'll admit I've not checked your pricing recently but last time I used your service I was paying something like $70/yr for <100G storage

@atYevP Do you guys do anything special regarding HDD vibrations? I am assuming your special case is built with this in mind. It is crazy how many things one has to consider at such scale. Things people normally do not care about!

Not Yev but I also work at Backblaze.

Most of the anti-vibration tech is straight forward, like the fans aren't bolted to the chassis, they are held by rubber rivets, and we have slowly but steadily improved the "pressure" that drives are held in place to limit failures, etc.

From time to time, a fan will "half fail" causing large vibrations which can massively increase drive failures in the rack that bad fan is in.

Thanks. Another question I got for you is:

What is louder? The fans or all the disks together? I had a workstation with 6 spinning disks and that was already quite loud.

Now it's Yev - you'd have to ask our DC techs. Every time I go to the datacenter it "feels" like the loudest part are the fans, but my ears aren't attuned to what is causing which ruckus :P

I imagine you'll have to reduce pricing on Backblaze B2 eventually in order to stay competitive. What's it like having to flip such a massive switch where your revenue is X one minute and then X - 20% the next?

I don't understand. B2 is currently about half the cost of S3 for outbound data and 1/4 the cost for data at rest.

What do you mean "stay competitive?"

They're not equivalent. B2 is singly homed.

So is S3. You have to pick the region (home) for the data when you create the bucket.

From https://aws.amazon.com/s3/ and https://aws.amazon.com/s3/details/

    Data is automatically distributed across a minimum 
    of three physical facilities that are geographically
    separated by at least 10 kilometers within an AWS Region
    
    Designed for 99.999999999% durability and 99.99%
    availability of objects over a given year.
    
    Designed to sustain the concurrent loss of data in two facilities.

From https://www.backblaze.com/b2/cloud-storage.html

    Uptime: 99.9% SLA

    Reliability: 99.999999% durability

So S3 is 10x more available, 1000x more durable, and datacenter redundant.

Eleven 9s of durability is an utterly farcical guarantee from a decade old service. A single big durability event in the next century would irrevocably decimate that record. They really might as well claim 100% durability, because it is no less plausible.

I mean, what are the odds that S3 will even be operating in 100 years? I'll certainly put $1 against $1,000,000,000 that it won't, and that's only one many ways it could fail the standard.

I mean that prices in the cloud space tend to drop regularly, they can't stay at this price point forever.

But why would they lower prices now? They are already the cheapest.

xhrpost said "eventually". Not now.

> you'll have to reduce pricing on Backblaze B2 eventually

Yes. We have already dropped B2 prices once, and the expectation is we'll probably be dropping them regularly as our cost per GByte drops.

> What's it like having to flip such a massive switch...

What's amusing is the switch is tiny (just one constant in a json table) and but the fallout is massive. But realistically we have a business strategy of not making egregious profits, so we only drop prices when our margin is too high.

Jeff Bezos of Amazon once said, "Your margin is my opportunity." Well, Jeff's margin is Backblaze's opportunity. :-) We survive and thrive because Amazon is being too greedy and marking up their storage (and outbound bandwidth) by way too much.

This, my friends, is the way it's done. Kudos!

Love these articles! Thanks for writing it. Must feel so good to buy that many HDDs at once...something really gratifying about that.

As the guy who signs the check, "gratifying" isn't quite the right word. :-) Some stress, some amazement that everything got this large.

When we started 10 years ago, we could only afford to rent a quarter of one cabinet at the datacenter at 365 Main Street, San Francisco. We had "right of first refusal" on the rest of the cabinet we occupied. We could pile all of the hard drives ever deployed in that cabinet on my kitchen table.

Now it is semi trucks filled with fork lift pallets of drives. But it just "snuck up on us". Each day it was a little larger, each drive order was just slightly larger than the previous.

Yes, something taking 10 years to sneak up on you is very sneaky.

How about security? If Backblaze can recover files for users then they can get the same files for any TLA.

That's the advantage of B2: straight storage, bring your own (encrypting) client. If Backblaze proper had a decent restore story (rather than typing your key into a web page and getting a zip), I'd recommend it wholeheartedly.

https://help.backblaze.com/hc/en-us/articles/217664798-Secur...

wonder if Backblaze is buying directly from HDD manufacturers or wholesalers and how much volume discount?

They've talked about this before -- despite orders like this, they're still not a big enough customer to be on the manufacturer's radar. So they're ordering from a middle-man.

Back during the hard-drive crisis after the tidal wave that disrupted production ... they were buying from anyone that had drives to sell, including people going into their local big-box store to source drives.

Yes I think the 'shucking' blog post was the one that caught my attention first.

I wonder which customers are big enough if this is the case. I can only think of PC manifacturers (Dell et al., but they are switching to SSD's) and companies with a truly massive amount of datacenters (Google, Microsoft, Amazon). Ipods and similar electronics have switched from HDD's for a while now.

So only Dell, HP, Apple and the likes have a direct line ? 10K units of the largest models feels huge to my layman self.

Still? 100PB of HDD is still not big Enough?

So the question would be, In unit of Backblaze, what are the size for Amazon, Apple, Google, Facebook?

Curious on that too. I wouldn't be surprised if Amazon needs 10,000 drives a day (many of which are probably going to replace dead drives, or upgrade lower capacity ones). Google is probably about the same.

Early on, we (I work at Backblaze) were told by the large drive manufacturers that to do a "direct order" we had to buy at least 10,000 drives in a single order.

If those are 12 TByte Seagates, that's about 120 PBytes? So at this point we are almost there and in fact we sometimes talk directly with the manufacturers, who turn around and send our order through a distributor or reseller. This is probably to prevent "channel conflict" where Seagate or Western Digital is undercutting it's distributors, but that's just a guess.

Link to their blog post on mad dashes to Costco after the floods [1]. It's an engaging read.

[1] https://www.backblaze.com/blog/backblaze_drive_farming/

Yev here -> it was a pain to drive around the Bay, let me tell you :P

Could you let us know about failure rates over time? Thanks for sharing.

Yev here -> These drives will be part of our ongoing "Hard Drive Stats" posts on our blog!

Curious where one would buy those, directly from the manufacturer?

Yev from Backblaze here -> Most of our purchasing is done through distributors. We'd love to buy direct at some point, but for now we have good manufacturer/distributor relationships! We're getting there though!

There is a simple fix - become a distributor. I worked for local European distributors that moved less merchandise per year and had direct relationships with Samsung/Fujitsu/IBM/WD/Seagate (you can tell it was a long time ago :P).

We've batted the idea around :P

Do you mean the storage pod? You want to look at https://www.backblaze.com/blog/open-source-data-storage-serv... and https://www.backuppods.com/

Reply on news.ycombinator.com