"It's the (Backblaze) restores that fail." | Hacker News Reader

"It's the (Backblaze) restores that fail."

seankilleen.com

31 pointsSeanKilleen54 comments

Please don't singly use cloud backups. Your first line should be a local backup solution like a Time Capsule or other NAS device. Blind trust in a black box backup is not prudent.

Precisely the point I'm making. It was hubris and laziness and I'm surprised that I let myself go that long. In a follow-up article I discuss an overview of what I'll be doing going forward: http://www.seankilleen.com/2014/01/my-newest-backup-system-p...

There is nothing wrong with trusting online backup providers provided your data is backed up on two or more of them and there not both using the same back-end (aka Amazon s3).

I agree, and I would add to that: "as long as you also have a local source." That was my failure here. Had I backed up to a local source, the slow restore times / issues would be marginal at best. So there are two lessons from my perspective: Backblaze needs to get its restore scenarios fixed for folks like me, and I need to get my shit together as far as a back up strategy. Good lessons to be learned for all I think.

For this very reason I've always backed-up to both an encrypted external HDD and another encrypted copy in the cloud.

Whenever a cloud service failure is posted to HN, there are always posts basically telling the OP that expecting the service you paid for is a silly idea.

Why isn't the SaaS industry in general roundly criticized as the fly by night hacks they always turn out to be when someone gets burnt?

Haha, I appreciate the defense. At the same time, I understand the economical aspect of the service, and there is a measure of "you get what you pay for."

To be fair, I wouldn't call Backblaze hacks by any means, but I understand your point about the business responsibilities of companies and not giving them a pass just because they're a price-friendly SaaS.

If you're going to only have one backup (which is the case for most people, realistically), your first choice should be cloud, not local.

As far as I know, the chances of something physically happening to your local backup (the hardware fails, someone knocks it off the shelf, it gets lost in a move, there's a fire) is vastly greater than the chance that your online provider loses your data.

Yes, obviously it's better to have both, but the fact that cloud is both off-site and stored redundantly off-site makes cloud the obvious first choice for most people (since most people are not setting up RAID arrays at separate locations).

> the chances of something physically happening to your local backup (the hardware fails, someone knocks it off the shelf, it gets lost in a move, there's a fire) is vastly greater than the chance that your online provider loses your data.

This is the key part: almost all of the failure modes affecting local storage also endanger local backups. Cloud backups are intrinsically less risky in that regard, although a local provider might still be affected by a massive natural disaster.

The remaining class of errors, however, are nasty ones like bit rot, human error or malice – all of which are tricky to prevent.

The more general advice would be to test your backups, whether they're online or offline (ideally both). Now I should go and practice what I praise ...

I wanted to post this here to bring attention to what happens when trusting a cloud backup provider with your data, as well as to spur Backblaze to hopefully pay more attention to this part of their business model.

Again, while I say it in the post, I think it's important to state again that the CEO has been great in allowing a dialog via Twitter. This is not a post meant to diss Backblaze as much as it is to share concerns and put the thought out there for consideration.

As the CEO Mr. Budman notes in the Storify link, they no doubt do hundreds of successful restores every day. But when I have two working systems fail for Backblaze restores but nothing else, it's still a cause of concern for me personally.

Their selling point is "Unlimited data backup for your Mac or PC, just $5/month." If you're in tech that should tell you it's not for storing "mission critical data" (paraphrasing the blogpost).

For $5/month you can also not expect round the clock, seven days a week personal attention.

Don't get me wrong, Backblaze seems like a kickass service for an off-site backup of my personal data.

A good point: "mission critical" is an overstatement and I should phrase that differently. "Critical" to me means personal value (i.e. family photos), and not necessarily in terms of timeliness (i.e. business related).

And to clarify, I don't expect personal attention. Contacting the CEO was because I thought there might be a legitimate problem with a server and support hadn't responded, and writing this blog came after I'd attempted to follow up with support but hadn't heard back for over 8 hours.

I've stated this elsewhere on the thread but I'm by no means out to shame Backblaze. I think there are some UX points worth revisiting, and I may be looking at another provider, but I'm not trying to damn the company. I think it's a discussion worth having on how they might do things better (and also how I might do things better).

I also lost a drive recently and found that while uploading to them was a breeze, the download was coming in at 14KB/s. I thought I was the one having a problem, but maybe not.

Yep, you weren't. Glad to know I wasn't the only one. Speeds varied for me, though the transfer did take much longer than the estimates. And that makes sense, given that I'm sure it's a ton of bandwidth to deal with all of those multiple restores. All the same, setting some expectations in those situations would be helpful, too.

FYI I left Backblaze for Crashplan for the same reasons.

I believe Blackblaze also removes the backup after some number of hours, so if you don't download the backup by then, you're in big trouble.

Crashplan also allows external and network drives to be backed up.

Could you elaborate? Same reasons in that you had slow or broken restorations?

Also curious: have you had to restore with CrashPlan, and if so, how has that process differed?

I had many issues with Backblaze over several years.

I found 1-2 times a year, when initiating a backup, the entire archive I had disappeared, as in, my computer would start to send all of my files all over again. This was very disconcerting.

I never liked seeing all files on an network drive disappear just because it was unplugged.

The final straw was when I attempted to restore a large archive, and saw the download speed was so slow, it would not finish before this restore was removed on Backblaze's servers.

Crashplan, has had zero issues over 2-3 years now. I've restored large files without issue, they store versions which I don't even need/want, and the desktop software works very well. Highly recommended.

I had the exact same situation three years ago with Carbonite. Had 80GB of data that I was never able to be restore. Their only means of amending the situation was offering a year for free. Never again. I had a local backup but it was unfortunately 3+ months out of date. Now I keep everything locally backed up with a NAS and a mix of dropbox/google drive.

There doesn't seem to be any attempt to confirm that this is actually a problem on Backblaze's end -- it's just as likely to be a problem with Sean's internet connection. One of the problems with downloading large zip archives is that a few scrambled bits or dropped packets can scratch the whole archive.

Backblaze offers the ability to get your data in the form of a flash drive or external drive (https://help.backblaze.com/entries/20224632-how-can-i-restor...). It's not free, but if your data is important, it's a pretty reasonable cost-of-hardware-and-handling price.

Hey there! Thanks for commenting. I think I commented on this aspect in a few other places on the thread, but while I glossed over the IT troubleshooting steps in my blog post, I've been in IT for 10+ years in the roles of support, sysadmin, and developer. I _of course_ did some troubleshooting. Two different PCs with internet connections that worked completely reliably both had different issues, and they were both in known stable states at the time. On each, I'd tested the internet connection, speed, etc. Nothing else was wrong with my internet connection at the time, at least in terms of things like loading web pages, downloads, and streaming HD video. In the first case, the speed seemed to drop to zero and then timeout. In the second, case the file took a long time to download and then came out corrupted anyway. I'm not saying these things don't happen; but I'm saying that, if it's a company's bread and butter, it would make more sense to put as many safeguards in place to make sure that after 12+ hours, a corrupted file doesn't happen.

Also, I'm aware that a few dropped packets can be an issue. This is why I used Backblaze's advertised downloader application -- I got the impression that it would provide better stability, checking, etc. to ensure that the zip file did not get corrupted.

I also mention the external HDDs in my post, but my point is that I pay them so that I can restore my files conveniently, not so that I can be upsold to a $189 drive. Normally, I would drop the cash, but after just spending several hundred dollars on a new motherboard, I figured I would use their download mechanism, which is offered. I think I'm allowed to have a reasonable expectation that it would work.

With all that said, this is for sure not some "grab your pitchforks!" moment. I'm doing this partly to call attention to my own stupidity at putting my faith in one cloud backup provider, and also to highlight that Backblaze's process for restoring could likely be a little bit smoother, given that this is when users would like things to be absolutely foolproof. I remain a fan of Backblaze's CEO who was responsive when support wasn't, and of the company itself which has done great things like open-sourcing its hardware.

I've since been in contact with support again and I will be sure to update my post once the matters are resolved.

The part that doesn't make sense right now is that Backblaze has been around for several years, and if they had some kind of infrastructural issue with transmitting large amounts of data, I'd expect this not to be the first such case to come across my news feed. (BB has been featured on HN a handful of times -- no reports that I know of so far with retrieving data from them.)

With all due respect to your expertise, in at least 9 out of 10 cases where I've encountered something similar to this, it's been a problem on the client's side of the link -- bad traffic shaping by the ISP, faulty modem, glitchy router or switch -- and those often don't show up in speed tests or media streaming or web browsing.

I have an idea that would satisfy my curiosity, if you're willing: how about renting a VPS on DigitalOcean (who charge in small increments), with enough storage for zipped file size + unzipped archive size, and then see if you can download the archive from Backblaze to the VPS and unzip it on the VPS? See e.g. http://victorwyee.com/shell/getting-backblaze-restores-via-c...

If it works, you can rsync over ssh. If it doesn't work, then I'll buy that Backblaze has got a problem.

> "This is why I used Backblaze's advertised downloader application -- I got the impression that it would provide better stability, checking, etc. to ensure that the zip file did not get corrupted."

I strongly suspect that their downloader just implements http resume in some slightly brain-damaged way.

> "...my point is that I pay them so that I can restore my files conveniently, not so that I can be upsold to a $189 drive."

Ah.

You're paying them $5/month to store your data. That's it. It's a bit of a nit to pick, but it's important: for $5/month, they're not guaranteeing that you can download your data. So providing alternatives to downloading is not an "upsell" (especially at those prices).

My current backup strategy:

MacBook & Mini -> Time Machine on Mini (second hd), everything is using FileVault

Pictures -> Bitcasa & local copies

Personal Documents -> TrueCrypt Volume on Dropbox with local copies, coding projects are on dropbox also, obviously I have code in SVN/GIT repositories.

I have 2 local copies, one in the safe deposit box and rotate them about every 3 months. This is pictures + personal files on a FileVault volume.

Currently I don't backup my windows gaming pc aside from game saves on Bitcasa.

This is a little disconcerting. I have 3 computers backed up with Backblaze. I had a successful 350 GB restore last week. I hope that wasn't an anomaly.

From what I gather, my situation may be the anomaly. Still concerning though. I'm still processing a 340 GB restore in the hopes that this one will go better. We'll see.

Your success has me hopeful!

Your situation is an anomaly, or this specific incident?

You _have_ actually tested restoring from your only backup solution at a service you've been evangelizing for years, right?

A fair point. Yes, I've tested and restored things before. However, those were normally smaller footprints and not the entirety of my data. Given the rest of the public's generally good experience with them that I could find (and kept seeing), I felt fairly confident that my data is safe (which is to say, I might not be able to get to it easily, but it's not going to disappear into thin air either).

I'm thinking given prior experiences, other users, etc. that my experience currently is the anomaly (and hoping so).

A question, because I'm unfamiliar with the way HN works in this regard. My article was #2 on the front page for a brief spell.

Now I look, and it's nowhere to be found.

Is this by design? I don't need it to be on the top page and this isn't about the page views, I'm just not sure how it would disappear. Did a moderator somewhere frown upon it?

Haha, nevermind, I'm an idiot. Just hadn't considered that the sheer amount of content would knock it off rather quickly (which makes sense, given that this is not nearly worth the focus that some other topics are.)

One other note: I think it's important to say that I'm a HUGE fan of what Backblaze does in terms of open-sourcing their storage box designs and how they accomplish things. I think that's a noble thing to do, and have always thought highly of them for that.

Wired or wireless connection? I would always do bit sensitive data over a wired connection.

This goes without saying.

Wired connection during both attempts (even on the laptop). But you're right and it does go without saying.

No it doesn't. Especially when it comes to HTTPS.

"No it doesn't" what?

That it's ok to trust https to keep a huge download completely error free?

It does not go without saying that you should use wired connections instead of wireless. Wifi has a perfectly good error detection mechanism that gets layered on top of the TCP mechanism.

Or over wireless. Absolutely.

If there was data corruption in the HTTPS stream, it wouldn't decrypt correctly or the packet wouldn't checksum correctly and it would be retransmitted.

TCP works just fine over wired vs wireless.

Backing up your data is not enough; you must also occasionally restore your data ( to a temp location )just to make sure your backup media or its governing database are not silently screwing up your backed up data.

Agree 100%. In the future I plan to do this with both my local and cloud personal backups. (I would always do this with any business-related backup; I don't know why I my standards be lower for my personal data.)

I wonder whether it's a header or an index on the compressed file, or the data itself that's damaged. Still not something you want from your backups.

Correct. They appear to basically zip the data using 7-zip and then transfer it, but I'm not sure what sort of hashing/checking they do to make sure the file makes it to me successfully.

Very possible that the next time I download the 6GB file it will work. But given how long it takes to do that, 1) I'd like to know it will work and 2) tougher for me to consider utilizing the free option of doing this for all ~400 GB of data.

Coming from working in IT support, it's frustrating to me that you've made a big public ordeal over something that, firstly was fixed by rebooting your PC, and secondly you haven't tried downloading the file a second time. I'm not necessarily defending Backblaze here, but posting this when it doesn't work quite right the first time seems a bit harsh. I feel like we need more information before we all grab our pitchforks.

There is no excuse for independent download software to fail because chrome is running, I don't care how fixed it is by a reboot.

The point the poster above is making is: Generally when you see posts with not a whole lot of troubleshooting which are decrying that the major component of your entire company (IE: restoring backups), is fundamentally broken, usually this leads to PEBCAK. After 10+ years dealing with users who claim that $ITEM is broken (yet has been working for years for the majority of customers) sets off the PEBCAK meter in that either they are doing something wrong, or more troubleshooting is needed before jumping to conclusions. If further troubleshooting does seem to point to an edge case, then finger pointing can begin...

I understand this concept. I also would not describe me as "decrying" Backblaze at all, nor saying that restoring backups is fundamentally broken. It's broken for me _right now_, and I think the UX and approach could be better.

I've been in IT for 10+ years, and while force killing chrome worked, it was a genuine surprise, given that every other site (including bandwidth-sensitive) activities performed fine. On that one, I should have dug deeper. Rather than closing and reopening the windows, I should have killed the processes sooner. I'm willing to give them a pass on that.

I'm more concerned about waiting 12+ hours for a 6GB restore that then can't be extracted and restored, even when I used their downloader to ensure better results.

These issues also happened on two different machines, both of which work fine in every other respect.

What makes you think the issue has anything to do with Backblaze, rather than something specific to the OP's machine?

He's not even sure that Chrome.exe was at fault. Why are you so quick to assume that it's Backblaze's fault in this one customer instance? He could have just reinstalled his machine, did a lot of weird configurations, etc. and not have even rebooted his computer a single time yet. Anyway, my point is that there is a lot of missing variables and information to make such assumptions. The customer is _not_ always right, you know.

This was a different machine, but you're right that that problem might not have been backblaze's fault.

But if he had problems downloading on both computers it's quite likely that backblaze is at the very least providing a low-quality downloading application, if not having outright broken features.

It's ambiguous but I think he did the 6GB download on the restored machine?

By "not being sure if it was Chrome", I mean that by any user's standards, I had closed and re-opened the application several times.

The first issue (downloads not working) was on an otherwise-fine laptop that I checked to ensure could do other bandwidth-sensitive tasks (streaming HD video, etc.) it was only the Backblaze site and downloader that showed 0-3kb/sec and then failed.

I could have done a lot of weird configurations, but just to be clear, in this situation, I did not. On the first download that didn't work, I was restoring to a laptop that I had just booted up and had not performed any configuration on. In the second case, I was downloading a 6GB file to newly restored system that I'd rebuilt from scratch, and I'd tested all of the components (including the network connection, disks, etc.) prior to considering the rebuild complete. After that and all of the updates, it had run in a stable condition for some time before I attempted the download.

I agree that the customer is not always right. But also note that I am not claiming that backblaze is evil. I'm claiming that this process and my experience makes me question their model as well as my own idiocies.

And I also should note that while I might not seem that adept in this scenario, I've spent years as an IT support person, systems administrator, and developer. Which is not saying I'm right all the time or to bolster my case, just to note that I didn't do _no_ troubleshooting before I wrote the post.

I agree with your point to a certain extent -- I've spent many years in IT Support as well. With that said:

* I went out of my way to avoid Pitchforks, merely calling attention to an issue and also ensuring I blamed myself enough * A 6GB download took 12+ hours. How many times am I supposed to wait for critical data that long to have it fail, and what happens when I try my 340 GB restoration, wait weeks, and it fails? * I think there are many potential better ways of doing it -- breaking into chunks and verifying each chunk as it is downloaded, any sort of check before the application says "complete", a little more finesse than merely using 7-Zip, etc. There are ways the experience could be improved is the point I'm making, since as customers this is the part where Backblaze really earns its keep.

I have no desire for pitchforks, which is why I pointed out that the CEO was extremely helpful and that I'm hopeful the problem will be resolved.

This is not an attempt to shame / dump on Backblaze; if they resolve the problem I'll very much still recommend and use them. However, both sides of the issue (my idiocy/negligence and Backblaze's areas for improvement) seem to be worth discussing.

I don't use Backblaze, so I don't know what kind of upload speeds they have on restores, but 12+ hours for 6gb seems very slow (or the bottleneck is your download speed). It doesn't seem all that unreasonable for someone that's storing 340gb to download 6gb twice. Of course, and it's been stated already, that if this is critical data you should really keep another backup.

I see that you've made some effort to be transparent and fair, but I think this should have stayed as conversation between yourself and Backblaze until there's more substance to the issue at hand.

With that said, it sounds like there could be some good discussion on their restoration process.

I appreciate this comment. And you have a great point that I could have waited longer to post this. I've never posted to HN before with any sort of results; my goal wasn't really to rush this out there. It just seemed like a conversation that was worth having, and I didn't think through the fact that it would look like a very ranty customer before posting it here. Your perspective has been very helpful in that regard.

As a (minor) side note, regarding the download speeds, the bottleneck wasn't my speed I don't think (though I do have Comcast and so I'm sure the mileage varies). Right now I'm pulling 56mbps down and even when when my laptop was dealing with slow internet it was still 13-15 mbps.

> a little more finesse than merely using 7-Zip, etc.

RAR works great for this type of situation. Download the data, if decompression fails, download the required parity files. This is how binary downloads on Usenet work (where random articles may be missing from server retention).

Does anyone have restore experience with Tarsnap? Tarsnap seems like a great service, but I've never had to do a restore.

I've pulled individual files from my tarsnap archives, and did a full restore of a server into a VirtualBox VM to verify that everything worked OK and can confirm that tarsnap works as advertised.

Reply on news.ycombinator.com