How Backblaze Got Started (2017)
backblaze.com
backblaze.com
I’ve used Backblaze since 2010 and in the past year contacted support twice. In both cases, they were only interested in quoting esoteric policies and technical limitations rather than actually solving my issue. As in, “Oh that’s because our system works like this and this and has this limitation.” Ooookay, that’s not what I asked; I asked how to solve this problem of mine that your poor system is causing.
For example, the most recent time was when I started getting emails warning that all of my external drives were offline and my data would be soon deleted. Instead of “Very sorry about that, here’s how to fix the issue,” I got this long response about the ways their system looks for new files in serial and it can get jammed and start ignoring everything, with no apology, no acknowledgement this was their issue, and no solution. I had to go fishing for solutions and drag the information out of them to finally figure out what I needed to do. Which it turns out is to get back an internal drive (totally unrelated to the other drives Backblaze abandoned) I had physically removed and repurposed, put it back the computer, wait a long time for Backblaze to see it, then uncheck that drive in Backblaze and remove it again. This was a good example showing how the client design is poor and the support is worse.
The other two major technical issues that convinced me I need a new service asap are:
1. The client will lie to you and you never know what’s really backed up. Even if you use the secret alt-click to force a full drive scan, it can still miss files and tell you fully backed up when files from days ago are still nowhere to be found. Luckily I’ve never actually needed to do a restore, but I almost thought I did one time and would have been furious at all the missing files I noticed.
2. This might not matter to some people but absolutely no meta data is backed up.
I hate to badmouth them since the company seems like a really good citizen, they project a very positive image, and their software seems good on the surface. But when you dig in, there are a lot of issues with the software and service, and I recently realized having Backblaze wouldn’t actually reassure me in the event of a local disaster, so migrated away and I wanted to warn others too.
Backblaze have not acknowledged my issues at all, even though I have tried to engage with the team numerous times.
I switched to CrashPlan, and have had to do two full restores since doing so, both of which went flawlessly and quickly. The Java client is a bit cumbersome, but it's reliable, and I am much happier with the performance and support I've received from them.
I've used them for a long while and previously had good experiences.
I was recently adding a new backup to CrashPlan and was a bit miffed that my backup is capping out at around 3 Mbps (and down to about 500 kbps at times). Opened a support ticket, went through a whole bunch of things, and they basically came back saying 1 - 5 Mbps is the expected range.
In 2018, I really don't think that's acceptable when you're trying to backup a couple terabytes of data. I have not had this issue with other backup services I use.
Not suggesting you switch back off of CrashPlan (we like those folks and they do good work) but we've changed quite a bit since your last interaction with us and the issue you were experiencing sounds like it may have been one of the burrs we've able to knock off. Again, I'm sorry you had a poor experience with us, but glad you're using an online solution that's working well for you!
Let me know if you find anything worth switching to :)
( tangent: word of warning: it seems to silently ignore folders with the same path except for case on case sensitive disks. Fringe issue, but I only accidentally saw my two most important folders weren’t backed up.)
> I suddenly got a big fat warning about something, > yada yada long story short > ... I can’t give you access to the diffs because > that’s how our system works
I would love to understand what the specifics were. Backblaze's whole design is based on a "log format" where you really SHOULD be able to understand the diffs and exactly what occurred at what days and times. In other words, a complete, total, absolute history of everything that occurred is found both on your local system, and then (encrypted) on our servers.
What I mean by "log format" is that when a file is backed up, a line is added to what we call the "bz_done" files that says something like this:
+ /pictures/puppy.jpg
The "+" means it was added to your backup. Then if you delete that file later we don't REMOVE that line, we add an additional line to the very end of the log like this:
- /pictures/puppy.jpg
The "-" signs means it was removed FROM YOUR LOCAL DISK (but not yet from the backup). Then 30 days later, the final line might appear like this:
x /pictures/puppy.jpg
That means the "30 day roll back history" period has expired and the file has been cleaned up from the Backblaze datacenter.
The whole point is we can determine exactly what occurred at what times, and you can understand the diffs yourself.
Last year they lost one of my drives — the whole backup of the entire drive 100+ GB and support just threw their hands up. Of course I only discovered that they lost it after the physical drive failed.
Still looking for a better replacement. Arq didn't quite do it for me.
> I gave Arq a go but usability wise it has some catching up to do. It's hard to find an arbitrary file that's been deleted to restore unless you know exactly where it lived on disk. The search didn't work too well for me.
https://news.ycombinator.com/item?id=16303122
Having to do a few restores like that was pretty painful with the Arq UI. I ended up giving up on it and finding the data another way.
I'm sorry you had a bad experience with support, if you can post your ticket number, I'll take a look at it and make sure that the head of support sees it so they can do training on it if necessary.
It matters to me. There's a lot of metadata on MacOS files and this is the reason I don't use Backblaze.
> absolutely no meta data is backed up
> There's a lot of metadata on MacOS files
To be perfectly clear, Backblaze backs up and restores TYPE and CREATOR metadata (on the Macintosh), and restores the file creation date and the file last modified date. Backblaze does not backup anything else, for example, what color you chose for your icons in the Finder, and Backblaze does not restore the file and folder permissions.
This works for most customers, but some customers won't be happy with it. We just want to be clear what we do and what we don't do.
The concept of "file creation date" and "file last modified" are fairly universal, and can be restored "across platform". What that means is if you backup on Macintosh you can restore to Windows and vice versa. If you restore a Macintosh TYPE and CREATOR to Windows it will lose that data since Windows (NTFS) does not have that concept.
Windows has the most sophisticated (complicated) set of ACL (Access Control Lists) that anybody has ever seen, and frankly I've never met anybody who could fully explain why Microsoft went crazy on these (and Microsoft. Backblaze does NOT backup the ACLs on Windows. But NTFS and Microsoft ACLs will simply never survive the test of time. In 30 years your iOS phone or Android tablet will never be able to understand what occurred and why "joe" had permissions as "System" while "jane" only had permissions as "Administrator" on a photo of a puppy even though those logins are long gone.
Most people who are worried about backing these sorts of things up should look into "Drive Cloning" software, which is different than "backup against catastrophic long term photo loss". On Macintosh you can look into "Carbon Copy Cloner". Not sure what to use for Windows?
They'll suit 80-90% of customers, but I'd say there's a portion that don't do their due diligence and test every use case they have (which is fair enough, most people don't and most people don't know every use case they will have going forward).
But that is not what they do exactly:
"we do not backup your operating system, application folder, or temporary internet files that are transient and would not be useful in the future. Backblaze also excludes podcasts in iTunes."
https://www.backblaze.com/remote-backup-everything.html
Is there an online backup service that makes a complete image that you can just pop a boot CD in, and restore from online? That seems like the holy grail.
Granted, there's still benefit to doing a traditional "snapshot" style backup. Something like Backblaze is still a good idea. But arguably it's more of a belt-and-suspenders thing at this point. (Especially if your cloud storage offers version history.)
While I've loved Backblaze's blog posts over the years, seems like this was not on their radar ten years ago. They were seeing it just as a traditional backup/restore problem. I see they've since introduced a cloud storage product (B2), but it doesn't sound that competitive.[1] Here's iCloud for comparison:
- $0.005/gb/mo (2TB plan) with unlimited data transfer
- Baked deep into the OS, totally automatic
- Intelligently purges local files when storage runs low
- Integrates with Photos app for optimized media storage
- Syncs while the device is asleep (Power Nap)
- HSM-protected keychain backup (geeky but groundbreaking security feature)[2]
(Not so nice: doesn't do version history. Dropbox does.)
Again, I've really enjoyed Backblaze's posts and hope they continue thriving in this new landscape.
[1] https://www.backblaze.com/b2/cloud-storage.html
[2] https://www.blackhat.com/docs/us-16/materials/us-16-Krstic.p...
Then an unaccountable algorithm closes your Google account, and suddenly all that data is inaccessible. Your cloud storage provider's backups are not your backups.
Then a fire destroys your local backups, and suddenly all that data is inaccessible.
Like I said, traditional snapshot backups are still a good extra step, but cloud storage has brought very good backup practices to the vast majority of the population. The difference versus ten years ago is probably several orders of magnitude.
Note: Backblaze charges a fairly hefty $0.02/gb to download, I believe, so restoring your 1TB backup will actually cost you $20.
But my bigger point is, people are moving to cloud storage for other reasons (i.e. syncing devices, optimizing local storage) and they're getting pretty good offsite backup along with that "for free". iCloud is nicely priced if you're actually interested in cloud syncing. At 2TB it's even the same storage price as Backblaze, which is amazing, because it's a full-on intelligent syncing service that doesn't charge for data transfer either.
I'll also note that intelligent purging is completely optional. There's a checkbox to turn it off for documents and/or media (separately). But if like most people you have fairly reliable internet, it's a real benefit. It makes it feasible to put your whole photo library in the cloud (mine is >1TB) and have it all immediately accessible without having to splurge on massive local storage.
> Note: Backblaze charges a fairly hefty $0.02/gb to download, I believe, so restoring your 1TB backup will actually cost you $20.
Backblaze has two product lines, priced two separate ways:
1) Online backup for $5/computer. THIS IS FREE TO DOWNLOAD.
2) "B2" - this is "object storage" for half of one penny per GByte per month. This costs 2 cents/GByte to download. If you want to use B2, you cannot use the traditional "Online Backup Client" from Backblaze, instead you can choose a piece of software from this integrations page: https://www.backblaze.com/b2/integrations.html (scroll down, it is long). You can see pricing "per transaction" here: https://www.backblaze.com/b2/b2-transactions-price.html
Side note: the $0.02/GByte download price is definitely over-charging, and we're working to lower that. We pay 1/10th that amount wholesale for bandwidth. The fear was that if we drop our prices much more, we would become the world's cheapest CDN (Content Delivery Network) and if that occurs too fast it will overwhelm our download servers until we build out that infrastructure and gain some experience with running the world's least expensive CDN.
:-) I share your pain. I just found my own typo in a different comment and now my shameful proof reading skills will be preserved for all of time.
How recent is this research? Starting with Sierra (late 2016) Apple is including the entire Documents and Desktop folders in iCloud Drive by default.
They are more than happy to share what they build and we've used their storage pods plans to build 2 for work.
I also really wish they would let customers buy a hard drive with multiple backups on it instead of one hard drive per backup. It would be great to purchase one drive every year with all the backups on it and not have to worry about downloading backups for retention.
> Since the server (and not the Internet connection) was the bottleneck, it wouldn't have helped
As Yev mentioned above, we MASSIVELY sped up restores fairly recently (August 2017). We understood waiting for restores for a few days was an inconvenience for customers and I'm personally very proud of the results. Would LOVE for you to try it again, but I fully understand if you don't want to. Here is the blog post about the faster restores (and faster backups): https://www.backblaze.com/blog/cloud-backup-5-0-rapid-access...
Would you mind taking the time to respond to the less positive comments about Backblaze rather than just thanking the two positive commenters? That would make me feel much more comfortable relying on Backblaze in the future.
I also have plenty of complaints about the service (posted above), but there’s nothing weird about this policy as written.
What’s weird is when permanently connected drives get classified as disconnected (see my other post).
In fact, their marketing claims "Never lose a file again. Get unlimited cloud backup for your Mac or PC. Just $5/month."
Only on reading the help does it seem that if the client doesn't connect with the server within 6 months, then all your backups are gone. I'm not saying this is unreasonable, but it seems a bit counterintuitive given you are still paying for the service. I just still haven't found the bit about disconnected drives.
I really like Backblaze's articles on HDDs and they really seem to be customer focused, but I had to dig in the help to find this.
For something as critical as data backups, I prefer that those companies stay in business and charge something that's a bit more proportional to their underlying costs. That's honesty and fairness, in my view.