Backblaze lost over 13TB of my data
reddit.com
reddit.com
But ... 17 TB total? For online backups? That's just not what they're designed for, unless you've got one hell of an internet connection. The complainant says it's going to take over a year to re-upload the 13TB. That's insane.
I think you have to give up on online backups for that use case and settle for a pretty beefy home rig in a fire-resistant box. Moving that kind of data up and down an internet connection just doesn't work real well yet.
I don't have any kind of special connection, but when I switched computers had to go through the process of deleting my entire online backup in order to use my new machine. Took a few weeks (not a year) for 5TB from scratch, all the while on pins and needles hoping nothing happened to my local disk.
I love Backblaze, but their confusing and weird policy re: switching machines made for a nail biting and slow, painful process. After reading this, I'm now considering alternative offsite backups.
This isn't too much for online backup in general. 17t is not so bad in a lot of cases.
It might be too much for Backblaze. I don't know. I don't know if they can effectively back up at line rate. They couldn't for me.
3 days on a 500 mbit/second connection---which is about what you'd really be able to get if you had a 1-gigabit line.
It's fairly easy to get them in any US metro area so long as you are willing to spend. It's also percolated pretty well through out Japan and Korea.
To lower that with a significant up front cost, you could buy the drives yourself, perhaps shucking external ones and then send them over to Vapornode https://vapornode.com/storage-vps so they can host it at 10 USD a month per disk. If you have a Fry's close by, they have 8TB units for 200, recently Best Buy had them for 180 but I think that ended. So that's 200 + 10 / month vs 4.5 * 8 = 36 USD / month, you are saving 26 USD a month, it takes eight months for this to be cheaper. I would say it's not worth it because replacing dead disks are on you but it's up to you. Also getting many smaller disks make for better RAID.
Finally, lowendtalk regularly has discussions on bring your own disk providers. You need to know what you are buying but it's doable.
But it's very funny how it is practically the same 4 EUR a month a terabyte as the rent-a-server-for-its disks solutions.
Buy a used SC846 second hand, that costs ~$900 inclusive of rails and a SAS2 backplane upgrade. Then buy drives. You can get 8TB drives for ~$230 (Seagate Archive or shucked Reds) or $0.027/GB. Add cheap colocation for say $70/mo.
Assuming the hardware lives about 5 years, total cost is $900 + (70x12x5 = $4200) + num_gb x 0.027.
If you fill all the bays you're looking at 192TB for 5 years for $10284, or $171.4/month or $0.90/TB/mo.
>>> So, all in all just a waste of time and money.
Dude wanted to backup 18 TB of backup on a home connection until he realized it's not going well.
Messaged the customer support to complain => he got a timely reply, then decided to leave the service and got a refund.
That looks like good service to me.
P.S. I feel sorry for companies who have to deal with that kind of customers.
I mean it's not an ideal circumstance, but not the end of the world. He really just needs a faster connection to support that much data.
I have about 80GB with Backblaze right now -- just documents, photos and music on my laptop -- and the metadata it stores in /Library/Backblaze.bzpkg takes up 46GB, or about half the size of the actual backup. Most of this is in uncompressed plain-text log files showing every file which has been backed up, going back years. On a 256GB SSD that overhead hurts.
I asked their support, but their only solution was to delete the backup, reinstall the client, and start from scratch, which seems... less than optimal.
(On the other hand I do understand the impulse to squirrel away terabytes of data... I now make a point of archiving all of my mail accounts and PDFs of papers I read, knowing that with good search, they'll be valuable for years to come. But I haven't quite read 13TB worth of papers yet.)
Crap, just between the main drives on computers in my house I easily have 20-30tB of raw disk space.
It might be a lot to transfer...yeah. At 50Mbps that's what...a month of sustained transfer? But how valuable is your data?
1 - https://www.newegg.com/Product/Product.aspx?Item=N82E1682217...
UHD Movies? Why would you save those in the age of streaming? Family pictures/videos? At 20TB, you must have hundreds of thousands of pictures. There's not enough time in your life to look at all those! The only thing I can think of is that you shoot photo/video professionally, or maybe do some kind of 3D/video rendering?
But I agree with you - when you have this much data, why the hell are you transferring it to the cloud? Buy a couple harddrives, load them up, and stick them in a safe.
Heck, my parents have >10tB of raw storage between their home and work computers. Even though they aren't data hoarders, between software, important documents and email and videos and pictures of grandkids and friends they probably easily sit at 4 or 5 tB of total storage that deserves backup.
17TB is a lot to send over the wire. If you've got a residential ISP connection it'll probably take months.
You don't want to deal with cloud on a home connection, it's gonna take forever.
If you are a company, you pay for storage on S3, Google Storage, OVH or IBM Softlayer. It's great quality yet simple and moderate costs.
Companies keeping data in house pay for NetApp, NetBackup and tape hardware.
Lose data locally, restore from local NAS
Lose data on local NAS, restore from remote NAS
Lose data on remote NAS, restore from local NAS
I'm not sure what happened in this user's case - but my guess is that something happened on their machine where our backup client decided that data on the machine didn't match with what is in the DC and restarted backing up the data so that we'd have a clean copy, resulting in 13TB of data getting re-scheduled for backup.
Not the cloud if you have a typical home internet connection
Sia = $2/TB/month.
Amazon Glacier = $4/TB/month.
Time4VPS Storage = $4/TB/month.
Backblaze B2 = $5/TB/month.
Backblaze B2
[0] https://www.reddit.com/r/DataHoarder/comments/6j06qp/how_do_...
However, I'm not sure how many of their clients have single drives with 17TB of data. Perhaps this is an edge case that they didn't adequately test for.
And since a vast majority of their customers will not have more than 3TB on any single drive, I can see why this might not be well tested.
Not sure what happened in this particular case as I'm not familiar with it, but I'd venture to guess that something occurred on the computer that caused the client to think it needed to re-upload those 13TB of data, so it scheduled them for re-upload.
People like this are why Amazon's unlimited cloud storage just became limited. The people out there abusing the spirit of the offer know who they are. The finance world is FULL of those people, and it's a good part of why the economic world is really fucked up.
Be a team player. Be thoughtful of your peers and neighbors. We'll all end up better off, even you. /rant
And I don't think they're at fault like you're suggesting. Suppose I have a legitimate use case for storing 17TB of data and need to do it on a budget. During my shopping, I discover ten $1000/yr services for 20TB storage and three $100/yr services for unlimited storage. To me, the $100 services seem like scams or that they have no idea what they're doing. Amazon, Backblaze, and Dreamhost are literally giving off these vibes! Regardless, I'm on a budget so I purchase Backblaze and upload my 17TB. But in the back of my mind I'm nervous that they will shut off my service, despite the contract I entered with them to keep an "unlimited" amount of data for the price I paid. Or, if the contract says they have the right to terminate my service if they deem the amount to be too much, that would make nervous even as a normal user of the service, since the definition of "ridiculous" might be decided by some judge in a court if the service decides to cancel the contract and my company decides to sue. Would you really want a judge to decide that, or would you prefer a hard limit to be placed on the website and the contract in the first place? I truly don't understand the appeal of "unlimited" plans compared to simple hard limited, tiered plans.
Can you give me a theoretical example? Seems like saying, "Suppose I have a legitimate case for needing 45 trips to the buffet line."
I get your points, but I think outliers are outliers, and correspondingly they should be thinking as such. So if I had, for whatever reason, a need to store 17TB of data, I would need to be as clever about how I stored it as I obviously was clever enough to even possess 17TB.
When I was younger I worked at a recording studio. Each song consumed about 10GB, and it was convenient to return to the sources at a future date once the mix was delivered to the artist. Each artist recorded about 10 songs, and in a year we could do 50 artists.
I can't imagine what animation and video production houses go through per day. Having a backup is completely necessary for these folks. I could go on, but I feel 17TB is pretty common for businesses, so I'm not sure why you're asking the question. These days you can get 17TB for only $700.
So either the guy has a legitimate business reason (in which case he's definitely chosen the wrong service (level), or he's not using it for business.
I can give you a practical example. I do a lot of photography/videography in my spare time (not professional, have never been paid for it). I shoot RAW photos on a Sony A7RII. Every photo is ~85MB and I can shoot 700 photos in a day. Assuming I shoot one day every weekend, that's 3TB of data a year (that's actually a close estimate when I account for vacations and such too). Then there's video. I own a Panasonic GH5 and record on an Atomos Ninja Inferno, which outputs 4K 10-bit ProRes video at 60fps. This consumes ~1TB per hour and I tend to store it for archival purposes since it only costs ~$25 to do so for pretty much ever.
For a less extreme case, one of my old school teachers has been doing photography as a hobby for ~15 years. She's amassed ~10TB of photos over that time and currently stores them on piles of HDDs and a few spindles of DVDs scattered around her house.
And yes, I absolutely accept that I'm an outlier but any service that offers an unlimited plan is explicitly advertising itself to outliers. They're not putting themselves out there as someone with a 1TB, 2TB, 10TB or even "enough for 99.9% of people!", they're saying that they're giving storage with zero limits.
Your buffet comparison by the way isn't a good one. A buffet isn't truly unlimited, it has a time restriction, a condition that you can only eat what you take, you can't leave with it and the fact that a human being can only physically eat so much. They haven't defined a limit but there certainly is one.
Unlimited bandwidth, free drinks, unlimited storage, etc., are all offered with some expectation of limits, with occasional spikes in use.
But let's back up a moment. 17TB is a lot of data. If you were a professional photographer or videographer, I could possibly imagine that need. Or if you were doing some truly big data work, or some financial algo backtesting (on a really grand scale), maybe. Those people should be able to afford storage prices in line with their amount of data.
Secondly, assume you lost your local drive. How are you going to restore 17TB? Whose network (that you're taking advantage of) are you going to saturate with your restore?
We should be grateful these early outliers exist and break things for us in advance.