Hacker News Reader

> Client writes every backup chunk to disk before uploading instead of holding the data in memory, reducing the lifespan of customer SSDs for no reason? It’s fine, don’t worry about it[2][3].

There is basically never going to be a meaningful reduction in SSD life from that amount of wear.

Every file is written once (when you change it) and then a second time (when the client writes the same file into a backup chunk). Halving the SSD lifetime is not something I’d personally call meaningless, especially for QLC drives which have two orders of magnitude fewer P/E cycles than SLC drives[0], especially since there’s no good reason for writing these chunks to disk in the first place.

[0] https://www.techradar.com/news/nand-and-cells-slc-qlc-tlc-an...

> Halving the SSD lifetime is not something I’d personally call meaningless ...

No, it doesn't halve the SSD lifetime. It is a little odd of you to make that claim.

First of all, you are assuming that 100% of SSDs die from being written, and die so quickly and so often from being written to that this is an enormous concern everybody using an SSD in a laptop should be worried about. In reality, a modern SSD you buy today might last you 10 years of normal use (including running Backblaze), but the rest of your computer simply won't last that long. Personally, I've never had one of my SSDs die. For any reason, including too many writes. Often I throw them in the trash perfectly working because I have upgraded their size for less money. Other times I get a whole new computer which comes with a whole new SSD. Backblaze runs SSDs in all of our "monitoring" computers that write the ever loving bejeebus out of their SSD drives 24 hours a day, 7 days a week FOR YEARS. I think we've had 1 SSD go bad after YEARS of writing at a rate 10x up to 100x higher than any home laptop user could achieve. And we're not sure it died due to too many writes.

So even if Backblaze doubled the writes (it doesn't, see below) it wouldn't have any measurable effect on your SSD's life. This is just provably false by watching servers that aren't running Backblaze write 10 or even 100 times as many times for 5 years as any consumer could ever achieve, and they still don't kill the SSD drives.

So, is Backblaze doubling the writes on your computer? Heck no. Backblaze only backs up valuable data files, not your operating system and certainly not your log files. The vast majority of writes to an SSD are not writing your one photo that you took today to disk - they are database transactions and log writes and system log write and system registry writes, etc. Backblaze puts in a lot of effort to exclude "chatty" log files (log files that are written all the time, all day long) because it saves our customer's network bandwidth and saves Backblaze space in our datacenter. So IN REALITY Backblaze's behavior is adding maybe a small single digit percentage of additional writes. The exact profile will matter what you use your computer for, but take email -> if you get 200 emails a day saved into an Outlook PST Inbox you might have 600 write transactions per day. But Backblaze won't back that up after every email, it would waste too much customer bandwidth. Backblaze might only back that up once per day. And only the part of the PST file that changed! So in reality, Backblaze only added 3 or 4 writes out of 600.

> especially since there’s no good reason for writing these chunks to disk

Different customers have different needs and different profiles, and Backblaze has to accommodate them. One reason a full snapshot is taken of the large files is that Backblaze wants a consistent "snapshot" of a file. On slow network connections it takes some customers 4 days to upload their Outlook PST file ONCE. Meanwhile they keep getting mail. If Backblaze didn't take a clean snapshot at one moment of time of this file, you'd get this corrupted file where the first 10 MBytes were from a file on Monday, and the last 10 MBytes were copied on Thursday from the same filename but totally different contents. Surely you see how that might be an issue?

And customers may or may not have the RAM required to hold a 10 GByte file in RAM, and none of them have enough RAM required to hold a 500 GByte file in RAM. Backblaze needs to store these "snapshots" someplace, so it stores them on disk.

Saying this is "for no reason" is disingenuous.

> And customers may or may not have the RAM required to hold a 10 GByte file in RAM, and none of them have enough RAM required to hold a 500 GByte file in RAM. Backblaze needs to store these "snapshots" someplace, so it stores them on disk.

But at the same time, customers are very likely to have enough RAM to hold a 100 MB file, and definitely have enough for a 10 MB file in RAM. It might be better (faster, easier, less abusive to the drive) to store small files in RAM (for some well-chosen value of “small”, of course. Likely one that depends on the system RAM size and bandwidth.)

> to store small files in RAM (for some well-chosen value of “small”, of course...

The Backblaze client code currently puts that dividing line at 100 MBytes. Any file less than 100 MBytes we call a "small file" and those don't get subdivided into 10 MByte chunks on disk, they get slurped up into RAM and just held there for the entire duration of being sent. For that code path, Backblaze can be configured to send up to 30 of these 100 MByte files in 30 separate threads which means (if the customer configures it this way) it can eat up 3 GBytes of RAM at peak. If you have that much RAM, it can really rip. If you don't, we recommend using fewer threads. :-)

The sub-division into 10 MByte chunks taking a one time "Snapshot" of the large file is for all files larger than 100 MBytes.

Wtf am i reading here:

"HTTPS doesn't hurt anything and actually has some nice side effects (it separates backups onto a separate port than the majority of your web traffic, allows for traffic shaping if you want to do it)."

Disclaimer: I work at Backblaze so you should check up to see if what I say is true and keep me honest.

> Zero-knowledge encryption? You don’t need that[0][1].

This is not our position, and I feel it is disingenuous of you to say that. If you read YOUR TOP LINK from TWO YEARS AGO I explain that Backblaze specifically offers 4 levels of security, one of which is Zero Knowledge, and we think that is a perfectly valid decision for some customers. If you want zero knowledge, choose it at Backblaze! But some customers have other requirements, and you are insisting that we remove part of our product line up that is very useful to other people.

Here are the four levels of security Backblaze offers, most of this is from this post 18 days ago I wrote: https://www.reddit.com/r/backblaze/comments/kroqhn/private_e...

1) Security Level 1 - no security. Backblaze B2 can serve public websites, on purpose, the way stuff that you want to go viral and share with everybody. https://www.ski-epic.com is supposed to be readable, not locked. These are totally open files for anybody to download.

Security Level 2 - username/password/2-factor. This is a good choice for the customers who would rather error on the side of recovering their passwords than losing all their backups. In this level of security, your Online Backup is secured by your username and password, and every file is "encrypted at rest" (all the files are always encrypted when stored on disk). In this mode, all it takes to decrypt your backup is to sign into the Backblaze website with your username and password, and 2-factor verification, and you can prepare a ZIP file restore to download. You can ALSO prepare an encrypted USB restore hard drive to be sent to your home. This particular level of security has the advantage (or disadvantage to the security sensitive) that if you forget your password, you can "recover" it through your email account. If you use 2-factor (like we recommend), a hacker with your username and password will STILL not be able to gain access to your files. This is a good choice for a customer who is not super overly concerned about hackers possibly getting their data, and just wants to backup a public website like https://www.ski-epic.com (which anybody could get from the website anyway), or some photos of their wedding. It errors on the side of being able to recover the data no matter what. Some things you want BACK more than you want to destroy the files in the event of a hacker breach, or if you forget your password.

Security Level 3 - Backblaze Personal Backup with a "custom" unrecoverable private encryption key. In this mode, your account is protected with your username, password, 2-factor like the above "security level 2", but also an ADDITIONAL "unrecoverable" passphrase that Backblaze does not know in any way, shape, or form for years. Without the passphrase your files cannot be decrypted. You only provide the "passphrase' in the event of preparing a restore, and then your passphrase is never stored on disk anywhere at Backblaze, it is held in RAM. For years your files are encrypted at rest where even if Backblaze is ordered by government subpoena to hand over your files Backblaze cannot comply even if we wanted to, we have no way to decrypt your files. If you choose this level of security, DO NOT FORGET that passphrase because there is no possible way to "recover" it, and without it your files are GONE. You cannot recover them, Backblaze cannot recover them, the CIA or FBI cannot recover them - they are GONE. Now, as long as you don't forget that passphrase, then years later when you actually need to prepare a restore, there is a security "window of exposure" for as little as 20 minutes ONLY IF (and when) you go to restore. If you are under arrest -> just don't prepare a restore, and the FBI simply cannot get the contents of your files. An alternative strategy is if you have some particularly sensitive files, like incriminating evidence of your crimes or your tax returns or a file with all of your passwords to your bank accounts, put these few files in a small encrypted file on your laptop, and EVEN IF you prepare a restore the FBI (or Backblaze, or hackers) cannot get the contents of those files. Now, the reason we allow the customer to provide this passphrase is it is STILL relatively friendly, and we can prepare 8 TByte USB restore drive (that is encrypted) and sent to the customer's home. While there is that tiny exposure if the restore servers were ACTIVELY hacked during the 20 minutes while the restore is being prepared, some customers (especially if they are just storing wedding photos and cat pictures and public websites) prefer this option. Many of our customers are naive (not computer expert) customers, and many, many, many customers find this particular security level, convenience, and tradeoff useful.

Security Level 4 - "Zero Knowledge". Customers can use Backblaze B2 with a zero knowledge product such as some of the products listed on this web page: https://www.backblaze.com/b2/integrations.html Some of those 3rd party tools are even open source if a customer doesn't trust commercial products and wants to read the source code. This is a very useful security level for customers that would rather LOSE THE DATA than ever have it intercepted by law enforcement or a hacker. Backblaze offers this level if you want it! However, there are some very real world drawbacks of this level of security. First of all, Backblaze cannot prepare an 8 TByte hard drive with all your files organized correctly as they were backed up and send it to you fully organized, because "zero knowledge" demands Backblaze never, under any circumstances, know any of your file names. This is a more secure system, but it is less convenient to restore. Also, some of our customers don't have the bandwidth to download 8 TBytes conveniently, so you may have to pay more money for a faster internet connection to make this type of backup work for you. The other thing that is "dangerous" or less convenient and might lead to data loss in this scenario is that if a customer stores the "Private Key" on the laptop that is being backed up, and the laptop SSD dies, they actually lose the backup also, because you need the keys to decrypt the backup. So any customer who chooses this security level needs to make several copies of their "Private Key" they never give Backblaze, probably on multiple different external hard drives in their home (in case one of those copies of the key "goes bad" you need multiple copies). Beware of a house fire that destroys the laptop, and all the extra copies of the security keys, because this will result in loss of the backup also! So one idea is to store the keys in a DIFFERENT online service (or two or three online services) elsewhere on the internet (not at Backblaze, because that is what is demanded by "zero knowledge"). Again, this is a great choice for customers that PREFER DATA LOSS and extra time and thought and effort and cost over allowing the FBI or a hacker to gain access to their files. This is a perfectly valid choice, and Backblaze offers it.

Now sometimes people accuse Backblaze of not being a "zero knowledge" backup. What they are saying is that only #4 is "valid" and they really want Backblaze to stop offering #1, #2, or #3. I reject their insistence that we not offer easy backups for cat pictures and that customers must all share their technical ability (and bandwidth, and time) to download the encrypted data instead of getting a USB restore hard drive prepared with all their files in a friendly fashion sent to them. Some customers have different use cases, and Backblaze strives to support all four of these use cases! Pick which is right for you!

> This is not our position, and I feel it is disingenuous of you to say that.

Every time I’ve seen someone bring up ZKE, you demur or say that most of your customers don’t need it because they are non-technical and their data is not sensitive enough to warrant that level of protection. Every time, you strongly imply that the only people who would need zero-knowledge encryption are criminals. (You’ve just done all of this again here.)

If your position is not what I claimed, and you do think ZKE is something users need, I’d recommend that you spend your engineering budget on implementing it, instead of using that time to do things like rename your Git branches[0].

> Now sometimes people accuse Backblaze of not being a "zero knowledge" backup. What they are saying is that only #4 is "valid" and they really want Backblaze to stop offering #1, #2, or #3.

Could you please give a link to an example of someone actually saying this?

> Security Level 3 - Backblaze Personal Backup with a "custom" unrecoverable private encryption key.

The private key for decryption is automatically generated and sent to Backblaze without any passphrase at install time. It doesn’t get replaced when the passphrase changes, so the original unprotected private key could just be stored and reused later. Setting a passphrase sends it to Backblaze (necessarily, since the private key is only stored on Backblaze servers), so that too could easily be stored (accidentally or intentionally). Restoration requires decryption on Backblaze’s servers, which both requires the passphrase to be sent and also puts all the user’s decrypted data onto persistent storage.

All these design flaws make “Level 3” not meaningfully more secure than “Level 2”. Because of this, it seems to me to exist primarily as a marketing tool, rather than to meaningfully protect users from threat actors. I think we will probably have to agree to disagree on this point.

> Security Level 4 - "Zero Knowledge".

This is not a thing that Backblaze offer. Saying “hey, we have storage, use some third party software to get actually encrypted backups” is no different than me selling some server space and advertising it as “zero knowledge”.

> This is a more secure system, but it is less convenient to restore.

Outside of the issue of forgetting your backup passphrase, there’s nothing more inconvenient about a properly designed zero-knowledge backup. In fact, I would say that the way Backblaze works right now—requiring users to go to a web site to download a zip file, or have a disk sent in the mail—is way more inconvenient than most other backup software which restores directly from the client, including ones with ZKE like Arq and SpiderOak.

[0] https://www.backblaze.com/blog/code-and-culture-what-happens...

The security issues are disconcerting. And their attitude is not encouraging at all.

What are some decent alternatives to Backblaze?

"What are some decent alternatives to Backblaze?"

If only there were a cloud storage provider that gave you an empty UNIX filesystem to do anything you want with, using any tool that worked over stock-standard SSH.

If only ...

To be apples-to-apples, you (2-2.5c/GB) do charge 5x what B2 charges (0.5c/GB) (for just the storage part) for small customers.

For a lot of people, a 5x price difference disqualifies the term "alternative", although you are in line with industry/S3 pricing. It's just that Backblaze has specifically differentiated by being super duper cheap.

It's why I use them for my personal projects.

Agree. It’s relatively straightforward to underprice ‘brand name’ cloud object storage, S3 and the like.

rsync is likely making a margin choice here rather than marketshare choice.

For people who, like you, may not be aware - there is a "HN Readers" discount as well as other (.edu, FOSS author, etc.) ways to chop the headline price down substantially ...

Your snark aside, I pay $5 per month for Backblaze to backup my entire PC. I would pay many multiples more for rsync.

Just an aside, but I found that I was overpaying for backblaze backup. I switched to duplicati backed by backblaze b2 and went from $5/mo. to more like $1.70/mo. for the backup. As always though, YMMV.

You shouldn't really worry about object storage security, and your approach to using it should be as if everything you store in/on it will be published in the newspaper tomorrow.