Hacker News Reader

Yev from Backblaze here -> rest assured that we do read what you're writing on these posts and they've spurred some internal process discussions. I believe the bugs you mentioned were cleared/fixed with version 7.0.0.439 which was released in Q1 of 2020. We did leave HackerOne and switched over to BugCrowd to handle our bug program. It's private at the moment, but easy enough to get invited (by emailing bounty@backblaze.com). While we spin that program up (it's a new vendor for us) we may stay private, but hopefully that's not a permanent state.

Edit -> I just noticed the Daniel Stenberg libcurl citation. Oof, yea that certainly a whiff on our end. Luckily though we were able to make up for it (he has a write-up here: https://daniel.haxx.se/blog/2020/01/14/backblazed/).

> rest assured that we do read what you're writing on these posts and they've spurred some internal process discussions.

OK, that’s good to hear. Nobody from the company has reached out to me so this is the first time I’ve been made aware. The only public replies I’ve seen up until now seemed to focus exclusively on just the public bug bounty part, which is really the least important part of this whole thing.

> I believe the bugs you mentioned were cleared/fixed with version 7.0.0.439 which was released in Q1 of 2020.

It’s really critical to be transparent about this stuff and tell your users. You published release announcements for subsequent versions and there were no mentions of security issues being fixed. When you don’t do this, it looks like you’re intentionally trying to hide vulnerabilities from the public. This is not how any company should act, especially not one that promotes how radically transparent it is[0].

> I just noticed the Daniel Stenberg libcurl citation. Oof, yea that certainly a whiff on our end. Luckily though we were able to make up for it […]

I reported license violations in a ticket and nobody replied.

You did fix the libcurl violation, which is great, but it took a letter from the author, which is less great.

You are still violating the OpenSSL license.

It honestly baffles me that nobody at Backblaze thought to check the licenses of the other OSS libraries that you’re distributing after receiving a notice that one of them was being violated. It’s not like there’s a huge compliance burden or complicated dependency tree to evaluate—as far as I can tell, it’s zlib (which requires no acknowledgement), libcurl (which does), and OpenSSL (which also does, for the version you are using[1]). This would’ve taken like 30 seconds.

[0] https://www.backblaze.com/blog/transparency-in-business/

[1] https://www.openssl.org/source/license-openssl-ssleay.txt

Disclaimer: I work at Backblaze.

> as much as I hate it I'm following Backblaze around and posting incorrect information about them

I get the impression Backblaze did something to upset you. Can you let me know what it is so I can try to fix it?

If there wasn't a pandemic on I would invite you to come to our office and I could buy you lunch and I could try to make up for whatever we did to upset you.

Hi Brian,

I'm not the person you responded to and I'm a happy user of both Backblaze and B2, but I wanted you to know that this response by you reads as quite disingenuous. You seem to want to shift the reason for his disgruntlement with Backblaze from all the reasons he already mentioned to some other, imaginary slight that you indicate you'll do your very best to fix. How about just reading his very real gripes and responding to those?

Let's take his twitter thread for some highlights, these seem like very real reasons to get upset, maybe you "can try to fix" those? * Backblaze changed their client to add an allowlist some time after my report, while also intentionally breaking their TLS code so it would accept INVALID TLS certificates. Thereafter, the local code execution vuln became a full blown RCE vuln.

* When I submitted my report 11 months ago, they told me they already knew about the problem, downplayed its severity, dodged follow-up questions, didn’t seem to understand how CVE IDs work and refused to issue one after being asked four times. It was not confidence-inspiring. The CVE ID for the vulnerability I gave to them is CVE-2019-19904. They should’ve announced it, but they never did. Actually, they never seem to voluntarily disclose any security bugs… there are a lot of verified, closed, undisclosed bugs on their HackerOne account.

* This is all in stark contrast to their security page (https://backblaze.com/security.html) which makes many claims about best practices, and their blog & social media which present a sense of radical openness. I used to like their blog, but it all feels so gross and dishonest to me now.

* Backblaze mislead users about PEK. The decryption key is sent to their server, and so is your password. The only way to restore data is to decrypt it on their servers first. It is not a zero-knowledge system. PEK data is not ‘inaccessible’ to them. They don’t care.

At face value (and I haven't done any digging of my own) these all seem like valid reasons to distrust Backblaze. Not necessarily because they happened, but because of the way Backblaze has addressed them (read: apparently not)

> this response by you reads as quite disingenuous

It wasn't intended as such, I really meant it. I'd like to get to the bottom of this, understand what this person's true issue with Backblaze is.

> these seem like very real reasons to get upset, maybe you "can try to fix" those?

What the user is doing is called "Gish gallop". This is a technique where somebody makes a rapid fire list of unrelated half truths or misrepresentations, each of which takes CONSIDERABLY longer to address than to claim. And I've repeated explained why they are invalid, but the user just shows up a day or two later and makes the same exact list of complaints. No edits, no admitting that even one of the complaints is invalid. Gish gallop.

This is not the behavior of somebody that is genuinely interested in having Backblaze address or fix that list of issues. There is something else going on, and I personally would like to know what it is. First of all because I'm curious what the issue is, second of all I hope I can fix whatever the real issue is.

I'm not going through the whole list because I've done that maybe 10 - 15 times so far? But let's take this one, because it's spectacularly false, this person KNOWS it's false, but this person repeatedly makes the claim over and over again:

> Backblaze mislead users about PEK. The decryption key is sent to their server, and so is your password. It is not a zero-knowledge system. They don’t care.

Backblaze has 4 security levels, one of which is zero-knowledge, and we ENCOURAGE customers to pick the correct level for themselves. You can read my longer, in-depth answer to this same user just 2 days ago here: https://news.ycombinator.com/item?id=25904473 or you can read my longer, in depth answer 18 days ago here: https://www.reddit.com/r/backblaze/comments/kroqhn/private_e... or you can read my answer TWO YEARS AGO in the link this person supplied you (!!!!) or you can go back to the beginning, 13 years ago, when Backblaze started, where we explained EXACTLY how our encryption worked the same as the Microsoft Encrypted File System ("EFS") here: https://www.backblaze.com/blog/how-to-make-strong-encryption...

Now, despite it being a spectacularly false accusation that has been documented and explained so many times in so many forums, this user will undoubtable show up in another couple days and make this claim again. All the user's claims are like this. Obviously something else is going on.

I just wish that user would tell me what the real issue is. I can't fix what I don't know about.

> I just wish that user would tell me what the real issue is. […] I hope I can fix whatever the real issue is.

It is exactly what the parent already told you. That’s it. That’s “the real issue”. There is nothing more. Everything I’ve said already is, in fact, what the issue is. Please stop trying to read between the lines.

That you refuse to accept mine and others’ arguments about PEK and ZKE and SSDs is one thing. It’s an entirely different and more alarming thing when you refuse to accept that these issues are the issues and insist on continuing to spin a story about how I must be really angry about something different when other people are telling you it’s not so. I also can’t even imagine how it would’ve seemed like a good idea to fabricate a quote and attribute it to me in the way you just did. You did this on some of your earlier posts, too.

As for me, I don’t use eristic techniques, I don’t tell intentional falsehoods, and I don’t do things as you imply in saying I’ll “show up in another couple days and make this claim again”. Anyone is free to look to my comment history and see that I’ve only made a handful of comments here about Backblaze[0], and in all cases, I try to make sure my comments are fair and well-researched and backed with citations whenever possible.

I understand how this company is like your baby and so it may feel emotionally like I’m trying to kill your baby with criticism, but please understand that that is not my goal. My goal is, and always has been, to keep users secure. If that means I can help a receptive vendor improve their software, excellent. If that means I have to warn users to stay away from a vendor who behaves poorly, that sucks, but I still feel an obligation to do that too. If some of the negative publicity gets a vendor to start doing the right thing, good. That’s the whole reason I have to talk about these things. It certainly doesn’t do me any good otherwise.

[0] https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

> you refuse to accept mine and others’ arguments about PEK

Not at all! I agree with you COMPLETELY. You want a zero knowledge backup product, Backblaze offers that and makes a large amount of money from it (millions of dollars annually actually), and we think you (csnover) should use that product because that's what you want. I understand the arguments, and you are COMPLETELY correct and I accept your arguments - that's why we built it and sell it, because you are correct.

One of our OTHER product offerings (in addition to the zero knowledge backup offering) is to host public websites. Public websites just can't have zero knowledge. We offer both products SEPARATELY, I've explained this over and over again, but you'll just post in two more days saying "Backblaze thinks zero knowledge is bad" (spectacularly false). Public websites (by definition) cannot be zero knowledge, do you comprehend this?

> I don’t tell intentional falsehoods

You keep saying we don't have a zero knowledge backup system (wow, spectacularly false) and say we think zero knowledge is bad (again, spectacularly false since I've stated repeatedly that zero knowledge systems are THE BEST security). Then you say you don't tell intentional falsehoods. Come on, it's obvious something is going on.

> I’m trying to kill your company with criticism

Is that it? Why do you care? Did we do something to you? I have a hard time believing you put in all this effort because you flipped a coin and decided you would try to kill Backblaze. Why us? I can't fix what I don't know about.

I look forward to your 200 future posts about how Backblaze doesn't offer zero knowledge backups.

I also genuinely hope that I can incentivise them to actually start following software development best practices, since they do provide a valuable service and I’d like them to succeed.

I read thru your HN posts and twitter and I completely understand where you're coming from. Very good points.

But here's my response to you: You're paying them $6 per month. That's not enough money for them to do better than what they're doing now. They literally can't afford to do better, given their paltry income stream. (Just my guess, I have no insider info about them).

If they keep doing what they’re doing now, I absolutely expect to see a massive data breach and/or data loss event at some point

I guess that's a chance they're prepared to take, though they probably haven't thought about it in such stark terms. I'm sure their development practices will improve, maybe in time to prevent a massive breach or loss.

Disclaimer: I work at Backblaze so I'm biased and you should keep me honest.

> they probably haven't thought about it in such stark terms

Backblaze takes security incredibly seriously, and I assure you we have thought about it in such stark terms.

Expounding on that: Backblaze never raised any significant VC funding, we survive entirely on sales of our products, and most of that is keeping our customer's data utterly private, confidential, and safe. As this is the business we are in, our reputation is INCREDIBLY important to us. If we have a major breach we'll most likely lose our customer's trust, lose all our customers, and go out of business.

Since this is all I've done for the last 14 years, and my income and life savings is all wrapped up in Backblaze, I take this issue as seriously as a heart attack. So do my business partners (Backblaze was founded by 5 equal partners.) We're also up to around 200 employees who all would suffer greatly if we lose the trust of our customers.

Internally, Backblaze has a "Security Council" of software engineers and technical operations people with something like a combined 150 years of security experience and obsession. One of these council members got his CS degree from MIT and is both one of the smartest people I've ever worked with (we've worked together at 3 separate companies so far over 25 years) and also deeply paranoid and stressed out all the time. Another of the security council members has a PhD in computer science. And so on... They watch over all the design proposals, the APIs, the technical infrastructure, everything at Backblaze. They propose and implement new procedures, new programs like our BugCrowd program where we have external white hat hackers constantly trying to break in. We are also going through an internal security audit right now paying consultants to get yet another perspective.

In addition to the Security Council, all software engineers and all technical operations people are expected to worry about security all of the time. It is quite possibly our most talked about, most worried about, most important thing we do.

> I'm sure their development practices will improve

We always, ALWAYS strive to do better and do more.

This seems on the money: https://news.ycombinator.com/item?id=25913516