Hacker News Reader

Comment by benja123 | Hacker News Reader

It’s bad that Backblaze did not do their due diligence while integrating with Facebook pixel, but the bigger problem is the tendency of third party integrations having default settings that are overly aggressive when it comes to access of user/site data.

This should be a warning to every developer, when you integrate with any third party - don’t just copy the default snippet of code they recommend, read the docs thoroughly and then test/monitor what is being sent back to the third party. And if you are writing a third party widget, be considerate and make the defaults the least aggressive possible when it comes to accessing user/site/server data.

View on HN

At this point, every developer should (and needs) to be aware of how invasive any tracking code they blindly copy/paste into a website. I think it's required from people who operate normal websites, blogs, etc... but missing it is somewhat forgivable, especially from people just starting.

But the level of irresponsibility with customers' most private data from a company who's MAIN JOB is to protect it is absolutely shocking. Yes, it's the freaking pixel that does the tracking, BUT IT'S THEIR RESPONSIBILITY TO KNOW WHAT THE HECK IS HAPPENING ON THEIR WEBSITE. Don't they have any sort of vulnerability assessment or security code review? Their reply tweet is almost as infuriating as how something like that could even happen to begin with. Like yeah, sorry, ain't our fault! It's astounding.

I considered using backblaze a couple of times and now I'm very happy that I eventually didn't.

> Don't they have any sort of vulnerability assessment or security code review?

I bet people just don't realize that the frontend code could end up being a source of major data confidentiality vulnerability. The threat modeling, auditing etc. usually just concentrates on attack scenarios involving the backend to save money and keep frontend development a bit lighter on the security review process side.

Does not make it excusable of course, just means their threat modeling was inadequate. But probably explains how this was able to sip into production.

I dunno, you don’t need to be particularly concerned with security to understand that you do not need a facebook tracking pixel on your ‘paying customer’ UI.

Facebook will be pitching to your Marketing folks that's exactly where you need it.

Facebook want data on what actions users took before signing up, which users actually signed up and started paying, and how that relates to revenue. This UI is exactly where they can determine these types of actions.

Whether this actually makes Facebook better at marketing or not is a good question.

That's why it's a killer mistake to let "marketing folk" dictate _anything_ that has to do with the app. They can suggest, but not dictate. If this points to anything it'd the dysfunctional development process at Backblaze.

>That's why it's a killer mistake to let "marketing folk" dictate _anything_ that has to do with the app.

Killer mistake of your revenue, or killer mistake of what we'd prefer apps to be?

In this case it's certainly going to cost them some revenue, not to mention the EU might slap them with a nice GDPR fine for leaking PII.

> Don't they have any sort of vulnerability assessment or security code review?

I haven’t seen any evidence that they do. The last time I brought up their history of bad security practices on HN, one of their co-founders decided that the correct course of action was to come on here, accuse me of being a bad actor, and repeatedly make up quotes I didn’t say.[0] All because I tried to warn others in the community that something just like this was likely to happen again. And now it has. So, you know.

[0] https://news.ycombinator.com/item?id=25919105

Wow. After reading about the FB tracking I was wary about Backblaze but teetering on the edge of willing to give it a pass if they fixed it. But after reading brianwski's comments in that thread, the arrogance and unprofessionalism (especially in his last comment) just completely turned me off. That attitude goes beyond a technical fuckup or bad marketing move. I'm moving my backups out of Backblaze today and won't be looking back.

not to me. sounds like someone who is tired of dealing with a user that has an axe to grind for years

OK, except I’m not a Backblaze user and I never have been (except for the 14-day free trial). I haven’t had any private correspondence with them since reporting the vulnerability I discovered in 2019. This exchange on HN was the first time I’ve ever knowingly[0] interacted with this guy. If he has actually been ‘dealing with [me] for years’ in the way you imply, it has been a very one-sided relationship.

As far as having an axe to grind goes… if wanting to protect others is grinding an axe, I guess I’m guilty of that. I don’t feel like a handful of topical messages warning people of a legitimate and clearly ongoing problem is some abusive behaviour on my part, but maybe I’m wrong. I’m happy to learn from others’ perspectives, since I’m sure I could be a more effective communicator.

[0] I suspect he was the one who replied to my vulnerability report since the same attitude was on display in those messages too, but I don’t know since that account was just named “bbqa”.

for years

Astonishing arrogance. Their replies on current incident are also dismissive and arrogant. I can’t even imagine what kind of culture they have internally.

> I considered using backblaze a couple of times and now I'm very happy that I eventually didn't.

Same. I ended up going with spideroak because it was too hard to figure out what other providers were even promising for privacy, and it’s been fine. I’m thinking maybe of trying rsync.net with duplicity one day

> It’s bad that Backblaze did not do their due diligence while integrating with Facebook pixel

Here's the thing though, being a third party, due diligence means you constantly have to check what they're doing.

Because otherwise how do you know when they suddenly change their script to do something entirely different?

This is why having any 3rd party scripts on a dashboard service like this is in my view entirely inexcusable.

When I go to visit my cloud-based dashboard for Acronis' backup service[1] there's one single domain involved, and that's how it should be.

[1]: https://www.acronis.com/en-us/products/true-image/

> Because otherwise how do you know when they suddenly change their script to do something entirely different?

https://developer.mozilla.org/en-US/docs/Web/Security/Subres...

Not bad, but as far as I can see it requires you are _really_ sure no further scripts are dynamically loaded.

Or is there a way for the server to specify all resources must have SRI?

Yes, by using CSP

Nice, at least it's possible.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Co...

You can combine CSP it with SRI hashes and also report violations to the backend.

Why would you allow third party scripts to be updated by anyone but you?

This trend of dynamically linking to other people's shit needs to stop.

If you're updating them then they're not 3rd party scripts.

> Because otherwise how do you know when they suddenly change their script to do something entirely different?

Even though it's inconvenient maybe we should treat it as just another 3rd party dependency that needs to be downloaded, screened, and then used from the internal store. Pretty dangerous to dynamically load a script from a site like facebook.com.

>Because otherwise how do you know when they suddenly change their script to do something entirely different?

(a) Sandbox it.

(b) Have them sign a contract they won't do so but only do these N things, and if change those without telling you, sue them.

As much as I loathe and despise Facebook, this one is on Backblaze: they integrated with a well-known evil (seriously this surprises absolutely nobody) and should have been more careful with their settings. Or, you know, not have done it at all.

Why does Backblaze even need Facebook integration anyway? That seems absurd to me.

> but the bigger problem is the tendency of third party integrations having default settings

It's certainly a problem, but the biggest problem is simply that Backblaze doesn't need to integrate Facebook pixel to their web interface, especially when users are logged in.

There is absolutely zero benefit to this for their users, but I also fail to see what it could bring to them as a company??

They have now created a major PR problem for themselves, and what did they get in exchange?

(Ok, as the saying goes, there's no such thing as bad publicity. But still.)

The FB pixel is how FB gets to know about conversions. If your company runs FB adds, FB encourages you to let them know about which users converted to paying customers so it can improve the targeting of your ad campaign. And if you offer a free trial and the upgrade to paying customer happens inside the user dashboard, presto-bingo Marketing now needs you to install the FB pixel there.

It's disappointing that this is what the internet has become, and I'm happy to see issues like this brought forward. I can see value in feeding back conversion events to an ad network, but the "just let us run code on your page" style of integration needs to stop. Give developers some API to explicitly send such an event, if they really need to.

Almost every ad network does give you some type of API like what you're talking about, but unfortunately they usually actively discourage use of it.

Well yeah, the API isn't going to let them keep tracking your users.

The real warning is this one: Disable any third party tracking scripts on all user account pages. Assume they are hostile party's attack vector.

If you want to retarget me on FB, fine, do it on a marketing pages of a website, but don't let the trackers anywhere close to my private data.

>And if you are writing a third party widget, be considerate and make the defaults the least aggressive possible when it comes to accessing user/site/server data.

Unless your business model is literally collecting all personal and private information, then don't do it.

>>It’s bad that Backblaze did not do their due diligence while integrating with Facebook pixel

How much "due diligence" should be required to know that adding anything from facebook INSIDE your backup / file storage product is a bad idea... hell adding anything from facebook at any product should be considered a bad idea...

> and then test/monitor what is being sent back to the third party.

Trust, but verify. Especially FB.

I see a lot of comments about a paid service using tracking pixels. I understand the backlash but this is also how companies that are taking your hard earned money improve and iterate on their products. I know people don’t like tracking but usage analytics are invaluable to improving the product. There’s only so much customer engagement you can do, and often that leaves really unexpected outcomes in the dark (if you asked most customers, they’d ask for a faster horse).

The issue here is that Backblaze injected untrusted code into their core product which by its very nature is extremely sensitive. I’m not really sure what they were thinking. Rolling your own tracking is a pain, but some security contexts demand it.

> Trust, but verify. Especially FB.

In the particular case of Facebook many of us simplify to "don't trust if not absolutely necessary" ;-)

I removed all Facebook many years ago.

In the particular case of Facebook many of us simplify to "don't trust"

There is no upside for Facebook to have anything about me.

Unfortunately Facebook has lots of information about you whether you have an account or not. You’re a big black hole with known characteristics inferred from those acquaintances of yours who interact with you in a manner Facebook can track. You might not have a name and an account, but there’s a big you-shaped golem in their data and there’s absolutely nothing you can do about it, unfortunately, and they’ll use it as they see fit with or without your consent.

Completely agree. I block Facebook at the router but I can't stop other people giving away my PII.

But I do make choices where I can.

Makes me think of that Junji Ito comic with the person-shaped holes in the rock after an earthquake. Only now, it's an allegory for social media giants and their corrupting influence on the people they lure into their system.

You can I'd you live in a jurisdiction which gives you the right to demand for all information a company has on you, and the right to be forgotten ( GDPR in the EU, maybe California as well?)

I totally trust Facebook. To fuck me over at every possible opportunity, even if they’ve pinky promised “we’d never do that!”. Because they have a track record of doing exactly that.

‘Trust’ is defined as wilful forbearance toward those agencies that we can conceive of as being sources of harm to us.

If you “trust Facebook to fuck [you] over” then you’re not trusting Facebook. You’re expecting Facebook to fuck you over and minimising your exposure to that process.

If you ask a computing security (or indeed any security) professional, "trust" has a different definition. If A trusts B, then B has the capability of doing something bad to A. This is regardless of whether A has granted B that capability. So, when you drive down the road, you trust the oncoming car not to swerve into your path and crash into you. This is just another way of saying that the oncoming car has the capability to swerve into your path and crash into you. There's no wilful forbearance involved.

I disagree insofar as the scenarios you describe (implicit trust in computer security, trust that another driver won’t swerve into your vehicle’s path) are perfectly good examples of the overall definition I posited above.

Indeed, if one didn’t have forbearance of (say) a software vendor or perhaps a dependency then one would be entirely free to not use that vendor’s product or that particular package. Similarly, if you don’t swerve right to prudentially make space for an approaching vehicle to serve into your path then you’re showing them forbearance and trusting them (or swerve left, depending on one’s location on the Earth what side of the road is legally mandate).

So, yep: those are perfect examples of Trust as embodied in the definition I presented above. And of course it lies at the root of initiatives such as Trusted Computing Initiative & cetera.

A necessary evil is no less evil merely because it is necessary.

> I know people don’t like tracking but usage analytics are invaluable to improving the product.

I wasn't aware that Facebook offered a Google Analytics style service, I was only aware of the Facebook Analytics for your Facebook page.

Can you provide links to this service, as my Google-fu doesn't return anything.

Is it not this? https://www.facebook.com/business/learn/facebook-ads-pixel

Could be, I wouldn't touch a Facebook property to test it, but the description implies this is all advertising tracking rather than usage analytics.

Fine, then use Matomo (https://matomo.org/) which even offers a free tier.

> I’m not really sure what they were thinking.

My guess would be that developers installed a single Google Tag Manager script and left the tracking to the marketing/analytics team. From that point they manage what third party scripts are added to the site, not engineers.

This situation cases a tunnel vision which can unfortunately lead to situations like this.

this is also how companies that are taking your hard earned money improve and iterate on their products.

What analytics do you think they can do using Facebook that they cannot do from their own httpd access logs, a technology that's 30 years old at this point?

Or, if that is not sufficient, one of the increasing number of ethical and/or self-hosted services, like Plausible? There is no need whatsoever to ship this kind of data to an outside party.

Facebook doesn't make an usage analytics service. And I really, really doubt 99.9% of the people using these abundance of tracking services ever discover any worthwhile corner cases you extol their virtues for. "Analytics" is just the 2000 word for "Reporting" - clueless middlemanagers demand shiny graphics.

Tracking is not ethical. Corporations only get away with it because most users don't realise it's even happening.

If you went into a retail store and an employee followed you around the whole time with a notebook and stopwatch writing down everywhere you walked and every product you looked at, you would rightly be creeped the fuck out and tell him to stop.

This is exactly what online tracking is, but done virtually.

> If you went into a retail store and an employee followed you around the whole time with a notebook and stopwatch writing down everywhere you walked and every product you looked at

I hate to break it to you, but retail companies are doing this today using security camera footage, to figure out what parts of the store customers start at, or spend the most time in..

They're also doing it with Bluetooth beacons / pings, explaining why every store has an app they want you to install:

https://www.nytimes.com/interactive/2019/06/14/opinion/bluet...

This is one of key applications of GDPR in Europe - the fact that you can collect data for one purpose (e.g. security cameras) does not necessarily imply that you're permitted to use the same data for any other purpose (e.g. marketing analysis of customer movements).

For the former purpose, it would generally be sufficient to inform visitors with a sign on the entrance with legitimate interest clause; for the latter example IMHO the only practical compliant solution would require anonymization of the data, so you could make and store density data iff you don't have any way to tie them back to customer identities including the purchases they made, which is a key difference from the facebook example, which (as far as I understand) uses unique IDs to link the conversions to specific FB accounts.

The difference is that with a human near by I feel like I am being judged and when they are not I do not feel that kind of pressure. The grocery store I've gone to my whole life has always had security cameras tracking people for reasons I assume to deal with theft. It does not effect me at all and the store has useful data that they can use. It benefits all parties.

>The difference is that with a human near by I feel like I am being judged and when they are not I do not feel that kind of pressure

I will offer you another perspective to consider.

Technology is extremely subversive in that it bypasses all of our brain's instinctual responses. Someone or something monitoring and tracking you should be setting off warning sirens in your brain. At best they are trying to study you, at worse they are trying to exploit or harm you.

Through hundreds of thousands of years of evolution our brains have built up warning systems to make us feel fear and unease when we realise we are being tracked. But since humans have spent 99.99% of evolution entirely in the physical world these systems have no concept of the digital.

The reason you feel extremely uneasy when being monitored by a person, but not when being monitored by a computer system that is collecting the exact same information (or more), is that your subconscious brain doesn't understand computers.

>our brains have built up warning systems to make us feel fear and unease when we realise we are being tracked

>The reason you feel extremely uneasy when being monitored by a person

I don't though. If someone walked up to me and asked me what my favorite color was and they wrote it down I don't feel any negative feeling.

Even if it was a true thing just because we have a warning system that doesn't mean that something is actually bad. Tracking just gives people more information to allow for better decisions to be made and it can make things more efficient.

Pet peeve of mine: "Trust but verify" is an oxymoron. If you "trust" someone/something you don't need to verify. I understand the sentiment, but it's based on a faulty assumption that trust is a binary state.

It's most exemplary case is the use done by the security teams in most IT shops: "Yes, we did put endpoint protection on your laptop, it's not that we don't trust you -- but you know what they say...". I wish they would simply say:"Of course we don't trust you, not on this. You are going to click on that shady link or download and run random executables from the interwebs. But we do trust you're good at your job".

The historical usage of the saying "trust but verify" is that the verification happens after they had the chance to do the thing (possibly badly), as opposed to mistrust where you'd verify everything that they'd do before allowing the thing to be done.

It's essentially "audits instead of pre-approval".

Yev from Backblaze here -> I chimed in below but wanted to let you know hat we've finished our root cause analysis and have updated our blog post with some additional information: https://www.backblaze.com/blog/privacy-update-third-party-tr....

"but the bigger problem is the tendency of third party integrations having default settings that are overly aggressive when it comes to access of user/site data. "

Our permission schemes are way too broad in general. I have done some tests with Zapier and IFTTT. For every integration you consent with them seeing ALL your data, being able to modify and so on. You also don't get a log how often the permission was used.

We're looking to invert this equation at Transcend. All client side network emissions can be regulated in accordance with tracking consent following a block/quarantine-by-default model.

This means that any network emissions you add to your site would have to be categorized by URL or domain instead of relying on shaky special-cased integrations that can fail as soon as an API changes.

Due diligence when installing a tracking script? Is this not a third party script which runs in the page context?

Reply on news.ycombinator.com