Hacker News Reader

Comment by yorwba | Hacker News Reader

This is really a "damned if you do, damned if you don't" kind of situation.

They can either use Tencent's Safe Browsing API as a drop-in replacement for Google's API, relying on k-anonymity to leak as little information as possible. That leaves them open to accusations that they allow Tencent (or, for that matter, Google) to track the browsing history of Safari users.

Or they can essentially turn off Safe Browsing in China. (Google's API is collateral damage of the Great Firewall.) That leaves their users unprotected against all kinds of malware and scams.

I think they made the right call here by protecting users against the most common threat (most people are not dissidents), while giving advanced users with a different threat model the opportunity to opt out.

View on HN

"Or they can essentially turn off Safe Browsing in China."

The OP as well as the associated blog post[1] as well as the Apple-provided fine-print language do not make it clear to me that this "feature" is exclusively enabled for Chinese users (or, perhaps Chinese IPs).

Could someone point to a source that confirms a US person, in the US, with a US-purchased iphone, would not have their browsing history transformed and sent away for analysis to tencent ?

[1] https://blog.cryptographyengineering.com/2019/10/13/dear-app...

If this source is to be believed, it's either going to Google or Tencent, but never both:

https://twitter.com/eromang/status/1183422784082530304/photo...

You can try yourself by going to one of the IOS Safe Browsing test pages on your phone, and when the warning pops up click "Show Details". It'll either say Google or Tencent on the warning message, which should let you know which one got chosen for you.

https://testsafebrowsing.appspot.com

I just tried it, and it says Google for me in the US.

Great. I disabled safe browsing probably back when it first appeared on my iPhone 3G or 4 and this test confirms I’m still not sending urls to anyone whilst surfing on my iPhone 11. Nice job preserving these settings over countless device upgrades.

For anyone else wanting to disable it (or at least learn more about it), the feature is labeled in iOS settings under Safari > Fraudlent Website Warning.

Interestingly, none of those links triggered a warning for me on my Mac…

You've probably switched off Safe Browsing. I wouldn't advise anybody to do that unless they're _so_ sure that they don't need Safe Browsing that when (most likely rather than if) they get infected by Malware or fooled by Phishing they are confident they'd tell everybody they know what an idiot they are.

I have it switched off on my home PCs (but on for work). But then I also don't carry home insurance and when I was flooded I believe my first Facebook message began "This is probably a good time for you to say 'I told you so'" because that seems like the right sentiment.

I haven't.

https://news.ycombinator.com/item?id=21242628

Alternatively, they could purchase the data from Tencent or another company, and operate their own version of the service. That may even be what they’re doing —- but we don’t know, since they launched the service with no details or publicity.

Yet another approach is to send the entire list of all malware URLs to each client and let the client do all the processing on their end.

This way no data (hashed, anonymized, truncated, or otherwise) would need to be sent to Tencent, Apple, or Google, or anyone else.

Why is this getting downvoted? I'm also interested in why this approach isn't taken.

Chrome does something like this, you can learn more about it here: https://codereview.chromium.org/6286072/

A couple possibilities I can think of:

* the list may be prohibitively large

* it exposes to the bad actors exactly which of their scams is detected, so they can simply refine their methods until their sites don’t make “the list”

Bloom filters take care of the first. There will always be an arms race between attack and defense, so I'm not concerned about the second issue.

Bloom filters can give false positives, and to eliminate them, you'd need to send "data (hashed, anonymized, truncated, or otherwise)" to some entity that has the full list. That's exactly how Google's Safe Browsing API works.

Can't the bad actors already check each of their sites individually by pretending to be a normal user?

Wouldn’t it be pretty easy for the bad actors to check the database anyway? I can’t imagine they would need to query often enough to hit any rate limits.

Bad actors can also occasionally poll the safebrowsing API.

Exactly how large is it?

OS vendors already make a habit of regularly sending gigantic OS updates. I'd have a hard time believing that a compressed list of malware URLs would be noticeably bigger, by comparison.

Also, once the list is sent the first time (or just included with the OS so it'd be already present on your device when you bought it), they could just send the deltas as the list changed, and those deltas (especially once compressed) should be relatively small even compared to the original (probably not that large) list.

Google Safe Browsing transparency report lists 40k new bad URLs per week - how large do you think the list is now? It is far, far too large for local processing but k-anonymity is perfectly trustworthy when used with cryptographic hashing.

Downvoters who think that might be too much data don't know about Bloom filters.

People who think they know about Bloom filters should consider what attack vectors a false positive with such would allow.

(The end result of the thought experiment will be basically what Google does now.)

Bloom filters are likely useless in this situation - following facts for phishing only:

1. Phishing sites have a lifecycle of about 15 hours.

2. Most malicious links are hidden within benign domains.

3. About 400,000 phishing sites are created each month.

From: https://www.itgovernance.co.uk/blog/4-eye-opening-facts-abou...

I haven't run the numbers, but I am guessing that a clientside solution would have a lot of bandwidth sucking and avoiding false positives is very important.

Also with a clientside solution, how are new phishing URLs detected?

PS: perhaps try to assume HNers know what a Bloom filter is (I've seen them come up lots of times in comments).

400,000 sounds like a lot, but I wonder how many new URLs Tencent adds to its database each month. I expect they don't add every phishing URL but some small subset of them (possibly even a very small subset.. we'll proably never know).

But let's say it is 400,000. I took the URL you linked and made a file of 400,000 copies of it. The file size was 28 MB. I didn't bother compressing that particular file since the URL is the same in each instance, but I expect a file full of actual phishing URLs would probably compress pretty well, so it would probably be significantly less than 28 MB.

Considering that OS vendors regularly ship multi-gigabyte size updates, having to download less than 28 MB extra every month shouldn't even be noticeable. If updates needed to be done more frequently, the client could subscribe to get regular updates as they become available.

> 28 MB extra every month shouldn't even be noticeable

Parent comment suggests phishing site life-cycle <15hrs, at 400k a month that's 8333 every 15 hrs. To give an idea of how frequency sensitive this is, assume URLs are added equidistributed in time: that would be a new one every 154ms - for such time critical information it makes no sense to attempt to synchronize clients, it would require constant polling or push updates to have _any_ chance of catching a malicious URL.

At such a frequency, efficiency becomes less about bandwidth and more about the overhead of continuously synchronising so many clients (think of that 28 MiB spread out over 400k separate messages over one month, one every 154ms, that not only inflates the size, but causes a constant network usage and processing that is far less efficient than a single 28MiB download).

Or you could just send the URL hash when you visit a URL... (do you request any where near 8k URLs every 15hrs?, 1 URL every 154ms? no), it's so clearly a simpler solution that will be faster for everyone without letting bad URLs slip through before a latent sync.

Do you really think Tencent is detecting a new phishing site every 154ms?

I'd seriously question how many of the total new phising sites they detect to start off with, and then how frequently they do so.

If a user only downloads the deltas periodically they'd risk being out of sync with the master list (which might not be updated even once a month or at all, for all we know), but that's the price they'd need to pay to have to send any information about their web browsing to parties they don't trust.

One other thing to consider is the likelihood that the URL you happen to be surfing is both a phishing URL to begin with and one of the ones that just appeared since the last delta download you did, compared to the likelihood that it's one of those already in the entire phishing URL database you've already downloaded. I'd expect those odds to be very low.

> the likelihood that the URL you happen to be surfing is both a phishing URL to begin with and one of the ones that just appeared since the last delta download you did, compared to the likelihood that it's one of those already in the entire phishing URL database you've already downloaded. I'd expect those odds to be very low.

Ignoring the first condition (otherwise why bother with a list at all)... Consider that this information is very transient (average 15hrs), this is pretty simple: deltaT / 54000

This is still horrible, because your safety is determined by how frequently you can sync with the DB.

> If a user only downloads the deltas periodically they'd risk being out of sync with the master list (which might not be updated even once a month or at all, for all we know).

Being updated monthly doesn't match the statistic of average 15hr lifecycle, because it would be useless after that length of time. And while I don't claim to know 15hrs as a fact, it is intuitive that the average will become ever shorter as malicious URL checkers become updated ever faster.

> but that's the price they'd need to pay to have to send any information about their web browsing to parties they don't trust.

Full URL information need not be sent, a hash of the URL domain and path would probably suffice... if that's not enough then it's a dilemma, but that doesn't make continuous syncing a good or fail safe replacement.

"Being updated monthly doesn't match the statistic of average 15hr lifecycle, because it would be useless after that length of time."

And maybe it is useless. We don't actually know, but we should at least recognize that there may be a difference between how frequently phishing sites allegedly appear and how frequently they appear in Tencent's malware URL database.

"This is still horrible, because your safety is determined by how frequently you can sync with the DB."

And being identified by the Chinese government as someone who surfs to forbidden websites might be even more horrible, for some.

There's no need to sync every time a new phishing URL is added - only every time a URL is visited by a client.

The delta can be derived just from the version number of the client's URL database, and should be a total of 1 MB in size for a whole day's worth of updates. So ~1 MB for the 1st URL visited in a day, and considerably less afterwards. Compared to average webpage size, that's nothing.

Really, only thing that changes is instead of sending a URL hash, you send the URL DB version, and the reply is the list of changes since that version.

> Really, only thing that changes is instead of sending a URL hash, you send the URL DB version, and the reply is the list of changes since that version.

Or none at all and a simple confirmation that the list is up to date. Yes this is a way better idea.

Although it's always going to be less efficient. For instance i'm not sure how it would scale into the future. Checking URLs server side is optimal, it's always going to be relatively constant in proportion to the URL size, but with DB deltas each URL is now related to both the URL size and the DB update frequency, i.e as the malicious URL rate increases over time, individual URL lookups will incur greater network cost... this is probably not a big deal for the client, but It would make a significant difference for the provider of the deltas - or maybe network caching would disolve it again? I mean there would be a lot of duplicate deltas flying around every minute... basically a content distribution problem but with a high frequency twist.

Google's safe browsing API is probabilistic too. The idea is that you do so many rounds of checking to get closer and closer to the mark. You start with a fairly high false positive probability, high-privacy check, then if you get a positive, you try a lower false positive rate check that also loses you some privacy, and the trade-off is that you don't have to have the full malicious site DB with you at all times (and keep it up to date).

Why did you assume I'd not know about false positives?

Local handling seems more Appley too.

Running their own service would be most ideal. They are definitely not. https://github.com/Igalia/webkit/blob/9777baa3db09cad7ed5b2c....

> This is really a "damned if you do, damned if you don't" kind of situation.

Why? Simply, when setting up the device/browser, let the user choose what safe browsing API the browser shall use (both, one of them, or none).

Letting the user make a conscious choice is the best way to handle "damned if you do, damned if you don't" kind of situation. To make the choice as conscious as possible for the user, provide additional material that explains the advantages and disadvantages of each option for the user so that the user is well-informed before he/she makes his/her choice.

That’s entirely against the entirety of Apples modus operandi. It’s always make the user have as little choice as possible and assume that users are idiots. The only exception they made is to developer with cli abilities, and even that they have began to restrict

> assume that the users are idiots

Microsoft didn’t make this assumption. Apple did. My entire extended family are now all Apple users. My tech support calls per year can now be counted on one hand.

In my extended family, the users are idiots, and thanks to Apple, I’m poorer financially but significantly richer in free time.

That is exactly my experience too. My parents just kept calling me when they were using Android phones. I bought them iPhone to get my free time back.

There is not much difference there. iPhone might be more confusing for people who are used to android.

On an Android device, there are lots of relatively simple to follow guides to be able to download 'free games'.

Rooting your phone is all well and good if you know what you're doing. It's not so good when you don't, and I'm your tech support phone call.

I didn't say it was a poor choice, rather, it's a great choice.

I'd rather be an idiot in the areas that I don't have an expertise in, I have no interest in plumbing and I would blindly follow the suggestion that the plumber who came to my house made. And I guess this strategy worked well for Apple.

I believe an argument could have been made that it was the right call if they had publicized it. Seeing as how it was implemented in the background, I am less inclined to give them the benefit of the doubt.

Do you think a decentralized database of unsafe URLs could exist?

Reply on news.ycombinator.com