µBlock dynamic filtering: default deny | Hacker News Reader

µBlock dynamic filtering: default deny

github.com

141 pointsXylemon77 comments

As the author was kind enough to pick us as an example, here's some background on what some of those requests are doing.

hits.theguardian.com points to our Omniture implementation, which is the main tracking suite used for macro-level reporting (like when we say we have X unique monthly browsers, or whatever). So if you want to be invisible to that, leave it blocked.

ophan.theguardian.com points to our own analytics tool, Ophan, which does things like tracks whether you "read" the article. It's for journalists to work out if people like their stuff. All the views of the data are aggregated, but if an analyst really wanted to they could go write some SQL to look at the behavior of individual cookies. So if you want to be invisible to that, leave it blocked. A quick Google [0] will turn up lots more about Ophan and how it works.

Our only calls out to Facebook and Twitter are to retrieve share counts for the current URL (besides articles with embedded tweets, for now). These are probably relatively safe to unblock, but if that information doesn't interest you they're equally safe to block.

api.nextgen.guardianapps.co.uk is handling most Guardian stuff that gets ajaxed onto the page, like suggestions for what to read next. It's pretty harmless and required for a bunch of functionality. All the guim stuff is obviously just static assets.

It would be cute if there was some way of us hinting to the plugin which domains were needed to not break the site, though likely impractical in the real world.

[0] https://www.google.co.uk/search?q=ophan%20guardian

> Our only calls out to Facebook and Twitter are to retrieve share counts for the current URL... These are probably relatively safe to unblock

Wouldn't this expose you to tracking by Facebook and Twitter? I would think people are more interested in blocking third-party tracking than first-party.

Thanks for the detailed explanation either way!

> some way of us hinting to the plugin which domains were needed to not break the site:

I had collated this so far to mostly un-break the site -- for viewing at least:

    www.theguardian.com guardianapps.co.uk * noop
    www.theguardian.com guim.co.uk * noop
    www.theguardian.com theguardian.tv * noop

Whenever I un-break myself a site, I add the proper rules in there: https://github.com/gorhill/uBlock/wiki/Dynamic-filtering:-de...

There were 16 3rd-party root domains I found were not needed [1]. Probably fonts.googleapis.com would make the site looks better, but it's up to users whether they want to let Google know they have been visiting what article on the Guardian.

[1] adnxs.com criteo.com doubleclick.net fonts.googleapis.com google.com googleadservices.com googlesyndication.com googletagservices.com gstatic.com imrworldwide.com krxd.net mathtag.com moatads.com ophan.co.uk outbrain.com revsci.net scorecardresearch.com

> It would be cute if there was some way of us hinting to the plugin which domains were needed to not break the site

No advertiser or site looking to cash in on ads would abuse that.

Yeah… I was told by someone that if everyone started blocking 3rd-party domains, the advertisers would start figuring out how to deliver their content through the server-side of the 1st-party so that it would be even less clear this was happening. In other words, there's already a way to hint at the plugin that something is needed: deliver it from the primary domain. This is obviously a complex battle for the long haul.

As someone who has built and exited an adserving SaaS in the past, I get asked this question almost never -- the advertisers that spend the most money never want to abuse the law.

The same goes for the SuperCookie; everyone knows the technology exists, but it's only the rotten apples in the industry ruining it for everyone else by actually making use of it. I got asked whether we would support this a few times, but it was always asked by the most shadiest of our customers, and a simple "we want to listen to the visitor's intent" sufficed.

The only real risk I see for online privacy is that this sort of stuff will happen en-masse and there will be a powerful lobby to illegalize this behaviour.

We do seem to have to worse of all worlds at the moment. It is very difficult for users to exercise control over good sites, and it is practiaclly impossible to control bad sites.

I wonder if a solution would be to tie third party cookies to the parent page. So that by default a Facebook cookie on a Guardian page could only be retrieved when the user is on the Guardian website. You could then have options within ther browser to explicitly allow cross domain cookies if the user wants (and send the actual Facebook domain cookie).

There is no reason limiting adservers from doing what you describe using first party cookies: they all make use of Javascript, so it is trivial to just set a first-party cookie.

We actually took that approach to be compliant with the EU's cookie law; if a visitor rejected third party cookies, we fell back to first party cookies.

> It would be cute if there was some way of us hinting to the plugin which domains were needed to not break the site, though likely impractical in the real world.

Sounds like RFC 3514, only negated.

Denying 3rd-party resources by default is nice, but definitely experts-only. Pages will break in weird ways when half their rat's nests of JavaScript are missing, and it will take some coding knowledge to figure out which parts are necessary. I may switch to this myself, but I think I'll keep telling friends and family to use Ghostery

The feature is available only to those enabling "I am an advanced user".

Ghostery breaks so much stuff, I don't know how people can bother. I've found that Privacy Badger does a great job of blocking third-party tools in a way that rarely breaks anything - in which case it's easy to turn off.

That and Ad Muncher for Windows.

I've been using Privacy Badger for 6 months and I can't recommend it enough for non-technical people (or technical people that don't want to debug websites all day long).

The one problem I've found is that certain services that use a bookmark/iframe combo (instapaper, for instance), create problems because you want to whitelist instapaper, but aren't given a chance. For that, I recently found: https://tsibley.net/provoke-the-privacy-badger/

But it's based on AdBlock code. I started using uBlock instead of AdBlock Plus in order to save some resources. It seems that adding Privacy Badger would introduce that performance hit to the equation again.

The only really noticeable impact I've seen is that it blocks Disqus, which too many people seem to use for comments. That's usually not a problem, and I can always disable it in the rare cases where I really want to comment on something. What sites are breaking for you?

It's been a long time since I used Ghostery, but the most common problem were JS trigger events from clicking a button that didn't fire. It made it a bit of a pain to debug.

What does it break apart from google ajax search?

Ghostery developer here. As someone else pointed out, it's usually analytics tied to certain events like button presses, ads that trigger the main content to load after it shows, or fonts loaded from a 3rd party like Adobe that break sites. When we block the javascript and the developer didn't use a try/catch around that analytics/ad call, it breaks.

Most of the time there's no way around that breakage except by allowing the script (unless you use surrogates). So if Ghostery breaks the page and another blocker doesn't, usually that means that the other blocker is not blocking that script.

Adblock Plus lists, for example, contain extensive exceptions to allow scripts/ads on certain broken sites. This improves usability greatly and is a big reason some people are happy with ABP and not Ghostery. This isn't done in the most transparent way though, which leads to people unjustly accusing them of accepting money to allow ads when they find out about these exceptions. (This has nothing to do with the non-intrusive ads feature, which is a separate issue)

I've been using Ghostery for a year or so now (and prefer its UI over μBlock), and I've had minimal issues. If something breaks, I paste the URL in am incognito window. If it works there, I know it's a Ghostery issue.

Reminds me a lot of the first time using NoScript. Figuring out which domains added functionality, and which just unleashed another tree of third party domains that may or may not unlock the content. Fun, for certain definitions of the word.

It's really satisfying to see the websites load and render faster after you get it all set up though

I came up with a simple rule for that. If I've got to temp unblock more than 3 things in NoScript to get to the content, it's not worth the effort.

I still havent figured out how to unblock resources with ublock.

In the "options" page for the Chrome extension there's a whitelist tab.

Does that whitelist the entire host? I would like to block everything but a specific JavaScript file on specific host. This is very easy to do with ghostery

Perhaps, you should just switch to µMatrix.

µMatrix is sadly not available for Firefox.

> a specific JavaScript on specific host. This is very easy to do with ghostery

I don't see any feature to accomplish this with Ghostery. You can block hosts from its pre-built library of hosts, not more, and not at a higher granular level then whole hosts.

uBlock will report all hosts encountered on a page, not just those part of an internal database.

"unblock resources" is too vague, can't provide an accurate answer.

You need to turn on advanced features if you want to do more than unblock everything from a site.

I don't know RequestPolicy but I've been using μBlock instead of ABP + Ghostery for a few weeks and I'm quite satisfied.

µBlock or Ghostery, whoever comes up with the tool to block garbage on mobile devices (preferably Chrome), will be a winner!

Ghostery supports Firefox for Android [1]. For µBlock, there are two tickets open on GitHub [2,3] about Firefox for Android support.

Unfortunately, Chrome for Android doesn't support extensions [4]. But if you have a rooted device, you can block ads/trackers/etc. across all apps via AdAway [5], which manages a custom HOSTS file for you.

[1] https://www.ghostery.com/en/download

[2] https://github.com/gorhill/uBlock/issues/524

[3] https://github.com/gorhill/uBlock/issues/556

[4] https://developer.chrome.com/multidevice/faq

[5] https://sufficientlysecure.org/index.php/adaway/

(Disclaimer: I'm a Ghostery developer)

In addition to the extension working on Firefox for Android, Ghostery has it's own browser:

https://play.google.com/store/apps/details?id=com.ghostery.a...

It's pretty new and has some key features that are still in development, but it's getting there.

Hey, I'm like millions of those lay users who do not have rooted mobile devices.

WRT Firefox, it does not play nicely with Google services, like Gmail. WRT, browsers...I have tried many of them under the sun and all of them have a consistent disadvantage over Google, that is, speed.

I've got many lay users friends who still look for an Ad blocker out of the box, for their mobile devices. I have hope from Ghostery and uBlock makers, fingers crossed.

Oh please, listen to my sentiments. I would LOVE to use that browser of yours, but its unexplainable to me how anyone could stand using a mobile browser without text reflow.

I know Google is not making things easier[1]. Before every app that used WebView had it. Now Opera is alone with this feature on Android. Implementing it would give you a nice competitive edge and make many users happy. Somehow Opera is able to to do it super elegantly with WebKit code.

[1] https://code.google.com/p/android/issues/detail?id=62378

I would pay good money to have this problem fixed!

I can't quite tell: is this a complementary or overlapping feature to µMatrix, from the same developer? As an aside, I have noticed much better browser performance with the µ plugins as opposed to things like ABP. Keep up the good work!

I was under the impression µBlock was a fork of uMatrix, made simpler for users.

µBlock and µMatrix are parallel offshoots of gorhill's earlier HTTP Switchboard. They're refactored forms of HTTP Switchboard, where each extension focuses on different forms of blocking (µBlock for pattern-based filtering, µMatrix for matrix-based filtering).

See:

https://github.com/gorhill/uMatrix/wiki/Changes-from-HTTP-Sw...

https://github.com/gorhill/uMatrix/wiki/Using-%C2%B5Block-wi...

Okay, I'm probably stupid. This would allow me to throw away NoScript, Disconnect and Adblock.

But I can't figure out how to enable deny by default mode?! (Yes I have advanced mode enabled)

There's a tiny little + by the "Requests Blocked" label on the Button panel. Click that.

Then red the left side of the top five boxes (images and scripts and the like).

If I understand correctly, it's not really a mode, it's just several broad filters. 0.8.5.7 on Firefox doesn't have an 'all' that I can see.

Oh, I see. I was under the assumption that these settings were still site specific, got it, thanks!

Edit: Would be nice to be able to set those global settings once and not have them pop up on every page. And/or have a visual distinction between global and site specific rules, right now there is no option to globally disable images and then whitelist on a per site basis, right? I can just whitelist a domain which will then load everything?

My initial impression wasthat I could globally blacklist everything I don't want per default and then enable it on a (sub)domain basis.

I'll certainly give this a shot though. My firefox is really slow with ~20ish addons.

If you go into the settings and look at "My Rules", you can get a pretty good sense of the scope of the rules you have running.

> right now there is no option to globally disable images and then whitelist on a per site basis, right

Yes you can, it's in the quick guide[1]:

> First column: what is to be dynamically filtered

> Second column: global dynamic filtering rules

> Third column: local dynamic filtering rules

[1] https://github.com/gorhill/uBlock/wiki/Dynamic-filtering:-qu...

I don't know if uBlock works the same way RequestPolicy does, but RP is not a replacement for NoScript. You may want to load images from a host but not allow that host to run Javascript or Flash: https://www.requestpolicy.com/faq.html#faq-noscript

Does uBlock provide any way to write shims/surrogates? Sites often break because they just can't deal with certain JS libraries being missing, but there are many cases where it looks like just providing a stub implementation would be enough.

Enabled default to deny last night. And I actually think it works great! Yes, some sites, are broken, but with some tinkering around, it works.

This will be my default setup from now on..

So with this, I no longer need Ghost and Adblock?

Yep, I just switched. I initially made the jump solely because of uBlock's performance claims, but I have to say, it already seems much better overall than the haphazard Ghostery + Adblock combination I had before.

Just did this. µBlock does seem faster than the ghostery/adblock combo. I've also enabled Spam404 and Dan Pollock's hosts and everything is perfect so far.

The question is: is it more resource-efficient than RequestPolicy?

Probably?

https://news.ycombinator.com/item?id=9018229

Policeman is like RequestPolicy, only much better (and it imports RequestPolicy's rules). I haven't tried uBlock/uMatrix yet, but if you like RequestPolicy do give Policeman a try.

Policeman is awesome and is now replacing RequestPolicy for me - thanks for mentioning it. I can't see it available for Chrome yet, which means I have to stick to HTTPSwitchboard I guess.

Does ublock still block crash reporters like newrelic and self hosted party statistics like piwik?

PS, I had major issues on two mainstream travel sites with ublock. It made then unusable with ublocksl's default settings.

Piwki would be fine if it was hosted on a subdomain of the first party domain.

If you visit test.com, piwki.test.com would work. However, piwik.not-test.com would be blocked.

Last time I check piwik is always blocked no matter the host.

Yeah, you're right. In the latest release, piwik is blocked. I hadn't tried it recently.

Do you want it to block first-party trackers? I am much more tolerant of trackers (and ads) if they are per-site rather than being run by a few huge data-gatherers, as there's much less privacy impact that way.

I don't but last time I checked ublock blocks all instances of piwik no matter where.

Using EasyPrivacy results in piwik being blocked, yes, regardless whether EasyPrivacy is used in uBlock, ABP, or whatever other blocker. I suggested you to bring the issue to EasyPrivacy maintainers if you think it's wrong[1], but by all appearances your grudge is solely aimed at uBlock.

[1] https://github.com/gorhill/uBlock/issues/564#issuecomment-70...

Easy privacy is not included as the defaults in other tools like abp. Thus ublock does differ here in a big way.

By default ublock blocks piwik and abp doesn't. That is a major difference.

Edit: why is a factual response down voted?

I downvoted you before you edited out the axe grinding.

Easy privacy also blocks pusher.com which is used by thousands of sites for realtime stuff. It's far too aggressive and should not be on by default.

> which is used by thousands of sites

That's probably the reason it is in EasyPrivacy.

In EasyPrivacy, I see: "||stats.pusher.com^"

When I put "stats.pusher.com" in the browser address bar, I get a page which only says, in plain text:

> Pusher client stats collection service

"stats.pusher.com" is also present in MVPS and hpHosts.

Edit:

The stats.pusher.com issue was brought to EasyPrivacy maintainer: https://forums.lanik.us/viewtopic.php?f=64&t=20744

still not able to block youtube ads, thanks, but no thanks. I´ll stick with adblock edge.

Did you enable the 3rd Party Filters?

I have never seen a single YouTube ad with uBlock and Chrome.

IME, youtube ads seem to come on dynamically, with dynamic amount of "forcible viewing" time, eg 5sec, 30sec, 60sec, etc. It does a pretty damn good job of showing me the most ad without my walking away in disgust. For example, I was on a binge of science videos and the ads went from 0 sec, to 15, to 30, in lock-step with my curiosity.

for me i've noticed they only show up on embedded videos on 3rd party sites

I believe Safari/Youtube/Flash won't block ads. But Safari/Youtube/HTML5 does.

Reference: https://github.com/gorhill/uBlock/issues/706#issuecomment-73...

You must be missing some of the block lists (or just speculating). The amount of stuff that uBlock handles is the same as Adblock Edge.

>I´ll stick with adblock edge.

Well, you can use what you want. We have no vested interest.

>still not able to block youtube ads, thanks, but no thanks.

But you're wrong, of course. And you've not given enough information for people to accurately help you, if they wanted to do so despite your tone.

On both Chrome and on Opera, with uBlock I don't get YouTube adverts. (Albeit with some more filter tickboxes ticked, which may or may not make a difference.)

If you want to try and get uBlock blocking your YouTube adverts, I'm sure lots of us would be willing assist. More so if you weren't an ass about it.

Edit: I'm using HTML5 instead of Flash, that could be the key difference.

> I'm using HTML5 instead of Flash, that could be the key difference.

The ads with Flash is only a problem on Safari, due to API limitation on Safari (Chromium/Firefox can block ads with Flash).

As for the filter lists, I think only _EasyList_ is needed for Youtube ads.

youtube-dl fixes all ads on youtube. And gives you a backup to boot! win/win really.

Reply on news.ycombinator.com