A JavaScript browser library fails due to an adblocker blacklisting 'beacon.js' | Hacker News Reader

A JavaScript browser library fails due to an adblocker blacklisting 'beacon.js'

github.com

78 pointsvbrandl77 comments

Another protip: don't name your images "banner.jpg"; "ads.js" and "analytics.js" etc are also not exactly great names.

Another protip: Always make sure that critical flows in the app won't fail if analytics event fails. You probably want to wrap calls to analytics provider with your own function, and in that function check for existence of global analytics object like "window.ga" which may fail to load due to adblockers and similar tools. In other words, don't 'ga("send", ...)' all over your code.

I had airplane ticket booking fail mysteriously because "pay" button had a analytics listener bound to it, and it would throw an exception when adblocker was on. Without devtools, I would not understand what's going on.

Yup, had an "analytics.js" in a project recently (related to an app dashboard, nothing to do with tracking) and it got blocked. Luckily I caught it right away because I have uBlock installed in the same browser I do development in.

Moral of the story: blocklists are a bit shitty and prone to overblocking due to excessively generic rules. Take that into consideration when naming things.

> Another protip: don't name your images "banner.jpg"; "ads.js" and "analytics.js" etc are also not exactly great names.

This is good advice even if you are designing banner ads or intrusive analytics!

I often wondered how many ads get blocked by the very simple rules usually seen at the top of adblock lists: /ads/ etc. Ads vs adblockers is an arms race, and it seems like renaming the ads files to a random per-deployment name would have been the first escalation by the "bad guys". But maybe many developers of ad software don't even bother to fight adblock, preferring to optimize for users who don't use one.

They do.

I've recently encountered a video ad startup that runs a proxy wrapping VAST responses into a VMAP. So, a website would request an ad not from DFP, but from an inconspicuous CloudFront URL. These CF distributions are deployed for each publisher and you'd have to block them one by one, not being able to block the whole ad platform.

They said that some adblockers can discover ad servers by parsing responses. If a response, say, is a VAST tag, they tag and blacklist a server.

Adblockers are starting to employ heuristics to analyse what a server does, rather than relying on just URL pattern matching. Just like Virus VS. Antivirus arms race.

Fascinating!

Like all arms races, one day it will all blow up in our faces. Since it is not a literal arms races, it will metaphorically blow up in our computers. Hope somebody resolves it before it really does. (Not gonna happen though)

Arms races are ubiquitous , from US Vs China to bats Vs the moths.

How exactly do they blow up?

Usually for money reasons?

I blogged about a fun one a while ago[1], pornhub using WebSockets to dodge adblockers.

[1]: http://blog.bugreplay.com/2016/11/pornhub-bypasses-ad-blocke...

CloudApp had this type of problem where the CPU would crank up to 100% usage after my Pi-Hole blocked their segment.io code from functioning.

IMHO it'd be much better to just mock it than to litter your code with if() all over the place, similar to how polyfills do it (but without the actual functionality):

    if (!window.ga) {
      window.ga = function(){};
    }

This shit drives me up a goddamn wall.

I had a web app a few years ago that I was using webpack to build the JS with, and the output filename was just a SHA hash of the contents.

After an update, I started getting reports that it wasn't working for some people. Turns out they had adblockers that were blocking *ad.js and it just so happened that the SHA hash of the file ended in ad.js

Medium had the same problem where hash-named paragraph divs would disappear if they contained "ad".

Wow. Thank you for this story. Word that this could happen should get out there. I would have never thought to look for hashes that happen to end in "ad"

Hmm I wouldn't think of this aspect but it's really insidious!

Looks like it could be interesting to submit a PR to some popular hashing libs to have a 'adblock-safe' mode (append some postfix if hash ends with 'ad' etc) or at least document the thing so that more developers are aware!

> Looks like it could be interesting to submit a PR to some popular hashing libs to have a 'adblock-safe' mode (append some postfix if hash ends with 'ad' etc)

Why is that preferable to a PR that fixes the bad regex?

I would suggest `[^A-Za-z0-9]ad.js` is better - it's not just hashes that would fall foul of this, any word that happens to end in 'ad' is going to break. If it's actually about ads, the prefix is probably going to be something like 'blah-' or 'main.' or whatever, surely?

In making a card game site (http://www.goodloesolitaire.com/), I got bit by ad.png; the "ad" was referring to ace of diamonds. Fixed it by using a single image for the whole deck and changing the position of the image to show the card that was picked. But man, tracking down that bug was a head scratcher for awhile. That was in the early days of adblockers.

>Fixed it by using a single image for the whole deck and changing the position of the image to show the card that was picked

Couldn't you have renamed the file?

Good excuse to refactor I'm sure.

Probably did do that at first (10ish years ago makes it a bit of a haze), but it was being served up by a PHP script that was creating the card name dynamically.

I wanted to move it off being tied to a server anyway and since anyone playing this would probably see a lot of cards, it made since to just download it all as one file rather than 52. At the time, it was all the rage to do such things.

Yup. I have to manually screw around with doordash URLs because the login button on their site breaks when I block analytics libraries.

Always make sure that critical flows in the app won't fail if analytics event fails.

Monoprice's site won't allow checkout with uBlock unless you define a function they're trying to call: _satellite = { track: function(){} }

With inventions like Google Tag Manager that allow marketing departments to inject JS into the production site, things like this may not even be the developers' fault.

The user is running an extension that breaks random pieces of the web page. Obviously that's not the developer's fault.

Monorprice is a business. At the end of the day, anything that stops an order is their PROBLEM. The fault doesn't really matter. No sale = no payday.

If it prevents you from making money it is still a problem.

If you develop for something that is not ad based you need to be aware of and test for these issues.

> I had airplane ticket booking fail mysteriously because "pay" button had a analytics listener bound to it, and it would throw an exception when adblocker was on. Without devtools, I would not understand what's going on.

This is likely intentional - break the website if viewed with ad-blockers, the consumer thinks 'the adblocker must be buggy', and disable the adblocker for the site.

In fact, I'd love to have analytics and as such on sites where I'm the paying customer, such as airline sites, amazon, etc - this enables the website developer to get the website code in a better shape.

Here's @vsund, author of this commit message. I understand that the title may was a bit clickbaity and now the title got renamed to a more rational one (which is totally ok).

Unfortunately the new title "A JavaScript browser library fails due to an adblocker blacklisting 'beacon.js'" doesn't reflect the situation very well in my opinion. The library didn't fail (mainly because there is no library yet), what instead happened is that the adblocker blocked GitHub internal requests which resulted in a unusable repository. The adblocker blocks requests that simply has "beacon.js" anywhere in the URL.

I see this rather as an issue of adblockers blocking to much than bad naming of projects :)

As one of an extremely large group of people who will never use that library but DO get protected from some ads and tracking by blacklisting beacon.js, I disagree that it's an issue with ad blockers.

The internet is a toxic place, to put it mildly, and if you're getting into any kind of public facing service on it, you need to know exactly what you're doing.

I totally agree with you, but I think adblockers could do better than just block all requests that have "beacon.js" in it. But yeah, I definitely prefer many protected users about one working library :)

There are so many possibilities to block content, I'm sure that we could do better with protecting users against tracking while ensuring that less content gets blocked false-positive.

Wait, you're saying that a false positive in an ad blocker is _not_ an issue with the ad blocker? Of course it is! Blocking files based solely on their names is a brittle and error-prone way of doing things, and this false positive is a clear example of that.

For sure it's a false positive and I wish it didn't happen. But it does happen and there's no way I'm ditching my ad blocker just because of some small number of false positives.

One time I came across a github user with a username that started with a hyphen. I could browse their repos fine, but I couldn't view their github.io site on Linux.

It turns out that Windows and OSX allow domain components to start with a hyphen, but Linux does not. There is at least one DNS RFC that disallows this though.

Relevant Ubuntu bug. https://bugs.launchpad.net/ubuntu/+source/resolvconf/+bug/66...

An interesting related anecdote on this is that at least in Chrome, adblockers can also interfere with extensions' attempts to load scripts, load stylesheets, or open windows. I maintain an extension with a smallish active userbase (~20k people) and a couple different times I've encountered mysterious failures that were the result of clumsy keyword blocklists in ublock/adblock/etc - for example an extension asset (packed into the extension, not on a web server!) containing the word 'popout' caused it to get killed silently, and renaming it made it work.

Naturally you can troubleshoot a lot of this with 'does it work if you disable adblock?' but it's a real pain to have to convince users that they can't use your software along with their broken adblocker.

Shouldn't adblockers whitelist local request to prevent something like this?

Why whitelist, when its easier to rename a file?

Also the filter in question "/beacon.js" is 3 years old in Easyprivacy, to be exact "Feb 23, 2014". I could understand if it was a recent commit but its not, the onus shouldn't be on the List authors to fix it, but the site owner (yourself) to fix it.

Is this meant to be sarcastic? becaon.js is a perfectly valid name for a JavaScript filename, being forced to name your files something different from their content just to placate ad blockers dodgy blocklists is not right.

Even more not right though, is every developer having to check every blacklist to see if their component name happens to also be used in advertising. Is this really what your suggesting?

Then there's the whole *ad.js blacklist another commenter brought up, this is frankly just a sign of a terrible blocklist.

And new entries that are just as bad get added all the time, so now every developer has to check every blocklist every single day if they want to keep their site running smoothly.

(A happy user of an ad blocker, who is aware that 9 time out of 10, when a site I'm using is broken, it's the ad blockers fault - not the sites)

> it's a real pain to have to convince users that they can't use your software along with their broken adblocker

The reality for most extension developers is that our extension is less essential to users than their adblocker, and if they had to choose between them they would choose the latter.

To be honsest, I might do the same. Surfing the web without an adblocker nowerdays isn't much fun, to say the least

Yes. We seem to have moved from a position of "four different browsers with differing behaviour" to "four consistent browsers with consistent behaviour, each with hundreds of potential content-rewriting extensions that may arbitrarily break your site in unexpected ways". I'm not convinced this is an improvement.

It seems to be an improvement for the user.

Well we also used to have mostly non-HTTPS, where wifi networks and ISPs could and did insert crap into pages. I much prefer now being able to guarantee that what the user downloads is what the site provided. At least then the "turn everything off and see if it works now" option should be viable.

Can you really blame them? Websites became hostile to users and users reacted. I don't know what your software is, and this certainly isn't directed at you, but if I'm forced to make a choice between using an ad blocker or using some software, I'll always choose the ad blocker (unless I was forced to use said software), especially if there are alternatives.

There are no real alternatives to using an ad blocker.

Don't assume most people are polite and reasonable like you. Many people are arseholes.

You're choosing to use either the ad-blocker, or the software, and you've chosen the ad-blocker.

You can see how tiresome it would be to have a flood of rude people saying "your software sucks", when there's nothing wrong with the software. There's a problem with the ad-blocker that the user has. The bug report should go to that ad-blocker.

You can also see how it must be a bit frustrating to have people saying "ZOMG why can't I use both? Fix your software so that I can use both!", when the fix needed is only achievable by the ad-block creator.

> when the fix needed is only achievable by the ad-block creator.

It seems to me that there are two possible fixes. One, the ad-block creator updates their list, yes. But the other is the extension writer renames the file that is triggering the ad-blocker so the ad-blocker is no longer triggered.

The only one of those two fixes that is within the direct control of the extension writer is renaming the file within the extension that is triggering the ad-blocker.

Except software which doesn't work with an adblocker does "suck". Sure, potential users could phrase it more politely, but for many of them (me included), it's a minimum requirement.

At least "sucking" is better than being totally useless.

But in the example given the reason the software doesn't work isn't because it needs ads. It's because the ad blocker is broken.

It's applying a heuristic that triggers a false positive in many cases. Most ad blocking rulesets do tend to include very basic rules like that. I don't know how many ads they actually still catch but if it's a significant number then it's probably a justified one.

The thing is, the software does suck some of the time. Here too, I'm not saying your software sucks, but web developers have developed several bad habits that put users at risk for no particular reason other than sheer laziness.

here are some examples:

1) Instead of putting third party scripts under your own version control and hosting them yourself, you host them from a "free" CDN which then stores tracking cookies in the origin of the user loading the script. How hard is it to serve all the js libraries you need from your own server?

2) You like some nice widget, like Google Maps. But hey, to use this widget, you need to load a third party script in your origin. No biggie, it's better than paying for a maps widget, right? Except now the user has a tracking cookie in your origin and you didn't disclose this or give the user an option to opt out. What you could have done is loaded that map inside an iframe that is served from a content domain or throwaway domain, so that there is a separation between your cookies and your scripts and the third party cookies and their scripts. This goes for all other widgets as well.

In the above two examples, the developers aren't being paid or are receiving revenue in exchange for installing tracking cookies in their origin, they are just letting it happen because they don't care and they want things to be easier on them.

I get that there are ad supported websites, and if this is your business model, then you are free to give it a go -- but I get upset when the justification for the tracking cookie is developer convenience rather than any kind of business decision to drive revenue.

WRT #1, while it certainly isn't difficult to serve libraries from your own server taking advantage of a CDN allows you to take advantage of browser caching and offloads bandwidth.

I'm not saying that CDNs should be setting tracking cookies, mind - I'm just pointing out that as a developer a CDN helps me improve the experience of my users and as a user a CDN helps you reduce my costs to provide whatever value I am to you.

Honestly, if browsers were able to quickly determine that my "reactXX-X.min.js" was the same as Facebook's I would probably just eat the occasional edge costs.

Well, that's the thing -- there are good CDNs out there, but they expect you to pay. But why is the developer making the decision of making the user pay for it with their privacy? Without any kind of notification or agreement, or even discussion, really, with the people who should be deciding this stuff.

Now perhaps I'm wrong, and this is not a developer decision at all but a decision from the business side, where they weighed the privacy issues against the slower performance against the cost of paying for a CDN, and decided this was the right way to go, then they instructed their developers to start loading all these scripts from third party free CDNs.

Except, I've been doing this thing for over a decade, and developers keep sneaking this shit in, and then acting surprised that it's not OK to pull jQuery from ye-favorite-free-CDN, but they need to stick the thing into static resources where it will be served from our CDN. And I keep finding devs doing this also when I do pentests of third party sites, and the response from the security POC is usually "we had no idea this was being loaded from the free CDN. We even have a contract to use Akamai, we just didn't know".

At the same time, I keep seeing this in open source software, even in examples and tutorials, where supposedly the user speed is not so crucial, as well as stackoverflow, and I also know there is a lot of cut and paste going on, so I still think this is just bad hygiene on the part of the developer community, where people are just very cavalier about letting third parties inject javascript into your origin.

But you do have a fair point. End of rant.

If extensions can interfere with each other's internal resources then the real breakage is in the browser's extension mechanism.

Well normally you wouldn't want one browser extension to interfere with another extension. However, there are contexts where that is very beneficial. Take a security or privacy extension which correctly identifies a harmful resource load. It doesn't matter whether that resource is being loaded in a content tab, loaded by another extension behind the scenes, or is local vs remote. It it still a harmful load. So a protective extension should be able to detect it, then inform the user and/or perform an appropriate blocking action.

This, like some other special powers, is something that the user should have to explicitly grant to an extension via permissions interface.

I'm puzzled. Content blocking extensions can't interfere with your extension's internals, and can't interfere with network requests made by another extension.

If your extension injects resources in the DOM, than a content blocker can interfere with the proper loading or rendering of these resources.

However I am very surprised about your statement that a content blocker extension intefered with your extension assets, my understanding is that should not be possible. You mind disclosing what is your extension?

You can't inject a content script into another extension's page but the last time I used it the webRequest API could intercept internal requests, including those from extensions. Although they're planning to crack down on e.g. extensions redirecting search requests from the new tab page and the omnibox.

Most Chrome extensions need to interact with content, so if you want to do that you have to run code inside the content context. Adblockers can intercept any requests you make, even if it's to extension content (i.e. not over the internet, not to an ad/tracking server). This goes so far that they will silently block attempts to open a popup window (in this case, in a click handler to open extension-specific UI for the user.)

The content was under chrome-extension:// and was just an html file. Incidentally I use ublock. I'd be happy to help you troubleshoot this if it's possible to fix the issue.

Using Little Snitch can be pretty eye-opening; I wouldn't for the life of me have imagined that my terminal installations of packages with stuff like Homebrew would send calls to Google Analytics. It's bloody everywhere.

That one is actually documented in their repository: https://github.com/Homebrew/brew/blob/master/docs/Analytics....

And aroused considerable controversy, here and elsewhere, when it was new. The two major active discussions here, 7 months and a year ago respectively:

https://news.ycombinator.com/item?id=13034936

https://news.ycombinator.com/item?id=11566720

I'm not too terribly fussed over it, especially considering what it must cost to support Homebrew - if analytics data helps the maintainers make more effective choices around how to allocate limited resources, that seems to me like a good goal, and I'm not sure how providing information on packages I've installed via Homebrew, to be included in anonymized reports on packages ~everyone has installed in Homebrew, poses any meaningful risk to me.

That said, if you feel otherwise, see https://github.com/Homebrew/brew/blob/master/docs/Analytics.... on how to make Homebrew not send analytics data any more.

Had similar issues while debugging why the advertisement team had trouble deploying their updates using our CI system. Took a really long time to discover the xhr requests for their projects where blocked by the ad-blockers, the irony :)

This seems like a good case in favor of Privacy Badger's model, where content blocking is based on the number of times a domain tries to track you across websites, rather than a blacklist of URL patterns and file names.

Privacy badger has a nasty habbit of breaking login for sites using a login subdomain though. In my experience privacy badger has vastly more false positives than ublock for instance.

In the same vein, I was recently working on a web app with a path at /advertisers, and discovered that ad blockers were preventing any AJAX requests to resources in that path. You might think twice before naming your paths to something containing the word 'advert'.

A couple years ago I was bucketing hash filenames into subfolders by their first byte, like "b328ab391bbeb686b5fc.jpg" into the "b3" folder.

Every once in a while, an image wouldn't load for some people on my website, and I couldn't figure out why until I listed out the paths of the images that had been reported over the last few days.

    /ad/ad67f2864a97aa80be22.jpg
    /ad/ad381cbc443a4fc56e7a.jpg
    /ad/adc225a332e69437a3b5.jpg

Solution: remove the "e" from the filename... "bacon.js", everyone loves bacon.

Already taken, of course: https://baconjs.github.io/ (and they have a picture of Francis Bacon, just to be contrary!)

Happened to me as well when I published the source for a analytics browser extension on github (https://chrome.google.com/webstore/detail/google-analytics-l...)

Anything to do with JavaScript on the page would fail. Thankfully, github's support team was very helpful in pointing out that uBlock was the cause.

Funny you mention that: I can’t access this page due to my adblocker :P

Which adblocker do you use?

I heard that Google's AdWords management portal ran into a similar problem because their web pages had HTML and CSS using names containing the word "ad" because that is related to the domain of ad management. AdWords customers running ad blockers couldn't manage their own AdWords campaigns. :)

Had a similar thing on my website, I put all my images (regular jpegs, pngs, etc.) in a directory named ads (which contained more than ads) and the adblocker was dutifully hiding them - took me a while to figure out the culprit was the adblocker + folder name.

Had the exact same issue so I've recently started working on http://blockedby.com to check regularly if my content didn't get blocked.

It's still very early but any feedback is welcome

its also common for antivirus to block arbitrary JS. not fun to debug or tell users to disable AV on your site.

This extreme vetting made me disable all of those adblocking lists. Now I manually go through web scripts and block the stuff I don't like. It turns out that majority of ads on the web are served by just a handful of advertisers. So my list of blocks include only 27 websites. And these site turn out to be the notorious displays of junk on the internet. I don't see ads on the web anymore. I also don't get called foul on by sites that block adblockers.

Could you publish that list somewhere?

Sites constantly break because their codebase rely on some tracking lib. AirBnb currently fails to display the map because of uBlock.

> So the moral of this commit message is: Yes, objectively bad names exist.

more like objectively bad content blocking algorithms...

i'm not going to pretend i have a better solution than hardcoding file names. but i'm also not going to pretend it's not the responsibility of the ad blocker to eliminate false positives.

Well, I guess between the rising popularity of ad-blockers and issues like these I'll actually install one and make sure to test with it active. May as well take note of the trends and adapt to them.

Reply on news.ycombinator.com