Ad Blocking Extensions Tested for Performance
raymond.cc
raymond.cc
- you can't use the load event to compare advert-heavy vs. advert-free load times. The load event does not fire when the page is usable by the viewer, it fires when every last request is done, in particular even requests fired from various advert iframes. However, those requests may not impact page usability. The load event may even never fire - any offscreen 1px spacer gif that times out will cause that! I don't think there is a built-in event for "done-enough", but the load event is certainly misleading.
- opening the developer tools causes various side-effects; those can cause the page to load more slowly than usual. You shouldn't benchmark with developer tools open (unless you're explicitly targeting usage with devtools open...)
- comparing load times across browsers as reported by the browsers themselves may be valid, but it's not obvious. You definitely want to check that carefully.
- Measuring "peak" CPU usage is almost meaningless without considering how long the CPU is used.
- Measuring the chrome extension process CPU usage and memory usage isn't very helpful, because running this kind of extension causes CPU usage and memory usage in every content tab. Both of these statistics for chrome in this use-case are meaningless. You'd need to measure the memory usage and CPU-time of the entire chrome process tree to get meaningful results. Even in Firefox without e10s it's not valid to measure just the main process CPU and memory usage because plugins are in separate processes (and things like flash or h264 decoding can definitely use CPU and memory).
The only thing this page really makes a decent case for is that Chrome loads pages faster than Firefox - but even there, it's not clear we're dealing with an apples-to-apples comparison.
I don't see why not. It's exactly what they want to show, how much quicker pages can load if you don't have to load the ads. And I personally am not satisfied with a page that has loaded to the extend of usability. Either a site loads fast or it doesn't.
> opening the developer tools causes various side-effects
As long as it's done for all the tests it should not matter. Whatever "side-effects" are introduced will be applied for all tests. Of course this means that the values themselves can't be used to compare the result against other tests, but the goal here was anyways to compare between the different ad blocks and not to generate values for general statistics.
The event does not have to come from within the vanilla browser, without human intervention. It could be from a plugin you create to collect human feedback on load times.
Of course, the human element will significantly decrease the number of trials you can do within a fixed time period.
Even that is not enough. A number of sites (especially ad-heavy sites on mobile devices) look ready to read, but as soon as you try to scroll, are jerky and semi-unresponsive. It's particularly a problem on ad-laden sites of small news publications. Blocking ads sometimes mitigates this, though not always, as some sites insist on using Javascript to implement "fancy" scrolling which inevitably breaks.
(Semi-related sidenote: please, for the love of God, don't use Javascript to hijack browser scrolling. It's almost certain you will not get it perfectly right and some users will suffer. It's almost never worth it. This is the #1 reason I end up having to switch to Reader Mode in Firefox).
The only reason I can think of where I'd be interested in how quickly people can navigate a site without having loaded all the content yet, is when optimizing a site for people with really slow/bad internet connections. But that seems like a whole other topic that has barely any influence on the goal of the comparison.
And then the page completely re-flows because some big advertising asset popped into existence further up that you weren't expecting.
For example, you might try to wait for all stylesheets to have loaded, and then run requestAnimationFrame until the latency is below some threshold (say, 100ms) to represent a somewhat-loaded-and-not-too-laggy page.
The point is, you want some measure that plausibly excludes ad-related resources since clearly you don't care how long it takes to load those. It'd also be polite to be transparent about the limitations of your measure.
Comparing times between browser-defined events in the same browser opening the same site on the same day is about the most apples-to-apples as you can get.
For your first point, the measurement wasn't "usability", it was how long it took for the load event to fire.
Second, having dev tools open should affect all results the same way. The absolute number isn't important, it's the relative ordering, which should be the same whether dev tools are open or not.
The rest of your points are just nitpicking. Maybe he didn't use the absolute best way to measure CPU or memory usage, but I'm willing to accept they're good enough proxy measures that the results are relevant.
Its sort of like measuring profit by revenue - sure, there's some correlation, but it's far from obvious.
How is that in any way helpful? If anything, the proper, comparable load time for the adblocked version is infinite - they never complete loading the ads.
If you're going to run this comparison, you need to at least compare the same set of resources being loaded - after all, if you're willing to omit ads entirely, you clearly don't care how long it takes for them to load.
We don't need a perfect measurement, but this measurement is extremely biased. DOMContentLoaded would be better, but still far from good - you want to measure the load time of the resources the adblocker would not have blocked.
Running WPRUI with first level triage will measure CPU along with a bunch of other stuff. Enabling ReferenceSet under memory Resource Analysis will measure memory usage.
Chrome has an ETW provider, I poked around a bit and couldn't find any for FF (it looks like they register one for the JS engine?)
I'm not familiar with how to solve the "page loaded enough but not done" problem, but IMO if you measure end-to-end times it's probably a close-enough proxy. Maybe measure several times and report on those statistics.
(Edit: Taking a trace with/without the adblocker enabled should allow you to exactly count # of sampled cycles spent in the adblocker - easy to compare in WPA, as long as there are events emitted indicating page load start/stop).
(Edit2: Using UIforETW along with Chrome 46+ will let you see ETW events:
Measuring peak CPU is very, very stupid. The best Ad Blocker should have near 100% CPU but for an extremely short amount of time. Using 100% CPU is making the most efficient use of the hardware, not being inefficient. Now, I think due to the way they're measuring, it's actually closer to being average CPU because task manager samples over a period.
Also: "Firefox didn’t like this page at all and we couldn’t get consistent readings to run tests, on every refresh the browser would simply not respond or crash."
Huh? I think somethings wrong there. Firefox obviously renders tmz.com. A bug in the thing they used to monitor page load times? Did they check the performance impact of it?
Also, Privacy Badger reports 47 trackers on raymond.cc but just 20 on tmz.com ;)
that makes me question the validity of the test in general... also it'd be interesting to benchmark tracking protection vs ublock on Firefox.
You'd be surprised about how many people perceive this as a bad thing. I've more than once had tickets from customers complaining about high CPU usage under load scenarios. Can't wait to see what journalists make of Servo (which, if up to scratch, will hopefully peg CPU near 100% for a brief period).
However, higher CPU usage associated with longer response times is a different story entirely.
> it's actually closer to being average CPU because task manager samples over a period.
Forget about that: they're even using the wrong tool. Perfmon should have been used. Eyeballing it is completely useless.
> Firefox obviously renders tmz.com.
Renders perfectly for me, too.
What they're really saying is not "you should make the CPU usage lower by spreading it over a longer time", but that the CPU is at 100% for an excessively long period of time. In other words, that 100% period should ideally be so short that it's below the sampling interval and does not appear as 100%.
Not necessarily with modern processors. A power-managedCPU at 50% clock-speed actually uses less than half of the power as at 100%.
The relationship of speed to power is not linear, so for a laptop user the most efficient use of hardware might actually be running one job for twice as long at half-speed.
I personally dislike it when an application makes my "PC" lag every now and then. CPU time and RAM are shared resources, thus applications should make sure to not act selfish unless asked for (e.g. encoding a video, etc).
But it'll take longer to render the page, which you also don't want.
That's ignoring the overhead of not being able to sleep the whole CPU, including uncore stuff for twice as long if you're only computing at half the speed.
I think it's pretty obvious that the expression "peak CPU usage" implies "peak average CPU usage over a period of time", for the simple fact that literal "peak CPU usage" makes absolutely no sense -- CPU usage is always over a period of time.
I have 4 cores and another 4 through hyper threading.
If 100% of my CPU is ever used up to render what amounts of a couple kilobytes of text content, then I start to wonder what is wrong.
I previously worked at a digital agency and one of my final projects was building the website for a large TV channel. At one point I had to have a meeting with the ad network to go through the integration.
The technical people involved didn't know anything about how their products worked or the implications of integrating them. No idea if they were blocking or async.
At one point I was trying to understand what, if any, changes I might need to make to handle their ads that did full background take overs. "No no no, it's an expanding mpu" - sure, fine, what does that even mean!?
(Don't get me started on having to swap using js for switching images in the gallery for using individual pages because the onmiture page tracking numbers were the metric that everything was measured by for selling ad space)
/rant
You'd think with all the smart people that have put their minds to ads on the internet, there would be lightning fast, targeted ads that actually worked by now.
It's called Facebook Ads, and Google AdWords. Another name for what you seek is "product placement in movies". Yet another is PR.
For instance, looking at the trailer for the new Transporter, one has to wonder how much Audi paid to have that movie made.
And if you look closely at most journalism on the modern web, it is nothing but a thinly veiled ad for a thing.
Indeed the longest events on the page - as reported by Firefox dev tools - are connections to https://tpc.googlesyndication.com/ which are 0.7 and 0.8s; connection to https://www.google.com/pagead/drt/ui which is 1.3s; and connection to http://fls-eu.amazon-adsystem.com/ is 1.4s.
Meanwhile on a commercial page I manage the second hint on my 94/100 score on Pagespeed is to "leverage browser caching" - all the resources listed, like api.js, are part of Google's analytics. Under minify javascript there are Facebook and Google resources (for Like and +1 buttons).
This is of course not a scientific analysis; instead I say it to illustrate that the smart people at Google and Facebook don't appear to be focussed on this end of things - I'm guessing the minify thing saves them resources, that they minify based on a balance of delivery costs vs. processing costs and that full minification is not worthwhile to them ... I'm just annoyed I can't get a 100/100 pagespeed score without removing Google's ads/analytics from pages!
If you give me a link I can file bugs and follow up with teams. This shouldn't happen and PageSpeed is right to call us out on it.
(The initial loader js you reference on your page does need to have a short cache lifetime so it can be updated, but everything else should be longcached. And everything should be minified.)
Sorry to hear about your failing business model.
http://www.adweek.com/news/technology/how-ad-blocking-could-...
This is an interesting thought. Working in a environment close to advertisement (travel recommendation engine), I have a few possible explanations on that.
- First, working in with a product that have no (or close to) added value for end users can be very bothering, and top engineers care about doing useful things.
- Second, to design a fast, simple and honest ad network you must fight your own business/marketing team (because they always want more data, higher click through rate, etc). Since they have most of the power here, they end up hiring more diligent engineers who won't argue about privacy, curated content and other stuff.
- Third, top engineers are working in ad companies, and secretly conspire to make the whole system collapse from the inside.
It's not necessary to get computer science geniuses for this, and frankly I need people who get stuff done rather than trying to save the world through code. But the talent has been subpar because most of the companies outside of the top few (google, facebook, twitter) are smaller and not tech driven. They just pick up whoever can code up some simple server and get an ad on the page. Add in all the shady companies and it gets worse. Optimizing for the user just hasn't been a requirement.
We have a honest and fast ad network, probably the best you'll find. Quick, simple, static ads that are non-intrusive. And we actually perform quite well. It's not somehow at odds with the model, it's just harder... the question is, who's willing to work harder at this?
In other words, use referers to figure out if they came in based on keywords from a search engine (ads targeted more towards that, but still biased to the page viewed) or otherwise just based on the content of the page.
That's it; the first can be quickly memoised across a set of common terms for a page (and while doing so the faster content of page based results can be served instead).
No more silly tracking, just showing what someone is likely to be interested in at that moment.
However, what about pages _without_ ads? Am I going to experience a significant performance hit when running an adblocker on genuine websites? This is important. Because, assuming the advertising industry is going to self-destruct tomorrow morning, am I going to experience a slower web if I let my add-on activated, and thus letting an opportunity for the ads to reappear later?
That is very different from a page that fires DOMContentLoaded as quickly as possible. That metric is pretty much irrelevant to real people.
Which is part of why ads aren't quite as nasty as a lot of tech industry people make out.
Check how well http://www.theguardian.com/ loads and you can see that it's perfect possibly to have plenty of ads while still having a great reading experience.
It's not really the backend that matters, these days its the front-end setup that makes the biggest difference. Less javascript, less huge unnecessary images, less "interactive" crap and streamlining the loading of content first then ads async can make a site load pretty quick. They also style the ad slot dimensions (since all banners are standard sizes) which keeps the content from moving around when an ad does load.
Its just simple effective optimizations, something that should be done but for some combination of bad talent, bad management and poor priorities, just isnt.
Around this time in 1997, the PBS Newshour website[1] was an outrageous 113.7KB, which at the very top end of the GSM GPRS 2G networks of the time, 115 kbit/s, would take 8 long seconds to download. Why? Because of all of those obnoxious Sun Netra banner ads![2]
What do you get in 8 seconds on LTE today? Lord of the Rings in 720p. Or a whole bunch of Buzzfeed pages with some ad trackers, sure.
[1] https://web.archive.org/web/19970605132107/http://www1.pbs.o...
[2] https://web.archive.org/web/19970605132107im_/http://www1.pb...
I often think that companies like Ghostery are missing a trick by not advertising themselves as a method of speeding up your internet connection because that's really what it does.
https://mailman.stanford.edu/pipermail/liberationtech/2015-A...
(I worked as CTO of an adserving platform and had to reject sales calls from Evidon on an almost monthly basis)
With surf+tabbed, each tab is it's own process, so you can enable and disable JS and or plugins per page.
Something may be broken on 95% of web pages (e.g. commenting or menus), but the main bit of the web - rendering hypertext and following links - mostly works.
Web applications less so, of course. But I don't think they're 95% of the web.
If you really need scripting on a page, it's very uncomplicated to add an exception in Chrome: just click the "Javascript blocked" icon in the address bar, and choose "Always allow Javascript on example.com".
In my experience using NoScript for the past year on all my desktops and laptops, this is at best only partly true.
Load times do definitely decrease, but this is dwarfed by the effect of having to reload the page for most new sites after re-enabling Javascript (since so many sites are unusable without Javascript[0]). In the end, it's a much slower experience browsing the web, not faster, and even with aggressive whitelisting, it's still rather frustrating.
I actually have separate Firefox profiles that I use for various websites that tend to be particularly problematic with extensions (like online banking), so I'm not even including those in this summary.
I wish using NoScript provided a faster browsing experience overall while providing better security, but the simple fact remains that right now, it does not. It provides better security at the expense of slowing down your browsing experience.
[0] So many static blogs now literally fail to display static text without Javascript!
Why would someone want to use javascript on the client-side for things where it's not necessary?
Year after year, the browser and the web are turning into "something else", and while the possibilities are exciting, it seems the use cases where javascript is used for things that actually make a positive difference, are vastly outnumbered by the cases where it is used to re-implement in a more flashy, complex and resource-intensive way things that have been usable, performant and functional, 10 years ago.
IMO displaying content should be done just by HTML, using javascript if it's not to provide useful functionality, is wrong.
Whether it is browser HTML/CSS incompatibilities, flash atrocities or complex script bloat, it seems the popular web always manages to fix one problem and create another.
When I click a link and see a blank page, I often click the back button or close the tab rather than allow it to run scripts. You do not need a script to display text. That functionality has been present since Sir Tim wrote version 0.1 .
Would it be so hard to provide alternate text for no-script browsers that says "Some content on this site is only viewable to users that allow scripts hosted by www.example.com and js.example.com to run on their browser clients."? No. It's not. There is no excuse for serving a blank page full of invisible scripts.
what justifies the usage of "sketchy" for these particular addons? Many of them are open source, you can audit them yourself. Or you could hope that the reviewers did a decent job.
Works wonders on all your devices. https://wijvrij.nl ( dutch )