Why we challenge Microsoft’s battery test | Hacker News Reader

Why we challenge Microsoft’s battery test

opera.com

302 pointsdoener210 comments

Edge platform team employee here. A couple thoughts on methodology, focusing on our scripted test (under "stay productive longer" at https://blogs.windows.com/windowsexperience/2016/06/20/more-...), which is most comparable to Opera's test; nobody seems to challenge our Netflix results:

* We did not enable ad blockers, because we were testing the efficiency of the network stack, browser rendering pipeline, etc.; if you cut off ads, you're effectively skipping half the assignment. This is a valid way to improve the user experience, but not a valid way to measure browser efficiency, especially because it will disproportionately impact some sites (like the news sites that Opera's test focused on) and have no impact on other sites (like Netflix, where Edge demolishes the other browsers). It's basically skipping a lap of the race and then bragging that you finished faster.

* It's also worth noting that the ad blockers are often detected by many news sites and will actually prevent the main page content from loading at all. Not sure if Opera's test accounted for this.

* Our test was designed to mimic real-world behavior: Watching YouTube (foreground and background), shopping on Amazon, browsing the Facebook news feed, searching on Google, opening email in Gmail, and reading Wikipedia. To reduce variability, we used WebDriver to instrument the tests (supported by all four browsers tested) and made sure each task was timed rather than just a loop of consecutive tasks (which could disadvantage or advantage browsers depending on other factors like pageload performance or network conditions, in a way that doesn't reflect user experience, which is likely to linger on a page once it's loaded). We then used the Maxim 34407 power instrumentation built into the Surface Book (which is why we chose the Surface Book for this test) to measure actual instantaneous power consumption at the hardware, sampled once per second and then averaged across the duration of the test. We feel strongly that this is a highly scientific and defensible test setup which mimics typical user behavior and, significantly, measures the same markup and the same duration, on the same hardware, in every browser.

The way I read the blog is that the adblocking is enabled to show that even _with_ an expensive extension enabled it still beats Edge, but that was in my opinion not the gist of the post.

The post was about that microsoft is not transparent about the methodology/setup/scripts/target websites used. A third party should be able to support Microsoft's claims.

Is it possible for the setup to be published to your github.com/microsoft so that it may be executed automatically? Heck, go the extra mile put it in a CI and publish the data on regular basis :)

Adblock isn't an "expensive extension". Ads are the expensive thing, so blocking them saves power.

Until we know exactly what websites/videos etc are loaded (also on the Opera benchmark!); you, me and all consumers do not know if this is true :)

Also Opera has built-in blocking. It's not a JavaScript-heavy extension to begin with. ;)

It's not quite that simple [0][1], or at least it wasn't always. AdBlock Plus in particular used to rely on a massive style sheet that was injected into all tabs and frames, which degraded really quickly on pages with a lot of iframes (which, as it happens, is typically pages with lots of ads).

I believe Firefox partly resolved the problem on their end later on, although I can't tell you the exact status of things. Nevertheless, there's definitely some precedence for claiming that ad blocking is an expensive operation. Intuitively, you'd think that network based blocking would be enough, but it won't work against same domain ad sources (a typical example being facebook ads), while css selectors are able to capture a bit more depth. Nevertheless, I don't know exactly what has happened since, although I do recall Firefox making some adjustments on their end that improved the situation.

[0] http://www.extremetech.com/computing/182428-ironic-iframes-a...

[1] https://blog.mozilla.org/nnethercote/2014/05/14/adblock-plus...

No, it really is that simple[1].

The only reason I can think of for someone not using uBlock Origin is because they've never heard of it.

[1] https://github.com/gorhill/uBlock#performance

I got rid of it because it made using the web nearly impossible and I got annoyed with constantly having to manage exceptions just to do things like view my Twitter analytics page. Downloading ads is less annoying than uBlock.

Click ublock icon, click power icon, done.

yeah, so why have it installed at all?

So that it blocks the 99.9% of unwanted crap.

Try dynamic filtering (in medium mode)[1] and raise that number even higher.

[1] https://github.com/gorhill/uBlock/wiki/Blocking-mode:-medium...

So you can browse the web on your own terms.

The power button only disables it for one site at a time (or even just a single page, if you Ctrl-click it).

Clicking the power button only disables it for that specific site. Every other site would still have blocking enabled so that sounds like a pretty good reason to have it installed.

What filter lists were you using? uBlock Origin's behavior is identical to Adblock Plus given the same filter lists are used for both. uBlock uses a few more filter lists by default although I still think it's on the conservative end of the spectrum in terms of what gets blocked.

I've only ever had to whitelist two things in uBlock in all my time of using it and that's pretty good considering that I use basically all of the lists that aren't the language specific ones, the anti-anti-adblock one (which requires a user script) or the merged (ultimate) lists.

But yes, my browsing habits are clearly different from yours and what works for me may not for you. I understand that but what uBlock does is not any different than any of the other adblocks that are out there. It just happens to do it more efficiently and with a better UI than the rest of them. The only difference in behavior that you might encounter is likely to be related to uBlock Origin's strict blocking[1]. In this case exception filters are very easy to create since you literally just have to press the disable permanently and it will forever be disabled for that site.

I've said this before the the best way to use uBlock Origin if you've never used something like NoScript/uMatrix before (or couldn't be bothered with the whitelisting approach) is to try and use what it calls "medium mode"[2]. Using its dynamic filtering in this way should net you with the largest gain with the least amount of effort. If you're looking for something with more control than you may want to look into uMatrix since I think the interface will is nicer for that sort of control.

[1] https://github.com/gorhill/uBlock/wiki/Strict-blocking

[2] https://github.com/gorhill/uBlock/wiki/Blocking-mode:-medium...

Doesn't the Netflix test really test GPU decoding vs software decoding?

I'm not 100% sure on this, so any clarification welcome, but I thought that Netflix content contracts DRM requirements allowed GPU decoding via Playready, but not on some others like Widevine. Would the Netflix test still show such an advantage for Edge if there weren't any DRM?

It seems to me that if you're running on the same device, the gpu should not be a variable.

IIRC everything except Edge and Safari are forced to use Silverlight for Netflix streaming. I'm not exactly sure why this is, but it probably has something to do with the mess that is video codec licensing.

It's for DRM purposes

Netflix plays fine in Chrome (maybe Firefox too) using HTML5, it was opt-in at one point not sure about now. I watch in Linux all the time, no Silverlight there.

Starting with Firefox 47, it works fine as well (macOS here), but you need to switch your user agent to Chrome (before you go to Netflix). I'm guessing Netflix hasn't updated their compatibility checker.

Chrome uses widevine (DRM) for Netflix which uses HTML5. Really the more significant difference is that Chrome gets capped to 720p. Edge actually gets the 1080p streams (as does IE11 on Windows 8+ and the Netflix Windows Store app).

Seems arbitrary, what's the reasoning for that? Isn't this exactly what everyone warned about with DRM in html5?

Your guess is as good as mine. Blame Netflix, Microsoft or the people pushing DRM. Maybe it's some combination of all of the above or none of them. I'm not sure anyone really knows why we have this behavior.

It's not necessarily DRM in HTML5 which is the issue here although I suppose the fractured nature of it may be partially to blame. It's the fact that it's seemingly arbitrary that Chrome gets capped to 720p and no one knows why. Did Microsoft pay a big chunk of change to Netflix for exclusivity or something along those lines? Do the content creators prefer Microsoft's DRM implementation over Widevine's?

It would be interesting to see this test run again with ad blockers enabled on each browser (the latest version of Edge has support for extensions).

So where is your script? Or are we just supposed to trust you?

you don't have to do anything.

How did you control for the specific ads the browsers were served? Ads are highly dynamic. Unless you (can) include the differences in your methodology you better eliminate that influence.

Ding, ding, ding and to be fair this is the problem in both methodologies. The only way to accurately perform this test is to spider a bunch of sites, save the contents to a locally hosted HTTPD and ensure all third-party JS calls are resolved locally and test both against the exact same sites as they were spidered at a point in time. You can simply not account for changes which may happen to the markup, ad network beacons, ads, metrics code, or even network routing, all of which could influence the test in one way or the other if not run from an identical static cache in a controlled environment. If I were doing this test I'd then skip the whole scripting automation part and simply add a meta-refresh to every page in the cache to sequentially take the browser through the content, giving each page something like 10 seconds to load and render. Simple, simple, and far more accurate.

You don't need to go to such complicated lengths. Just perform enough tests (as in, a statistically large enough amount) and a distribution will form. That also captures the variability of real world network effects.

It's hardly complicated. I've put such tests together in an afternoon. In fact, whatever is added in complexity is gained by the fact fewer tests are necessary. Via this mechanism you can also remove any questions about compression, use of HTTP/2, etc., which could impact the tests based on server-side choices when it comes to serving data to either platform. Equal always equals better.

But those metrics are important, if servers serve more optimized pages to Edge users for some reason that a freaking important fact to know. This is about real world data and real experiences and how it affects actual real users. You can normalize the tests to the point where there is absolutely zero difference between the browsers, of that I'm sure, but that will not reflect any actual cases that real users experience.

That would be a whole different test then. Not about efficiency of the browser itself but about what the browser gets served.

The test was which browser gives you the better battery life while browsing the internet like a normal user.

Why is a different question and it's not that relevant, I often don't care why something I USE works better I just care that it does.

If it something I BUILD then I would care much more but again this is a whole different issue.

Within those specifications, the objection about the ad-block become irrelevant. If the browser justs works better, then users don't care and can simply enjoy more battery time.

The case for more normalized tests is to find out which browser is factually better designed/written.

But this is not repeatable.

What about different ads served to different browsers? Someone running, say, Opera will have a different ad profile than a Chrome user even when completely blank cookie-wise.

Maybe have them give the same user agent?

Ads are not that much of a problem ads will even themselves out and if for some reason MSFT Edge users receive less ads or ads that are less resource intensive it's also an important metric.

I don't see anything that would somehow create a bias in favor of a specific browser as far as ad networks goes, if anything the stigma/stereotyping of IE/Edge users would probably mean that ad networks are more incentivized of sending the baity apps towards those browsers.

As for the network part well again that's an important metric if certain browsers perform better at adverse network conditions it's an important factor to know, you do not want to give them the best case scenario every time.

Giving a page a fixed amount of seconds to load is also completely the wrong approach you want to see how browsers behave when they can't load a page properly or when it takes more time than usual, maybe some browsers expend more resources by resubmitting the entire request, maybe some browsers do not parse the DOM tree from scratch when some of the requests stall, maybe some browsers have less resource intensive placeholders for DOM elements, maybe some browsers are better at adjusting the DOM preprocessor for network congestion than others.

So no I can't really see how would your approach would be any better, the approach that MSFT took was quite good, netflix, wikipedia, youtube, facebook etc. with what seems to be realistic user behaviour. What you want to do is to put in test that would produce fair results for fairness sakes that's not how you evaluate anything because it would not yield you any real world data.

That would be as far from real world as you can get.

What you need here is a big enough sample.

You'd probably need to run multiple tests to account for the variance of ads served.

You run enough tests to even things out and if you still have a bias towards certain browsers then it's a statistically important metric on it's own.

Given that basically everyone should be blocking ads (if for no reason other than that they are a vector for malware), native ad blocking is really an energy saving mode, a user experience improvement, and a crucial security measure all rolled up into one feature.

If the focus of this test is meant to be on the user experience, as the tone of the Microsoft blog post seems to suggest, then said native ad blocker is a killer feature that really might mean Opera users can "stay productive longer".

If not, then sure, Edge's underlying implementation is probably faster. It would be interesting to see a comparable benchmark of Edge with the best available ad blocking turned on.

Wow this is all fascinating! If it is not too much trouble, would you mind releasing the script that you used in your testing so that independent parties may verify your results.

Could you disclose/open source your methodology?

> nobody seems to challenge our Netflix results:

Of course, this is Lotus vs Office all over again. You're using the OS advantage (Windows 10 and its DRM platform) to show that you can do things more efficiently.

You chose the technology platform (I'm guessing Play Ready), you forced GPU vendors to comply, and now you give competitors a bad reputation for daring to use their own (or a competitor's) technology.

Or, to put it another way: the have built a well integrated technology stack and are now demonstrating that it has superior performance on a metric we all care about.

Apple has a different approach (own the entire stack) but they're going for the same thing and it has made them a lot of cash.

Netflix is a rather specific example — don't they use Silverlight (as in "the only site anyone in the world cares about that uses Silverlight"?)

It's like Apple stressing how well their browser works on -- oh wait, Apple doesn't produce a proprietary plugin. Well, let's say the iTunes page, if it were specifically tuned to work well in Safari.

Edge and Safari run Netflix under HTML5's protected media path. Silverlight is only used for backwards compatibility.

It may not be fair to enable adblocking if you want to compare other engineering components, but these results still mean users in the real world will see more battery time with Opera than with Edge.

This is

* not based on observed user behavior

* not reproducible by you (recorded/replayed network where possible)

* not reproducible by peers

* in stark contrast to reproducible tests provided by others

You describe it as "highly scientific" and jump straight to a marketing campaign. Honestly, who's skipping a lap of the race?

It's pretty clear the data generated is useful from an engineering point of view. It can help identify problem areas. It's also absolutely clear that the conclusions being marketed are blatant misrepresentations of the work.

Microsoft didn't just do a battery test, they also release statistics based on Windows 10 telemetry showing that Edge has better battery life than Chrome, Firefox:

"In addition to testing battery life internally, Microsoft also looks at tens of millions of PCs that send telemetry data to the firm, Weber said. And what it sees there is simple: A small advantage over Firefox, but massive gains of about 50 percent additional battery life when compared to Chrome."

https://www.thurrott.com/windows/windows-10/68176/microsoft-...

Not to discredit their results, but what it shows is that users who used Edge consumed less power than Chrome or Firefox.

Whether that means Edge is more power efficient, or devices where Edge is used more often are more power efficient, or users who select for Edge tend to visit sites that allow Edge to be more power efficient isn't clear.

To be fair to both sides, this is a really hard thing to measure.

> To be fair to both sides, this is a really hard thing to measure.

Is it though? It's more the fact that Big Data is totally useless because it contains thousands of uncontrolled variables that confound the results.

If you really want to know which browser is more power efficient, get a sample of the commonly visited sites and then visit the same sites on the same common devices in different browsers and see which one uses less power.

You can still get the wrong outcome by using a biased sample on purpose, but then the outcome is wrong because you fudged it, not because it's hard to get the real answer.

That's what opera just did!

That's also what Microsoft did.

It's quite possible that both results are valid. It sounds like there's big difference in the sites that Microsoft visited compared to Opera.

It seems an even better methodology would be to record actual user browsing sessions, and then replay them through different browsers and measure power consumption.

Opera's simulated browsing session does not sound particularly sophisticated IMO.

Couldn't you just make a giant virtual "browsing session" with hundreds of thousands of different sites , and crunch the numbers to minimize the background noise? MS has the computing power, that's for sure.

Except that this has an interesting failure mode: unless you're visiting the sites by hand you're presumably using some browser automation framework, with different implementations for different browsers. And _those_ can have nontrivial power usage, depending on how they work. I know for a fact some of them do power-hungry continuous polling...

It seems pretty unlikely that the data wouldn't be normalized by some metric of intensity of use since this is pretty much the first thing anyone would think of. Normalizing even by something as simple as "number of websites visited" would, I imagine, significantly reduce the contribution of such factors to the differences observed.

Or they just ignore normalization, since it's for PR and not an actual scientific study meant to withstand scrutiny.

The fact that they spent so much money in hardware to test the battery usage and the amount of effort they put in to improve doesn't seem to imply that this was just a PR stunt.

You underestimate the value attributed to PR by large corporations.

> Not to discredit their results, but what it shows is that users who used Edge consumed less power than Chrome or Firefox.

The workloads Microsoft simulated were held constant for every platform and instrumented via a common command-and-control testing library. It is very much apples-to-apples. The additional telemetry they released is even more confirmation.

Opera replied by enabling a pair of special default-off features, both of which significantly change the workload and the user experience for their browser, put it on a random laptop pair which might not even be from the same lot or subtly different models, and then made a timelapse.

> The workloads Microsoft simulated were held constant for every platform and instrumented via a common command-and-control testing library. It is very much apples-to-apples.

I'm not doubting this part at all.

> The additional telemetry they released is even more confirmation.

The additional telemetry doesn't really tell much of a story either way, given that it's presented as a single chart and little context is provided. Unfortunately there's no link to the data analysis to understand what factors were considered when looking at the data sets involved.

Personally I've found that it's hard to come to these high-level conclusions about performance from large data sets, because users and their devices are so incredibly diverse. It's really hard to draw definitive conclusions given all the data.

It's dangerous to take your data, aggregate it at a high level and look at the results from that point of view - you lose the context under which it was gathered and can come to the wrong answer (for example, browser X is popular in a country with great internet, so page loads are faster, and browser Y is popular in a country with terrible internet, so page loads are slower).

Which is impressive I guess given that Firefox users often use extensions of varying quality?

I'm sure there are a lot of Firefox extensions of questionable quality but I would expect that at least the popular ones are of fairly decent quality. I also would assume that only a small minority of users know about and actually use extensions.

Even people who do use extensions probably don't really use more than an adblocker.

An extension doesn't have to be popular to be significant. 1% of people accidentally installing some spyware crap that slows down the browser by 50,000% is going to move the needle.

Adblock Plus is very popular and very inefficient: https://github.com/gorhill/uBlock/wiki/uBlock-vs.-ABP:-effic...

Is that telemetry data able to determine how much of the battery is being used by just the browser? It would seem difficult to me to distinguish between battery power used by a browser, and say, code compiling in the background while I surf the web.

People who use the Edge browser may do less on their computers while web browsing than people who use chrome?

> they also release statistics based on Windows 10 telemetry showing that Edge has better battery life

It's such a pity I don't use Microsoft Windows 10, because I'd be so thrilled to know that through the magic of telemetry, everything I do on a Microsoft Windows 10 enabled computer could help Microsoft win some crucial PR battles.

As opposed to your Apple iPhone where through the magic of telemetry every app is dumping your every finger motion into services like Mixpanel and Kahuna for later analysis.

For clarity, they captured the extremely detailed telemetry from their test devices. Most of the telemetry that Microsoft ships back to the mothership is the exact kind of telemetry that apps from the Apple Store, iOS Store, Android Store and Chrome Web Store submit: app specific telemetry. Many of these include signals that help understand battery life impact, so every major application platform (including the web) is doing this.

That stuff–stuff you agree to in the EULA as it stands– is absolutely essential to improving app experience to the standards of the market. The reason the mobile ecosystem has developed so fast (and that the web ecosystem can develop so fast) is a whole lot of telemetry on user usage patterns, habits, device failures, device selection, etc.

I'm not sure why it's so controversial. Even the Ubuntu store apps can administer telemetry.

If you'd like to complain about the pervasive surveillance state in your national scale, there are more fruitful avenues.

> As opposed to your Apple iPhone where through the magic of telemetry every app is dumping your every finger motion into services like Mixpanel and Kahuna for later analysis.

This is not true at all. And there is a difference between your smartphone and your laptop, even if you can't see it.

Just one of many links on Google for an appropriate search string. This one is dated and no longer accurate, but the phenomenon still occurs.

http://lifehacker.com/lets-talk-about-apples-privacy-issues-...

They have and they do, on every device. And that's ignoring app specific telemetry, which is explicitly allowed on all app stores.

It is one thing to not want to share that data. It's another entirely to invoke a 7+ year old corporate culture as evidence Microsoft is the sole "bad" actor in the space when every vendor openly discusses their telemetry collection.

How much did we hear about Chrome telemetry at I/O and PWAS? A lot. More to come. And it's necessary to make software products affordable o produce, unfortunately.

Yes, telemetry from my smartphone is even more invasive to my privacy, having my location at all times, as well as being the single location with all my communications.

I wonder if we can make a bot that posts a telemetry comment on every article with "Windows 10" in the title.

Don't worry, for your convenience, that Telemetry technology was backported to Windows 7 and Windows 8 as well.

There are some programs out there that can help block/disable it.[0]

[0] https://www.safer-networking.org/spybot-anti-beacon/

So Microsoft is great based on some secret data that we have no access to. Which microsoft collects from my computer without my knowledge. And it is all collected by their opaque closed source os. But hey they may be right. Nothing stops them from adding code in windows 10 that intentionally wastes power whenever the user is running chrome.

The more i read this the more pissed off i feel about using windows 10. Unfortunately, they made it very hard to install another OS on new laptops.

You can intercept this data with a proxy and your own root SSL cert if you want. MS doesn't do certificate pinning unlike some people in this space.

Opera with an ad blocker vs Edge without it doesn't seem like an ubiased test.

And they say "Opera Developer (39.0.2248.0)" - which is presumably a dev build of the browser.

Like it would be fine if they say "in next version we will retake the top" or whatever but don't try to dispute their results by providing something they had no way of measuring.

The developer stream is public, and so is the exact build they used[1].

[1] http://get.opera.com/pub/opera-developer/39.0.2248.0/

I wouldn't switch to an unstable browser just to improve battery life. It just shows that they're not really trying to get a fair test. I'm running ad blocker in Chrome and it works perfectly. I'd be interested if they could improve on that significantly, but I trust Google more in creating a stable software.

I think you are unaware how marketing works (the whole power saving competition from both Microsoft and Opera is just marketing).

Essentially they did some power saving improvements and want to create a bit of noise about it to increase their popularity.

It's kind of same as how Apple, Google etc announce new features/products before you even can use them.

At least Opera was transparent about the details of their test. Was Microsoft?

It doesn't.

But there's no reason anyone should be running a browser without an ad-blocker in the first place. Edge should have a lightweight ad-blocker built-in by default.

How would that fair with Bing's ad network?

How is that the user's problem?

Microsoft adding an ad-blocker by default, while also owning an ad service seems to be a set of conflicting interests.

It's not, but it's Microsoft's problem. The reason Edge exists is to try to win back market share from Chrome. The reason Chrome exists is to perpetuate the Google ecosystem by making Google offerings cross-platform, but more importantly Chrome-platform -- not just Google ads.

But you're right, Microsoft will have to decide which one is more important: driving ad eyeballs to Microsoft properties at the expense of Google properties -- or -- decreasing the share of users using Chrome and by proxy, the attractiveness of Google's integrated cloud email/drive/docs offerings.

Considering Microsoft recently added ads to the Windows 10 login screen, I wouldn't hold my breath on Edge coming with an ad blocker out of the box.

EDIT: Downvotes? Really this is factually accurate:

http://lifehacker.com/windows-10-is-showing-ads-on-your-lock...

The article you linked to specifically talks about how easy it is to turn them off.

I knew someone would make this lame excuse. That's like saying "Hey if you don't want me to take a shit on your sandwich it's really easy to order it without the shit."

OP never claimed it wasn't easy to turn them off.

The thing is, ad blocker is disabled by default in Opera.

Even with the ad blocker being a feature in opera out of the box? They were trying to show that opera can achieve better battery performance based on it's features and optimizations. Does that not include the ad-blocker?

In that case you might as well browse different websites because using adblocker makes your browser render different content.

Microsoft is welcome to include a native content blocker in their rendering engine

While I applaud their work and appreciate that there is an extension framework at all, I seriously doubt this will happen unless they plan to also redesign their home pages to remove the ads.

I just loaded up MSN.com and scrolled down the page. I count 3 ads total: one gigantic banner at the very top of the page before all of the content, and two smaller panners that pop into place as you scroll. These are served by AdChoices. When I turn my ad blocker back on, these all disappear, and the site loads much faster.

I think if Microsoft were to release a default ad-blocker, they would need to re-code their own website to use native advertising rather than a third-party network. I believe this to be an excellent practice, but I just don't see Microsoft going this route. I'd love to be pleasantly surprised though.

I assume they would implement a white-list allowing "some non-intrusive adds" similar to Adblock.

Sure, but then my browser with native procrastination protection will beat the pants off both: when you type in a url it just renders "get back to work, maggot."

It even works without a network connection.

Not necessarily true if both ad blockers are using the same filter lists. I find Adblock Plus and uBlock Origin (I prefer the latter) to be relatively on par in terms of things they block and things they tend to miss, even cross browser. I think the initial pageload would be the biggest differnce, as it relies on the performance of the adblocker in applying its filters to the DOM once it's loaded. The underlying extension framework probably affects this a lot too.

I'd think the user agent string is more likely to cause a huge discrepancy, as many frameworks will be utilizing polyfills and employing different technologies depending on what each browser supports. (Or, in the case of the user agent string, what the framework thinks the browser supports, regardless of how correct its assumptions are.)

His point is that only one of the browsers was using an ad blocker so they were rendering different content because one rendered ads and the other did not.

Oh, I totally missed the part where Opera has the ad blocker turned on by default. Ignore my rambling then; my sleepy brain thought he was talking about two browsers that both had different ad blockers installed.

Not sure if Opera has it on by default - all they say is that it is build-in so that somehow makes it okay to use it and compare apples to oranges :)

If I understand both tests correctly, Opera ran with adblock and battery saver enabled (those are apparently not default settings), versus Edge without adblock, on sites which supposedly are ad-heavy - this will bias the results towards browser with adblock enabled.

Exactly, it's like saying your bus with no passengers gets better gas mileage than your competitors bus with 60 passengers.

yeah it's like comparing MPG of similar cars but Opera test is done in the middle of the night and competitors at rush hour

Your analogy collapses then though because you need people to be able to choose whether to travel at day/night at will for the comparison. Perhaps a better (but still flawed) way would be to say if the Opera browser included a free pass for toll roads and Edge didn't - so with the standard Opera offering you'd be getting a faster journey.

The point to me is that the situation is more nuanced than Microsoft apparently chose to present it.

Well, when you are the person performing the test you can somewhat control the conditions. ;) Opera just chose to drive in the night but sent Edge through rush hour.

It's not on by default though.

Their power saver isn't on by default either

They could have turned on an ad blocker for Edge as well.

Limiting the test to features built into the browser is pointless and doesn't represent real usage. People who turn on the Opera ad-blocker will install an ad-blocker in Chrome, Edge, or Firefox.

Perhaps the fairest approach would be to compare results across the different browsers with the presence or absence of various adblockers:

* Edge (no adblocker) * Opera (default adblocker) * Opera (adblocker disabled) * Chrome (no adblocker) * Chrome (uBlock Origin)

Otherwise you're just lying through omission to make your product seem better.

I wish they had done this instead of just said "look ours is faster when we enable adblocker on one browser but don't use it on another."

So do the WebKit/safari teams' respective efforts to reduce power consumption not count? It seems odd that they didn't compare against safari given Apple makes a dedicated effort to reduce power consumption.

That it's a "mode" which defaults to off says a lot - the things it does aren't acceptable all of the time apparently, so you choose between features/performance, and battery life.

Well you can't really compare Safari with Edge as they are are exclusive to different platforms.

It wouldn't be too hard to dual-boot Win10 on Apple hardware and compare them, but it might be a little awkward for Microsoft to do so officially.

The OS is a pretty obvious confounding factor, don't you think?

I think it would still be interesting. Win10 + Edge vs macOS + Safari.

In the end the user does not care which OS runs in the background, just which platform offers the best experience - in this case the longest battery time.

Depends really. I prefer using ubuntu but win 10 + edge gives nearly double the battery life, so when I'm out and about I have to use win 10.

There are plenty of suitable comparisons from which one can make an educated guess. In general, Macs do better than comparable Windows machines in browsing endurance tests, and Safari clobbers every other browser it has been tested in for battery life (I have _not_ seen a comparison of Safari to this new version of Opera, but judging by its performance relative to Chrome it could be competitive with Safari).

Safari, of course, has lots of support for ad blocking short of Apple actually doing the ad blocking itself. I suspect Safari + ad blockers would be the best option across the board (it absolutely rocks on iOS devices).

It is, but with the Boot Camp drivers, it seems like a fair fight. Obviously both OSes do some things behind the scenes that impact battery life, but we hope both are tuned to conserve battery when not charging. Though to turn the argument around, given that ChromeOS isn't known for wasting a lot of battery while browsing with Chrome, having less OS is probably a boon.

That wouldn't work. You don't get the same battery life out of a mac on windows as you do on macOS just sitting idle. I don't know how you would control for that.

But they could do a separate test of Opera, Safari, and Chrome on the Mac.

Opera was specifically aiming at Edge in this, it seems.

Opera was responding to something published a few days ago by Microsoft's marketing/Edge team. That's why it's directed at Edge.

Ok so for those who didn't quite get it - my point was that Opera are claiming to be the first 'battery saving' browser, and then completely ignore Safari in their tests.

Obviously this is anecdotal but I've been messing around with my laptop a lot recently trying to get to a full day of work on battery, and the best setup I've found is windows 10 Insider with Edge + adblock.

Opera battery saver is great but they should enable it automatically whenever you're on battery, as it has no performance impact that I've noticed.

Edge allows extensions now or are you using an OS-level adblock?

Edge allows extensions now.

https://developer.microsoft.com/en-us/microsoft-edge/extensi...

I like how RES is one of the first options. I remember when the Windows 10 team did an AMA and a lot of the redditors said they'd try edge as soon as RES was available. Apparently that's now.

It's great to see battery efficiency becoming something browser vendors fight over (even if it does come with a side of the usual benchmark gaming). Hopefully it shames Google into improving Chrome's battery performance.

Definitely not a fair test... Edge was rendering a lot more without an ad-blocker.

especially if there were animations going on in those ads. Most battery savings come during (even super short) periods of idle time, when various hardware components can enter lower power states

You might argue that not rendering ads is a valid method to save battery.

Correct, it is a valid method to save a battery. It makes the comparisons invalid though, since you are not testing with the same input.

You are testing more or less the same input just producing different output. This was a common strategy where graphics cards for example would degrade image quality for speed when a driver noticed you where running a benchmark.

Which is one of the reasons benchmarks are so difficult.

PS: A more insidious problem is when you specifically optimize for something because it's a benchmark.

Not neccessarily same input - all depends how adblock works. I assumed that with adblock, my browser will not even send a request to fetch the ads, so I save on network traffic, and the browser will not have to render whatever would have been fetched - this to me is a different input (to the browser's rendering engine) than you would have without adblock.

With all this said, "proper-er" benchmark would be to show each of the browser's power consumption with or without adblock, with or without battery saver in Opera's case, all this done on a set of websites which are ad-heavy and ad-light. :)

Server sends an HTML and browser fetches what it needs. Not fetching content is simply an advanced form of caching.

More importantly even if you grab the content it's rendering that takes power downloading files is very low energy. Even just disabling auto play sound / video is going to save power.

PS: Assumptions about how browsers interact with websites is basically a failure to understand what's going on IMO. Telnet to port 80, send fetch request on a web-server and you will get HTML unless they require HTTPS.

> Server sends an HTML and browser fetches what it needs. Not fetching content is simply an advanced form of caching.

I didn't understand this part. Caching is basically "not fetching content that you already have", and I don't see how it is relevant here. I assume that when using adblocks the browser will not even issue a request to fetch the data (be it HTML, JavaScript, flash videos, images, etc.) that is ad-related, so it is not the same as caching. And even if it did choose to fetch the data but then skip rendering, it boils down to the same thing: browser engine has different (presumably smaller) input in case when you use adblock.

By telnetting I will get a response, but this is only a first step. HTML will have references to huge amount of other content that also needs to be fetched (unless everything is embedded in the HTML - possible, but unlikely for any bigger website). Open developer tools in any browser and go to network tab and see for yourself how the amount of fetched data (and number of requests / number of fetched files) differs when you use adblock and not. This is the difference in the input for the browser. Opera mentions mlive.com - with uBlock, FF fetches 2.5MB of data (caching disabled), without uBlock I get 5MB (caching disabled). We can even take the example to the extreme - let's go to the website that has only ads. Of course Opera will be better there - it will have to render nothing, everything will be blocked by adblock.

You can also cache intermediate results from calculations, the idea is don't fetch things you don't need. And as you know the result from fetching an add is going to be blank you don't need to fetch it. Much like a screen reader not downloading a JPEG.

As to downloading you can try it you self your talking fractions of a watt savings from not downloading a few Meg's.

In terms of other downloads from a page, most of that is things like tiny images which you can mostly skip. The Amazon homepage sends lots of junk but all you actually care about is the search bar and links to other areas.

But how is it all relevant to the discussion? Bottom line is that using adblock in one browser and not using it in other browser makes the comparisons meaningless...

People are objecting to add blocking as a perfectly normal browser feature. There is a mindset that as soon as people start wearing augmented reality glasses they are going to want context sensitive advertising in their day to day lives. IMO, most people would much rather block real world adds and overlay them with blank space or even nature scenes.

Browser advertising fits into the same argument as adds on cable television. Subscribe to Hulu and no the adds don't go away.

Let me reiterate as you did not address anything from my comment: But how is it all relevant to the discussion? Bottom line is that using adblock in one browser and not using it in other browser makes the comparisons meaningless...

Adblock is a browser feature just as much as GPU acceleration.

Suggesting you need to add third party GPU acceleration to make things fair is ridiculous. But, you seem to think Adblock is somehow different.

No, I can make things fair by doing another test. Let's test Opera with adblock and Edge without, but only use websites that do not have ads. Sounds good?

Problem with Opera's comparison is that it is not fair because they don't display the same content - they (probably) need to render less. Of course they will be better if they render less (unless they are really bad at what they do). So to make the comparison meaningful, make them display exacly the same thing - then we can talk about who is more power-efficient.

Of course it is. But if you care a lot about battery life, you might wonder if you should use Edge with an ad blocker or Opera, and the provided benchmarks are not very useful for determining which of those options is superior.

and that is true - but enabling the (default disabled) ad-blocking in Opera to test comparable metrics means that Opera, in my eyes, just lost any credibility they had.

Given that even with the ad-blocker it barely beat Edge I'm almost certain they tried it initially and found the result was true so enabled this feature so they could write this fluff piece.

Edge is free to implement an ad-blocker, too.

It's supported if you're running a Windows Insider build. Since the Opera test was using a dev channel build, I don't think it's a stretch to say they could have used the insider build of Edge with Adblock or Adblock Plus (uBlock isn't ported yet, but it shouldn't be too hard).

https://developer.microsoft.com/en-us/microsoft-edge/extensi...

Opera being better than Edge is a claim I might believe but I don't think Edge is almost as bad as Chrome.

I stopped using Chrome when I discovered it took hours off my battery life on a MacBook. Edge can't be almost as bad. Intuitively I would believe the ranking in terms of battery life would be Safari > Edge or Opera > Firefox > Chrome.

opera likes to boldly claim it was the first browser to do power management on pages but truth is Microsoft has done this for many years prior and has the operating system integration as an advantage.

http://www.sevenforums.com/tutorials/159269-internet-explore...

Now that Opera Software is owned by a Chinese company I've avoided their browser anyways, regardless of potential battery/performance benefits.

Isn't that just xenophobia?

No, I don't think so. Maybe I'm a bit paranoid but I tend to avoid software from Chinese companies because of all recent stories of a apps stealing/sending data that they don't necessarily need to function. But I suppose software from any country could do the same thing though...

And what is your position about hardware?

Most hardware we use here, though manufactured in China, is still wholly owned by American companies though.

I'm personally just as wary of Edge...

Nope: it is Statephobia and States have no right to honor, just people eith individual bodies.

That seems like splitting hairs to me. Would you want people to reject your software just because of your nationality or that of your company? Probably not. So it seems to me that an implication of the golden rule is that we shouldn't do that.

I would understand if people from another country did not use my software for fear of an NSA backdoor. It'd suck, but I would not think twice about their decision.

Software written by Chinese people -- no problem.

Software written by persons and companies subject to the will of the Chinese government -- huge problem.

It's really no different than European countries not wanting to do business with US companies because of the NSA.

On my nationality? No.

On the jurisdiction in which I, or the company employing me, operates? That can well be legitimate. You're discriminating against the laws (or unwritten government policies) in that jurisdiction, not against people.

The fact that I do not like it does not change my opinion.

Only in the same vein as the more extreme example of avoiding flight itineraries with a layover in North Korea is.

There are two sides to any story and countries have different versions of reality that they believe in and promote. Nonetheless it's foolhardy to not notice which countries take an overly keen interest in people's communications and the "correctness" of their views... and jail those saying things they don't like.

If you're completely unfamiliar with the history of recent Chinese-western software collaborations, try googling "TOM-Skype"

Saying something like that is only bad and wrong if it's about Israel.

I think it's just that the investment firm who bought Opera also owns Qihoo 360, a notoriously sketchy “antivirus” (and I use the term loosely) company. There's probably nothing to worry about, given that Qihoo doesn't actually own Opera.

That said, I think a general lack of trust toward the PRC is understandable.

Only if you're Norwegian.

I recently switched from Opera to Vivaldi and have been pretty happy with it.

> Reducing frame rate to 30 frames per second

Many users may not like this.

Only enabled in power saver mode

probably depends what you're doing, right? e.g., if i'm just sitting here reading hacker news i can't tell the difference between 60 fps or 30 fps repainting, right?

Same, I stopped reading after that. I got it, they got more battery life by reducing performance, not something I'm interested in.

What really surprised me were the run times of that laptop. Man that is crappy battery life for both instances. My thinkpad T220 with extended battery and 13" Macbook pro both decimate that battery life.

They had the screen brightness at 100%. I imagine that eats through a lot of battery...

The Lenovo Yoga (about 550 €) is hardly on the same price level as the ThinkPad or Macbook Pro (1500 €) ... And you even mentioned the extended battery.

I have found that safari gives best battery time on OSX (like 9 hours of use vs 6-7 when on firefox).

But i generally dont use safari as battery time is not the most important factor in my common use cases.

>But i generally dont use safari as battery time is not the most important factor in my common use cases.

I use Safari for almost everything. It's by far the best browser for browsing and just reading & enjoying the web. Reader interface and the speed of it is still unbeatable.

I use Chrome for web development, however.

I agree Safari is the best. Every other browser is a clunker IMO. The Safari web inspector keeps getting better. You might want to take another look at it.

I agree here, its excellent for when I'm low on battery or just don't want to use the charger. It manages my tabs well, I was even watching a 5 hour movie on Youtube once and my battery was at 80 percent after I was done

Safari has the same problem that Edge/IE has - it's only on one OS.

If it's on the OS that you use, that's hardly a problem.

I wonder how much battery would be saved if not every site you visited was reported to bing api or opera sitecheck. <g> But most power is is probably used to light the screen, so an extension to make every page use a black theme would probably beat most optimizations.

I don't think it would take much of a difference on an LCD display because your backlight is going to be on regardless.

Opera need to market themselves as the browser that extends your battery life so you can surf without the range anxiety. Testimonials are better than tests plus word of mouth.

This is the first time I've heard the term "range anxiety" applied to laptops, as opposed to electric vehicles.

Where's firefox in the perf?

In the battery-benchmarks that I've seen so far, Firefox was always the leading browser, so granted those benchmarks did not test Opera's battery saving mode, Firefox should still be at least in second place.

And Firefox does also have a native ad-blocker, so if you enable that, too, I could imagine that it maybe even still outperforms Opera's battery saving mode, but yeah, a proper benchmark would be good...

> Firefox does also have a native ad-blocker

I don't think it has one natively. Which one do you mean? Or are you referring to tracking protection (private mode only, but available in Fx 49 also in "normal" mode iirc)

Yeah, I'm referring to Tracking Protection. It's not supposed to be an ad-blocker, but with almost all ads using tracking code, you rarely encounter ads when it's enabled. It's probably even better for battery life than just an ad-blocker, as a lot of webpages also execute tracking code which isn't bound to ads.

And you could always enable it in normal browsing by setting "privacy.trackingprotection.enabled" in about:config to true...

On a side note: I've been testing edge on a Dell 1012 netbook (atom, 2gb) and it beats Chrome for rendering YouTube videos (480p max!) which I found surprising (video stutters on Chrome). However on my main laptop when running scratch (ok flash based), cloud based games are better sync'd when using Chrome (it's a slither.io clone). Overall battery life is exceptable for both browsers.

After I read this I've installed Opera mobile and it's now my default browser. The ad blocker runs pretty well, along with a customized hosts file mapping advertisers' domains to localhost. Too bad DuckDuckGo is only available as a search engine in Opera Mini beta. Looking forward to get rid of Google search.

Coming from the embedded world, I have always concern about CPU loading which translates to how much power you are sucking.

I can't not understand why minimal CPU load is not on top of design requirement of many software products, totally ignored if on a desktop system.

Because PC Master Race. We only want 300FPS+ and 4k graphics, we don't care about $5 extra to our electricity bill, so why would software developers even bother with this and spend extra time and money for improving power drain?

Thanks for the laugh!

Seriously, imagine the world collective savings on energy if everyone reduces their power consumption by small modest 5%.

We're even seeing websites now care about how they cause power consumption. Folks at PWAS did hallway talks about measuring it. Very cool.

Ad-blockers are taking over the internet. It seems that most people here believe that Ad-blockers must be enabled by default in browsers which is quite strange.

They should also release control videos which show both machines with battery saver on or both off.

Reply on news.ycombinator.com