Consumer Reports Retesting MacBook Pro Battery, Apple Says Safari Bug to Blame

Thats correct-the funny bit is they were using a hidden developer feature-not one of the 'Developer' drop down features but an actual hidden flag. They also observed huge internal discrepancies with their testing methodology-but instead of reporting that to Apple or trying to figure out why their test was broken CR reported their results as 'facts'. Thats why people and Apple are irritated about this.

This is a good point, which seems to be ignored by other commenters nearby.

Additionally, this seems to be a case where CR (a venerable publication with many good qualities) used results in a home-brew single-point test to characterize a system's quality.

This is not an isolated problem with this particular review. There's a lot of snake oil in audio systems, but CR reviews of audio equipment were a joke for many years because of over-reliance on specific lab measurements. CR has had issues over the years with mistaken testing of child safety seats, pet food, and other products over the years (see their wiki page).

This caveat is apparent after subscribing for a few years.

Websites change over time -- they'd have to download a static copy of the websites and store them locally for repeatable tests (but then it no longer reflects the real world, since websites often tailor themselves for each user).

And since browsing activity changes over time too, they'd have to regularly update the test, which means that tests are not comparable over time.

I put my MacBook Pros through a grueling battery test (basically build and destroy a VM successively, sustaining 100%+ CPU, tons of disk I/O, and maxing out WiFi bandwidth intermittently) for my blog post elsewhere on HN.

Worst-case, the 13" Pro with Touch Bar gets 3.5 hours. The Function Key model gets 4 hours. And the 2015 Pro gets 5 hours.

In light usage (e.g. while writing said blog post, and watching an occasional HTML5 YouTube video, and listening to some music in iTunes through AirPods), I can get 8-10 hours without issue.

1-2 hours sounds like you may have an app that's more buggy than the worst that I'd allow to run on my laptop...

Edit: Link to said test script: https://github.com/geerlingguy/macbook-pro-battery-test

Until the most recent batch, Apple laptops have just had straight up the largest batteries of everything except Thinkpads with batteries that stick out. E.g. the much touted "bigger battery" of the new HP 15" x360 only brings it even with the new Touchbar MBP, which has a 25% smaller battery than the previous MBP. Samsung's current 15" laptop has a battery that's about half the size of what is in the new MBP.

Non-Apple apps don't kill battery life. What kills battery life is shitty apps like Chrome (or anything based on Electron). VSCode for example will cut battery life to 6 hours versus 10+ for Emacs. Just simple stuff like typing and scrolling takes huge amounts of CPU. Even just sitting idle it's using 1% or so CPU.

>1.5 hours is probably closer to reality

Ughh.. no. I'm on an 6 year old Macbook with SSD which had fresh battery put in a year ago. Running IntelliJ, web server, JS build tools (they run in the background), and over a dozen apps (i.e. slack, browser etc.) lasts me about 3 hours.

> I think I'd be lucky to get two hours of productive development time on my laptops.

Is this speculation or have you actually measured it?

This isn't a Mac specific issue. Most of the efficiencies in battery life That have taken place in the last decade are all about CPUs and their components lowering their idle power consumption.

I have a circa 2014 MacBook Pro that I use as my daily driver. The battery life is great -- so long as I don't light up VMs, run OneDrive for Business or a few other misbehaving apps.

I'd challenge you to identify another laptop that's better without making other sacrifices. My old ThinkPad T series had great battery life, but you had to deal with the bulk and weight of a huge battery.

Can you recommend a company that has good battery life for high intensity workloads? I'm genuinely curious.

As usual, another comically hyperbolic diatribe against Apple possessing little to no basis in actual fact.

Read Apples statement to TechCrunch in the article.

I think they removed it because the battery life of the new MBPs has been getting a lot of negative attention, deserved or not, and in general people don't seem to be capable of understanding why the estimate would change for different workloads. Maybe the best PR move was just to remove it altogether. I wish they hadn't.

It's not the same machine, but I routinely get > 10 hours on my 2016 12" Macbook with a workflow that includes moderate to heavy browsing in Opera, coding in Tmux/Vim, and various compilers and interpreters invoked periodically. Have you had a look in Activity Monitor to see where all your power is going?

I'm using two non-Apple apps as the core of my workflow (Opera and Alacritty for terminal: https://github.com/jwilm/alacritty) but my battery life is still excellent. However, I chose them carefully on the basis of their efficiency. Things written in Javascript seem to be real resource hogs (Atom comes to mind), and Chrome is notorious for inefficiency in memory, at least.

Click on the battery icon in your menu bar, and see if there are any programs using "Significant Energy". Some programs will switch on the discrete graphics card without needing it, for example.

I'd suggest switching to Safari. I easily doubled my battery life while on El Capitan doing that.

Sierra has wrecked my battery life tho...

Chrome is pretty bad for battery life.

I've used them for baby equipment, for mattresses, and for cars. They were spot on with cars, baby car seats and cribs, but their mattress ratings could have been better at the time. Overall, I've found them to be pretty high quality, and most importantly, independent. Their reviews will often differ from more public sources. And I could be mistaken, but I think they don't even accept free products from companies. They buy the products they test.

Generally I've found them to be pretty good with the reviews. The testing is thorough and they explain what to look for when buying something (I've used it for appliances).

Its subscription only (they don't do ads), which means they're not beholden to the companies they're reviewing for money (a good thing)

They generally make a grid of the items you're looking at with dots indicating the ratings. Its very easy to nagivate and intuitive.

You'll have to pay for it (or find someone to give you their old copies, or many libraries have subscriptions).

Consumer Reports is pretty good for household products that they have expertise in. For example, I'd trust their dishwasher or toaster recommendation. However, when they test something that's outside of their sphere of expertise, they can miss the mark by failing to focus on criteria that are known to be important by real experts. For example, from time to time they'll review bicycles. Anyone who really knows about bikes will be amused/exasperated by the aspects of the bikes being tested that they make their judgement on. They've been testing computers for long enough that I'm sure they're not completely clueless, but I think their judgement criteria are based more on a casual computer-as-appliance home user, and a power user might feel that they focus on superficial things.

In general, they do well, but there are a few categories which they don't seem to "get". They do a good job of trying to remove a lot of barriers to objectivity, even when useful reviews require subjective knowledge. They also try hard to find equivalent, reproducible tests so that they can use them on multiple products, even if they're not exact equivalents. This means, for instance, that they might end up testing ice-braking tires by measuring the stop distance in an indoor ice rink where they can control the conditions... that makes it easy to measure the tires against other tires, but it's not real-world. They also purchase their products, rather than accepting them from the companies, which helps increase the odds that they're getting the same things off the shelf you do. Again, they seem to be clueless about a few categories, though, such as certain electronics. Just look at the ratings for the Google Home and Amazon Alexa--very poor--because they get lumped in with wireless speakers where sound quality is the utmost importance.

I found them useless for high-end dishwashers (both Mieles they recommended worked poorly), audio equipment, and high-end vacuums. I specify high-end because I am very willing to pay top dollar for appliances that will last many years, not because I'm a snob.

Not really... most people I know use things like the Wirecutter instead.

This became Hacker News fare because lots of people were unhappy with the new MacBook Pros for other reasons.

My mother and sister are very huge fans of Consumer Reports.

I can see where you're coming from but I disagree. In that case you are testing the battery hardware, while the software actually makes a big difference.

Say you have two phones: one with crap software, and one from a company that spends time on writing proper, efficient software. If you run a stress benchmark, they will both be running at 100% CPU all the time and you'll only be measuring how much mAh is in the battery divided by how much power the CPU draws. All of this is information is already available in the product specification.

The only reason my Galaxy Note 2's battery life is any good (still the original battery which is, what, 5 years old now?) is because I did software tweaks to keep it from holding wake locks when apps are inactive.

This is Apple's battery life test:

>The wireless web test measures battery life by wirelessly browsing 25 popular websites with display brightness set to 12 clicks from bottom or 75 percent. The iTunes movie playback test measures battery life by playing back HD 1080p content with display brightness set to 12 clicks from bottom or 75 percent. The standby test measures battery life by allowing a system, connected to a wireless network and signed in to an iCloud account, to enter standby mode with Safari and Mail applications launched and all system settings left at default. Battery life varies by use and configuration.

http://www.apple.com/macbook-pro/specs/

> for example - who do you know that only browses 25 sites in a single charge?

You're right, most people probably browse fewer than 25 sites.

So the test if flawed. You can't simply disable cache and call it a good test.

Do they do this on other browsers? Browse the same site over and over while disabling cache?

Why did this not cause a problem when they ran Chrome (did they also disable cache there)?

The CR tests raise more concerns about how CR runs their tests (and specifically how they run this test) as opposed to the MacbookPro.

>> Consumer Reports had Safari Developer Mode enabled

I always have it enabled and get 8+ hours from my previous model Pro.

If a laptop battery capacity is cut in half because you turn off caching in a browser that is important information to know, as it suggests the battery times are the result of carefully sterilized conditions that don't withstand the test of real world use. They didn't disable power management or something, they turned off caching in a single app. Come on.

Incorrect - the performance hit was from a bug related to that:

"the variable battery performance we experienced is a result of a software bug in its Safari web browser that was triggered by our test conditions."

Changing a testing approach in my opinion is fine, but should result in the retroactive re-testing of previous models used in any in-article benchmark comparisons.

Why?