Hacker News Reader

Comment by titzer | Hacker News Reader

Speaking from the perspective of a (former) V8 engineer, SunSpider is crappy benchmark, and I can't believe people still track it. It has misdirected JavaScript performance engineering work for going on 15 years now, leading to all kinds of weird contortions in JavaScript engines that do not benefit real world sites.

Years ago, V8 switched away from microbenchmarks for the most part. See:

https://benediktmeurer.de/2016/12/16/the-truth-about-traditi...

edit to add: Don't take the above to mean that Firefox's performance improvements aren't significant, and welcome. It's just that SunSpider is the absolute worst benchmark to look at.

View on HN

To be clear: we track it in the sense that we still have it running as part of old dashboards. It's cheap, and every so often it catches an unintended regression.

The SpiderMonkey team is very much on board with ignoring microbenchmarks. We care about Speedometer, because it correlates reasonably well with real-world performance. We pick and choose which parts of Jetstream2 we bother with. Anything older than that is strictly a measuring stick, rather than a target: it's nice when the numbers go up, but it will only happen as a side-effect of work we're doing for other reasons.

Notably, all the SunSpider wins on this graph are side-effects of Speedometer work we've been doing this year.

I understand the logic, but I never understood why we (and I mean, collectively, the JS VM engineers across the browser vendors) didn't just rewrite the benchmark (or fix it) and name it SunSpider 2. I suggested this many times. It's not that much code.

Presumably because replicating the mix of operations used in real react/jquery code all over the web accurately isn't trivial...

Yeah, I've been making this argument for years too.

Speedometer 3 is in the works, and it's moving in this direction. On multiple occasions we've done performance analysis of the latest draft version, noticed a particular chunk of code is hot, considered what it would take to optimize it, and then changed it because we don't think it reflects real-world usage.

EDIT: And I guess the answer for "why not fix SunSpider specifically?" is that once you've done the work to turn it into something worth optimizing, it no longer bears any resemblance to the original. It probably looks a lot more like Speedometer, honestly.

> EDIT: And I guess the answer for "why not fix SunSpider specifically?" is that once you've done the work to turn it into something worth optimizing, it no longer bears any resemblance to the original. It probably looks a lot more like Speedometer, honestly.

I get that, yeah, but superseding the old name is probably worth it though; the internet is full of people who never get the memo except via version numbers :)

I tongue-in-cheek proposed that we match the Sunspider ASTs in V8 and either make it infinitely fast or infinitely slow. Now that will kill a benchmark once and for all!

JSOP_SUNSPIDER was apparently a running joke back in the day inside SM.

The link you posted promotes Speedometer, where Firefox reportedly already surpassed Chrome.

https://news.ycombinator.com/item?id=36770883

Firefox tracks their performance across many different benchmarks. Here is the performance vs Chrome over the past year on many different ones. It is encouraging to see steady improvements being made on a substantial number of them over time.

https://arewefastyet.com/win10/benchmarks/overview?numDays=3...

It appears that Chrome still wins most of these. Even the touted Speedometer win has apparently been trumped by Chrome over the last couple of weeks. The competition doesn't stand still. Which is good, imagine if Firefox threw in the towel, Chrome would become IE 6 in short order.

Safari is faster than either across almost all dimensions, lighter on battery, nicer UI, better standards support generally than FF and even matches chrome for what I care about. It’s the clear best browser imo.

On Mac? Probably because only Safari has access to the private api's

Just compare the results from Windows and macOS.

Lol... Doesn't make any sense

No safari on Windows. And I don't have a Mac.

It sure does. Did you even open OPs link? There you can compare everything.

And no need for Safari on Windows since you can compare Firefox on Windows (where those 'private api's' are not protected by Apple) with Safari on macOS

Generally, no.

Safari is only well maintained on MacOS and iOS. That makes it a non-starter for most people.

More than half of US users run iOS or MacOS on one or more of their devices.

Less than 5% of people live in the US. Not sure how your statement is relevant to my statement about most people in general.

Which is irrelevant if 95+% of your users are coming from the US…

Unless you’re equally interested in non English speaking users then the global average is absolutely meaningless.

What matters to organizations is some subset of everyone and what their specific access looks like.

We're talking about web browsers. Pretty much everyone with a personal computer(including smartphones) needs one. The US makes no sense as a subset here. If we were discussing what markets for a company to target, then sure. But we're not, are we?

I and many people here abandoned Chrome, so it’s only relevant to me from a software development perspective.

In terms of software development a US specific audience isn’t uncommon. Local utilities and many government agencies etc just don’t care about foreign users.

> Unless you’re equally interested in non English speaking users then the global average is absolutely meaningless.

The US is not even the largest English speaking population, it’s India. Additionally, if a subset is to be chosen, why is this subset from the US? Why does it get to be the centre of the universe, especially for a general-use product like a web browser?

The point was it’s rare to care about global numbers, not that the US is the only meaningful subset.

As to why a US only subset may be reasonable vs a language specific subset, some US government agencies care about a global audience but many are US specific. They may have multiple languages because Americans aren’t all fluent in English.

The US is hardly representative. Globally, 25.3% of website visits happen from a device running iOS or MacOS. That's impressive, but still a mere quarter. Windows (29%) and Android (40%) each account for more users.

https://gs.statcounter.com/os-market-share

Which isn’t actually relevant to any specific website as they all favor either some subset of users and generally a subset of their devices. Ie: Fewer than normal stack overflow users are coming in on cellphones let alone PlayStations.

You need to know your target audience, not simply lookup overall statistics. I’ve seen Windows make up less than 4% of all users and I’ve seen it be over 90%.

Statscounter is a popular statistics for platforma usage, but I strongly doubt that it is heavily biased. I've never found that any websites use statcounter (though I haven't aggressively investigated), and never found installation guide in Japanese. Perhaps is userbase biased to smaller websites on some countries?

I can run Chrome and Firefox on my Debian laptop and on my Android devices. How do I benchmark Safari there? I can't, so Safari is barely relevant to me. Nice to know that it's faster than the other two browsers on Macs, but I'm not very surprised. Apple should be able to integrate it with their hardware and OS more than Google and Mozilla.

Customization and freedom means Firefox is always the best for me.

How's the safari world of add-on's? Do you have tab containers? What about mobile? Can you block ads?

See also, significant regressions in Firefox's memory consumption: https://arewefastyet.com/mac/memory/overview?numDays=365

In general I’ve preferred Firefox’s lower memory usage than pure speed.

This isn’t an agreement or disagreement:

Having benchmarks based on real-world sites seems like both a good and a bad idea.

Good for obvious reasons. Bad because it creates a feedback loop. People develop based on performance. Performance is improved based on what people develop.

Did you/your colleagues tackle this in some way?

Here's one example from 2016: https://v8.dev/blog/real-world-performance

In fairness, when sun spider was first released it did accurately reflect the core pain points in JS engines, as present on the web. That applied to V8 as well (though V8 introduced its own benchmark suite as well on release, it was overly focused on property access, and insufficiently on the parts of initial page load that sun spider just happened to cover).

Of course all of that was _15 years ago_, no benchmark from back then is really reflective of anything useful, and they are largely no longer valuable for anything other than catching regressions.

I agree, but speaking as someone who's been using JS for computationally medium-intensive tasks: Firefox and Chrome don't differ that much. For most users, the difference is negligible.

As the saying goes any compiler engineer can make their compiler the fastest if they know what the code is.

Reply on news.ycombinator.com