Hacker News Reader

Comment by BackBlast | Hacker News Reader

How can you trust a list that places JavaScript an order of magnitude lower than TypeScript? Seems fundamentally broken.

View on HN

I didn't read the original paper[0] but it is based on the Debian Benchmark Game[1] as it was in 2017 and the game relies on user submitted code for each language separately, so most likely when the researchers checked the results, the TypeScript tests didn't have as optimized tests as the JavaScript tests - which makes sense as the latter is way more popular than the former, so there are less people to bother with it (and in fact the benchmarks nowadays do not include TypeScript at all).

You can confirm it via archive.org too: the TypeScript[2] page shows both less implementations and the implementations that are shown are often slower than those in the JavaScript[3] page.

As for if that makes sense, well, IMO using the benchmark game for judging how good a language is at optimizations is flawed in the first place as not only there is a bias towards the more popular languages but also a lot of the top entries use approaches that in practice you wouldn't find in real projects.

[0] https://greenlab.di.uminho.pt/wp-content/uploads/2017/10/sle...

[1] https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

[2] http://web.archive.org/web/20170425064751/https://benchmarks...

[3] http://web.archive.org/web/20170425114504/https://benchmarks...

I fixed the Typescript implementation 3 years ago when this paper was debunked back then.

Didn't submit it though because they make it difficult.

https://news.ycombinator.com/item?id=24826453

(the whole thread is worth a read)

And-yet your program was accepted and included, and then with a later TypeScript update it stopped working.

    spectralnorm.typescript-6.ts(115,29): error TS2794: Expected 1 arguments, but got 0. Did you forget to include 'void' in your type argument to 'Promise'?

Something like that is probably what stopped the authors of "Energy Efficiency across Programming Languages".

https://sites.google.com/view/energy-efficiency-languages

Yeah Typescript doesn't guarantee perfect backwards compatibility. You normally have the compiler as part of your dependencies and pin it. I guess they just didn't know.

I don't know what you mean with your last sentence. That's the same paper...

I appreciate the thoughtful reply. Every time I see this list trotted out I can't get past the obvious red flag. Nice to have some more background on it.

    --alwaysStrict

So when the JavaScript doesn't type check, the authors measured a different program that does type check.

Even so, that only messes up the results because the mean is used rather than the median, and the data tables published with that 2017 paper, show a 15x difference between the measured times for a single outlier the selected JS and TS fannkuch-redux programs.

That single outlier is enough to distort TS and JS "mean" Time difference.

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

A single outlier is enough to distort the TS and JS "mean" Time difference.

https://news.ycombinator.com/item?id=36524073

Check the pages i linked at, or even better the one that compares the two[0] around 2017. While there is a single case with wildly different results, there is more than one with results that also have a large difference in JavaScript's favor.

[0] http://web.archive.org/web/20170715120019/http://benchmarksg...

Please read the original paper and base your comments on the repo that the authors provided.

Like you, most readers haven't seen the paper.

Like you, most readers have only seen "Table 4. Normalized global results for Energy, Time, and Memory" taken out-of-context (often without any way to find the original source).

Most readers notice the too-large differences C/C++ and JS/TS and start speculating about how those differences were caused (because that's fun).

I went back to the time measurements the authors provided and calculated the Mean, Geometric Mean and Median. Simply using a more appropriate summary statistic would have presented average values which readers would have found acceptable:

    JS 7.25 times slower than C
    TS 7.8 times slower than C

:even though they were based on different programs and included an outlier. (Similar story with C/C++.)

> How can you trust a list that places JavaScript an order of magnitude lower than TypeScript?

Simple: start with the same data, make the same calculations, see the same results.

When we make assumptions about how measurements were made and analyzed, our assumptions may be wrong.

> How can you trust a list that places JavaScript an order of magnitude lower than TypeScript? Seems fundamentally broken.

The list places JavaScript at a score of 4.45 and TypeScript worse at 21.5. For reference, C, the most energy efficient it 1.0

> The list places JavaScript at a score of 4.45 and TypeScript worse at 21.5. For reference, C, the most energy efficient it 1.0

The point is that TypeScript is JavaScript with typing syntax added on top. There is a transpile step into JS. That's how TypeScript works. The runtimes are exactly the same. Unless we're also measuring the build step? Which seems silly.

The difference should be near zero. And it's not. They clearly do not understand exactly what they are measuring.

One explanation is that the typical JS output by the TS compiler is more verbose (so more code to load, parse, and run). On the other hand, one can imagine that the TS compiler’s catching classes of errors at compile time means fewer runtime exceptions.

I think the takeaway, however is the theme of a spectrum of energy efficiency, with compiled languages being more efficient than those that are not.

But I still maintain that the overall efficiency of a system is more a function of other factors.

My take away is that this list is very unreliable.

I don't know what the usual speed difference of TypeScript programs vs JavaScript programs.

But the argument that TypeScript generates a JavaScript, so it must have similar speed doesn't hold in general.

If the compiler in question is a whole-program compiler, it can make optimizations that a normal person couldn't do.

As an anecdote in [1] a raytracing program were implemented in both Scheme and C. The Stalin compiler (a whole-program Scheme compiler) produced an executable 45% faster than the one produced by g++. The Stalin compiler produced the excecuable by compiling Scheme to C, and then used a C compiler to produce the final executable.

The price of a whole-program compiler? Well, the compile time are huge.

[1] Scroll to Siskind's comment. https://groups.google.com/g/comp.lang.scheme/c/NJxRsdMNKz4

For those curious about the actual programs and results: https://web.archive.org/web/20071011073406/http://www.ffcons...

> with typing syntax

Programs that didn't type check were excluded.

https://github.com/greensoftwarelab/Energy-Languages/issues/...

The authors chose to summarize their data using the mean, and that highlights the variance.

Using the Geometric Mean or the Median with the time measurements from that table would have highlighted the middle value, like this:

    JS 7.25
    TS 7.80

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Reply on news.ycombinator.com