Which Programming Languages Use the Least Electricity? (2018)
thenewstack.io
thenewstack.io
For example, Javascript and Typescript. Theoretically, I would assume those to be very close, since one compiles to the other. And for memory usage, they are very close. But for running time, Typescript is an order of magnitude slower. That looks suspiciously like entirely different algorithms were used in each implementation (with one being obviously superior in terms of performance), and if that's happening in this specific case, where else is it also causing problems in the analysis?
This also isn't a criticism of the benchmark game, it's well known that not every implementation is equivalent in the time and effort put into optimizing it. It serves its purpose about as well as can be expected. Unfortunately, using it as the base of further calculations can lead to some of the known quirks of the benchmarks being exaggerated into results that are not always obviously an artifact of the underlying system, as I suspect this is. Making that obvious by pointing it out can be useful.
That's actually one of the reasons why the TypeScript/Javascript divide jumped out at me. They were mentioning that the bottom of the list was dominated by interpreted languages, and mentioned TypeScript by name (which surprised my given the focus JavaScript VMs have gotten), and then when I reviewed JavaScript's standing (which was more in-line with what I expected), I noticed the difference between it and TypeScipt was very pronounced, which is odd when (to my knowledge) TypseScript compiles to JavaScript, and not because it's doing a lot of convenience stuff that would slow it down. That said, I don't use TypeScript, so maybe I'm overlooking something.
So as a regular writer of TypeScript I had the exact same question as you.
https://web.archive.org/web/20070503205039/http://shootout.a...
However, the original research has been posted multiple times to proggit and HN since 2017; and I don't recall whether or not anyone noticed this problem until now --
https://news.ycombinator.com/item?id=15249289
https://www.google.com/search?q=energy+efficiency+programmin...
It's a study presented at some conference so while not exactly the Higgs boson, they're showing other people data and the conclusions they derived from it. It's 100% their job to understand what their data measures and to notice that one of the measurements is completely bogus for their purposes. The fact other people hadn't necessarily noticed on messageboards before is mildly curious but it's not really their job.
On the contrary; we can see from archived web pages that other measurements showed the same relatively-poor performance, with those old versions of TypeScript.
Unsuitable because?
Let's say I want to measure the 'energy efficiency' of x86 assembly and JS. I'll use sorting an array of 1000 integers. In my JS implementation, I call Array.sort. In my x86 implementation, I randomly shuffle the array and check if it's sorted, if not repeat until it is. Does measuring the execution times of these tell me anything about the 'energy efficiency' of Javascript vs x86 assembly?
A few TypeScript versions later, that happens to be only 1.6 times slower.
There are good questions to ask about how to handle possible outliers in a study that takes a snapshot of a changing situation and then seeks to make more general claims.
You seem not to have considered the possibility that the authors may have simply made a mistake, unrelated to the origin of the programs.
The authors presented at an Oct 2017 conference. Archived benchmarks game web-pages from 2017 do not show the 10x fannkuch-redux differences that the authors report --
https://web.archive.org/web/20170918163900/http://benchmarks...
I think the Sep 1st 2017 benchmark does, though.[1] At that point it's 1,204.93 seconds, compared to the 131.39 seconds on September 18th you referenced. That makes sense, since the paper could have been finished quite a bit prior to the conference.
1: https://web.archive.org/web/20170901020804/http://benchmarks...
5 July, Node 8.1.3, TypeScript 2.4.1
https://web.archive.org/web/20170715120038/http://benchmarks...
1 Sep, Node 8.4.0, TypeScript 2.5.2
https://web.archive.org/web/20170922144419/http://benchmarks...
----
How should we now assess your "suspiciously like entirely different algorithms were used in each implementation" comment?
Also, typical TypeScript programs are faster than typical JavaScript programs, because JavaScript JITs like predictable object shapes and monomorphic functions for the same reasons that other language implementations require them. Libraries like lodash and bluebird take advantage of this fact without using TypeScript, but TypeScript steers you towards these patterns.
Person who works on js engines here :)
You would think this is the case, but in actuality, js and ts are about on par in performance (assuming similarly written code). This is in fact due to the fact engines optimize for idiomatic js patterns, not idiomatic typescript patterns. often these will align, but in some cases (usually revolving around generics and inheritance) well written js will actually fall through the optimization pipeline faster due to following more patterns that have specific optimization checks.
After having worked on JS for some large web apps that need good graphics performance, the two rules of thumb in my head for making JS fast are: 1- avoid dynamic memory allocation, and 2- avoid using the functional primitives like map.
The first one is more or less true in all languages, memory allocation always costs a lot, and if you can pre-allocate memory and/or re-use memory along the way, the code will run faster. This means paying attention to what things in JavaScript will allocate memory under the hood, use of dicts, use of 3rd party library and framework functions, etc.
The second one is a bummer; I love using the functional primitives. But map is slower than a for loop, all else being equal. It has gotten relatively faster over time. I mostly use functional everywhere, and only resort to for loops in performance critical code.
I understand that map requires creating a new array, and that is already included in point 1. What overhead are functional primitives subject to apart from memory allocation? e.g. forEach
BTW, it's easy to test the basic primitives. I use Chrome snippets.
test = (name, fn) => {
const timeLimitMs = 1000
let start = Date.now(), count = 0
while (Date.now() - start < timeLimitMs) { fn(); count++ }
console.log(name, count)
}
var N = 1000000
let a = new Array(N)
test('for loop', _=> { for (var i = 0; i < N; i++) a[i] = i })
test('map', _=> { a = a.map((x,i) => i) })
test('forEach', _=> { a.forEach((v,i,a) => a[i] = i) })
for loop 941
map 37
forEach 69
Like so?
const N = 1000000
let a = new Array(N), b = new Array(N)
for (let i = 0; i < N; i++) a[i] = i
test('copy loop', _=> { for (var i = 0; i < N; i++) b[i] = a[i] })
test('copy map', _=> { b = a.map((x,i) => a[i]) })
test('copy forEach', _=> { a.forEach((v,i,a) => b[i] = a[i]) })
Are you using a different browser? I know that sometimes other browsers have very different results.
In any case, it's somewhat irrelevant if there are cases that optimize and cases that don't. When idiomatic functional code is sometimes up to 30x slower than a for loop, it can't be used in perf critical sections. Even if it's only Chrome and only certain cases. The forEach perf needs to be always reliably performant before I can use it without worry.
Often times, these sorts of benchmarks are done with prejudice (not necessarily malice). The benchmarks are written by someone with something to prove: my chosen tech stack performs better, and let me show you why. A favorite of mine is Perl vs Python comparisons, where you see an idiomatic Perl implementation vs a non idiomatic Python implementation (or other way around). Typically in a head-to-head comparison, the benchmarks are developed by the same individual whom likely has above average knowledge in their favorite and below average in the target they're trying to show as inferior.
You'll see this time and time again in internet benchmarks comparing performance. Unless you can see the code from all benchmarks involved, my suggestion is to avoid them. I mean, for all I know, the author of the benchmark was unaware of the built in sort and instead bubble sorted.
The test that lowered TypeScript’s score in the paper is called fannkuch-redux, and here are the sources in question:
https://github.com/greensoftwarelab/Energy-Languages/blob/ma...
https://github.com/greensoftwarelab/Energy-Languages/blob/ma...
They are both contributed by the same person, and there is no bubble sort involved. So now you know.
I don’t see an obvious reason one would be slower, but they’re also quite different. Maybe the algorithmic complexity is different. Maybe the cross-compilation is doing something bad with memory allocation. Note the input sizes for this test are very small, it would be easy for a difference in temporary variables the compiler injects to cause a serious problem.
What is not obvious is any prejudice, malice, or incompetence.
What are you talking about? Where did that happen?
> when you admit to having zero idea what the actual specific problem here is?
This is the comment section for a submission about an article referencing the paper. I brought it up for discussion. It is perfectly valid to bring up a question that you don't know the answer to.
> What is not obvious is any prejudice, malice, or incompetence.
Please stop.
Edit: From another comment, and some deeper digging of my own from that, you might find the archived results of the fannkuch-redux interesting. From 2017-08-01[1] to 2017-09-18[2], the benchmark changed from a running time of 1,204.93 second to a running time of 131.39 seconds. The paper was released in October 2017.
1: https://web.archive.org/web/20170901020804/http://benchmarks...
2: https://web.archive.org/web/20170918163900/http://benchmarks...
I was responding directly to @hermitdev. Did you get your threads crossed? What I'm talking about happened immediately above in the parent comment, beginning with "Often times, these sorts of benchmarks are done with prejudice" https://news.ycombinator.com/item?id=19527057
"You'll see this time and time again in internet benchmarks comparing performance."
> It is perfectly valid to bring up a question that you don't know the answer to.
I agree. It's a bummer that's not really what happened here.
>> What is not obvious is any prejudice, malice, or incompetence. > Please stop.
The parent comment explicitly stated an assumption of both incompetence and prejudice and I responded directly to that.
From the HN guidelines: "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith."
If you'd like me not to call out speculation, then please assume good faith and don't speculate next time.
> From 2017-08-01[1] to 2017-09-18[2], the benchmark changed from a running time of 1,204.93 second to a running time of 131.39 seconds.
Yes! Now we are getting somewhere. It appears that would change the outcome of the paper. Perhaps it was a mistake. That might mean it was nothing more than an oversight that already got fixed. It doesn't mean there is any other coloring of the study at all, nor that there was any intention or agenda to make TypeScript look bad, right?
Perhaps I misinterpreted what you said. You started the paragraph referring to the top level comment, which is me. I took the "you're" in "You’re assuming incompetence when you could just go look it up." to be a general "you", and commentary on my original comment.
> From the HN guidelines: "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith."
I actually looked this up before the GP comment, and almost included it myself. I can see now that you were implicating the comment you replied to. I didn't think that was the case, because I apparently didn't interpret that comment remotely in the same way you did.
> Perhaps it was a mistake. That might mean it was nothing more than an oversight that already got fixed. It doesn't mean there is any other coloring of the study at all, nor that there was any intention or agenda to make TypeScript look bad, right?
I never implied it was. For that matter, I didn't really interpret the comment in question as stating that either. The more charitable interpretation is not that they are trying to make another language look bad, but that they are trying to make their favorite language look good. That doesn't require purposefully tanking one benchmark, it just requires them to be much better versed in optimizing one language than another and a lack of awareness about this. As they say, never attribute to malice what can be explained by incompetence. In fact, if you read the comment carefully, they even call out to this with the "not necessarily malice" remark.
The project doesn’t talk about favorites or seem to want to make certain languages look good. Jumping to the conclusion that bias is involved isn’t the good faith interpretation, even if you state with a positive sounding framing. The good faith interpretation is to take the stated project goals at face value, and assume that the participants have done a good job.
I didn't see anywhere that the comment in question called any project bias into question, but instead noted that in a situation where work is crowd sourced, people with their own intentions and motivations will put out bad benchmarks, either in the case of the benchmarks game, or a specific benchmark or comparison put forth in an article or blog. I've personally been witness to the latter multiple times just from HN submissions.
I just want to end with, as someone that's brought up viewing comments in an uncharitable light, you seem to have done a lot of that in this discussion. You've repeatedly taken your interpretation of a comment, rephrased it in a harsher way, and the stated it as what the other person was saying as fact, and then responded to that. I would think actually trying to find a charitable interpretation should at least include a question at the beginning to confirm whether what you think is being said is entirely correct. Note that I started with that when I thought you were attributing statements to me that I did not say. My first words were a solicitation "What are you talking about? Where did that happen?" to confirm what was going on. You've been doing this from your first response to my top level commend, when you stated "But you’re using that assumption to cast slippery-slope doubt on the whole project without knowing anything specific." That's a very uncharitable rephrasing of what you think I was doing, and it certainly wasn't my intention. I've already outlines in specific exactly what I was trying to do and why, and in doing so I also stated that I felt you were misinterpreting me. There's a clear trend here as I see it, and you repeatedly bringing up good faith assumptions just puts it into clear highlight.
I think we've covered about all there is to say on this (these) topics. I'll let you have to the last word if you wish. I'll read and promise to consider any points you raise, but I don't think me responding would be very fruitful, and this discussion has digressed far enough.
5 July, Node 8.1.3, TypeScript 2.4.1
https://web.archive.org/web/20170715120038/http://benchmarks...
1 Sep, Node 8.4.0, TypeScript 2.5.2
https://web.archive.org/web/20170922144419/http://benchmarks...
----
How should we now assess your "suspiciously like entirely different algorithms were used in each implementation" comment?
Yes.
> How should we now assess your "suspiciously like entirely different algorithms were used in each implementation" comment?
The suspicion was incorrect. That's why it was presented as a suspicion, not as fact. I have no reason to defend it if it's incorrect, but I still defend that it was valid to raise questions, given the facts on the ground. We've now shown there was something that changed very drastically at that time, and while it's less likely it's the benchmarks themselves (unless one or both of those are fairly out of date Node versions)[1], it still points towards something to be aware of in the results presented. Namely, they rely on a lot of underlying assumptions which should be looked at if you care about the numbers.
1: Also, I imagine the V8 devs probably considered the performance of TypeScript in that case to be a bug, given how horrible the performance regression from JavaScript is and that it's still javaScript running. It's possible that TypeScript was doing something really odd, but given the exposure and Microsoft's backing and developer time, I think that's a less likely scenario than some optimization that should have been triggered was missing, which happens quite often.
> I still defend that it was valid to raise questions
Of course, it's valid to question a measurement that looks strange but your comment went further than that -- your comment, without evidence, assumed a cause; and, without evidence, implied that assumed cause led to widespread problems with the analysis.
In other words -- innuendo.
Corrections are for facts. I put forth a theory. People being misled by a theory are not something I have limited power to affect. People representing theories read on the internet as fact have larger problems that that will solve.
This discussion is the correction, and a better one than someone would be willing to read. Were it within the 2 hour edit window, I would through in an edit, I've done so numerous times in the past. I will ask Hn to amend it's rules so I can correct a statement I made about something I suspected.
> Of course, it's valid to question a measurement that looks strange but your comment went further than that -- your comment, without evidence, assumed a cause
This is incorrect. I had evidence, I had numbers that did not line up with my understanding of how things should have been given my knowledge of the subject. I presented that as a theory, by using the word "suspect". All I implied is that if that theory was correct, which I made sure to not assert as fact, then it might affect some other languages. I did not assume a cause, I assumed a possible cause, and presented it as such.
I am very particular with my language. I try not to state things as fact when they are not. I try my absolute hardest (and I believe I succeed) to always speak in good faith, where I'm trying to raise a point I think is worthwhile or ask a question where I think there is benefit. I'm actually rather bothered by how some people interpreted my words and intentions, and that includes you. I'm bothered by how you've interpreted my words. Since you're not the only one (although I do believe you're in the minority), I'll assume there's something I could have done better to represent my point. I don't think all the blame lays with me though. There should be some way for me to posit a question and advance a theory without people assuming bad faith, so my question to you is, what way is that? How could I have expressed concern over the results without triggering that interpretation from you? Because I don't think doing personal research on a problem is an acceptable prerequisite for raising a question. In this case, I could have spent hours looking into something I was unfamiliar with and come away with more answers, but many people may not have the knowledge to do so but have enough to think something is wrong. Should they just keep their mouths shut? Are we in a time where raising a concern that turns out to be unfounded (or in this case, just more complicated and slightly misdirected) is unacceptable under any circumstance? I refuse to accept that.
The honest question is -- Can someone please confirm that those programs implement the same algorithm?
No! I think this is a very useful project and analysis. I just think that some languages might have extremely optimized versions (or possibly more likely, some languages don't quite yet have that extremely optimized version that has propagated throughout the others) and that might be affecting specific languages in the analysis.
I think the first 5-10 entries are probably very accurate, as they are generally with very performance centric and often optimized for performance languages. As the languages and VMs/interpreters do more and are optimized less, it's much easier to miss a performance difference caused by an benchmark submission and attribute it to an inherent aspect of the language.
> I am at least giving benefit of the doubt and wondering what might be wrong with TypeScript.
I did no such thing. Note how I used the phrases "Theoretically, I would assume" and "That looks suspiciously like". I simply raised an issue of concern, in a way where it was obvious that I did not know if my concern was correct, and wondered that if it was, what else it might affect.
That’s exactly what I mean by casting doubt. If the concern might not be correct, why lead into speculation about further concerns?
Let’s find out what the actual reason that TypeScript is measured slower, rather than guess as what else could be wrong if unverified theoretical assumptions might be correct.
You mean, I should not have explored the ramifications of what my suspicions might mean to so people might think it's worth actually looking into? What's wrong with that?
I can't help but feel that you feel compelled to defend your original position that I believe was based on a misinterpreting my point and intention.
I saw what I believed might be a problem. I noted it. I noted why. I noted what it might mean to the analysis because if my suspicion as to the reason was correct, it might not be isolated and other items might need a closer look. I did so in a way where I was sure not to claim something as factual when I wasn't certain. What part of that do you think is an inaccurate assessment of what I did or was unwarranted?
Edit: Changed misconstruing to misinterpreting, as that's what I was trying to express, and misconstruing might be interpreted as a purposeful action, which is not what I was trying to express.
The part that was unwarranted was the negative speculation.
The authors presented at an Oct 2017 conference. Archived benchmarks game web-pages from 2017 do not show the 10x fannkuch-redux differences that the authors report --
https://web.archive.org/web/20170918163900/http://benchmarks...
https://sites.google.com/view/energy-efficiency-languages/ho...
Many benchmarks have no TS implementations, TS/JS results are about the same except for fannkuch-redux which is about a zillion times slower in the TS implementation. When looking at that kind of massive discrepancy in similar languages with the same runtime, the guess kbenson made was a perfectly sensible one. The authors of the study should have examined that kind of crazy outlier more closely and it's, as pointed out, not that unusual when using the benchmark game as a starting point.
In fact, on the benchmarks game website, those same programs do not show a "massive discrepancy" --
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
The issue is that one of the metrics in the suite is lines of code, so people write fantastically obscure and concise functional programs in Perl when the imperative one would be 2x the LOC, but much, much faster.
(This is from a spot check years ago. Maybe they’ve fixed this somehow).
The benchmarks game does not measure LoC --
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
In that case, the characteristics of the system will dominate.
I don't think that's a safe assumption. The Javascript results would be a lower bound (assuming the same algorithm, etc.), but there's no telling what "extra" Javascript (and thus overhead) might be inserted by the Typescript compiler.
As far as individual results go, there's no point jumping to conclusions when the results and code are available online and easy to try for yourself:
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
Typescript is still 2-3x slower than Node in several of the benchmarks, and the results could have been different in May 2018 when the article was written.
I would be very surprised if the fastest C++ and C are actually different.
You can see 4 or 5 C++ fannkuch-redux programs "compared"
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
That's true, but the fact we're talking about an order of magnitude running time difference is what lead me to think it's not as easily explained as that.
> there's no point jumping to conclusions
Respectfully, I don't think I did. I put out a theory, with my reasoning, fully acknowledging where it might be wrong. As someone that doesn't use TypeScript (and I wouldn't consider myself in expert in JavaScript either), I don't feel qualified to assess the respective algorithms for those languages in the benchmarks game, so I brought the issue to a larger audience so someone else might take a look if so inclined.
> Typescript is still 2-3x slower than Node in several of the benchmarks, and the results could have been different in May 2018 when the article was written.
That doesn't really invalidate my theory, as it's possible those are also instances where it's lagging in a good submission. Given that there are several tests where it actually beats the fastest JavaScript algorthim submitted, that seems at least plausible.
TypeScript doesn't do this. With only a few exceptions, the way to turn TypeScript into JavaScript is to remove the type annotations, which leaves you with JavaScript. This is an important design goal of TypeScript: it doesn't change your code, it's just JavaScript with types. Like Babel, TypeScript can be used to compile to earlier JS versions like ES5, but I'd consider that a misconfiguration in an environment like this.
Saying that TypeScript is slower than JavaScript is almost as silly as saying "C++ with comments is slower than C++ without comments". If you see a C++ program with comments that's slower than another one without comments, it's almost certainly because they're different programs, not because the comments have any effect. From a glance through your links, it seems likely that all of the running time differences are just because it's different code being run.
...which also seems to be reflective of how much programmers using language X care about (execution) efficiency, so I'd say these results are not far off from reality.
C and C++ are another interesting pair --- proponents of the latter always love to claim "zero cost abstractions" that allow you to write very abstract code which they say the compiler can then optimise (or clean up the mess, depending on your viewpoint...) to the same output as if you did it manually in C, but the results show a very different picture.
(You can then have a debate about whether you want to see idiomatic C++ in benchmarks or purely the most performant C++ possible. Personally if I'm shopping for a language for a project then I'm far more likely to be interested in the former.)
I jest, (and I'm a Rust admirer myself), but my more serious point is: so many different kinds of electricity go into a plush tech company with its well paid developers and our copious brain food that powers us through all our developing and debugging. If you want to talk about sustainability, ask about the lifecycle maintenance of a software base. These benchmarks are cute, but academic, and only tenuously related to any green solutions. Especially if people in this thread are taking this seriously in terms of "This is exactly how we should all be thinking about server engineering moving forward, as we aim to drastically reduce carbon footprint within 11 years", then this is an feels like an awful, awful way to measure it.
What language is most conducive to writing algorithms that are smart in terms of Big O, or designing systems that can be refactored intelligently instead of throwing boxes at the problem?
Anyone from Twitter care to comment on this?
1- This is exactly how we should all be thinking about server engineering moving forward, as we aim to drastically reduce carbon footprint within 11 years. Efficiencies at the language level are one of biggest bangs for buck here. Just by redeploying, you can reduce energy consumption by perhaps double digits. Imagine how hard that is to do at the hardware or energy farm level.
Think about it: You, brave software engineer, can literally make a significant contribution to saving the world by adopting a more energy-efficient language, if you are fortunate enough to deploy something at scale.
2- Rust is killing it in these metrics but developer productivity / friendliness is important to overall success. Looking at these results I have a conjecture that is maybe provocative: The top two candidates for long-term success at supplanting Java on the server in my eyes are
a) Go
b) brace yourself.. Swift
Swift is extremely young on the server, to the point where I'd expect your natural reaction to be "WTFLOL?! never heard of it". But here's some food for thought: the Netty team, one of the top performing Java server stacks, has been recruited by Apple and is chewing through all that stuff and just launched their NIO2 release[1] which I've heard is already very close to Netty.
Go has an amazing concurrent garbage collector and has really pushed the envelope with that.[2] Swift is unique in the server world in that it uses reference counting which sidesteps the whole GC collection problem, which could translate into very low, very consistent latencies as well as memory usage. It's still quite early days for Swift, but these are the two languages I'm watching the closest.
That's a pretty cool idea. I read something on HN the other day about using swift in the back end, but haven't used it myself. Hopefully Apple gets support for non-ubuntu linuxes as well as windows out the door (especially since they're no longer making their own hardware). I've heard anecdotally that it has very slow compile times, so that might be another issue.
> Swift is unique in the server world in that it uses reference counting
Out of curiosity, why has no-one done automatic reference counting on the server side before? C++11 has smart pointers, and I believe python does as well (can't speak to this one personally).
I also feel like most languages using the llvm backend could achieve performance at least close to c++; wasn't that part of the purpose? It seems like Swift might be especially ripe for this because Apple is the big developer of both LLVM and Swift.
Those examples are both interpreted languages, and they have a higher runtime cost for memory management because they're actively breaking reference cycles, so it's not apples-to-apples, but it's still reference counting.
My guess is you fall into one of two categories:
1- You don't care a ton about efficiency because you can scale out to infinite servers that are relatively cheap and affordable. So you can run on django or rails or whatever and it's fine at "pre-IPO" scale.
2- You do care, but Java is good enough.
So you kind of have to be at Google's scale to care about creating a new server language. In Swift's case, the motivation was super constrained mobile devices, so bringing it to the server side is more of an incremental engineering cost against that massive investment.
Also take note that the scale of the graphs is different. E.g. Figures 5 & 6 makes it look like Java energy is super low when it's really just similar to the compiled languages.[1]
[1] http://greenlab.di.uminho.pt/wp-content/uploads/2017/10/sleF...
1 - Why would you want to replace java on the server? (Seems obvious that it provides the best balance of productivity and performance per watt.)
2 - Assuming a company would want to replace jave, why, on earth, would we not use C++? (Or, perhaps, rust if we want to use a newcomer?) Why are we better off going all the way down this list to go and swift?
Well one reason, if you look at the paper, is memory consumption. One of the ways Java gets good performance despite being a GC'd language is less efficient memory usage. (Although there are many advancements in this area akin to what Go has accomplished but as the paper's metrics show, often Java uses a lot of mem.)
Also probably lots of people will agree that it has accumulated some cruft over the years and predates lots of modern trends and things like concurrency and functional programming and whatnot require more complicated code patterns.. a classic example is the getter/setter verbosity of a simple value type (not trying to start a flame war here).
However it's fair to say Java is the thing to beat in terms of "goldilocks" languages that blend productivity/safety with performance.
> 2 - ...would we not use C++?
Because it's a huge regression in productivity and safety. The goal is to reduce memory consumption while achieving great performance and, crucially, consistently low latency which is a greatly underappreciated server metric and really should be the number one benchmark IMHO.
???
You're recommending a slower language, that uses more watts, on a presumably 24/7 backend...
to save a few bucks on memory?
As I said, if you want a company to switch to c++ or rust because you want to save money on memory IN ADDITION to all the money you're saving on watts, while getting the same or better performance? That might make some sense.
Switching to go makes, No, sense. You end up paying more money per user session to do the equivalent work. (Even worse, each user waits longer because the work is done slower.)
And then, like typical engineers, we'd proceed to explain to our bosses that all of this is actually better...
because our new program uses less memory.
Also, a request. Try to take it easy on the ???, all caps and excessive italics.
And golang does what exactly for this? As a matter of fact, Java is superior here to golang on these fronts, and is only going to get better. There is a lot of unsubstantiated golang hype and people should know better.
Not sure what you're responding to but I specifically said "at scale" and "developer productivity / friendliness is important to overall success."
If it were up to me, no matter the energy consumption, a piece of software in that realm should be proven correct (both crash-free and doing what the specification says).
You do realize that a lot of autonomous driving software uses ML techniques that make providing guarantees like that difficult, right? (To be honest, oftentimes we can't even provide an explanation of why an ML decision was made, let alone "prove" why an ML decision is made. We only show tha most of the time, 98.3995% or whatever, the system should do "something like this".)
But in any case, none of that would affect the languages at issue here, because no one would write self driving software in any of these languages. There's about a 10,000% chance that any such startup would use GPU languages to do the meat of that work. Almost 0% chance any reputable company relies on something like go or swift to drive an automobile out on public roads.
You couldn’t be more wrong. Tesla hired the creator of Swift to run Autopilot software.
He’s now at Google where he’s working on making Swift a primary language for TensorFlow.[1]
[1] https://en.wikipedia.org/wiki/Intergovernmental_Panel_on_Cli...
Just pointing out this distinction because the “can’t predict next week’s weather” is a disingenuous soundbite that has been used by climate change deniers to discredit climatologists in the eyes of the of the public who don’t understand the difference. I’ll give you the benefit of the doubt that you weren’t trying to be disingenuous, but the distinction still makes your question somewhat irrelevant.
JVMs and CLR had those a decade ago. The state of the art are concurrent, compacting, region-based pauseless or millisecond-pause collectors.
I'm sure you're also aware that recent Go's GC pauses are sub millisecond for most use cases:
"We now have an objective of 500 microseconds stop the world pause per GC cycle." - 2018 Go team
https://blog.golang.org/ismmkeynote
My personal experience with microservices is to expect STW pauses in the 350 microsecond range. The best part is that it requires zero tunning or developer's attention while still being light on memory usage. Can't say the same for Java's default GC.
OpenJDK's default collector - parallel or G1GC, depending on version - is not the best available among JVMs and if your goal is pause times then yes, it will be worse than Go's. But if you switch to say C4 or ZGC you'll get comparable pause times and compacting on top and being able to scale to terabyte heaps.
10 years ago we had Metronome and CMS, which are more comparable to Go's collector.
And pause-times are not everything. Throughput and fragmentation resistance matter too. Compacting collectors fare much better on the latter metric. I don't know how the former is now, but those slides talked about 25% GC overhead in older versions of Go, that's utterly terrible.
Go is facing one challange that java doesn't: internal pointers. But the CLR's collectors have to deal with those too, so that's not terra incognita either.
> 25% GC overhead in older versions of Go, that's utterly terrible.
25% overhead of what? And compared to what? Just throwing numbers in the air and saying it's terrible makes no sense.
The only 25%'s if could find in the slide were these: https://blog.golang.org/ismmkeynote/image6.png
2014 Go: 25% of CPU used by GC
2018 Go: 25% of CPU used during 2x STW GC of < 500 microseconds
So even if STW GC occurred as frequent as every second (which it doesn't in my use cases), this would amount to 0.025% of the CPU being used for GC, not 25%.
> Of course gains are to be expected when switching from Java's default GC to something specialized.
So? That's irrelevant for what's state of the art.
-Note- I'm not saying one is better than the other. ARC works great on iOS for creating applications.
I think Rust w/ macros like you see in Rocket (https://rocket.rs/) looks promising. I haven't used it, but the guarantees, simplicity, and performance (possibilities - it doesn't use async yet) are really interesting.
Reference counting, by contrast, is entirely predictable. It doesn't defer any work. I would argue that CTOs are a lot more interested in consistently low latency than in requests per second. So it's very interesting to see an approachable, performant language take the ARC route on the server. It is early days though.
ARC would be just one my issues using Swift outside of iOS.
I think you are overstating the pause duration of a modern GC. I have used both Go and the JVM server side, not at a huge scale but enough to see GC effecting response times. They add some fluctuate, but nothing compared to network latency or the multitude of other factors that fluctuate heavily. It was never significantly relevant for response times. I'm looking at my server logs right now and theres not even a real correlation between GC and response time. Unless you are considering a 0-10ms fluctuation.
If you are interested in using an ARC server side I know Kotlin Native is using ARC however their implementation eliminates the cyclical reference issue.
Funny you say that because just this week there has been an investigation into what seemed to be “leaks” but turns out it’s memory fragmentation. It’s a fascinating read into how to debug a server problem if you’re into that sort of thing.[1]
Agree that modern tracing GC can be very good in a wide range of cases but there are some where it’s not. Very dependent on the case. Ultimately you are deferring work and hoping to find some time in the future to squeeze it in unnoticed. ARC is a cool paradigm to explore on the server as it doesn’t have this problem to begin with.
[1] https://forums.swift.org/t/memory-leaking-in-vapor-app/22209...
Memory fragmentation is another legit concern I guess, as far as I can remember iOS has no memory compaction. Again not a necessarily an issue for a short live user space application, it is a larger one than memory leaks at least at my company. In some hot spots of our app we specifically slow down reading of some queries to reduce memory fragmentation.
Frankly Chris Lattner's claim that a GC leads to 2x-3x memory consumption over ARC is unfounded and sorta shocking coming from someone held is such high esteem. It's something thats continually shown to be untrue.
It always seems Swift's biggest selling point is it uses ARC instead of a GC, which is either not a large issue or a GC is actually more beneficial. Other than that you still haven't dealt with the toxic "Swifty" community, the terrible tooling situation, the immature libraries and frameworks, etc, etc.
There seems to be so many better solutions to writing server side code. This is all coming from someone who uses the language on a daily basis.
(And then you have ref cycles which is another kind of headache. And poor cache behavior from all those refcount updates.)
My rule of thumb now is that memory access, not compute, is the primary consumer of energy. That would tend to confirm the above statement while also supporting the data that scripting languages aren’t super energy efficient, since they tend to do a lot more dynamic allocation than compiled languages, when generally broadly speaking about common programming practices in each language.
This is mainly colored by GPU usage and a paper/presentation some friends made: https://www.researchgate.net/publication/324217073_A_Detaile...
In the GPU case, memory access costs sometimes 10x more than compute, meaning that minimizing average (not peak) memory traffic is more or less the only path to significantly reduced energy. (See figs 7 & 10 in the linked paper)
What's important is wattage per unit of useful end product, in my example pricing an option. For others, it might be wattage per web page served or anything else.
The JS/TS comparison was directly fishy. I found no mention in the paper of how they managed to "run" the tasks in Typescript, which AFAIK is not possible, or at least not how one would do it in a real life.
Unless you have very large numbers of [users|servers|whatever] to amortize the cost of going one way vs another over, it's probably not worth even considering trying to change anything on your end. More likely, your personal or team productivity is the much more important metric to optimize for anyways. If you're doing things at FAANG scale, it's an entirely different story.
Nope but for iot devices or anything in a scenario with scarce energy resources or difficulty to remove heat it's an interesting datapoint. Although I think the results conform mostly with intuition. I doubt anybody would voluntarily compile a lot of haskell on their space satellite
https://spectrum.ieee.org/computing/hardware/the-future-of-c...
Glad that it's still moving, sad that it's at an even slower pace than the "functional/immutable will save the parallellization worries" stuff I encountered in the same period.
I don't know enough about the performance of Javascript vs. Typescript to form a strong opinion about it, but if history is any guide, it's likely that it improves software development efficiency at the expense of performance.
I don't know about javascript's usual JIT compilers, but LuaJIT's JIT compiler (trace compiler) does a pretty good job of figuring out abstractions.
For example, saying that Python uses much more energy is a foolish thing to say. The “same” algorithm written in pure Python is a very semantically different thing. It involves allocation of flexible objects that obey certain attribute lookup protocols, operator protocols, dynamic attribute mutation / creation, iteration protocols, etc. It is presumed that if you wrote in pure Python, you need this dynamism and ability to introspect at runtime, modify data structure layout arbitrarily, utilize an automatic garbage collector, etc. So you’d need a benchmark test that requires all that functionality before it could possibly make sense to test Python. Otherwise you’re penalizing Python for a bunch of expensive overhead, which is totally unfair and foolish because the whole point is that such overhead exists for use cases where either the costs are negligible or the flexibility that necessitates that overhead in any language happens to be a desired and important part, such that to write the same functionality in other languages would first require building all the protocols, garbage collection, etc. machinery that essentially defines Python.
If instead you have some benchmark problem that can be solved in e.g. C or Rust without needing any runtime dynamism or heavy machinery of certain protocols or garbage collection, then to write it in Python you would just write it in Cython or as a hand-made C extension module to specifically bypass the overhead of the interpreter or Python protocols / dynamic lookups / etc.
Basically, it never makes sense to compare Python and C on a benchmark that doesn’t require reimplementing most of Python in C first. Because to solve that problem in Python, the ubiquitous, basic, Python 101 way to do it would be to use Cython or numba or write your own extension module.
Python is basically a special C language DSL for dynamic types, garbage collection, and a set of conventions and protocols.
“Pure Python” is a linguistic trick for saying “a huge ton of tools that make C-level polymorphic structs easy to use.”
* edit- thinking about it it's probably negligable
As for any other product, this is a trade-off between fixed and variable cost: want to do some quick processing on an hourly basis? Just write a quick python script with a Cron. Want to compute something billions of time? Use C or something more modern that compiles. Need even more savings? Build an ASIC doing the job.
IMHO it's not quite right to evaluate energy efficiency just by looking at the runtime performance. Compiled (incl. JIT) programs have to be compiled. So for compiled programs, you have upfront costs (e.g. rust, haskell much more than c), you would have to include in the calculation. Then you have programs/script that run once (a day maybe). I wouldn't be surprised if taking this into account would change the calculation.
We have wanted to do a similar conversion from the results data in our TechEmpower Framework Benchmarks [1] to energy efficiency. I've wanted to go as far as converting watt-hours to some average carbon emission rate for electricity generation in a given region.
I imagine the carbon emission per request would be amusing, if nothing else.
I’m all for figuring out what’s the most efficient option while also ensuring it’s an effective one as well. If you kept a knife in its sheath to keep it nice, but never take it out to use it, then it’s not servicing its purpose as a tool (ignoring ornamental ones). However, no matter how cool it might seem, you really don’t need a samurai sword to pear apples.
I believe that's because they're all using a regular expression library which is probably written in C/C++, maybe even with some pieces in optimised Asm, so none of the actual RE-work is being done in the interpreted language itself. If you wrote actual RE processing in the interpreted language and let the interpreter interpret/JIT it, I bet they would be as (in)efficient as the "other scenarios" where the "heavy lifting" is running through the interpreter.
Of course, using Python means you have a lot of low-hanging fruit to pick. We have a Python service where we moved ONE ~60 line recursive function to Go and overall CPU consumption dropped to 15-20% of what it used to be.
So there's no actual Python-Go interaction in our code.
This was quicker, cleaner and has made the page more responsive for users.
To get to the required two decimal places of precision, each call would recurse 50-60 or more times so this was more or less all that the server was doing.
A VERY low hanging fruit.
https://sites.google.com/view/energy-efficiency-languages/se...
The LISP evaluated by the Benchmark Game - and, apparently, this paper - is indeed SBCL [3], which seems about on par with Java.
“Lisp, on average, consumes 2.27x more energy (131.34J) than C, while taking 2.44x more time to execute (4926.99ms), and 1.92x more memory (126.64Mb) needed when compared to Pascal.”
[1] https://sites.google.com/view/energy-efficiency-languages/ho...
[2] https://github.com/greensoftwarelab/Energy-Languages
[3] https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
What surprised me is how much of a hit the OO programs took. I had thought they would compile down to something reasonably similar to the imperative languages, but C++ ended up being 50% slower.
I wonder if anyone has attempted to rate how idiomatic or typical CLBG entries are, and whether choosing the most idiomatic implementation for each language would have obtained different results?
These benchmarks are all single-threaded, pure computation. Erlang is pretty slow at that (alas).
For being a mostly pure functional language it performs pretty well in that category. To the parent comment’s question, the reputation for solid performance likely comes from Ruby/Python developers. When dealing with web server applications it really shines due to how it handles concurrency. Also NIFs are really easy to write, especially in Rust.
From http://erlang.org/doc/man/erl.html:
+sbwt none|very_short|short|medium|long|very_long
Sets scheduler busy wait threshold. Defaults to medium. The threshold
determines how long schedulers are to busy wait when running out of work
before going to sleep.Rust was specifically created to use the paralell hardware resources well, and it excels at it.
Something like 90% of all credit card transactions.
Much higher on the list of concerns is finding capable COBOL developers, ongoing maintenance, and security. Those are all things that you can't just throw money at.
Java is also compiled.
For instance, switching to ASICs or even FPGAs that are optimized to handle certain types of computations.
Just look at datacenters or mobile devices. Sure, optimizations are made to runtime environments to improve performance, and increase battery life. But you are going to see bigger gains through chip architecture. And that is what vendors focus on.
Or to put it another way. Why do you think mobile devices almost exclusively use ARM? Why do datacenter operators invest heavily in R&D for ASICs instead of engineering new languages or runtimes? Because languages play a secondary role in energy consumption.
This study is just like countless of other meaningless benchmarks seen in language war flame posts. Pointless and misses the big picture.
Facebook and Google both have in house chip designers these days, specifically for their data centers.
You will see different energy outputs per language based on their compiled results depending on chipset architecture. And you will see different deltas between them; e.g. language x may use 10% more energy than language y on chipset A, but use 20% more energy on chipset B. And usage will vary depending on which kind of task is performed on which chip architecture, and with what language.
So you did a poor job setting your controls. The effect of the programming language was not well isolated in your benchmark trials, so setting chosen language as an independent variable is flawed. Who knows whether the overall ranking of languages would be affected had the benchmarks been done differently.
https://www.cvedetails.com/vulnerability-search.php?f=1&vend...
https://www.cvedetails.com/vulnerability-search.php?f=1&vend...
Pedantic note: quite a few of these vulnerabilities don't technically contradict the GP's claim because they're in C++, not C, but I hope there are enough C ones to disprove the point.
Google & AWS pay money to develop & acquire technologies that save energy because of their scale. The energy used in dev is scratch compared to the energy used to run programs at scale. Google, for example, has a PHP compiler that makes all PHP web pages execute in a fraction of the energy usage of running the PHP interpreter.
Perhaps awareness and concrete data of energy usage has to come first? Yes, the article is pointing out something that all the developers you talk to might not know. Doesn’t that make it a good thing? Maybe in the near future you will hear them start to talk about energy. Maybe now that you’re becoming aware, you can be the first among the people you know to talk about the energy usage of your software choices.
Anecdotally, my own experience is that I didn’t hear a lot or think a lot about energy usage until I switched from games & web development to working at a hardware company. My peers now do talk about energy, even though most only write software.
You’re right that most people do choose based on convenience, especially when they lack any other reasons to make the choice. Historically, choosing solely on convenience has contributed to global environmental problems, and people globally are only just becoming aware of the environmental costs of their choices. Separately, Moore’s law only just recently stopped working, so it’s not surprising to me that energy usage has suddenly become more important. Reducing energy use is now one of the primary ways we can increase speed & efficiency, unlike the recent past.
But HipHop is certainly another good example that demonstrates that this matters in practice.