Intel Fortran now 20% faster than C, fastest language on Shootout | Hacker News Reader

Intel Fortran now 20% faster than C, fastest language on Shootout

shootout.alioth.debian.org

72 pointsmarkokocic49 comments

Intel Fortran vs gcc ???

How about Intel Fortran vs. Intel C ? How about gcc C vs. gcc Fortran ?

Hyper-optimized compiler for Intel architecture beats compiler designed to be portable to many, many target architectures? Not very shocking.

This. ICC is known to beat gcc usually.

Yeah, I'd suspect that better use of SSE instructions is the big win for number crunching.

https://alioth.debian.org/forum/forum.php?thread_id=14867...

Cool. Yeah, seems like it's better inlining and use of XMM registers more than the SSE instructions themselves.

Most of Fortran's gain is on mandelbrot, but it sucks on binary-trees (10x slower than C). It also tends to use 2x more memory (with a 23x outlier on fannkuch): http://shootout.alioth.debian.org/u64q/benchmark.php?test=al...

binary-trees is rough to beat C on. It involves allocating and traversing some pretty gigantic data structures, and the C code is optimized by using region-style allocation to avoid all of the individual calls to free (or garbage collections in a GC'd language). From the CPU usage, it also looks like they also haven't parallelized the Fortan version yet, though even at ideal speedup it'll still lose.

I've been doing some tuning work on our parallel dialect of ML, and many of the C programs are fairly well-optimized, though mandelbrot still has some room for improvement.

From a simple look, the Fortran code seems to have an additional optimization that allows it to avoid a lot of checking for overflows in the innermost loop.

This is a really terribly done study, with a misleading title to boot.

First, even if we take the conclusion as given, "Fortran" isn't faster than "C". The median of one set of programs, compiled with one Fortran compiler with one set of options and run on one processor, is faster than the median of another set of programs compiled with one C compiler with one set of one and run on that processor. This tells us approximately nothing.

If you only expand your consideration to include the whole distribution of programs they tested, GCC actually looks better than iFort (eg. there is a single program which ran 2x faster with iFort, but multiple programs ran more than 2x slower). Median is a spectacularly poor choice of metric, especially with so few samples in the distribution.

But also: Intel also makes a C compiler. Why wasn't that used? How were these benchmark programs chosen? Etc.

This study says close to nothing about "Fortran" and "C", or about any of the other languages involved.

This study says close to nothing about "Fortran" and "C", or about any of the other languages involved.

Sorry, but I disagree here. It's one of the best resources out there for getting numbers on the relative performance of languages, despite the pitfalls. The implementations of popular languages are well tuned (sometimes extremely so), the used compilers and settings are quite sane, and the benchmark programs are fairly well distributed in functionality.

The median is a reasonable choice because it's less sensitive to outliers and was probably chosen for that reason. "Spectacularly poor metric" seems to be an opinion, not a fact.

I admit it would have been better to compare Intel C with Intel Fortran. Why wasn't this done? Perhaps some of the submitted programs don't compile correctly with Intel C, or perhaps Intel C doesn't actually perform well on them. Who knows.

But to say that this is a "terribly done study" because of that is doing it injustice.

The median really is a spectacularly bad choice when there are so few data points, and especially when thinking about performance. If A and B get 3x faster, but C gets 3x slower, then your overall workload gets slower, even though the "median performance improvement" is 3x. Obviously the effect is magnified with only 3 data points, but the number of points used in the examples on that page is not so much greater as to eliminate this effect.

Adding to this is that significant slowdowns, the "outliers" that you're keen to disregard, tend to dwarf performance improvements that the median might show (because of the multiplicative nature of these values: if A/fortran is 10x faster than A/C, and B/fortran is 10x slower than B/C, then A+B/fortran is 5x slower than A+B/C).

What's really right is to show the complete distribution of speedups and slowdowns (as one of the detail pages on that site does). If you look at the actual distribution for the comparison of C and Fortran in question, C actually comes out looking much better than Fortran does (which shows precisely how misleading using the median can be).

If you must to boil it down to a single number, there are several choices better than the median. I would be happier with a mean, despite its imperfections.

Mean would be a better choice if a typical real-world application were split approximately equally between each of these tasks. Which is of course patently untrue, particularly so because writing number-crunching code in Fortran does not prevent you from writing your hash-table code in something else (note how the best C binary-trees code is actually calling an external library).

Comparison of medians is nice because it shows that for approximately half of fairly diverse tasks one language is faster than another, which imo is a much more interesting statistic.

> What's really right is to show the complete distribution of speedups and slowdowns

The page the OP linked DOES show all the distribution - that's the beauty of box plots.

Median is a bad choice, because it completely ignores just how bad the worst performance holes are for a compiler. Discarding outliers is bad methodology in this case.

The outliers are not discarded. The outliers are shown both in the chart and in the table (Fortran Intel 25.26).

http://shootout.alioth.debian.org/u64q/which-programming-lan...

Median can be described as what you get when you discard all but the centremost value or two as outliers.

You don't seem to have even looked at the web page.

For each programming language implementation, that web page shows 7 descriptive values - not just the median!

As I've already said - The outliers are not discarded. The outliers are shown both in the chart and in the table.

"7 descriptive values" - Indeed that is true. But the median is used for the ranking and that is why I criticise it.

The worst value for Fortran, one of the included values, is more than five times worse than that for C. This suggests (not proves) that the Intel Fortran compiler has worse performance holes than gcc C that show up for some, but not all, of these benchmarks.

> that is why I criticise it

You criticise but don't even suggest an alternative, let alone evaluate how well or badly that alternative compares to the median.

That's empty criticism.

What is bad about median is that a compiler that performs well on most benchmarks but then falls off a cliff for many other, but less than half, performs well altogether.

I think the distance-weighted estimator would serve the purpose of the ranking better.

http://en.wikipedia.org/wiki/Distance-weighted_estimator

This measure finds a measure of the centre of the distribution that gives less weight to items further away from that centre without discarding them. In this way, the measure is not dominated by outliers, but they still contribute to the final result.

I don't say that this is the best measure, only one that is better. A factor analysis might get at the extent to which the different benchmarks are doing the same thing. Some attempt to find measures of how these activities are represented in larger pieces of code might suggest weights for the benchmarks (there's a literature on analysing loads that might be relevant here). But this would make their analysis more complex and more prone to bias leaking into their analysis, so I don't say they ought to do this; my point is that median is simply a bad population measure for the purposes of their survey.

You have argued against me without trying to defend the choice of median as the measure used for the ranking. Is that denialism?

> Is that denialism?

It isn't any kind of ism.

> What is bad about median is that...

Again, that's what's bad about the median as the sole characterisation of the measurements - but the median is not presented as the sole characterisation of the measurements.

The outlier Fortran measurements are there for all to see.

Some variation on the distance-weighted estimator might work here -

http://shootout.alioth.debian.org/u64q/which-language-is-bes...

True, but what is centremost still depends on the data you just threw away. So this is pretty lame and meaningless.

What the outliers are is always distribution dependent.

This is a really terrible criticism of the linked page, with a misleading first statement to boot. Where do you see the word study? The title at the top of the page says "Computer Language Benchmarks Game". How is calling it a benchmark game misleading? That's what it is, and it doesn't pretend to be anything else - it is a sorted list of benchmarks run on one particular configuration.

They conducted a series of experiments, and reported their results. If that's not a "study", I don't know what is. The fact that the authors acknowledge that it's a "game" doesn't change that. What is misleading is the title on the link: "Intel Fortran now 20% faster than C".

What makes you think the title shown on Hacker News has anything to do with the linked website?

Can you find the statement "Intel Fortran now 20% faster than C" anywhere on the linked website?

> the authors acknowledge that it's a "game"

See http://shootout.alioth.debian.org/dont-jump-to-conclusions.p...

With Fortran this is expected, since for language semantics the compiler can perform more assumptions compared to C, resulting into more optimizations.

For the shootout, all the source code is visible to the compiler (though at least GCC needs to be told so with a flag). So I don't think there's much more assumptions that can be made.

AFAIK GCC doesn't add "restrict" to pointers by itself. Fortran semantics mean that pointers are "restrict" by default.

How does this matter when the compiler can see all the source code? It can see what the pointers are pointing to and determine if there is aliasing.

Feel free to point out the source code in the shootout where this would somehow not be the case. Shouldn't be hard to find if you focus on the benchmarks where Fortran beats C heavily.

Additionally, the argument about restrict is flawed because it can be added to the C code too, so it can't be an advantage of Fortran.

It is impossible write a compiler that reads a C source code and for every point in program decides if two names visible from that point cannot alias each other. You cannot do that even under assumption that every code path may be visited.

I agree it is impossible in the general case (it's obviously impossible when input data controls program flow, for example). But I don't think that has much meaning for the benchmarks under consideration, which, by the way, one would expect to sidestep the obvious pitfalls here.

The impossibility of solving aliasing in the general case doesn't stop C compiler writers from implementing alias analysis.

I assume xyzzyz was referring to the undecidability of the halting problem, not just user input. Comparisons to the halting problem are often made inappropriately, as would be the case here, since the compiler doesn't need to answer whether an arbitrary program contains aliasing, but only identify programs that admit arguments (often simple ones) demonstrating that they don't alias. It's still undecidable, though.

void* p1; void* p2;

p2 = function(p1);

Do p1 and p2 alias? Note: function() is in a *.so file, no source available.

How many times do you have all of the source code available for analysis?

I pointed out specifically that all source code is available exactly for this reason. The original claim was that "for Fortran this is expected" which isn't true at all in this condition. So aliasing information can't explain the performance difference.

These reports are worse than useless. Any person with good understanding of computing knows that implementation plays a huge part in it. Why people wasting their time comparing speed of different programming languages while they should focus on different implementations. I guess the latter one is difficult so people all try to avoid it, and the first one is just so easy to do and everybody is doing it.

Sorry, but your Ruby code will never the same speed as C code, even if it is the same implementation. We'd all like to believe all programming languages are equal, but that's not the case.

Which of course the web site freely acknowledges. At the bottom of that page states the real question is "Which of these programming language implementations have the fastest benchmark programs?"

And at the top of that page ;-)

That fact stands out immensely. The Scala code and java code should be equal as the first can be written exactly like the second. Don't even see Clojure. Hate to see what times they get there.

Except Scala has no 'break' (that isn't a hack using exceptions), this makes imitating imperative style difficult and some syntaxical sugar should no doubt ensure that Scala will always be slower than Java.

What would be interesting about Scala programs written exactly like Java programs?

Hint: If you want to check for some text on a web page, use page search.

Clojure is there but where is clang?

clang is waiting for someone like you, who wants to spend their time building language implementations on the LLVM toolchain and measuring program runtimes -

http://shootout.alioth.debian.org/demo2/which-programming-la...

http://shootout.alioth.debian.org/help.php#llvm

Fortran was designed to munch numbers. As most programs are written by people who will not spend much time getting that last 3x of performance out, a language like Fortran makes sense for the number crunchers. Its easier to screw up in C. These kinds of tests are not all that great, as the code is optimized. Its un-optimized code that matters, since that powers the world.

FWIW, This Developer's Life has an interesting interview with a fortran programmer this month: http://thisdeveloperslife.com/post/2-0-7-dinosaurs

Is there some reason that gfortran from the GCC project is not on the list of compilers for this language shootout? Instead only the branded "Intel Fortran" compiler shows up. Hmm.

Fortran's always been faster than C

You have to hit F5 to reload page, since otherwise the old page will be displayed from browser cache. Probably something wrong with page cache expiration settings.

Reply on news.ycombinator.com