NASA: High Performance Fast Computing Challenge
herox.com
herox.com
Edit: on the other hand graduate students and postdocs are notoriously underpaid and often have a lot of freedom in how they spend their summer time. I guess that is the target competitor pool?
With this sort of code you also tune it to the machine architecture back when I did F77 for BT / Dialcom the code was tweaked to suit the PR!ME Architecture to get extra speed
i've heard people refer to c90 as modern before.. (in the last two years) and the nasa/astronomy guys are famously clingy to 'known good' tech from the olden days.
only the very most recent astronomical dataset i acquired was in .csv. everything is made for fortran from what i've seen over the last 10 years...
Not quite, the submission deadline is already in 8 weeks. And before you see any code, you need to go through their approval process, which they say takes several weeks. So you only have maybe 4 to 6 weeks to get the actual work done...
That same organizer is now gnashing their teeth, realizing full well how the organization extracted all the value out of that 2-year grant just forcing the proposal through (the money no doubt expires on 30 September but NASA bean counters no doubt refuse to cut checks after 31 August, or maybe even 31 July).
A hundred GS-7 employees, with an average education of a high school graduate, extracted the value of this poor bastard's grant with dozens of crappy PDF Standard Forms, emailed to distribution group of idiots and revised a dozen times each, and no, we can't even consider Form 2 until the President of these United States himself has signed off on Form 1.
I gotta tell ya boys, working for the Government sucks, but if you don't start stepping up, you're going to find your government, complete with nuclear weapons and responsibility for your clean water and bridge safety, is totally run by complete idiots.
i do not see this opinion espoused, nor lived by often enough today.
kudos.
This is not at all ready for a competition. The least they could do, if they want professional volunteer help, is to open-source the platform that everyone would be deploying on. It was probably built using public money, after all.
Sigh, no shame:
> FUN3D is: Code developed by the US Government at US taxpayer expense, ...
I'm also somewhat amused by the claim they expect a 1000x performance improvement.
No... New software. Huge, mission critical, core project, software.
The best part: No documentation, no maintainer, and no migration plan.
Scientists think in terms of how Units of Science per Units of Work. Translating code, learning new languages, testing, checking correctness, software validation, bug testing, or even just using external (non-science) libraries yield a very low science/work ratio.
Just opening a text editor, writing a language you already know, and calling it "good enough" is high science/work.
Their method of software validation usually boils down to "spit out your data, graph it, and see if it looks like what you expect".
There's also this notion that "Fortran is fast" that MANY people hold for some reason. They don't know about what kinds of modern compiler optimizations they're missing out on and what new IPC frameworks are available. They know Fortran is fast, MPI is parallel, and that's what they use.
Right now I'm looking at a codebase that is actively being developed and is written in Fortran. It's about 48,466 lines in total. The source is about 2.3M in size. No one seems to have formula listings for it. I want to rewrite it but pulling apart fortran is very difficult.
It's very scary that if I were to write the software in C that it would be seen as modern! This software is tax-payer funded and included in real time systems and production software.
(On that note if anyone is good at Fortran and can document the formulas in this software so I can rewrite them I'd be happy! Please email me)
Compared to C or C++ it has the advantages of better complex floating point number support and better control of reproducible math results. And there is better possibility for compiler optimisation because the langauge semantics let the compiler know one array will not alias another.
Compared to Java, C#, or Python, or any other JVM language, it has the advantage of compiling to machine code for every relevant supercomputer and cluster architecture.
And all of the well-tested high performance computing libraries are written in or support Fortran. The ecosystem is there.
Don't assume things are done purely by inertia -- for big projects there are real reasons for the choices made!
No it does not have better complex types. C++11 added std::complex which is a standardized implementation of complex types for all data sizes.
> And there is better possibility for compiler optimisation because the langauge semantics let the compiler know one array will not alias another.
These optimizations will only ever be made if Fortran has wide adoption.
These optimizations are probably already done in GCC/Clang. Fortran also misses out on a HUGE optimization that can be provided by the existence of explicit inlining from C/C++ and implicit optimizations from things like C++/Rust. There's also constexpr from C++ which, in C++17, will offer huge benefits to non-trivial code.
Most formulas can be expressed as an inline function or constexpr and as a result can, in some cases, see huge performance improvements from compiler optimizations.
> Compared to Java, C#, or Python, or any other JVM language, it has the advantage of compiling to machine code for every relevant supercomputer and cluster architecture.
You can do this with Python. Or you could juse use Julia which is built for clustered environments, performance, IPC & machine-to-machine communication.
> And all of the well-tested high performance computing libraties are written in or support Fortran. The ecosystem is there.
So is with most of these languages. C++ is just as developed in this aspect, Julia is getting there but it's already ready for most applications.
> Don't assume things are done purely by inertia -- for big projects there are real reasons for the choices made!
I'm not. I'm assuming these decisions are made because people in physics/ME/chemistry are not programmers and don't know about the latest and greatest our field has to offer. It's our fault for not advertising our successes to these people.
I think you completely fail to realize that Fortran does have wide adoption, and that there is a big market for commercial and open-source Fortran compilers, and that these optimizations have already been made.
I also don't think you realize how much more productive a scientist is with a language that explicitly lacks all the bells and whistles of e.g. C++. Just like how many fiction authors prefer to use old and simple text editing software, hell even mechanical typewriters, while you would argue an iPad with keyboard cover has so much more functionality.
If you want some examples, go have a dig through the OpenFOAM C++ source code [1], and compare it to e.g. the Pencil Fortran code [2]. They're both CFD codes, though the specializations are different. The former can best be characterized as "templates all the way down", throwing you 800-line error messages when you miss a semicolon, while the latter is downright pleasant to install, use, understand and debug in comparison.
100 times this. Few years ago I personally edited Makefiles in one HPC project to test GCC and clang as possible replacements for PGI. Lemme tell you this: they were joke. Like, 2x slower.
First of all, commercial Fortran compilers can do all the simple optimizations like inlining, loop unrolling etc. And then on top of that, they come with loads of HPC-specific optimizations like autovectorization, loop tiling, loop fusion and hell knows what else. GCC recently seems to be developing in this direction, maybe they'll become competitive at some point or maybe already are, clang was behind GCC last time checked.
In any case, state of the art Fortran compilers are good. And guess what, GCC has a Fortran frontend too. No need to speculate, you can go and test GCC on your codebase now.
In fact, most modern high performance compilers, open source or commercial, have multiple frontends for multiple languages, usually including C and Fortran.
You're assuming wrong. Feel free to join us in scientific computing and make your case. Many have, and yet we still use fortran alongside the newer stuff.
Perhaps people who disagree with you do so out of something other than ignorance. Just a thought.
Also, people are now moving away from Fortran. There's a huge push in my field to use python and many students (like me) are't using Fortran.
Julia is a JIT language. Whatever edge you get in pretty code is going to be paid for in speed. Also, it's pre-1.0 and breaks compatibility frequently. 40-year-old Fortran code runs fine; 2-year-old Julia code is a syntax error.
Rust is a nice language, but all of its numerical computing is experimental 0.x stuff. You'll probably end up wrapping another library to do your number crunching. It's probably written in Fortran.
C++ has pointer aliasing shenanigans, so it's hard to make it as efficient as Fortran. I have seen a couple of projects that tried to port Fortran libraries to C++. They came out slower, buggier, and less readable. The only reason you'd want to do that is a specific hatred for Fortran -- which you are not the only one to have, but over the decades it hasn't proven to be a good motivation for redesigning software.
Of course people like to use Python, but you don't get to use Python instead of Fortran. That would be incredibly slow. You use Python and Fortran. For example, you use NumPy and SciPy, which make Python a great high-level platform for numerical computing because they wrap large amounts of Fortran code. If you use Python for numerical computing, you are not "moving away from Fortran", you're embracing its success.
I don't actually know Fortran, but I have huge respect for the people who do. They're the reason most of the things I do are possible and fast.
1.0 will be out around midway through the year, so that will ease the growing pains.
That is incorrect. This nonsense that JIT is slow is an old belief that's hung around because it sounds nice. The fact is that the numbers don't line up [0].
> Rust is a nice language, but all of its numerical computing is experimental 0.x stuff. You'll probably end up wrapping another library to do your number crunching. It's probably written in Fortran
It's new and it's growing. It's also a single example for a single case.
But it even still have a comprehensive numeric library [1].
> C++ has pointer aliasing shenanigans, so it's hard to make it as efficient as Fortran
I don't know how this could be the case (especially with the addition of shared pointer) but you shouldn't be writing code with huge shared memory dependencies. Use constexpr, use inline, and use functions. There's no need for 20 functions to have shared memory access to a piece of data (which I have seen scientists do).
> I have seen a couple of projects that tried to port Fortran libraries to C++. They came out slower, buggier, and less readable.
That is on the programmers who wrote it, not the language's fault. There's no magical reason that Fortran should be faster then other programming languages. That's like me saying "Some of my friends tried to design a new car engine. Their car came out slower and buggier then my Ford Model T which still runs perfectly". It's anecdotal evidence that doesn't say anything about C++.
> The only reason you'd want to do that is a specific hatred for Fortran -- which you are not the only one to have
I don't have a hatred for Fortran. I just don't see any job where I'd prefer using Fortran over a more-sane alternative. Fortran was great for 77 but isn't great for 17.
> over the decades it hasn't proven to be a good motivation for redesigning software.
Comparability with modern software, improved performance from modern language features, pool of developers, further abstraction of your implementation to remove cruft from the code base using modern language features, and easier deployment are the reasons why we shouldn't be using Fortran.
I mean... when people use Fortran they usually don't even make use of Fortran 90's abstraction features. It's insane.
> Of course people like to use Python, but you don't get to use Python instead of Fortran. That would be incredibly slow. You use Python and Fortran. For example, you use NumPy and SciPy, which make Python a great high-level platform for numerical computing because they wrap large amounts of Fortran code.
You are mistaken. NumPy does not contain Fortran in it's code base. SciPy does but it only makes up 25% of the SciPy code base. I have no doubt that within the next 20-50 years that will slowly be removed bit by bit.
Wrapping a legacy library is the first step to replacing it. Once you remove the hard dependancy you will remove
> If you use Python for numerical computing, you are not "moving away from Fortran", you're embracing its success.
No you are definitely moving away from Fortran and towards better tools. The old cruft may still be there but as time moves it will slowly be replaced. You'd never need to use Fortran to get Fortran style speeds with modern software architecture using just python and C [2]. I've seen some crazy projects use this approach and outperform some native C applications.
[0] - https://julialang.org/benchmarks/ [1] - http://rust-num.github.io/num/num/index.html [2] - http://cython.org/
Fortran is not the same in 17 as it was in 77. They are different languages.
> I don't have a hatred for Fortran. I just don't see any job where I'd prefer using Fortran over a more-sane alternative. Fortran was great for 77 but isn't great for 17.
Others do. Plenty of scientific programmers consider Fortran to be a perfectly sane language.
> I mean... when people use Fortran they usually don't even make use of Fortran 90's abstraction features. It's insane.
False. My group does.
> > Of course people like to use Python, but you don't get to use Python instead of Fortran. That would be incredibly slow. You use Python and Fortran. For example, you use NumPy and SciPy, which make Python a great high-level platform for numerical computing because they wrap large amounts of Fortran code.
> You are mistaken. NumPy does not contain Fortran in it's code base. SciPy does but it only makes up 25% of the SciPy code base. I have no doubt that within the next 20-50 years that will slowly be removed bit by bit. Wrapping a legacy library is the first step to replacing it. Once you remove the hard dependancy you will remove
GP never said numpy contained Fortran in its code base, that was a strawman you added. Of course, that is what it means for numpy to provide a high-level platform for numerical computing: it means wrapping current code.
It's still rather important in SciPy.
I think, by number crunching, they may mean blas/lapack, which is nowhere near replaced by that numeric library...
>> C++ has pointer aliasing shenanigans, so it's hard to make it as efficient as Fortran
>I don't know how this could be the case (especially with the addition of shared pointer) but you shouldn't be writing code with huge shared memory dependencies. Use constexpr, use inline, and use functions. There's no need for 20 functions to have shared memory access to a piece of data (which I have seen scientists do).
Not sure you are understanding what is meant by pointer aliasing. Explicitly marking pointer arguments as 'restrict' in C can have a pretty dramatic speed benefit as the compiler can assume that the memory spaces do not overlap. C++ does not (strictly) have such a feature, unfortunately, and I don't see how shared_ptr/constexpr can fix this (although inline might in some cases)
> I'm not. I'm assuming these decisions are made because people in physics/ME/chemistry are not programmers and don't know about the latest and greatest our field has to offer. It's our fault for not advertising our successes to these people.
And how many FORTRAN HPC programmers do you know?
Also, it's not correct to say that C++ only just got a feature fortran had. It only just got the feature in it's standard library. There have existed complex number libraries for C++ for years.
Things like the GSL have been around since 1996.
People who grouse about Fortran have almost never used it for its intended purpose. It solves a problem, and does so very well. People who mention Julia, Python or Rust as replacements for Fortran simply do not know what they're talking about. You can do equivalent things in C++ or C, but you really shouldn't most of the time. Just as you shouldn't try to code up fancy data structures in Fortran.
I can understand having some of the other opinions you expressed, but this just seems egregious in the casual assumption that "the latest and greatest" is somehow universally applicable, or specifically applicable to scientific contexts.
Sometimes in some cases new isn't better.
In this case age has something to do with it. There are far fewer people who know fortran, there are far fewer people working on good fortran libraries, and there are languages that can do the same thing that fortran can do.
My issue is that fortran can do the same things (we can argue about speed because that's not really the language's fault) but it has a detraction that other languages don't have. No one outside of very limited subsets of the software development community uses it.
I see that as a huge detraction. Maybe others don't, but I do.
People in my field don't use PHP, either. Doesn't make it a bad language for its purposes, but certainly limited subset.
All the 20-somethings (including myself) in my research group use Fortran.
Well, sure, but there's nothing wrong with that, it doesn't mean the language is bad, and it certainly doesn't mean that it's wrong to use it.
Python has taken off in the scientific community because of the huge amount of code written by non-scientific programmers. Fortran will never benefit from this level of community support.
Scientists aren't interested in abstracting problems, improving libraries, or software organization but those things are important for having a productive, easy to use, and fast language.
Scientists are missing out on the benefits of community support for, in my opinion, no articulable or valid reason.
Some in this thread have said "It's because of number libraries" and others have said that "It's because of speed". I think I have shown, and that cursory research will show, that neither are the case.
Fortran offers nothing that C++, C, Rust, Julia, and Python have that can't be achieved using the respective language. What it does have is a small following who your work can only be appreciated by.
To me writing a program in Fortran is like an English major setting out to write a modern day English dictionary but choosing to write it in 17th century Celtic English.
You can definitely do it, it might be easier for an expert to do it this way, but it will never leave it's audience. Anyone who wants to use your work will need to rewrite it or painfully wrap it. You'll also have perpetual trouble finding people who can work on your project in the future.
Rust: does it even have arbitrarily-indexed multidimensional arrays, again? Are there good compilers? What's the performance on loop-heavy and cache-heavy code? (C at least has this covered). What are the floating point semantics and how do they affect accuracy and optimization opportunities?
Julia: now that's just a Fortran of the 21st century. In few decades bikeshedders will bitch how it doesn't support the latest C++42 features, completely disregarding that understanding C++, especially code written by others, requires pretty much devoting one's life to understanding C++. Not to mention that Julia looks visually different from JavaScript and doesn't even run in the browser. What's this early 21st century Celtic relict even doing here? Don't they get that developers want to use the Web nowadays?
Python: by your own standards it's worse than C because it lacks explicit inlining. QED. Never mind that C doesn't have explicit inlining either :p (hint: read docs).
We are currently preparing to develop and program new classes of algorithms (statistics) for use with (upcomming) Intel MIC processors. I recently succeded with implementing a customized synchronisation procedure based on atomics using Fortran 2008 coarrays. (In my opinion, customized synchronisation procedures allow to enter a new level in parallel programming). To make efficient and safe use of atomics, it was required to make use of several Fortran specific programming techniques, and tricks, that are all together hardly available in any of the other programming languages, at least not easily. (To be honest, Intel ifort does currently not support all of them neither -possibly a bug, I did request on this recently-). Also, to achieve our goal: C++ is too much OOP (runtime polymorphism) whereas C has too less support for objects to allow for highly efficient parallel programming (that's just my personal opinion).
That's not a correct statement anymore. There are a few examples of why this statment doesn't hold true but I think the biggest thing to point you to is this talk: https://www.youtube.com/watch?v=zBkNBP00wJE
With constexpr, const, and -O3 you're not going to beat C++ (or C) for any form of embeded-y systems development.
Also.... Phis are supported by OpenCL [0] [1] [2]. This lets you use a number of languages and libraries to better handle your computational tasks [3].
Why use Fortran, C++, or C when Haskell has Accelerate or when Python has ViennaCL [4].
Benifit from the computer science world's work. Don't write your own MPI code that you'll spend 4 months debugging when higher levels of abstraction exist for the problems you want to use. My best bet for you is: either use OpenCL's compute language which is very simple and allows you to do very complex operations completely in parallel on (cheap!) commodity hardware like GPUs for 4 years ago, or learn C++ just enough because the same things are doable using C++ and OpenCL libraries.
It's not too object oriented. Layers of abstraction don't slow code down if they are correctly developed. C++'s compilers and OpenCL have been correctly developed to give you frankly amazing performance for very high level features.
[0] - https://software.intel.com/en-us/iocl_tec_opg [1] - https://software.intel.com/en-us/blogs/2012/11/12/introducin... [2] - http://www.techenablement.com/portable-performance-opencl-in... [3] - http://www.iwocl.org/resources/opencl-libraries-and-toolkits... [4] - http://viennacl.sourceforge.net/
Nevertheless, our task is not just to write codes that can be executed on the MIC. We must efficiently handle the large number of MIC cores on upcoming HPC systems. What makes the MIC so special is: (a) an increasing number of cores in the near future and, more importantly, (b) we can execute all our existing, highly optimized (vectorized) library codes (C, Fortran 77, IMSL, NAG, or whatever) on the MIC cores. With that we may face two different main approaches for using such many-core hardware: Firstly, we may use massively increasing amounts of data to make better predictions (Big Data, weather forecasting, ...). And secondly, we may use the many cores to develop and program new algorithms to process small data in new ways: With an increasing number of MIC cores, evaluations based on ‘exact values’, as opposed to approximations, will become much more attractive. (Don't take the term ‘exact values’ to literally, in most cases these values are based on approximations as well). That is an important option in those situations where approximations are not available (which is the case with the more sophisticated statistical methods). With today's Fortran, or with PGAS in general, we make a development (away from threading) towards merely (remote) data transfer (through PGAS memory). This allows to write the most sophisticated parallel codes with simple sequential syntax style and brings the amount of required codes for parallel computing to a minimum. Sounds good so far? But the situation is not quite that simple: The (remote) data transfer must be synchronized, i.e. you can only consume the remotely transferred data after some synchronization point. The problem is, if the programmer wants to use the MIC cores efficiently, she/he will need to use some of the cores for distinct purposes and thus, execute distinct parallel codes with different synchronization points on them. This will almost certainly break with what is called ‘ordered execution segments’ (Fortran terminology). ‘Ordered execution segments’ are a very severe limitation for parallel code execution (more precisely: remote data transfer) that is not based on a restrictive programming language but due to some extreme limitation of upcoming HPC hardware. With unordered (user-defined) segment ordering the programer loses nearly all the remote communication channels among the involved cores. Then, the only way left to transmit values remotely are the atomic data types: (binary or integer) scalar values only. Until recently, I had not much hope to circumvent the limitations of such user-defined segment ordering using atomics and thus, not much hope to make efficient use of the MIC cores with current Fortran. Yet, we can not offer professional style programming techniques, but instead some simple, rudimentary Fortran techniques and ‘tricks’ to overcome the limitations of atomics: https://github.com/MichaelSiehl/Atomic_Subroutines--How_the_... Anyone who complains or doubts about Fortran: Try this with your favorite parallel programming language or tool. cheers.
Why worry about writing specialized vectorized code? Just by using OpenCL your code will work like that. That's what OpenCL does. You give it chunks of data, you give it a kernel to run, then you run it on your data.
An extremely old AMD cards you can easily find 1280 "core" systems that can push GFLOPS of data through them.
OpenCL is the tool you should use for this. Not a hodge podge of messy fortran code that's been ""vectorized"".
Don't do the compiler's job.
Fortran is fast and has no reason to be worse than C. In fact, it may be easier to write compiler-friendly code in F thanks to "restrict by default". And are there really HPC-grade C compilers with whatever "modern optimizations" that don't come with a Fortran frontend? Don't think so.
BTW, I have seen an attempt to rewrite a few dozen KLOC Fortran project to C and it wasn't pretty. The code was still a bunch of for loops crunching through a bunch of arrays, just in a different language, and then it had some new bugs that were introduced in translation and also it produced slightly different results because of different language rules, compiler and optimizations translating the same math formulas to different machine code. And the formulas were written to produce expected output when compiled with, you guessed it, Fortran.
Don't get me wrong, the old-school Fortran with 3-letter variable names, 72 character columns, GOTOs and stuff sucks donkey balls. But you don't have to write like that - while I'm no real Fortran expert (sorry), I've heard that modern standards of Fortran are vastly better and look much closer to C. I would certainly research this before rewriting everything in C.
Finding developers and maintainability. Those are two reasons.
I did have a quick hours training on how to operate the PDP 11 though.
I can understand legacy libraries. I cannot understand new projects and mission criticle systems.
For Fortran's use case you want to turn math into running code and all this C /C++ OOP stuff gets in way.
Also any language that doesn't have arrays built in is not going to get the time of day from a technical programmer.
You want the maximum performance out of your code so Python might be good for prototyping but you want to convert that to the fastest code in production.
Turning math into code is just as easy in both languages. The only addition that is different is adding types is mandatory in C++. You'd see more "float" or "long double"s in C/C++ code. No other difference.
> Also any language that doesn't have arrays built in is not going to get the time of day from a technical programmer.
Julia and Python have arrays built in, C++ has an array type, and C/C++ have pointers.
> You want the maximum performance out of your code so Python might be good for prototyping but you want to convert that to the fastest code in production.
With very little work you can turn prototype python code into production code with Cython. C speeds with python orchestration.
This leaves out the other languages on the block like Haskell and D which are better suited for other tasks.
Your whole post is just complaining about Fortran still beeing used, but you don't really explain why. The only point I can find is
> The best part: No documentation, no maintainer, and no migration plan.
which is in no way language specific.
Have you looked at modern Fortran? You have object orientation, polymophism and even parallel features like coarrays (I have to admit that I never really used them, because the gcc support is somehow incomplete).
Hint supersonic 1:1 wind tunnel time is very expensive so do you want to spent millions to which is why NASA uses CFD rewriting every thing in C++ would require a lot of back testing
Big frown over here
Just optimize our old code, and we'll pay peanuts! If you win, that is.
We were going to hire someone to shephard the old code, but why bother when you geeks will do it for nearly free because it's so cool!!!
OK I have my cynical hat on, and I can't really blame them for trying a cost-effective approach, but is this the future of work?
> is this the future of work?
Not just the future: many people have contributed to open source projects that have ended up being used in commercial projects, and never received any payment for their work.
I'll pass, but hopefully someone else can do this as a labor of love.
Yes, except you are not allowed to run the software on their cluster, instead you are expected to run it on your own machine.
Is it really from NASA? Are they lacking HPC programmers? Or any one knows HPC? Either their code is really bad such that optimizing on your home commodity computer is enough to get it run faster, or they are just making fun of it.
Even if the code gets much faster with their aging clusters, it wouldn't necessary be any better with skylake or knights landing.
I'm wondering if maybe NASA would be better off over the long term, if they ported their models to a more accessible platform. It sounds like they are having trouble hiring people who can work on their current stack.
The difference here is that this is a solver, not a library. So it is focused on a specific set of PDEs, by doing so you can apply optimizations that are specific to this PDEs and the space representation that it uses, this optimizations are not general to the underlying method of solution (which can actually be a combination of methods), so it will be extremely hard for this generic libraries to compete in performance with a focused solver.
Each generation has brought different optimisations, tweaked instruction sets, and performance characteristics. You'll have to optimise for the most common scenario.
> To catch integration errors, the suite of codes are repeatedly checked-out around the clock from our central repository, compiled, and several hundred regression and unit tests are run. Email and cell phone SMS provide instant notification of any errors.
also their us government tie... this is just a coded message for me that i am not welcome because i am too arabic. at least at first glance. probably its not but this is my experience of trying to do anything involving america in the last several years (i can't speak about the trump administration - but i doubt they are more arab friendly given the bad press).