The State of Fortran
arxiv.org
arxiv.org
Generics are necessary to bring performant data structures to Fortran, and yet they are nowhere near Go's generics.
For the usual naysayers doubting Fortran's place in a modern world: whenever you are reading or watching a weather forecast, that's decades of Fortran staring at you. Whenever you drive past a nuclear power plant, or an airbase with 'igloos', remember that nuclear safety codes run on Fortran.
True, the fact that it is still running says something, but not overmuch about whether it would be pleasant to maintain, extend or otherwise modify.
Put all the damn goto statements in there you want, as long as I don’t have to look at at, great!
Two out of three of those are controversial because of Spectre. https://en.wikipedia.org/wiki/Spectre_(security_vulnerabilit...
Just curious, how do you know that? I would have thought many establishments would have moved on by now. That said, I've a few of my personal programs that still run on IV - no need to change them as they still work fine (but then they're not running under the auspices of some institution where updating has relevance).
I suppose a more relevant question would be what is the breakdown type of institution still using 77 - educational, research, government, commercial etc. (I know airline bookings and traffic systems used a lot of 77 until recently). I note from another poster CERN is using 95.
I'd reckon legacy systems would be more likely to be found in govt. and commercial and updated more likely in research. (I'd expect there'd be a large disparity between the different types of institutions.)
50% of Fortran in numpy is 77 https://github.com/numpy/numpy/search?l=fortran&p=1
ARPACK, ATLAS, LAPACK, ODEPACK, SLATEC, UMFPACK are all notable Fortran libraries that are still fortran77.
A search of github suggest that by repository, there are roughly, 150 f77, 450 f90, 180 f95, and less than 230 for fortran 2003, 2008 and 2018 combined (fortran 2018 is especially dire there are only 6 repos with over 10 stars). however, it's worth noting that github is a site is biased towards newer versions since it only became a relatively dominant platform in the past 10 years or so.
All in all, I think my initial statement is probably wrong by 1 version. The majority of Fortran appears to be f90 or older, and only about 1/5th of fortran code was written on a version of the language from this millennium.
While I do work for JuliaComputing, my posts on HackerNews in no way reflect the views of the company. I'm not sure why you think paying software engineers to bash languages would be an efficient use of resources.
The way I see it, Fortran exists today because of inertia, not because it still has technical superiority.
These mountains of Fortran code represent decades of investment and undocumented "features" that you could never code around during a rewrite.
And so today, new code is bolted on, with a prayer that nothing breaks, may god have mercy on your soul.
College software engineering textbooks that explain why global memory is generally a terrible design choice were written off the lessons learned from legacy Fortran.
What you end up with is a language with a reputation for being faster for computation
But I can assure you, the vast majority of legacy Fortran is not vectorizable. At least not without a bit of refactoring.
Oh and did I mention, most of this legacy code was hand optimized for memory utilization. You see, back in the day 8k of memory was cutting edge HPC and about half that went to the OS and compiler. Well we all know, everything is a balancing act between time and space - the old timers traded time for space just to be able to do the computation at all on the hardware they had.
1) Nobody puts "restrict" everywhere, and in C it can be very dangerous to do so unless you're exceedingly careful.
2) The fact that few codebases use it means it's riddled with compiler bugs even if you are exceedingly careful. Just look at how many times Rust has had to disable noalias due to LLVM bugs and regressions.
Just because Fortran does the equivalent of implicitly using restrict everywhere does not mean the compiler actually statically checks for abuse. You can just as easily write bad pointer aliasing code in Fortran and depending on what optimization level you built with you may or may not experience memory corruption and/or segmentation fault at run time.
So the standard just says "don't do it, LOL" but then you are able to write Fortran violating this. Which is how you end up with bugs that go away when you switch optimization levels.
In fact, only C++ is able to confortably beat Fortran, despite not having official support for restrict, because of the stronger type system and compile type metaprogramming that offer optimization opportunities out of reach to C compilers.
Usually HPC places like CERN and Fermilab don't migrate their aginging Fortran code to C, rather C++.
Inertia is not a bad thing, but a long living code evolves in strange ways, because of us, humans. First of all, programming languages and software design has a symbiotic relationship. They feed each other. Support for required patterns and new technology is added to languages, and designs are made in confines of language capabilities.
Moreover, technology changes with every generation. Hardware change, capabilities change, requirements change, designs and languages change. More importantly mindset change. So you're bolting modern designs over "vintage" designs. Inevitable shims get coded (sometimes implicitly), and thing get complicated, even with best documented designs and developers with well intentions. The code pushed down solidifies, knowledge fades, and documentation get lost unless it's bundled with the code repository.
As a result, legacy code becomes something of a monster, where developers don't want to see, touch, or work with.
My research was coded with C++11 in the beginning. If I want to modernize with C++14 or add new parts with C++14, these parts will probably look so different that it'll look like two different languages glued together with black magic. It's the same thing with long living FORTRAN code. Only with longer legacy.
The culture and dynamics around scientific and academic programming is different from both FOSS and commercial software. This needs to be taken into account.
I'm currently teaching modern scientific Python programming, if that's a thing.
You raise an excellent point, the software development life cycle is quite different in academia. Do you have resources in mind on that topic?
I was thinking of dividing academic code in 3 "Ps":
- Playground: throw-away scripts for data analysis, fast moving code where any rigidity slows the process. Internal use by a single researcher usually.
- Prototype: code that serves as a foundation to the above, or is reused frequently, but is otherwise not shared outside a research group.
- Product: code, often open source, that is shared with the community.
Most tutorial about "good coding practices" do not make distinctions between these three stages.
For Python, there are practices that work at any level: auto-formatting code and sorting imports, gradual typing. But things like tests, API documentation, dependency management only apply to steps 2 and/or 3.
- Petrified: code that's set in stone and everyone's to scared to touch in because "it works, but God knows how".
In my experience, what happens is that some prototype works quite well and gets used in the next project. But the next project is building a new prototype, not improving the previous one. After a few projects, the person who wrote that first prototype already left the department, but there are people who depend on it to get results. If enough time passes, you will have some piece of legacy code that very few people can understand.
The problem with academia is that, while the prototype is an excellent topic for an article, the real product, a maintained (and maintainable) software project, is not. And getting funding in academia to do something of no academic interest (ie. that cannot be turned into a paper) is very difficult. If the software is of no interest to some industrial partner willing to spend money, that code will just keep moving between the USB drives of PhD students, probably branching into slightly different versions, until someone is brave enough to start working in a new prototype and the cycle starts again.
I understand how code bases change and warp but it really is a push and pull for when you should abandon something vs. keeping it around. Moreover, the other alternative, learning to actually use old code and understand it in the mindset it was developed avoids the frankenstein-ization you refer to becase if people actually understood the old code, they can add to it in a way that meshes well with the existing code rather than it being a bolt-on. That said, I can understand if you inherit something that already has the bolt-ons such that you're not really responsible for that, and that can be harry, but really I don't really feel like that is something unique to computational science in the abstract. Bolt-ons are common across CS I feel like.
The main thing I am railing against and have been doing for a long time is the tendency for developers to have more on an emphasis on writing code as opposed to reading code that already works. In particular, taking time to understand so-called legacy code, learning to think in the way it was written, then modifying said code in a way idiosyncratic to it. Unironically, we focus way too much on creativity in CS. That sounds a bit funny but the fact it does already demonstrates the reality of that mindset's (ironically) strangehold on software in general. It's really funny because creativity is actually not that important for the majority of users of computers but is very much valued by developers in general because they develop computers for a living. On the other hand, something that works and is stable is actually something people don't even know they want but even better (or worse), they rely on or at least grow accustomed to given they bitch and moan once the familiar is broken often to fill some need of some developer to chase the new and shiny.
This is a long comment, but there is one last thing I'll touch on: one of the places where I do agree somewhat is new technology. For example, for people in modeling laser-plasmas (where I hail), people have still not really adopted GPUs even though that's all the rage (and has been for years already actually), because the main tools (PIC and MHD) do not map well to GPUs and the algorithms were developed with a large memory space accessible across a node assumed. There are efforts being made now but it's still considered hot shit for the most part. So, there is one place I'll grant you that it does require some willingness to "write more" so to speak, to be able to take advantage of new technologies. That said, "writing more" in this case still requires rather deep knowledge of the old codes and particularly why they made the choices they did in order to save yourself a few years of recreating the same mistakes they did (which btw, is literally what I see whenever people do attempt that sort of thing today).
My first language was Fortran. Whilst I've some lingering affection for the language and that I still use it occasionally I also use other languages, Lisp for instance. It's significantly different to Fortran so I reckon my comments below come from having a wider experience than just Fortran alone.
First, there are things about Fortran that I don't like and have never liked the main culprit being the obtuse formatting and I/O (I'd have altered it if I could!). I'm not alone, one of the reasons Kemeny and Kurtz invented BASIC was that with BASIC all you have to do to print is to use a simple 'Print' statement. I also agree that Fortran hasn't fully kept up with the times but that's understandable given its large and significant role in science and engineering (I shouldn't need to explain that). At least it's lost some of its more egregious points—for instance, the infamous 'GOTO' instruction has long been kaput (and so on).
However, criticizing Fortran isn't the thrust of my argument here, which is the ongoing need to use existing Fortran programs without the need to change them. There are many, many thousands of programs—mathematical and scientific subroutines, etc.—that have been developed over the past 70 or so years which have been shown to be debugged and highly reliable by the fact that they have been and are still used repeatedly to good effect in critical industries such as nuclear, space and civil engineering just to mention a few.
Translating and or rewriting these routines into more modern languages risks introducing errors and bugs, and whilst modern programmers would likely find this ascetically pleasing that ought to be their secondary consideration—the primary one of course being the correct operation of the program they're working on. Even if all that coding were translated successfully, we'd likely still have issues with different compilers interpreting the translated code in subtly different ways to the original Fortran ones, [as I mention below, authentication of the translated code and similarly the new compilers to meet necessary standards alone would be a nightmare].
In short, we cannot guarantee the translated routines will do exactly the same thing or behave in exactly the same way as they did under a native Fortran compiler—at least not without one hell of an effort!
As mentioned, the key issue for keeping this huge library of Fortran routines operational is that it is huge—truly vast (nearly 70 years is tied up in developing Fortran code/libraries across a multitude of disparate industries and endeavors). Moreover, much of it was programmed by scientists and engineers who had a different attitude to today's programmers in that their primary work involved working and rearranging atoms which is much harder than just recompiling the source upon discovery of an error.
In short, engineers and scientist were used to designing hardware that worked properly and reliably the first time—bridges falling down and nuclear reactors melting down the moment they are commissioned would have serious consequences that today's programmers will never experience; often, the first they know of a failure mode in their code is when it comes in from the field after it was supposedly deemed to work. Thus, this 'work-first-time' exactness attitude spilled over into early Fortran programs and has been the mainstay for much of the code written in the language since (it's a significant reason why so many of these programs have been so reliable). Remember, John Backus, Fortran's chief instigator, was trained as an engineer.
Ideally, it would be nice to see new intelligent compilers that could accurately compile all of Fortran's variants as well as other languages, Algol, COBOL, Ada, PL/I, etc. including C etc. in one package as that would allow easy reuse of old 'solid' code. Of course, this will never happen for a multitude of reasons, the first of which is that it would take years and years to fully standardize and authenticate any such compiler to full ANSI/ISO standards comparable to past Fortran ones (especially so when multiple languages are involved). Then there's the problem of who would do the programming (I'd reckon most of today's programmers would hate such work—they'd see it as much worse than writing 'hated' hardware drivers not to mention the need for them to maintain standards and endure rigid discipline across the length of the project—and that could be for quite some years.)
That brings us back to where we started. As noobermin said, old software isn't broken as is so often assumed, moreover for reasons stated, I contend that history has demonstrated that it's often much more reliable than much of today's new code.
For these reasons, I reckon Fortran is going to be around for much longer than many of us care to imagine. History has shown it has staying power whether we like it or not.
I say that as someone who has worked in nuclear safeguards.
In essence, that means the simplest approach is to use dedicated and or standardized systems (i.e.: in measurement/accounting) wherever possible then there's little or no argument when the inspectorate arrives. (From my comment you'll know my background.)
Jesus wept!
-Signed, a physicist with 20 years LabView and Fortran experience.
Part of the reason Julia can do this is due to support for parametrized types and JIT compilation which I can make it possible to in-line a lot of code which you cannot inline with AOT compilation.
[1] Numeric age for D: Mir GLAS is faster than OpenBLAS and Eigen:
http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/...
Same thing for ADA.
However, my understanding is that its main focus is on array computations. Now with hierarchical data structures and graphs becoming ever pervasive, I wonder if Fortran will ever try to compete in this space (when these higher-order concepts are integrated into domains of Fortran's past dominance in numerical simulation).
From that point of view Fortran may be even more relevant for high-performance computing now than even 10 years ago.
It is evolving towards a great language for parallel HPC, integrating neatly both shared-memory and distributed computing in a very nice framework. Much better than the usual hodgepodge of OpenMP and MPI (or whatever API du jour NVIDIA or Intel happen to be pushing).
But the quick answer is that people programming in Fortran use the usual editors - Visual Studio Code, Emacs, Vim, CLion, etc.
There are also a couple of Fortran REPLs out there, the most promising is probably LFortran which is only in alpha stage of development but has Jupyter integration and will be the next cool thing I think.
"With proper use of overlays, it is possible run the system on a minicomputer with only 32K bytes of memory." -- Cleve Moler
Fred Weigel
Microsoft BASIC-80 (around the same time) was written in assembler. You also don't write in assembler when using BASIC-80. But, BASIC-80 would be extended by assembler, in the same way that MATLAB would be extended by FORTRAN.
Fortran was absolutely run on punch cards, which also informs some of the strangeness of its formatting, especially pre-fortran-90.
https://upload.wikimedia.org/wikipedia/commons/1/18/FortranC...
Print that in color on 8½x14" (legal size) paper and get to work!
Of course there were a few other steps in the programming process back then.
1. A Systems Analyst took business requirements and turned them into an overview diagram of the system structure.
2. A Programmer took those diagrams and drew detailed flowcharts. These included every "if" statement, every loop, every detail. They were in effect code, but in a diagram form. Don't forget your IBM Flowcharting Template! https://ids.si.edu/ids/deliveryService?id=NMAH-AHB2012q05389
3. A Coder read the programmer's flowcharts and wrote out the equivalent FORTRAN code on multiple pages of Coding Forms.
4. A Keypunch Operator read the Coder's forms and punched them onto cards, one card for each line of a Coding Form.
5. Finally we get to the High Priest of this operation. A Computer Operator - the only person allowed to touch the Computer! - ran your deck of cards through the machine and blessed you with a printout. Which was usually a core dump.
Check your debug output, modify (or insert) a few cards and repeat.
Don't forget the JCL (job control language) on the first card. That is how you tell the mainframe what resources your code needs and how to run it. If you get that wrong nothing happens.
We were taught to draw a diagonal line across the top edge of the deck of cards so you could quickly put the deck back together correctly if it was dropped.
It has many of the usual IDE expectations built in like autocomplete, project management, debugging, Git integration, etc. Our Windows distribution adds a few niceties like a Fortran-oriented (though basic) GUI library, a Windows-native coarray implementation (that doesn't use MPI), Windows-native OpenMP and threading, and a binary package manager for pre-built libraries.
We're decidedly a small player in a niche market, but I like to think Simply Fortran makes life easier for Fortran developers. Sorry for the self-promotion!
Am.. am I about to teach myself Fortran?
Traditionally, these things were published in ACM SIGPLAN Fortran Forum (or on NAG’s servers for historical reasons). Arxiv is really natural, given the direction scientific and technical publishing is taking. This is a heavy duty language used by scientists and people like mechanical and nuclear engineers. Not the latest JavaScript framework or full stack doodad from Google.
For another take, Jack Dongarra was just awarded 2021 ACM Turing prize. He’s the creative force behind LINPACK and BLAS, and says that Julia is a good candidate for the “next computing paradigm”: https://www.zdnet.com/article/jack-dongarra-who-made-superco...
> I sometimes wish [Julia] had just appeared one day as version 1, and made periodic updates every year or three.
> Its “multi-paradigm” nature. Yeah, I know other languages have this too, but it does tend to bloat a language.
And some are just funny:
> The language is not stable, and by that I mean it seems to have a lot of bugs [links to github issues with label:bug]. You can’t really call something that has bugs stable.
No, that is not a common thing post Julia v1.0.
A lot of newer libraries are not well documented in Julia, but the core stuff is much better documented than Python IMHO. So his view on Julia documentation lacks nuance. It is both good and bad depending on what you look at.
The post talks about the complexity if importing stuff with using and include. Yes, if you start from scratch it is more complex, but the benefit is that it scales better. I have helped compete beginners with Julia who had totally given up on Python due to the complexity of package management, modules and environments. Julia has a beautiful solution to this which is far easier to get right and manage.
A novice will find it easier to import packages they want to use with Julia compared to Python. But very often people evaluating this stuff are experienced Python developers and they don’t see the Python hassle easily. If you take somebody who is a beginner in both languages then experienced will be quite different.
The “multi paradigm” comment makes zero sense. Sure Julia is multi paradigm but it is far more streamlined than the common OOP-functional mess you see in other mainstream languages today such as Python , Java, C#, Swift.
In Julia it is all functions. You don’t distinguish between free functions and methods. For a beginner this is much easier to deal with.
Nothing overstated; just the circumstance in the real world that these four languages are the only ones used in the most demanding HPC.
C and Fortran are also not as fast as they could be if you use them incorrectly. Using concrete types is not “significant tweaking”.
There the "naive" Julia code, simply implementing the code like I would in Fortran is a factor of 10 or 15 slower than the optimised cython version (which would be the same as a regular C version), the optimised Julia version is still a factor of 5 slower than the cython and pythran version. Can you show me how to optimise it so that Julia performs on par with pythran or cython?
apply_filter(x, y) = vec(y)' \* vec(x)
function cma!(wxy, E, mu, R, os, ntaps)
L, pols = size(E)
N = (L ÷ os ÷ ntaps - 1) \* ntaps # ÷ or div are integer division
err = similar(E) # allocate array without initializing its values
@inbounds for k in axes(E, 2) # avoid assuming 1-based arrays. Just need a single inbounds macro call
@views for i in 1:N # everything in this block is a view
X = E[i*os-1:i*os+ntaps-2, :]
Xest = apply_filter(X, wxy[:,:, k])
err[i,k] = (R - abs2(Xest)) \* Xest # abs2 avoids needless extra work
wxy[:,:,k] .+= (mu \* conj(err[i,k])) .\* X # remember the dots!
end
end
return wxy, err # note order of returns, seems more idiomatic
endEither way, there's still often at least some native code you have to write and wrap. That's where a professional programmer might get involved. But probably more often than not, it's the engineers and scientists themselves.
[edit] Just to be clear, I don't think Fortran has these.
It's an easy language to get to grips with, but the users somewhat less so. At one point doing consultancy, I came across people declaring variables like I1,I2,I3,I4 to IN where N was a large number.
I asked "why?" and they said "we might need them" - Fortran programmers will see at least two reasons why this is daft.