Hacker News Reader

Comment by thicknavyrain | Hacker News Reader

"Professors usually have this legacy code on hand (often code they wrote themselves decades ago) and pass this code on to their students. This saves their students time, and also takes uncertainty out of the debugging process."

This is so true. I'm a PhD student in physics using Fortran for pretty much that reason. At the start of my PhD, in response to my supervisor telling me I should learn Fortran to modify our current codebase, I asked if I could rewrite what I'd be working on into C++ first, since I was already familiar with it and wanted to bring future development into a more "modern" language.

His response was "You could do that and it would probably be enough to earn your PhD, since it'll take you at least three years. But I suspect you'll want to work on something else during that time".

He was right. I later learnt one of our "rival groups" attempted the same thing and it took three phd students working full time for a year to rewrite their code from fortran to C++.

View on HN

"Within a month of his arrival, Randy solved some trivial computer problems for one of the other grad students. A week later, the chairman of the astronomy department called him over and said, “So, you’re the UNIX guru.”

"At the time, Randy was still stupid enough to be flattered by this attention, when he should have recognized them as bone-chilling words. Three years later, he left the Astronomy Department without a degree, and with nothing to show for his labors except six hundred dollars in his bank account and a staggeringly comprehensive knowledge of UNIX."

– Neal Stephenson, _Cryptonomicon_ (p. 78). HarperCollins.

I am unfamiliar with the passage.

But it describes two options:

1. "A staggeringly comprehensive knowledge of UNIX", three years of domain-specific education, and a network of people who trust you to--no, who depend on you to be able to get things done with that knowledge

2. A piece of paper just like everyone else in the department.

And while the former might be more difficult to make use of, I think it could be much more valuable in the long run.

No, he left without a degree.

A good point, but the text suggests there are ways to get this knowledge.

  >> except six hundred dollars in his bank account

  >> are ways to get this knowledge.

Edit: are ways 'better' to get this knowledge.

More like two options:

2. A piece of paper just like everyone else in the department, three years of domain-specific education, and a network of people who trust you to--no, who depend on you to be able to get things done with that knowledge (and probably a girlfriend)

In context, he was underpaid tech support who was taken advantage of and got no grad school education while he paid for grad school. A bunch of scientists who know you as "that guy who fixes my email and doesn't publish or know anything" is not a great network.

> three years of domain-specific education, and a network of people who trust you to--no, who depend on you to be able to get things done with that knowledge

This might be the case, but he's now like the janitor who unplugs the toilet. His work may be appreciated, and it might be necessary, but it's not respected or well remunerated. I guess he gets a little bit of respect for unplugging more theoretical pipes, but a little respect isn't a PhD.

Don't forget his girlfriend.

To paraphrase: "Usually a comprehensive knowledge of UNIX and a girlfriend are mutually exclusive."

Not useful if he wanted to be an astronomer, but a much more marketable set of skills if his goal was to find a job in industry. Probably also made a more positive (albeit uncredited/unrewarded) contribution to astronomy than most of the grad students.

Trusting legacy code with few users is a dangerous proposition. My roommate was given some "state-of-the-art" code and told to run simulations with it. The only graphical output was postscript (for some reason), so every frame was 150 MiB and took minutes to dump - so usually, this was only done at the end to show the result.

I managed to hack in a step which just dumped the memory of resulting frame to a file, and then we wrote a Python script to read that and produces a PNG.

We combine the PNGs into an animation and show someone else in the department because the supervisor wasn't in that day. "Cool! But, hrmm, those boundary conditions look wrong." Sadly, this was 2/3s into his Masters.

Like much now in Physics, the original code was simply incorrect. This happens all the time, and papers get retracted because of it - well, that's the best case if somebody notices.

At the end of the day, programming languages come with eco-systems, and must be chosen solely as a tool. The problem is not with Fortran, but that often Fortran == outdated development practices and in Physics horrible code hacked on by 10+ people without prior programming experience.

It's going to have few users anyways, since it's so specialized. And that's also why it will be legacy.

Another problem is that most people who wrote this code aren't programmers - they don't write clean code, no tests, etc. They don't really know those are important. Sometimes the code that is used and updated for years looks like a dirty prototype.

I don't know what can be done about it except hiring programmers to write code, which wouldn't be either easy or cheap

Well, I guess one moral of the story is that doing image manipulation is easier in Python than Fortran, so it's again using the right tool for the job. And I think the push for Physicists to use Python if possible is good, as the learning curve is less steep. And once you have that tool at your disposal, you might use it more often (instead of Excel).

I can see many projects being improved by providing a Python pre-processor that writes out e.g. a binary config file, the hard-code Fortran/C simulation code reads that, and spits out the simulation results, and then having a Python post-processor that does the pretty stuff at the end.

Academically, it seems that pairing CS undergrads with Physics undergrads to do e.g. a molecular dynamics (MD) course would be cool. The Physics behind MD isn't too hard, and given the right parameters the programming part would be manageable. Then again, CS undergrads aren't necessarily great programmers either...

Clean code and tests are overrated. Version control and a big eco-system is underrated. It's like maths, physicists don't understand maths, they just use it like a carpenter uses a nail without understanding metallurgy, or a programmer uses a CPU without understanding solid-state physics. And that's okay.

Why not using both ? You can reuse existing fortran code withing Python, and benefit from the prior experience and speed while enjoying the qualities of Python.

There are several ways to bind Python and fortran together. E.G http://arogozhnikov.github.io/2015/11/29/using-fortran-from-...

If you don't have clean code and tests, vsc would allow you to switch between old and new bad, probably non working code. That's nice, but probably not what you'd want. If I'd need to choose one out of all, I'd choose tests. From my experience, it's nearly impossible to implement big system without tests. Maybe I'm just not good enough as a programmer

I would have thought a more natural candidate for scientists would be languages like Java or C# rather than python: you get both a language that is relatively easy to write, with no memory to manage, no hornet nest of pointers on pointers, nice debuggers, lots of 3rd party libraries, while a the same time getting the performance of static typing / running compiled code once the program has been JIT (which for long running code is a negligible cost). It's never going to be as fast as a C++ or Fortran, but you get a compromise between performance and ease of use, while I understand python is going the other extreme, i.e. all ease of use at the expense of performance. Doesn't matter for a simple simulation, but for some large data analysis projects, I assume it could make a big difference.

But that doesn't seem the case. The choice seems to be between python and c++/fortran. Does anyone know why?

1. Nice bindings already exist for numerical libraries in python. It is more accurate in some ways to say that scientists are using a domain specific language based on numpy than they are using python.

2. The choice is either "fastest possible" or "I just don't care how long it takes" - for either development or run time. There is usually no middle ground.

3. Tooling - the scientist is most likely using a text editor not an IDE (especially when they start working). Fortran and python are both low enough on boilerplate to not need IDE support.

4. Abstraction level. Scientists in general don't bother care about abstraction level at all. Procedural programing's abstractions are usually more than enough for them (and may actually be the correct level for some numerical work - think cache misses vs. hits).

5. Some areas of science do use Java (check out imagej for example).

They where not back checking against real word experiments?

I've heard horror stories of supposedly quite serious software modelling programs behaving quite differently when their input problem was rotated a few degrees.

Ugh - I remember spending a lot of time checking our pump simulator module against real word experiments back when I worked in technical computing

I've heard horror stories of supposedly quite serious software modelling programs behaving quite differently when their input problem was rotated a few degrees.

Sounds perfectly normal :)

The standard advice here at work for one of our modelling packages is: if it crashes try rotating your input model by one degree and try again. Most of the time that will fix it. If it doesn't fix it try one degree in the other direction.

Do people publish work based on this thing?

Is correctness somehow assured provided it doesn't crash? Or is the whole thing just a joke?

Do people publish work based on this thing?

I'm sure lots of people do. We're a commercial company selling consultancy service around this software (technically around some inhouse tools built on top of this software), among many other things.

Is correctness somehow assured provided it doesn't crash?

On the whole well tested PDE solvers either crash, give answers that are off by several orders of magnitude, or give a correct answer. (If the answer they give is relevant to what you're trying to model is left as an exercise to the reader)

We're reasonably sure that, if the calculations converge, the solution provided is a correct numeric approximation (within the error bounds given) of the PDEs we're claiming to solve. We also believe that the PDEs we're using provide a reasonable balance between modeling what we claim we're modelling and our computation running in a reasonable amount of time.

They probably did at some point in the misty past. The problem is that running relevant experiments is often very expensive (otherwise they wouldn't need to do it in software) and they can only test a tiny handful of cases. More often than not your calibrating your model against another model that was calibrated against another model that was calibrated against a set of simple experiments done in the 60's.

Yep. Most of my programming classes were either in pascal or c when I was in undergrad, but in my physics courses, all the way through my graduate education, I used FORTRAN. It's because physics doesn't change that much over the years, FORTRAN code that is time-tested and that works exists and there's no need to re-invent the wheel in another language when more interesting and important problems exist to solve.

David Baker's Rosetta code, which made him a pile of money, is IMO a spectacular example of craptastic C++ written by people who really didn't understand C++ but didn't let that deter them from using every single feature of the language, badly. Some years back we tried to port it to CUDA but there were so many levels of indirection, dereferencing, and virtual functions that it was nearly impossible to make any progress.

In contrast, porting the 30+ year old molecular Dynamics package AMBER to CUDA took about 3 months and probably established me as a CUDA expert. In my opinion its well-maintained Fortran 90 code was far easier to understand and refactor.

While my primary languages today are C++ and CUDA, there is something clean about Fortran when it comes to understanding underlying algorithms. I have a similar opinion about well written C code.

That's the whole trick. Good code makes complex stuff look simple. Bad code makes simple stuff look complex.

That quote should be the metric by which code/languages are judged.

How do you measure the complexity of the code, and most importantly, of the problem that the code is solving?

Complexity analysis.

I wonder if you can formalize this. Assume TDD, and then measure the compressed size of the codebase vs. the compressed size of the test suite. Code is "worse" when it requires more "stuff" (informational entropy) to do less "stuff" (pass test cases.)

The compression would presumably remove the redundancy of the language itself as a factor (including differences in idiomatic cyclomatic-complexity "depths" of various stdlibs), and also remove any redundancy in the way the test cases were specified. So it'd be down to a measure of how much circumlocution and over-engineering you did in the process of implementing the solution.

I'd worry slightly that code-golf solutions would be rewarded, though. Maybe pass everything through an obfuscator + linter before computing the metric, so that things like identifier lengths and spaces aren't considered.

> using every single feature of the language

I constantly run into code where someone takes an obscure corner case in the language, and builds an entire edifice around it.

> which made him a pile of money

May I ask how? I don't see ads or any other obvious monetization channels.

Rosetta is a protein folding bit of software: https://els.comotion.uw.edu/express_license_technologies/ros...

GP is not talking about http://rosettacode.org/wiki/Rosetta_Code

Its a product of the UW, so no surprise the quality is crap. IMO their Cybersecurity course (which is mandatory for CS students) is the biggest load of shit I've ever seen in a class.

The head of information security at the UW calls the PSTN bulletproof secure, but VOIP insecure, then babbles on about how he shares classified info with his buddies casually (happens to be a felony to share said info).

Needless to say, UW's Avaya phone system barebacks the internet, with no regard for using silly security things like TLS or SRTP.

Oh, you're that Scott Grand? Allow me to thank you for the work you did on pmemd.cuda, my old research group wouldn't be where they are now without you.

You're welcome, it was ironically based on ideas drawn from a C Port of the Amber potential function I wrote back in grad school. While I'm not proud of the code these days, I put it on sourceforge a long time ago.

https://sourceforge.net/projects/folderol/

Why write the port in the first place? Because back in the days of yore, I was doing the equivalent of adversarial search to try and design a better potential function for predicting protein tertiary structure. I ultimately arrived at the result that there were too many adversaries to make linear models and single hidden layer neural networks work.

And unfortunately, my postdoctoral advisor at the time didn't consider this publishable research.

Maybe it's simply the fact that the C++ code was object-oriented?

OO allows you to make a much worse mess than procedural programming in my experience.

Now you need to do a follow on study to see how much science the 'rival' group does with a more modern codebase than your 'legacy' group does. Would you know if there are enough examples of two groups who have diverged like this to get meaningful (as in statistically significant) results on the cost benefit of porting / not porting?

> Now you need to do a follow on study to see how much science the 'rival' group does with a more modern codebase than your 'legacy' group does.

I would guess that a C++ codebase written by PhD students, not by seasoned C++ experts, is more complicated and much slower to debug, than a corresponding Fortran codebase.

Well, that Fortran codebase probably also was written in large parts by PhD students, not export Fortran programmers.

C++ is a more efficient language to write difficult to track down bugs in than Fortran.

Fortran is a much simpler and safer language than C++. And faster to learn. Fortran is even somewhat simpler than C or Java, whereas C++ is probably the most complicated language in the known universe.

So especially in the hands of non-experts, Fortran should produce less bugs.

Funny as hell. Perl, though.

C++ is complex. Perl is complicated.

They are both typically pretty bad - but there are many more ways to shoot yourself in the foot in c++

Can verify that if you give numerical methods people C you may get wonders like (e.g.) main() functions that are 30 pages long.

No, I'm not kidding.

I shudder to think what they'd have done with C++. :-)

Don't be so assumptive that a 30 page long main function is bad code. The code still can be well organized and readable. Function is not the only method to organize code. Knuth's literate programming for example is invented to better organize code.

Function has its side effects. It's use as only once case is really questionable, justified by lack of other means.

The very fact that the article itself, with a completely straight face, has all that malloc nonsense in there as "the way you have to do it in C++" is very telling of how easy it is to get the wrong end of the stick with C++ unless you are au fait with the history of the language and its evolution to better more modern idioms

I would also really want to see a comparison of bugs and error rate, although that's probably a nightmare to try and work out.

> Now you need to do a follow on study to see how much science the 'rival' group does with a more modern codebase than your 'legacy' group does. Would you know if there are enough examples of two groups who have diverged like this to get meaningful (as in statistically significant) results on the cost benefit of porting / not porting?

I don't know of any studies like this. But there are two aspects to consider. One is what you've pointed out, which is the long-term scientific gains and the productivity of the research group as an entity. But the other aspect is whether those students who rewrote the code into C++ were more or less successful than the students who weren't rewriting code.

Obviously, how you judge success in the latter will be tricky, since there may be differences in the interest of students who rewrote the code and their desired outcome after the PhD (e.g., stay in academia versus go into industry).

That is a good point, if the majority of the code is living in Fortran then there is definitely some value in understanding large legacy Fortran code bases.

This is true. Though I realized I wasn't specific enough in what I meant. I was more thinking about the limited available time to students and that rewriting code might mean the students do less science and so are less able to obtain a good postdoctoral position (or whatever position typically follows a PhD in the particular field).

True, but the students that re-implement these codebases also might have a much greater understanding of the underlying techniques, theory, and the interaction between them; compared to others who produce more results, but use these codebases as black boxes. I think both pathways lead to equally good academic careers, they just branch out into separate paths.

His response was "You could do that and it would probably be enough to earn your PhD, since it'll take you at least three years. But I suspect you'll want to work on something else during that time".

as someone who had contact with that codebase, do you have any insights as to why that is?

Was it the sheer size of the thing? Was it some nuance that Fortran had as an advantage over other languages? was the math just difficult to follow?

Genuine curiousity.

Pretty much just size! There was about 20,000 lines of it, it's legacy code that has been gradually added to since the 1980s, so it would have taken a while to rewrite all the parts to work with each other. Perhaps some day though.

What's the code do? [physics grad level description would be fine]

It must surely be generic equipment management, you're not running experiments in physics that are > 30 years old, surely?

Started in the 80s doesn't mean the code is 30 years old, as they keep adding to it and modifying it in Fortran. Plus most of the analysis part is probably just math routines that don't need modification ever after.

Doesn't mean it isn't c.37 years old either ... I think we need specifics to progress this conversation usefully.

Only 20k I have worked on bigger Fortran systems doing billing systems for a telco oddly enough.

20,000 lines of code really doesn't seem like that much. I probably output that much in about 2-3 months of biomedical research so there has to be more to it than that.

20,000 lines of code casually churned out in 2-3 months will be a nightmare to a) understand, b) maintain, much less to rewrite...

It sounds like he’s talking about scripts generated for research, rather than code for components intended to be persistent.

Precisely this.

>> legacy code that has been gradually added to since the 1980s

> 20,000 lines of code really doesn't seem like that much.

It's not "20k lines of code" but "n lines of code that grew organically over decades to 20k lines" - with the help of probably way more than 100 people who all are not trained as developers. I think it's a safe bet to say the current state only has a faint memory of being a consistent code base.

> 20,000 lines of code really doesn't seem like that much.

Once I spent close to a month hunting down a subtle but nasty bug in a number crunching module which was perhaps around 2k LoC. Writing code is very easy. Verifying and validating number crunching code is very hard and very time consuming.

20k of fortran code is a lot. there are no beans/interfaces/adapters/mediators/listeners - just the code that does business.

20,000 lines of your code is one thing. 20,000 lines of someone else's code is another thing entirely. 20,000 lines of many other people's code can be insane.

I have reimplemented all Fortran code of my professor at CERN in C during several undergrad classes during my Physics major. Then I have quited physics and became a developer.

I'm taking (educated) guess, because people involved estimating rewrite where all physicists and not developers. And certainly not developers with 5+ years of experience. Which is minimum I'd trust with archtechting major rewrite.

To do such a rewrite one needs domain knowledge much more than you need software engineering knowledge. This means you need to get research physicist to contribute major effort into such a rewrite. To get there one needs to make them see a clear benefit of translating a system (that works fine today, thank you), from language A to language B.

A Ph.D. student or a young researcher might be more interested if a rewrite lets them run experiments using Amazon / CUDA / whatever for a few thousand dollars instead of spending scarce grant money on dedicated hardware.

They are talking about large scale massively parallel code to run on a super computer. Most grad students in physics that need it have have access to a system much faster then Amazon. For example my friends code runs on Cori @ NERSC which is a 622,336 core Cray.

http://www.nersc.gov/systems/cori/

I completely agree on the domain knowledge statement. It is 100% required more then the CS degree.

Yes, C++ does not buy you that much. Sure, it is a more modern language, but when used by scientists (not software engineers) the advantages are hardly earth shattering. Much of complexity is hidden (as it should be) into libraries.

IMO, Python stands a better chance of breaking the Fortran's lock on physics related computing. Give it a few more years and enough numpy-based libraries might make Python a real competitor. My 2c

I have to be honest with you, as someone who writes in both C++ and Python, I really do not see Python being more of a candidate than C++ for displacing Fortran.

Can you clarify why you think Python might be able to do it? For scientific computation with high performance requirements, Python is not competitive with Fortan or C++. For work that continues to happen in Fortran due to "academic inertia", my impression (and vague experience) is that researchers find the convenience of what they're accustomed to (Fortran) to be greater than the convenience of things like rapid prototyping offered by the various Python scientific computing and data analysis libraries. There is a mental overhead is switching, and I think most academics are sort of okay writing code in whatever is familiar and battle-tested if it means they can focus more on the research at hand.

In other words, I'm not saying you're wrong, but I'm not following your reasoning.

In a previous discussion of this kind (I'm not in this area myself), a scientist had said that the speed of the languages is less important than iterating the algorithms, and it was a lot easier for non-developer scientists to work with python than with C++.

I worked as a programmer in a molecular dynamics research group for a while. I was asked to work in Python because it was what the boss was familiar with - so there's some of the same inertia happening again, just with a new(er) language, I guess.

Speed is not a huge issue if you're happy to leave your simulation running overnight anyway, or if you have the option to just throw more and more cores at the problem (or in our case, both). The goal is to get your papers published. Code developing/interpreting/debugging time is generally far more serious an obstacle to that goal than simulation speed. I was the only developer there. Everyone else in the group coded on an almost daily basis, but none of them considered themselves programmers, and very few of them actually learn about good programming practices. They're biophysics researchers.

Having not worked in Python much before I was also pretty pleased to find that, having determined that I needed to use Dijkstra's algorithm for path-finding and then working out the smallest Standard Deviation between certain sets of data points, Python came with libraries to do both of those things off the cuff. It's just so easy, I can see why it has a favoured place in this field.

As long as Fortran is able to keep up with Python's capabilities (or any other language's for that matter), there seems to be absolutely no reason to make a change considering Fortran's history and familiarity in Physics. In other words, other languages need to make a big enough leap forward or Fortran to lag behind enough to justify a change. Your example just shows that it's fine to use other languages, and I agree, but it's not very compelling in promoting a change in language.

> Speed is not a huge issue if you're happy to leave your simulation running overnight anyway, or if you have the option to just throw more and more cores at the problem

What about those cases that it's the matter between throwing at 1000 core cluster and wether we can have results before next conference in a couple months. That is what really defines the scientific programming -- it's about feasible and infeasible. For other jobs, isn't it just matter of taste?

Depends not all scientific programming is at universities

when I worked for BHR group (a Hydro dynamics research organisation) some times we had emergency projects that had < 24 hour turn around. I recall one where a clinet had had a serious issue at a plant and we ran the simulation and produced a report in a single day

Just because python is slow doesn't necessarily mean that every library is also slow. With numpy/scipy, which are written in C with python bindings, scientific computation is pretty fast. Python is fast becoming standard language for data science with so many tools like notebooks that make it extremely easy to do many things. I am not sure how well it compares to Fortran but it is no brainer to use it over C++, especially for research needs.

Isn’t numpy written in Fortran?

The native code python extensions for numpy are written in C, but numpy often links to matrix libraries written in Fortran like BLAS and LAPACK

Scipy is where a lot of the fortran is.

A disclaimer -- I do not work in physics, however a couple of recent physics Ph.D. that I work with expressed similar views.

Speed is not nearly as important now as it was 10-15 years ago. A typical scientist's worstation has 20 CPU cores. If I want to use 100 CPUs for a few days, it is trivial and 1000 is easy to get. Thus the fact that Python is slow(er) does not bother me unless I am setting up something major.

What matters though, is the availability of libraries that let me reliably run my experiments. If there is a bug, it must be in my code, not the library -- a "discovery" caused by a software bug is humiliating. This is where Fortran shines and Python is not quite there yet -- the decades of beating on those libraries made them very well understood and trusted.

"Speed is not nearly as important now as it was 10-15 years ago." Please try telling that to other users sometime. They do not seem to agree.

What users? I do work in research and this is what I see almost every time -- doubling the number of CPUs available for a simulation is trivial; getting someone to rewrite existing code to make it twice as fast on the same CPU is hard.

I am not talking about interactive programs -- a slow browser is annoying. I am talking about scientific computing. This is just my experience, can you provide some counterexamples?

So more cores means you can simulate to a higher resolution

I don't see Python displacing Fortran or C++, but I do see Python showing up in HPC as a platform for DSLs. Instead of handling the expensive part of the computation, Python acts as a UI layer / configuration manager / code generator. For an example, see fenics: [https://fenicsproject.org/]

>Give it a few more years and enough numpy-based libraries might make Python a real competitor.

A lot of Numpy is written in FORTRAN.

Hardly any of numpy is written in Fortran. It's basically all C and Python. Numpy does however links to and provide wrappers around some existing Fortran libraries.

Potentially your "rival groups" will have a long term advantage now though. This might be bad for the graduate students that did the work, but could be good for the professor and group over the next decade.

Graduate students need to think in terms of 4-6 years while forward looking professors might want to think in terms of 5-15 years.

That was without a doubt pretty bad for those grad students' careers. That kind of work shouldn't have been done by anyone without tenure.

Not necessarily. It was definitely bad if they want to become academics and needed to publish papers. But if they knew from the start what they were getting into and wanted to transition from physics to software development then it might not have been so bad (although there might have been a more optimal path).

A bad advisor wouldn't care what his students wanted, but a good advisor might still have students work on this kind of project as long as they were aware it wasn't going to help them get a tenure track position in the future. If they wanted to go work at a national lab doing HPC work and programing it might have been plenty ok for their career (This is what I am transitioning to now) or if they want to go work for a hedge fund or apple it might also be an okay option.

I might actually disagree on this one. If it was the first year of their PhD, then yes, it does seem like grunt work which is dissatisfying, but they will have fully learned the ins and outs of their group's code framework.

In the process, they would have understood nearly every approach taken by the group towards producing the results that it does, and I bet that has helped them when they've ended up modifying that same code later for their research. And with fewer supervisor meetings to work out exactly what X, Y and Z part of the code does because they will have worked on it themselves.

I'd say I've easily spent 6 months just spending time getting my head around all the code we work with anyway, so at the very least, I hope that made their following years of research more productive.

Or you could spend the time reading the papers of many groups.

I don't know. Having "rewrote X kloc of scientific Fortran to modern idiomatic C++" on your CV should get you to the head of the line in many places when looking for a job.

Only if you go after run of the mill coding jobs after your phd, in which case why bother at all. When applying for a postdoc, you'll very much want to bury that part of your work.

Or you could be applying for a HPC research job in one of the countless non-university setting that also does that sort of thing. Then having both a relevant PhD and some relevant hands on experience will be extremely helpful.

Plenty of high-paying quantitative jobs requires such a degree and experience. The majority of phd’s don’t go on to support their family with a tenure-track academic salary.

Why would you rewrite something that works well instead of just linking to it?

Because you need to modify or extend it in some nontrivial way.

I do wonder about this a little, but so far I haven't seen anything implemented elsewhere in my field that hasn't been possible to do with our existing code. In fact, I have spent a fair bit of my time replicating others' results with our code and getting sub-percent level agreement.

However, I guess one drawback is that a lot of the things we currently implement are all written from scratch (for very standard things such as numerical differentiation and parameter optimisation), which has the advantage of having "control" and more understanfing over the code, but less time saving/potentially not as efficient as using pre-existing libraries.

I agree, it is very project/field dependent. There are times when a painful redesign might pay off in the future, but I am sure there are also times when it is a waste of effort. In your case a redesign might have been a waste, you are in the best position to judge this.

I use Python for most things and write C extensions which can use OpenMP or CUDA and which are hooked in via Cython for the slow parts. Find this works well, although it can lead to you duplicating things unnecessarily sometimes (needing a C function to be callable by Python requires you to write a wrapper for it).

>>I later learnt one of our "rival groups" attempted the same thing and it took three phd students working full time for a year to rewrite their code from fortran to C++.

I wonder if there are any transcoders for Fortran to C++? I wonder if there is even a market for something like that? I've written alpha versions of transcoders for C to Java and Java to Objective-C and I think I could do the same for Fortran to another language but why?

Writing a transcoder first is the right way to do it and it should take you a couple of months to do it, especially if you already have some experience with this kind of stuff. Definitely less than a year for a single person. I would not try to translate the code by hand. It would be a never ending project full of bugs.

Edit: Apparently there are plenty of Fortran to C++ transcoders. Here is just one I found during my google search: http://cci.lbl.gov/fable/

No idea how good they work.

When I was working for LROC, most of the planetary geologists used IDL (Fortran-ish in many ways) but there was still plenty of actual FORTRAN floating around. Oh, look, in the last couple of years this legend of photometry worked up this new method and here's the associated Fortran code. That kind of thing.

I did actually rewrite both IDL and Fortran, but it was always smallish, single-purpose programs or functions.

Note that the SPICE Toolkit[1] is still written in Fortran today, translated to C using f2c, and the C version used as the base for other language support. The Fortran part is unlikely to ever go away, since support for processing past missions is crucially important and all that processing code was written in Fortran. Also, talk about your stable, backward compatible APIs...

1. https://naif.jpl.nasa.gov/naif/toolkit.html

Just before I left JPL, the SPICE team had announced a plan for a complete rewrite in C++. The SPICE team is exceptionally small (especially considereing how widely used and impactful the software has been). IIRC since the team’s inception, to the time I left, it has been about 4-5 people. Their primary goal has always been stability and correctness over speed. Recalling a conversation with Boris, they have something like 2.5 million lines of test code. So it would take some time to port over. The codebase is probably the most documented I have ever seen, every mathematical deduction is described in great detail.

That being said, as someone who has integrated CSPICE into several C++ and python projects, the modernization would be a very welcome change. The current arch depends far too much on global state, and none of it is threadsafe.

https://naif.jpl.nasa.gov/naif/Toolkit_evolution_R2

> Recalling a conversation with Boris

Good to know Boris is still there! Was Chuck still heading up the team when you left?

> a plan for a complete rewrite in C++

Oh, no! That's a real shame. I'd be all for a rewrite, especially considering their insane test coverage, but that decision worries me.

> That being said, as someone who has integrated CSPICE into several C++ and python projects, the modernization would be a very welcome change.

As someone who worked mostly with CSPICE in C but also with some other non-Fortran toolkits, I agree.

I recall that a local aerospace outfit was looking for C++ programmers to re do a metric F%^K ton of Fortran into c++.

I actually rang the recruiter to query why would not be simpler to train your existing staff to use Fortran - the add stayed up for years and years I always wonder if it ever got ported.

Did you get an answer about why not just train them to use Fortran?

Not really its very odd as when I worked at on of this organisations peers straight from high school I was told to get a fortran book from the company library and learn it I also had an hours basic instruction in how to boot the PDP11

"...and it took three phd students working full time for a year to rewrite their code from fortran to C++"

There's a salient lesson here. Clearly, this was a stupid decision for two reasons:

(1) there are many thousands of scientific routines that have had 40-60 years of fine honing and careful debugging and they just work! (For instance, we send voyagers—Voyager, Cassini, etc.—to the end of the solar system and they invariably get there; the Fortran routines that get them there do exactly what they're supposed to do (unlike much of today's poorly written C code)!

(2) Rewriting that already-reliable fully-debugged Fortran code into any other language will almost certainly make it far less reliable, thus it's a no-brainer to stick with the original .F source.

Yes, Fortran is simpler than C [consider it BASIC on steriods] and years ago it had long since evolved well past John Backus's 1954 incarnation of the language into solid workhorse that both physicists and engineers use regularly.

Just because something is old and out of fashion doesn't mean that it's broken or doesn't work well. (Longevity ensures that there's been sufficient time for many hands to make it reliable.)

(Oh, BTW, I'm reminded that decades ago when I was just beginning to learn Fortran using punch cards on an IBM360 mainframe with the WATFOR FORTRAN-IV compiler, that I made four errors in only six lines of code. After dutifully printing ERROR against each offending line, the compiler finished with the message: "YOU NEED TO SEEK ANOTHER CAREER " or words to that effect [yes, it was in uppercase]. Eventually, I got considerably better.)

It's not just that the code is already written and debugged; in many cases results have been published using analysis done with this legacy software. Using the same code ensures a certain amount of consistency between experiments from the same lab.

From this point of view, rewriting code is an extremely high-risk proposition. As we know, the likelihood of discovering bugs during this process is quite high.

Lawrence Berkeley Lab has a C++ to Fortran compiler called FABLE: http://cci.lbl.gov/fable/

Curious if anyone in your group considered using it?

>> Professors usually have this legacy code on hand (often code they wrote themselves decades ago) and pass this code on to their students

So cut-n-paste code. The students are running code that they don't really know what it does, it might not even be correct...it's like StackOverflow in academia.

Did you write your own libc and libm? Compiler? We are ALL using code we didn't write.

Daniel J. Bernstein did that for all of his projects (qmail, daemontools, etc) and I don't see him getting a ton of flack over doing that.

He certainly used to back when qmail vs postfix as the better sendmail replacement was a real debate 'we' where regularly having. If he's not getting any flack today it's most likely because he's fallen more or less into irrelevance when it comes to practical day to day operations.

Why isn't Julia a good choice?

Reply on news.ycombinator.com