Evaluation of C, Go, and Rust in the HPC environment [pdf]

The author has a "lack of equal experience in the three evaluated languages", especially C, but is comparing them for development time and SLOC based on a sample size of one developer and one very small program.

He rewrites a distributed program as a threaded one, because Rust and Go didn't have a distributed computing library, but says Go and Rust have "similar performance" in HPC.

He says he was surprised at the C performance, but there's no investigation of a cause. Maybe just because C is slow...

I don't mean to be overly critical, but these results in the paper mean absolutely nothing. It's a waste of time.

The problem with reading this in BS thesis form is that it hasn't gone through the wringer enough. Most of the performance comparison results vs C are bogus because (to its credit, as noted), the author accidentally compiled some important support libraries for the C version without optimization and used those on the cluster. Oopsie.

As briefly mentioned in the previous chapter this performance regression might have been caused by the two unoptimized libraries that were compiled on the development laptop and copied to the cluster.

Keep in mind, this is a bachelors thesis. It's not a peer reviewed research article in a respected systems journal or conference. I mean this in both a positive and a negative sense. Positive in sense that the purpose of a bachelors thesis is to introduce undergraduates to an extended, self directed project with an element of research to it. As someone who has supervised many such projects, I think that this is a pretty good one. Great work! At the same time, since the author is an undergraduate, presumably with a full coarse load, this is not a high quality research publication. The results are probably a bit shaky and the methodology is perhaps a bit off. But that's ok. Every excellent piece of work has to start somewhere.

GitHub repo with all of the code: https://github.com/mrfloya/thesis-ba

Most interesting part of the thesis, for me, is page 42:

> In this case it was even an advantage that the Rust version was developed last since it revealed a critical error in the other implementations.

Null strikes again!

I feel that it's a shame to test C rather than C++, because the latter offers so many advantages to developer productivity -- the paper cites memory management and type-related errors as being major causes of developer unproductivity, and C++ has automatic memory management and templates and such.

Also, it doesn't restrict the use of C libraries, so even if you've been writing C for HPC code all these years, there should be minimal technical difficulties in switching over, except for the time it takes to learn C++, which shouldn't be long if you know C well.

Looking through the links, and methodology makes me question a lot of things, most of them regarding his choices in how he allocated memory. First and foremost, he was using the glibc malloc, which belongs in a museum, not in a running system. One of rust's big advantages in performance comes from that alone, given all the memory that's being allocated and destroyed. Which leads to the second question of how he did allocate objects. He knew he was going to allocate a number of objects, yet he didn't pre-allocate a large chunk of memory ahead of time to make it so he wouldn't have to call malloc and friends as often. I wouldn't be surprised if spending a bit of time just tuning memory didn't cut significant time from how long execution takes. I really wonder how much time was spent in the system context of all three of those; I'd be willing to bet that the final part of the examination would probably fare much different had he used the same malloc for the C program that rust and go already use.

Any HPC language evaluation that doesn't include vs FORTRAN is pointless. It's still king of computation for a reason.

I'm a bit surprised that the author didn't mention julialang.org in the report - it's much more suited for technical / scientific computation, while Go and Rust seem better for systems-level concurrency/design.

My biggest criticism of this work is that it does not consider C++. Most of the big HPC projects today are written in C++, not C, and to the best of my knowledge this transition happened in the 90s.

An interesting case study, though not really HPC or scientific computing related. It would be good to see some profiling to understand exactly why each implementation was slower in particular ways. From a HPC context, linking with Fortran and C codes, and exchanging data with them, is very important.

Would've been nice with some internal links in that pdf, especially in the table-of-contents (links to sections and subsections). This is simple with hyperref.

"In times where compilers are smart enough to basically rewrite and change code for performance reasons it is completely inexcusable that the order of source arguments to process is still that relevant."

After reading the comments on C and the difficulties the author had in the compilation process.... All I can say is Garbage in garbage out. I have worked in the HPC enviornment for a number of years using both Fortrash and C. The errors the author encountered would have been avoided by someone with any experience developing large softwares. I learned these things after working on my first project and now they barely, if at all, factor into my development time. Make sure you build and keep a dependency graph while you are developing and this becomes a non issue.

Interesting work.

It has some pretty basic flaws that are pervasive that the reader should be aware of. First off is the "productivity" measure: the last time I looked around, this was nearly impossible to quantify for software development. The author chose SLOC and development time as stand-ins for productivity. SLOC has a connection with code quality and time of development, but development time is well known to vary[1].

In particular, development time in this thesis is linked to results in a pretty classic "Psychologist's fallacy"[2]; the author generalizes their experiences as conclusive. This implies in particular that "development time" in this thesis should be thrown out for meaningful conclusions, as the sample size is 1. It is, however, an interesting experience report.

Another basic flaw is the connecting of 'modern' with 'good'. The author remarks the main disadvantage of C and Fortran are their age; this crops up here and there. Workflow tooling in Go and Rust are major focuses by the developers: C workflow tooling is usually locally brewed. This does not make C-the-language worse.

More on a meta level, my advisor in my Master's work drummed into me that I should NOT insert my opinion into the thesis until the conclusion. So I found the editorializing along the way very annoying.

And finally, and very unfortunately, the experience level of the author appears to be low in all three languages; this is significant when it comes to implementing high performance code.

---

Now, for the interesting / good parts of this experience report.

Standout interesting for me was the Rust speedup on the 48-core machine. I did not expect that, nor did I expect Go to make such a good showing here as well.

In general Go performance/memory made a surprisingly good showing (to me) for this work. I shall have to revise my opinion of it upward in the performance axis.

I am both surprised and vaguely annoyed by the Rust runtime eating so much memory (Would love to hear from a Rust contributor why that is and what's being scheduled to be done about it).

Of course the stronger type system of Rust catching an error that C and Go didn't pick up is both (a) humorous and (b) justifies the type community's work in these areas. I look forward to Rust 1.0!

One note by the author that is worth calling out stronger is the deployment story: Go has a great one with static linking, whereas C gets sketchy and Rust is... ??. I believe Rust has a static linker option, but I havn't perused the manual in that area for some time. For serious cloud-level deployments over time, static linking is very nice, and I'm not surprised Google went that route. It's something that would be very nice to put as a Rust emission option "--crate-type staticbin".

Anyway. I look forward to larger sample sizes and, one day, a better productiv

[1] http://www.amazon.com/Making-Software-Really-Works-Believe/d... The situation is actually far worse than just varying developer time.

[2] https://en.wikipedia.org/wiki/Psychologist%27s_fallacy

Well, Go and Rust can do better threading on this problem than a not-really-optimized C code with OpenMP. Thats a start. Message passing has to be shown next, and then one can decide whether the installation and runtime of Go/Rust/Python/Julia on something like an IBM Blue Gene/Q with its custom compute node OS is easier than a link flag.

80 pages for that, with one user and one implementation of one algorithm ? Sorry but the rate knowledge/size is very low, so don't be impressed by the page number, it harms the paper more than anything else. Even for a bachelor thesis, it's mostly "filling"

Why are there so many bold terms and phrases throughout the text? It's extremely distracting.

I wonder what would've happened with the results if biicode would've been used as deps/package manager for C.

Wildly offtopic- reading papers like this bring to mind for some reason, that the next evolution of CPython, the one that deals with the Python2/3 rift should be written in Rust.