What can Rust do for astrophysics?
arxiv.org
arxiv.org
This are not valid benchmarks at all. What is the point to run N-Body sim with all particles set to 0.0 ?
Where is the result of the sim ?
Why it will be not validated at all ?
To make this look like real benchmark one will need to use the same start condition and then validate that result for all languages is the same.
It would be great to have C++ code using Eigen library for example ? To show the difference .
Also this is naive O(N^2) sim that could be speed up using Fast multipole method (FMM), but of course this would be much more complicated to do this in Rust or Go as in C++.
Not sure about language it self, my knowledge about Rust is limited. But st is at least harder because we already have optimized libraries that can do this in C++.
Do some one know such library for Rust ? It would be great too look at it.
The Rust community is actively hurt by people doing things like this, no-one takes this type of work seriously.
Edit: please come back when you've replicated and validated at least something like a basic, well-understood finite difference solver of single-phase incompressible Navier-Stokes, and shown that it's either equivalently fast and much easier code, or significantly faster (very unlikely). Do note that a readable Fortran version is going to be ~400 lines, so there's not a terribly large room for improvement. Also, we already have a good language that gives us slower and much easier code, namely Python.
(Protip: don't use if's inside tight loops. Just don't.)
In table 1
Rust Fortran C Go
They did have a note that C could be faster if language specific features could be used.
Is rust really that performant compared to C? Or did they neglect to even bother checking C's performance?
Which would be the equivalent of writing a comparison between a sledgehammer and jackhammer, without turning the latter on.
https://github.com/marblestation/benchmark-leapfrog
It would be great if somebody have a fresh look at the C version, since I completely agree that it is not normal that its execution time is so far away from Fortran or Rust.
gcc -O3
1m40.302s
gcc -O3 -ffast-math
0m5.110s
rustc -C opt-level=3
0m9.278s
rustc 1.15.1 (021bd294c 2017-02-08)
Edit: Looking into this a bit more, I think 'rustc -C opt-level=3' is optimizing out the actual integration. If I put a println!("{}", x[0][0]) at the end, I end up with 1m43s. Not sure what 'gcc -O3 -ffast-math' is doing; I haven't looked at the disassembly.
It's surprising that gcc does such a poor job at -O3 though.
Yes, without that it's basically cheating. There is now a GitHub issue: https://github.com/marblestation/benchmark-leapfrog/issues/6
The submitter writes: "final output [for Rust] is 134 seconds, and also the output is [[NaN, NaN, NaN], [NaN, NaN, NaN]] (batman)"
Rust is able to avoid allocations and de-allocations in many areas where C programmers would manually use alloc and free. I must admit though, that's less relevant to scientific programming, where the heaviest work is mainly matrix matrix multiplication and the sizes of everything is known in advance.
Without reading through the paper, I expect work on the Rust/Fortran side is being vectorized and/or being kept in L1/L2 caches, whereas the C and Go compilers don't use SIMD instructions and/or are generating code that triggers cache misses.
http://benchmarksgame.alioth.debian.org/u64q/compare.php?lan... -- in this benchmarks C in most cases faster than Rust, but only in few they have the same performance.
Edit: Looks like using -Ofast instead of -O3 makes a big difference: https://news.ycombinator.com/item?id=13634119
Offering context? ;-)
2. How many years already has SIMD not supported on stable been posted as a response in these discussions ?
2. 1.75, which is coincidentally how long a stable version of Rust has existed. :P
Clang produces practically the same between the two levels.
On OS X, so can't test if this actually makes a performance difference.
GCC 6.3, -Ofast: 5.93s
GCC 6.3, -O3: 93.06s
GCC 6.3, -O2: 134.43s
Clang 3.9, -O3: 346.92s
Clang 3.9, -Ofast: 5.87s
Intel 15.0, -O3: 110.84s
Intel 15.0, -Ofast: 106.88s
The `time` command is part of base OS X.
Also the output is [[NaN, NaN, NaN], [NaN, NaN, NaN]], which is a bit worrying...
Not sure how much this affects the benchmark, but it'd probably be smart to randomise the starting positions.
LBB3_1:
addsd %xmm1, %xmm0
addsd %xmm1, %xmm0
ucomisd %xmm0, %xmm2
jae LBB3_1
In other words, it's measuring how long it takes to count to 3.6525E+8 by adding 0.04 repeatedly. LLVM (which both Rust and clang use as a backend) optimizes away the actual work of the program.
GCC doesn't seem to optimize things out as aggressively, so that probably explains the difference: https://godbolt.org/g/lBRIIW
Never trust a benchmark you didn't falsify yourself.
printf("x[0][0]=%f\n", x[0][0]);
Benchmarking is hard.
It's not in the general case, but can be for particular code.
Unless the C program was written badly, the fact that Fortran is also faster seems to point to compiler optimizations not easy/possible in C (e.g. involving aliasing assurances).
That said, while I'm really excited to see rust go more places, astrophysics seems like one of very few fields where practitioners learn a high performance language as a matter of course, and so rust may have a less empowering effect for them. I would wager that after paying the upfront cost to learn it, students and researchers may see improved iteration time from fewer bugs. But it seems like rust may have an incremental improvement to offer which is not as powerful as the transformative nature of its potential in other computationally intensive fields.
void gravity_calculate_acceleration(int n_particles, double m[], double x[][3], double a[][3]) {
double G = 6.6742367e-11; // m^3.kg^-1.s^-2
for (int i=0; i<n_particles; i++){
a[i][0] = 0;
a[i][1] = 0;
a[i][2] = 0;
for (int j=0; i<n_particles; i++){
if (j == i) {
continue;
}
double dx = x[i][0] - x[j][0];
double dy = x[i][0] - x[j][0];
double dz = x[i][0] - x[j][0];
double r = sqrt(dx*dx + dy*dy + dz*dz);
double prefact = -G/(r*r*r) * m[j];
a[i][0] += prefact * dx;
a[i][1] += prefact * dy;
a[i][2] += prefact * dz;
}
}
}https://github.com/astrogo/fitsio
(a pure Go version)
1. Compile C version with better optimization flags or ask an expert to tweak it for faster performance.
2. Include sample run on some known / meaningful data to test that software runs correctly (not only fast).
IMO rust doesn't have to be faster than C to make a valuable article. If it is as fast or almost as fast it is still a good data point.
Part of the point they are trying to make is that you don't need to be an expert to reap performance benefits in certain languages. They explicitly state that they did not do things that C experts would know how to do. This is pretty reasonable in the context of astrophysics. The whole idea of the thing is that if there is a language that allows you to write faster code than C without having to "ask an expert to tweak it", you will be more productive using that language.
That said, your point 2 is entirely valid; the paper fails to demonstrate pretty much all of its claims, and it never makes a case for needing Rust's safe dynamic memory allocation features in a program that doesn't use dynamic allocation at all.
For whatever it's worth, the astrophysicists I most recently talked to wrote their simulations in Python (with SciPy or similar) or even something called IDL: https://en.wikipedia.org/wiki/IDL_(programming_language) and they also tend to have mad Bash skills for glueing different data-processing programs together.
At this point it is not the language expertise, but knowing about your tools.
Rust's safety features aren't tied at all to dynamic memory allocation. You can write a Rust program with pointers (into the stack, for example) with no dynamic memory allocation at all and still benefit immensely from the safety features.
It appears that it doesn't run correctly (rust optimizes out the actual integration, and values are all set to NaN!)
https://github.com/marblestation/benchmark-leapfrog/issues/6
a) access to invalid memory regions, b) dangling pointers and attempts to free already freed memory, c) memory leaks and, d) race conditions.
The first three benefits also come with essentially any GC'ed language. d) is interesting. What are the costs that come with such a guarantee? Presumably it prohibits certain kinds of parallelism? Are the ownership/borrowing mechanics interesting programming tools, or are they a hindrance?
People talk a lot about static typing, and I can see the benefits for critical/user-facing applications, but not for numerical code. The nightmare with numerical code is finding out that I forgot a minus sign somewhere, and that it invalidates my last 6 months of published results.
Actually it enables parallelism you couldn't do before, because now you're confident the compiler will prove that what you're doing is safe.
Like any sort of static analysis, some see them as a tool, some see them as a hindrance ;)
> also come with essentially any GC'ed language.
Ownership and borrowing can extend to arbitrary resources, not just memory.
Also, C and D aren't correct; while Rust helps with memory leaks, it absolutely does not prevent them, and Rust prevents _data races_, a specific form of race condition, but cannot prevent race conditions generally.
Where is the discussion of the library availability? The libraries for Fortran are maintained and improved during the decades?
https://github.com/marblestation/benchmark-leapfrog
And indeed, Fortran has decades of advantage in terms of libraries.
The C code:
void integrator_leapfrog_part1(int n_particles,
double x[][3], double v[][3],
double half_time_step){
for (int i=0;i<n_particles;i++){
x[i][0] += half_time_step * v[i][0];
...
int main(int argc, char* argv[]) {
const int n_particles = 2;
...
while(time <= time_limit) {
integrator_leapfrog_part1(n_particles,
x, v, half_time_step);
const N_PARTICLES: usize = 2;
...
fn main() {
...
while time <= time_limit {
integrator_leapfrog_part1(N_PARTICLES,
&mut x, &v, half_time_step);
...
fn integrator_leapfrog_part1(n_particles: usize,
x: &mut [[f64; 3]; N_PARTICLES],
v: &[[f64; 3]; N_PARTICLES],
half_time_step: f64) {
for i in 0..n_particles {
x[i][0] += half_time_step * v[i][0];
The N-Body on this site, with other implementations, has clearly faster C than Rust:
https://benchmarksgame.alioth.debian.org/u64q/performance.ph...
As noted elsewhere in here, the C implementation is using SSE, whereas the Rust implementation isn't. It's just as much of an unfair comparison as the one you're describing. :P
https://benchmarksgame.alioth.debian.org/why-measure-toy-ben...
The OP doesn't even specify the rules for its own benchmark, as far as I understand doesn't verify the results? People here get NaNs?
If the NaNs are produced as results, then the OP code is not measuring the speed of calculations at all but the speed of the failed calculations. Which is especially problematic as the main argument is "attractive for the scientific community" "it guarantees memory safety." Which is presented as good because not having it "can produce random behaviors and affect the scientific interpretation of the results." If the result here is NaN the calculation doesn't even have to be performed after the first NaN that affects the result appears, as anything + NaN is NaN etc.
Edit: kibwen, did I write somewhere that I "refute" something? I gave a link to the rationale behind the "benchmarksgame" site. The rest is about the OP, surely not about your post.
I'm not sure what you're refuting?
Verifying that the results match is a necessary but insufficient quality for ensuring comparability. If the algorithm could be the same in each language but isn't (e.g. quicksort vs. bogosort), then that's not a valid comparison if your objective is to determine the overhead imposed by the language implementation itself (and if you're not trying to determine language implementation overhead, then what are you measuring?). Likewise if the implementation details could be the same in each language but aren't (e.g. if one uses 64-bit integers and the other uses 32-bit integers).
The computer language benchmarks game was initially conceived to determine a ballpark for how slow interpreted and managed languages are compared to C. Quantifying the overhead of interpreters and runtimes is its raison d'etre, and it shows. When it comes to comparing low-level systems languages that have no runtime to speak of, the best it can do is attempt to quantify the quality of each backend's code generator (it's a missed opportunity that it doesn't include Clang for comparison with GCC).
(And yes, I understand that the benchmarks game contains repeated massive disclaimers that people should not take the performance results as a means of serious comparison. Internet commentators remain undeterred.)
If you're just trying to argue that the methodology used in the OP is poor, then obviously we're in agreement (was there ever any doubt?).
Today the benchmarks game is referred to from the Rust FAQ. What is one to do?
https://www.rust-lang.org/en-US/faq.html#performance
Back in the previous millenia -- "[Doug Bagley's] goal was to compare all the major scripting languages. Then [Doug Bagley] started adding in some compiled languages for comparison…"
http://web.archive.org/web/20010125021400/http://www.bagley....
Actually one small disclaimer --
"Non-motivation: We are profoundly uninterested in claims that these measurements, of a few tiny programs, somehow define the relative performance of programming languages."
There's much more about PL benchmarks in general.
If so then an equivalent Rust program would be welcome.
If not then it's hardly unfair to compare what actually is provided in current Rust stable to what actually is provided in an old C compiler (gcc 5.4.0).
Yes, stable Rust provides equivalent SSE support to the language specified as ISO 9899:2011. :P
Apparently this is just Rust stable not providing the functionality.
* Safer than C
* Almost as fast as C
* Good concurrency support
* High productivity due to simple abstractions and good tooling
Is the downside of a GC language really relevant? Does astrophysics suffer from "stop the world interrupts" or is it just because of the performance? The Go GC is already really fast.
So safety gets you faster development: less debugging, easier parallelisation etc., but it does not give you correct results, nor does it give you those results by spending less computational time.
And you legitimately can't imagine how reducing the likelihood of making bugs by using a language+compiler which can identify/prevent more of them would help?
I'll leave you to it then.
> For instance, a common source of error is the access to invalid memory regions, which produces random execution behaviors and affects the scientific interpretation of the results.
Such errors are astronomically more likely to cause crashes than silent errors in calculations.
Fortran 2008 is already quite modern in language features and doesn't suffer from C safety issues.
EDIT: typo.
Then again, Fortran would cover these scenarios and is better established.
And your point about Fortran is great. I used that and C++ for my research and never once encountered an issue that Rust's guarantees would have helped with.
That's the key point. Judging by the (wrong) benchmark above where both C and Go seem to do some work, Go is about half as fast as C.
A grad student (if paid properly) costs maybe 60k €/year. In comparison, we regularly spend more than 250k on new and faster computers, not including maintenance, the electricity bill etc. If you can make software even 50% faster by increasing development time, this is nearly always worth it in an academic setting. Note that the benchmark implementation shown here is a very simple example, normally computing jobs take days to weeks (multiplied by however many cores you can sensibly use) of CPU time.
Decreasing runtime also increases productivity a lot, since the turnaround time becomes shorter; waiting 3 versus 6 weeks for results is a noticeable difference.
Additionally, these tools rarely have safety concerns: In my own code, there is no "untrusted user input". There is correct user input (good) and incorrect user input (mostly it will crash, with common mistakes it will try to notify the user). Safety is handled by the operating system (such that you can’t overwrite someone else’s data, for example).
This means that the only advantage Go could have over C/C++ in academia is the "good concurrency support". However, concurrency in HPC is usually handled via OpenMP (shared memory) or MPI (distributed memory) parallelisation. This takes a while to get used to, but is very different (and in a sense much easier) than e.g. the typical case for a web server, where you wish to serve as many users as possible at the same time using as little CPU time as possible – a busily waiting loop is horrible in the latter case but perfectly fine in the former (under some circumstances).
So overall, languages are not interesting in the academic HPC crowd if they sacrifice speed for anything, which, incidentally, makes Rust also interesting, because that is precisely not the case (apparently, under some circumstances, etc.pp.).
> Judging by the (wrong) benchmark above where both C and Go seem to do some work, Go is about half as fast as C.
I think this is what is most flawed in the paper. For (some) concurrent problems Go is about as fast as C [1].
But then again, I don't know what kind of computing requirements they have. Is there a reason why GRP computing isn't mentioned? They are really efficient and CUDA isn't that hard to learn.
> Rust allows the user to avoid common mistakes such as the access to invalid memory regions and race conditions
They seem to care about safety.
[1] https://benchmarksgame.alioth.debian.org/u64q/compare.php?la...
Certainly, this was really only taken as a ballpark estimate of the performance difference. Looking at your link, it seems to have been slightly overestimating the difference, though the general point still stands: C is (in most cases) faster and there is (nearly) no case in which Go beats C.
> Is there a reason why GRP computing isn't mentioned?
> They are really efficient and CUDA isn't that hard to learn.
Sorry, not sure if you mean GPU instead of GRP in the first sentence. CUDA helps to some degree, but not always, e.g. when you are memory bandwidth bound (common in tensor networks in physics). I have no experience with Monte Carlo methods and can’t comment on whether they substantially benefit from CUDA; I know of at least one (Quantum Monte Carlo) code which runs much faster on a standard Xeon than on the Xeon Phi, though.
> They seem to care about safety.
Yes of course safety is nice to have and I’ll gladly learn Rust when I have some free time to get that safety at hopefully zero cost. But sacrificing performance is simply not competitive, if you can throw someone with gdb at the problem and get essentially the same "safety".
That's an odd tradeoff. That time of debugging could be significantly longer than just writing the software in a safe language. Seems like a bad tradeoff, and in many scenarios, bugs can lead to very bad things that you can't recover from. I've experienced all of these, having to fix them over long periods of time (not all my code, but sadly some was): data loss, concurrency (tough to debug in gdb), major memory leaks (even from std libs), corrupted data because of misused non null terminated c strings, array out of bound issues. Each one of these took weeks to track down, maybe because I'm not smart, which is a valid criticism; with Rust I've never had issues with any of those (2 years and running), and given that I'm not smart, it helps me by telling me where I got something wrong.
Be safe out there people...
It is easier to debug and improve a fast program when you can have some result in 1 day versus same algorithm implemented in a slower language that takes one week.
It depends. If it takes 8 weeks for the program to run using a high-level language, but 1 week for it to run using a faster language, dropping down to C to cut the running time down to 6.5 days doesn't help if it take several weeks to do it, and you are not sure that you won't have to rerun the program due to hard-to-find bugs.
> If you’re using more than one word in your filename, use an underscore: hello_world.rs rather than helloworld.rs.
Is the use of underscores a recommendation or a limitation?
I don't see how Go would be a better option than it.