A new programming language for high-performance computers
news.mit.edu
news.mit.edu
Source: was involved in development of multiple HPC frameworks, attended multiple HPC conferences, despite wailing and gnashing of teeth all actual software continues to be written the same way it's always been.
C lets you control memory alignment, in a way that very few other languages do. Until the competitors in HPC catch up to that, C is going to continue to outrun those other languages.
(Now, for doing the tweaking to get the most out of the algorithm, do I want to do that in C? No. I want to do it in something that makes me care about fewer details while I experiment.)
C isn't the Lingua Franca of fast software anymore. The reason why C programs can sometimes be faster, today, however is not an intrinsic property of the language but rather its inability to allow incompetent programmers to hide their bad data structures.
C also encourages programming styles that are harder to optimize so some loop optimizations are no longer possible.
Strong disagree. I don't see any reason why a competent programmer would hand-code a linked list, which is more of a hassle to make than a simple array (e.g. using buffer pointer + capacity and/or length field), unless the linked list makes actual sense for performance.
Try writing a type that can automatically switch layout from AOS to SOA in C, you just can't do it.
The only language I've actually done it in, is D. It's probably doable in many other nu-C languages these days, but D at very least can make it basically seamless as long as you do some try-and-break-shit testing to make sure nothing is relying on saving pointers when they shouldn't. This obviously constrains the definition of automatic ;)
I don't have my implementation to hand because it grew out of patch that failed due to aforementioned pointer-saving in code that I'm not paid enough to refactor (), but here's one someone else made https://github.com/nordlow/phobos-next/blob/master/src/nxt/s... there's another one in that repository too. I've never used those particular implementations but they're both by people I know so hopefully they're not too bad.
A more subtle thing, which I haven't used in anger, but would like to try at some point is to use programmer annotations (probably in the form of user defined attributes) to try and group things so things are stored such that temporal locality <=> spacial locality, but I've never bothered to actually do it.
There are some arrays of structs in an old bit of the D compiler that are roughly the size of a cacheline, and aren't accessed particularly uniformly. I profiled this and found that something like 75% of all LLC misses (hitting DRAM) were due to 2 particularly miserable lines... inside an O(n^2) algorithm.
struct Point { float x, y, z; };
soa<8> Point pts[...];
for (int i = 0; i < 8; ++i) {
pts[i].x + pts[i].y
}
struct Point { float x[8], y[8], z[8]; };
Point pts;
for (int i = 0; i < 8; ++i) {
pts.x[i] + pts.y[i];
}
I've not yet had an excuse to use ISPC myself, unfortunately :(
https://zig.news/kristoff/struct-of-arrays-soa-in-zig-easy-i...
(you could use a macro to encapsulate the layout though, or in C++ could code a function that returns a reference)
In what situations are linked lists the path of least friction??
C is a bad programming language in 2022
Anything fast is in C++ or C++ like languages.
What is it, then?
> The reason why C programs can sometimes be faster, today, however is not an intrinsic property of the language but rather its inability to allow incompetent programmers to hide their bad data structures.
I'm not sure I manage to parse that properly. What's wrong about not being able to hide bad data structures? And how is this making C faster?
Just a random example, I wrote a two lines Python function using Numba jit, because I know in my mind that some Python behavior is going to make it less performant and it is a high performance kernel so we need that fast (and also lower memory footprint.) Compiling using Numba jit is no brainer because I just add one more line there with like 10 seconds effort.
But the code base I am merging to has a policy against Numba (reasonably as we are targeting HPC platform where Numba has some performance problem related to oversubscribing if not setting up carefully.)
So I end up rewrote that in C++ and wrap it with pybind11. And the result is that it is faster by around 30%.
Since the algorithm is entirely trivial, the only explanation I have is exactly memory alignment, where I can control that in C++, but in Numba jit there's no way (both to guarantee the array allocated is aligned, or tell the compiler to assume that.)
(The 30% number is also reasonable in textbook examples.)
P.S. But it has a cost:
> Showing 14 changed files with 230 additions and 36 deletions. > from https://github.com/hpc4cmb/toast/commit/a38d1d6dbcc97001a1ad...
where the 36 deletions are mostly just documentations. So it's an ~200 lines effort comparing to a 4 line effort...
Hats off for making a good shot. But don't be surprised to see reluctance to move.
(See e.g. the Mario 64 decompilation — https://github.com/n64decomp/sm64 — , whose reverse-engineering process at all times kept a source tree that could build the original byte-identical ROMs of the games, despite being increasingly [manually] rewritten in an HLL.)
I even suppose that the orchestration layer can be in basically any convenient language, even in Python, while the underlying compute modules may be written in highly-optimized compiled languages; see Thensorflow and PyTorch for hihgly successful examples.
But you're right: today, MPI is dominant. I suspect this will change, if only because HPC and data center computing is converging, and the (much larger) data center side of things is very much not based on MPI (e.g., see TensorFlow). Personally, I find it reassuring that many of the dominant solutions from the data center side have more in common with the upstart HPC solutions, than they do with the HPC status quo. I'd like to think that at some point we'll be able to bring around the rest of the HPC community too.
Using Python on HPC system, depending on who you ask, is kind of cheating as the main work is always C/C++ (or Fortran). So it is still really just C/C++ and Fortran.
Julia is really promising as it is the first new language that can efficiently run on HPC and can use other form of message passing.
Also we should not forget UPC, a special kind of C that has its own parallel programming model (not MPI.) It has been used on HPC system as well but perhaps only from a selected few. (This probably reveal where I come from.)
Real work was always C++ and Fortran.
In terms of scripting, a professor once showed us a workflow script from his old days written entirely in bash and is very very long and very complex. He basically said if your script gets that complicated it should be written in Python nowadays.
But Python’s HPC presence is deeper than scripting in that sense. Machine learning frameworks definitely is a contributing factor, but even more traditional HPC workflows are benefited when stuffs have Python API, especially with mpi4py.
I once used something called mpi bash that I need to compile it myself on an HPC platform. It is much less well known and used, perhaps because of what the professor said—if it needs MPI, probably even for scripting it should be scripted in Python instead.
Obviously it's not very performant, but the lessons learned on how to program with MPI in mind is where the value lies.
Draft paper: https://arxiv.org/pdf/2008.11256.pdf
Sample code (looks like a DSL embedded in Python): https://github.com/gilbo/atl/blob/master/examples/catenary.p...
I might be wrong but they take the principle of dual numbers: a + b epsilon, and change the b component into its linear dual. Therefore, b is a function, and not a number. If b is a function instead of a number, then there's more flexibility regarding flow of control, and that flexibility allows you to express reverse-mode autodiff.
Just consider garbage collection. Trivial reference counting is easy to explain and implement. Not going to win too many points in use, though.
https://github.com/ChezJrk/verified-scheduling
OTOH, a Coq-like DSL written in python would be exciting!
The coq repo is covered by the paper but is a verified version of only the scheduler (a part of the parent python repo, i presume)
Then the repo you linked to seems to be a rewrite/generalization of another(?) lower level part of the main ATL.
IMHO the main issue of HPC today is not performance per se, but rather not embracing software best practices...
On one hand, I hated that there wasn't some fancy framework (like, a database?) to hide the footguns behind... on the other hand, the learning curve was very low and I'm happier about that on the balance.
- Usually no testing on a small test case / data set before going to the big one;
- Very old fashioned workload management systems with CLI-level job setup and automation prone to trivial errors as not handling escape sequences;
- Poor environment setup, little to no use of containerisation and when in use, HPC-specific solutions (as Singularity or Slurm containerisation support) tend to be more palliatives than addressing the root issue;
- A generic "not invented here" syndrome plus a strong belief that standard software best practices cannot be adopted in the HPC space (for no reason).
This, more or less.
I don't understand how "no testing on small test case" will resulted in "loads of compute hours get wasted because of accidentally overwriting output data"?
> Very old fashioned workload management systems with CLI-level job setup and automation
I think this means they are rolling their own workflow manager? By workflow manager, I mean like those listed in https://docs.nersc.gov/jobs/workflow-tools/ and https://docs.nersc.gov/jobs/workflow/other_tools/
> ...standard software best practices cannot be adopted in the HPC space (for no reason)
I agree for no reason is bad, but then sometimes there's reason. Just a random example, for example using snakemake with SLURM has the following caveat at NERSC:
> Astute readers of the Snakemake docs will find that Snakemake has a sluster execution capability. However, this means that Snakemake will treat each rule as a separate job and submit many requests to Slurm. We don't recommend this for Snakemake users at NERSC. > from https://docs.nersc.gov/jobs/workflow/snakemake/
It could depends on how large the HPC is. The case mentioned above is NERSC, which I think always has a top 10 HPC at release time in top500.org. When dealing with such a state of the art HPC, some "standard software best practices" doesn't scale well enough for that many nodes.
In the particular example above, it is related to the SLURM scheduler, and when there are many more nodes for it to schedule, some usage pattern in smaller HPC platform can break it.
> Poor environment setup, little to no use of containerisation and when in use, HPC-specific solutions (as Singularity or Slurm containerisation support) tend to be more palliatives than addressing the root issue;
But how this would "loads of compute hours get wasted because of accidentally overwriting output data"?
While container could solve some problems in setting up environment, it is not the final answer. E.g. many HPC (if not all) does not use docker itself but roll their own Singularity or Shifter solution is because they cannot give you root access. Also using container does not automatically provide you optimized binaries, meaning if one want to run HPC applications at best performance (say `-march native -mtune native` or even O flag) will need to build their own container. (Compiling specifically to the native arch is very important to HPC. e.g. For NERSC Cori II using Intel Xeon Phi KNL, AVX512 is essential for performance, and in this specific case, we really need the Intel compilers for peak performance as KNL is odd and GNU compiler can't optimize them as good, given they don't have much intensive to optimize for such niche product, unlike Intel has to deliver that to sell the product.)
And if we now agree we need to compile specific to the HPC, then container is not that attractive comparing to just compile using the HPC environment (with Cray wrapper compiler for example.) It does has other advantage related to IO as shown in https://docs.nersc.gov/development/shifter/images/shifter-pe... from https://docs.nersc.gov/development/shifter/
Going back to another statement in the parent thread,
> IMHO the main issue of HPC today is not performance per se, but rather not embracing software best practices...
The main issue of HPC is really performance. Again it might depends on how big a HPC is (in terms of top500.org ranking), but for really high performance computing, which always is at the cutting edge, they are really testing new technologies that brings us to the next level. (say exa-scale at the moment) Think about the tech used in upcoming exa-scale or near-exa-scale machines, some uses AMD CPU + NVIDIA GPU, some AMD CPU + AMD GPU, some Intel CPU + Intel CPU, and some pure ARM CPU... (and the previous generation GPU-like CPU such as Intel Xeon Phi) They are all quite exotic (compare to previous ones) and very difficult to squeeze peak performance. One simple metric is compare the theoretical peak performance to that realized in https://www.top500.org/lists/top500/2021/11/ , it is very difficult for these large system to have high realization especially as the number of nodes are increasing at that speed (communication bottleneck is more apparent.)
(Just to mention that is a simple metric that no longer tells the whole story. That measures only in terms of FLOPS, whereas nowadays IO can be a bottleneck in "big-data" applications. And there are less well known metric targeting that.)
E.g. in the current generation NERSC system, Cori, some systems with lower theoretical peak ranks higher than that essentially because of the headroom from Intel KNL (everyone who uses that system hates Intel KNL... Even Intel themselves hate Xeon Phi so much that they give it up. And their replacement solution is, well, just copying NVIDIA and give you a GPU.)
The next issue, or the same issue in disguise, is energy. It is well known that we can build exa-scale machines a few years back. What prevented it is the amount of energy (or power as in MW) it needs. The main thing we need to solve to go to exa-scale is basically FLOP per watt, whereas in the old days may be more FLOP per $. (There's a rule of thumb that a Million Watt machine needs a Million dollar per year energy cost.)
Frankly, I think many users of HPC continues to have bad software practices and continue to be successful. Bad software practice are going to bite them when like you say (generated) data are overwritten, which basically is just wasting computer hours. (It is rare to lose original data, but I think there's a recent news say a lot of actual data are loss in an HPC center / university.) Or that bad software practice resulted in a lot of manual labor in baby-sitting jobs. But in today's Science world, that's entirely irrelevant. While I was speaking with Scientific computing in mind, it probably is also true say in financial computing, etc. Well, people just cares about the outcome and who cares how messy you get there.
Halide has had transparent autodiff for some time now. I am guessing, this is follow-on work in the usual academic style. (Not trivializing at all.)
I've been watching Halide for a while as I really like it, intuitively. Also it wasn't that hard to bring up on FreeBSD with a few patches, where it performs more or less exactly as on Linux when configured cpu only. That's solid generic programming, to say the least.
However... I have no recent decades experience with HPC but I would be surprised if dense tensor operations were a significant component of current workloads. Could be though!
And just reminding people this is one of the very few cases when it is ok to create a new language.
In Julia, there is already Tullio.jl [1], which is basically a front end for tensor expressions, which can target both CPUs and GPUs with AD support and automatically parallelizes the computation. It doesn't really optimize the generated code much right now though, so something like this could be interesting.
They have some framework that achieves high level compute!
1. You come up with notation for tensor spaces, and tensor product operations, and a way to specify the indexes of pairs of dimensions you would like to contract, either using algebra or many-argument linear maps.
2. You write down a symbol for the tensors and one suffix for each of the indices. For an arbitrary rank tensor you write something like $T_{ij\cdots k}$. You write tensor operations using the summation convention: every index variable must appear exactly once in a term (indicating that it is free) or twice (indicating that it should be contracted – that is you should take the sum of the values for each possible value of the index). Special tensors are used to express special things.
So for the product of two matrices C = AB, you could write:
c_{ik} = a_{ij} b_{jk}
v_i = a_{ij} u_j
t = a_{ij} \delta_{ij}
d = a_{1i} a_{2j} \cdots a_{(n)k} \epsilon_{ij\cdots k}
The great advantage is that you don’t spend lots of time faffing around with bounds for the indices or their positions in the tensor product. But to some extent this is only possible because the dimension (ie the bounds) tends to have a physical or geometric meaning. This is also why when the notation is used in physics you also have subscripts and superscripts and you must contract one subscript index with a corresponding superscript index.
I wonder if it is possible to achieve a similarly concise convenient notation for expressing these kinds of vector computations.
You can imagine abusing the notation a bit but leaving most loops and ranges implicit, e.g.
let buf[i] = f(x[i]);
let buf' = buf or 0;
let out[i] = buf'[i-1] + buf[i];
out
A convolution with 5x5 kernel could be something like:
/* in library */
indicator(p) = if p then 1 else 0
delta[[is]][[js]] = indicator(is == js)
windows(radii)[[is]][[js]][[ks : (-radii..radii)]] = delta[[is]][[js + ks]]
/* in user code */
out[i,j] = in[p,q] * windows([2, 2])[p,q,i,j,k,l] * weight[k,l]
/* implicit sun over all k, l, p, q, but only loops over k, l as p, q values implied */
I don’t know if it would be possible to sufficiently generalise it and avoid having to specify the size of everything everywhere, or whether a computer could recognise enough patterns to convert it to efficient computation.
the idea that speed, safety and usability are mutually exclusive has always been wrong. trivially so imo... but clearly not given how long the myth has persisted.
https://www.cambridge.org/core/books/abs/crystal-field-handb...
Is Rust considered slow these days outside of compilation speed?
I wonder how this kind of thing will fare a 100 years in the future, where X is drawn from a set of functionalities that is much and much larger. We would essentially get stuck on a single language.
Considering the small size of the project, I wouldn't consider that fast and in fact, makes me very turned off on the prospect of the project growing any bigger.
I doubt your compile times would really ramp up that much from your project itself getting much bigger. If you are at 1.5k lines of code, the ridiculous compile time is probably due to other causes.
Thanks for the pointers though!