Hacker News Reader

Comment by gigatexal | Hacker News Reader

If I recall correctly the AMD hardware is competitive or better in raw compute than their nVidia counterparts but Cuda makes developing software so easy so I’m curious what ORNL’s take on that is.

View on HN

I don't work for ORNL and don't know what their take in this is, but a lot of codes that the DoE uses are being ported to the Kokkos framework out of the Sandia National Lab. The application developer at that point basically doesn't have to care if the code runs on KNL, on Nvidia cards using Cuda or on AMD cards using ROCm. The work of tuning Kokkos well on AMD cards is cheap compared to getting stuck on a single vendor.

Is it open source? It’d be huge to have something compete with cuda so that the best hardware may win

Kokkos (as in the C++ template frame work and a lot of tools to get insights into bottle necks) is opens source. See https://github.com/kokkos/kokkos and https://github.com/kokkos/kokkos-tools . They take pull requests and react to bug reports. They even plan to upstream some things from kokkos (such as multi-dimensional arrays) into future C++ standards.

The mission related codes based on kokkos are very much not open source.

There are some non-weapons codes like Albany and NaluCFD based on Kokkos and Trilinos that are open source.

[1] https://github.com/SNLComputation/Albany

[2] https://github.com/NaluCFD/Nalu

Most everything we develop (I work for the DOE) has to be open source.

At LLNL the story is similar, but the framework is RAJA: https://github.com/llnl/raja. RAJA is a bit simpler than Kokkos in that it does not require you to adopt its data structures.

The Kokkos and RAJA teams are also working together on some common utility libraries now.

Is the implication, that applications must use C++, rather than a language people like me can understand? A while ago, I heard the LLNL CTO talk about performance portability for Sierra with the message, I thought, that it was focussed on OpenMP 5, with features driven by CORAL requirements. (He was around for an OpenMP standards meeting, though...) Is it not actually working that way -- if you can talk about such things?

A shocking fact I heard recently is that on these large GPGPU supercomputers, only about 10% of the applications actually use the GPU at all.

The labs are not likely to be a big fan of CUDA, since the reality is that most scientists do not have the bandwidth to rewrite their software (even to use GPGPUs in the first place, see above), and Nvidia tries very hard to make sure that CUDA is impossible to use for other GPU vendors. The labs have a requirement to source from multiple vendors, so the CUDA lock-in is not something they are thrilled about.

I'm wondering the same. I'd guess that 99% of science code that uses the GPU now depends on CUDA. The other 1% use other third-party GPU frontends which might also be tuned for Nvidia GPUs. I feel that it's good to use the AMD ecosystem to mix up the competition a bit, and the majority of other clusters will still use Nvidia if your application requires it, but I'm just not aware of any momentum at all (in the form of software ecosystems) that target AMD, so library-writers will have to basically start from scratch if this AMD cluster is confirmed.

>... so library-writers will have to basically start from scratch if this AMD cluster is confirmed.

I doubt library writers will care beyond anyone involved in the project. That said, it's possible to port CUDA code already with a bit of work, so not quite from scratch in any case:

https://github.com/ROCm-Developer-Tools/HIP