Porting HPC Applications to AMD Instinct MI300A Using Unified Memory and OpenMP

ROCm or HIP? Does it start out with porting a lot from CUDA etc. or starting fresh on top of the AMD APIs?

How much of the project time is spent on that compute API stuff in comparison to "payload" work?

LANL might disagree given that they just unveiled a new supercomputer with NVIDIA chips [1]. NVIDIA CEO Jensen Huang was even at the unveiling.

[1] https://ladailypost.com/los-alamos-national-laboratory-unvei...

Both Frontier and Aurora bet on unproven future chips. Sometimes it pays off and sometimes it doesn't.

They are competent people, just not in the fields techies want.

When you're a national laboratory and your wallet is taxes from fellow Americans, it is very important that you find a balance between bang and buck. Lest you get your budget slashed or worse.

nvidia absolutely gives deals to national labs and universities. See Crossroads @ LANL, Isambard in the UK, Perlmutter @ LBL. While AMD is being deployed at LLNL and ORNL, Nvidia isn’t done with their HPC game. Maybe not at the leadership level, but we’ll see how Oak Ridge and LANL decide their next round of procurements

This is a great question. In the sense that ROCm is pure userspace it's never necessary - make the syscalls yourself and the driver in the Linux kernel will do the same things ROCm would have done.

In practice if you go down that road on discrete GPU systems, allocating "fine grain" memory so you can talk to the GPU is probably the most tedious part of the setup. I gave up around there. An APU should be indifferent to that though.

There will be some setup to associate your CPU process with the GPU. Permissions style, since Linux doesn't let processes stomp on each other. That might be rather minimal and should be spelled out in roct.

Launching a kernel involves finding the part of the address space the GPU is watching, writing 64 bytes to it and then "ringing a doorbell" which is probably writing to a different magic address. There's a lot of cruft in the API from earlier generations where these things involved a lot of work.

Game plan for finding out goes something like:

  1. Compile some GPU code and put it in the host processs
  2. Make the calls into hsa.h to run that kernel
  3. Delete everything unused from hsa to get an equivalent that only uses roct
  4. Delete everything unused from roct to get the raw syscalls

Roct is a small C library that implements the userspace side of the kernel driver. I'd be inclined to link it into your application instead of drop it entirely, but ymmv. Rocr / HSA is a larger C++ library that has a lot more moving parts and is more tempting to drop from the dependency graph.

Going beyond that, you could build a simplified version of the kernel driver that drops all the other hardware. Might make things better, might not. And beyond that there's the firmware on the GPU which might be getting more accessible soon, but iiuc is written in assembly so might not be that much fun to hack on. And beyond that you're on the silicon, where changing it is making a different chip really.

>Explicitly copying data to/from GPUs is a definite nuisance.

CXL allows fine grained shared memory, but people look at the shiny high bandwidth NVLink and talk about how much better it is for... AI.

If you're doing simulations, or poking big matrices continuously on CPUs, you can saturate the memory controller pretty easily. If you know what you're doing, your FPU or vector units are saturated at the same time, so "whole system" becomes the bottleneck while it tries to keep itself cool.

Games move that kind of data in the beginning and doesn't stream new data that much after the initial texture and model data. If you are working on HPC with GPUs, you may need to constantly stream in new data to the GPU while streaming out the results. This is why datacenter/compute GPUs have multiple independent DMA engines.

Afaik those unified memory architectures are mostly neither cache coherent nor do they support virtual addresses efficiently (you have to trap into privileged code to pin/unpin the mappings) which means that the relative cost is lower than a dedicated GPU accessed via PCIe slots, but still to high. Only the "boring" old Bobcat based AMD APUs supported accessing unpinned virtual memory from the L3 (aka system level) cache and nobody bothered with porting code to them.

OpenMP has been around for a long time. People know how to use it, and it has gained many features that are useful for scientific computing.

The consortium behind OpenMP consists mostly of hardware companies and organizations doing scientific computing. Software companies are largely missing. That may contribute to the popularity of OpenMP, as the interests of scientific computing and software development are often different.

>> Is language support why people like OpenMP?

I use it sometimes with C++ because it is super easy to make "embarrassingly parallel" code actually run in parallel. And by using nothing but #pragma statements it will still compile single threaded if you don't have OMP as the pragmas will be ignored.