Thanks for the great reply and for linking the research paper - I am going to check it out. Could you suggest a good review paper for someone new to get into modern HPC?
https://hpc.llnl.gov/documentation/tutorials/introduction-pa...
This is a very detailed free book focusing on programming:
But HPC is very diverse. Some care about compute performance, others about memory bandwidth and others about IO performance. Some run a ton of small jobs while others run a single large job.