Programming on Parallel Machines; GPU, Multicore, Clusters and More | Hacker News Reader

Programming on Parallel Machines; GPU, Multicore, Clusters and More

heather.cs.ucdavis.edu

122 pointsabhi9u30 comments

If you feel like you've finally groked GPU/massive parallel software programming and need more challenges, I highly recommend playing around with digital circuits! The level of parallelism available to you in hardware is truly unmatched and it's incredibly fun, especially once you start really pushing implementations of your designs on FPGAs. Granted, FPGAs are frequently less useful than what you could do on a GPU due to the higher clock speeds available on ASICs (if your GPU core clock is 3GHz and your FPGA design maxes out at 500MHz [which would be admirable!], the GPU has nearly 6x the number of cycles to match or beat your implementation!).

How do you get started with that?

have a look into hardware descriptive languages, VHDL or verilog and go from there

How is the best way to programm with them?, IDE or LSP?

The OSS CAD Suite [0] is a good open-source toolchain for this stuff. You can then write hardware designs in the SytemVerilog language (VSCode has some plugins, I believe, but I've just been using a basic text editor) and use the build toolchain to compile ("synthesize") and program e.g. an FPGA with your designs.

(FWIW, I've only just taken a class on Verilog this past Spring, but we used oss-cad-suite and I found it pretty straightforward to use. The bundled version of Verilator had some issues on my Mac though, so I had to compile my own copy of Verilator.)

[0] https://github.com/YosysHQ/oss-cad-suite-build/

I started by reading “Digital Design and Computer Architecture”. There’s new RISC-V edition https://a.co/d/imzGBK5. The book starts from Boolean logic and transistor technology and goes all the way to assembly programming with everything in between. Most importantly gives great introduction to HDLs. Next I played with a bunch of hardware projects specifically targeting inexpensive Arty-A7 board to get comfortable with FPGA tooling.

Had a friend who fell into that black hole, it's so time consuming for peoplecwith little time

https://dl.acm.org/doi/book/10.5555/2815529

Their ARM edition (2015) is available for free via the ACM.

FPGAs are hardware, you generally program them with hardware description languages like Verilog.

If you don’t want to buy hardware (reasonable, IMO, since HDL is kind of niche and the boards can be pricey), you could try out the language with something like Verilator. This will let you write Verilog, and then compile it to generate C++ classes which simulate your design.

I know it depends on the analysis, but I often am doing somewhat embarassingly parallel things. So just knowing GNU parallel for mid-scale things (and R/python basically parallelism, although shared memory is a bear), and how to temporarily scale across the cloud to like 500 core, is huge.

500 cores is just a handful of nodes nowadays. Living in the future is weird.

If you count hyperthreads as cores, AMD 9754 dual socket is 512 in one node.

You can’t count a hyperthread as a core, that’s cheating.

I guess we’ll have to get four sockets.

Fwiw, a single u-12tb1.112xlarge has 448 cores.

Woof, $109/hour on demand pricing (~$950k/year on demand, ~$590k/1year reserved). Super curious what a similar spec machine would retail.

The problem is you will need to hire 10 guys to use that machine well. And that easily costs you 2M a year.

448 logical processors.

Each logical processor is a hyperthread on 224 cores.

A logical processor is the number of the processor's cores multiplied by the number of threads per core.

2 threads per core.

vCPUs are not cores and those are "new 2021" instances based on freakin Skylake. Yikes. Green-IT, huh?

For anyone who knows both R and Python well - Request: I think it would be nice to translate this book into Python from R.

The book is in C, not R

It says R there. There is even a chapter named "Why R" explaining why R was the choose language for the book.

Page 321 goes into how to translate R things to C/C++ things. It’s trivial to do the same for python. ChatGPT could do it.

I like the Parallax Propeller for this if you need a microcontroller. $8 for 8 cores.

Is MPI still widely used?

What would replace it?

Chapel[0] was created, in part to replace MPI in HPC communities. However, MPI remains dominant.

[0] https://chapel-lang.org/

I dont expect a new PL to replace a library anytime soon.

I would say Chapel was created less to replace MPI and more to provide a higher-level alternative to it that is amenable to compiler optimization.

MPI is industry standard at this point.

When I was in the university is what still pretty much PVM vs MPI fighting for attention.

I bet no one remembers what PVM was all about now.

Reply on news.ycombinator.com