Does not look like very fancy or innovative.
In the end, they just put a dedicated coprocessor directly with their memory chip. They named it AI because buzzword and marketing bullshit...
In the end, they just put a dedicated coprocessor directly with their memory chip. They named it AI because buzzword and marketing bullshit...
They named it AI because it massively boosts embarrassingly parallel workloads. You can think of Processing In Memory as rendering the mapPartitions() operation free in Spark's MapReduce ML workloads.
Some algorithms like DNA sequencing have a tradeoff between map and reduce [1]: you spend more time generating higher quality matches between the short sequences (map), before sending them for global matching (reduce). And PIM lets you exploit that.
For an order of magnitude: the average Intel has about 60GB/s of RAM bandwidth per socket. 256 GB of UPMEM's RAM let you have 2.5TB/s of local bandwidth to a computation unit (to 2560 'dumb' cores @400Mhz) [2].
[1] https://www.researchgate.net/publication/346703874_Variant_C...
It strikes me these processors would be most helpful in pre-multiplies, filter operations, and perhaps for scatters. All that stuff is not just relevant to tensorflow / pytorch stuff but also databases. While I’m sure the “AI” labeling is pure marketing, I’d imagine Samsung would love to target workloads beyond deep learning training and inference.
Yes! and that's the beauty of it. It is not an accelerator, these are fully generic cores.
Not equivalent to 'smart' Intel cores with all the branch prediction, prefetching and caching magic; but with massive computation capabilities nonetheless.
GPUs do have massive amounts of memory (both in RAM and registers), but you have to have preloaded your stuff into it beforehand. And what you can actually do efficiently are SIMD operations.
I'd liken PIM to a better GPU-CPU blend: you get to keep your CPU doing its things with massive parallel operations concurrently. Also, these seem to be mostly independent cores, so you would not be limited to SIMD.
Let's bet: in 10 years, AWS will have a new offering: the 'nano lambda'. You get a PIM core share, with 10 MB local 'persistent' RAM (keeping your data + a continuation of your code when it is not running), running your tiny Loom thread [1], at the edge, billed at 1us granularity, only when it is running, and for 0.0000000000000001 USD per us.
[1] https://cr.openjdk.java.net/~rpressler/loom/Loom-Proposal.ht...
When could it happen? My guess is S3 favors very old compute hardware but I could be wrong. If it does, though, that could also mean it's just very cheap to replace ;)
They felt the wind turning, saw the market ask for it, saw companies like UPMEM having a lead.
If you want more details, UPMEM has more. If I recall correctly they etched their own core right into the same silicon as the DRAM.
Here is a link that I think will satisfy your curiosity: https://www.upmem.com/nextplatform-com-2019-10-03-accelerati...
Looking at the ISSCC program, I presume this is what they're talking about: "A 20nm 6GB Function-In-Memory DRAM, Based on HBM2 with a 1.2TFLOPS Programmable Computing Unit Using Bank-Level Parallelism, for Machine Learning Applications" [1]. I suspect this is where there will be a deep dive into the gory technical details of the architecture and performance. It's tomorrow so it won't be a long wait.
[1] https://submissions2.mirasmart.com/ISSCC2021/PDF/ISSCC2021Ad... (p 38)
Also, a GPU requires transport of data to it. PIM lets the CPU do concurrent work at very low latencies.
(Meaning, relatively small chiplet AI processors with ram stacked on top of them.)
The reason for that is that as the precision used for the coefficients has gone down, the relative energy cost of doing computation on them has turned into a rounding error when compared to the cost of moving data to the alus, and in AI there is very little cost of distributing the processing power into many small chips, which are relatively far away from each other.