Amazon's Trainium2 AI Accelerator Features 96 GB of HBM, 4x Training Performance | Hacker News Reader

Amazon's Trainium2 AI Accelerator Features 96 GB of HBM, 4x Training Performance

anandtech.com

45 pointsksec31 comments

AMD is coming out with the MI300x on Dec 6th, it has 192 GB of HBM3 and you put 8 of them in a chassis all connected to each other with super fast connections (5.2 TB/s of memory bandwidth) and each chassis is networked with 800-900G connections.

We don't know what the tflops will be for the 300x yet, but it seems like the limitations here are more about memory than it is about tflops.

Those are all impressive numbers, but did they actually fix their software stack? Are there people here that have actually trained with those and if so how was the experience?

I'm genuinely curious because it seems like every other AI Accelerator/GPU thread on HN mentions MI300X, yet I haven't talked to/met anyone who trained anything with AMD.

The MI300x aren't released yet, that's why it isn't popular. Azure is building a cloud [0]. I know someone else personally who's building a cloud too.

As for training, it has been happening a bit on MI250's [1]. There is also A LOT that happens that doesn't make the larger news [2] and [3] and [4].

[0] https://techcommunity.microsoft.com/t5/azure-high-performanc...

[1] https://www.databricks.com/blog/training-llms-scale-amd-mi25...

[2] https://www.amd.com/en/resources/case-studies/kt-cloud.html

[3] https://www.lamini.ai/blog/lamini-llm-finetuning-on-amd-rocm...

[4] https://twitter.com/cHHillee/status/1730293330213531844

https://news.ycombinator.com/item?id=38479616

The issue with AMD (and frankly AWS' custom accelerators) is more on the software side usually. ROCM is getting better but still lags far behind CUDA in adoption and often performance.

With things like inferentia, although the performance numbers they have reported are really impressive-- in practice, model compilation is super finnicky. I'd love to see how seamless they can make usage of trainium.

What you said gets repeated many times.

Fact is that AMD was caught out of the initial AI race and is catching up. Lisa has said over and over again that she is committed to it.

Not sure of the relationship between MI300x and MI300, but MI300 is claimed to be 2.2 PFlops FP8 (with structured sparsity, which I did not know AMD was implementing!!!): https://www.amd.com/en/claims/instinct

There are 3 versions of MI300.

X is pure gpu, A is more like an APU (cpu+gpu), C is pure cpu.

Will be a long time before a large number of these are available in clouds though, not until summer at least.

I predict sooner, but that's cause it is what I'm working on now. ;-)

In my experience AWS Nvidia machines are extremely overpriced. $30/hour for 8 A100s, and you have to pay for all 8 at a time. Most competitors I have seen are around half that price.

At $30/hour I’m literally spending a dollar or two just installing dependencies.

Whether AWS custom hardware is reasonably priced, and how much of a pain in the butt it will be to port my code, and how training speed compares to Nvidia GPUs will be my main deciding factors. I’m not locked in to AWS so Trainium will need to be faster and cheaper (cheaper per epoch) to be competitive for me.

Can't you install dependencies on a potato instance then change the instance to the GPU when you're ready to roll?

That only helps if your dependencies are slower moving targets than whatever research/product you are working on. Otherwise, the dependencies may be different each experiment or attempt.

Or whatever cloud provider you use does something funky and the setup on one machine is less portable to a GPU machine than you might expect.

I was wondering the same thing of why not have an image ready to go, but then realized why take away the easy complaint with rational logic?

The $2 dependency installation cost isn’t the problem, it’s the $30/hour training cost.

A month with an a2-ultragpu-8g is going to cost you ~$6400/month or about $36.92. Where are you getting half price?

As someone from outside the AI space, how useful are chips like this from Amazon? Meaning am I going to have to rewrite my entire stack to make use of this and then be locked into Amazon forever? Or are workloads at least portable in some capacity?

We tried Inferentia & though they claim their compiler should "just work" on your existing models it was producing a bunch of NaNs for us & was opaque enough that we didn't really have a path forward so we gave up on it.

If we were operating at 100x or 1000x the scale it might make sense to spend the extra engineering time, but we just pay for the NVIDIA chips (especially because a big chunk of our customers run at the edge where NVIDIA compatibility is even more important so we'd be doing double the engineering on an ongoing basis).

There’s semi lock-in. You can use torch, but you also have to use their framework. Not much different than using cuda I guess.

It's a bit of a mixed bag. NeuronSDK is implemented with Torch XLA so there's not real lock-in there. Your code will usually just work on CUDA if it works in Trainium. However for larger models (ZeRO, Sharding, etc...) they have their own libraries that are more coupled to the actual accelerator topology because using something like DeepSpeed was not possible.

Overall the challenge is often more with making your code work on their accelerator, not porting it away from their accelerators so I wouldn't be too concerned about lock-in.

not very useful, but it's getting better with XLA. It does not just "work" by default for any given model, and if you're missing an HLO that has an optimized kernel written you lose benefits of accelerated computing thus people won't use it. I would not put any effort into it until they truly have it where you just cast `to_device(..)`.

I hope the development of all these accelerators drives developers and frameworks to move away from CUDA only and even better to move to a open standard of some sort (eg. ROCM, if it can atain parity with CUDA in terms of usability). It would be such a waste for each framework to need to have a seperate backend for each hardware vendor. Then we will end up in the same desperate situation as game engines with Vulkan, Metal and DX12 with a side of OpenGL and Intel OneAPI (for things like eg. Blender). That sadly seems to be the way it's going right now.

Trainium and Inferentia sadly aren't going this way, they have their own approach https://github.com/aws-neuron/transformers-neuronx . I think the best case scenario is some middleware like Triton https://openai.com/research/triton becomes the standard and we can build adapters to all of these backends. It is disappointing though.

650 TOPs Is surprisingly low for narrow precision datatypes - I assume they are quoting the fastest performing datatype (Int8 or maybe even lower!). That's on par with an A100, an H100 hits 2 PETA TOPs (4 PTOPs 2:4 sparse!) of Int8!

Everyone was trying to get their hands on A100s right until the H100 was released. If Amazon can replace A100s that’s a big win.

What happens to all this data center hardware when Amazon or others get rid of it? Do they sell it or do they shred it?

The answer is 'yes'. It probably depends on how they accounted for it on their taxes. Some sold, some destroyed. You can usually find bunches of it on ebay. Unless it is 'custom' then it will most certainly be destroyed as there is no market for it.

How do they know there's no market for their outdated in-house silicon? It's a shame that all this compute is being manufactured only to go to the shredder.

who would need something very used/abused without software support?..

Also some companies consider the bespoke ones 'IP' and do not want it getting out.

Its not necessary they don't want, IP has cost attached in accounting, and they would need to explain for tax and investment reporting why they give this IP away.

Reply on news.ycombinator.com