Ask HN: What local machines are people using to train LLMs?

I am using attention weighted pooling strategy to generate cross-attentions across N transformer heads, this allows us to visualize which parts of an output were caused by which part of the input -- this is used to debug summarization. Only works for encoder/decoder networks - i.e. ones where you have cross attentions

Thank you! The animation at the top is a GPU shader of the transition between 1s, 2p and 3d electron wave functions in a hydrogen atom, the animated crystal of sulpher I made using Maya and then animated into/out of wireframe by loading it and spinning it in WebGL, the other ones further down are just screen recordings.

You might like this too - this last weekend I finished writing a topological contour GPU shader for visualizing electron density grids, here is a molecule of buckminsterfullerene - the contour generator is running on the GPU, which is how I can keep the framerate so high.

https://drive.google.com/file/d/1xZkXsWDZtpe0H3BKSkdzvjOA3UE...

There's little between the M2 and M3 at this point with regards to LLM work, and given no M3 Ultra exists yet, the M2 Ultra would be the way to go considering this task in isolation, and indeed the only option if you wanted to go up to 192GB RAM. If you needed to be portable or had budgetary concerns, then your choice may differ.

No, they have far less bandwidth. However, I am not sure if getting Apple CPUs would be a good idea. You can get some toy projects running with them, but a pay-as-you-go cloud solution would have been cheaper and faster. I don't find my M1 Max to be particularly competitive.

AMD and Intel allow their customers to choose lots of different memory configurations and so they prioritize compatibility over bandwidth. Apple's architecture has no support for external memory, they put their memory directly on package and can prioritize for bandwidth.

i'm not sure, but I bet you could get an answer to a more specific hardware question like this over at /r/locallama

Much better these days and easier to do. Fans are aso much quieter these days too. A few years ago I was twisted into building a water cooled GPU system because it was the only 30 series card I could get in a 6 month period. Wasn't too hard to build into a system that doesn't even have enough room for the water pump to stand vertically. Soft tubing is annoying though and will fatigue with time, restricting flow rates. So I suggest if you do that to take more time to ensure you have angled fittings where you need them and to zip tie straight paths where you can (and thicker tubing is better). Might just be easier to go hardline instead tbh.

Thermals are great though. But tbh now what I do is put my machine behind my TV and I have the side open and place the pump outside to stand vertically like old systems that had them outside the case lol. That's because if I'm not playing games on it then there's no difference in just sshing into the system because I live in the terminal anyways.

Pretty straightforward these days! Though I've been doing it for awhile. There a lot of little gotcha's like tubing choice and fittings but once you get the hang of the options it's pretty fun.

Though cooling 4090s even with water is hard - you need a good flow rate to dissipate 450W. They'll hit near 70C during training if the fans are kept below audible, which works for me. 55-60C with fans at 50%. This is with 2x420mm (3x 140mm fan) rads and one 280mm rad (2x 140mm fans).

The stock heatsinks actually were actually able to cool them just as well, just they where much louder doing so!

Power is the other tricky bit. The computer is living on the dedicated AC circuit for now... 1600W is a lot. When it was on the single outlet circuit in my house, I could see each iteration as the lights dimmed in sync with the power draw surging.

For inference and fine tuning yes, not training.