Hacker News Reader

Comment by knicholes | Hacker News Reader

I've got a machine with 4 3090s-- Anyone know which model would perform the best for programming? It's great this can run on a machine w/out a graphics card and is only 3B params, but I have the hardware. Might as well use it.

View on HN

Try mistral 8x7b, which some human evals place above gpt-3.5 and you have enough VRAM and compute to make training a LORA either on your own dataset, or one of the freely available datasets on huggingface worthwhile, or at least interesting

AFAIK deepseek coder family are the best open coding models.

I haven't tested, but I think deepseek coder 33b can run in a single RTX 3090 when 4-bit quantized. In your case you might be able to run the non quantized version

The coding models are all small because speed is crucial. If you need to wait 2 seconds for an autocomplete it becomes near useless.

Here is a leader board of some models

https://huggingface.co/spaces/mike-ravkine/can-ai-code-resul...

Don't know how biased this leaderboard is, but I guess you could just give some of them a try and see for yourself.

This is a much better leaderboard: https://evalplus.github.io/leaderboard.html

I've seen the CanAiCode leaderboard several times (and used many of the models listed), but I wouldn't use it to pick a model. It's not a bad list, but the benchmark is too limited. The results are not accurately ranked from best to worst.

For example the deepseek 33b model is ranked 5 spots lower than the 6.7b model, but the 33b model is definitely better. WizardCoder 15b is near the top while WizardCoder 33b is ranked 26 spots lower, which is a wildly inaccurate ranking.

It's worth noting that those 33b models score in the 70s for HumanEval and HumanEval+ while the 15b model scores in the 50s.

Thanks

Did you build a machine with 4x 3090 ? I looking for a way to build such a machine for ML training.

Wondering the same here

I did! I started by going to vast.ai. I was able to look at the specs of the top-scoring machines. I started with the motherboard (as I knew it could support my 3090s, because some PCIe busses can't handle all that data). Then of course I copied everything else that I could. I ended up using PCIe extenders and zip-tieing (plastic, I should use metal zip ties instead) the cards to a rack I got from Lowes. I'm not too pleased with how it looks, but it works!

BTW, depending on where you're at in your ML journey, Jeremy Howard from FastAI says you should focus more on using hosted instances like paperspace until you really need to get your own machine. Unless, of course, you enjoy linux sysadmin tasks. :) It can get really annoying trying to match the right version of CUDA with the version of Pytorch you're trying to get running for the newest model you're trying.

Here are the parts I can find CPU: https://www.newegg.com/amd-epyc-7252-socket-sp3/p/N82E168191... Motherboard: https://www.newegg.com/asrock-rack-romed8-2t/p/N82E168131400... SSD: https://www.newegg.com/samsung-2tb-980-pro/p/N82E16820147796... Computer Power Supply (the GPUs have their own power supplies, run apart from the computer, which I'm told is bad, but... seems to work for years now for me): https://www.newegg.com/corsair-hx-series-hx1200-cp-9020140-n... PCIe 4.0 x16 Risers: https://www.newegg.com/p/N82E16812987068?Item=N82E1681298706... Tower: https://www.newegg.com/black-fractal-design-meshify-2-compac... RAM: https://www.newegg.com/nemix-ram-16gb-288-pin-ddr4-sdram/p/1...

Total, without GPUs and their power supplies: $2900

Thanks man! Super helpful.

Reply on news.ycombinator.com