Hacker News Reader

Comment by chillee | Hacker News Reader

Hey, author of the blog post here. It's mentioned in the blog post, but one of the intentions of this repo is that it's more of a "tutorial" than it is a library/framework. My hope is that people will copy-paste and modify it for their own needs :)

Code can also be found here: https://github.com/pytorch-labs/gpt-fast

And a twitter thread summary here: https://twitter.com/cHHillee/status/1730293330213531844

View on HN

Great work! Do you know if it's possible to port this over to pytorch's Apple Silicon/MPS support?

Unfortunately it's a little bit tricky today. The main issue is that we rely heavily on torch.compile + Triton for performance in this repo, and there isn't an Apple Silicon backend either for torch.compile or Triton.

For example, there's an AMD backend for Triton (and it's also integrated into torch.compile), which is why we can mostly do the same optimizations on Nvidia and AMD GPUs.

Ideally, there'd be an Apple Silicon backend for Triton, and then this repo would mostly work out of the box :)

What kind of workstation would you build/buy for local GPT development with a budget of $3000? Is remote dev a viable alternative to local workstations?

Local workstation is much cheaper in the long run.

Even ignoring that, most of the development is running experiments. You're gonna be hesitant to run lots of experiments if they each cost money whereas when you pay upfront for the hardware, you're gonna have the incentive to fully utilize it with lots of experiments.

I'd go with rtx 4090 and deal with memory limitation through software tricks. It's an underrated card that's as performant as cards that are magnitude pricier. It's great way to get started with that budget.

I agree with you but right now RTX 4090 cards are pushing $2000, which doesn't leave much budget left. I'd suggest picking up a used 3090 card from eBay, which are currently around $800. This will still give 24gb of VRAM like the 4090.

i've seen some blog posts saying if you buy a used 3090 that has been used for bitcoin mining then there is a risk of thermal throttling because the thermal paste on the vram is not great and worse if it was run hot for a long time.

any recommendations on how to buy one? e.g. 24GB model, any particular model to run LLMs? what is the biggest baddest LLM you can run on a single card?

have been thinking about it but was sticking with cloud/colab for experiments so far.

The good deals are gonna be on local ads. Facebook Marketplace in most of the US.

Craigslist and eBay have some great deals.

I remember videos (on youtube likely) of thermal paste replacement, that was upgrade to stock card. So, average person should be able to do it. It'll cost a few $$ for the paste. I would go with local workstation, then don't have to think much about while running stable diffusion. Plus, if it's used from ebay, prices cannot go much lower, you'll get something back at the end. Also, for image things training dataset can be quite big for network transfers.

Strong endorse here. I pick up used RTX 3090s from Facebook Marketplace and eBay at $800 maximum. Can usually find them locally for $700-750, and typically can test them too, which is fine (though I've had no issues yet).

Depending on what you're doing, 2x used 3090s are the same price and offer you more VRAM. That's what I'm planning on doing, in any case - being able to run 70B LLMs entirely on the GPU is more useful than being able to run 34B faster.

Yeah multiple 3090s is the best budget way to go for sure. Also older server boards with tons of PCIe lanes if you can swing rack mounted hardware and have some technical skills.

Agreed. I recently completed a new build with two 3090 GPUs and really appreciate being able to run 70b models.

which cpu did you go with?

i7-14700k

z790 chipset w/ mobo that supports x8/x8 bifurcation

96gb ddr5 @5600mhz

Not OP, but I asked myself that same question two years ago. Then I looked at the energy prices in Germany and knew I had no chance against cloud GPUs. Maybe you live in a country with lower energy prices, like Bermuda (or any other country on earth), in which case this may not be as important to you. A side benefit of going cloud that you can pick and choose the right GPU for whatever project you’re working on, and you’re really just paying while you’re running them. Also, no hardware or Cuda drivers that may divert your attention.

I’d go with a remote dev solution. Training/finetuning of large models requires much more resources anyway, so the GPUs in the local machine would be unused most of the time.

I got a 13900k + 4090 workstation for ~$3500. But I hear what people are doing is getting 2x (or more) 3090s instead, because they are cheap used, and having more VRAM and VRAM bandwidth is the important thing at the moment, even if it is split between cards.

I'm happy with my 4090 though. Dealing with splitting between GPUs sounds like a chore and also I like the gaming abilities of the 4090.

I would do remote dev using vast.ai and other cheap cloud computing resources to ensure you want to do this and have utility for it, then build your own. 3090s are typically the most budget friendly, and if you have any IT chops (and tolerance for noise), then server rack-mounted hardware, PSUs, and riser cables tend to be the most efficient with tons of PCIe lanes (which is a hidden issue people have with consumer-grade gaming PCs as they scale).

I made a custom pc in May with a 4090 for playing around. It's good and fast but for production workload or serious training you want more ram. It was less than ~3k in components (~2k after considering removing vat and expending it / depreciating it in a limited company)

If you also want a Mac consider an m3 MacBook with maxed ram.

I'm using an AMD 6900XT with ROCm and it's fast enough ti be usable, for a fraction of the price of a 3090 or 4090.

Just make sure you’re buying something with a warranty. Hardware failures during training are quite common.

A regular gaming PC will do, about a grand, then slot in a 3090 or something off eBay. Congratulations, you're a grand under-budget, but already have a viable development machine.

Just wanted to share, the charts and gifs are exceptionally well done. Informative, concise, and easy to read.

Thanks! I've also written a couple other things along a similar vein you might like at https://horace.io/writing.html (particularly https://horace.io/brrr_intro.html) and also some of the things I've tweeted: https://twitter.com/cHHillee/highlights

Hi, Thanks for Open sourcing the code! I was trying to reuse the code especially the dynamic quantization per channel (int8 on gpu) but couldn't get it to work, i also checked out torchao package but it looks like it has dependency on the nightly channel and SAM's dynamic implementation with triton has other issues, is there any clean implementation of int8 dynamic post-training quantization that you can point too ?

What’s the issue with getting int8 dynamic quantization to work? As in, you’re unable to get it to quantize or to run with speedups?

What's the difference between, say, Karpathy's nanoGPT and your GPT implementation?

Is it just a difference in speed, or are there some new theory details to learn?

Thanks for sharing your work.

Just a difference in speed. This repo is primarily showing how you can get really good inference perf with just native pytorch.

Great work and a really useful resource! Comprehensive guides on improving PyTorch performance are pretty hard to come by, and I learned a couple new tricks from this!

Fantastic article, great job. One small note: eke and eek are not the same word.

What GPU was used when testing this?

Is this faster than HuggingFace's Text Generation inference container?

We used an A100-80GB GPU. We didn't compare explicitly to Huggingface TGI but I think you should be able to compare the tokens/s achieved.

One note is that this release is optimized for latency, while I think HF TGI might be more optimized for throughput.

Reply on news.ycombinator.com