How to run Deepseek-R1 locally for only $6000

https://app.bsky.cz/profile/carrigmat.bsky.social/post/3lgso...

At last, a Bluesky option. :)

That didn't work for me. This works though: https://bsky.app/profile/carrigmat.bsky.social/post/3lgsoqsx...

https://threadreaderapp.com/thread/1884244369907278106.html

Any idea about the number of tokens/s this build will give?

He said 6-8 tokens/second

So a single RTX 4080 ain't gonna cut it. :-(

This is about the price of 3 5090's, but it can run the full 670B model with Q8 quantitization, albeit at a few tokens per second. Doing this on GPUs would require 24 5090's (if nvlink still works, but I don't think so) or 10 H100's.

login-walled

Motherboard: Gigabyte MZ73-LM0 or MZ73-LM1. We want 2 EPYC sockets to get a massive 24 channels of DDR5 RAM to max out that memory size and bandwidth. (https://www.gigabyte.com/Enterprise/Server-Motherboard/MZ73-...)

CPU: 2x any AMD EPYC 9004 or 9005 CPU. LLM generation is bottlenecked by memory bandwidth, so you don't need a top-end one. (https://www.newegg.com/p/N82E16819113865)

Get the 9115 or even the 9015 if you really want to cut costs

RAM: This is the big one. We are going to need 768GB (to fit the model) across 24 RAM channels (to get the bandwidth to run it fast enough). That means 24 x 32GB DDR5-RDIMM modules. Example kits (https://v-color.net/products/ddr5-ecc-rdimm-servermemory?var...) and (https://www.newegg.com/nemix-ram-384gb/p/1X5-003Z-01FM7)

Case: You can fit this in a standard tower case, but make sure it has screw mounts for a full server motherboard, which most consumer cases won't. The Enthoo Pro 2 Server will take this motherboard (https://www.newegg.com/black-phanteks-enthoo-pro-2-server-ed...)

PSU: The power use of this system is surprisingly low! (<400W) However, you will need lots of CPU power cables for 2 EPYC CPUs. The Corsair HX1000i has enough, but you might be able to find a cheaper option (https://www.corsair.com/us/en/p/psu/cp-9020259-na/hx1000i-fu...)

Heatsink: This is a tricky bit. AMD EPYC is socket SP5, and most heatsinks for SP5 assume you have a 2U/4U server blade, which we don't for this build. You probably have to go to Ebay/Aliexpress for this (https://www.ebay.com/itm/226499280220)

SSD: Any 1TB or larger SSD that can fit R1 is fine. I recommend NVMe, just because you'll have to copy 700GB into RAM when you start the model, lol