Hacker News Reader

All those choices seem to have very different trade-offs? I hate $5,000 as a budget - not enough to launch you into higher-VRAM RTX Pro cards, too much (for me personally) to just spend on a "learning/experimental" system.

I've personally decided to just rent systems with GPUs from a cloud provider and setup SSH tunnels to my local system. I mean, if I was doing some more HPC/numerical programming (say, similarity search on GPUs :-) ), I could see just taking the hit and spending $15,000 on a workstation with an RTX Pro 6000.

For grins:

Max t/s for this and smaller models? RTX 5090 system. Barely squeezing in for $5,000 today and given ram prices, maybe not actually possible tomorrow.

Max CUDA compatibility, slower t/s? DGX Spark.

Ok with slower t/s, don't care so much about CUDA, and want to run larger models? Strix Halo system with 128gb unified memory, order a framework desktop.

Prefer Macs, might run larger models? M3 Ultra with memory maxed out. Better memory bandwidth speed, mac users seem to be quite happy running locally for just messing around.

You'll probably find better answers heading off to https://www.reddit.com/r/LocalLLaMA/ for actual benchmarks.

View on HN

Started building llama.cpp from source, history: 21 git clone https://github.com/ggml-org/llama.cpp 22 cd llama.cpp 23 which cmake 24 sudo apt list | grep libcurl 25 sudo apt-get install libcurl4-openssl-dev 26 cmake -B build -DGGML_CUDA=ON 27 cmake --build build --config Release

28 cmake --build build --config Release -j 16 29 ls 30 ls build 31 find . -name "llama.server" 32 find . -name "llama" 33 ls build/bin/ 34 cd build/bin/ 35 ls 36 ./llama-server -hf ggml-org/gpt-oss-120b-GGUF -c 0 --jinja

37 clear;history 38 ./llama-server -hf Qwen/Qwen3-VL-30B-A3B-Thinking -c 0 --jinja --port 11434 39 ./llama-server -hf Qwen/Qwen3-VL-30B-A3B-Thinking.gguf -c 0 --jinja --port 11434 40 ./llama-server -hf Qwen/Qwen3-VL-30B-A3B-Thinking-GGUF -c 0 --jinja --port 11434 41 clear;history

* Claude in December: 91 million tokens in, 750k out * Codex in December: 43 million tokens in, 351k out * Cerebras in December: 41 million tokens in, 301k out * (obviously those figures above are so far in the month only) * Claude in November: 196 million tokens in, 1.8 million out * Codex in November: 214 million tokens in, 4 million out * Cerebras in November: 131 million tokens in, 1.6 million out * Claude in October: 5 million tokens in, 79k out * Codex in October: 119 million tokens in, 3.1 million out

* most fixed cloud subscriptions will run out really quickly and will be insufficient (Cerebras being an exception) * if paying per token, you *really* want the provider to support proper caching, otherwise you'll go broke * if you have local hardware that is great, but it will *never* compete with the cloud models, so your best bet is to run something good enough, basically cover all of your autocomplete needs, and also with tools like KiloCode an advanced cloud model can do the planning and a simpler local model do the implementation, then the cloud model validate the output