Qwen coder 32b instruct is the state of the art for local LLM coding and will run with a smallish context with that on a 64GB laptop with partial GPU offload. Probably around .8 tok/sec.

With a quantization of it you can run larger contexts and go a bit faster. 1.4 tok/sec at 8b quant with offload to a 6GB laptop GPU.

Speculative decoding has been being added to lots of the runtimes recently and can give a 20-30% boost with a 1 billion weight model running the speculative token stream.

The free version of chatgpt is better than your 70b LLM, whats the point?

Partly because I can, because unless you go absolutely wild with excess it’s the RAM equivalent of fuck-you money. (Note it’s unified though, so in some situations a desktop with 48GB main RAM and 16GB VRAM can be comparable, and from what I know about today’s desktops that could be a good machine but not a lavish one.) Partly because I need to do exploratory statistics to say ten- or twenty-gigabyte I/O traces, and being able to chuck the whole thing into Pandas and not agonize over cleaning up every temporary is just comfy.

I have 128gb in PC (largely because I can) and android studio, a few containers and running emulators will take a sizable bite into that. My 18gb MacBook would be digging into swap and compressing to get there.

Memory can get eaten up pretty quickly between IDEs, containers, and other dev tools. I have had a combination of a fairly small C++ application, clion, and a container use up more than 32GB when combined with my typical applications.

Running 128 GiB of RAM on the box I am typing on. I could list a lot of things but if you really wanted a quick demonstration, compiling Chromium will eat 128 GiB of RAM happily.

Nobody ever regretted having extra memory on their computer.

Several Electron apps and 1000+ Chrome tabs. (just guessing)

Maybe because of electron apps