> on a decent speed
But you said 7-9 tokens/second, that's not a decent speed. I'm not an expert by all means but in my local experiments, less than 12 to 16 tps is too slow.
But you said 7-9 tokens/second, that's not a decent speed. I'm not an expert by all means but in my local experiments, less than 12 to 16 tps is too slow.
https://mikeveerman.github.io/tokenspeed/?rate=10&mode=text
You think of an idea that you want to have the LLM process, queue it up, and go back to what you were doing. Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete. It's kind of like using a 3D printer: It doesn't matter if a print takes 10 hours, because when you come back in the morning it will be done.
Yes, with top-tier GPU farms you can hit hundreds of tokens per second. But if the old Xeon in the closet can get useful work done at 5 tokens per second, there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source.
These are fundamentally different tools for entirely different applications. They only look similar to people who don't understand the tools or their purpose.
I swear, tech culture has gotten people wanting to work for the machines, rather than the other way round.
Let the machine do its work while I relax, I’ll check up on it later.
At 5 tokens per second and unknown prompt processing speed, you may need a very extra long lunch break depending on your codebase.
Let LLMs write the corpo code, as it will be unlikely to still be running in 5-10 years. Frontier AI is already at the point where it writes fewer bugs per LOC than humans. By a lot.
Go ahead and do your bespoke coding on your side-project loves and core libraries... The stuff that will last, anyway.
But if you're working for a corpo and still doing bespoke... That's... not gonna last, I'm afraid. Well, either you remaining there, or that, as it were.
The correctness of an application is limited by your ability to understand and describe what you need. We have a word for an application specification tool so detailed it eliminates all ambiguity. It's called a "programming language".
The mistakes are always in the transfer from human to machine. I still find a high-level programming language to be the best way to express my intent. Humans will make mistakes in the hand-off to AI just like they make mistakes in the hand-off to code, but at least code is deterministic.
This has been being claimed for at least 2 years now. Wouldn't we already be seeing disasters if this was the case? It's certainly been around long enough to cause some real damage. Instead there is a slow trickle of things that makes the news by people who didn't bother instituting a single fucking control. For example, I am completely immune to `rm -rf` style fuckups because I wrote rm-safe years ago and it is mapped to `rm` in any environment that the LLM will run in: https://github.com/pmarreck/rm_safe
In any event, it's not a replacer, it's an augmenter. Nothing will replace humans, because we are the stakeholders; it may shift them around, though.
> The correctness of an application is limited by your ability to understand and describe what you need.
If you write code in Lean 4 or Idris 2, you may not completely understand why it is or isn't correct, but their respective compilers will certainly prove it to you one way or the other.
We already are perfectly functional with incomplete understandings. An entire generation of web developers have had successful careers without having a single clue how compilers work or how machine code works. Now is there a type of problem that sometimes comes along that DOES require deep understanding? Of course, that's when you call in the heavy artillery. Or the advanced LLM. Or both. See my point?
No, the prover can only prove that the implementation matches the formal language specification. That's different from the application being correct.
Likewise for image assets for CMS and ecommerce sites, now instead of hiring agencies, many of those images are AI generated.
Just because it hasn't come yet to some coder jobs, doesn't mean others haven't been already displaced by it.
Keep your mental context in your brain is critical
With a fast enough model, I can iterate on the UI of a given screen 4-5 times before Opus finishes its first attempt.
Working with models that are super fast, but slightly dumber (like mimo-v2.5-pro-ultraspeed) is amazing, I feel like I'm still the one that's actually making every decision.
If I spend 10 minutes reading an article, that would only generate 3000 tokens.
That’s not counting the prompt processing time.
We have very different expectations for LLMs if your tasks only take a couple thousand tokens and you’re happy waiting 10 minutes for it.
> Yes, with top-tier GPU farms you can hit hundreds of tokens per second
My 5090 gets hundreds of tokens per second with this model. No farm needed. I’d have to double check but I think even a $1000 Intel B70 might break 100 tokens per second.
> But if the old Xeon in the closet can get useful work done at 5 tokens per second, there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source.
If that old Xeon pulls 200W from the wall and you pay national average electricity costs, it’s going to cost $0.90 per day to run it.
I would rather pay a dollar per day, get my answers 100X faster, and not have an old Xeon heating up my house.
This is an option if you must run local inference, you’re not sensitive to speed, and the budget is low.
It’s not going to be cheaper than paying API prices for the model though.
It is common for agents to just stop because overload or some API error hijinks.
Or you get a TUI question that is blocking.
In general you’re right though, staring at tokens from agentic is not time well spent.
Some of these I’ve built custom harness around in iterm2 though.
I watch tokens to see if it goes in right direction. If model goes off the rails, then it is time to stop and adjust prompt.
This is how I used to think about my 3D printer, but FWIW the way my actual thinking and planning works, print speed really matters. Not for the final print, but for iterative work and test parts, it is obvious that either having a fast printer helps. Having multiple slow printers also helps, but there are only so many areas of a design you can iterate on at once.
At the moment my own LLM use is experimental and iterative, and I definitely favour the faster MoE models for much of what I am doing, even if I might in principle prefer to get the final work done in the slower ones.
That Xeon is going to be devouring power.
Pending any new hardware or radically different LLM architectures, we're going to be waiting a lot longer than 2027 for 200B models on local hardware. SOC platforms like Apple Silicon are hamstrung by their obsession over raster performance, the hardware lacks the fundamental GPGPU hardware to be a replacement for real-world inference.
Instead of coding agents like tool for local models I would like to see more "docker ps" like tools where you can queue up tasks that get processes incrementally (maybe when the pc i idle) and are specialized for doing retries and caching as much work as possible to work decently with slow local models.
It says so right in the readme. They’re not hiding anything.
Like, obviously faster is better and the entire point of computers is to do things faster, but I find it kind of surprising how many people consider 9 tokens per second—heck, even much less than that—to be unusably slow. It's still automating a class of task that virtually no one before 2022 was able to automate, and it's faster than basically any human can write code.
I guess the better question is, why would you use a 9 token per second system when you could use a much faster cloud model. Obviously, if you want speed, that's the way to go. But a lot of people seem to find the idea of sending their requests to a third party server untenable. If 9 tokens per second is the best you can do—I don't know, that seems usable and useful to me.
The biggest argument in favor of running local models doesn't seem to be privacy at all, it's the fact that you can't run out of tokens even if it is a bit slow.