It was better when they had gemma at 1k. Inco does DS flash at about 600. A few places will do K3 and GLM in the hundreds.
Such a tiny model at that t/s is less impressive than it would have been four months ago.
Such a tiny model at that t/s is less impressive than it would have been four months ago.
I genuinly only see these speeds being useful for customer service/transactional workflows. Of which much smaller models can do the job (but those dont make tons of money for companies like Cerebras that need to pay off massive amounts of debt).
Nobody needs to code at 600 words per second. Using a 100tps model for an hour or so will leave you with 4-8hrs of code review and revision work.
And when things go south, you blame AI?
I would be very curious to see how you explain it to your customers.
Is it going to sound similar to this?
> You see, our well-meaning AI-generated code has caused all your data to be permanently deleted. In case you are confused as to who to blame, we would like to clarify that we did not write, nor review the code. So we cannot possibly bear any responsibility for its mistakes. The responsibility lies with the LLMs, not us. We have already fired the LLM which did the coding and the review. And we are already using their main competitors. Hopefully that settles your concern with the quality of our service and we are looking to have you on board of our next products.
This seems to work well enough for people who deploy cloud instances without redundancy to us-east-1 then blame AWS when there's an outage. I say this somewhat unironically because if one pushes the "move fast and break things" slider all the way to the right then they're necessarily assuming that type of risk. Of course some people will try to have their cake and eat it too [1] as regards velocity and quality but that's a separate discussion.
[1] I've never understood this idiom because if one isn't in possession of their cake before eating it then they're eating stolen cake which is a decidedly anti-social activity and orthogonal to the point of the saying. It should be "eat their cake and keep it too" or something.
I do. I used to use haiku for the speed. Now its just as slow as the rest. Speed is my #1 ranking of how good a model is
What is your workflow?