I think a lot of people would be very content if they never got smarter, and just kept getting even cheaper/faster. Of course, both things continue to happen on a seemingly monthly basis
It was so amazing to get advices and reflect that it struck me : I could use this model forever - it’s clever enough to help me tons and do lot of work for me - even if ai would stop evolving I would love it
And this is inherent to how LLMs work.
Having a natural language interface where you need to go out of your way to specify that you want an accurate answer rather than just a plausible one defeats the entire purpose of it being a natural language interface for normal people. In certain professional contexts, it can be useful, but I don't buy it at all that it makes sense to ask everyone in their everyday lives to go out of their way to specify that they actually want correct answers to their questions.
You don't. It goes in the system prompt.
Take that away and you'll barely be able to make an app that display a pigeon riding a bicycle (or whatever you ppl are doing these days).
There's a lot of truth to this. I think we're starting to approach the point where increased intelligence has declining marginal returns, such that it might not even be worthwhile to improve models unless it can be done cheaply.
All current devices used to run AI are very far from an efficient solution to the problem. What you really want is a pure dataflow architecture, instead of a von Neumann machine. The reason people aren't really making them yet is that when you build one, even if you use SRAM for the weights, you are binding yourself to the dimensions of the model you target -- your chip is only ever going to run variants of that specific model. And SRAM is much more expensive than ROM, so if you want to make a cheap version, you need to design a specific model into silicon.
Once model improvements taper off, the next thing that will happen is everyone will chase speed. There is no physical reason why a mid-sized model could not run at >1 million tokens per second on leading edge silicon, if all computation that can be parallelized, is. No-one will go straight to that, even for a mid-sized model that's like 20 distinct reticle-limited chips. But something like the next version of Taalas HC1 (presumably called HC2?) will probably boost a ~30B parameter model to ten of thousand of tokens+ per second from a single stream within 12 months.
It's highly likely that over the next 10 years we find demand and loss of production further constrains supply.
Designing a specific model into silicon sounds like one of the worst possible ideas. No better way to freeze assumptions and limit growth. Software defined solutions dominate for a reason, because adaptability is key.
Models were never the answer. Eventually we'll get past the nonsense of observationally inefficient neural nets.