Quite a few people saw this coming.
It is still early to tell if we reached AI winter again or not, but at least we can see that news are slowing down.
It is still early to tell if we reached AI winter again or not, but at least we can see that news are slowing down.
We're just scratching the surface of what's possible with the current state of the art. Even if there are no major advances or breakthroughs in the near future, LLMs and associated technologies are already useful in many use cases. Or close enough to useful where engineering rather than science will be sufficient to overcome many (though not all) of the shortcomings of current AI models. Never mind grandiose claims about AGI, there's enough utility to be gotten out of the limited LLMs that we have today to keep engineers and entrepreneurs busy for years to come.
But I think the term "AI Winter" usually refers to the underlying research programme and the economics around it. Soaking up many billions of dollars of industry money and public grants on the pitch that AGI might be just around the corner, and then being unable to deliver on that pitch can induce a hangover effect that makes it much harder to raise money for anything that smells even a little bit like the failed pitch. Investors and administrators feel burnt and turn very skeptical for a very long time.
Meanwhile, the actual productive applications which shook out of the initial boom just get renamed to something else so that they don't carry that smell.
We'll see how it goes here, but that's the familiar road and where the terminology comes from.
But you would think after 70 years AI practitioners would learn some humility! It is very obvious that GPT-4 is dumber than a honeybee, let alone a cat, let alone a crow. But for over a year I've heard dozens of tech folks insist it's smarter than most people.
Why? The people doing AI now are not the same as those that did AI 60 years ago.
We are very clearly on that same path again. What leads to the conclusion that another winter is coming. But even the fact that people are talking about it is evidence it's not here yet, and as always with political phenomena, there's no guarantee history will repeat.
Anyway, none of it means people will stop applying their knowledge or studying AI. The entire thing happens on funding and PR, and the world is not entirely controlled by those two.
The funding is there but we're letting hype drive everything and not calling out the con artists.
The problem is we've had recent great success but still don't know how to get to AGI. But because we're afraid of winter we're not willing to try new things. We want to only compare to sota and think it's fair to compare a new method with a handful of papers to the status quo. That's not how the S-curves of technology work. Sure, maybe things don't scale but that doesn't mean they don't have merit or can't scale if someone finds some modification.
The problem is we treat research like products, not academic work. You need to produce everything from hard core theory to robust products to have an effective chain. But we seem hyper focused on the middle area. And for some reason people think products can be placing a nice interface around research code. There's still a lot of work you need to do and all those models can be optimized. They absolutely do not have optimal hyper parameters or even parameters.
Yes sheer quantity has a quality of its own but that won't produce optimal results.
Incremental improvement of proven approaches, driven by profit motive, will surely continue regardless of whether there is an AI winter or not.
It's the rage because it's a way to practically _do work_ from LLMs that generally provide wow from conversationally accurate, often factually accurate responses.
There are some major limitations to LLMs that aren't going to be "optimized" away. At the end of the day LLMs are Monte Carlo samplers over a latent, compressed representation existing human text data. Many of the tasks people hope LLMs will achieve require major leaps in our current understanding of language modeling.
One great example limitation (which shocks me sometimes when I think about it): generating output is still ultimately stuck in looking at the probability of the next token rather than the much more useful, probability of the generated statement. There are techniques to improve this, but we're missing a major piece of generating highly probable statements with no hint about how to really get there. Consider how you might write SQL. You conceive of the high level query first and start sketching out the pieces. An LLM can only look at each token and can't, statistically speaking, think in terms of the entire query.
Personally I think LLMs are very underutilized/exploited for what they are good at, and there is way too much focus on what they can't do. Hopefully we'll dodge an AI winter by using LLMs to solve the wide range of classical NLP problems that make many tasks that just a few years ago nearly impossible, rather simple today. Unfortunately the irrational hype around these models makes me skeptical of that scenario.
ie., P(mat|the cat sat on the) is a distribution over, say, 100k words. Whereas, P(the cat sat|on the mat) is a distribution over 100k^3 words.
Part of the illusion of an LLM is that we produce create text in such a highly regular way that a mere distribution over 100k iterated, say 500 times, gives you a page of text as if modelling 100k^500.
There's always some space in high-dimensions to put that extra context somewhere. So yes, only one token is predicted, but there's a lot of information squeezed into the "condition" (as in conditional probability, as mjburgess' comment shows)
> [hype instead of NLP]
yes, exactly. we'll see what remains after the bubble bursts.
But I do feel confident that an AI winter is not on the horizon solely due to the overhead of implementation that we currently have. Just with currently existing AI, it would take years for the economy to fully leverage the abilities that are available. I'm confident we have transformative AI. So it won't feel like a winter for several years as we actually succeed in optimizing, implementing, and productizing current technology in other industries.
Only the poor souls who survive the fall, at the very bottom of the precipice, will get to experience the AI winter.
Sure we already have some interesting applications, but they are not exactly printing money.
Even if academic progress stalls, we're going to be inundated for at least another few years.
Again?