Unfortunately, because ML models went brr some time ago (Norvig was at the leading edge of this when he worked on the early google search engine and had access to huge amounts of data), we've since seen that probabilistic approaches produce excellent results, surpassing everything in the NLP space in terms of producing real-world sysems, without addressing any of the issues that the NLP folks believe are key (see https://en.wikipedia.org/wiki/Stochastic_parrot and the referenced paper). Personally I would have preferred if the parrot paper hadn't also discussed environmental costs of LLMs, and focused entirely on the semantic issues associated with probabilistic models.
I think there's a huge amount of jealousy in the NLP space that probabilistic methods worked so well, so fast (with transformers being the key innovation that improved metrics). And it's clear that even state-of-the-art probabilistic models lack features that NLP people expected.
Repeatedly we have seen that probabilistic methods are the most effective way to make forward progress, provided you have enough data and good algorithms. It would be interesting to see the NLP folks try to come up with models that did anything near what a modern LLM can do.
powerful response but.. "fit for what purposes" .. All of human writings are not functionally equivalent. This has been discussed at length. e.g. poetry versus factual reporting or summation..
Nevertheless there is something to be said for classical linguistic theory in terms of constituent (or dependency) grammars and various other tools. They give us much simpler models that, while incomplete, can still be fairly useful at a fraction of the cost and size of transformer architectures (e.g. 99% of morphology can be modeled with finite state machines). They also let us understand languages better - we can't really peek into a transformer to understand structural patterns in a language or to compare them across different languages.
Although I'd also offer a slightly different lens through which to look at the reaction of other researchers. There's jealousy, sure, but overnight a ton of NLP researchers basically had to come to terms with the fact that their research was useless, at least from a practical perspective.
For example, imagine you just got your PhD in machine translation, which took you 7 years of laboring away in grad/post grad work. Then something comes out that can do machine translation several orders of magnitude better than anything you have proposed. Anyone can argue about what "understanding" means until they're blue in the face, but for machine translation, nobody really cares that much - people just want to get text in another language that means the same thing as the original language, and they don't really care how.
Tha majority of research leads to "dead ends", but most folks understand that's the nature of research, and there is usually still value in discovering "OK, this won't work". Usually, though, this process is pretty incremental. With LLMs all of a sudden you had lots of folks whose life work was pretty useless (again, from a practical perspective), and that'd be tough for anyone to deal with.
Note that the author has a background spanning a lot of the timespans/topics discussed - much work in multilingual NLP, translation, and more recently at DeepMind, Cohere, and Meta (in other words, someone with a great perspective on everything in the top article).
Re: Machine Translation, note that Transformers were introduced for this task, and built on one of the earlier notions of attention in sequence models: https://arxiv.org/abs/1409.0473 (2014, 38k citations)
That's not to say there weren't holdouts or people who really were "hurt" by a huge jump in MT capability - just that this is a logical progression in language understanding methods as seen by some folks (though who could have predicted the popularity of chat interfaces).
Very much like when I moved from tightly coupled to "embarassing" parallelism. A friend said "don't call it embarassing... it's pleasant not to have to think about hard distributed computing problems".
AI is obliterating the usefulness of all mental work. Look at the high percentage of HN articles trying to figure out whether LLMs can eliminate software developers. Or professional writers. Or composers. Or artists. Or lawyers.
Focusing on the NLP researchers really understates the scope of the insecurity induced by AI.
In other words, what is progress for the field might not be progress for you !
This reminds me of Thomas Kuhn’s excellent book ´the structure of scientific revolutions’ https://en.wikipedia.org/wiki/The_Structure_of_Scientific_Re...
https://en.wikipedia.org/wiki/Average_and_total_utilitariani...
Take that moral position and then extend the calculation to cover all potential generations of humans to come, and the value of maximising your utility now is infinitely small.
I have a bit of background in this field so it's nice to see even people who were at the top of the field raise concerns that I had. That comment about LHC was exactly what I told my professor. That the whole field seems to be moving in a direction where you need a lot of resources to do anything. You can have 10 different ideas on how to improve LLMs but unless you have the resources there is barely anything you can do.
NLP was the main reason I pursued an MS degree but by the end of my course I was not longer interested in it mostly because of this.
I think you're confusing problems, or you're not realizing that improving the efficiency of a class of models is a research area on it's own. Look at any field that involves expensive computational work. Model reduction strategies dominate research.
Well, they're unhappy that an unrelated field of research more-or-less accidentally solved NLP. All the specialized NLP techniques people spent a decade developing were obviated by bigger deep learning models.