Transformer == Modern AI?
Wow, I dont know where should I start
My mind literally stuck at the absurdity
Besides that, are you denying that transformers are the fundamental piece behind all the AI hype today? That's the point of the article. I think the fact that a mainstream publication is writing about the Transformers paper is awesome.
Modern AI should anything after https://en.wikipedia.org/wiki/Dartmouth_workshop Not transformer
AI has been people's dream since written history. i.e., everyone in their own sense would want to invent something that can do things and think for themselves. That's literally the meaning of AI.
It had been a relatively gradual and accelerating progress, with number of people working in the field increasing exponentially, since 2010 or so, when Deep Learning on GPUs was popularized at NIPS by Theano.
Tens of thousands people working together on Deep Learning. Many more on GPUs.
Consider "modern" to mean NN/connectionist vs GOFAI AI attempts like CYC or SOAR.
I guess it depends on how you define "AI", and whether you accept the media's labelling of anything ML-related as AI.
To me, LLMs are the first thing deserving to be called AI, and other NNs like CNNs better just called ML since there is no intelligence there.
Well this is what Im trying to say too!
I dunno. The earliest research into what we now call "neural networks" dates back to at least the 1950's (Frank Rosenblatt and the Perceptron) and arguably into the 1940's (Warren McCulloch and Walter Pitts and the TLU "neuron"). And depending on how generous one is with their interpretation of certain things, arguments have been made that the history of neural network research dates back to before the invention of the digital computer altogether, or even before electrical power was ubiquitous (eg, late 1800's). Regarding the latter bit, I believe it was Jurgen Schmidhuber who advanced that argument in an interview I saw a while back and as best as I can recall, he was referring to a certain line of mathematical research from that era.
In the end, defining "modern" is probably not something we're ever going to reach consensus on, but I really think your proposal misses the mark by a small touch.
The modern era of NNs started with being able to train multilayer neural nets using backprop, but the ability to train NNs large enough to actually be useful for complex things AI research, can arguably be dated to the 2012 Imagenet competition when Geoff Hinton's team repurposed GPUs to train AlexNet.
But, AlexNet was just a CNN, a classifier, which IMO is better just considered as ML, not AI, so if we're looking for the first AI in this post-GOFAI world of NN-based experimentation, then it seems we have to give the nod to transformer-based LLMs.
Am not expert, do you have some links about this? i.e. a neural net construction that outperforms a transformer model of the same size.
Transformer+Scaling+$$$ triggered the current hype