Hacker News Reader

This is an uncharitable and oddly dismissive take (i.e. perfect for HN, I suppose).

Today's incredible state-of-the-art does not exist without the transformer architecture. Transformers aren't merely some lucky passengers riding the coattails of compute scale. If they were, then the ChatGPT app which set the world ablaze would've instead been called ChatMLP, or ChatCNN. But it's not. And in 2024 we still have no competing NLP architecture. Because the transformer is a genuinely profound, remarkable idea with remarkable properties (e.g. training parallelism). It's easy to downplay GPTs as a mostly derivative idea with the benefit of hindsight. I'm sure we'll perform the same revisionist history with state-space models, or whatever architecture eventually supplants transformers. Do GPTs build on prior work? Do other approaches and ideas deserve recognition? Yeah, obviously. Like...welcome to science. But the transformer's architects earned their praise -- including via this article -- which isn't some slight against everyone else, as if accolades were a zero-sum game. These 8 people changed our world and genuinely deserve the love!

Question for you, as someone relatively new to the world of AI (well, not exactly new - I took many courses in AI, including neural networks, but in the late 90s... the world is just a tad different now!)

Is there any good summary of the history of AI/deep learning from, say, late 00s/2010 to the present? I think learning some of this history would really help be better understand how we ended up at the current state of the art.

I've been following a data science course called "The Data Science Course 2023: Complete Data Science Bootcamp" at Udemy.

The course starts all the way back from basic statistics and goes through things like linear regression and supposedly will arrive at neural networks and machine learning at some point.

So I don't know if something like this is exactly what you're looking for, but I think that, in general, if one wants to learn about (the history) AI, then it might be a good idea to start from statistics and learn about how we got from statistics to where we are now.

Not sure if this is wayy to off of what you ask: https://www.youtube.com/watch?v=OFS90-FX6pg

Maybe this? https://arxiv.org/abs/2212.11279

The worlds I see by fei fei li is exactly what you're looking for, intertwined with a beautiful story of a brilliant scientist's perseverance

> And in 2024 we still have no competing NLP architecture.

No, we do. State space models are both faster and scale just as well. E.g., RWKV and Mamba.

> Transformers aren't merely some lucky passengers riding the coattails of compute scale.

Err... they are, though. They were just the type of model the right researchers were already using at the time, probably for translating between natural languages.

It’s reply’s like this that make HN a good place.

> but we didn't see what another group found: that scale in compute with simple algorithms were just better

The bitter lesson [0] strikes again.

[0] http://www.incompleteideas.net/IncIdeas/BitterLesson.html

I kind of wonder if the reason this seems to be true is that emergent systems are just able to go a lot farther into much more complex design spaces than any system a human mind is capable of constructing.

I think this is overly general. A more accurate statement is that, on tasks where we don't actually understand how something works in precise detail, it's more effective to just throw compute at it until a system with "innate" understanding emerges. But if you do actually know how something works (rather than vague models with no clear supporting evidence), it's still more effective to engineer the system specifically based on that knowledge.

I argue they definitely 'changed AI' but definitely agree they didn't 'invent modern AI'.

Personally, I think both compute and NN architecture is probably needed to get closer to AGI.

I’m studying neuroscience but very interested in how ai works. I’ve read up on the old school but phrases like memory graph and energy minimization are new to me. What modern papers/articles would you recommend for folks who want to learn more?

For phrases, Google's TF glossary [0] is a good resource, but it does not cover certain subsets of AI (and more specifically, is mostly focused on TensorFlow).

[0] https://developers.google.com/machine-learning/glossary

If you are in neuroscience I would recommend looking into neural radiance fields rendering as well. I find it fascinating since it's essentially an over-fitted neural network.

Someone put this link up on another discussion the other day and I found it really fascinating:

https://bbycroft.net/llm

I believe energy minimization is literal, just look at the size of that thing and imagine the power bill.

He is most likely referring to some sort of free energy minimization.

In fairness, an article that’s about “the Google engineers who incrementally advanced AI” wouldn’t sell many advertisements.

They didn’t invent it, they advanced it. I agree.

However, some advances can have huge consequences to the field compared to others, even if at the technical level they appear comparable.

One example that comes to mind is CRISPR.

This is a classic... This isn't the first time this piece has been posted before either. Here's one from FT last year[0] or Bloomberg[1]. You can find more. Google certainly played a major role, but it is too far to say they invented it or that they are the ones that created modern AI. Like Einstein said, shoulder of giants. And realistically, those giants are just a bunch of people in trench coats. Millions of researchers being unrecognized. I don't want to undermine the work of these researchers, but that doesn't mean we should also undermine the work of the many others (and thank god for the mathematicians who get no recognition and lay all the foundation for us).

And of course, a triggered Yann[2] (who is absolutely right).

But it is odd since it is actually a highly discussed topic, the history of attention. It's been discussed on HN many times before. And of course there's Lilian Weng's very famous blog post[3] that covers this in detail.

The word attention goes back well over a decade and even before Schmidhuber's usage. He has a reasonable claim but these things are always fuzzy and not exactly clear.

At least the article is more correct specifying Transformer rather than Attention, but even this is vague at best. FFormer (FFT-Transformer) was a early iteration and there were many variants. Do we call a transformer a residual attention mechanism with a residual feed forward? Can it be a convolution? There is no definitive definition but generally people mean DPMHA w/ skip layer + a processing network w/ skip layer. But this can be reflective of many architectures since every network can be decomposed into subnetworks. This even includes a 3 layer FFN (1 hidden layer).

Stories are nice, but I think it is bad to forget all the people who are contribution in less obvious ways. If a butterfly can cause a typhoon, then even a poor paper can contribute to a revolution.

[0] https://www.ft.com/content/37bb01af-ee46-4483-982f-ef3921436...

[1] https://www.bloomberg.com/opinion/features/2023-07-13/ex-goo...

[2] https://twitter.com/ylecun/status/1770471957617836138

[3] https://lilianweng.github.io/posts/2018-06-24-attention/

I guess the same thing happened with special relativity and Poincaré was forgotten.