I don't see anything wildly different now, other than scale and youth and the hubris that accompanies those things.
What's your best examples of this? Some of the most impressive examples I've seen ended up being likely in the dataset, or very close to being so. I've yet to see something where it definitely wasn't approximately in the dataset and was solved in a way that seemed to use some sort of novel process, but open to being wrong.
Every single day, I get immense use out of modern language models. Even if an output is similar to something it's already processed, that's fine! Such is the nature of synthesis.
They are not novel if there is an equivalent pattern in the training dataset. I guess you are not really trying anything that isn't available already on github or google in some form. If you think you do then please show an example of "entirely novel and original idea", that GPT-4 developed for you. I had at least 4 cases in which ChatGPT failed to produce correct solution (after pushing it for hours to correct itself in many ways) in an actual novel problem (solution not longer than 200 lines of code) for which there was no solution on Google or github. But you can't blame statistical model that was trained to create the most probable outcomes based on it's training data.
Usually, drawing from existing knowledge is the whole appeal of using GPT. Over time, you actually begin to get a sense of what the model is good at and bad at, and it's good at an incredible amount of things. I get it to write novel code constantly and I think that playing with it and confirming that for yourself is better than me showing you.
The human brain is a salient point because often we are using AI so that the human brain can do less. Get this GPU to RTFM instead of the human. The human time is more valuable. All the while making the human brain probably less effective (compare someone who learns another language vs. someone who speaks it through an AI translator only).
I hold both points of view that AI is both marvelous, but also concerning in terms of energy use.
To nitpick - in " NNs are to emergent behavior what crypto is to cash " applies more to large language models. Simpler NNs for easy tasks that don't consume much power wouldn't apply (that might be like a VISA card?)
Is it that this behavior is the result of any system at scale? That is undeniably preposterous.
Is it that the human brain is more efficient? At energy usage, sure, but for me to find an individual who is capable enough to assist me in the manner GPT does, at the speed and level of breadth and depth that it does, would be next to impossible. If I did, their required compensation would be astronomical.
What are you arguing for or against? Are you aware that these systems will, like all previous computationally intensive systems, become drastically more efficient over time?
Apparently, sit there and write plain English at a billion-dollar cluster and wait for astonishing answers at 300bps.
Like, a whole decade of ML: better optimisers, better init, residual connections, tokenisation and token embeddings, training with large batches over thousands of machines, the attention mechanism, causal masking, flash attention and other memory optimisations, and even having the foresight to train on the totality of web text.
Not seeing the intermediate steps doesn't mean they are not essential and needed.
If you still believe a toy 10x10 fully connected net is the same with current models (bar scaling), then what is you opinion on MLP-Mixer? That was an "MLP is all you need" moment but it didn't lead to adoption.
You too might be a Hubris News celebrity one day.
Reminds me of the old dudes in the gym who come to you to tell you how they used to bench four plates when they were young. In their mind, they are badasses. In their mind only.
BTW, awesome "flex" about how much time you spend at the gym, you sound like a guy who knows what he's talking about ;)