The convolution empire strikes back
gonzoml.substack.com
gonzoml.substack.com
Similarly, humans achieve about the same results when they have the same training. Small variations. What matters is not the brain but the education they get.
Of course I am exaggerating a bit, just saying there are a multitude of architectures of brain and neural nets with similar abilities, and the differentiating factor is the data not the model.
For years we have seen hundreds of papers trying to propose sub-quadratic attention. They all failed to get traction, big labs still use almost vanilla transformer. At some point a paper declared "mixing is all you need" (MLP-Mixers) to replace "attention is all you need". Just mixing, the optimiser adapts to what it gets.
If you think about it, maybe language creates a virtual layer where language operations are performed. And this works similarly in humans and AIs. That's why the architecture doesn't matter, because it is running the language-OS on top. Similarly for vision.
I place 90% the merits of AI on language and 10% on the model architecture. Finding intelligence was inevitable, it was hiding in language, that's how we get to be intelligent as well. A human raised without language is even worse than a primitive. Intelligence is encoded in software, not hardware. Our language software has more breadth and depth than any one of us can create or contain.
I believe LLMs will train mostly on synthetic data engineered to have extreme diversity and very high quality. This kind of data confers 5x gains in efficiency as demonstrated by Microsoft in the Phi-1.5 paper.
overparameterization is fine
Maybe vision is special enough that convnets and approache transformer level performance or it could be generalized to any modality. I haven't read enough papers to know if someone has already done something like this but everywhere I look on the application side of things, vanilla transformers seems to be dominating
That's misleading. Small average variation overall, but some outliers are dramatically better and dramatically worse. Some brains produce fields medals, some can barely count change. Unfortunate but true.
Ideas are self replicators - usually by being useful to people, but their lifecycle is different from that of people, they evolve much faster. It's an evolutionary process. Modern man and that of 10,000 years ago are biologically not much different, the difference comes from cultural evolution.
Maybe it's worth pointing out: That's a core idea of memetics, i.e. the study of memes, in the original sense of that word.
Yes, but the Gausses and Ramanujans of the world disprove the claim that architecture doesn't matter, and that only training data matters. The long tails that describe human variance I think show that architecture can matter a lot.
I think you might be right that architecture for some machine learning algorithms doesn't matter so much given enough data, but the fact that they need so much data to perform well I think implies that there's a better architecture awaiting discovery.
So they may have an IQ advantage, but they definitely also have a data/training advantage.
I am sure it helps alot if your parents are professors!
Even if that were true, you'd need only one Ramanujan to disprove the claim that architecture doesn't matter. We've seen many more than one.
If Ramanujan were raised as a goatherd in an illiterate society I don't think he would've accomplished much mathematically.
It's as if you trained an LLM on children's books and it converged to Terence Tao.
Despite hard evidence to the contrary, some people really really want to pretend that all human brains are equally capable.
For example, instances of cognitive disability due to macroscopically visible brain structure deviations or brain damage are usually brushed away as "Well that's just a disability and therefore it doesn't count! All the other brains are the same!"
That's true, but is that due to all the adults and "forces" around them conspiring to treat them different, or is it due to them behaving in a way that elicits certain responses? Surely some of each, no? So then, nature may have a way of manipulating nurture in a sense.
Fraternal or identical?
> Small average variation overall, but some outliers are dramatically better and dramatically worse. Some brains produce fields medals, some can barely count change. Unfortunate but true.
We don't actually know which part of the “can barely count change” situation comes from the “brain” compared to the situation people were raises in. Sure you'd always have outliers (good or bad) for purely genetic reasons, but if you looked at sport performance for instance you'd find that the theoretical difference between people is much lower than the practical one for many environmental reasons: in theory, once you factor in rare medical conditions, almost every human between 18 and 40 should be able to run a marathon. The reason why most people cannot is entirely environmental (not enough physical activity, overweight, work related health issues, etc.).
And it's also true in terms of performance: almost any human ought to be able to run a 100m in less than 14 seconds, which is only 30% slower than the world record. Yet, for purely environmental reasons, only a small fraction of the population is actually able to do it.
I suspect that in terms of intelligence the results would be roughly in the same ballpark: almost everyone, except rare mental development disorders, ought to be 70% as intelligent as the most intelligent person in history (and the enormous majority fitting in a much narrower band), but most end up far worse for environmental reasons alone.
For some reason (mostly politico-philosophical), most people in the “nature” camp of the “nature vs nurture” debate are completely oblivious to the evidence for this.
I think this too is misleading. We've quantified the heritability of intelligence to quite a significant degree. Twin studies of twins who have been raised apart even shows that outcomes are largely the same despite different environments. Nurture is very much oversold. Basically, as long as you don't traumatize one of two twins they will develop very similarly.
> And it's also true in terms of performance: almost any human ought to be able to run a 100m in less than 14 seconds
But it's literally not true of all humans, and it won't be true if you drop that below 13 or 12 seconds. Which shows that the specific biomechanics you're born with will influence your natural physical abilities. By extension, we also have natural cognitive abilities with similar consequences.
> most people in the “nature” camp of the “nature vs nurture” debate are completely oblivious to the evidence for this
I think you're way overselling this claim. "It's all nurture" is a far more common opinion, and far less evidence supports it.
Not really. I can't find the link again but there was a survey of cognitive science experts, and the question of inheritability of intelligence is still an open question with lots of conflicting opinions in the academic community. Also, “heritability” doesn't mean what you think it means.
> Twin studies of twins who have been raised apart even shows that outcomes are largely the same despite different environments.
I'd love to know your source, because I doubt the number of “twins that have been raised apart ” is large enough to be a statically significant cohort even if you could have them all in your study.
> But it's literally not true of all humans
Yes, some people are born without legs. And some are born with Down syndrome, nobody disputes that.
> and it won't be true if you drop that below 13 or 12 seconds. Which shows that the specific biomechanics you're born with will influence your natural physical abilities. By extension, we also have natural cognitive abilities with similar consequences.
Nobody questions that either (although 13 secs is probably close to what's reachable by anyone, and 12 by any male), the question is about scale: if you take a random sample of 20 Americans of the right age, you'd have insane disparities, with a significant number not being able to run the distance at all and struggling to finish in less than a minute, despite their completely functional genome.
> I think you're way overselling this claim. "It's all nurture" is a far more common opinion, and far less evidence supports it.
That's a strawman, a very common one in the mouth of the folks of the ”nature” camp. Nobody argues that it's literally “all nurture”, everyone know that Down syndrome exist.
The actual argument is that when in my previous 100m example you have genetics variation with an impact of at best 30%, and a observed variation of more than 500%, it's fair to say nurture is the dominant factor.
Doesn't mean nature has no role at all, but it doesn't have an explanatory role: the reason for observed variation is essentially nurture.
Stop taking The Bell Curve seriously, this book has been written in bad faith by people with a political agenda (which even stated with a straight face in the book). The reason why these people argue for the genetic impact trumping everything is because they don't like the idea of state intervention to reduce poverty. When you say “these people are poor because they are genetically dumb” then you can claim there's no point in helping them, and without surprise that's exactly what Herrnstein and Murray are advocating in their book…
This, among others: https://pubmed.ncbi.nlm.nih.gov/2218526/
> Yes, some people are born without legs. And some are born with Down syndrome, nobody disputes that.
I see you haven't debated many blank slaters. The original post I responded to said, "What matters is not the brain but the education they get". That is literally false.
> if you take a random sample of 20 Americans of the right age, you'd have insane disparities, with a significant number not being able to run the distance at all and struggling to finish in less than a minute, despite their completely functional genome.
Yes, and you will find cases of people who ate basically the same diet and were just as active or sedentary who fall on both sides of that line of finishing/not finishing. Some people will just have a more efficient metabolism, or a more robust skeletal structure, or greater lung capacity, or more type II muscle fibers, and the positive confluence of all of these factors makes a great natural athlete, and a negative confluence on all of them makes for an unlucky individual who will likely never be athletic no matter how hard they try. Which doesn't reduce the importance of being fit, or reduce their human worth, but there's no point denying the obvious.
All of these factors fall within natural human variability, and all of these factors are influenced by environment of course, because genetics and environment can't be trivially separated, but the point is that it seems like these factors have multiplicative effects. Such factors exist for cognition as well, and the same conclusions follow. This counts as architecture.
> The actual argument is that when in my previous 100m example you have genetics variation with an impact of at best 30%, and a observed variation of more than 500%, it's fair to say nurture is the dominant factor.
Nurture is probably the dominant factor in almost all cases throughout the world because of malnutrition. It remains the case that even if you eliminate all environmental disparities, humans will still exhibit considerable variability on all traits. Some will still be natural gifted athletes and naturally gifted mathematicians, and some significant fraction will simply be below average on many categories, and some unlucky few will be below average on all categories, and not due to any obvious deficiency like Down's.
I'm not even going to bother addressing the irrelevant comments on Murray and his book, mostly because I haven't even read it. This is all standard statistics and genetics.
The original matter I disputed was literally false, as I said above, but the spirit of the comment was that data matters more than architecture. I think that's a very premature conclusion. It would be more correct to say that a) you can equate outcomes by varying data despite the architecture, but b) some architectures scale better (do more with less data), and I would add c) no known architecture scales as well as human cognition yet, which suggests better architectures remain to be found, and this suggests d) that architecture does matter.
Bingo.
> What matters is not the brain but the education they get". That is literally false.
No, it's oversimplified, but it's still less false than the opposite view you're trying to promote.
> and the positive confluence of all of these factors makes a great natural athlete, and a negative confluence on all of them makes for an unlucky individual who will likely never be athletic no matter how hard they try.
Only if you count as “athletic” the performances that are outliers, by a few 10%s. But not by reasonable standards for being athletic. Almost anybody that starts practicing endurance running before their 40 can be among the top 5% of the whole human population (and be “the athletic guy”), just because the enormous majority is desperately bad for environmental reasons (including: they have something else to in their damn life than running 2 hours a day). Likewise, everyone who can read can read a book a week and quickly become among the 5% most cultivated. Except the few disabled, there's no genetic barrier you cannot overwhelm with some work, and win the game that the majority isn't even playing.
> It remains the case that even if you eliminate all environmental disparities, humans will still exhibit considerable variability on all traits.
Yes, 30% is still considerable by most metrics (you're never ever gonna win a race against someone who's 30% faster than you) but it's also very small difference if you compared to what are the disparities in the real world. If everyone was in the 30% expected margin, then pretty much everyone would be over 150IQ (whatever this metrics means). Sure, there would still be a few people above 200IQ, but the world would still be unrecognizable.
> This counts as architecture.
No, this is barely akin to hyperparameters in the ML world, but any comparison between deep learning models and the brain is completely meaningless anyway. The original take was very bad, in the ML topic, but yours is equally bad on human cognition. Yet another example that one can refute a poor argument with an even worse one.
> I'm not even going to bother addressing the irrelevant comments on Murray and his book, mostly because I haven't even read it. This is all standard statistics and genetics.
It's not irrelevant, it's still the main source of inspiration for most people who share your arguments and opinion, and it's likely that you have been at least indirectly influenced by it through different means.
https://arxiv.org/pdf/1712.00409.pdf
The slope of the power law is determined by the problem and dataset. Compute, parameter count, and data move you along the curve. Change in architecture/bias is a constant offset.
So architecture can give an advantage, but that advantage can be overcome by scale.
Genetics may play a huge role.
I assume you mean architecture doesn’t matter much, as long as it’s roughly sensibly designed.
The basics of architecture do matter though: a two-layer fully connected network won’t get very far no matter how many parameters it has. Architectural improvements like ResNets have definitely made a difference to the field.
Ironically, convolution provides some unification in physics too, e.g. renormalization is a convolution.