I also wonder - is the compute the main limiting factor today ? Let's imagine there is an unlimited number of NVidia chips available right now and energy is cheap - would using a cluster x100 of current biggest one result in a significant improvement ? My naive intuition is that not really.
I’m sure a few companies/models are optimized enough to fit ideal case but it’s rare.
Edit: Another aspect of this is nature of model architecture that are good today is very hardware driven. Major advantage of transformers over recurrent lstm models is training efficiency on gpu. The gap in training efficiency is much more dramatic with gpu than cpu for these two architectures. Similarly other architectures with sequential components like tree structured/recursive dynamic models tend to fit badly for gpu performance wise.
Let me reframe the question - assume it's not only 100x GPUs, but all the performance bottlenecks you've mentioned are also solved or accelerated x100.
What kind of improvement would we observe, given the current state of the models and knowledge ?
So are we now at performance plateau? I know people at openai like places that think AGI is likely in next 3-5 years and is mostly scaling up context/performance/a few other key bets away. I know others who think that is unlikely in next few decades.
My personal view is I would expect 100x speed up to make ML used even more broadly and to allow more companies outside big players to have there own foundation models tuned for their use cases or other specialized domain models outside language modeling. Even now I still see tabular datasets (recommender systems, pricing models, etc) as most common to work in industry jobs. As for impact 100x compute will have for leading models like openai/anthropic I honestly have little confidence what will happen.
The rest of this is very speculative and not sure of, but my personal gut is we still need other algorithmic improvements like better ways to represent storing memory that models can later query/search for, but honestly part of that is just math/cs background in me not wanting everything to end up being hardware problem. Other part is I’m doubtful human like intelligence is so compute expensive and we can’t find more cost efficient ways for models to learn but maybe our nervous system is just much faster at parallel computation?
the term for it is "grokking", amusingly. There's some indication that we are actually undertraining by 10x
I had never thought about this before but how do I recognize faces? I mostly recognize faces by context. And I don't have to match against a billion faces, probably a hundred or so? And I still suck at this.
The fact that human brain works with 0.3 kW per day likely doesn't mean much. How do we even start asking the question - is a human brain thermally (or resource in general) constrained?
Frustratingly, it’s also the hardest part to solve too. Throwing more compute at the problem is easy but diagnosing and then solving those other bottlenecks takes a great deal of time and not to mention experience across a number of specialty domains that aren’t typically mutually inclusive.
So much of this is a blackbox that you have to wing it for lots of things and try stuff. The more compute you have, the more YOLO runs you can do.
For example, the research team I work with has about 1/20th the compute that a Google researcher has. This gives them a massive advantage because they can afford to train lots of random ideas and see what kind of advantage they get. We have to be much, much more measured and predict our outcomes better.
The search space could easily be so large that a less disciplined approach might yield fewer useful outcomes even with the compute advantage. Being forced to be focused and disciplined might actually be a big advantage to you.
Isn't OP talking about this _blackbox_ of gains?
But this is highly disappointing too - there are speed gains and tradeoffs to be had at every blackbox layer, but instead of doing that and actually gaining these improvements across all future experiments, we do YOLO runs and don't garner them at all.
I've seen very compelling results in small scale experiments. I've personally achieved basic tool use (reading a prompt buffer and writing to an output buffer) in a SNN simulation. The biggest takeaway for me is that the fitness function is the most important thing followed by selection criteria and simulation parameters. Figuring out how to compact the total measure of fitness into a single float as the simulation progresses is non trivial. I've found that dynamically adapting the fitness function and simulation parameters over time is essential.
To be clear, I don't know if I've achieved learning/generalization by the formal definition yet. I do know I can minimize the loss over a training set by merely mutating the weights of neurons with otherwise fixed delay and connectivity. I'm not even doing things like crossover or elitism yet. My next sprint will attempt varying the delays as well as the weights. I've also got some really strong opinions about the ratio of excitatory to inhibitory neurons now. I think inhibitory neurons are a bandaid for physical limitations. In a simulation, we can simply constrain the total allowed ticks or global activations per candidate. The selection pressure will reject candidates that can't get the job done with the allotted resources.
I think you can do the real deal on one powerful workstation. I've got networks with 10k neurons and I can run through about 4000 generations x 250 candidates on a single threadripper in a day. The main bottleneck being memory bandwidth as networks grow in size. Cloning them each generation starts to take a long time once you get into 6 figures. You really want as many generations per unit time. Being able to watch it evolve at meaningful rates of speed can unlock new understanding about your fitness function.
GPT-2 needed around 500,000 neurons and couldn't use tools.
RL style training can let you train against a vector of losses... Custom loss functions are a thing...
> I've found that dynamically adapting the fitness function and simulation parameters over time is essential
Did you just figure out epsilon greedy? - very well known technique...
Spiking / time domain... Non linearities can already be captured by stacking linearly activated layers, or connecting layers. Time domain, CNNs.. RNNs.. ResNet, U-Net as (now almost ancient) examples cover alot of the same ground.
10K neuron networks are tiny. I dont know what your trying to accomplish but I would suggest reading more literature, because it sounds like your stuck in a local optima of old ideas...
Small scale experiments are approximately meaningless in ML. A lot of the clever tricks that improve results at smaller scales have their gains BTFO'd by the scaling you give up as you increase the size of the model.
The key insight from deep learning was that the gains from doing a smarter thing were dwarfed by doing a dumber thing a lot more.
For a smarter trick to be valuable, it needs to not only improve results at the small scale, it needs to not become a bottleneck in your system as you jack up the size.
But throw 100x more compute to an LLM (through an increase of both parameters and tokens) and it will undoubtedly beat current models.
And as llama is trained on 15T tokens, we have lot more tokens remaining on the internet. And different modalities like video will have orders of magnitude more token and more information.
But still they would be extremely useful, even if very different than AGI
There are probably plenty of undiscovered architectures out there which might be the next thing.
It is quite a problem if every experiment takes days weeks or month.
Video of reference (check especially the ending summary where they calculate more or less how much would be necessary just to replicate a human mind's way of achieving conciousness if microtubules in fact contribute towards it):
2. quantum is really weird.
3. what if the two were really connected?
edit: oh god, I just had a horrible thought: the universe is a simulation, and the reason consciousness requires "quantum" spookiness is because it's running on a different computer and plugged in as an adhoc skill.
I think it depends on the framing of the question, especially how you define compute.
- No, compute is not the limiting factor. The limiting factor is poorly optimized software (there's a joke: "10 years of hardware advancements have been entirely undone by 10 years of software advancements")
- No, compute is not the limiting factor. The limiting factor is that the electron is too big and the speed of light is too slow.
If we're talking about ML, then no, compute is not the limiting factor.
At least if we're define compute as the number of FLOPs we can process and not in terms of algorithm or resultant abilities. Though I'll admit that I'm an outlier in this respect[0]. But I think it is worth recognizing that we now have exaflop machines, that use tens of megawatts of energy, and they pale in comparison to what a 3 lb piece of fat and meat that only uses a handful of watts. In fact, our exascale computers aren't even seemingly sufficient to simulate far smaller and far less intelligent creatures. Certainly scale is a factor (we do see this pattern in apes too), but clearly there is more. And I think it should be obvious that scale isn't all you need, since we're the only ones. If it was that simple, we should see it more often. And if scale is indeed all we need, well we neither do we have an idea of how much scale that actually is nor does it mean that this is the best path forward as that scale may be ludicrously large. But what we do know, is that incredible feats can be done with what would constitute a rounding error to current scales (let alone future). I think we just want to believe this is the path forward because if it is, then there is a clear direction. But if it isn't, then we have to admit that we're still lost. But I think the problem is that we think that there's a problem in being lost. Or that we think that admitting we're lost somehow undermines or rejects the progress that we have made. But research is all about exploring the unknown. If you aren't at least a little lost, well then you're not exploring, you're reading a map. But the irony in this is that "scale is all you need" denies a lot of significant advancements we've made. Many smaller models perform far better that previously, and this is not due to knowledge transfer from larger models. Just look at any leaderboard, they aren't size is not the determining factor.
So I'd argue that if you want to advance AI, you should focus on smaller models. After all, smaller models are far easier to scale than larger models. They're also far easier to analyze and interpret, which is what gives us more information on how to lighten the way forward. But also don't expect a smaller model that is more successful to immediately be better than larger models. I far too often see a mistake even by reviewers/experts, where a method is dismissed because it was developed by some poor grad student with limited compute and did not unilaterally defeat the big models. Of course that doesn't mean the proposed methods are better, but that's orthogonal to what I'm arguing.
[0] Obviously I'm not alone. Yann is a clear believer and it's why he's looking at JEPA models (I don't think this will be enough but I think it is better). And Collet became more well known (at least outside the ML research community) and is a clear dissenter.
On another thread*, I mentioned that the XC2064 FPGA design is the paradigmatic example of an optimized minimal "hardware API", that, if you were interested in making new kinds of logic and memory-- okay you are prob thinking of higher level designs, but I want to throw out a relevant memorable example ASAP