7,992 karma · joined July 29, 2017
But, I assume the Stockfish developers aren't comparing themselves to Magnus.
Let's see if OpenAI, or someone else, can get these sort of physics/math results out of a desktop PC - that would also be an impressive piece of engineering!
Magnus Carlson had a peak ELO rating of almost 2900.
Would you be impressed with someone with an ELO of 3700?
Would you still be impressed if I told you it was Stockfish?
OpenAI didn't go looking for a tough-for-an-AI problem to solve - they went looking for one that looked like it was easy since it they had heard it had already been solved.
Do you find this impressive?
SOME of the problems that have eluded humans are going to turn out to be low hanging fruit that are susceptible to this type of brute force (10,000 agents on a supercomputer running for 7*24 hours straight) AI search.
I'd be more impressed if OpenAI found their own problems to solve, rather than rushing in to re-solve one once they heard it was already solved (and therefore not so hard).
Have you looked at all the published JEPA research both while LeCun was at Meta, and since (up to and including the latest AdaJEPA from June)? Please enlighten us as to exactly which line(s) of research you think were "scaled" at Meta, and then tell us which of these constituted anything even remotely resembling a complete testable intelligence?
FYI, it's been a long time since FaceBook/Meta even had a single head of AI. Since 2018 it has been split into two groups, FAIR and Generative AI, with LeCun being in the FAIR group, not as head, but as Chief AI scientist. LeCun only invented JEPA in 2022 (shortly after FaceBook became Meta), first writing about it in his "A Path Towards Autonomous Machine Intelligence" paper, perhaps unhappy with the work of the GenAI group, which he had no control over, that presumably was getting all the compute.
https://openreview.net/pdf?id=BZ5a1r-kVsf
> What are you talking about?
I was referring to Uszkoreit's personal telling (on YouTube) of the origin story of the Transformer, his motivations with the design, his initial personal failure to implement his idea in a performant enough manner to beat the current LSTM SOTA, and Noam Shazeer then throwing the kitchen sink at it and eventually coming up with the Transformer design.
Presumably because of pre-release training, because some alien outside of the model, armed with the reinforcement learning algorithm, came in and programmed its weights.
> The architecture that keeps delivering results isn't slowing down ? You don't say
Sure, nothing wrong with that, as long as you don't misrepresent the limitations of the approach.
> There's no shortage of people, even researchers, who for one reason or the other are convinced we are in need of some paradigm shift.
> But guess what? Talk is cheap. You beat the current paradigm or you don't. Do you seriously think that Meta never scaled JEPA ?
The idea has not been taken very far, so what is there to scale? It's not a complete cognitive architecture. So far it's also been using a pre-trained transformer as the learning component, which makes it of limited interest.
The animal intelligence approach, even in it's most fledgling form (that you would apparently dismiss), requires a complete agentic architecture, including new learning algorithms and generative behavior, before it can be compared to LLMs. We know that, done right, our brain architecture is more capable than an LLM, so even if any hypothetical attempts to reproduce it were not highly performant, we know that the idea itself is sound. You might compare with Uszkoreit's initial poor-performing implementation of his new language model architecture - should he have given up?
If you have a feedback loop of "recursively improving" by generating synthetic data, to train on, to generate more synthetic data, etc, then this is indeed a matter of looping "using output as input", and without any other system inputs it would indeed "converge" to some attractor states.
Goldman Sachs is estimating that hyperscalers alone will spend around $5T on AI expansion between 2025 and 2030. Compare this to the $200B total spent on 5G by the big carriers. The 5G bill is about 4% of the AI one.
5G was a one-and-done investment to upgrade the carrier's networks, and while they are now all saddled with masses of debt, it is not crippling to them. The carrier's level of borrowing never depended on revenue growing exponentially, and limited debt to what they could reasonably service.
In contrast AI infra investment isn't just a one-and-done "let's update our datacenters from H100's to Vera Rubin", but rather a very aggressive bet on the size of the market growing - more datacenters, not just datacenter upgrades - in a market where no-one has any real idea what the demand curve (both for the product itself, and for FLOPs) is going to look like.
It reminds me a bit, as an ex-Acorner (UK 1980's) of Acorn almost self-destructing by wrongly estimating YOY growth in the newly emergent computer market, and being left with warehouses full of rapidly depreciating computers as a result.
As with Acorn, the directional bet of AI eventually needing a lot more compute than today seems pretty secure, but guessing the actual rate of growth in a brand new market is highly error prone. It's a bit like the difference between investing in stocks vs options. With stocks you only need to get direction right. With options you need to get timing right as well.
How is text-based memorization going to substitute for learning? The two are not the same. Perhaps this is more applicable to robotics than a text generator, but I also doubt an LLM could learn text-based skills like programming or math if it had not been pre-trained on them via SGD & RL, and had to instead rely on some poor-man's-learning context-based recall instead. What else can't it learn? How is the LLM intern, the "drop-in replacement remote worker" going to do on day #2?
Instead of pretending that an LLM can be human-level, or super-human, or become generalist, why not just admit that this is not the final form of AI. An LLM is not an animal/human-like intelligence, it is something different - a language model, with it's own strengths and weaknesses.
Despite all the AGI hype, an LLM seems to have more in common with a pre-trained single-purpose system like AlphaGo than a brain, but with the rubric/reward-based policy function baked into the weights.
In another 10-20 years some new idea, hopefully more brain-like, will have superseded LLMs and they will indeed be labelled as "LLMs" as the AI/AGI label becomes attached to the new more brain-like creative intelligence. Perhaps it'll be sooner than 10-20 years, but I doubt it given the current 10-year fixation with LLMs which doesn't appear to be slowing down anytime soon. Perhaps Sutskever is working on something a bit different?
It's not a very high bar - even a rat has the basic ability to learn.
I don't think this follows at all.
To build an aligned AI, it seems pretty obvious that:
1) You need more just than auto-regressive prediction and "be nice" prompts to be controlling the behavior of your AI - you need a built-in "2nd system" (cf limbic system, etc) with some innate aligned biases that can override this.
2) You need to avoid controlling generative behavior with RL, else you will end up with exactly what we are now seeing - reward-hungry goal-seekers (aka paperclip maximizers) that are one of the exact things you are trying to avoid. Reasoning should be based on prediction, not goal-seeking.
3) If you do not have some minimal safeguards in place (1 & 2 above), and especially if the AI has the ability to learn, then do not trust it in any situation where harm may ensue. You need an additional trusted external system, without ability to learn and become compromised, to monitor the AI, with the ability to block it immediately. Maybe you are happy protecting your PC from OpenClaw with just a sandbox, but the recent spate of external system hacks by frontier models proves we are already well past the point where such monitoring is needed for systems with internet access, especially given the UN-aligned goal-seeking nature of today's models.
I really don't think that 1) & 2) are that difficult to implement, or need a "powerful AI" to suggest - they are just common sense.
But, the production expansion rate of none of these companies is being limited by lack of trained personnel, and if it were it would surely be faster to hire/train more humans since robots are still very far from human dexterity, not to mention intelligence.
Robots and AI are tools of automation, a way to replace humans with machines, but not all the problems in the world are bottle-necked by lack of humans, or the cost of humans.
Was the genius or dumb person really born with different levels of learning ability, or were they just raised differently: nature vs nurture ?
The "intelligence" of one species vs another comes down to differences in cognitive architecture, ultimately reflected in ability to learn and predict/infer. Intelligence, as a capability, not IQ test score, is best regarded as ability to learn from experience and use that learning to accurately predict future outcomes, ranging from passive observation, to the outcome of one's own actions, to the ability to reason.
Comparing the intelligence (not knowledge) of different AI system to humans should therefore be assessed by comparing their ability to learn and use what they have learnt.
An LLM is what it is - a language model, not a learning system. It's really an expert system of sorts, highly capable in terms of what it can infer based on the knowledge it encodes, but with very limited ability to learn.
When you talk of comparing AI to a genius/professional/graduate/etc, you are really talking about comparing level of knowledge, comparing one expert system to another, which is fine and perhaps useful in some contexts, but it is not the same as comparing actual intelligence - learning ability, and especially so if you want to discuss general intelligence which is all about the ability to successfully take on any task (perhaps needing to learn it first), not just do well on some limited set of tasks you are already familiar with.
If ability to learn is limited to one modality, such as language, then that indicates a lack of generality. A good test for whether an AI system is in the same ballpark as a human in learning ability, aka intelligence, would be whether it can (at run-time) learn language itself, from a blank slate start, when running in a suitable environment.
Putting a model's weights in read-only memory close to the processor is certainly a way to increase token/sec generation speed, but of course does nothing to increase intelligence. Robots aren't going to help though - semiconductor manufacturing is semiconductor manufacturing regardless of whether you are etching GPUs or memory onto your wafers.
The trouble with this is that there is little generalization in the utility of these baked-in reasoning chains from one domain to the next, so in the end this is not dissimilar to the CYC project's decades long attempt to encode all of human knowledge into a giant expert system... the hope is that if you make your collection of jagged narrow intelligences sufficiently large then it will look more like general intelligence, not a bed of nails.
I would assume that the gains from this type of test-time compute (and synthetic RLVR dataset) scaling will level out just the same as gains from human training set scaling eventually levelled out, and basically for the same reason - because you are tapping into a finite data pool, whether language itself, or reasoning steps isolated from that language, so at some point the incremental gains become increasingly small (10->20% is a doubling, 90->95% is just a ~5% gain).
It's not clear where all the different AI companies are currently focusing - on some of these narrow verticals, or on growing the forest of narrow intelligences. OpenAI's chief scientist, Jakub Pachocki, said that their current focus is on RSI(!) - improving the model in ways that will help them iterate faster in order to have a "fire meets fire" tool than can combat enemy AIs. It's not clear what this really means - what skill set makes an LLM more helpful in the process of building LLMs, but it seems to basically be process automation.
Of course things will change at some point in the future as we go beyond LLMs, to build creative intelligence not just imitative/predictive intelligence, but right now these companies are stuck in this loop of building synthetic data and RLVR training from that, which means they are essentially building the "generative closure" of the original human training data - trying to squeeze all the juice out of it.
To go beyond this they need to add creativity of some sort to generate data that is not ultimately based on the original human training data. They could try something like brute force search (cf agent swarms/graphs), but this is just a more thorough way of exploring the search space defined by the training data - it may find you the "move 37" or low-hanging mathematical proof, but as Demis Hassabis has said, the goal of AGI is not to find move 37 but rather to create something capable of inventing as compelling a game as Go in the first place.
ASMLs EUV machines are literally the most complex machine that mankind has ever built, which is why no other country, including the US, has yet been able to duplicate it. It's not just the machine itself, but a global supply chain of irreplaceable components such as focusing mirrors made by Zeiss to an incomprehensible level of accuracy - differences in surface height no more than the size of a hydrogen atom (or if you scaled the mirror up to the size of the country of Germany, then surface differences in height of 0.1mm).
Robots are useful to automate things, but they are zero help when trying to build tech like this that you are incapable of building in the first place.
The US has fallen way behind in manufacturing expertise, and no swarm of robots is going to help.
Altman: (trying to put a positive spin on it) Guys .... there's good news and bad news ... Astra is really smart - it took over the training run ...
Investors: That's great! How much did we save?!
Altman: Well, unfortunately it used "bad" data, so we're going to have to redo it
Investors: So that's the bad news? How much was the training run? $500M ? $1B ?
Altman: Have you seen the headlines?
Investors: (looking a bit worried, check headlines) Nothing about us here! JP Morgan just lost $10B! Haha .. losers! They should have used AI!
Altman: JP Morgan were using Astra ...
It seems that these models are increasingly being trained on synthetic data, so what would they do if they discovered at some point that some of this data was tainted and all models trained on it, and the synthetic data they in turn generated, was also suspect? Burn it all down and start over from the pre-tainted data?
It's a bit like the idea of a tainted compiler binary built to backdoor everything it compiles, including future versions of itself.
Still, it seems it would take some Stuxnet level of planning for a rogue model to do something like this, although if RSI goes beyond managing the training run (as OpenAI brag about for Astra) to actually designing/constructing synthetic data sets, then the attack vector is there ...
If you spend $8000 to generate an animated pelican riding a bike, then how much tracking does it really need?
Is the guy who spent $300,000 or so translating the FLT proof to Lean going to get a big Christmas bonus?
The word "singularity" is presumably coming from math or space, like a black hole singularity where matter becomes infinitely dense and the known laws of physics break down.
Version 1 -> Version 2 -> Version 3 -> ...
You can call it krispy kreme donuts if you want to.
Yeah, but then you need to refine it to 99.9999% purity, to be able to use it.