Why Fei-Fei Li and Yann LeCun are both betting on "world models"
entropytown.com
entropytown.com
LeCun is right to say that continuous self supervised (hierarchical) learning is the next frontier, and that means we need world models. I'm not sure that JEPA is the right tool to get us past that frontier, but at the moment there are not a lot of alternatives on the table.
So the world of mathematics is really the only world model we need. If we can build a self-supervised entity for that world, we can also deal with the real world.
Now, you may have an argument by saying that the "real" world is simpler and more constrained than the mathematical world, and therefore if we focus on what we can do in the real world, we might make progress quicker. That argument I might buy.
Sure mathematics can be said to be at the core of most of that but you’re grossly oversimplifying.
In theory I think you are kind of right, in that you can model a lot of real world behaviour using maths, but it's an extremely inefficient lense to view much of the world through.
Consider something like playing catch on a windy day. If you wanted to model that mathematically there is a lot going on: you've got the ball interacting with gravity, fluid dynamics of the ball moving through the air, the changing wind conditions etc. yet this is a very basic task that many humans can do without really thinking about it.
Put more succinctly, there are many things we'd think of as very basic which need very complex maths to approach.
Firstly, games aren't mathematics. They are low quality models of physics. Mathematics can not say what will happen in reality, mathematics can only describe a model and say what happens in the model. Just mathematics can not say anything about the real world, so a world model just doing mathematics can not say anything about the world either.
Secondly, and far worse for your premise, is that humans do not need these mathematical models. I do not understand the extremely complex mechanical problem of opening a door, to open a door. A world model which tries to understand the world based on mathematics has to. This makes any world model based on mathematics strictly inferior and totally unsuited to the goals.
The real world however is far more complex and perhaps rooted in a universal language, but in one we don’t know (yet) and ultimately try to describe and order by all scientific endeavors combined.
This philosophy is an attempt to point out that you can create worlds from mathematics, but we are far from describing or simulating ‘Our World’ (Platonic concept) in mathematics.
"If people do not believe that mathematics is simple, it is only because they do not realize how complicated life is."
In Dreamer 4 they are able to train an agent to play Minecraft with enough skill to obtain diamonds, without ever playing the game at all. Only by watching humans play. They first build a world model, then train the agent purely in scenarios imagined by the world model, requiring zero extra data or experience. Hopefully it's obvious how generating data from a world model might be useful for training agents in domains where we don't have datasets like the entire internet just sitting around ready-made for us to use.
Between us and human-level intelligence lie many problems. They can be summarized as that of succeeding in the "common-sense informatic situation". [1]
And the search continues...
Perhaps paradoxically, if/as this becomes a consensus view, I can be more excited about AI. I am an "AI skeptic" not in principle, but with respect to the current intertwined investment and hype cycles surrounding "AI".
Absent the overblown hype, I can become more interested in the real possibilities (both immediate, using existing ML methods; and the remote, theoretical capabilities follow from what I think about minds and computers in general) again.
I think when this blows over I can also feel freer to appreciate some of the genuinely cool tricks LLMs can perform.
That being said there have been models which are pretty effective at other things that don’t use language, so maybe it’s a non issue.
But only some groups have the ability to systematically encode language as writing.
Writing is a technological marvel.
It is the most impressed I've been with an AI experience since the first time I saw a model one-shot material code.
Sure, its an early product. The visual output reminds me a lot of early SDXL. But just look at what's happened to video in the last year and image in the last three. The same thing is going to happen here, and fast, and I see the vision for generative worlds for everything from gaming/media to education to RL/simulation.
What you get is a 3D room based on the prompt/image. It rewrites your prompt to a specific format. Overall the rooms tend to be detailed and imaginative.
Then you can fly around the room like in Minecraft creative mode. Really looking forward to more editing features/infill to augment this.
I feel LeCun is correct that LLMs as of now have limitations where it needs an architectural overhaul. LLMs now have a problem with context rot, and this would hamper with an effective world model if the world disintegrates and becomes incoherent and hallucinated over time.
It'd doubtful whether investors would be in for the long haul, which may explain the behavior of Sam Altman in seeking government support. The other approaches described in this article may be more investor friendly as there is a more immediate return with creating a 3D asset or a virtual simulation.
Text really hogged all the attention. Media is where AI is really going to shine.
Some of the most profitable models right now are in music, image, and video generation. A lot of people are having a blast doing things they could legitimately never do before, and real working professionals are able to use the tools to get 1000x more done - perhaps providing a path to independence from bigger studios, and certainly more autonomy for those not born into nepotism.
As long as companies don't over-raise like OpenAI, there should be a smooth gradient from next gen media tools to revolutionary future stuff like immersive VR worlds that you can bend like the Matrix or Holodeck.
And I'll just be exceedingly chuffed if we get open source and highly capable world models from the Chinese that keep us within spitting distance of the unicorns.
Fundamentally what AGI is trying to do is to encode ability to logic and reason. Tokens, images, video and audio are all just information of different entropy density that is the output of that logic reasoning process or emulation of logic reasoning process.
No? The Wason selection task has shown that logic and reason are not really core nor essential to human cognition.
It's really verging on speculation, but see chapter 2 of Jaynes 1976 - in particular the section on spatialization and the features of consciousness.
This is wrong. The vast majority of revenue is being generated by text models because they are so useful.
Enterprise doesn't know how to use these models to achieve business outcomes.
These subscriptions will unwind, and when they do, it'll be a bloodbath.
I don’t think many of the companies running these make a profit right now
Which companies are using these.models to run at a profit?
Maybe I need to re-read reports; last I checked, none of those companies were operating at a profit.
You could say the same thing about AGI. Ultimately capital will realize intelligence is a drawback.
Just because people aren't spending money on them doesn't mean it won't eat your lunch.
I am developing a p2p program where the model runs on the end user's computer. So I don't even need to pay money for each user and have a bunch of infrastructure monetize them. It is a game changer and allows for a completely different architecture.
Edit: I’d like to add that I personally get a lot of value out of the models. They’ve helped me learn to do frontend development very quickly at my job. That said, that hasn’t translated into higher pay. The expectations have risen with employee capacity.
I would say this: in the future I think we are gonna have all sorts of robotics that will be able to use LLMs and vision models and stuff to do basic reason and coordination to automate a ton of tasks. The average person is basically going to be able to fit a micro-factory in their house that can knit all of their clothes, make circuit boards for all of the computers they need, stitch their wounds together, and such.
In the future, we won't even need to engage in the economy of mass production, and we will basically all be low effort self-sufficient sustainable farmers and manufacturers due to AI reducing the effectiveness of economies of scale.
No one will have conventional jobs, so we will each recreate the old economy on a tiny scale to avoid the expensive monopolies. A single person's job would be like operating a tiny factory that produces a certain type of insulin or a certain antibiotic, or some sort of resistor or tobacco or something. Like the idea of family farms extended to the industrial domain.
And all of this progress is being taken on for free at massive cost by these AI companies that think it will have the exact opposite effect, which is monetizable.
I think that LLMs can be used as a far more advanced search than google. Imagine you have some project that requires a certain part. You could spend hours browsing the internet for the best deal, or you run a local LLM that scrapes websites and does the shipping calculations and runs a reasoning model to decide if it is a good fit based on the criteria you give it, etc. You essentially have the shopping done for you, it is just a matter of one person designing the framework and open sourcing it.
Most searching isn't so much finding a direct answer to your query, but scoping out a general field of information where you don't even know what it is you want to know. LLMs give us the opportunity to script general reasoning tasks.
Maybe it is bad or neutral for labor in the short term, but in the long run I think it is worse for capital. A lot of the moat that capital has is the ability to organize labor. If anyone with a computer can do the work of 100 men, then when the 100 men get laid off they will all ask themselves "why can't I also just start a competitor where I automate all the tasks in the company?".
Thanks for reading my TED talk.
We’re willing to fork over money for things because those things require human effort to obtain, and we’d rather not expend it. In this new world, everything from the extraction of raw materials to the production of advanced technology would require no human effort. If our modern notions of property still persisted, however, then this doesn’t mean that people would simply have whatever they wanted. You need trees to get apples, you need a hole in the ground to get coal. Ultimately the limiting factor on everything would come down to land. Labor-time is replaced with land-time, because the land works itself. Not having land in this society would be like not having limbs or a brain in ours. You would have nothing to exchange in order to get the things you needed.
So I’d say that either the notion of property itself would change, or people without property would die until everyone had some amount of it, and people would generally occupy their time with various games that shuffled around the limited amount of land available as a proxy for social status. The flawed assumption that you make is that people would all have some amount of land in which to make their microfactory, but this would only be the case after lots of people died.
Not really. It would be good at doing generic, well-defined tasks but bad at doing specialized, novel tasks. You would still need some humans in the loop to get to the bottom of niche problems.
I agree that it would still go to hell without some type of Georgism or UBI or socialism. I agree that wealth will transfer to companies that control industrial means of production (like 3M or mining companies or intel or something), but it will also transfer out of companies whose moat is based on control of human capital (like accounting, software development, and law).
I think that even before AI, we are already seeing this sort of "land is everything" economy. Physical labor has largely been automated in the industrial revolution. Intellectual labor has been displaced not by newfangled AI mechanisms, but by information storage mediums and general pre-AI automation. If you are an artist, you are competing with all of the art that came before you. If you are an engineer, you are reinventing the wheel working on some sort of project that, if open sourced, would only need to be done once.
A major sense in which AI eliminates jobs is by acting as a bypass for copyright, it allows you to plausibly make a near-copy of something without a license. There is simply not an infinite amount of demand in the economy for intellectual labor. The thing that destroys the world as we know it is not so much AI, but information sharing and de-duplication of work. Open source would have destroyed the economy if AI didn't.
So everyone is working in the service sector now, it's unsustainable. Property prices keep going up, fertility rate keeps going down.
“Taking images and turning them into 3D environments using gaussian splats, depth and inpainting. Cool, but that’s a 3D GS pipeline, not a robot brain.”
A lot of vfx today is automated and things are possible that we’re just too cost prohibitive before. You could say “who wants to see digital art”. The moat is the artist realizing their vision - for the same $ spend you get significantly more art or higher quality art (eg first pass by AI with humans doing the refinement steps).
The boom in television is because of plummeting production and distribution costs for example