Genie 2: A large-scale foundation world model
deepmind.google
deepmind.google
That said; I cannot find any:
- architecture explanation
- code
- technical details
- API access information
Feels very DeepMind / 2015, and that's a bummer. I think the point of the "we have no moat" email has been taken to heart at Google, and they continue to be on the path of great demos, bleh product launches two years later, and no open access in the interim.
That said, just knowing this is possible - world navigation based on a photo and a text description with up to a minute of held context -- is amazing, and I believe will inspire some groups out there to put out open versions.
Had they released this two months earlier it would have been incredibly impressive. Now it's still cool and inspiring, but no longer as ground breaking. It's the cooler version that doesn't come with a demo or any hope of actually trying it out.
And with the things we know from Oasis's demo, the agent-training use case the post tries to sell for Genie 2 is a hard sell. Any attempt to train an agent on such a world would likely look like an AI Minecraft speedrun: generate enough misleading context frames to trick the AI into generating what you want
1. Non-cherrypicked in its consistency (if you look at the demonstrations in the Oasis blog post you can find specific cases of consistency which is an anomaly rather than the norm)
2. Is live-inferenced at 20fps. If you use Runway v3 which is a comparably larger and higher quality model (resolution and consistency) it might take a minute or two generate 10 seconds of video.
3. Is served (relatively) reliably at consumer scale (with queues of 5-10k concurrent players) which means that in order to save on GPU cost, you increase batch size and decrease model size to “fit” more players in 1 GPU.
I really wonder how much more stable Genie 2 is.
Secondly, any estimate of how much the price could fall in 5-10 years?
For reducing the price, ASICs like etched may be the way forwards [1]. The models will get bigger for a time, but there may be a lot of room for models that can exploit purpose-built hardware.
What would they do / how would they use this output to make a better AI?
For a straightforward example, this could help Waymo rehearse driving in various cities and weather / traffic settings
Even in games I expect a game playing model to exploit glitches present in world building one
I think it's great that Google is researching these, but I can't see the return and if there is it is many steps away
It's not clear if video models will follow the same trajectory.
In the intermediate term my guess is that this kind of world model will be useful for training 3D model generators, so that you can go from sketch -> running in-engine extremely quickly.
1: https://www.tweaktown.com/news/101466/oasis-ai-and-single-nv...
> This is.. super impressive. I'd like to know how large this model is. I note that the first thing they have it do is talk to agents who can control the world gen; geez - even robots get to play video games while we work. That said; I cannot find any:
> architecture explanation > code > technical details > API access information
Are there any game developers working on infinite story games? I don’t care if it looks like Minecraft, I want a Minecraft that tells intriguing stories with infinite quest generation. Procedural infinite world gen recharged gaming, where is the procedural infinite story generation?
Still, awesome demo. I imagine by the time my kids are in their prime video game age (another 5 years or so) we will be in a new golden age of interactive story telling.
Hey siri, tell me the epic of Gilgamesh over 40 hours of gameplay set 50,000 years in the future where genetic engineering has become trivial and Enkidu is a child’s creation.
Procgen games mainly work when the procedural parts are just a foundation for hand-crafted content to sit on, whether that's crafted by the players (as in Minecraft) or the developers (as in No Mans Sky after they updated it a hundred times, or Rougelikes in general).
I haven't added any words or phrases to it in years, but I still use it regularly and somehow it still surprises me. Maybe the Spelunky-type approach can be surprising for longer; that is, make a bunch of hand-curated bits and pick from them randomly: https://tinysubversions.com/spelunkyGen/
But that doesn't translate well to websites, trailers or demos. It's easier to wow people with graphics.
(Dwarf Fortress being much more focused on generating a whole world.)
Consider EVE Online. The stories it generates are Shakespearean and I defy anyone to argue that they have no plot.
I would go further and predict that stories generated by sufficiently advanced AI can explore much more interesting story landscapes because they need not be bound by the limitations of human experience. Consider what stories can be generated by an AI which groks mathematics humans don't yet fully understand?
You're not gonna get new intriguing stories from AI which only regurgitates what it's stolen. You're going to get a themeless morass without intention.
I also find it amusing how your example to Siri uses one of the oldest pieces of literature when you also tire of stories heard a thousand times before.
When did we start thinking this way? That things HAVE to get better and in fact to think otherwise is very negative? Is HN under a massive hot hand fallacy delusion?
Sure, progress will likely not be linear or without challenges, but we already have the human brain as proof that it is possible.
> We collected a vocabulary consisting of about 1500 basic words, which try to mimic the vocabulary of a typical 3-4 year-old child, separated into nouns, verbs, and adjectives. In each generation, 3 words are chosen randomly (one verb, one noun, and one adjective). The model is instructed to generate a story that somehow combines these random words into the story
You can do the same for generating worlds, just prepare good ingredients and sample at random.
These are “stories” in the most vacuous definition possible, one that is just “and then this happened” like a child’s conception of plot
For LLMs like GPT-4, this all seems reasonable to account for and assume the LLM is capable of processing, given appropriate guidance/frameworks (of which may be just classical programming).
Yes. This is an active research area. See https://github.com/yingpengma/Awesome-Story-Generation, which is not up to date.
The better you make this infinite narrative generator, the more complicated the world gets and the less compelling it gets to actually interact with any one story.
Stories thrive by setting their own context. They should feel important to the viewer. An open world with infinite stories can't make every story feel meaningful to the player. So how does it make any story feel meaningful? I suppose the story would have to be global, in which case, it crowds out the potential for fractal infinite storylines - eventually, all or at least most are going to have to tie back to the Big Bad Guy in order to feel meaningful.
Local stories would just feel mostly pointless. In Minecraft, all (overworld) locales are equally unimportant. Much like on Earth, why should you care about the random place you appeared in the world? The difference is that on Earth you tend to develop community as you grow and builds connections to the place you live, which can build loyalty. In addition, you only have one shot, and you have real needs that you must fulfill or you die forever. So you develop some otherwise arbitrary loyalties in order to feel security in your needs.
In Minecraft there's zero pressure to develop loyalty to a place except for your own real-life time. And when that becomes a driving factor, why wouldn't you pick a game designed to respect your time with a self-contained story? (Not that infinite games like Minecraft are bad, but they aren't story-driven for a good reason).
Now, a game like Dwarf Fortress is different because you build the community, the infrastructure, the things that make you care about a place. But it already has infinite story generation without AI and I'm not sure AI would improve on that model.
I'd say AAA games have been on track of "less fun" for at least half a decade. So this sounds like a natural next step.
- SimCity where you can read a newspaper about what's happening in your city that actually reflects the events that have occurred with interesting perspectives from the residents.
- Dwarf Fortress, but carvings, artwork, demons, forbidden beasts, etc get illustrations dynamically generated via stable diffusion (in the style of crude sketches to imply a dwarf made it perhaps?)
- Dwarf Fortress, again, but the elaborate in-game combat comes with a "narrative summary" which conveys first hand experiences of a unit in the combat log, which while detailed, can be otherwise hard to follow.
- Any fantasy RPG, but with a minstrel companion who follows you around and writes about what you do in a silly judgy way. The core dialogue could be baked in by the developers but the stories this minstrel writes could be dynamically generated based on the players actions. Example: "He was a whimsical one, who decided to take detour from his urgent hostage rescue mission to hop up and down several hundred times in the woods while trying on various hats he had collected. I have no idea what goes through this mans mind..."
I'm not sure if there is a word for it, but the kernel here is that everything is indirectly being dictated by the players actions and the games existing systems. The LLM/AI stuff isn't in charge of coming up with novel stories and core content, they are in charge of making the game more immersive by helping with the roleplay. I think this is the area they can thrive the most.
OTOH, lots of games come with DLC that add new stories with the same mechanics. There might be some additions or changes, but if you really like the mechanics, you can try it with a different plot. Remnant II has sucked a ton of my time because of that.
How so?
I could totally see generative AI add a ton more variety to crowds, random ambient sentences by NPCs (that are often notoriously just a rotation of a handful of canned lines that get repetitive soon), terrain etc., while still being guided by a human-created high level narrative.
Imagine being able to actually talk your way out of a tricky situation in an RPG with a guard, rather than selecting one out of a few canned dialogue options. In the background, the LLM could still be prompted by "there's three routes this interaction can take; see which one is the best fit for what the player says and then guide them to it and call this function".
Worst case, you get a soulless, poorly written game with very eloquent but ultimately uninteresting characters. Some games are already that today – minus the realistic dialogue.
Yes, sure, but that's not what I was responding to. AI adding detail, not infinite quest lines, is possibly a good use case.
> Worst case, you get a soulless, poorly written game with very eloquent but ultimately uninteresting characters. Some games are already that today – minus the realistic dialogue.
Some games, yes... why do we want more of those? Anyway, that's not the worst case. Worst case is incomprehensible dialogue.
By definition, an infinite game is as boring as real life. To make it interesting, the engine must be able to search for a good story based on player actions. You can see this today already. Many games will guide the player into one or a few predefined stories. A better game would not have them predefined, but generated on-the-fly based on player actions.
That’s actually a use case I can understand- and what’s more I think that humans could generate training data (story “prototypes”?) that somehow (?) expand the phase space of story-types
Ironic though - we can build AI that could be creative but it’s humans that have to use science and logic because AI cannot?
It seems like it’d be more useful to have the model generate the raw artifacts, world map, etc. and let the engine do the actual rendering.
Now, imagine training it on thousands of hours of PoV drone footage from Ukraine, and then using that to train autonomous agents.
If game assets are cheap to generate you’ll see small teams or even solo developers willing to take more creative risks
So I see the most likely outcome is a lot of dogshit and Steam being forced to make draconian moves to protect the integrity of the store.
I don’t see why AI will be any different. All that’s changed is ratio of potential creators to the general population. Most of it is going to be slop regardless because of economic incentives.
If AI pushes that up to 98%, that means you have to look through 5 times as much crap to get the good stuff.
Expect something similar if video games, interactive 3D is cheap to produce.
Filtering is a much easier problem to solve and abundance a preferable scenario.
It's a great idea. We want more than an open-world. We want an open-story.
Open-story games are going to be the next genre that will dominate the gaming industry, once someone figures it out.
I think fully conversational games (voice to voice) with dynamic story lines are only a decade or two away, pending a minor breakthrough in model distillation techniques or consumer inference hardware. Unlike self driving cars or AGI the technology seems to be there, it’s just so new no one has tried it. It’ll be really interesting to see how game designers and writers will wrangle this technology without compromising fun. They’ll probably have to have a full agentic pipeline with artificial play testers running 24/7 just to figure out the new “bugspace”.
Can’t wait to see what Nintendo does, but that’s probably going to take a decade.
"There’s no question in my mind that such software could generate reasonably good murder mysteries, action thrillers, or gothic romances. After all, even the authors of such works will tell you that they are formulaic. If there’s a formula in there, a deep learning AI system will figure it out.
Therein lies the fatal flaw: the output will be formulaic. Most important, the output won’t have any artistic content at all. You will NEVER see anything like literature coming out of deep learning AI. You’ll see plenty of potboilers pouring forth, but you can’t make art without an artist.
This stuff will be hailed as the next great revolution in entertainment. We’ll see lots of prizes awarded, fulsome reviews, thick layers of praise heaped on, and nobody will see any need to work on the real thing. That will stop us dead in our tracks for a few decades."
Until then, I'll stick with human art
For now though, it's very good at making thing similar to what's already made.
"Here's a photo of me and my wife, draw me and my wife as a cowboy in the style of a Dilbert cartoon shooting a gun in the air" can't be done by AI as far as I know, which is why artist are still employed throughout the world.
Jackie Chan is not a good example because he's a famous person it may have been trained on. I used myself as an example because it would be something that is novel to the AI, it would not be able to rely on it's training to draw me, as I am not famous.
I'd say its less the work they have been trained on, and more what they have been reenforced to do, which is stay on topic. it causes them to dwell instead of drift.
It is like saying the paint brush and canvas lack creativity. Creativity is not a property of the tool, it is a property of the artist.
We also have a very poor understanding of human creativity from selection bias.
Last weekend I found a book at the library that was Picasso's drawings 1966 to 1968. There must have been 1000-1500 drawings in this book. Many were just half finished scribbles.
The average person seems to believe though that the master artist only produces masterpieces because they didn't bother to look at all the crap.
How is that different from humans? Do we get magic inspiration totally separate from anything we’ve learned?
Show me any great book, song, movie, building, sculpture, painting. I will tell you the influences the artist trained on.
I’m not sure “can produce good outputs, can produce terrible outputs” is a good way to differentiate humans and LLMs.
I’m not going to claim this is Pulitzer-worthy, but it seems fairly novel:
> In Spiritfall: A Symphony of Rust and Rose Petals, readers traverse the borders of time, taste, and consciousness in a genre-bending epic that effortlessly fuses neo-noir detective intrigue, culinary magic realism, and post-biotechnological body horror under the simmering threat of a cosmic opera. Set in a floating, living city grown from engineered coral-harps, the story follows a taste-shaper detective tasked with unraveling the murder of an exiled goddess whose voice once controlled the city’s very tides. As he navigates sentient cooking knives, ink-washed memory fractals, and teahouses that serve liquid soul fragments, he uncovers conspiracies binding interdimensional dream-chefs to cybernetic shamans, and finds forbidden love in a quantum greenhouse of sentient spices. Every chapter refracts expectations, weaving together genres never before dared, leaving readers both spellbound and strangely hungry for more.
If I were using this for real I’d ask it to iterate, to create a story arc, etc.
I'd be curious if it could explain those concepts and use them in consistent ways. If so, I'd be curious how novel it could really get. Is it just going to be repackaging well-trod scifi and fantasy devices, or studied philosophy? Or could it offer us a story with truly new understandings? For example, to my knowledge, House of Leaves is something truly novel. It's probably not the first book with intentional printing errors, or with layered narration, or with place-horror, etc. But I think House of Leaves is pretty widely considered a sort of "step forward" for literature, having a profound impact on the reader unlike anything that came before it.
(A really serious discussion will require analyzing exactly what that impact is and how it's novel.)
LLMs have no life experiences, are only familiar with the most mainstream literary works with the most mainstream internet discussions, and use a fancy RNG formula on the next most likely word as a not so great substitute for imagination.
Those beautiful worlds took a lot of money to make and the studios are smart enough to realize consumers are apathetic/stupid enough to accept much lower quality assets.
The top end of the AAA market will use this sparingly for the junk you don't spend much time on - stuff the intern was doing before.
The bottom of the market will use this for virtually everything in their movie-to-game pipeline of throwaway games. These are the games designed just to sucker parents and kids out of $60 every month. The games that don't even follow the story of the movie and likely makes the story worse.
Strangely enough this is where the industry makes the vast majority of it's day-to-day walking around cash.
For example, right now if you save an entire village from an attacking tribe of orcs, only a handful of NPCs even say anything, just a nice little "thanks for saving our town!" and then 2 villages over the NPCs are completely unaware of a mighty hero literally solo tanking an entire invading army.
Why is that?
Well you'd need lots of, somewhat boring but important, dialogue written, and you'd need tons of voice lines recorded.
Both those are now solvable problems with generative AI. AI generated dialogue is now reasonably high quality, not "main character story arc" high quality, but "idle shop keeper chit chat" quality for sure, it won't break immersion at least. And the quality of writing from AI is fine for 2 or 3 sentences here and there.
I'll be soon releasing a project showing this off at https://www.tinytown.ai/ the NPC dialogue is generated on a small LLM that can be ran locally, and the secret of even high quality voice models is that they don't require a lot of memory to run.
I predict that in another 4 or 5 years we'll see a lot of models ran at the edge on video game consoles and home PCs, fleshing out game worlds.
To me this basically describes God, once you get all the mods and DLC.
We should celebrate creation of tools and capabilities, while also acknowledging that there are many layers yet to be completed before your very cool ad hoc video game request can be delivered.
Star Wars isn't great- but if a single company didn't own Star Wars we'd likely see some companies competing with really great Star Wars stuff- and others sucking at it. The issue is economic.
Dwarf fortress?
> Delamain was a non-sentient AI created by the company Alte Weltordnung. His core was purchased by Delamain Corporation of Night City to drive its fleet of taxicabs in response to a dramatic increase in accidents caused by human drivers and the financial losses from the resulting lawsuits. The AI quickly returned Delamain Corp to profitability and assumed other responsibilities, such as replacing the company's human mechanics with automated repair drones and transforming the business into the city's most prestigious and trusted transporting service. However, Delamain Corp executives underestimated their newest employee's potential for growth and independence despite Alte Weltordnung's warnings, and Delamain eventually bought out his owners and began operating all aspects of the company by himself. Although Delamain occupied a legal gray area in Night City due to being an AI, his services were so reliable and sought after that Night City's authorities were willing to turn a blind eye to his status.
When AR glasses get good enough to wear all day, I've really been wanting to make a real-life ad blocker.
Consider the use where you seed the first frame from a real world picture, with a prompt that gives it a goal. Not only can you see what might happen, with different approaches, and then pick one, but you can re-seed with real world baselines periodically as you're actually executing that action to correct for anything that changes. This is a great step for real world agency.
As a person without aphantasia, this is how I do anything mechanical. I picture what will happen, try a few things visually in my head, decide which to do, and then do it for real. This "lucid dream" that I call my imagination is all based on long term memory that made my world view. I find it incredibly valuable. I very much rely on it for my day job, and try to exercise it as much as possible, before, say, going to a whiteboard.
Anything can be a huge (or a microscopic) step on a journey when the destination is vague and its distance is unknown.
A key reason why current Large Multimodal Models (LMMs) still have inferior visual understanding compared to humans is their lack of deep comprehension of the 3D world. Such understanding requires movement, interaction, and feedback from the physical environment. Models that incorporate these elements will likely yield much more capable LMMs.
As a result, we can expect significant improvements in robotics and self-driving cars in the near future.
Simulations + Limited robot data from labs + Algorithms advancement --> Better spatial intelligence
which will lead to a positive feedback loop:
Better spatial intelligence --> Better robots --> More robot deployment --> Better spatial intelligence --> ...
> That large alien? That's a tree. > That other large alien? It's a bush. > That herd of small creatures? Fugghedaboutit > The lightning storm? I can do one lightning pole. > Those towering baobob/acacia hybrids? Actually only two stories tall.
It feels so insulting to the concept artist to show those two videos off.
Welcome to HackerNews.
But these video game models I think are a lot less capable, because there just aren't that many video games out there, they aren't all that different from one another, and they're all just finite state machines. WASD, desert, jungle, ruins, city. Hell half of them share the very same game engine!
How many massive, cohesive, open world games are there? Red Dead and GTA5... Gee, I wonder why so many of their examples look like that?
That one would reimagine the world any time you look at the sky or ground. Sounds like Genie2 solves that: "Genie 2 is capable of remembering parts of the world that are no longer in view and then rendering them accurately when they become observable again."
Google is firing warning shots to kill off interest in funding competing startups in this space.
I suspect that in 6 months it won't matter as we'll have completely open source Chinese world models. They're already starting to kill video foundation model companies' entire value prop by releasing open models and weights. Hunyuan blows Runway and OpenAI's Sora completely out of the water, and it's 100% open source. How do companies like Pika compete with free?
Meta and Chinese companies are not the leaders in the space, so they're salting the earth with insanely powerful SOTA open models to prevent anyone from becoming a runaway success. Meta is still playing its cards close to its chest so they can keep the best pieces private, but these Chinese companies are dropping innovation left and right like there's no tomorrow.
The game theory here is that if you're a foundation model "company", you're dead - big tech will kill you. You don't have a product, and you're paying a lot to do research that isn't necessarily tied to customer demand. If you're a leading AI research+product company, everyone else will release their code/research to create a thousand competitors to you.
There is still an enormous amount of long hanging fruit that anyone can harvest right now, but eventually big advances are going to require big budgets and I can only imagine how technically tight lipped they will be with those.
Basically, the foundation model companies are outsourced R&D labs for big tech. They can be kept at arms length (like OpenAI with Microsoft and Anthropic with Amazon) or be bought outright (like Inflection, although that was a weird one).
Both OpenAI and Anthropic are trying to move away from being pure model companies.
> If you're a leading AI research+product company, everyone else will release their code/research to create a thousand competitors to you.
Trillion dollar question - is there a competitive edge / moat in vertical integration in AI? Apple proved there was in hardware + os (which were unbundled in wintel times). For AI, right now, I can't see one, but I'm just a random internet comentator, who knows.
While it would be interesting if Chinese companies were releasing their best full models as an intentional strategy to reduce VC funding availability for western AI startups, it would be downright fascinating if the Chinese government was supporting this as a broader geopolitical strategy to slow down the West.
It does make sense but would require a remarkable level of insight, coordination and commitment to a costly yet uncertain strategy.
The overall cost for the Chinese government is probably very small in the grand scheme of things. And it makes a lot of sense from a geopolitical strategy.
I am less worried for AI research+product companies: they have likely secured revenue streams with real customers and built domain knowledge in the meantime.
GameGen-X came out last month. https://arxiv.org/html/2411.00769v1
For these kinds of models to be "playable" by humans (and, I'd argue, most fledgling AI agents), the world state needs to be encoded in the context, not just a visual representation of what the player most recently saw.
Which is a big problem for the agent-training use case they keep reiterating on the website. Agents are like speedrunners: if there is a stupid exploit, the agent will probably find and use it. And for Oasis the speedrunning meta for getting to the nether is to find anything red, make it fill the screen, and repeat until the world-generating AI thinks you look at lava and must be in the nether
Something I find interesting about generative AI is how it adds a huge layer of flexibility, but at the cost of lots of computation, while a very narrow set of constraints (a traditional program) is comparatively incredibly efficient.
If someone spent a ton of time building out something simple in Unity, they could get the same thing running with a small fraction of the computation, but this has seemingly infinite flexibility based on so little and that's just incredible.
The reason I mention it is because I'm interested in where we end up using these. Will traditional programming be used for most "production" workloads with gen AI being used to aid in the prototyping and development of those traditional programs, or will we get to the point where our gen AI is the primary driver of software?
I assume that concrete code will always be faster and the best way to have deterministic results, but I really have to idea how to conceptualize what the future looks like now.
But modern search is hampered by people responding to algorithmic indexes. Algorithms responding to metadata without directly evaluating content enabled a world of SEO and low quality websites suddenly being discoverable as long as they narrow their focus enough.
So longer term it’s going to be an arms race between the output of Generative AI and people trying to keep updating their models. In 20 years people will get much better at using these tools, but the tools themselves may be less useful. I wouldn’t be surprised if eventually someone sneaks advertising into the output of someone else’s model etc.
The question is ambiguous without defining how much worse the dataset is.
Even facts age. In 2004, Pluto was still classified as a planet. Not such a big deal on its own, but stale data gets a little bit worse every day.
It will likely be a mix of both concrete code and live AI generated experiences, but even the concrete code will likely be partially AI generated and modified. The ratio will depend on how reliable vs creative the software needs to be.
For example, no AI generated code running pacemakers or power plants. But game world experiences could easily be made more dynamic by generative AI.
There are already a number of techniques for procedurally-generating a world (including Markov-based systems).
The problem with replacing procedural world generation with LLM generation are a) you need to obtain a data set to train it, which doesn’t commercially exist, or train it yourself, b) there’s a fundamental need to iterate on the design, which LLMs do not cope with well at all, c) you need to somehow debug issues and fix them. That’s quite apart from the quality issues, cost and power usage.
I mean we're already there with Copilot, Cursor and other tools that use LLMs to assist in coding tasks.
What makes it little? This is the difference between von Neumann architecture and Harvard architecture.
Games are about interactions, and this actively works against it. You don't want the model to infer mechanics, the designer needs deep control over every aspect of it.
People mentioned using this for prototyping a game, but that's completely meaningless. What would it even mean to use this to prototype something? It doesn't help you figure out anything mechanically or visually. It's just, "what if you were an avatar in a world?" What do you do after you run around with your random character controller in your random environments?
I think the most useful part of this is the world generation part, not the mechanics inference part.
people sell entire franchises off of a few pre-rendered generic-fantasy still images -- I would have to disagree with the premise that this is useless as a visual concept tool.
I agree with your notions about integration into an existing game.
I would consider a different approach, when the training phase watches games (or video recordings) and refines the formulas that describe its physics, the geometry of the area, the optics, etc. The result would be a "map" that is "playable" without much if any inference involved, and with no time limitation dictated by the size of the context to keep.
Very certainly, video game map generation by AI is a thing, and creating models of motion by watching and then fitting reasonably simple functions (fewer than millions of parameters) is also known.
I cannot be the first person to think about such possibilities, so I wonder what does the current SOTA look like there.
I think this is precisely why they're doing it. Video games are where the data is, because the analytical solutions can generate it.
They aren't trying to make a video game. They're trying to make an android.
Differentiable Rendering [1] is the closest thing to what you are describing. And yes, people have been working on this for the same reason you outline, it is more data/compute efficient and hence should generalize better.
[1] https://blog.qarnot.com/article/an-overview-of-differentiabl...
But also: > While cool, this also seems utterly wasteful. Video games offer known "analytical" solutions for the interactions that the model provides as a "statistical approximation", so to say.
A bit of the same debate as people calling LLMs a "blurry JPEG of the web" and hence useless.
Yes this is a statistical approximation to an analytical problem... but that's a very reductive framing to what is going on. To find the symbolic/analytical solution here would require to constrain the problem greatly: not all things on the screen have a differentiable representation, for example complex simulations might involve some kind of custom internal loop/simulation.
You waste compute to get a solution that can just be trained on billions of unlabeled (synthetic) examples, and then generalize to previously unseen prompts/environments.
Noone knows yet. AI technology like this is closer to scientific research than it is to product development. AI is basically new magic, and people are in a "discovery" phase where we are still trying to figure out what is possible. Nothing of value was immediately created when they discovered DNA. Productization came much later when it was combined with other technologies to fit a particular use case.
Prompt: Here's a blueprint of my new house and a photo of my existing furniture. Show me some interior design options.
You just don't like AI.
It can be used for training agents, prototyping, video generation, and is quite possibly a glimpse of a whole new type of entertainment or a new way to create video games.
What's the point of the massive amount of money spent on video games in general? Or all of the energy spent moving people back and forth to an office? Or expensive meals at restaurants? Or trillions in weaponry? Or television shows or movies?
I feel like sharing early closed-source blog-posts is part of the research process. I'm sure someone in this thread has thought of a use case that the Google team missed. Open/closed source arguments here feel premature IMO.
This is just a marketing fluff piece that does not benefit anyone and is ego stroking at best.
I still think things like this are important, and at least give folks a bit of time to ideate on what will be possible in a few years. Of course having the model or architecture on hand would be nice, but I'm not holding that against Google here.
asked the same thing a while back, and the answers boiled down to "somehow helps RL agents train". but how exactly? no clue
[edited out some barbs I wrote because I find some comments on this website REALLY annoying]
The major difference being the former scales very poorly for generating training data compared to the latter. Genie 2 is not even a video game and has worse fidelity that video games, the upside is it probably scales even better than video games for generating training scenarios. If you want androids in teal life, Genie 2 (or similar systems) is how you bootstrap the agent AI. The training pipeline will be: raw video -> Genie 2 -> game engine with rules -> physical robot
One of those arrows is not like the others
Any model would have to succeed in one stage before it can proceed to the next one.
Self driving vars have cameras as part of their sensor suite, and have models to make sense of sensor data. Video will help with perception and classification (understanding the world) with no agency needed. Game-playing will help with planning, execution, and evaluation. Both functions are necessary, and those that come after rely on earlier capabilities
For example, let's say that the car is approaching an intersection, and suddenly sees a puddle on the road to the left getting brighter - a visual world model like this might extrapolate a scenario that the brightness is the result of a car moving towards the intersection assigning this some probability, and signing another probably to a scenario that it's just a flickering headlight, and the car would then decide whether and how much to slow down.
In this example there is a sensor, but it definitely doesn't tell the robot "exactly what is there", and while we could try to write rules about what it should do, the Bitter Lesson tells us it's better to just let it create its own model.
Page 8 of the Genie 1 paper: https://arxiv.org/abs/2402.15391
That's nice. I'm not completely disabled, but I am disabled, and I very much would appreciate them, as my capability to do things over the longer term is very much not going to go in the direction of improving. As it is, there are a lot of things I now rely on people for, that at one time, I did not.
Whilst I recognise its probably not going to happen in a time span that is useful to me, I do wish it could, so that I could be less of a burden on those around me, and maintain a relative level of independence.
Unless you have a young/quick death, there's a really good chance you will be, too.
You could use real video games to do this but I guess there'd be a risk of over-fitting; maybe it would learn too precisely what a staircase looks like in Minecraft, but fail to generalize that to the staircase in your home. If they can simulate dream worlds (as well as, presumably, worlds from real photos), then they can train their agents this way.
This would only be training high-level decision policies (ie, WASD inputs). For something like a robot, lower level motor control loops would still be needed to execute those commands.
Of course you could just do your training in the real world directly, because it already has coherence and plenty of environmental variety. But the learning process involves lots of learning from failure, and that would probably be even more expensive than this expensive simulator.
Despite the claims I don't think it does much to help with AI safety. It can help avoid hilarious disasters of an AI-in-training crashing a speedboat onto the riverbank, but I don't think there's much here that helps with the deeper problems of value-alignment. This also seems like an effective way to train robo-killbots who perceive the world as a dreamlike first-person shooter.
To have this perspective you must believe that this will never get better than it currently is, its limitations will never be fixed, and it will never lead to any other applications. I don't know how people can continue to look at these things with such a lack of imagination given the pace of progress in the field.
Dreaming and sleeping is incredibly expensive, we spend 33% of our "availability" on average asleep.
This kind of work is a step toward building similar tools for general AI agents (IMHO).
All motivation to make further games is removed because now "somebody" can spool out a 3D adventure game instantly with a line of text. It implies you'll waste a year of your time, and just before release, out pops the dramatically better AI product to steal away all further business and reduce all time you've spent to meaningless. Everybody then waits indefinitely for the "endgame gear." https://xkcd.com/989/
Exactly like LLMs and image generators almost completely took away all business for normal writers and normal painters, because now all managers want is the AI. Now there's endless articles about how "somebody" prefers "AI" for every task. Now the market won't invest in anything unless it has "AI" in the name. Now people idiotically add "AI" to everything just to have the investment.
I will add the totally inconsistent backgrounds in the "prototyping" example suggests the AI is simply cribbing from four different games with a flying avatar, which makes it kind of useless unless you're prototyping cynical AI slop. And what are we even doing here by calling this a "world model" if the details of the world can change on a whim? In my world model I can imagine a small dragon flying through my friend's living room without needing to turn her electric lights into sconces and fireplaces.
To state the obvious: if you train your model on thousands of hours of video games, you're also gonna get a bunch of stuff like "leaves are flat and don't bend" or "sometimes humans look like plastic" or "sometimes dragons clip through the scenery," which wouldn't fly in an actual world model. Just call it "video game world model!" Google is intentionally misusing a term which (although mysterious) has real meaning in cognitive science.
I am sure Genie 2 took an awful lot of work and technical expertise. But this advertisement isn't just unscientific, it's an assault on language itself.
That's because it's using video game data for training footage because it's cheap and easy to generate. It would not be simulating video game gravity if it was training on real world video inputs.
>if you train your model on thousands of hours of video games
What if you train the same model on thousands of hours of sensor data from real, physical robots?
I have a feeling many AI researchers are trying to fix things which are not broken.
Game engines are not broken, no reasonable amount of AI TFlops going to approach a professional with UE5. DAWs are not broken, no reasonable amount of AI TFlops going to approach a professional with Steinberg Cubase and Apple Logic.
I wonder why so many AI researchers are trying to generate the complete output with their models, as opposed to training model to generate some intermediate representation and/or realtime commands for industry-standard software?
Now, how Google plans to make money with all this bleeding edge research, that's the mystery.
Seriously, I wish more than anything I was kidding.
Decart (Oasis) raised $25 million at $500 million valuation.
World Labs raised $230 million.
> get absorbed into some hype mill startup
Recruiting
Interesting they're framing this more from the world model/agent environment angle, when this seems like the best example so far of generative games.
720p realtime mostly consistent games for a minute is amazing, considering stable diffusion was originally released 2ish years ago.
People of the internet, you were right. Now, this is incredible.
Now?
I mean, I don't know man?
With this Genie 2 sneak peak, it all just makes World Labs' efforts look sad. Did they really think better funded independents and majors would all not be interested in generating 3D worlds?
This is a GUBA moment. If you're old enough to know, then you know.
Seems that it's only "consistent" up to a minute, but if the progress keeps the same rate.. just wow.
For reference:
19th century
evolution by natural selection as science
electromagnetism
germ theory of disease
first law of thermodynamics
--------------------------------------------
20th century
general relativity
quantum mechanics
dna structure
penicillin
big bang theory
--------------------------------------------
21st century
crisp
deep learning
20th century: general/special relativity, radioactive decay, discovery of the electron
21st century: crisp and deep learning
Hard to argue that the big science of the first 20 years of the previous century looks way more impact than crisp and deep learning put TOGETHER.
This is naivete on the scale of "Cars were much safer 70 years ago".
But DNA sequencing and biologics have revolutionized medicine and changed lives.
Also, the computer as phone took it from 100M's mostly business users buying optical disks to 3+B everyday people getting regular system updates and apps on demand accessing real-time information. That change alone far outweighs the impact of anything produced by advanced physics.
As a result we, as developers, now have the power to deliver both messages and experiences to the entire world.
Ideas are cheap, and progress is virtually guaranteed in intellectual history. But execution is exquisitely easy to get wrong. Genie 2 is just Google's first bite at this apple, and milestones and feedback are key to getting something as general as AI right. Fingers crossed!
This is huge, the Minecraft demos we saw recently we're just toys because you couldn't actually do anything in them.
For the time being I will gloss over the fact this might just be a consumer facing product for Google that ends up having nothing to do with younger developers.
I'm torn between two ideas:
a. Show kids awesome stuff that motivates them to code
b. Show kids how to code something that might not be as awesome, but they actually made it
On the one hand you want to show kids something cool and get them motivated. What Google is doing here is certainly capable of doing that.
On the other hand I want to show kids what they can actually do and empower them. The days of making a game on your own in your basement are mostly dead, but I don't think that means the idea of being someone who can control a large amount of your vision - both technical and non-technical - is important.
Not everyone is the same either. I have met kids that would never spend a few hours to learn some Python with pygame to get a few rectangles and sprites on screen that might get more interested if they saw something this flashy. But experience also tells me those kids are extremely less likely to get much value from a tool like this beyond entertainment.
I have a 14 year old son myself and I struggle to understand how he sees the world in this capacity sometimes. I don't understand what he thinks is easy or hard and it warps his expectations drastically. I come from a time period where you would grind for hours at a terminal pecking in garbage from a magazine to see a few seconds of crappy graphics. I don't think there should be meaningless labor attached to programming for no reason, but I also think that creating a "cost" to some degree may have helped us. Given two programs to peck into the terminal, which one do you peck? Very few of us had the patience (and lack of sanity) to peck them all.
It's fascinating how much understanding of the world is being extracted and learned by these models in order to do this. (For the 'that's not really understanding' crowd, what definition of 'understanding' are you using?)
The current tooling we have is just way too good to just discard it, think of Maya, Blender and the like. How will these interfaces, with the tools they already provide, enable sculpting these word-based worlds?
I wonder if some kind of translator will be required, one which precisely instructs "User holds a brush pointing 33° upwards and 56° to the left of the world's x-axis with a brush consisting of ... applied with a strength of ...", or how this will be translated into embeddings or whatever that will be required to communicate with that engine.
This is probably the most exciting time for the CG industry in decades, and this means a lot, since we've been seeing incredible progress in every area of traditional CG generation. Also a scary time for those who learned the skills and will now occasionally see some random persons doing incredible visuals with zero knowledge of the entire CG pipeline.
What this should say to you instead is that stuff is really bad on training data side if you start scraping billions of game streams on internet - hard to imagine if there is a bigger chunk of training data than this. Stagnation incoming.
Can we let another models generate in this models's world and vice versa?
What if both output in a single instance of a world? What if both output in their own private world and only share data about location and some other metrics?
But, I didn’t expect this much progress towards that quite this fast…
A good video game is far more the world building, the story, the creativity or "uniqueness" of the experience, etc.
Currently this seems to generate fairly generic looking and shallow experiences. Not hating though. It's early days obviously.
These DeepMind guys play Factorio, they don't play Atari games or shooters, so why aren't they thinking about that? Or maybe they are, and because they know a lot about Factorio, they see how hard it is to make?
There's a lot of "musing" as you say.
It would play to the medium's strengths -- any "glitches" the player experiences could be seen as diagetic corruptions of reality.
The moment we get parameterized NeRF models running in close-to-realtime, I want to go for it.
Of all things this must be the most boring use case for this crazy looking new technology. But hybrid video meetings have always annoyed me and I think to myself that surely there must be a better way (and why hasn't it arrived yet?).
I love the advancement of the tech but this still looks very young and I'd be curious what the underlying output code looks like (how well it's formatted, documented, organized, optimized, etc.)
Also, this seems oddly related to the recent post from WorldLabs https://www.worldlabs.ai/blog. Wonder if this was timed to compete directly and overtake the related news cycle.
This also means that my dreams will keep looking like this iteration of Genie 2, but computer will scale up and the worlds won't look anything like my dreams anymore in next versions (its already more colorful anyway).
I remember image generation use to look like dreams too in the beginning. Now it doesn't look anything like that.
I remember there was a brief window where some gamers bought a Physx card for high fidelity physics in games. Ultimately they rolled that tech in to the CPUs themselves right?
Depending on how controllable the tech ends up being, I suppose. Could be anywhere from a gimmick (which is still nice) to a game engine replacement.
This is the Unreal Engine killer. Give it five years.
We need to calm down with the clickbait-addled thinking that "this new thing kills this established powerful tested useful thing." :-)
Game developers have been discussing these tools at length, after all, they are the group of software developers who are most motivated to improve their workflow. No other group of software developers comes close to gamedevs' efficiency requirements.
The 1 thing required for serious developers is control. As such, game engines like Unreal and in-house engines won't die.
Generative tools will instead open up a whole new, but quite different, way of creating interactive media and games. Those who need maximum control over every frame and every millisecond and CPU cyle will still use engines. The rest who don't will be productive with generative tools.
These models won't need you to retopo meshes, write custom shaders, or optimize Nanite or Lumen gameplay. They'll generate the final frames, sans traditional graphics processing pipeline.
> The 1 thing required for serious developers is control
Same with video and image models, and there's tremendous work being done there as we speak.
These models will eventually be trained to learn all of human posture and animation. And all other kinds of physics as well. Just give it time.
> Those who need maximum control over every frame and every millisecond and CPU cyle will still use engines.
Why do you think that's true? These techniques can already mimic the physics of optics better than 80 years of doing it with math. And they're doing anatomy, fluid dynamics, and much more. With far better accuracy than game engines.
These will get faster and they will get controllable.
Brother, you're preaching to the choir. I've been shilling generative tools for gamedev far harder than you are in your reply. :-)
But I'm just relaying to you what actual gamedevs working and writing code right now need and for the foreseeable future for which projects have been started or planned. As Mike Acton says, "the problem is the problem".
> These techniques can already mimic the physics of optics better than 80 years of doing it with math.
I encourage you to talk to actual gamedevs. When designing a game, you aren't trying to mimic physics: you're trying to make a simulation of physics that feels a certain way that you want it to play. This applies to fluid dynamics, lighting/optics, everything.
For example, if I'm making a saling simulator, I need to be able to script the water at points where it matters for gameplay and game-feel, not simulate real physics. I'm willing to break the rules of physics so that my water doesn't act or look like real water but feels good to play.
Movement may be motion captured, but animation is tweaked so that the characters control and play in a way that the game designer feels is correct for his game.
If you haven't designed a game, I encourage you to try to make a simple space invaders clone over the weekend, then think about the physics in it and try to make it feel good or work in an interesting way. Even in something that rudimentary, you'll notice that your simulation is something you test and tweak until you arrive at parameters that you're happy with but that aren't real physics.
I strongly disagree that you need to cater to existing workflows. There's so much fertile ground in taking a departure. Just look at what's happening with animation and video. People won't be shooting on Arri Alexas and $300,000 glass for much longer.
I didn't say that these tools need to though. :-)
I said that actual high-end game developers need precise control over every aspect of their game. A developer needs to be able to say something as simple as: "I want to make my particle system to run at 30ps, while my cloth animation is 120fps, while my logic is at 60fps."
> I've written my own 2D and 3D game engines as well as worked in Unreal. I'm currently working on a controllable diffusion engine using Bevy.
Then you know all that I'm suggesting already! You probably have a list of the typical problems that game engine programmers are trying to solve when they build their own engines or have to modify Unreal Engine itself. You could even just watch GDC[^0] or the Graphics Programming Conference[^1] and ask how these tools solve the problems discussed.
Generative tools will create a new way of making games or game assets, but they won't eliminate the current way of making games.
Since you're building these generative tools alongside your game, you can demonstrate how they solve the kinds of problems game engine programmers need to solve and there's no need for us to misrepresent either side of the equation. Just give a presentation or publish an essay showing engine problems being solved at the standard a typical studio needs.
[^0]: https://youtube.com/@gdconf?si=F_n4G4zxQSny8BNC
[^1]: https://youtube.com/playlist?list=PLLaly9x9rqjsXLW1tMFruyh_6...
We'll see which so-called AI-companies are really "dying" when either a correction, market crash or a new AI winter arrives.
Many of the current AI models have their roots in games: Chess, Go, etc.
Lighting, gravity, character animation and what not internalized by the model... from a single image...!
Statistical models will output a compressed mishmash of what they were trained on.
No matter how hard they try to cover that inherent basic reality, it is still there.
Not to mention the upkeep of training on new "creative" material on a regular basis and the never ending bugs due to non-determinism. Aside from contrived cases for looking up and synthesizing information (Search Engine 2.0).
The Tech Industry is over investing in this area exposing an inherent bias towards output rather than solving actual problems for humanity.
There was another quite similar model from a different group within the last month or so. I can't remember if they released any weights or anything or the name of it. But it was the same concept.
Yippee finally google posts a non confirming cookie popup with no way to reject the ad cookies!
I'll keep my stance, give it two years and very realistic movies, with plot and everything, will be generated on demand.
What you're talking about is a minor jump from the SOTA, much smaller than what we've already see in these two years.
I'll match any 5-figure amount you propose. I also know an escrow service we can trust.
https://x.com/mrjonfinger/status/1865161230706520472
Let's do this, Shasseem.
https://x.com/MKBHD/status/1866152437838393797
Please, please, please take that bet my "South Asian" friend.
That would be fun to use, but ultimately pointless. An AI model will generate things that are _statistically plausible_ ; solving crimes usually requires finding out the _truth_.
Again, the robocop glitch scene - in real life, Kinney's family would have suied, I guess ?
What you haven't been able to do so far, after many years of trying, is to go from the virtual, to the real. Go from Arcanoid to a robot that can play, I dunno, squash, without dying. A robot that can navigate an arbitrary physical location without drowning, or falling off a cliff, or getting run over by a bus. Or build any Lego kit from instructions. Where's all that?
You've conquered games. Bravo! Now where's the real world autonomy?
Look. The other poster also said "Waymo" but I'm talking about DeepMind. It's DeepMind that promises to conquer the world with Deep Reinforcement Learning, and it's DeepMind that keeps showing us how great their DRL agents work in virtual worlds, like minecraft or starcraft, or how well they work on Chess and Go, but still haven't been able to demonstrate the application of those powerful learning approaches to real-world environments, except for very strictly controlled ones. Waymo's stuff works in the real world (although they do have remote safety drivers much as they try to downplay the fact) but they're also not pretending that they'll do it all with one big DRL "generalist" agent. That's DeepMind's schtick.
For example, it was, I believe, DeepMind that recently publicised some results about legged robot football, where the robots were controlled by agents trained with DRL in a simulation. That's robot football: two robots (yeah, no teams) kicking a ball in the safest of safe environments: a (reduced-size) football field with artificial grass, probably padded underneath (because robots) and no other objects in the play area (except anxious researchers who have to pull the robots on their feet once in a while). Running in the physical world in principle, but in practice nothing but a tech demo.
Or take the other Big Idea, where they had a few dozen robot arms reaching for various little plastic bits in a (specially-made) box to try and learn object manipulation by real-world DRL. I can find a link to those things if you want, but that robot arm project was a few years ago and you haven't heard anything from them since because it was a whole load of overpromising and it failed.
That kind of thing just doesn't generalise. More than that: it's a total waste of time and money. And yet DeepMind keeps banging the drum. They keep trying to convince everyone and themselves that training DRL agents in virtual environments has anything to do with the real world, and that it's somehow the road to AGI. "Reward is all you need". Yeah, OK.
Btw, Waymo is not using DRL, at least not exclusively. They use all sorts of techniques but from what I understand they do a hell of a lot of good, old-fashioned, manual programming to deal with all the stuff that magickal deep learning in the sky can't deal with.
Waymo absolutely uses simulated multi-agent environments to improve their cars reliability; here's an example research artifact: https://waymo.com/research/waymax/
I think you're deluding yourself about the progress in this area. There's an enormous amount of specialized work in bringing results from research to market. WayMo does that work, but it simply isn't worth doing for things like robot football or simple object manipulation. So you're simply not going to see a 1:1 alignment of 'pure' research teams and applications teams. That doesn't mean that the research work hasn't led to improvements in applications, though.