Hacker News Reader

Comment by vessenes | Hacker News Reader

This is.. super impressive. I'd like to know how large this model is. I note that the first thing they have it do is talk to agents who can control the world gen; geez - even robots get to play video games while we work.

That said; I cannot find any:

- architecture explanation

- code

- technical details

- API access information

Feels very DeepMind / 2015, and that's a bummer. I think the point of the "we have no moat" email has been taken to heart at Google, and they continue to be on the path of great demos, bleh product launches two years later, and no open access in the interim.

That said, just knowing this is possible - world navigation based on a photo and a text description with up to a minute of held context -- is amazing, and I believe will inspire some groups out there to put out open versions.

View on HN

We already knew it's possible from AI minecraft (https://oasis.decart.ai). This is just a more impressive version of that, trained on a wider range of games and with more context frames (Oasis has about a second of context, this one a minute). Even the architecture seems to be about the same.

Had they released this two months earlier it would have been incredibly impressive. Now it's still cool and inspiring, but no longer as ground breaking. It's the cooler version that doesn't come with a demo or any hope of actually trying it out.

And with the things we know from Oasis's demo, the agent-training use case the post tries to sell for Genie 2 is a hard sell. Any attempt to train an agent on such a world would likely look like an AI Minecraft speedrun: generate enough misleading context frames to trick the AI into generating what you want

This is far beyond Oasis. Oasis had approximately 0 continuity, and the generated world was a blurry mess. This on the other hand actually approaches usability.

I don't know what the pipeline looks for these, but I assume that's due to the costs associated with training and running. Oasis had a context of only a couple of frames, while this genie model apparently runs for a couple of minutes. I guess they have a couple tricks up their sleeve to optimize this though.

And it works on a wide variety of games, instead of just a single one with a relatively consistent art style. On the other hand, Oasis was realtime, while this one is offline; IMO getting the inference speed down was their most impressive feat, as even most decent video gen models are slower than that.

Oasis is playable so therefore:

1. Non-cherrypicked in its consistency (if you look at the demonstrations in the Oasis blog post you can find specific cases of consistency which is an anomaly rather than the norm)

2. Is live-inferenced at 20fps. If you use Runway v3 which is a comparably larger and higher quality model (resolution and consistency) it might take a minute or two generate 10 seconds of video.

3. Is served (relatively) reliably at consumer scale (with queues of 5-10k concurrent players) which means that in order to save on GPU cost, you increase batch size and decrease model size to “fit” more players in 1 GPU.

Here is a thread of videos from my tests of the recent DIAMOND model: https://x.com/chrisoffner3d/status/1845436198254227590

I really wonder how much more stable Genie 2 is.

That AI Minecraft feels like playing a dream, which is insanely cool.

Any estimates of how much one of these cost to generate and keep a minute of context?

Secondly, any estimate of how much the price could fall in 5-10 years?

Oasis (the Minecraft world model) can serve about 5 players on 8 H100 in real-time at 20fps in 360p. This is a much more capable model with two orders of magnitude more context. They pretty much say it can't be played real-time, which I read as they generate less than 15fps@240p on 8 GPUs. Probably why they talk so much about using it for AI training and evaluation rather than human use. There is a distilled version that works in real-time, but they don't show anything from that version (which is a statement in itself).

For reducing the price, ASICs like etched may be the way forwards [1]. The models will get bigger for a time, but there may be a lot of room for models that can exploit purpose-built hardware.

1: https://www.etched.com

> Probably why they talk so much about using it for AI training and evaluation rather than human use.

What would they do / how would they use this output to make a better AI?

Embodied cognition is a core theory for AGI; this would enable a vast array of bodies, environments, and situations, that high level of diversity can empower AI adaptability.

For a straightforward example, this could help Waymo rehearse driving in various cities and weather / traffic settings

Not meaning to pick at that example but a broader question the value of these, what use cases outside of games are they willing to let AI that is meant to interact with the real world be trained on AI synthetic data, that is like black box on black box, double the training and inference cost

Even in games I expect a game playing model to exploit glitches present in world building one

I think it's great that Google is researching these, but I can't see the return and if there is it is many steps away

I bet the military is keenly interested.

Hey! I'd love to know how this performs on 8xMI300x in comparison. Reach out to me?

The price of LLMs has fallen 1,000 times in the last year for the same quality tokens.

It's not clear if video models will follow the same trajectory.

I saw a demo of stable diffusion work so fast the images change as you type

They don't give much info on parameter count, etc so it's hard to say concretely: Oasis (AI Minecraft) apparently runs on a single H100 [1], but this is presumably much larger — both due to higher fidelity, and due to the 60s context window instead of 1s context window for Oasis. But in 5-10 years regardless of what it takes to run now, the price will drop massively, and my bet is this would be playable in real-time. Context length will be solvable simply by increased VRAM (i.e. an H200 has 141GB per GPU, vs 80GB for an H100). Although Google is probably running these on TPUs, TPUs should follow a similar trajectory.

In the intermediate term my guess is that this kind of world model will be useful for training 3D model generators, so that you can go from sketch -> running in-engine extremely quickly.

1: https://www.tweaktown.com/news/101466/oasis-ai-and-single-nv...

While this is impressive, yet still looks like a very early prototype. The overall nuance seems that it doesn't try to be a standalone product but a part of broader R&D projects toward general agents... I doubt if they even have any productionized modeling pipelines for this project yet and pretty sure that we won't have an open access anytime soon.

there are lots of 3D modelers spending hours on 3D worlds and assets to use in training, this seems to automate a lot of that work

GDM is a research lab. They are not set up for production. There are other teams in Alphabet doing productionization stuff.

This kind of demo is probably great for hiring top talents: come work here, we have the best models and you'll have your name on the best papers.

I asked this in a similar thread the other day but what is with this pattern as well exemplifies with the below quote

> This is.. super impressive. I'd like to know how large this model is. I note that the first thing they have it do is talk to agents who can control the world gen; geez - even robots get to play video games while we work. That said; I cannot find any:

> architecture explanation > code > technical details > API access information

Reply on news.ycombinator.com