The Matrix: Infinite-Horizon World Generation with Real-Time Interaction | Hacker News Reader

The Matrix: Infinite-Horizon World Generation with Real-Time Interaction

thematrix1999.github.io

210 pointslnyan71 comments

What does "world" mean here? How does the spatiality fit into some latent space? Or what constitutes the "world"? If the answer is, there is none, the world is just frames of video and any consistency quickly blurs out after a few seconds. That's not a world generation, that's just generation of video frames following frames. Not that it isn't cool, but it has almost zero usability for generating a "world" simulation. The key to a realistic world is that you can reliably navigate it. Visit and revisit places. If you modify anything, those modifications are persisted. If you leave a room and re-enter it hours later, the base expectation is that the same objects are in that room.

Wouldn't a working approach be to just create a really low resolution 3D world in the traditional "3D game world" sense to get the spatial consistency. Then this crude map with attributes is fed into frame generation to create the resulting world? It wouldn't be infinite, but on the other hand no one has a need for an infinite world either. A spherical world solves the border issue pretty handily. As I understood it, there was some element of that in the new FS2024 (discussed yesterday on HN).

What you really want is a story telling engine informing the world model and the humans need to generate power.

It's basically a dream. Maybe that's what dreams are? World prediction engine running with no sensor input and memory in read only mode?

When you go to sleep, your brain stem disconnects from your body and your brain enters a feedback loop. The sensors are very much connected, just to whatever. Hence why you can grow dragon wings in your dreams and feel them. Memories can be made as well. I used to be really into lucid dreaming and time compression. My longest dream was nine years compressed into a 12-hour sleep period.

Where did you get the 'brain stem disconnects from your body'? Because thats not how it works in the brain.

We have the part which controls your muscles and we have another part which simulates the movement. Not executing on it has nothing to do with the brain stem disconnecting from the 'body'.

Its the same mechanism as you thinking about a movement but not doing it.

I believe it's a much more fundamental difference than just the distinction between ideating and acting.

Many people occasionally experience the transition between "conencted" and "disconnected" states as a sudden jerk or loud noise just at the moment of falling asleep.

Sleep paralysis is another "failure mode" of this mechanism that reveals what's going on. (I'm not sure if there is a reverse to it, i.e. whether sleepwalking could be explained as a drastic fail-open of the same mechanism).

I read a book, just a few weeks ago regarding this specific topic and my explanaition is directly from that book. We do have specific brain areas for this.

This sudden jerk you can read about it here: https://en.wikipedia.org/wiki/Hypnic_jerk and its not scienctificly clear why and how it works, the best assumption is that its a reflex.

You are not disconnecting your brain from anything.

Your brain disconnects: https://article.imrpress.com/bri/Landmark/articles/pdf/Landm... is probably the best paper I know of that describes the actual process going on in detail.

> My longest dream was nine years compressed into a 12-hour sleep period.

Does the brain change in response to that sleep period? Or is there no change because there's no new information input?

I dunno, I wasn't hooked up to any kind of measurement device to measure the changes in my brain. From a subjective point of view, I miss that place. I've been writing a book about my adventure there, off-and-on for years now. Maybe someday, I will finish it. If I could return, I'd do it in a heartbeat at any cost.

I _feel_ older than I am because there are a couple extra decades in my brain than in real life. Most of my time compression experiments were only a few months or weeks. That one long one changed me forever, and I've never done it on purpose since then.

I still have time compressed dreams from time to time, and when I wake up, two or three weeks have subjectively passed, but only a night has passed in the real world. There's a period of time, no more than 10-30 minutes, while the brain tries to reconcile two different and overlapping pasts. It can be a bit disorienting. My wife knows when I have these dreams because when I wake up, she says I look around surprised or confused to be there. The absolute worst is when you lay down to go to bed in the dream and wake up in the real world. Those will mess you up.

So, maybe my brain did change. Who knows? Maybe someone should study it.

This is maybe the coolest thing I've ever read on HN.

"Awoken By A Lamp" vibes, definitely:

https://www.reddit.com/r/Glitch_in_the_Matrix/comments/30t9k...

Thanks for share.

What like inception? Like you experienced 9 years and then woke up and went to work??

In this particular case, I was in the hospital after a major motorcycle accident. So, I didn't go anywhere. To be honest, after a couple of months in the dream, I had determined I had died and found the afterlife. I had never thought it was a dream.

The trick to time compression in dreams is two things:

- being able to generate false memories

- being able to skip the passage of time

It requires acknowledgment that there is no proof you existed 5 minutes ago, only your memories of existing (and surviving the existential crisis that may cause) matching up with the current perceived reality. So, to have a time-compressed dream means to simply 'skip ahead' for a period of time and have access to the memories in-between. This last part is the part that needed the most practice for me. I was able to skip ahead, but it took years before I'd be able to create false memories with coherency. These days, it isn't uncommon to have a dream with an entire lifetime of memories that aren't mine. Luckily, these are forgotten within seconds of waking up, making it easy enough to determine which of my dreamt experiences are fake and which are real.

On a normal night, you only have a couple of hours to dream (more or less depending on sleep deprivation and need for deeper sleep). So it works kinda like a movie that covers a greater period of time, skipping ahead to the interesting parts. Then the access to the memories in-between the interesting part to make decisions and sense out of what you are experiencing.

Not an expert, but I think procedurally generated terrain is generally fractal in nature, and is reproducible in that sense from a seed that is used in the generation. It is therefore recursive, as fractals are recursive.

A traditional neural network is a universal function approximator, however it is not recursive in nature, unless it is some sort of RNN. The transformer architecture, which this seems fairly similar to this one, is also not recursive in nature; although I believe, limited recursion can come about through CoT.

Therefore, I don't believe this could match the reproducibility, in an infinite sense, of a traditional procedural generator.

"Fractals are not self-similar"

https://m.youtube.com/watch?v=gB9n2gHsHN4

If you want to learn more about the fascinating world of fractals.

The parent comment didn't mention self-similarity; did you maybe intend to reply to another comment?

Procedurally generated terrain is whatever it wants to be. It isn't necessary for it to be fractal in any particular sense. Or even arguably all that useful, at least at the present time.

Luanti [1] (Minetest) has a fractal map generator ("mapgen"). You can test it for yourself. It's funny at first but becomes eventually boring.

Its other mapgens massively some kind of Perlin noise in various ways, so that you can have "realistic" landscapes (e.g. the Carpethian mapgen) or landscapes with impossible mountains and floating rocks sometimes (e.g. the V7 mapgen) that are good for fantasy/sci-fi worlds.

Noise is a pretty efficient way to fake complexity, and it's not a coincidence [2].

[1] https://www.luanti.org/ [2] https://en.wikipedia.org/wiki/Kolmogorov_complexity

"Wouldn't a working approach be to just create a really low resolution 3D world in the traditional "3D game world" sense to get the spatial consistency. Then this crude map with attributes is fed into frame generation to create the resulting world?"

That's Minecraft. It does work. And when you turn around, then turn back to where you were facing, you don't feel like you're in a fever dream, because the landscape hasn't completely shifted.

The tech is neat and I'm sure worth publishing. The rhetoric accompanying it is terribly overblown. Of course that's as likely to be their University PR department as the authors.

> Wouldn't a working approach be to just create a really low resolution 3D world in the traditional "3D game world" sense to get the spatial consistency. Then this crude map with attributes is fed into frame generation to create the resulting world?

If you go "ultra-low" resolution, that's basically Minecraft or Luanti (formerly Minetest). There some mods for Luanti that let one generate a less "squary" world by adding slopes, but that's still heavily polygonal.

> It wouldn't be infinite, but on the other hand no one has a need for an infinite world either.

Minecraft has an infinite world (with glitches when you go very far from origin due to floating point errors). Luanti's world is finite (around 64000 m^3 because 16 bits coordinates; players move at 2-4m/s usually), some people are working to make it infinite. In my opinion, you are right that nobody needs an infinite world; the argument for an infinite world is that a finite world can be a problem on popular servers but in my eyes servers where this is a problem don't use enough the available space (in particular the huge vertical space that can be used to create multiple worlds) or shoot themselves in the foot by providing players with ways to move fast (mainly transports); increasing the player's speed actually shrinks the world.

I think there is a conundrum from the players and gameplay perspective. On one hand, you want a lot of space, but on the other hand you want to be close to other players (for commerce, play together, etc.). It's not enough to have an infinite world, you also have to have the gameplay that goes with it.

Another thing to consider are the interaction of mobiles (NPCs, enemies) with the terrain. Because nobody likes an infinitely empty world either. I haven't played an AAA game recently, but a decade ago even with precomputed terrain, some mobiles would look silly sometimes with steep terrains.

I'm into VR and mixed reality, and I think this is headed to making the Holodeck real in an immersive way. That's the concept of the Matrix and what they are demoing, just in 2d.

I am guessing the main thing holding this stuff back in terms of fidelity and consistency or generalization is just compute. But the new techniques they have here have just dramatically lowered the compute costs and increased the generalization.

Maybe just something like the giant Cerebras SRAM chips will get to the next 10 X in scale that smooths this out and pushes it closer to Star Trek. Or maybe some new paradigm like memristors.

But I'm looking forward to within just a few years being able to put on some fairly comfortable mixed reality glasses and just asking for whatever or whoever I want to appear in my home (for example) according to my whim.

Or, train it on a lot of how-to videos such as cooking. It just materializes an example of someone showing you exactly what you need to do right in your kitchen.

Here's another crazy idea: train on videos and interactions with productivity applications rather than games. In the future, for small businesses, we skip having the AI generate source code and just describe how the application works. The data and program state are just stored in a giant context window, and the application functionality changes the instant you make a request.

I think we need devices that can produce gaussian splat videos to become as common as 2d cameras.

I think fever dream AI stuff like this would be nightmarishly terrifying in VR lol.

Still, would be super cool to try

I wish researchers would spend more time on using generative models to create level geometry, rather than trying to generate video from scratch. It would be both cheaper and more effective for stable gameplay.

There's so many researchers working in AI now that we can afford to explore all avenues. This probably isn't going to lead to a fully generative AI game engine, but I bet there'll be useful learnings along the way.

I think Ubisoft are leaders in that space. I saw algorithmic generation of New York style geometry for some behind the scenes video for The Division 2.

I wouldn’t be surprised if epic has similar.

This is the future I was trying to pitch in 2018 when we had built Ayvri and had every paraglider in the world, the world's largest ultramarathons, drone operators, and lots of other users of our real-world 3D environment.

Though we were using map tiles at the time, we were developing a model that took photos and a GPS track to add information that better matched environmental conditions (cloud, better lighting, etc).

People still ask me to open-source or give them our source code, but the code was acquired, so that isn't possible. But I do regularly say that if I were to rebuild Ayvri today, I'd do it as an interactive video rather than loading tiles.

Why would you want to generate all the pixels using this model instead of generating all the art, physics, and objects in the world using a game engine? The engine does so much of the physics and keeps everything stable for very cheap.

Generating coherent objects in an infinite world is probably harder.

Because you wouldn't have to wait for a human to write the engine and write the game

We generate 2d art, 3d models, some other in-world data, and soon animations at our game studio. The huge problems of generating pure pixels without a persistent game state behind it is illustrated well by the recent MinecraftAI craze (where it's a cool demo, but absolutely unplayable for anything other than novelty), and exacerbated on the dev side by missing out on tons of "free" stuff you get from existing engines that already do almost every kind of physics ops for you.

I didn't fully grok what this was about from the website. Though just last night I was talking to a friend about that quote from the Matrix that Morpheus tells Neo, so some nice synchronicity there. The sense I got from this is that they are developing a triple AAA type virtual world that can get generated on the fly based on text prompts? When the authors say frame level control do they mean that at any point, the next frame can be manipulated, either to be completely new or to influence the current story or context that is being played out?

I’m really excited for where this is going. From the demo videos, it seems to be a step up from Oasis, which itself came out only 2 weeks ago. I expect to see a lot of innovative use cases in this field

unreadable website

> Click to play

Clicking - nothing works.

Click to play [...the video]

No source, no playable demo, just promises of.

Could be total vaporware for all we know.

It is an ad, a statement of achievement in case someone else states it first, or what?

Seems like it would be better on Youtube, it really doesn't offer much of use right now.

Definitely used Cyberpunk2077 footage to train

"Welcome to the Matrix" with matrix-like rain seems like an invitation for Warner Bros to sue you into oblivion.

Prediction: in 20 years, I’m going to be reading about some dude who wrote a program to drive the car continuously until it ran into some surreal edge condition, and finally hit it. There will be a subculture of “matrix glitchers” who spend much of their time doing these kinds of experiments.

People have been doing that with Minecraft for over a decade. In the old days, once you got far away enough, the terrain generation would go haywire. Lots of videos from that time period of people exploring the "edge of the world".

Personally, these were the kind of glitches which made games feel magical and "real" to me as a kid. Being able to analyze a system by breaking it made it seem so much more tangible, like an actual place I had an NTSC-sized porthole into.

Cf. the "Minus World" in Super Mario Bros. for the NES.

https://en.wikipedia.org/wiki/Minus_World

Ha! I remember being either 5 or 6 when my uncle showed me Minus World and it blowing my mind. That might have actually been my first exposure to "backrooms" glitches like that. What an amazing glitch. It even worked on my combo Super Mario Bros / Duck Hunt cartridge

MissingNo. is another good example. I have fond memories spending untold hours in my favorite game engines trying to break free. The Jak and Daxter series were some of my favorite to break, due to the uniqueness and flexibility of the engine and the weird ways that the chunk loading system could be broken.

Ahh, "Mountain King" on the Atari 2600 was the game for me finding a cool bug. If you bounced just right, you'd soar over the mountain into the glitches far above. Games didn't crash, they just worked with what they had.

I didn't have Mountain King for my 2600 so I looked it up. What a neat glitch. Platformer glitches are fun, I really enjoyed breaking the early Sonic games for things like the Hyper Sonic glitch, or some of the map glitches.

I think this is one thing about Super Mario Bros. 3 that felt so magical to me. With the addition of the hidden whistles and intentional "glitches" like crouching for an extended time on a white platform, running behind map elements, etc. you felt like some kind of plane walker just bending time and space to your will. Fantastic implementation of a level skip mechanism for veteran players. It gave an already incredibly expansive game quite a lot of extra replay value, just like Minus World.

Thank you for the reminder of MissingNo! Takes me back to when I was a child and received a Gameboy Color without any games. I spent months just watching the start up animation on repeat before I got Pokémon yellow.

That is one of the saddest thing I've ever heard. Did your parents just not know it needed games, or was it a budget thing?

I was extremely poor growing up but I did get lucky and get a Gameboy Color for Christmas with a copy of Pokémon Gold at age 5, right before my guardians went insane and forbade any non-Christian media such as "Pocket Demons" or any fantasy content. That game expanded my mind so much, introduced me to a lot of things I'd never encountered before. It seemed so mysterious and huge, especially with the entire extra Kanto campaign. Still one of the greatest and most complete games ever made.

Far Lands or Bust is a great YouTube channel for this. He’s been walking in one direction in a Minecraft world since 2011.

"there's no bugs in there!" a chick with a star on the cheek probably.

That community already exists because the current version of these types of AI game engines are constantly running into a surreal edge condition since they don't track things consistently when they go off frame.

This is surely really cool. Just a bit sad that, as phrased by the authors, the "First Real-Time" virtual world created for the demo is a fat & fast SUV driving on virgin lands.

Why is that sad?

When you think of "The Matrix" - An infinitely generated and extremely complex world, I don't think most people would picture this, and especially not in a car like that. The car choice makes sense when you consider there aren't any roads at all! Of course, it is a very new technology, but then that does bring into question the paper author's frankly ambitious title choice.

Interestingly, this website does remind me of something I played a while back - check it out! [https://slowroads.io/]. It has roads!

What would you expect it to look like based on the title? (I don't understand the title and have no expectations)

Well, the Matrix movie showed a perfect depiction of the world stuck in 1999, with people given freedom to do as they wish (or what they thought was freedom). So... I guess more than just driving on barren plains haha.

Sigh, agreed!

Someone should ban "AI" articles on Hacker News.

But then there would be no articles!

Why? It's a mostly democratic news aggregator site without much editorial overview, if you don't like it, downvote / don't upvote it, or write a client that filters topics you don't like out based on some keywords. You didn't need to open this page and comment on it if you don't like it, nobody's making you read things you don't like.

I don't know if you're actually new here, or you've been reading HN for years, but your account is only 11 months old.

We talk about a lot of things here, but when we do talk about AI, we tend to prefer talk about things with code or papers.

This particular project has a paper. They're expecting to publish their code soon.

Here's the paper:

https://thematrix1999.github.io/article/the_matrix.pdf

You don't have to read it, but you may want to consider it if you want to learn something and/or contribute something meaningful to the conversation.

Yes, and that someone should be you. Be the change you want to be.

Check lobsters

Reply on news.ycombinator.com