Hacker News Reader

Comment by speps | Hacker News Reader

Note that the dataset from the video is called Matrix city. It's highly likely extracted from the Unreal Engine 5 Matrix demo released a few years ago. The views look very similar, so it's photorealistic but not from photos.

EDIT: here it is, and I was right! https://city-super.github.io/matrixcity/

View on HN

Epic acquired the photogrammetry company Quixel a while ago, so it's quite likely they used their photo-scanned asset library when building the Matrix city. Funnily that would mean the OP is doing reconstructions of reconstructions of real objects.

Or just rendering it mixed with some splats, we don’t know because they didn’t release their source code. I’m highly skeptical of their claims, their dataset, and the fact that it’s trivial to export it into some other viewer to fake it.

The Matrix Awakens demo had some of the assets released to the Unreal Engine marketplace as separate packs, [1]the map, [2]buildings, crowds, and vehicles.

The matrixcity map is different, but is somewhat similar to that of the map in Matrix Awakens. You can see from the design breakdown on [3] this page by a technical lead on the Matrix Awakens project.

edit. If you look further into the [4]github codebase, under the MatrixPlugin section it explicitly states that they used the city-sample project.

[1] https://www.unrealengine.com/marketplace/en-US/product/city-... [2] https://www.unrealengine.com/marketplace/en-US/learn/city-sa... [3] https://quentinmarmier.artstation.com/projects/xYeKNO [4] https://github.com/city-super/MatrixCity

Just want to add that using data from a game engine gives you perfect camera poses associated to each image. That makes training of nerfs and GS a little easier since there is no error from camera pose optimization. That's also the reason early Nerf papers used the famous yellow Lego excavator rendered with blender.

How so?

Through pixel perfect alignment? Resolution?

one of the fundamental problems in photogrammetry is determining the position of the camera in 3D space, with a game engine you just have a concrete value for your camera position, removing that entire problem. I don't know too much about photogrammetry but i'd imagine once your camera position is 100% accurate it's a lot easier to construct the point cloud accurately.

Replying to myself with a question, as someone could have the answer: Would it be possible to create the splats without the training phase? If we have a fully modelled scene in Unreal Engine for example (like Matrix city), you shouldn't need to spend all the time training to recreate the data...

Yes, and then it gets interesting to think about procedurally generated splats, such as spawning a randomized distribution of grass splats on a field for example

To me the big issue is image quality versus generative efficiency. If splats make rending complicated scenes efficient without requiring a lot of data/calculation "scaffolding" then you could do almost everything procedurally, maybe using AI models to fill in definitional detail.

Yes, it's possible to create gaussian splats from a mesh. See for example step 3 in SuGaR: https://imagine.enpc.fr/~guedona/sugar/

Are you referring to the gaussian splat rasterizer?

I'm referring to using the modeled scene to bind gaussian splats to an existing mesh.

> Binding New 3D Gaussians to the Mesh

> This binding strategy also makes possible the use of traditional mesh-editing tools for editing a Gaussian Splatting representation of a scene

I could be wrong, but being able to remove the step of estimating the camera position would save a large amount of time. You’re still going to need to train on the images to create the splats

> If we have a fully modelled scene in Unreal Engine for example...

No images involved, so no training required.

Of course! And this was done many times in the past, probably with better results than current deep learning based gaussian splatting where they use way too many splats to render a scene.

Basically the problem with sparse pictures and point clouds in general is their lack of topology and not precise spatial position. But when you already have the topology (eg with a mesh), you can extract (optimally) a set of points and compute the radius of the splats such that there are no holes in the final image (and their color). That is usually done with the curvature and the normal.

The 'optimally' part is difficult, an easier and faster approach is just to do a greedy pass to select good enough splats.

Not all of the videos are Matrix City, some are real places.

Reply on news.ycombinator.com