Hacker News Reader

Comment by airstrike | Hacker News Reader

Wow, these look amazing. I'm a layman, but I think this is what everyone's been thinking about ever since the first NeRF demos?

EDIT: I went looking for those threads and found my own comment wishing for this 5 years ago https://news.ycombinator.com/item?id=22642628

The next step is to automatically add "nodes" to the 3D images where the model can pivot, rotate and whatnot and then boom, you have on-demand animated, interactive content.

Feed it some childhood photos and recreate your memories. Add an audio sample from a loved one and have them speak to you. Drop into VR with noise-cancelling headphones for extra immersion. Coming soon! Click here to join the "Surrender Reality" waitlist

View on HN

>The next step is to automatically add "nodes" to the 3D images where the model can pivot, rotate and whatnot and then boom, you have on-demand animated, interactive content.

The next step is to generate models with higher quality mesh topology that allows animation and editing without breaking the mesh. I've done a lot of retopologizing and if I (or AI) were to rig these models as-is there would be all kinds of shading and deformation issues. Even without animating they are glaringly triangulated up close. But I suspect really high quality 3D asset generation is just around the corner because all you'd have to do is join up the approach seen here with AI quad re-meshing based on estimated direction fields and feature detection, which is also getting scarily good.

Anywhere you'd recommend a hobbyist can learn more about remeshing or feature detection?

https://blog.yarsalabs.com/blender-topology-guide/

https://cgcookie.com/posts/guide-to-clean-topology

This is a technical paper, but it has a quite conversational abstract and introduction that is easy enough to follow if you have some experience with mesh modelling: https://www.graphics.rwth-aachen.de/media/papers/337/learnin...

At this point maybe meshes are not the best representation for animation and editing. We can just use latents of neural networks

> The next step is to automatically add "nodes" to the 3D images where the model can pivot, rotate and whatnot and then boom, you have on-demand animated, interactive content

My gut says a 3D engine + this would be a superior solution to the current approach of rendering rasterized video directly from the latents (coincidentally, Sora got released today).

It may not be tractable to train a network to rig and animate meshes, as well as setting up an entire scene to be a "digital twin" of random videos, bit I imagine such a set up would have finer-grained control over the created video while keeping everything else in it the unchanged

> "The next step is to automatically add "nodes" to the 3D images where the model can pivot, rotate and whatnot and then boom, you have on-demand animated, interactive content."

Well not really sure what you're talking about here wrt nodes (adding in arbitrary rotation/zoom sounds great in theory if all you're looking for is a lazy susan or spinning exorcist heads), but the next steps will likely be more around ensuring sane symmetrical topologies, better UV maps, and automatically building rigging (FK/IK) to allow for easy animation.

I meant rigging, but I'm a layman so I don't know the terminology. But yes, symmetrical models with simpler meshes and better UV maps would definitely be needed to make it work as I'm imagining it

Mixamo is pretty close to this (Auto-rigging tool from Adobe). https://www.mixamo.com/#/

It's limited to mostly human shapes, but I've personally used it in combination with the 3d-pack in comfyUI to generate 3d models and rig them, starting from a text prompt.

Trellis looks like a more capable model generation tool than TripoSR and marching cubes, which is what I was doing in Comfy: https://github.com/flowtyone/ComfyUI-Flowty-TripoSR - It worked but models ended up having a slightly "melted wax" appearance.

This type of flow is definitely already here for low quality assets (think mobile games). I'm excited to go play around with Trellis, looks like a significant bump up in quality.

Super cool stuff! Thanks for sharing. It's exciting to see how fast this space is developing.

Hopefully one day we get some open source alternative to Mixamo that plays nicely with the rest of the open ecosystem.

I'm interested to see how this affects 3D artists in game development studios. Will those studios use these tools and keep their artists, allowing them to push out more and more content, faster, and easier, or just keep a bunch of artists around, drop the other 80% of them, and use the tools to _replace_ those artists?

Last time I looked at these the lighting was in the textures, also the meshes were asymmetrical and insane. Not usable by a game dev studio.

These models will get better. And in answer to the previous question, of course they will get rid of artists. They will keep just enough to do the work necessary with the help of generative models, and let go of the rest.

The rest of the artists are not dumb, and they have a lot of talent. I'm sure many of them will use the same models to come up with their own games.

How this whole thing will play out is anyone's guess. Long term, I'm hoping for new types of jobs to come into being. But who can say?

I hope they use it to create a bigger variety of assets. In lots of large games you start to notice where they've reused assets that could have used some more variation.

Studios don't have that many artists already - most of the "heavy lifting" is outsourced to asset production companies in China. I can see a future where these are replaced by AI and the main job of the in-house artists is to fix problems with the generated output.

> ... and then boom, you have on-demand animated, interactive content.

And in addition to that it's also useful for rending still pictures. 2D generated images by AI so far have incorrect lighting and many errors. Once it's a 3D scene rendered by something like Blender (which is free), it's going to look flawless: lighting is going to be correct (and configurable) and all the little details that are wrong are going to be easily fixed.

We already have insanely powerful tools and apparently from here it's only going to get way more powerful real quick.

As a newly minted 3d printer owner next step is accurate dimensions, material and nozzle diameter awareness ;) then some CAD-like support where you can specify constraints on… things?

This isn't anything parametric but I 3d printed a model from Trellis https://news.ycombinator.com/item?id=42375951

Reply on news.ycombinator.com