Small neural network enables realistic rendering of woven fabrics in real-time
techxplore.com
techxplore.com
I think that the 60fps on a consumer grade card is the win, but that says a lot about the power of consumer grade cards in 2024.
Interesting to hear that, consumer market has been well past 30 fps for years, to my knowledge. For example:
- Phones are widely at 120 fps as of a couple years ago.
- Gamers have been targeting 240 fps for years and years, since 2018?
- On the low end, ex. Steam Deck requires you manually opt-in if you're happy with less than 60.
I'm curious for some groups you're aware of that commonly accept 30. (note: not a trick question, ex. we know movies are 24 fps, and professional 3D renderers are happy with not-real-time, but we're thinking about 3D rendering for games)
PS5 came out 4 years ago (November 2020) and has 4 games at 30 fps. Of 3140.[1]
PS5 and Series X both have an infamous problem with ports from older generations coming over at 30fps, because they were render-locked to that in the previous console generation, and no upgrade work was done during the port.
Beyond that, 30 fps comes about via opting into it via choosing raytracing at 4K.
It's a massive reach to describe 30 fps as commonly accepted. And that's without considering the context: it's being used to argue for the usefulness of a cloth simulation neural network that takes 18 ms to render on a 3090, not 2020 consoles.
Moreover, those screenshots aren't showing a game with some clothing simulation. They're showing a nearly-empty render except with some cloth simulation.
I think steelmanning is good and important for conversation, but its hard to swallow here. Blithe while taking too many liberties that are likely to shade rather than shed light.
[1] https://www.reddit.com/r/PS5/comments/1biskb4/there_are_3140...
Toy Story was rendered at 24 FPS and I don't know a single person who refused to accept it as-is. In fact, I hear more people complain when cinematics aren't run at a filmic framerate.
Consoles, hand helds usually aim for 30+
Real time rendering, video streaming etc, are 24-30.
If the nit is my phrasing, let me rephrase, the point is that it's usable in an interactive/real-time application. It's perhaps at the lower bar, but 30 fps is far from a rarity. Though not always ideal, it's certainly acceptable.
They're only briefly at 120fps during important things like full screen scrolling. Phones are battery powered and passively cooled, so rendering at 120fps would be very wasteful even if you can do it.
The important thing about 120fps is that a lot of things are factors of it; you can't play a 24fps movie on a 60fps screen.
> Many of these tools are now widely used by graphics designers, animated film creators and videogame developers worldwide.
perhaps if someone builds a liquid NN version, can reduce the size dramatically. but that has it's own problems, e.g., operations in that network class less supported.
> Woven fabric patterns […] tend to be regular and repetitive. Chen and his colleagues set out to leverage their repetitive nature
Is this “just” using the NN to approximately compress and decompress things? That doesn’t seem to explain the performance gains, unless some work can be done in the reduced space?
Possibly it's RAM rather than compute limited but still. You couldn't use that in a game. Existing techniques like PBD or verlet integration would be far more efficient.
Maybe on future devices with separate neural processor units this could be useful. But even then I bet those same chips could run standard real-time cloth algorithms really really fast too.
Edit: I downloaded their video and there's no movement at all, only still images. Maybe I misunderstood the purpose of the technique. But I re-read the article and it certainly seems like they're claiming to do full simulation, not just static scenes.
That scene with the sofa isn't even getting 60fps! It kinda feels like a disingenuous article after watching the video.
> Furthermore, our network can be applied to three typical types of woven fabrics once trained, but NeuMIP have to be trained per material which takes a long time
> Specifically, we introduce a simple encoder-decoder structure, where the encoder compresses the fabric pattern and other parameters into a material latent vector, and the decoder interprets the material latent vector with a spatial fusion component and a small angular decoder. Thanks to the lightweight encoder and decoder, our network is able to achieve real-time rendering and editing. Meanwhile, our network only occupies a small storage of 5 MB, even for scenes with multiple fabric materials.
Yeah. This is really just a simple solution that can be quickly trained and that compresses the problem space better at the cost of some fidelity.
You are trying to map inputs to outputs in an efficient way space-wise or time-wise. The NN is doing something which allows it to skip complex calculations at the expense of fidelity in a way which is not far from a more advanced look up table.
The fact that what it does is so opaque remains the main flaw of it.
That, and the fact that it takes rather a lot of numerical calculations to do it.
Like, actually really pseudo intelligent companions you can converse with. Not gonna happen in the foreseeable future, and the job markets are gonna be... Well... essentially gone once that exist, so it's probably for the best that this tech isn't actually anywhere close to AGI
It's already possible to play with actual other people, and quite popular too. Sure, those guys might not be very in-character or prioritise you having fun over them, but there's genuine intelligent interaction.
The other way in which games make us interact with other humans is by experiencing things designed by them. Story, dialogues, game design, atmosphere etc. I think we might see more ad-hoc generated world/story, but I think many people would want to experience the _same_ game, in some way, being social animals.
I'm not sure if and in what way generative AI or even AGI is going to disrupt the game industry. I can certainly picture the jobs of many people changing, from artists to voice actors.
But I'm not sure I see a holy gray here. Lots of interesting novelties that I'm sure will be tried, but not really a fundamental change to why we play games and consequently, what games are like.
Good point on the procedural aspect, Minecraft and similar games came to mind when I wrote my first post. I'd argue they are still fundamentally a designed, shareable experience, and that's my main point really. In Minecraft, you can't talk about the world much with someone else since it's generated, but the mechanics, materials, recipes etc. are the same, and provide exchange between humans playing the same game.
I believe we'll even see some successful AI heavy games in the near future, but I also believe we'll cycle back to games being, first and foremost, designed upfront. I might be wrong of course and we might see some kind of holodeck 100% custom game experience become dominant. It's just my hunch that we won't, even if it's feasible. Thinking as far back as I can, people always seem to have wanted two things from games: Fun, and connection to other people.
For me it’s either Stardew Valley or Paradox type games for my single player itch or in the past I’ve ventured off to MMORPGs like Eve or Wurm and play in “single player” mode.
MMORPGs have a greater sense of world change as players can carve mountains and build estates while slaying champions and such in Wurm online and you can stick to your plot of land and have nothing to do with it while enjoying the benefits. This doesn’t exist in single player games but it could and should.
Although I don’t agree AI will be taking jobs but it will just be another tool used in the industry. We need to embrace change and use these new tools to make better content and not just assume jobs will be lost.
From what people say about them Bethesda games try to do this, but it seems like the result is they're impossibly buggy because noone can test all of that.
Undertale is low tech, but does track player deaths and use them in the story.
That’d be great for an RPG game, but if I’m playing a action adventure game or a style fighter, I generally care about how cool it looks and how smooth it plays than I do about the intelligence of the side characters.
In all seriousness though, I think RTS games (I only do single player) would be more fun for me personally if I could leave the economy/build order stuff to a reasonable AI and focus instead on military unit control. I’m just not someone that can handle all of that at once.
I think the potential for an enemy tailored to challenging and occasionally surprising me without simply overwhelming me would also be great.
I really like the economy / base building part, but care less for unit control. I’m more of a macro player than micro.
I think a lot of players fall on one side or the other.
Weirdly enough, I’ve never played it
Just pick an RTS game without base building? They exist.
1. Industrial Annihilation: a dual-mode game where you can build a base or manage an army and the AI will happily handle the rest.
2. Battle Aces: unit-management distilled by almost entirely removing building from the game.
Imagine if you could describe what is happening on the screen, including the state of the world and everything, with a really long prompt. You dynamically alter that prompt continuously based on inputs received.
There will be a point in the future where the prompt will render with 99% consistency. You’d probably still need some sort of barebones state server for things like hitboxes and to prevent cumulative divergence.
Still a ways away though :)
Neural network is the big hammer.
If the neural network is allowed to approximate, then why not?