The Gates of Argonath in VR at 50 FPS on Unreal Engine
old.reddit.com
old.reddit.com
Yeah, it's "just" 50 fps, but ... would you just look at it!
4 years ago there was an equally fascinating "Paris apartment" VR demo, but the movement was limited and the amount of work that went into appears to have been orders of magnitude more - https://www.benoitdereau.com/
Go back 10 years and seeing this back then would've been absolutely bananas.
Makes you wonder what we'll have in 10 years from now.
Recent versions handle real-time global illumination (Lumen) and dynamically render geometry to insane levels of detail (nanite). Both of these are groundbreaking achievements. But having had to install this in a bunch of university computers I can tell you that it comes at a cost.
Interestingly, the most significant issue is the space these installations require. We have optimized our installation as much as we can, but we nonetheless had to introduce systems whereby local storage was wiped clean every two weeks.
So, it's kinda a weird spot to be. Not every scene was streaming 250mb/s of assets, but when it did the worse drives took a notable hit to performance.
That being said, compression would be an interesting avenue to explore for SD cards and SATA drives. The performance add can be pretty marginal, but sometimes that's just what these drives need.
Our lab computors have 500gb of space. Each time a new profile is activated by a student, that eats up around 5 to 10 gig. In addition… Video editing eats up storage space like there is no tomorrow (the Cache requirements of Resolve are another issue). Nanite supports scene with very dance meshes (downloaded directly from Unreals online store). These add up quickly in terms of file size, especially as each comes in several versions, with multiple levels of detail.
Ideally we would have liked to install more than one version of Unreal. Unreal updates regularly and files created with each version cannot be guaranteed to be opened by subsequent versions. However, this was one saving we made.
There are sample files which are actually very instructive but which we did not install.
All of these issues compounded and towards the end of sem we were having a tough time.
There's your problem. Development and content creation machines need far more disk than this. It doesn't have to be the OS drive; a data drive or two is usually fine, but 500GB in total is far too little.
This has been true since 1998.
It will also lead to questions about the copyright of the resulting models, probably somewhat dependant on how automated this process is.
I'm sure there will be no questions there.
The questions about the copyright of models like DALL-E, Stable Diffusion, etc. exist solely because it infringes on the rights of thousands of random artists all around the world. There is no one single entity with enough power to clearly oppose it so people making the models can get away with "questionable copyright", "legal gray area", etc. kind of talk.
Try the same with a model trained on a single movie created by a multinational with infinite money and all this will be resolved and defined real quick and strongly worded cease & desist letters sent out by end of work day.
Frodo: What?
Sam: If I take one more step, I'll be the farthest away from home I've ever been.
Popup: And you can do that for just $29.99 when you buy The Hobbits Shire Expansion DLC!
The movie making studios already own those 3d "digital assets" from their films. IDK if there is a real "free-for-all" right now, but the monetisation play has already been planned for.
Never underestimate the power of a sufficiently motivated rich individual to create legal hell (see Gawker). There are many very rich artists. There just needs to be an obvious large enough irritant which will happen as these generative models start getting used beyond toy amusements. The class action lawsuit against Copilot was filed just as pricing models started.
This is going to be an interesting area to watch. Use in commercial projects at your own risk of exposure to liability.
It really feels like the textile factory worker problem all over again. We can't let the big players be the benefactors every time.
That is impossible (assigning credit to training images from the output is arbitrary, not deterministic) and wouldn't help anyone (imagine Spotify except it pays you even less).
> It really feels like the textile factory worker problem all over again. We can't let the big players be the benefactors every time.
Industrialization is better for society because it produces more customer surplus. Even Marx thought the Luddites were wrong.
Also my point.
>Even Marx thought the Luddites were wrong.
Sure they are wrong in that they confuse what the problem is. If workers owned the factory they would be happy to automate the work. But their feeling of being left out has some validity because if they weren't the total value would improve.
The real issue is that the revenue brought back through automation doesn't benefit the worker. If it did, the automation would happen sooner because incentives would align better and the total value for society would thus likely improve faster.
But what if you use only three films? Or two?
Where are they saying that? Nobody thinks it's a gray area, they think it's explicitly legal and they're probably right too. Claiming it's a copyright violation is actually a rather weak case artists are just saying is obvious.
Microsoft has licensed DALL-E. They're not venturing into legal gray areas.
The argument I heard most often goes along the lines of: "people also learn from a lot of examples and other artists, and that is not copyrighted".
It is an interesting argument, but AI is not people and the argument could easily be the opposite (not taking sides here). They might think it is legal, but it is far from clear, IMHO.
IANAL, but it will be interesting to see if this is enforceable and by whom.
Their history as litigation defendants would suggest your optimism may perhaps be unfounded.
How does it infringe? Are your memories of a movie copyright infringement?
Note that this doesn't mean that Stable Diffusion (and friends) are copyright infringing just because they're trained on copyrighted material. My brain doesn't infringe the LOTR copyright because of my memories of the films. If I turn my experience of fantasy epics into a new novel with a different storyline than LOTR, that's not copyright infringement. That's creativity. But those same memories can likewise be used to make works that are infringing. The question of infringement isn't in the creative act, it's in the artifact. I think it's plausible (though not certain of course) that IP questions will be settled for AIs in the same way: it's not infringement to train your AI on copyrighted material. It is infringement to use your AI to generate works that a reasonable person would conclude are intended to replicate or imitate copyrighted material.
If that's the case, it should probably should be codified in law. Copyright has a lot of exceptions, and if the existence of AI models trained with copyrighted data are in public interest then it could be an exception too, but this should not be up to the courts.
I guess you can pirate Lord of the Rings on bitorrent all you want then? Cause computers are just a bunch of brains remembering things?
This is not a winning legal argument. However this is resolved, no court is going to treat an intentionally designed piece of computer software as a "brain". Or the machine storage of information as equivilent to a human being's "memory".
Obviously if OpenAI or other AI researchers are committing media piracy as part of their research, that is something they're legally liable for. It doesn't mean that their AI "remembering" it is inherently infringing.
I predict A.I. trained on LOTR will be found to be infringing because it's a derivative work unless the court determines that use of the original work constitutes a fair use.
My general point in my intentionally silly brain metaphor is brain metaphors are inapplicable and unconvincing.
Okay, but that's just a prediction, not an argument. I don't agree with the prediction. I think it makes some sense to think about the AI training data on analogy with memories, because they contain impressions of a thing without being able to reproduce it exactly. They can both be the source of both real creativity as well as the source of infringing works.
I don't see an argument for considering the training data produced by AI training a derivative work, at least not in any copyrightable sense. ChatGPT is "software"; the training data that it operates on is not "software" in the strictest sense and exists in an uncomfortable middle space at present with respect to copyright law. That's what the memory analogy is supposed to show: just like human memories aren't subject to copyright, AI training data that is similar in structure and capabilities to human memories is likewise not subject to copyright. More precisely, my point is that that's one entirely reasonable way courts could rule: AI training data represents "fair use" of its sources (provided they are legally obtained of course), but the products of an AI trained on those sources might not be.
It also has no precedent in it's favor. As far as I know no court has ever used "Is it like a brain?" to settle a copyright issue.
If you build your own replica of the bridge of the Enterprise, then yes.
This is pretty settled law.
As a society we ask people to branch out into various expertise… through no fault of their own maybe someone’s expertise is randomly obsoleted by a new technology. Just as copyright laws incentivize new creations, we as a society should incentivize people to embrace new disrupting technologies by safety-netting those displaced by them since it’s a gamble more or less of who is next to be displaced.
The point of copyright in general is to protect the economic value for the producer. Being so grabby about "IP rights" that they literally claw back the experience of consuming it means that the economic value of the IP drops to zero. It only has value in the context of being consumed, remembered, etc., and where the person purchasing the experience has confidence that such a thing will be allowed.
I suppose hypothetical future simulated humans could be shaped and molded until they give up some real value for an "experience" they won't be allowed to remember tomorrow, but by then they have passed so far beyond what is "human" that I have no further guesses what they may act like. (That's the true meaning of the "singularity" term, not any particular future, but the point at which our predictions are meaningless.)
Copyright also has nothing to do with the tool.
What a piece depicts and how it is used are much more important than how a piece was created. If I paint a picture by hand of a particular scene from a Disney movie, that's clearly a derived work and may be subject to copyright protection (it depends, there could be a fair use argument in specific situations, so even "it's copyright!" is not a cut-and-dried argument). If I paint a picture of Hercules in the style of Disney, that's trademark violation. But I can paint Hercules in any other style and Disney can't say shit (though they certainly try).
This is an age-old argument. Some artists think they should have a monopoly on certain ideas. As a society, we've already said no, you only get to own your expression of your ideas.
And as ideas go, "like a trending artist on ArtStation" isn't a particularly strong one.
Me, but I doubt it will be the same experience, if the scene is highly optimized by hand like the movie is vs. some AI generation.
I would take the opposite bet, I doubt most movies have complete enough 3D scene to be AI enhanced, let alone something good enough to walk around, you wouldn't model what you won't film after all and nobody wants to see hallucinated aliens in middle earth.
It can fill in the blanks (i.e. provide textures for areas that are invisible in the film).
Sure, like once or twice? I could see it getting old fast, though.
Spend a week preparing for a trip on how to navigate the city before actually going.
nb: you may need practice to navigate Shinjuku station
But I think the better is to train AI to generate scenes based on description such as "dogfight in P-51 with German ME-109 in WW2 for 15 mins" and after some computation the player gets to play this game in VR for 15 mins. The problem is how to make it possible at low cost as player will probably just pay a few bucks for such experience (but you can always resell the popular ones many times)
Obviously nobody is saying "VR is the same as real life" but surely to some degree it's the content of the experience that determines whether something is interesting. Some locations would be fascinating to walk around even if there's technically "nothing to do".
I'm not much of a gamer so my interest in VR has always slanted towards towards the more passive experiences. In fact quite often don't want my enjoyment spoiled by having to complete tasks or solve puzzles just to get around.
This is a good example of the fundamental limits of headset VR. Jumping into a lake will never feel real. Not by touch, obviously as you can’t get wet. Not by physical movement because you almost certainly can’t move downwards irl. Not by the weight of water around you or the buoyancy you expect to feel. Not by the rotation you would expect to have in water. Probably not by the stroke propeller movement or the correlation of your breath. And of course probably overly smooth movement to avoid motion sickness.
The difference between controlling a flying camera through water and believing you’re swimming is pretty big.
What you need is a home version of the 4DX cinema setup. If VR lasts another 5 years, I'd put money on someone making one of those in that timeframe.
We are basically at the Eliza stage of VR with a bunch of people pretending Eliza is not that far from passing the Turing Test.
Once we have virtual skin sensation and virtual physical movement there is no intellectual debate about any of this. Just instruction tutorials about how to have the experience because no marketing needed.
Once we have this I can think of far more interesting things than swimming around in the Avatar movie.
I very much doubt it. Within the Movie business, the ownership of "digital assets" such as 3d models, who keeps what, and what they can legally do with it, is the subject of long and detailed contracts between the film studio, VFX houses, subcontractors etc. Ownership is well-covered, if not exactly "clear" if you're not versed in the law regarding it.
A third party using any technique (AI included) to re-create a look-alike of such an asset, and to monetise this, would surely attract the "eye of Sauron" of the big studios to turn it's withering gaze upon them, and to send out a well-armed detachment of lawyers.
It will lead to questions - but those questions will be answered by the platform owners not the courts. The platform owners will just remove your models even in cases where a court would rule it fair use.
VR of real places is going to be huge - bigger than for imaginary places. My guess is that most VR worlds will be a combination.
Here's a question. What if you model the real world and an architect of a building. I know that copyright doesn't apply to buildings or other functional works. But again, this will be decided by the platform owners and not the courts.
Clearly there are good reasons to not have "platform owners" in the metaverse.
Me. It sounds boring. I couldn't give a damn about exploring someone else's fantasy world. This is a bit odd because I love open world video games. I guess I need a reason to spend time there.
Why hadn't it happened this year, or last? Nanite is great, but IMO doesnt change the equation that dramatically.
The cost for a studio to manually decimate game quality versions have been small enough that of the will had been there, we could have been doing it for years.
Having loosely followed this space, I think I might actually have some input!
First off, you need a VR-ready environment. Not many companies are making these, so the only people equipped to roll this out is Valve, Meta, and possibly Unreal/Epic. Valve already did this alongside Half Life Alyx; they released certain map portions so you could freely walk around them with friends without playing the game. Meta is getting there, but the combination of low-power hardware and headset attrition doesn't leave them with much of an audience. There's also Epic, but they don't really do much in this space yet either.
So, obviously a paradigm shift is needed. And frankly, I think Universal Scene Description will be that paradigm shift. TL-DR: Nvidia came up with a new standard for 3D environments that interoperates game technology with digital effects. In theory, you could take your favorite Andor or Rings of Power scenes, drag their file into another app, and just start walking around in-scene. You may end up adjusting a few things vis-a-vis scale, but once it reaches adoption I can see it being pretty easy to work with and popular.
Maybe 'next year' is a little close, but the stars have definitely aligned already.
WASM is neat but extremely finnecky at the moment. I've seen cool stuff done with it, but I'm not convinced it's funny production ready yet.
WebGPU is sorta in the same boat as WASM, if a little more realistic of a bet. People will want hardware-accelerated browser experiences soon, and WebGPU will probably do most of the lifting there.
The author predicted that players will play a game where they will be dropped into their favorite movies, and will be given points for exactly mimicking the dialogue, inflection, and movements of the characters.
Spoiler: the first one is War Games.
It's not as if things are going to move massively if you move your head a bit, you should be able to precalculate things to an extent.
For reference here is the scene from the movie https://www.youtube.com/watch?v=EN7kG7Ui7Bg
Have you tested the Valley of the Ancients demo on your computer? You will see what I mean.
This looks more like the actual in-game rendering: http://cdn.themis-media.com/media/global/images/library/deri...
What stands out to me is the low resolution textures everywhere, obvious seams between terrain and assets (runtime virtual texturing resolves that), light seams which have odd seams, and the entire ambience being vaguely similar to how light acts, but not really - primarily the lack of bounce lighting affecting anything. With the move to HDR, the more binary lighting of previous games looks weird too now.
Contrast that with the rich detail in lighting you can see in Fortnite's new update - https://i.redd.it/g2wqe10no44a1.jpg
Nanite, especially foliage, is something you have to experience for yourself though. Foliage pop-in not being so stark and alarming is huge for me, and the rich lighting capabilities improves immersion massively.
Regarding assets, high polygon assets have been created for ages too, but were never actually brought in to games. You bake those details into your normal maps, but with Nanite you can just bring them into games, and get the full detail when the player is up close to them. When games can fully embrace Lumen and Nanite, the developer iteration improvement will be huge.
What makes that shot look good are just the post effects, eg. depth of field and god rays.
edit: not to mention the lighting in the Xbox One screenshot was also pre-baked and probably manually tweaked, compared to real lighting updated in realtime in UE5.
Now please do the Ishtar Gate
The main focus of the game is you take an elevator to the top of a building, there is a plank an you jump off it.
I've found depending on the age of the participant the experience ranges from light fun to harrowing encounter with our own fears of death. I've seen people that, if it's their first time in VR, can take nearly an hour to finally do it.
0. https://www.oculus.com/experiences/quest/1642239225880682/