Hacker News Reader

Comment by pj_mukh | Hacker News Reader

Wondering if it would be better to abstract Vulcan away by Unreal, get even better photo-real quality.

I guess the better question is how much does photo-realism quality matter for this kind of sim2real work? A lot I would wager.

View on HN

Vulkan is a rendering technology, Unreal is an engine, that can render using Vulkan under the hood. You can absolutely do this in Unreal. I implemented this at Cruise using UE4 for integration testing, and it worked great for inference (we weren't doing training on sim at that point, but I was pushing for it! There was a paper out in 2018 or so that showed mixing in a bit of simulated data had an outside positive impact on the outputted model). There are companies out there right now doing this with even more modern renderers. I can't comment on how much the rendering realism gap matters here. I think there's some people out there using a variant of lower quality rendering + some kind of diffusion to get "better" images without having to do detailed modeling/lighting for their sims (fuzzy memory, I don't have a source on this).

Is there a good Unreal + Physics simulator OOB now that people can try?

I guess Isaac Sim is king?

Unreal will definitely get better results out of the box, but it's also possible to do photorealism with significantly less overhead (particularly UE shader compilation overhead) - useful for single purpose platforms. If you don't need to support lots of specific editor or game features, it may be a valuable investment.

UE is definitely used to obtain simulation data in other domains (this is coming from first hand experience in big tech), but usually through scripting UE handmade levels in python which also needed convoluted server systems at the time (hopefully this has gotten better now).

Reply on news.ycombinator.com