DSReality: Turns a DS game into a 3D image that floats above your controller | Hacker News Reader

DSReality: Turns a DS game into a 3D image that floats above your controller

github.com

241 pointsPhilpax83 comments

A few comments suggest watching the video, but since it's not quite obvious, the video is embedded in this tweet: https://twitter.com/zhuowei/status/1671532250020683787 / https://nitter.net/zhuowei/status/1671532250020683787

Thanks! This made absolutely no sense until the video

I still don't understand it after watching the video. Is the video recorded through a VR headset?

It's a recording of the phone screen running their app. The input of the app is the phone's camera, which they have pointed at their hands holding their Switch. The output of the app is the input plus the game overlaid on top of the Switch.

But is the idea that this would eventually run on a VR headset?

Oh hey! Nitter is working again! Good to know

What I find very cool is that it's extracting scenes from the running game and then projecting them into the VR space, allowing the scenes to be panned/rotated in a way that isn't possible in the original game.

If you haven't watched the video, check it out; my assumption from just reading explanations here was that it was just projecting a facade of the emulated device and its normal rendering of the game into VR.

The video is wild. Now we need an AI 3D infill to block in all the missing data at the edges of the view.

I actually originally assumed this was entirely done by some AI interpretation of the original 2D projection of the game scene, conjuring the entire new 3D scene. But of course, just extracting the vertex data from the emulator is way better.

I remember all the way back when this round of VR was new I was using a driver called VorpX to reinterpret non vr games into VR.

https://www.vorpx.com/

This Readme doesn't really explain anything. I don't understand what this is meant to do.

It's pretty clear from the title; it's an AR app that projects an emulator over top of your controller.

Sounds pointless if you're using a phone as your window into AR, but if you were to imagine this on a headset like a Quest or Vision Pro, it would basically "turn" any controller into a handheld retro console.

> It's pretty clear from the title; it's an AR app that projects an emulator over top of your controller.

It's not that clear at all if this is your take away -- its not just projecting a 2d surface emulators normally generate, but is instead taking that 3d data from the emulator and projecting _that_ into space. If you move the controller

> this on a headset like a Quest

None of the Quest give camera access to the apps, and there's currently no AR target support through the cameras. :(

The camera is only used to track the controller, the actual 3D data is pulled from the game files. The Quest API does let you layer elements onto the passthrough view[0], so it could work, the game would just hover in front of you instead of tracking the controller.

[0] https://developer.oculus.com/documentation/unity/unity-passt...

Then why isn't it running on Android? That would instantly work on the Quest. Why would someone voluntarily subject themselves to xcode? People need to treat themselves better.

ARkit presumably. Android just doesn't have an AR ecosystem that's as mature.

Which is interesting, because the release of ARKit doesn’t pre-date that of Android’s ARCore by even a whole year, and as I recall Google was dabbling in AR before Apple was. The difference is that ARKit has received consistent attention and iteration.

Well, there was a clear and specific product driving it in the Apple camp. Not just to get it ready for the dev kit for VisionOS devices but to get their dev teams a constant flow of useful data from any AR apps created in ARKit along the way that have likely fed into the design decisions and roadmap of the entire project.

I’m pretty sure ARKit has been 99.99% in existence as a way of getting as much real world data as possible for the headset design and ‘spacial computing platform’ project.

This sort of stuff also just seems to be way more in Apple’s wheelhouse. Apple is a device company, providing these kinds of developer libraries is, like, table stakes. For Google, supporting hardware isn’t in their DNA, right?

Just watch the linked video on twitter :)

There is a video link, but that link is not very easy to notice. I missed it when I first saw the readme

Ah, I see it now. Kind of a cool idea.

Repost here so more people can see it: https://twitter.com/zhuowei/status/1671532250020683787

I think this type of camera control has some big potential in the AR market. It's not fully submersive, but (1) (almost) no nausea because you're peaking into the world (2) screen everywhere (3) can be used in many genres ranging from casual to RPGs and actions (4) feel more natural to existing gamers.

Somehow this made me more convinced with the future of AR, but not VR.

Now that would be a compelling day one Vision Pro app.

It would be, except Apple don't allow emulators inside their walled gardens.

I can load up an emulator on my mac just fine. Whether the Vision Pro allows installs outside of the App Store is still unknown.

It's very unlikely Apple would allow side loading in MacOS if it was released today rather than 22 years ago.

"Walled garden" doesn't describe the Mac platform.

Don't use the marketing phrase 'walled gardens', its pro-corporate marketing jargon.

When has Apple ever referred to their ecosystem as a walled garden? I've only seen it used negatively.

To me "walled garden" sounds pretty negative. It's a wonderful place unavailable to pretty much everyone. Some pretty anti-democratic, capitalistic s*t. But I suppose it carries different connotations for different people.

What's your preferred terminology? "Monopolist prison" comes to mind as an alternative that's not possible to interpret positively.

Golden cage?

Don't call it an 'ecosystem': it's a zoo.

How do you figure? I've always seen walled garden used in a negative sense

This might work better on the XReal AR glasses that allows for multiple types of input sources.

This seems like an incredibly specific demo. Even they say that it doesn't work with games that aren't mario kart.

Unlike the NES which has only a few ways to layout the display, which the 3D NES emulator relies on to get good results, the DS can do arbitrary 3D stuff. It also has no API layer to reliably extract the camera information to generalize this kind of re-projection.

Neat looking trick but its sort of a dead end. It requires hand written code for any game you would want to do. I'm also curious about the rendering artifacts visible in the demo.

The DS is very much the wrong choice for this. They should try something similar with Dolphin and gamecube games, because the community there already does some stuff with camera manipulation.

There is a fork of Dolphin for VR. I played a few minutes of Zelda with my Oculus on several years ago. Pretty trippy to be transported to a game I enjoyed on a CRT 22 years ago, with a life-scale Link staring me in the face, 3 feet away.

This reminds me that there seem to be few "VR ports" or "cross platform VR releases" of normal games. I guess the gameplay of standard games doesn't necessarily work very well with the inherent limitations of VR. (E.g. nausea from moving around too much)

The nausea thing is quite a dealbreaker.

If someone had told me ~15years ago that one day I'd have the ability to play a VR version of Super Mario Galaxy, but that, when that time came, I wouldn't even bother to try and play it, I'd have (1) had a hard time believing it, and (2) assumed that future gaming must be godlike, in order to make me not bother with SMG VR.

But the reality of it, is so much more human and boring.

That is, I know from playing far-more-tame 3d motion games in VR, that it actively sucks as much as it rules. The eyes are saying Wow, and the rest of the body is saying No thanks, in a way not experienced before++. As such, when I heard of Dolphin VR last year, I didn't even dream of using it. Even the mere thought of trying to do SMG levels in VR is intimidating.

++I grew up loving roller coasters, and was the only local kid that never got sick on rough boats, but this, this is something else.

I still think the Metroid Prime games would work well in VR, giving you more depth perception, situational awareness, and ability to look around than the original 4:3 or 16:9 aspect ratios did.

I own a Quest 2 and I very rarely use it. I don't experience any nausea, it just hurts my face. :(

The NES also had many games with specific chips inside the cartridge, so I guess it is not too unusual for emulators to have game specific code.

There's way fewer NES mappers than there are DS games. It's 30ish vs hundreds. On an NES emulator, coding up just a few mapper chips gets you most of the library and probably 95% of the games worth playing, whereas with this, you need to write a "camera and 3D asset exporter" for each and every title you want to emulate. The work to give this effect to any game is nontrivial. Honestly interacting with the AR library and headset is probably the EASIEST part of this demo.

> It also has no API layer to reliably extract the camera information to generalize this kind of re-projection.

What do you mean?

Why do you have to be so negative about it? The README itself says it's a "prototype/concept". It's a neat hack, certainly the best thing I've seen on this website today. You make entirely valid technical points, but you don't have to present them as if you were scrutinizing someone's startup pitch. It's not that.

With ARKit you could probably get it to recognise and track the controller using it's object recognition, there's sample code here : https://developer.apple.com/documentation/arkit/arkit_in_ios... Then you wouldn't have to strap the phone to the controller

I'm not so sure about RealityKit, I found it doesn't have the flexibility and finer grained options of ARKit, it's annoying that Apple make 2 frameworks for AR but don't have the same features in both

This is extremely impressive! I was very confused until I saw the video, and then it all made sense.

Excellent work!

It would be cool to see something like this that could handle the 3D slider on the 3DS.

I've been thinking about making a stereogram viewer for my Hololens 1, I think you can make a left and right texture in Unity.

I feel like we are living in the start of a golden age of homemade hacks.

This makes me think, is there a Nintendo 3DS emulator for VR by any chance? Unfortunately non-VR screens which are able to display stereoscopic 3D are getting increasingly rare again.

I was hoping for some polarization and high frame rate trick to allow a passive, reflective screen to show the top screen.

not for DS/AR but for directx VR vorpx did it first.

https://www.vorpx.com/

I mean, that is pretty cool but... It seems kind of obvious that they could have just printed the QR code and taped the paper to the switch instead of using rubber bands to attach a phone to their switch...

having a spare phone is probably more common than a printer these days

Not sure why this is getting downvoted, I thought the same. Having worked on plenty of projects that rely on tracking a QR code, you'll get a much more stable experience scanning it on paper than an LCD screen.

Doesn't the phone control the accelerometer data?

Who needs an accelerometer when you're just interpreting polygon transform/coordinates in a camera view (the camera that recorded the video)

Oh, I thought those came from the switch. Then why not do away with the switch though. Ok, probably just very improvised, quick demo indeed.

SadlyItsBrad demoed something similar

This guy needs to cut his nails tho

Reply on news.ycombinator.com